About our crawler

If you found this page from a User-Agent in your server logs, this is us. Throughball is a college-soccer recruiting tool for families. It reads a small number of public web pages, almost all on college athletics sites, so a family can see current coaching staff without re-checking dozens of sites by hand.

What we read

Public college athletics pages, mainly coaching-staff and staff-directory pages for the college soccer programs in our catalog. To find a program's staff page, the daily sync may also open a few pages a web search returned for that program; these can include other pages, such as a team roster or a page on another site, and we keep only staff pages. From time to time we also read the NCAA's public member directory, and the index pages of the NAIA's public member and sport directories, for facts about an institution, such as its city, state and athletics website address; no person's data is read there. When a family asks us to research a school or prepare for a call, we may also read pages a web search returned for that school. A program's own camp page, when its staff page links to one on the same athletics site: we read the camp dates, location and price the school publishes there, and nothing about any person. A camp page on another site (a registration vendor) is kept as a link only and is not read. We never try to reach anything behind a login, and we never submit forms.

When

A roster sync, once a day. A check of one school's staff page when that school is added to a family's list: when the family adds it, when a coach's email to the family adds it, or when our staff add it on the family's behalf. Now and then, a check of one program's staff page that our staff start by hand. From time to time, and before a launch, our staff may also start that same check over the programs in our catalog whose staff page we have not found, each from a page address our staff located by web search, and may run it again; the pass reads at most a few pages a minute, one request per site every 30 seconds, and keeps only staff pages of tracked programs. Research, only when a family asks for it: a job runs every minute to carry out research families have queued, and when nothing is queued it makes no request to any school. A program's own camp page is read with the daily sync, at most a few pages a day, and in a one-off pass over the programs in our catalog that our staff start by hand. From time to time we run a one-off pass over the athletics sites in our catalog, or over the NCAA or NAIA directory, to refresh facts about institutions; it is paced the same way as everything else here, and it is not run for any one family.

How to identify us

Our page requests send:

What we commit to

We read and honour your robots.txt. Before we request a page from your host, we read your robots.txt and follow Disallow for the group that matches our User-Agent (or *). If there is no robots.txt (404 or 410), we take that as permission. If we cannot get a readable file, we request nothing from your host. That covers a server error, no answer, and a 401, 403, 429 or any other 4xx. The only request we make without first reading a robots.txt is the request for robots.txt itself. If your server redirects that request, we follow the redirect wherever it points, as the robots.txt standard (RFC 9309) asks — unless it points at a private or internal network address, which we do not follow — and apply the rules we find there to your site.

We space our requests. Each part of Throughball that reads your site leaves at least 30 seconds between its requests to the same host name. If your robots.txt sets a longer Crawl-delay, it leaves that long instead, once we have read it.

We identify ourselves. Every request we make for a page on a site we do not own carries the User-Agent above, with a link to this page.

If you ask us to stop, we stop. A Disallow for our User-Agent in your robots.txt stops our page requests to your site, without contacting us. It does not stop the requests we make while reading a robots.txt file — yours, or another site's that redirects to or through your site — because that is how we read the rules. Or email hello@trythroughball.com with the host name or the specific URLs. We switch your site off in our own records, and from then on our jobs stop requesting pages from it, except as noted under “Where we fall short today”.

Where we fall short today

Each of these is an exception to the commitments above. We will update this page as each one closes.

Our search provider's request. When a family's research needs a page, we first ask our search provider, Tavily, to read it. That request comes from Tavily's servers under Tavily's own identity, not ours, and we do not space it. Before asking, we check your robots.txt for our User-Agent, and our own switch, for the address we hand over, and we ask for nothing if either refuses. Tavily follows any redirect on its own side.

If you are listed

If you are a coach or staff member listed in our catalog, you can ask us to correct your entry, remove your email address, or remove your entry. Email legal@trythroughball.com with your name, institution and program. We act within 10 business days. A removal takes you out of the catalog families see. Our automated updates will not add you or your address back, including under a different spelling of your name. To make that possible we keep a record of your request — your name, role, institution and program, the date you asked and our reference for it — and use it for nothing else. It does not change a family's own record of messages it has already exchanged with you. Throughball never emails coaches; families write from their own accounts.

Anything else, such as a page we misread, a correction, or a question about what we stored: hello@trythroughball.com.