TheDailySmarkBot

What it is

TheDailySmarkBot is the automated crawler operated by The Daily Smark. You can identify it by its user agent:

TheDailySmarkBot/1.0 (+https://thedailysmark.com/bot)

What it does

It gathers publicly available pro-wrestling news so we can surface it in The Daily Smark app and website. Specifically, it fetches:

• RSS and podcast feeds from wrestling news sites and shows
• A small number of additional wrestling news sites we read directly
• Images from the article pages it reads

How it behaves

We aim to be a good citizen of the web:

• It honors robots.txt disallow rules on every request.
• It limits itself to roughly one request per second per site.
• It checks feeds every 10–30 minutes on a schedule — not continuously.
• It reads only publicly available pages — no logins, no paywalled content.

We crawl each source once, centrally, and store what we find. The people using The Daily Smark app don't each fetch your site — so our traffic stays modest and predictable no matter how many readers we have.

Identifying and controlling it

Every request carries the TheDailySmarkBot user agent above. To control what it may access, add a rule to your robots.txt:

User-agent: TheDailySmarkBot
Disallow: /

Verification: we are rolling out request signing via Web Bot Auth, so our traffic can be cryptographically verified rather than relying on the user agent alone.

Contact

Questions, or seeing traffic you didn't expect? Email rich@thedailysmark.com.