TheDailySmarkBot

What it is

TheDailySmarkBot is the automated crawler I run for The Daily Smark. You can identify it by its user agent:

TheDailySmarkBot/1.0 (+https://thedailysmark.com/bot)

What it does

It gathers publicly available pro-wrestling news so I can surface it in The Daily Smark app and website. Specifically, it fetches:

• RSS and podcast feeds from wrestling news sites and shows
• A small number of additional wrestling news sites I read directly
• Images from the article pages it reads

How it behaves

I try to be a good citizen of the web:

• It honors robots.txt disallow rules on every request.
• It limits itself to roughly one request per second per site.
• It checks feeds every 10 to 30 minutes on a schedule, not continuously.
• It reads only publicly available pages. No logins, no paywalled content.

I crawl each source once, centrally, and store what I find. The people using The Daily Smark app don't each fetch your site, so my traffic stays modest and predictable no matter how many readers I have.

Identifying and controlling it

Every request carries the TheDailySmarkBot user agent above. To control what it may access, add a rule to your robots.txt:

User-agent: TheDailySmarkBot
Disallow: /

Verification: I'm rolling out request signing via Web Bot Auth, so my traffic can be cryptographically verified rather than relying on the user agent alone.

Contact

Questions, or seeing traffic you didn't expect? Email rich@thedailysmark.com.