WPBuildAI-NewsBot

WPBuildAI-NewsBot/1.0 (+https://news-production-2090.up.railway.app/bot)

What it does

• Reads a fixed list of public feeds, APIs and pages, published here. It does not discover new sites by following links.
• Honours robots.txt, including Crawl-delay, and caches it for a day per host.
• Makes at most one request a second to any host, uses conditional requests (ETag, Last-Modified), and stops at 3 MB per page.
• Stores titles, text and links for editorial use. It never stores images, and nothing it reads is republished in full: the letter links to the source.

Keeping it out

Add this to your robots.txt and the bot stops within a day:

User-agent: WPBuildAI-NewsBot
Disallow: /

Questions go to hello@wpbuildai.com.