# robots.txt — Digital Faith multi-tenant site (Diocese of Kentucky) # # Public content (including news and events) is allowed for every crawler — # deliberately including AI assistants (GPTBot, OAI-SearchBot, ClaudeBot, # PerplexityBot, Google-Extended, Applebot-Extended, etc.), which # increasingly drive discovery. News/event articles, listing # pagination (?page=) and the past-events archive (?past=true) are all # crawlable. Do NOT add per-bot User-agent groups here: under RFC 9309 a # crawler obeys only its most specific group and would then IGNORE the # disallow rules below. # # AI reading list (curated wiki map): https://episcopalky.org/llms.txt # # Admin, search, and expensive/duplicate query-string views are disallowed # to keep crawl budget focused on content pages and to protect server load. # The same public pages are served under both /digital_faith/ and # /digital_features/, so every rule is emitted for both prefixes. User-agent: * Allow: / Disallow: /digital_faith/management/ Disallow: /digital_faith/search/ Disallow: /digital_faith/people/ Disallow: /digital_faith/parishes/ Disallow: /digital_faith/people.pdf Disallow: /digital_faith/calendar Disallow: /digital_faith/events/calendar Disallow: /digital_faith/campaigns/*?* Disallow: /digital_faith/dfcfiles/*?* Disallow: /digital_features/management/ Disallow: /digital_features/search/ Disallow: /digital_features/people/ Disallow: /digital_features/parishes/ Disallow: /digital_features/people.pdf Disallow: /digital_features/calendar Disallow: /digital_features/events/calendar Disallow: /digital_features/campaigns/*?* Disallow: /digital_features/dfcfiles/*?* Disallow: /*?*global=true Disallow: /*.dat$ Disallow: /*.php$ Crawl-delay: 1 Sitemap: https://episcopalky.org/sitemap.xml Sitemap: https://episcopalky.org/news-sitemap.xml