mirror of
https://github.com/waynesutton/markdown-site.git
synced 2026-01-11 20:08:57 +00:00
462729de58
Update version to 1.0.0 across package.json and changelog. Configure netlify.toml with Convex deployment URL (agreeable-trout-200.convex.site). Verify TypeScript type-safety for src and convex directories. Confirm Netlify build passes with SPA 404 fallback configured. Update TASK.md with deployment steps and files.md with complete file structure.
34 lines
455 B
Plaintext
34 lines
455 B
Plaintext
# robots.txt for Wayne Sutton's Blog
|
|
# https://www.robotstxt.org/
|
|
|
|
User-agent: *
|
|
Allow: /
|
|
|
|
# Sitemaps
|
|
Sitemap: https://your-blog.netlify.app/sitemap.xml
|
|
|
|
# AI and LLM crawlers
|
|
User-agent: GPTBot
|
|
Allow: /
|
|
|
|
User-agent: ChatGPT-User
|
|
Allow: /
|
|
|
|
User-agent: Claude-Web
|
|
Allow: /
|
|
|
|
User-agent: anthropic-ai
|
|
Allow: /
|
|
|
|
User-agent: Google-Extended
|
|
Allow: /
|
|
|
|
User-agent: PerplexityBot
|
|
Allow: /
|
|
|
|
User-agent: Applebot-Extended
|
|
Allow: /
|
|
|
|
# Cache directive
|
|
Crawl-delay: 1
|