x/wiki
1
0
Fork 0
mirror of https://github.com/waynesutton/markdown-site.git synced 2026-01-12 04:09:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: SEO/AEO/GEO improvements for AI crawlers and search engines

Browse Source
This commit is contained in:
Wayne Sutton
2025-12-22 00:07:24 -08:00
parent d623aed598
commit 94ba8fe184
12 changed files with 96 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
TASK.md
View File

@@ -4,10 +4,17 @@
## Current Status ## Current Status
v1.20.2 deployed. Write conflict prevention for heartbeat mutations. v1.20.3 deployed. SEO/AEO/GEO improvements for AI crawlers and search engines.
## Completed ## Completed
- [x] Raw markdown files now accessible to AI crawlers (ChatGPT, Perplexity)
- [x] Added /raw/ path bypass in botMeta edge function
- [x] Sitemap now includes static pages (about, docs, contact, etc.)
- [x] Security headers added to netlify.toml
- [x] Link header pointing to llms.txt for AI discovery
- [x] Preconnect hints for Convex backend
- [x] Fixed URL consistency in openapi.yaml and robots.txt
- [x] Write conflict prevention: increased dedup windows, added heartbeat jitter - [x] Write conflict prevention: increased dedup windows, added heartbeat jitter
- [x] Visitor map styling: removed box-shadow, increased land dot contrast and opacity - [x] Visitor map styling: removed box-shadow, increased land dot contrast and opacity
- [x] Real-time visitor map on stats page showing live visitor locations - [x] Real-time visitor map on stats page showing live visitor locations

21
changelog.md
View File

@@ -4,6 +4,27 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
## [1.20.3] - 2025-12-21
### Fixed
- Raw markdown files now accessible to AI crawlers (ChatGPT, Perplexity)
- Added `/raw/` path bypass in botMeta edge function
- AI services were receiving HTML instead of markdown content
### Added
- SEO and AEO improvements
- Sitemap now includes static pages (about, docs, contact, etc.)
- Security headers: X-Frame-Options, X-Content-Type-Options, X-XSS-Protection, Referrer-Policy
- Link header pointing to llms.txt for AI discovery
- Raw markdown files served with proper Content-Type and CORS headers
- Preconnect hints for Convex backend (faster API calls)
### Changed
- Fixed URL consistency: openapi.yaml and robots.txt now use www.markdown.fast
## [1.20.2] - 2025-12-21 ## [1.20.2] - 2025-12-21
### Fixed ### Fixed

15
content/pages/changelog-page.md
View File

@@ -7,6 +7,21 @@ order: 5
All notable changes to this project. All notable changes to this project.
## v1.20.3
Released December 21, 2025
**SEO, AEO, and GEO improvements**
- Raw markdown files now accessible to AI crawlers (ChatGPT, Perplexity)
- Added `/raw/` path bypass in botMeta edge function so AI services receive markdown, not HTML
- Sitemap now includes static pages (about, docs, contact, etc.)
- Security headers: X-Frame-Options, X-Content-Type-Options, X-XSS-Protection, Referrer-Policy
- Link header pointing to llms.txt for AI discovery
- Raw markdown files served with proper Content-Type and CORS headers
- Preconnect hints for Convex backend (faster API calls)
- Fixed URL consistency: openapi.yaml and robots.txt now use www.markdown.fast
## v1.20.2 ## v1.20.2
Released December 21, 2025 Released December 21, 2025

11
convex/http.ts
View File

@@ -23,12 +23,13 @@ http.route({
handler: rssFullFeed, handler: rssFullFeed,
}); });
// Sitemap.xml endpoint for search engines // Sitemap.xml endpoint for search engines (includes posts and pages)
http.route({ http.route({
path: "/sitemap.xml", path: "/sitemap.xml",
method: "GET", method: "GET",
handler: httpAction(async (ctx) => { handler: httpAction(async (ctx) => {
const posts = await ctx.runQuery(api.posts.getAllPosts); const posts = await ctx.runQuery(api.posts.getAllPosts);
const pages = await ctx.runQuery(api.pages.getAllPages);
const urls = [ const urls = [
// Homepage // Homepage
@@ -44,6 +45,14 @@ http.route({
<lastmod>${post.date}</lastmod> <lastmod>${post.date}</lastmod>
<changefreq>monthly</changefreq> <changefreq>monthly</changefreq>
<priority>0.8</priority> <priority>0.8</priority>
</url>`,
),
// All pages
...pages.map(
(page) => ` <url>
<loc>${SITE_URL}/${page.slug}</loc>
<changefreq>monthly</changefreq>
<priority>0.7</priority>
</url>`, </url>`,
), ),
]; ];

4
index.html
View File

@@ -3,6 +3,10 @@
<head> <head>
<meta charset="UTF-8" /> <meta charset="UTF-8" />
<link rel="icon" type="image/svg+xml" href="/favicon.svg" /> <link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<!-- Preconnect for faster API calls -->
<link rel="preconnect" href="https://convex.cloud" crossorigin />
<link rel="dns-prefetch" href="https://convex.cloud" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" />
<!-- SEO Meta Tags --> <!-- SEO Meta Tags -->

18
netlify.toml
View File

@@ -49,6 +49,24 @@
path = "/*" path = "/*"
function = "botMeta" function = "botMeta"
# Security and SEO headers
[[headers]]
for = "/*"
[headers.values]
X-Frame-Options = "DENY"
X-Content-Type-Options = "nosniff"
X-XSS-Protection = "1; mode=block"
Referrer-Policy = "strict-origin-when-cross-origin"
Link = "</llms.txt>; rel=\"author\""
# Raw markdown files with proper content-type
[[headers]]
for = "/raw/*.md"
[headers.values]
Content-Type = "text/markdown; charset=utf-8"
Access-Control-Allow-Origin = "*"
X-Robots-Tag = "noindex"
[context.production.environment] [context.production.environment]
NODE_ENV = "production" NODE_ENV = "production"

5
netlify/edge-functions/botMeta.ts
View File

@@ -42,12 +42,13 @@ export default async function handler(
// Only intercept post pages for bots // Only intercept post pages for bots
const pathParts = url.pathname.split("/").filter(Boolean); const pathParts = url.pathname.split("/").filter(Boolean);
// Skip if it's the home page, static assets, or API routes // Skip if it's the home page, static assets, API routes, or raw markdown files
if ( if (
pathParts.length === 0 || pathParts.length === 0 ||
pathParts[0].includes(".") || pathParts[0].includes(".") ||
pathParts[0] === "api" || pathParts[0] === "api" ||
pathParts[0] === "_next" pathParts[0] === "_next" ||
pathParts[0] === "raw"
) { ) {
return context.next(); return context.next();
} }

BIN
public/images/122.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 222 KiB

After

Width:  |  Height:  |  Size: 216 KiB

BIN
public/images/v17.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 560 KiB

After

Width:  |  Height:  |  Size: 562 KiB

2
public/openapi.yaml
View File

@@ -9,7 +9,7 @@ info:
url: https://github.com/waynesutton/markdown-site url: https://github.com/waynesutton/markdown-site
servers: servers:
- url: https://markdown.fast - url: https://www.markdown.fast
description: Production server description: Production server
paths: paths:

15
public/raw/changelog.md
View File

@@ -7,6 +7,21 @@ Date: 2025-12-22
All notable changes to this project. All notable changes to this project.
## v1.20.3
Released December 21, 2025
**SEO, AEO, and GEO improvements**
- Raw markdown files now accessible to AI crawlers (ChatGPT, Perplexity)
- Added `/raw/` path bypass in botMeta edge function so AI services receive markdown, not HTML
- Sitemap now includes static pages (about, docs, contact, etc.)
- Security headers: X-Frame-Options, X-Content-Type-Options, X-XSS-Protection, Referrer-Policy
- Link header pointing to llms.txt for AI discovery
- Raw markdown files served with proper Content-Type and CORS headers
- Preconnect hints for Convex backend (faster API calls)
- Fixed URL consistency: openapi.yaml and robots.txt now use www.markdown.fast
## v1.20.2 ## v1.20.2
Released December 21, 2025 Released December 21, 2025

2
public/robots.txt
View File

@@ -5,7 +5,7 @@ User-agent: *
Allow: / Allow: /
# Sitemaps # Sitemaps
Sitemap: https://markdown.fast/sitemap.xml Sitemap: https://www.markdown.fast/sitemap.xml
# AI and LLM crawlers # AI and LLM crawlers
User-agent: GPTBot User-agent: GPTBot