x/wiki
1
0
Fork 0
mirror of https://github.com/waynesutton/markdown-site.git synced 2026-01-12 04:09:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: SEO/AEO/GEO improvements for AI crawlers and search engines

Browse Source
This commit is contained in:
Wayne Sutton
2025-12-22 00:07:24 -08:00
parent d623aed598
commit 94ba8fe184
12 changed files with 96 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
TASK.md
View File

@@ -4,10 +4,17 @@
## Current Status
v1.20.2 deployed. Write conflict prevention for heartbeat mutations.
v1.20.3 deployed. SEO/AEO/GEO improvements for AI crawlers and search engines.
## Completed
- [x] Raw markdown files now accessible to AI crawlers (ChatGPT, Perplexity)
- [x] Added /raw/ path bypass in botMeta edge function
- [x] Sitemap now includes static pages (about, docs, contact, etc.)
- [x] Security headers added to netlify.toml
- [x] Link header pointing to llms.txt for AI discovery
- [x] Preconnect hints for Convex backend
- [x] Fixed URL consistency in openapi.yaml and robots.txt
- [x] Write conflict prevention: increased dedup windows, added heartbeat jitter
- [x] Visitor map styling: removed box-shadow, increased land dot contrast and opacity
- [x] Real-time visitor map on stats page showing live visitor locations

21
changelog.md
View File

@@ -4,6 +4,27 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
## [1.20.3] - 2025-12-21
### Fixed
- Raw markdown files now accessible to AI crawlers (ChatGPT, Perplexity)
- Added `/raw/` path bypass in botMeta edge function
- AI services were receiving HTML instead of markdown content
### Added
- SEO and AEO improvements
- Sitemap now includes static pages (about, docs, contact, etc.)
- Security headers: X-Frame-Options, X-Content-Type-Options, X-XSS-Protection, Referrer-Policy
- Link header pointing to llms.txt for AI discovery
- Raw markdown files served with proper Content-Type and CORS headers
- Preconnect hints for Convex backend (faster API calls)
### Changed
- Fixed URL consistency: openapi.yaml and robots.txt now use www.markdown.fast
## [1.20.2] - 2025-12-21
### Fixed

15
content/pages/changelog-page.md
View File

@@ -7,6 +7,21 @@ order: 5
All notable changes to this project.
## v1.20.3
Released December 21, 2025
**SEO, AEO, and GEO improvements**
- Raw markdown files now accessible to AI crawlers (ChatGPT, Perplexity)
- Added `/raw/` path bypass in botMeta edge function so AI services receive markdown, not HTML
- Sitemap now includes static pages (about, docs, contact, etc.)
- Security headers: X-Frame-Options, X-Content-Type-Options, X-XSS-Protection, Referrer-Policy
- Link header pointing to llms.txt for AI discovery
- Raw markdown files served with proper Content-Type and CORS headers
- Preconnect hints for Convex backend (faster API calls)
- Fixed URL consistency: openapi.yaml and robots.txt now use www.markdown.fast
## v1.20.2
Released December 21, 2025

11
convex/http.ts
View File

@@ -23,12 +23,13 @@ http.route({
handler: rssFullFeed,
});
// Sitemap.xml endpoint for search engines
// Sitemap.xml endpoint for search engines (includes posts and pages)
http.route({
path: "/sitemap.xml",
method: "GET",
handler: httpAction(async (ctx) => {
const posts = await ctx.runQuery(api.posts.getAllPosts);
const pages = await ctx.runQuery(api.pages.getAllPages);
const urls = [
// Homepage
@@ -44,6 +45,14 @@ http.route({
<lastmod>${post.date}</lastmod>
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url>`,
),
// All pages
...pages.map(
(page) => ` <url>
<loc>${SITE_URL}/${page.slug}</loc>
<changefreq>monthly</changefreq>
<priority>0.7</priority>
</url>`,
),
];

4
index.html
View File

@@ -3,6 +3,10 @@
<head>
<meta charset="UTF-8" />
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<!-- Preconnect for faster API calls -->
<link rel="preconnect" href="https://convex.cloud" crossorigin />
<link rel="dns-prefetch" href="https://convex.cloud" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<!-- SEO Meta Tags -->

18
netlify.toml
View File

@@ -49,6 +49,24 @@
path = "/*"
function = "botMeta"
# Security and SEO headers
[[headers]]
for = "/*"
[headers.values]
X-Frame-Options = "DENY"
X-Content-Type-Options = "nosniff"
X-XSS-Protection = "1; mode=block"
Referrer-Policy = "strict-origin-when-cross-origin"
Link = "</llms.txt>; rel=\"author\""
# Raw markdown files with proper content-type
[[headers]]
for = "/raw/*.md"
[headers.values]
Content-Type = "text/markdown; charset=utf-8"
Access-Control-Allow-Origin = "*"
X-Robots-Tag = "noindex"
[context.production.environment]
NODE_ENV = "production"

5
netlify/edge-functions/botMeta.ts
View File

@@ -42,12 +42,13 @@ export default async function handler(
// Only intercept post pages for bots
const pathParts = url.pathname.split("/").filter(Boolean);
// Skip if it's the home page, static assets, or API routes
// Skip if it's the home page, static assets, API routes, or raw markdown files
if (
pathParts.length === 0 ||
pathParts[0].includes(".") ||
pathParts[0] === "api" ||
pathParts[0] === "_next"
pathParts[0] === "_next" ||
pathParts[0] === "raw"
) {
return context.next();
}

BIN
public/images/122.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 222 KiB

After

Width:  |  Height:  |  Size: 216 KiB

BIN
public/images/v17.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 560 KiB

After

Width:  |  Height:  |  Size: 562 KiB

2
public/openapi.yaml
View File

@@ -9,7 +9,7 @@ info:
url: https://github.com/waynesutton/markdown-site
servers:
- url: https://markdown.fast
- url: https://www.markdown.fast
description: Production server
paths:

15
public/raw/changelog.md
View File

@@ -7,6 +7,21 @@ Date: 2025-12-22
All notable changes to this project.
## v1.20.3
Released December 21, 2025
**SEO, AEO, and GEO improvements**
- Raw markdown files now accessible to AI crawlers (ChatGPT, Perplexity)
- Added `/raw/` path bypass in botMeta edge function so AI services receive markdown, not HTML
- Sitemap now includes static pages (about, docs, contact, etc.)
- Security headers: X-Frame-Options, X-Content-Type-Options, X-XSS-Protection, Referrer-Policy
- Link header pointing to llms.txt for AI discovery
- Raw markdown files served with proper Content-Type and CORS headers
- Preconnect hints for Convex backend (faster API calls)
- Fixed URL consistency: openapi.yaml and robots.txt now use www.markdown.fast
## v1.20.2
Released December 21, 2025

2
public/robots.txt
View File

@@ -5,7 +5,7 @@ User-agent: *
Allow: /
# Sitemaps
Sitemap: https://markdown.fast/sitemap.xml
Sitemap: https://www.markdown.fast/sitemap.xml
# AI and LLM crawlers
User-agent: GPTBot