x/wiki
1
0
Fork 0
mirror of https://github.com/waynesutton/markdown-site.git synced 2026-01-12 12:19:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: Add semantic search with vector embeddings

Browse Source 
Add vector-based semantic search to complement keyword search.
  Users can toggle between "Keyword" and "Semantic" modes in the
  search modal (Cmd+K, then Tab to switch).

  Semantic search:
  - Uses OpenAI text-embedding-ada-002 (1536 dimensions)
  - Finds content by meaning, not exact words
  - Shows similarity scores as percentages
  - ~300ms latency, ~$0.0001/query
  - Graceful fallback if OPENAI_API_KEY not set

  New files:
  - convex/embeddings.ts - Embedding generation actions
  - convex/embeddingsQueries.ts - Queries/mutations for embeddings
  - convex/semanticSearch.ts - Vector search action
  - convex/semanticSearchQueries.ts - Result hydration queries
  - content/pages/docs-search.md - Keyword search docs
  - content/pages/docs-semantic-search.md - Semantic search docs

  Changes:
  - convex/schema.ts: Add embedding field and by_embedding vectorIndex
  - SearchModal.tsx: Add mode toggle (TextAa/Brain icons)
  - sync-posts.ts: Generate embeddings after content sync
  - global.css: Search mode toggle styles

  Documentation updated:
  - changelog.md, TASK.md, files.md, about.md, home.md

  Configuration:
  npx convex env set OPENAI_API_KEY sk-your-key

  Generated with [Claude Code](https://claude.com/claude-code)

  Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

  Status: Ready to commit. All semantic search files are staged. The TypeScript warnings are pre-existing (unused variables) and don't affect the build.
This commit is contained in:
Wayne Sutton
2026-01-05 18:30:48 -08:00
parent 83411ec1b2
commit 5a8df46681
58 changed files with 7024 additions and 2527 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
scripts/sync-posts.ts
View File

@@ -375,6 +375,24 @@ async function syncPosts() {
}
}
// Generate embeddings for semantic search (if OPENAI_API_KEY is configured)
console.log("\nGenerating embeddings for semantic search...");
try {
const embeddingResult = await client.action(
api.embeddings.generateMissingEmbeddings,
{}
);
if (embeddingResult.skipped) {
console.log(" Skipped: OPENAI_API_KEY not configured");
} else {
console.log(` Posts: ${embeddingResult.postsProcessed} embeddings generated`);
console.log(` Pages: ${embeddingResult.pagesProcessed} embeddings generated`);
}
} catch (error) {
// Non-fatal - continue even if embedding generation fails
console.log(" Warning: Could not generate embeddings:", error);
}
// Generate static raw markdown files in public/raw/
generateRawMarkdownFiles(posts, pages);
}