Run RAG Check

Run RAG Check helps you find retrieval problems in a documentation page before those problems reach an AI or RAG pipeline.

It takes the same AI-readable text shown in AI View, splits it into retrieval-sized chunks, generates test questions, and uses a built-in keyword retriever to see whether those questions can find the right content.

You get chunk-level findings and retrieval scores that help you identify problems such as:

• Content that is difficult to retrieve with likely user questions
• Chunks that don't make sense when retrieved on their own
• Headings that don't provide enough context
• Tables or code blocks that lose meaning when extracted
• Broken links that can stop an AI agent from following a documentation path
• Sections that are too long or too fragmented for effective retrieval

When to use Run RAG Check

Use AI View and Run RAG Check together:

1. Open AI View to see the exact text an AI pipeline receives.
2. Fix formatting or content problems you find there.
3. Run Run RAG Check to see how that content behaves when split into chunks and retrieved.
4. Use the findings to improve the page.
5. Run the check again to verify the changes.

Enable Run RAG Check

Run RAG Check is enabled by default during local development.

To configure it explicitly, add a ragCheck block under ai in camelmind.config.ts:

const config: CamelMindConfig = {
  // ...
  ai: {
    ragCheck: {
      enabled: true,
      localOnly: true,
      roles: [],
      defaultChunkSize: 500,
      defaultChunkOverlap: 80,
      maxGeneratedQuestions: 12,
    },
  },
}

Configuration fields

Requirements

Run RAG Check is available only when all of the following conditions are met:

• The dev server is running with NODE_ENV=development, or CAMELMIND_AUTHOR_TOOLS=true is set.
• The request's Host header is localhost, 127.0.0.1, or [::1], on any port.
• OFFLINE_MODE is set to false.
• ai.ragCheck.enabled is set to true.
• If roles contains values, the current session has at least one of those roles.

When these requirements aren't met, Run RAG Check is hidden from the toolbar and its route returns 404. This means a production reader does not receive a permission error or other indication that the author tool exists.

Run a check

The Run RAG Check screen has three panes:

• AI-readable text and chunks — See how your page is divided into retrieval chunks.
• Controls and summary — Configure chunking, run the check, and review RAG readiness scores and test queries.
• Findings — Review issues and recommended fixes.

Fixing common findings

Categories shared with AI View (long-paragraph, context-less-code, context-less-table, weak-alt-text, positional-reference, opaque-jsx, weak-heading-context) use the same fixes described on the AI View page.

Run RAG Check across every doc

You can run a site-wide diagnostic scan across every page in your navigation tree to find low-scoring docs in a single pass:

npx tsx scripts/rag-check-report.ts [threshold]

The rag-check-report.ts script runs the same chunking, question generation, retrieval, and scoring pipeline as the interactive tool against every document in nav.yml. It sorts the results by overall score and outputs pages scoring at or below the specified threshold (default 89).

For each flagged document, the report displays the overall score, the two lowest-scoring dimensions, and up to five findings:

--- /features/example-doc (Example Doc) — overall 72 — weakest: crossLinkQuality=40, retrievability=58
    [blocking] dead-link: ...
    [warning] weak-heading-context: ...
    ...and 3 more findings

Related tools

• AI View — preview the raw text a RAG pipeline ingests, without running chunking or retrieval.
• LLMs.txt — the underlying AI-readable output both tools are built on.