Knowledge
What the agent is allowed to answer from. Everything else it says is either a clarifying question or an admission that it does not know.
Source types
| Type | Use it for | Notes |
|---|---|---|
SITEMAP | A whole documentation site | Fetches the sitemap, then every URL in it. The fastest way to start. |
URL | One page | Re-crawled on demand from the dashboard. |
FILE | A PDF, Markdown or text export | Useful for policies that live outside your site. |
SNIPPET | A short answer with nowhere else to live | Pasted straight in. Ideal for the thing everyone asks that nobody documented. |
Lifecycle
A source moves through PENDING → CRAWLING → EMBEDDING → READY, or lands on FAILED with the error kept on the row. Only chunks belonging to a READY source are retrievable, which is why a source added seconds ago will not answer anything yet.
Re-syncing re-fetches and re-embeds. Deleting a source removes its chunks in the same operation — there is no window where the dashboard shows a source gone but the agent still quotes it.
Chunking
Documents are split on structure first — headings and sections — and only then on length, targeting roughly 600 tokens per chunk with about 15% overlap between neighbours.
- Structure first so a chunk is a coherent idea rather than an arbitrary window. Each chunk keeps the heading it came from, which is what a citation points at.
- Overlap so a sentence spanning a boundary is not lost to both sides.
Retrieval
Two searches run against your chunks and their results are fused:
- Vector search catches paraphrase — “my parcel is late” finding a page titled “Delivery delays”.
- Full-text search catches exact tokens — order numbers, SKUs, error codes, the things embeddings blur.
Fused candidates go to a cross-encoder reranker, which scores each one against the query properly rather than by vector distance. Only the top few survive into the prompt, and only if they clear the confidence threshold.
Writing docs an agent can use
Retrieval quality is set by your corpus. These are the changes that move the number most, in our experience:
- One question per heading. A chunk that answers exactly one thing retrieves cleanly. A 4,000-word page covering nine topics retrieves badly for all nine.
- Say the answer in the first sentence. Rerankers and readers both reward it.
- Use your customers’ words. If they say “my parcel is stuck”, that phrase belongs on the page — not only “shipment exception handling”.
- Delete the stale page. Two contradictory documents produce a confidently wrong answer roughly half the time.
Content gaps
Every time the agent abstains, the query is logged. The Analytics page clusters those into content gaps ranked by how often they recur — a prioritised list of the articles you are missing, written from real customer language rather than guesses. Fixing the top three is usually worth more than any amount of tuning.
Citations
Answers carry the heading of the chunk they came from, shown under the message in the widget and linked in the dashboard. If an answer has no citation, it was not grounded — and validation should have caught it. See The turn pipeline.