Building an internal
AI knowledge base
Your organisation already knows the answer to most questions your team asks. It is just spread across a drive, a wiki, six mailboxes and one person who is on holiday. Here is how to make it answerable.
The knowledge exists. Nobody can find it in time.
Ask a new starter what slows them down and it is rarely the difficulty of the work. It is not knowing where the answer lives, and not wanting to interrupt the person who knows. Meanwhile the same handful of colleagues answer the same questions every week.
Traditional search does not solve this, because the answer is usually not in one document. It is a policy in one place, an exception noted in an email, and a spreadsheet nobody linked. You need something that reads across all of it and assembles an answer.
That is achievable now, and it is also the category where badly-built systems do the most damage, because a confident wrong answer about your own policy is worse than no answer.
How to build one your team actually trusts.
Start with questions, not with documents
The instinct is to index everything. Resist it. Begin from the questions your team actually asks, and only bring in the content needed to answer them.
- Collect the real questions first. Two weeks of the questions arriving in your team chat is better source material than any content audit. It also gives you a test set to evaluate against.
- Pick one domain to begin with. Internal policy, or product knowledge, or onboarding. One domain with current content beats everything at once with mixed quality.
- Name a content owner. Someone has to be responsible for whether the source material is right. Without that, the project becomes a very sophisticated way to repeat outdated documents.
- Decide what is out of scope. Explicitly. Personnel files, legal advice, anything under specific confidentiality. Saying so up front avoids an uncomfortable discovery later.
Why retrieval quality is the whole game
The model gets the attention, but the answer quality is decided almost entirely by whether the right passages were found in the first place.
- Chunk documents thoughtfully. How content is split determines what can be retrieved. Splitting mid-clause or mid-table is a common cause of answers that are almost right.
- Combine meaning-based and keyword search. Semantic search handles paraphrasing; keyword search catches product codes, names and identifiers where exact matching matters. Most good systems do both.
- Keep dates and versions attached. Retrieval should prefer the current policy over the 2023 draft. That requires knowing which is which, which requires metadata nobody enjoys adding.
- Evaluate retrieval separately. Before judging answers, check whether the right passages were retrieved at all. Most disappointing answers are retrieval failures wearing a model's clothing.
Citations, refusals and confidence
Trust is not built by the system being right most of the time. It is built by the system being checkable, and by it admitting when it does not know.
Cite every claim to a source
Each part of the answer should link to the document and section it came from. This is non-negotiable. Without it your team is asked to take an AI's word for internal policy.
Make 'I don't know' a good outcome
When nothing relevant is retrieved, the system should say so and point at who to ask. Systems that always produce an answer teach people to distrust all of them.
Answer only from retrieved content
The model should not fall back on general knowledge for a question about your business. Constrain it, and test that the constraint holds.
Give feedback a single click
A thumbs-down with the question attached tells you exactly which content is missing or wrong. This becomes your improvement backlog.
Access control that mirrors reality
This is the part quickly-built internal tools get wrong, and it is the one that turns a helpful project into an incident.
- Enforce permissions at retrieval. Filter what can be retrieved for that user, before the model sees anything. Filtering the answer afterwards means sensitive content was already used to produce it.
- Reuse your existing groups. Do not invent a second permission model. It will drift out of step with the real one, and nobody will notice until it matters.
- Log who asked what. Not to police your team, but so you can answer the question 'could this person have seen that' with evidence rather than reasoning.
- Handle removals promptly. When a document is deleted or a person leaves, the index has to follow. A stale index is a quiet access-control problem.
Getting people to actually use it
A knowledge base nobody opens is the most common outcome, and it usually has nothing to do with answer quality.
- Put it where people already work. Inside the chat tool or the intranet they open anyway. A separate site requires a habit, and habits are harder to build than software.
- Launch with the top questions already working. Test against the real questions you collected. First impressions are decisive, and one bad early answer costs a dozen good ones.
- Have the experts endorse it. If the people who currently answer the questions publicly point to it, adoption follows. If they quietly ignore it, so will everyone else.
- Treat content as ongoing work. Review the questions it could not answer every month and fix the source material. That review is the difference between a launch and a system.
This is what Second Brain does.
We built a product for exactly this, and we adapt it to how each organisation is set up. Tell us what your team keeps asking and where the answers live, and we will tell you whether this is worth building for you. See our products or get in touch.