Below the waterline: what starts after the demo
Month 0 is the best month: the demo works, the team is excited and the business case looks solid. Then the work that no tutorial mentions begins: sources that change, embeddings that need refreshing, permissions that shift, and answers that slowly get worse without anyone noticing.
Chunking is the silent failure mode here: cut your documents up slightly wrong and the system gives confident answers based on half the context. You do not discover that during the build, but months later, in an answer that was just a little bit off. So when you build it yourself, do not just budget for the build sprint, but for ongoing engineering hours and infrastructure costs for as long as the system runs.
- Months 1-3Scheduling re-indexing and embedding refreshes
- Months 3-6Keeping rights and permissions in line with every change
- Months 6-9Evaluation: are the answers still right?
- Months 9-12Drift: sources have changed, answers are out of date
- OngoingEngineering hours and infrastructure costs
- Set up: structure, permissions and folders are in place before anything goes in.
- Populate: curate existing knowledge before adding it, rather than importing everything at once.
- Validate: an adversarial agent tests it with hundreds of edge-case questions until the answers hold up.
- Maintain: review flows and automatic updates keep the brain up to date.
Four steps, then a fixed monthly fee. Maintenance is our job.