How to Train an AI Agent on Your Company Data

To train an AI agent on your company data, you almost never retrain the model. You give a capable off-the-shelf model three things: a short, current set of written policies it always reads; a searchable library of your documents it can pull the right passage from when a question comes up (called retrieval, or RAG); and live connections to your systems so it looks up customer, order and stock data at the moment it needs them. Then you test it against real past questions and fix the sources, not the model. For most businesses this takes two to six weeks and costs far less than fine-tuning. This guide covers each step, what to leave out, and the mistakes that make agents confidently wrong.

This post is part of our guide to rolling out agents, which starts at wizeb.com/blog/how-to-implement-ai-agents-in-business. If you have not yet picked the job your agent will do, read wizeb.com/blog/how-to-build-an-ai-agent-for-business first: what the agent needs to know depends entirely on the job.

The Problem: "We Uploaded Our Drive and It Still Gets Things Wrong"

The most common request we hear from owners is some version of "connect it to all our files so it knows our business". The result is predictable. The agent quotes a returns policy from 2022, mixes up two price lists, or tells a customer a product is in stock because a PDF said so last spring. The model is not the problem. It is doing exactly what it was asked: reading whatever it finds and answering with confidence. A shared drive is a record of everything your company ever wrote, including the drafts, the superseded versions and the one-off exceptions. An agent needs the current truth, and only the part relevant to its job.

Fine-Tuning vs RAG vs Live Lookups: What "Training" Really Means

There are three ways to get company knowledge into an agent, and they solve different problems.

• Fine-tuning changes the model's weights using examples. It is good at teaching a style, a format or a narrow classification task. It is poor at teaching facts: the knowledge is frozen on the day you train, cannot be traced back to a source, and must be redone every time a price or policy changes. Very few business agents need it.
• Retrieval (RAG) stores your documents in a searchable index. When a question arrives, the agent finds the few most relevant passages and answers from them, citing where they came from. Update the document and the agent's answer changes the same day. This is the right home for policies, product information, procedures and FAQs.
• Live lookups (tools) let the agent query your CRM, ERP, helpdesk or database through an API at the moment it needs a fact. Anything that changes daily, such as balances, order status, stock, prices or appointments, belongs here. Copying it into a document index guarantees it will be out of date.

A well-built agent uses all three layers in proportion: a page or two of core instructions it always sees, retrieval for the reference library, and tools for live records. If you want the broader picture of how agents differ from knowledge chatbots that only do the middle layer, see wizeb.com/blog/ai-agent-vs-chatbot-vs-automation.

How to Train an AI Agent on Company Data: 6 Steps

Step 1: List the questions the job actually involves

Pull 100 to 300 real questions or cases from the job: support tickets, sales emails, internal Slack requests, whatever the agent will handle. Group them. You will usually find that 15 to 25 topics cover 80% of the volume. That list, not your folder structure, decides which knowledge the agent needs. It also becomes your test set in step 6.

Step 2: Find the single source of truth for each topic

For each topic, find the one document or system that is currently correct, and the person who owns it. Where two documents disagree, settle it before the agent ever sees them. Where the answer only lives in someone's head, write it down; these unwritten rules are where most agent mistakes come from. Expect this step to take longer than the technical build. It is also the step that pays off for your staff, not just the agent.

Step 3: Clean and structure the documents

Agents read best from short documents with clear headings, one topic per section, and dates and owners at the top. Convert scanned PDFs into real text and tables, remove drafts and old versions, and split 80-page manuals into sections a search can find. Tag each document with who may see it, which region or product it applies to, and when it was last reviewed. If the source material is mostly invoices, contracts or forms, extracting it into structured data first works better than indexing raw scans; that is what wizeb.com/services/document-ai is for.

Step 4: Build the retrieval index

Load the cleaned documents into a search index that combines meaning-based (vector) search with plain keyword search, because product codes, policy numbers and names are often missed by vector search alone. Keep the metadata from step 3 so the agent can filter by region, product or date. Set it up so that updating a source document re-indexes it automatically; an index that needs a developer to refresh will be stale within a month. Require the agent to cite the passage it used, so a reviewer can check any answer in seconds.

Step 5: Connect live systems and enforce permissions

Give the agent tools to look up the live records its job needs, scoped to read-only unless the job requires changes. Permissions must follow the user: an agent answering a customer should only retrieve that customer's records, and an internal agent should only show an employee what that employee could already open. Enforce this in the retrieval and tool layer, not with an instruction in the prompt. Our governance guide covers the controls in detail at wizeb.com/blog/ai-agent-security-guide.

Step 6: Test against real questions, then keep it current

Run the agent on your test set from step 1 and score every answer: correct, incomplete, wrong, or should have said "I don't know". When an answer is wrong, trace it. Usually the right passage was missing, outdated or not found, and the fix is in the source or the index rather than the prompt. Rerun the full set after every change. After launch, give each source an owner and a review date, and send every answer users flag as wrong back to that owner. Knowledge upkeep is a few hours a week, and it is the difference between an agent people trust and one they quietly stop using.

What Not to Feed Your Agent

• Whole shared drives, email archives or chat history "for context". They contain contradictions, personal data and confidential threads the agent will happily quote.
• Superseded versions of policies and price lists. Archive them outside the index.
• Live data copied into documents: stock levels, balances, order status. Use a tool lookup instead.
• Anything regulated (health, payment card or HR records) unless the job requires it and access is filtered per user and logged.

How Long Does It Take and What Does It Cost?

For one job with a few hundred documents and one or two systems, expect two to six weeks, with roughly half the time spent on steps 1 to 3. Build costs typically sit in the low-to-mid five figures depending on how messy the sources are and how many systems need connecting. Running costs are modest: the search index and model usage usually come to a few hundred dollars a month at small-business volumes. The ongoing cost people forget is the owner time to keep sources current; budget it from day one.

Case Study: An Agent That Answers From a Distributor's Real Catalogue

A 45-person industrial parts distributor wanted an agent to answer trade customers' technical and availability questions by email. Their first attempt, a chatbot pointed at the shared drive, gave wrong lead times and quoted discontinued parts, and sales staff stopped trusting it within two weeks.

We started with 220 past customer emails, which fell into 18 topics. Of 3,400 files on the drive, fewer than 300 were current and relevant; the rest were drafts, old catalogues and supplier correspondence. The team merged three conflicting returns documents into one and wrote down the substitution rules that only two senior staff knew. Product specifications went into a retrieval index tagged by product line; stock, pricing and lead times came from a live ERP lookup filtered to the customer's account and price tier. After testing against the 220 emails and two weeks of draft-and-approve mode, the agent answered 64% of inbound technical questions without edits, cited the source sheet on every answer, and first-response time fell from most of a day to under ten minutes. The cleaned-up catalogue also cut onboarding time for new sales hires.

How Wizeb Trains Agents on Your Data

Wizeb treats knowledge as part of the build, not an upload step. We map the questions the job involves, help your team pick and clean the sources of truth, build a retrieval index with permissions and automatic refresh, connect live systems through scoped tools, and test against your real history before launch. You keep the index, the test set and the documentation. See how we build at wizeb.com/services/ai-agents, and if your knowledge is locked in scanned documents and forms, start with wizeb.com/services/document-ai.