Fine-tuning vs RAG vs prompting: what should your team use?

People often hear “we should fine-tune it” in the same meeting as “we should connect it to our documents” and “we just need a better prompt.” These are three different tools for three different problems, and they are not equal in cost. The right way to choose is to start with the cheapest and move up only when it falls short. This page explains each, compares them, and gives a simple order to try them in. For deeper definitions see fine-tuning, RAG and prompt engineering.

The short comparison

	Prompting	RAG (your documents)	Fine-tuning
What you do	Write clear instructions and examples	Let the AI look up your files before answering	Retrain the model on example input and output pairs
Solves	Tone, format, steps, most everyday tasks	“Answer from our policies, contracts, data”	Consistent style or labels at high volume
Changes the model	No	No	Yes
Setup effort	Minutes	Hours to days	Days to weeks, with a developer
Keeping it current	Edit the prompt	Replace the document	Retrain
Can cite sources	If you ask	Yes, usually	No
Main risk	Inconsistent results between runs	Wrong passage retrieved, outdated files	Cost, upkeep, a confident wrong answer in your format

Prompting: the first rung

Prompting means giving the AI a clear role, the steps, the format you want and two or three examples of a good result. It is free to change, works in any chat assistant and solves the majority of work tasks: emails, summaries, reports, meeting notes, spreadsheet help. OpenAI’s accuracy guide says prompt engineering is typically the best place to start. Most “the AI does not get it” complaints are fixed here. If you can describe what you want in a paragraph and show two examples, you probably do not need anything more. Start with how to write AI prompts that work.

RAG: when the answer has to come from your files

RAG gives the AI your documents at the moment you ask; OpenAI describes it as something that is used to give the model access to domain-specific context to solve a task, and the technique comes from a 2020 research paper by Lewis and colleagues. It first finds the relevant passages and then writes its answer from them, often with a link to the source. It suits questions about a handbook, a contract set, a product manual or a shared drive, anything that changes over time. Many assistants now let you attach files or connect a drive, which gets you most of the benefit without building anything. Saved assistants such as custom GPTs or Claude Projects package prompting and reference files together. RAG reduces made-up answers but does not remove them, so check cited passages. See hallucination and how to stop AI hallucinations.

Fine-tuning: the last rung

Fine-tuning retrains an existing model on hundreds or thousands of your examples so it does one narrow job consistently. It earns its place for high-volume, repeated tasks, such as sorting thousands of tickets a day into set categories, where a prompt keeps drifting or where a smaller, cheaper model fine-tuned for the task can replace a bigger one. OpenAI’s guide notes that RAG only solves in-context learning issues, while fine-tuning addresses cases where the LLM has to learn a task so it can perform it consistently and reliably. It is a poor way to teach facts, it needs developers and good data, and it has to be redone when your needs or the underlying model change. Availability is also shrinking: OpenAI’s deprecations page says that since May 7, 2026, creating fine-tuning jobs or training is not available to organizations that have not previously run fine-tuning. Google, by contrast, documents that you can create a supervised fine-tuning job for Gemini using its console, SDK, REST API or Colab Enterprise.

Which to use: work examples

• Weekly report in the same format every time. Prompting, with last week’s report as the example.
• “Ask HR” assistant that answers from the handbook. RAG, so answers stay current and point to the page.
• Proposal drafts for a sales team. Prompting plus reference files in a saved assistant.
• Reviewing a stack of contracts for specific clauses. RAG-style: attach the contracts and ask for quotes and locations. See summarizing PDFs and contracts.
• Classifying 5,000 support tickets a day into fifteen categories, with years of staff-labeled history. A candidate for fine-tuning, after prompting has been tried and measured.
• “We want the AI to sound like our brand.” Prompting with a style guide and sample passages first; fine-tuning only if volume is very high.

How to decide

1. Write the best prompt you can, with examples. Test it on ten real cases.
2. If answers need your documents, attach or connect them. Test again.
3. If one repeated task is still inconsistent or too costly at volume, and you have hundreds of labeled examples and a way to measure success, talk to an engineer about fine-tuning.

Along the way, keep a habit of checking output; none of the three removes the need. See which AI tool to use at work for picking the assistant itself.

What the vendors’ pages say

Each line below is taken from OpenAI, Google, AWS or the original RAG paper, quoted in the Sources list.

• OpenAI says prompt engineering is typically the best place to start when optimizing an LLM
• OpenAI defines RAG as retrieving content to augment the LLM prompt before generating an answer
• AWS describes RAG as optimizing the output of a large language model so it references an authoritative knowledge base outside of its training data sources before generating a response
• OpenAI says RAG only solves in-context learning issues
• OpenAI says that for many use cases the issue will be ensuring the LLM can learn a task so it can perform it consistently and reliably, and that for this problem it turns to fine-tuning
• OpenAI says creating fine-tuning jobs or training is not available to organizations that have not previously run fine-tuning, as of May 7, 2026
• Google documents that you can create a supervised fine-tuning job for Gemini using the Google Cloud console, the Google Gen AI SDK, the REST API, or Colab Enterprise
• The arXiv paper on RAG explores a general-purpose fine-tuning recipe for Retrieval Augmented Generation models which combine pre-trained parametric and non-parametric memory for language generation

The first two rungs are the skills practiced hands-on in Learn AI Fast, a two-day in-person class in Boston for non-technical professionals, using provided email, spreadsheet and document case files.

Sources

• OpenAI, Optimizing LLM accuracy: “Prompt engineering is typically the best place to start”
• OpenAI, Optimizing LLM accuracy: “RAG is the process of R etrieving content to A ugment your LLM’s prompt before G enerating an answer”
• Amazon Web Services, What is Retrieval-Augmented Generation?: “Retrieval-Augmented Generation (RAG) is the process of optimizing the output of a large language model, so it references an authoritative knowledge base outside of its training data sources before generating a response”
• OpenAI, Optimizing LLM accuracy: “RAG only solves our in-context learning issues”
• OpenAI, Optimizing LLM accuracy: “for many use cases, the issue will be ensuring the LLM can learn a task so it can perform it consistently and reliably. For this problem we turn to fine-tuning”
• OpenAI, Deprecations: “Creating fine-tuning jobs or training is not available to organizations that have not previously run fine-tuning”
• Google Cloud, Tune Gemini models with supervised fine-tuning: “You can create a supervised fine-tuning job by using the Google Cloud console, the Google Gen AI SDK, the REST API, or Colab Enterprise”
• arXiv, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al.): “We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) – models which combine pre-trained parametric and non-parametric memory for language generation”