How to Build RAG Applications on Rails: Step-by-Step Guide

RAG (retrieval-augmented generation) is changing how Rails apps generate answers. Instead of relying solely on what an LLM model ‘knows,’ RAG incorporates fresh and relevant data to provide users with more accurate results.

In a Rails app, RAG works as a cycle of three processes: retrieval, augmentation, and generation. Each part fits naturally into Rails’s structure, making it easier to maintain.

The main components of a RAG system are retrievers, vector databases, embeddings, and LLMs, all working together. They are like building blocks you can swap for each other depending on your needs.

Common pitfalls when using RAG in Rails include token limits, slow queries, or inconsistent data handling.

Best practices in RAG production in Rails include proper monitoring, data hygiene, caching, and scaling as data and workloads grow. These keep your app reliable once real users are on board.

This step-by-step guide walks you through the practical process of piecing everything together, including setting up retrieval, embeddings, and generation in Rails.

Let’s get started!

What is RAG?

RAG (retrieval-augmented generation) is a technique used by AI models that combines external information retrieval and text generation.

Instead of relying only on what an AI model has been trained on, RAG can retrieve relevant information from external databases, documents, or websites. This helps fill in the knowledge gap for LLMs without the need to retrain them.

How does RAG work in a Rails app?

In a Rails application, RAG follows a straightforward workflow you can implement with existing APIs.

At a high level, the process has three main steps:

1. Retrieval: When a user asks an AI model a question, your Rails app searches through the knowledge base to find the most relevant content based on semantic similarity. This could be documents stored in the database, external APIs, or vector databases.
2. Augmentation: In augmentation, your Rails controller takes the retrieved information, cleans it, ranks and potentially reranks it, and combines it with the user's original question. It creates an enriched prompt for the LLM. This step can also include tasks such as trimming for token budgets, chunking documents, and attaching metadata.
3. Generation: The LLM receives the augmented prompt from the previous step and generates an answer.

Let’s see the key components of building a RAG system.

What are the key components of a RAG system?

A RAG system is built on several key components that work together to make responses more useful:

• Vector database: This is the knowledge base. It stores all your documents as mathematical (vector) representations, making it easy to retrieve content by meaning (semantic similarity) rather than only by matching keywords.
• Embeddings model: This is the encoder that converts text into the above-mentioned vector representations. These vector embeddings serve as numeric ‘fingerprints’ for the files, enabling the system to measure similarity between a user query and the stored documents.
• Retriever: The component that finds relevant information from your vector database based on a user query. It acts like a search function, asking, ‘What content is most similar to this query?’
• Large Language Model (LLM): This is the brain that takes the user’s question plus the retrieved passages as context to produce a well-structured response. The quality of the answer depends on both the LLM and the relevance and conciseness of the retrieved context.
• Orchestration layer: This is where your Rails app ties everything together – indexing documents, calling the embeddings model, running retrieval, and assembling context for the LLM. With Meilisearch, you can use a simple REST API and handy Ruby gems to do the heavy lifting.

Now, let’s see the common limitations of building RAG systems on Rails.

What are common pitfalls when building RAG in Rails?

While Rails provides a strong foundation, adding RAG tools involves additional components (vector embeddings, retrieval, prompt composition) that can cause issues if not handled carefully.

Here are the main pitfalls to watch out for:

• Token limits: Large Language Models have a limit on the amount of text they can process at once. The model may omit some vital information if you include too many documents in the prompt. To avoid this, it is advisable to set hard limits on your context length. You can also compress or summarize long inputs, or chunk long documents and index them with embeddings.
• Content mismatch: More information does not always mean better answers. Overloading the model with excess information can confuse the LLM, leading to inaccurate retrieved results. Instead, focus on using semantic ranking, applying simple heuristic filters (such as date, source, and type), and designing prompts that enable the model to ignore irrelevant context.
• Memory leaks: Rails apps can start to swell when they repeatedly handle large documents without proper cleanup. To avoid this, ensure your processes release resources when they are no longer needed, use streaming to handle big files in smaller chunks, and continually monitor memory usage in production.
• Performance issues: Retrieving information from large datasets can cause performance issues and mar the user’s experience if not properly managed. To keep your app responsive, offload heavy retrieval tasks to background jobs, and use data cache for frequent requests.

Step-by-step guide to building a RAG app in Rails

To demonstrate how to build a RAG app in Rails, we will create a Recipe Search Assistant that can answer cooking questions, such as ‘How do I make pasta?’ by searching through a recipe collection and providing personalized cooking advice based on the ingredients available.

Here is the flow:

User Question: "How do I make pasta?"
     ↓
Search Your Recipes → Find "Basic Spaghetti" recipe
     ↓
Send Recipe + Question to AI → Get specific cooking instructions
     ↓
"Based on your Basic Spaghetti recipe, boil water for 10 minutes..."

Let’s jump in.

1. Set up your Rails foundation

We need to create a new Rails application with the right dependencies for our RAG system. Let’s call it recipe_rag.

# Create a new Rails application
rails new recipe_rag --skip-javascript
cd recipe_rag

We will use meilisearch-rails for search functionality, httparty for API calls to OpenAI, and dotenv-rails for managing environment variables.

In the Gemfile, add these lines:

gem 'meilisearch-rails'  # For fast document search
gem 'httparty'          # For making API calls to OpenAI
gem 'dotenv-rails'      # For environment variables

Run bundle install in the terminal to install them.

2. Get Meilisearch running

We will spin up the Meilisearch server using Docker.

# Start Meilisearch with Docker (keep this terminal open)
docker run -it --rm -p 7700:7700 -e MEILI_ENV='development' -e MEILI_MASTER_KEY='aSampleMasterKey' getmeili/meilisearch:v1.4

The Meilisearch server should be running at port 7700.

Next, create a .env file and store the environment variables. This includes the Meilisearch host, API key, and OpenAPI key, which we will use later.

MEILISEARCH_HOST=http://localhost:7700
MEILISEARCH_API_KEY=aSampleMasterKey
OPENAI_API_KEY=your_openai_api_key_here

3. Connect Rails to Meilisearch

Rails needs to know how to talk to our search engine. We do this through an ‘initializer.’ This file will run when Rails starts up and configures our connections. Add this configuration to config/initializers/meilisearch.rb

require 'meilisearch'
MeiliSearch::Rails.configuration = {
  meilisearch_url: ENV['MEILISEARCH_HOST'] || 'http://localhost:7700',
  meilisearch_api_key: ENV['MEILISEARCH_API_KEY'] || 'aSampleMasterKey'
}

This instructs Rails on where to locate Meilisearch and how to authenticate with it using the credentials from our .env file.

4. Creating your Recipe model

In Rails, a ‘model’ is like a Python class that represents data in your database. By integrating it with Meilisearch, every time we save a recipe to the database, it automatically gets indexed for searching. No manual work required.

In the terminal, enter the command:

# Generate the Recipe model with database fields
rails generate model Recipe name:string ingredients:text instructions:text cuisine:string
rails db:migrate

This will generate a dummy Recipe model in app/models/recipe.rb. Edit the file to specify how Meilisearch should search. The meilisearch do block tells Rails which fields should be searchable and how to rank results.

class Recipe < ApplicationRecord
  include MeiliSearch::Rails
  # Configure which fields should be searchable
  meilisearch do
    searchable_attributes [:name, :ingredients, :instructions, :cuisine]
  end

  # Basic validations to ensure data quality
  validates :name, presence: true
  validates :ingredients, presence: true
  validates :instructions, presence: true
end

5. Load sample recipe data

We need some recipes to search through! Rails has a ‘seeds’ file specifically for loading initial data. We will add a few sample recipes that demonstrate different cooking styles.

Note: When we create these recipes, they automatically get indexed in Meilisearch thanks to the integration we set up in the previous step.

# Clear any existing recipes first
Recipe.delete_all

# Create sample recipes that will automatically get indexed
recipes = [
  {
    name: "Basic Spaghetti",
    ingredients: "spaghetti pasta, olive oil, garlic, salt, pepper",
    instructions: "Boil water, add pasta, cook 10 minutes. Heat oil, add garlic, mix with pasta.",
    cuisine: "Italian"
  },
  {
    name: "Chicken Stir Fry", 
    ingredients: "chicken breast, soy sauce, vegetables, garlic, ginger, oil",
    instructions: "Cut chicken into pieces. Heat oil, cook chicken 5 minutes. Add vegetables and sauce, stir fry 5 minutes.",
    cuisine: "Asian"
  },
  {
    name: "Chocolate Chip Cookies",
    ingredients: "flour, butter, sugar, chocolate chips, eggs, vanilla", 
    instructions: "Mix butter and sugar. Add eggs and vanilla. Mix in flour. Add chocolate chips. Bake at 350°F for 12 minutes.",
    cuisine: "American"
  }
]

# Create each recipe (automatically indexed by Meilisearch)
recipes.each { |recipe_data| Recipe.create!(recipe_data) }
puts "Created #{Recipe.count} recipes - all automatically indexed!"

Now, we can load the data into Meilisearch. In a terminal, run:

# Load the sample data
rails db:seed

We can now test whether everything works with a few sample searches.

# Open Rails console to test search functionality
rails console

# Test exact search
Recipe.ms_search("pasta")

# Test typo tolerance
Recipe.ms_search("spagetti")  # Should still find spaghetti

# Test semantic search across fields
Recipe.ms_search("italian noodles")  # Should find Basic Spaghetti

Output:

Notice that even with a typo – ‘spagetti’ – or with a search of ‘italian noodles,’ Meilisearch can find ‘spaghetti.’ This is the power of the Meilisearch search engine over basic keyword search engines.

6. Build the RAG service logic

Now we can create a service that searches for relevant recipes, then ask AI to generate a response based on what we found.

We are not just sending the user's question to OpenAI. Instead, we first search our recipe database, find the most relevant recipes, and then send both the question and the relevant recipes to the AI.

Create app/services/recipe_assistant_service.rb:

class RecipeAssistantService
  def initialize
    @api_key = ENV['OPENAI_API_KEY']
    raise "Please set OPENAI_API_KEY in .env file" unless @api_key
  end

  def answer_cooking_question(question)
    # Step 1: Search for relevant recipes
    relevant_recipes = search_recipes(question)
    
    # Step 2: Ask AI to answer using the recipes
    ai_response = ask_ai(question, relevant_recipes)
    
    # Step 3: Return both the answer and sources
    {
      answer: ai_response,
      recipe_sources: relevant_recipes.map { |r| r.name }
    }
  end

  private

  def search_recipes(question)
    # Rails + Meilisearch returns AR objects, not raw hits
    Recipe.ms_search(question, limit: 2)
  end

  def ask_ai(question, recipes)
    recipe_context = recipes.map do |recipe|
      <<~RECIPE
        Recipe: #{recipe.name} (#{recipe.cuisine})
        Ingredients: #{recipe.ingredients}
        Instructions: #{recipe.instructions}
      RECIPE
    end.join("
" + "-" * 40 + "
")

    system_prompt = "You are a helpful cooking assistant. Always base your answers on the provided recipes when available. If recipes are provided, reference them specifically and explain how they relate to the question."

    user_prompt = if recipes.any?
      <<~PROMPT
        Based on these recipes from my collection:
        #{recipe_context}

        Question: #{question}

        Please provide specific cooking advice based on the recipes above. Mention which recipes you're referencing.
      PROMPT
    else
      <<~PROMPT
        Question: #{question}
        
        I don't have specific recipes for this question in my collection, but I can provide general cooking guidance.
      PROMPT
    end

    response = HTTParty.post(
      'https://api.openai.com/v1/chat/completions',
      headers: {
        'Authorization' => "Bearer #{@api_key}",
        'Content-Type' => 'application/json'
      },
      body: {
        model: 'gpt-4o',
        messages: [
          { role: 'system', content: system_prompt },
          { role: 'user', content: user_prompt }
        ],
        max_tokens: 300,
        temperature: 0.3
      }.to_json
    )

    if response.success?
      response.dig('choices', 0, 'message', 'content') || "Sorry, I couldn't generate a response."
    else
      "API Error: #{response.code} - #{response.message}"
    end
  rescue => e
    "Error generating response: #{e.message}"
  end
end

7. Create the web interface controller

We can now create a control that handles HTTP requests. When someone visits our website or submits a form, the controller determines how to proceed with that request.

In a terminal, enter:

# Generate the controller with basic actions
rails generate controller RecipeAssistant index ask --no-helper --no-assets

Our controller simply takes the user's cooking question, passes it to our RAG service, and then displays the results. Create app/controllers/recipe_assistant_controller.rb:

class RecipeAssistantController < ApplicationController
  def index
    # Just show the main page with search form
  end

  def ask
    @question = params[:question]
    
    if @question.present?
      assistant = RecipeAssistantService.new
      @result = assistant.answer_cooking_question(@question)
    else
      @error = "Please ask a cooking question!"
    end
    
    # Render the same page with results
    render :index
  rescue => e
    @error = "Something went wrong: #{e.message}"
    render :index
  end
end

Now we need to set up our routes (URL patterns). Edit config/routes.rb:

Rails.application.routes.draw do
  root 'recipe_assistant#index'  # Homepage shows search form
  post 'ask', to: 'recipe_assistant#ask'  # Form submission goes here
end

8. Build a frontend UI to test

Finally, we can create a simple frontend that allows us to interact with the RAG we have just built. We will use ERB templates, which let us mix HTML with Ruby code to display dynamic content. Edit app/views/recipe_assistant/index.html.erb:

<!DOCTYPE html>
<html>
<head>
  <title>Recipe Assistant</title>
  <style>
    body { font-family: Arial; max-width: 600px; margin: 50px auto; padding: 20px; }
    .form-group { margin: 20px 0; }
    input[type="text"] { width: 100%; padding: 10px; font-size: 16px; }
    button { background: #007cba; color: white; padding: 12px 20px; border: none; cursor: pointer; }
    .answer { background: #f0f8f0; padding: 20px; margin: 20px 0; border-radius: 5px; }
    .sources { background: #f8f0f0; padding: 15px; margin: 10px 0; border-radius: 5px; }
    .error { background: #f8f0f0; color: red; padding: 15px; margin: 10px 0; }
  </style>
</head>
<body>
  <h1>Recipe Assistant</h1>
  <p>Ask me cooking questions and I'll help based on my recipe collection!</p>

  <!-- Question Form -->
  <%= form_with url: ask_path, local: true do |form| %>
    <div class="form-group">
      <%= form.text_field :question, 
          placeholder: "e.g., How do I make pasta?", 
          value: @question %>
    </div>
    <%= form.submit "Ask Assistant", class: "button" %>
  <% end %>

  <!-- Show Error -->
  <% if @error %>
    <div class="error"><%= @error %></div>
  <% end %>

  <!-- Show AI Answer -->
  <% if @result %>
    <div class="answer">
      <h3>🤖 Assistant's Answer:</h3>
      <p><%= simple_format(@result[:answer]) %></p>
    </div>

    <!-- Show Recipe Sources -->
    <% if @result[:recipe_sources].any? %>
      <div class="sources">
        <h4>Based on these recipes:</h4>
        <ul>
          <% @result[:recipe_sources].each do |recipe_name| %>
            <li><%= recipe_name %></li>
          <% end %>
        </ul>
      </div>
    <% end %>
  <% end %>

  <!-- Example Questions -->
  <div style="margin-top: 40px; padding: 20px; background: #f5f5f5;">
    <h4>💡 Try asking:</h4>
    <ul>
      <li>"How do I cook pasta?"</li>
      <li>"What ingredients do I need for stir fry?"</li>
      <li>"How long do I bake cookies?"</li>
    </ul>
  </div>
</body>
</html>

9. Test your RAG system

Time to see everything working together! Ensure Meilisearch is still running in a terminal, then start your Rails application in a separate terminal.

# Start the Rails server
rails server

Open your browser to http://localhost:3000 and try some questions. For instance, we can ask: ‘How do I make pasta?’

Notice the steps are based on the ingredients we have added to our database.

And this is how to build a RAG application that demonstrates the use of document indexing, semantic search, context augmentation, and grounded AI responses.

What are production best practices for RAG on Rails?

Running RAG in production means dealing with real users, data, and its problems.

Here are some best practices for producing RAG on Rails:

• Build for reliability: It is possible for APIs to fail and servers to crash. Implement proper error handling and circuit breakers to ensure optimal system performance. Always have a fallback response ready when your LLM fails you.
• Proper monitoring: Track key metrics like response times, search quality, API costs, and user satisfaction. Set up alerts for when things go sideways.
• Plan for data evolution: Your knowledge base will grow and constantly change. Build processes to update embeddings, refresh search indexes, and ensure that your new content does not break existing functionality.
• Data hygiene: Keep documents clean and up-to-date so the model generates accurate and relevant results.
• Implement robust caching: Cache as much as possible, including search results, LLM responses, and processed embeddings. Meilisearch stores your documents in an indexed and searchable format, so you can quickly retrieve the relevant data. You can combine this with proper expiration strategies and cache invalidation to ensure your users always get accurate results without overloading your system.
• Keep learning: RAG techniques are evolving fast. Always stay flexible with your architecture so you can swap components as better tools emerge.

Bringing RAG to life in your Rails applications

RAG is a practical way to make your Rails apps more useful. You can start small with a simple keyword search through Meilisearch and then layer in vector search and more intelligent retrieval as your needs grow.

What matters most is solving real-life problems, not chasing every new AI trend. If you have a solid foundation with good caching, background jobs, and monitoring, you will deliver relevant and reliable answers to your users.