Building an AI-Powered SaaS: Lessons Learned

TL;DR

Successfully integrating LLMs into a SaaS product requires streaming responses for UX, disciplined prompt engineering, and aggressive cost management through caching and model selection. I built an AI-powered content platform that used GPT-4o for generation, RAG for context grounding, and a multi-tier cost strategy that kept API spend predictable at scale.

The Challenge

A startup approached me to build an AI-powered content platform for marketing teams. The idea was simple: users would input a topic, select a tone and format, and the application would generate draft blog posts, social media copy, and email sequences grounded in the company's existing brand guidelines and knowledge base.

The real challenge was not generating text. Any developer can wire up an OpenAI API call. The real challenges were:

• ›Latency. GPT-4 class models take 5-15 seconds to generate a full response. Users will not wait staring at a spinner.
• ›Consistency. LLMs are probabilistic. The same prompt can produce wildly different outputs. Marketing teams need predictable quality.
• ›Cost. At scale, naive LLM usage gets expensive fast. A single GPT-4o request with a large context window can cost $0.05-0.10. Multiply that by thousands of daily users and the unit economics collapse.
• ›Grounding. Generated content needed to reference the company's actual products, pricing, and brand voice, not hallucinate facts.

The platform was built on Next.js with a PostgreSQL database, deployed on Vercel, with the AI layer integrated through the Vercel AI SDK.

The Architecture

Streaming UI: Making LLMs Feel Fast

The single most impactful UX decision was streaming. Instead of waiting for the complete response and rendering it all at once, I streamed tokens to the client as they were generated. This transforms a 10-second wait into an experience that feels responsive from the first 200 milliseconds.

The Vercel AI SDK makes this remarkably clean in Next.js App Router:

// app/api/generate/route.ts
import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';
 
export async function POST(req: Request) {
  const { prompt, context, format } = await req.json();
 
  const result = streamText({
    model: openai('gpt-4o'),
    system: `You are a marketing content writer. Write in a professional but approachable tone.
Format: ${format}
Brand context: ${context}`,
    prompt,
    maxTokens: 2000,
  });
 
  return result.toDataStreamResponse();
}

On the client side, the useChat hook from the AI SDK handles the streaming connection, message state, and error handling:

'use client';
 
import { useChat } from 'ai/react';
 
export function ContentGenerator() {
  const { messages, input, handleInputChange, handleSubmit, isLoading } = useChat({
    api: '/api/generate',
  });
 
  return (
    <form onSubmit={handleSubmit}>
      <textarea value={input} onChange={handleInputChange} />
      <button type="submit" disabled={isLoading}>
        Generate
      </button>
      <div className="output">
        {messages
          .filter((m) => m.role === 'assistant')
          .map((m) => (
            <div key={m.id}>{m.content}</div>
          ))}
      </div>
    </form>
  );
}

The streaming response renders character by character, giving users immediate feedback that the system is working. This alone eliminated the most common user complaint during beta testing: "Is it broken? Nothing is happening."

RAG: Grounding Outputs in Real Data

The most dangerous failure mode of an LLM-powered product is hallucination. If the AI generates content claiming a product costs $99/month when it actually costs $149/month, the marketing team publishes incorrect information.

I implemented Retrieval-Augmented Generation (RAG) using pgvector, PostgreSQL's vector similarity search extension. The pipeline worked in three stages:

1. Ingestion. When a company onboards, their brand guidelines, product documentation, pricing pages, and previous content are chunked into ~500 token segments and embedded using OpenAI's text-embedding-3-small model. The embeddings are stored in a pgvector column.

import { embed } from 'ai';
import { openai } from '@ai-sdk/openai';
 
async function ingestDocument(content: string, companyId: string) {
  const chunks = splitIntoChunks(content, 500);
 
  for (const chunk of chunks) {
    const { embedding } = await embed({
      model: openai.embedding('text-embedding-3-small'),
      value: chunk,
    });
 
    await db.query(
      'INSERT INTO knowledge_base (company_id, content, embedding) VALUES ($1, $2, $3)',
      [companyId, chunk, JSON.stringify(embedding)]
    );
  }
}

2. Retrieval. Before generating content, the user's prompt is embedded and a cosine similarity search fetches the top 5 most relevant chunks from the company's knowledge base.

SELECT content, 1 - (embedding <=> $1) AS similarity
FROM knowledge_base
WHERE company_id = $2
ORDER BY embedding <=> $1
LIMIT 5;

3. Augmented generation. The retrieved chunks are injected into the system prompt as context, grounding the LLM's output in actual company data.

This approach reduced hallucination rates dramatically. The model could reference real product names, actual pricing tiers, and genuine brand voice because that information was in the prompt context, not retrieved from training data.

Prompt Engineering: Treating Prompts as Code

I learned early that prompts are not strings you write once and forget. They are code. They need version control, testing, and iteration.

I built a prompt management system where each prompt template was stored in the database with a version number. This allowed A/B testing different prompt structures and rolling back when a new version produced worse outputs.

interface PromptTemplate {
  id: string;
  name: string;
  version: number;
  systemPrompt: string;
  userPromptTemplate: string;
  model: string;
  temperature: number;
  maxTokens: number;
  isActive: boolean;
}
 
async function getActivePrompt(name: string): Promise<PromptTemplate> {
  const result = await db.query(
    'SELECT * FROM prompt_templates WHERE name = $1 AND is_active = true ORDER BY version DESC LIMIT 1',
    [name]
  );
  return result.rows[0];
}

Key lessons on prompt engineering:

• ›Be explicit about format. "Write a blog post" produces inconsistent results. "Write a blog post with an H1 title, 3 H2 sections, and a conclusion paragraph. Use markdown formatting." produces reliable structure.
• ›Include negative instructions. "Do not invent product features. Only reference features mentioned in the provided context." was more effective than hoping the model would stay grounded.
• ›Temperature matters. For creative marketing copy, a temperature of 0.7-0.8 worked well. For structured data extraction, 0.1-0.2 prevented unwanted variation.

Cost Management: The Make-or-Break Layer

Without cost controls, AI features will bankrupt a startup. Here is the multi-tier strategy I implemented:

Response caching. If two users from the same company generate a "product launch email" with identical parameters, the second request returns the cached response. I used a hash of the prompt + context + model as the cache key, stored in Redis with a 24-hour TTL.

Model tiering. Not every request needs GPT-4o. I implemented automatic model selection based on task complexity:

function selectModel(task: string): string {
  const complexTasks = ['long-form-blog', 'whitepaper', 'case-study'];
  const simpleTasks = ['social-post', 'email-subject', 'meta-description'];
 
  if (complexTasks.includes(task)) return 'gpt-4o';
  if (simpleTasks.includes(task)) return 'gpt-4o-mini';
  return 'gpt-4o-mini'; // Default to cheaper model
}

Token budgets. Each company had a monthly token budget. The application tracked cumulative usage and warned administrators when they approached their limit. This prevented surprise bills and forced intentional usage.

Fallback chains. If the primary model's API returned a 429 (rate limit) or 500 error, the system automatically retried with a fallback model before surfacing an error to the user:

async function generateWithFallback(params: GenerateParams) {
  const models = ['gpt-4o', 'gpt-4o-mini'];
 
  for (const model of models) {
    try {
      return await streamText({ model: openai(model), ...params });
    } catch (error) {
      if (model === models[models.length - 1]) throw error;
      console.warn(`Model ${model} failed, falling back...`);
    }
  }
}

Key Decisions & Trade-offs

pgvector over Pinecone. I chose pgvector because the knowledge base was already in PostgreSQL. Adding a separate vector database would have introduced another infrastructure dependency, another point of failure, and another service to monitor. pgvector's performance was more than sufficient for the query volumes we handled. At very large scale (millions of vectors), a dedicated vector database would make more sense.

Vercel AI SDK over raw API calls. The SDK abstracts away streaming, message formatting, and provider switching. The trade-off is vendor coupling, but the developer experience improvement was substantial. Switching from OpenAI to Anthropic required changing a single import and model string.

Caching at the application layer over edge caching. AI responses are highly context-dependent, making traditional CDN caching ineffective. Application-layer caching with semantic cache keys gave me precise control over cache invalidation when a company updated their knowledge base.

Storing prompts in the database over hardcoding them. This added complexity, but it paid for itself the first time I needed to fix a prompt in production without deploying new code. The ability to A/B test prompt variations was a bonus.

Results & Outcomes

The platform launched successfully and the AI features became the primary selling point. The streaming UI eliminated user complaints about perceived slowness. The RAG pipeline grounded outputs well enough that the marketing teams trusted the generated content as a solid first draft rather than dismissing it as generic AI slop.

The cost management strategy kept per-user API costs predictable. Model tiering alone reduced the average cost per generation by roughly half compared to using GPT-4o for everything, with no noticeable quality difference for simple tasks like social media posts and email subject lines.

The prompt versioning system proved invaluable during iteration. When OpenAI released model updates that subtly changed output behavior, I could quickly test and deploy prompt adjustments without code deployments.

What I'd Do Differently

Implement evaluation pipelines from day one. I spent too long evaluating prompt quality by manually reading outputs. An automated evaluation system using LLM-as-judge patterns would have accelerated iteration. I would set up a test suite of representative inputs with expected output criteria and run them against every prompt change.

Build content moderation before launch. The first time a user found a prompt injection that made the system generate inappropriate content, I had to rush a moderation layer into production. This should have been a launch requirement.

Use structured outputs earlier. For tasks that required specific formatting (JSON metadata, structured email sequences), I initially relied on prompt instructions alone. Moving to OpenAI's structured output mode (or Zod schema validation with the AI SDK) would have eliminated parsing errors from the start.

Invest in observability. Tracking token usage, latency percentiles, and error rates per model per endpoint should have been instrumented from the beginning. I retrofitted this with Langfuse, but having it from day one would have caught cost anomalies earlier.

FAQ

How do you handle slow LLM response times in a SaaS app?

Use streaming to send individual tokens to the client as they are generated, keeping users engaged instead of waiting for the full response. The Vercel AI SDK simplifies this in Next.js with the streamText function on the server and the useChat hook on the client. The psychological difference is significant: a 10-second wait with a spinner feels broken, but watching text appear word by word feels fast and engaging. Beyond streaming, you can also implement optimistic UI patterns where the interface immediately transitions to the "generating" state with placeholder structure, giving the user a sense of progress before the first token even arrives.

What are the biggest challenges of building AI-powered SaaS?

The three core challenges are implementing streaming UIs for responsiveness, managing prompt engineering for consistent outputs, and controlling API costs at scale. Beyond these, hallucination management through RAG is critical for any application where factual accuracy matters. There is also the challenge of user expectations: people expect AI to be perfect, but LLMs are probabilistic systems that require guardrails, fallbacks, and human review workflows. Finally, the rapid pace of model releases means your architecture must be flexible enough to swap models without rewriting your application.

How can you reduce LLM API costs in production?

Use response caching for repeated queries, select smaller models when full GPT-4 capability is not needed, and implement token budgets to prevent runaway costs. Model tiering is the highest-impact strategy: routing simple tasks to GPT-4o-mini instead of GPT-4o can reduce costs by 10-20x per request with minimal quality loss. Semantic caching with Redis adds another layer of savings for frequently requested content types. Additionally, optimizing your RAG retrieval to include only the most relevant context chunks reduces input token counts, which directly reduces cost since you are paying per token in both directions.