AI Automation Fundamentals: Understanding LLMs

This is Part 1 of the AI Automation Engineer Roadmap series.

TL;DR

A foundational guide to understanding how large language models work and why they matter for AI automation engineers. You will learn about transformer architecture, tokenization, context windows, sampling parameters, and how to use the OpenAI and Anthropic SDKs to build your first AI-powered features.

Why This Matters

Every AI automation workflow starts with an LLM call. Whether you are building a chatbot, a document summarizer, or a full agentic pipeline, understanding how these models actually work under the hood is the difference between shipping reliable systems and wrestling with mysterious failures. Most tutorials skip the fundamentals and jump straight to "copy this prompt." That approach falls apart the moment you hit context limits, unexpected token costs, or inconsistent outputs. This post gives you the mental model you need to debug, optimize, and architect AI systems with confidence.

Core Concepts

What Are Large Language Models?

At the most basic level, an LLM is a neural network that predicts the next token in a sequence. Given the input "The capital of France is", the model assigns probabilities to every token in its vocabulary and selects one -- most likely "Paris." Stack this prediction loop end-to-end and you get coherent paragraphs, working code, and structured data.

The key insight: LLMs do not "understand" text the way humans do. They are extraordinarily powerful pattern-completion engines trained on massive datasets. Keeping this framing in mind will save you from anthropomorphizing the model and help you reason about its limitations.

Transformer Architecture (Simplified)

The transformer architecture, introduced in the 2017 paper "Attention Is All You Need," is what makes modern LLMs possible. Here is the simplified version:

1. ›Tokenization -- Input text is split into tokens (subword units). "unhappiness" might become ["un", "happiness"].
2. ›Embedding -- Each token is mapped to a high-dimensional vector that captures its meaning.
3. ›Self-Attention -- The model examines relationships between all tokens simultaneously. This is the magic -- it lets the model understand that "it" in "The cat sat on the mat because it was tired" refers to "cat," not "mat."
4. ›Feed-Forward Layers -- Transform the attention outputs through learned weights.
5. ›Output Projection -- Produce a probability distribution over the entire vocabulary for the next token.

You do not need to implement transformers from scratch. But understanding this flow explains why context windows have limits (attention is quadratic in sequence length), why token count matters for cost, and why the model sometimes "forgets" information in very long conversations.

Tokens and Tokenization

Tokens are the atomic units LLMs work with. They are not words -- they are subword pieces determined by the model's tokenizer. Some key facts:

• ›1 token is roughly 4 characters or 0.75 English words
• ›Code typically uses more tokens than prose for the same "amount" of content
• ›Different models use different tokenizers (GPT-4o uses o200k_base, Claude uses its own BPE tokenizer)
• ›You pay per token -- both input and output

Context Windows

The context window is the total number of tokens the model can process in a single request (input + output combined). This is one of the most important constraints you will work with:

• ›GPT-4o: 128K tokens
• ›Claude Sonnet: 200K tokens
• ›Gemini 1.5 Pro: 1M+ tokens

Larger context windows are not always better. Longer contexts cost more, increase latency, and models can struggle with information retrieval in the middle of very long contexts (the "lost in the middle" problem).

Sampling Parameters

When the model produces its probability distribution over tokens, sampling parameters control how the next token is selected:

• ›Temperature (0-2): Controls randomness. 0 = deterministic (always pick the highest-probability token). 1 = balanced. Higher = more creative/random.
• ›Top-p (0-1): Nucleus sampling. Only consider tokens whose cumulative probability mass reaches this threshold. 0.1 means only the top 10% probability mass is considered.
• ›Frequency Penalty (-2 to 2): Penalizes tokens based on how often they have appeared, reducing repetition.
• ›Max Tokens: Hard cap on output length. The model stops generating when it hits this limit.

For deterministic tasks (data extraction, classification), use temperature: 0. For creative tasks (writing, brainstorming), use temperature: 0.7-1.0.

Hands-On Implementation

Using the OpenAI SDK

import OpenAI from "openai";
 
const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});
 
// Basic completion
async function generateResponse(userMessage: string) {
  const response = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [
      {
        role: "system",
        content: "You are a helpful assistant that explains technical concepts clearly.",
      },
      {
        role: "user",
        content: userMessage,
      },
    ],
    temperature: 0.7,
    max_tokens: 1024,
  });
 
  return response.choices[0].message.content;
}
 
// Streaming responses for better UX
async function streamResponse(userMessage: string) {
  const stream = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [
      { role: "system", content: "You are a helpful assistant." },
      { role: "user", content: userMessage },
    ],
    stream: true,
  });
 
  for await (const chunk of stream) {
    const content = chunk.choices[0]?.delta?.content || "";
    process.stdout.write(content); // Write each chunk as it arrives
  }
}

Using the Anthropic SDK

import Anthropic from "@anthropic-ai/sdk";
 
const anthropic = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY,
});
 
// Basic completion with Claude
async function generateWithClaude(userMessage: string) {
  const response = await anthropic.messages.create({
    model: "claude-sonnet-4-20250514",
    max_tokens: 1024,
    system: "You are a helpful assistant that explains technical concepts clearly.",
    messages: [
      {
        role: "user",
        content: userMessage,
      },
    ],
  });
 
  // Anthropic returns content blocks, not a single string
  const textBlock = response.content.find((block) => block.type === "text");
  return textBlock?.text;
}
 
// Streaming with Claude
async function streamWithClaude(userMessage: string) {
  const stream = anthropic.messages.stream({
    model: "claude-sonnet-4-20250514",
    max_tokens: 1024,
    messages: [{ role: "user", content: userMessage }],
  });
 
  for await (const event of stream) {
    if (
      event.type === "content_block_delta" &&
      event.delta.type === "text_delta"
    ) {
      process.stdout.write(event.delta.text);
    }
  }
}

Model Selection: When to Use What

// A simple utility to pick the right model for the job
type TaskType = "extraction" | "reasoning" | "creative" | "simple" | "long-context";
 
interface ModelConfig {
  provider: "openai" | "anthropic";
  model: string;
  temperature: number;
  rationale: string;
}
 
function selectModel(task: TaskType): ModelConfig {
  const configs: Record<TaskType, ModelConfig> = {
    extraction: {
      provider: "openai",
      model: "gpt-4o-mini",
      temperature: 0,
      rationale: "Fast, cheap, deterministic -- ideal for structured data extraction",
    },
    reasoning: {
      provider: "anthropic",
      model: "claude-sonnet-4-20250514",
      temperature: 0,
      rationale: "Strong reasoning capabilities for complex analytical tasks",
    },
    creative: {
      provider: "openai",
      model: "gpt-4o",
      temperature: 0.9,
      rationale: "Good creative output with higher temperature",
    },
    simple: {
      provider: "openai",
      model: "gpt-4o-mini",
      temperature: 0.3,
      rationale: "Cost-effective for simple tasks like classification or summarization",
    },
    "long-context": {
      provider: "anthropic",
      model: "claude-sonnet-4-20250514",
      temperature: 0.3,
      rationale: "200K context window for processing large documents",
    },
  };
 
  return configs[task];
}

Estimating Costs

// Rough cost estimation utility
interface CostEstimate {
  inputCost: number;
  outputCost: number;
  totalCost: number;
}
 
function estimateCost(
  model: string,
  inputTokens: number,
  outputTokens: number
): CostEstimate {
  // Prices per 1M tokens (check provider pricing pages for current rates)
  const pricing: Record<string, { input: number; output: number }> = {
    "gpt-4o": { input: 2.5, output: 10 },
    "gpt-4o-mini": { input: 0.15, output: 0.6 },
    "claude-sonnet-4-20250514": { input: 3, output: 15 },
  };
 
  const rates = pricing[model];
  if (!rates) throw new Error(`Unknown model: ${model}`);
 
  const inputCost = (inputTokens / 1_000_000) * rates.input;
  const outputCost = (outputTokens / 1_000_000) * rates.output;
 
  return {
    inputCost,
    outputCost,
    totalCost: inputCost + outputCost,
  };
}
 
// Example: Processing 1000 customer support tickets
// ~500 input tokens per ticket, ~200 output tokens per response
const estimate = estimateCost("gpt-4o-mini", 500_000, 200_000);
console.log(`Estimated cost for 1000 tickets: $${estimate.totalCost.toFixed(4)}`);
// Estimated cost for 1000 tickets: $0.1950

Best Practices

1. ›Always set max_tokens -- Without it, the model may generate far more output than you need, increasing cost and latency.
2. ›Use streaming for user-facing features -- Perceived latency drops dramatically when users see tokens appear in real-time.
3. ›Start with the cheapest model -- Try gpt-4o-mini first. Only upgrade to larger models when the task demonstrably requires it.
4. ›Log everything -- Store the full request and response for every LLM call. You will need it for debugging, evaluation, and cost tracking.
5. ›Use environment variables for API keys -- Never hardcode keys. Use a secrets manager in production.
6. ›Set timeouts -- LLM API calls can hang. Always wrap them in a timeout or use the SDK's built-in timeout configuration.

Common Pitfalls

• ›Ignoring token limits: Your input + output must fit within the context window. If your input is 127K tokens on GPT-4o, you only have 1K tokens left for the response.
• ›Using high temperature for deterministic tasks: Setting temperature to 0.7 for data extraction will give you inconsistent results. Use 0.
• ›Not handling rate limits: Every provider has rate limits. Implement exponential backoff retry logic from day one.
• ›Treating all models as equivalent: GPT-4o and GPT-4o-mini have very different capabilities. A prompt that works on GPT-4o might fail on Mini.
• ›Forgetting about latency: A 5-second API call is fine for a background job. It is unacceptable for a real-time chat UI without streaming.

What's Next

Now that you understand how LLMs work and how to call them programmatically, the next step is learning how to give them the right instructions. In Part 2: From Prompt Engineering to Context Engineering, we will cover zero-shot vs few-shot prompting, Chain-of-Thought reasoning, structured outputs, and why "context engineering" is replacing "prompt engineering" as the key skill for AI engineers.

FAQ

What are large language models and how do they work?

Large language models are neural networks trained on massive text datasets that predict the next token in a sequence, enabling them to generate human-like text, answer questions, and perform complex reasoning tasks. They use the transformer architecture with self-attention mechanisms to understand relationships between tokens across the entire input.

What is the difference between OpenAI and Anthropic models?

OpenAI offers GPT models focused on versatility and broad capabilities, while Anthropic builds Claude models emphasizing safety, helpfulness, and extended context windows for complex tasks. In practice, both are excellent -- the right choice depends on your specific requirements around context length, pricing, reasoning capability, and integration ecosystem.

Why should developers learn AI automation fundamentals?

Understanding LLM fundamentals helps developers build more reliable AI-powered applications, debug unexpected behaviors, optimize costs, and choose the right model for each use case. Without this foundation, you are essentially copy-pasting prompts and hoping for the best -- which is not a viable strategy for production systems.