Module 1: LLM Fundamentals#

What an LLM actually is, from tokens to weights to the API you call: next-token prediction, sampling, context windows, the OpenAI API as the de facto standard, the model landscape and how to read benchmarks critically, and the practical craft of working with the context window. Compressed pace; the lab is where it sticks.

Questions this module answers#

• What actually happens when I send a prompt to an LLM — what are tokens, weights, and sampling?
• Why is “glorified autocomplete” both accurate and misleading?
• What is a context window, and why do long sessions degrade?
• If the API is stateless, why is resending a long conversation not ruinously expensive?
• What’s the difference between a system prompt and a user message, and how do I use each well?
• What actually makes the system prompt special? (less than you think)
• How do I pick a model — and when does small-and-fast beat the frontier flagship?
• How do I read benchmarks (and ArtificialAnalysis) without being fooled?
• Why do alignment and refusals matter for security work?

Slides#

Lab 1.1: Write a Chat Program (60 min)#

Goal#

Talk to the LLM API at the rawest level first (a curl command), then write a basic chat program yourself using the openai Python library. No frameworks, no scaffolding, no tool calling; just you, the wire format, and the messages array. When you finish, you understand the layer every AI tool is built on.

Keep your script: you’ll extend it into an agent in Labs 2.1–2.5.

Provided#

• $CLASS_API_URL and $CLASS_API_KEY, exported in your shell; your values are shown in the class portal tab. $CLASS_API_URL is the API base URL and ends in /v1: the Python client uses it directly, and a raw request appends /chat/completions.
• A checker script, already on your instance.

Everything else you need is in the reference section below; this page stands alone. The openai package is preinstalled on your instance. You write the program from scratch; there is no starter code.

The class API serves two model aliases: small (fast, cheap; the default for this lab) and large (stronger reasoning; used later in the class).

Reference: the API in one page#

The raw HTTP request. One POST, JSON in, JSON out:

POST /v1/chat/completions
Authorization: Bearer YOUR_KEY
Content-Type: application/json

with this JSON body:

{
  "model": "small",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello"}
  ]
}

Roles. Each entry in messages has a role and content:

The response shape (the fields you need):

{
  "choices": [
    {"message": {"role": "assistant", "content": "Hello!"}}
  ],
  "usage": {
    "prompt_tokens": 18,
    "completion_tokens": 3,
    "total_tokens": 21
  }
}

The Python client. The openai library builds the same HTTP request for you:

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["CLASS_API_URL"],
    api_key=os.environ["CLASS_API_KEY"],
)

response = client.chat.completions.create(
    model="small",
    messages=[{"role": "user", "content": "Hello"}],
)

print(response.choices[0].message.content)

Conversation history. The API is stateless. To give the model memory, send the complete conversation on every turn:

messages.append({"role": "user", "content": user_text})
# Send messages to the API.
messages.append({"role": "assistant", "content": assistant_text})

Resending the full transcript every call sounds wasteful, but providers cache the stable prefix of a conversation (prompt caching), so the unchanged earlier turns cost a fraction of fresh tokens.

REPL exit:

try:
    text = input("> ")
except EOFError:
    break

if text.strip() == "quit":
    break

Common failures:

Steps#

1. Connect to your lab instance. Read the reference section above.

2. Before any Python: send one request with curl. POST a JSON body (model, messages array with one user message) to /chat/completions, with your API key in the Authorization header. Read the raw JSON that comes back: the assistant message, the role, the token usage. This is the entire interface. Everything in this class (every chatbot, every agent, every framework) is wrapped around this one HTTP call.

   curl -s "$CLASS_API_URL/chat/completions" \
     -H "Authorization: Bearer $CLASS_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{
       "model": "small",
       "messages": [{"role": "user", "content": "Say hello in five words."}]
     }'

   What each piece does:

   • curl -s "$CLASS_API_URL/chat/completions" — -s is “silent”: suppress curl’s progress meter so only the response prints. The URL is your class API base (which already ends in /v1) plus the chat-completions path. Without -d, curl would send a GET; providing a body makes this a POST.
   • -H "Authorization: Bearer $CLASS_API_KEY" — -H adds an HTTP request header. This one authenticates you: the Bearer scheme means “whoever holds this token”.
   • -H "Content-Type: application/json" — a second header telling the server the body is JSON.
   • -d '{...}' — the request body. Two fields: model, the alias of the model that should answer (small is the fast, cheap tier), and messages, the conversation so far as an array of {role, content} objects — here a single user message.

3. Send the curl again. Different phrasing back? That’s sampling: the model picks among likely next tokens rather than always taking the top one.

4. Tokenizer detour: open https://platform.openai.com/tokenizer in a browser and paste each of these three payloads. Note the token count for each:

   • 100 W characters in a row (WWWW…)
   • ~100 characters of a famous poem
   • one word repeated until you reach ~100 characters (e.g., hello hello hello …)

   Same approximate character count, three different token counts. Why do they differ?

5. Now write the smallest possible Python program: construct a client pointed at your class API URL, send one hardcoded user message to small, print the reply. Compare with the curl: the openai library is building the same request you just typed by hand, nothing more.
6. Print the raw response object once and look at it: the same fields you saw in the curl output, now as Python attributes.

7. Turn it into a REPL: loop over input(), send the user’s message, print the model’s reply, repeat until EOF/quit.
8. Add conversation history: keep a messages list, append each user message and each assistant reply, send the whole list every turn. Verify the model remembers your name from turn 1 at turn 3. Those two appends are the difference between an API call and a “chat”: the model itself remembers nothing, and your list is the only record.
9. Now tamper with the record. Change the assistant append: after printing the model’s real reply, prompt yourself for what to record as the assistant’s message (plain enter keeps the real reply). Then lie: replace a reply with something the model never said and keep chatting. Ask the model about its earlier answer. It stands by whatever is in the list. You control what the model thinks it said; the transcript is just editable data.

messages.append({"role": "user", "content": user})
resp = client.chat.completions.create(model="small", messages=messages)
reply = resp.choices[0].message.content
print(reply)
edited = input("history> ")  # plain enter keeps the real reply
messages.append({"role": "assistant", "content": edited or reply})

The finished REPL (steps 7–9), as a flow:

flowchart TD
    R["Read user input"] -->|quit / EOF| Z([exit])
    R --> U["Append user message"]
    U --> P["POST full messages list to the API"]
    P --> A["Print assistant reply"]
    A --> E{"Keep or rewrite?"}
    E -->|enter: keep| K["Append real reply"]
    E -->|type a lie| L["Append edited reply"]
    K --> R
    L --> R

10. Add a system prompt as the first entry in the list (hardcoded or via flag). Test that it changes behavior (e.g., “only respond in JSON”).

import sys
from openai import OpenAI

client = OpenAI(base_url=BASE_URL, api_key=API_KEY)
messages = [{"role": "system", "content": "You are a terse assistant."}]

while True:
    try:
        user = input("> ")
    except EOFError:
        break
    if user.strip() == "quit":
        break
    messages.append({"role": "user", "content": user})
    resp = client.chat.completions.create(model="small", messages=messages)
    reply = resp.choices[0].message.content
    messages.append({"role": "assistant", "content": reply})
    print(reply)

11. Run the checker. It drives your REPL and verifies: replies come back, history persists across turns, the system prompt is honored.
12. Stretch goals, in order: streaming (print tokens as they arrive), a --model flag (try large; one string change), token counting per turn from the usage field.

stream = client.chat.completions.create(
    model="small", messages=messages, stream=True)
reply = ""
for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    reply += delta
    print(delta, end="", flush=True)
print()
messages.append({"role": "assistant", "content": reply})

Done when#

The checker passes.