Understand AI Better than 90% of People, Part 2: Context Windows

Understand AI Better than 90% of People in 2 Minutes

Part 2: Context Windows

In Part 1, we learned that AI models process information using tokens. Think of tokens as the building blocks that power your conversation with your favorite AI tools.

Now it's time to talk about where those blocks are stored.

When you start a new conversation with ChatGPT (or any other LLM), a new workspace is created for that chat. That workspace is called a context window.

The easiest way to think about a context window is as a whiteboard. The context window is the size of the board, and the tokens are the building blocks that fill it.

Every question you ask gets written on the board. Every response the AI generates gets written on the board too. Every document you upload, every spreadsheet you analyze, every block of code you paste into the chat uses even more space on the board.

At first, there is plenty of room. You can ask questions, refine ideas, upload documents, and build on previous responses. But eventually, even the biggest board starts to fill up.

When that happens, things can get weird.

To make room on the board, something gets erased off the board, but don't fret, the information isn't technically erased and gone forever, it may be summarized, compressed, or pushed out of the model's active memory to make room.

Most of the time, you won't notice. If the AI forgets that your favorite ice cream flavor is mint chocolate chip, life goes on.

But if it forgets a key requirement from the software architecture you've been designing for the last three hours, you may suddenly find yourself explaining the same thing over and over.

This is why context windows matter.

A larger context window allows the AI to keep more information available at the same time. It can reference more of your conversation, analyze larger documents, review more code, and maintain a better understanding of complex problems without losing track of important details.

So, when someone tells you that a new AI model has a larger context window, they're really telling you something simple, it has a bigger whiteboard to write on.

Next up, Part 3: Hallucinations 

---

Thanks for reading and if you have a question, please leave a comment.

With over 25 years of technical industry experience, Sarah Writtenhouse shares her insights about job market trends, emerging technology and software development.

Find her LinkedIn and Medium.