Maturity Assessment

This assessment evaluates organizational capability across multiple dimensions, tracking current state, target goals, and improvement pathways.

Foundational

Culture of Innovation

Process Maturity

IT Team

Automation

Integration Maturity

Automation Maturity

Data

Dashboard & Reporting

Data Maturity

Analytical Maturity

Governance

Security & Compliance

Innovation Strategy

Strategic vision and approach for organizational improvements

Innovation Opportunity

Company X is a small, nimble Company X operating in a period of rapid technological change, where the primary advantage is the ability to improve faster than larger Company Xs constrained by legacy systems and cultures. Its lean technology stack and flexible operations make it possible to change workflows and adopt new tools quickly. Early automation investments can both deliver immediate gains and build a long-term innovation advantage.

By making the right strategic bets Company X can have a much higher pace of innovation than larger law Company Xs.

Key Challenges

Recommended Investments

We recommend that Company X make two related investments:

The Team Investment

The goal is to build an organization where innovation is part of the operating model.

This requires Company X to develop three core capabilities:

The Technology Investment

The goal is to provide the technical foundation that makes sustained automation and operational visibility possible.

This requires building toward a state where:

Operational Modifications

Introduction of New Management Practices

Company X's improvement capability is built on four interlocking management practices. Each practice serves a specific purpose, and together they form a complete operating system for continuous improvement.

New Roles & Responsibilities

Operating Rhythms

Technology Enhancement Roadmap

Company X's technology strategy is to build capability progressively, starting with high-value automation and establishing the disciplines needed before making larger platform investments.

Implementation Timeline

Proposed Implementation Plan

The following sequence builds capability progressively, with each phase enabling the next:

Month 1-2: Foundation

1. Complete Gavel document automation project
2. Assign Ops Lead(s)
3. Establish bi-weekly improvement review
4. Begin building improvement backlog

Month 2-4: Process Visibility

1. Conduct facilitated process mapping sessions
2. Document current-state workflows
3. Identify automation opportunities

Month 3-6: Data Foundation

1. Select and configure case tracking system (e.g., Airtable)
2. Define data model: Clients, Matters, Case Stages
3. Begin capturing case data in structured format
4. Integrate with document automation

Month 4-6: Operational Metrics

1. Define 3-5 core lifecycle metrics
2. Build first operational dashboard
3. Establish baseline performance
4. Use metrics to guide improvement priorities

Proposed Success Milestones (6 Months)

By the end of six months, Company X should have:

✓ Delivered multiple projects — 3-5 completed improvement projects with measurable outcomes
✓ Established operating rhythms — Bi-weekly improvement reviews running consistently
✓ Built internal capability — At least 2 citizen developers who have successfully delivered projects
✓ Created data foundation — Basic case data layer in place and being used
✓ Enabled visibility — At least one live dashboard providing operational insights
✓ Maintained momentum — Active backlog with 5-10 prioritized opportunities

These outcomes indicate that Company X has built a sustainable improvement capability, not just delivered a few one-off projects.

Phase 1: Initiate

Align leadership and define a clear North Star for innovation.

The Initiate phase establishes the foundation for your AI journey by creating leadership alignment and a clear strategic direction. Without this foundation, AI initiatives often become disconnected pilots that fail to deliver organizational value.

What

Why

The Innovation Council serves multiple critical functions:

• Removes organizational blockers that slow execution
• Creates accountability at the executive level
• Ensures resources are allocated appropriately
• Provides air cover for teams driving change

How

Our approach to the Initiate phase includes:

Leadership Workshop Structure

The workshops are designed to be highly interactive, bringing together diverse perspectives from across the organization. We facilitate discussions that surface both opportunities and concerns, ensuring the resulting strategy has broad executive support.

The Innovation North Star

Your Innovation North Star serves as the guiding principle for all AI initiatives. It answers the fundamental question: What does successful AI adoption look like for our organization?

A well-crafted North Star:

• Connects AI to business outcomes
• Is ambitious yet achievable
• Provides clear criteria for prioritization
• Inspires and aligns the organization

Phase 2: Analyze

Establish a clear view of current capabilities, gaps, and opportunities.

The Analyze phase creates a comprehensive understanding of where your organization stands today. This diagnostic work ensures that implementation efforts are built on a solid foundation and targeted at the right opportunities.

What

Why

A thorough analysis phase prevents common AI implementation failures:

• Building on unstable infrastructure that can't support AI workloads
• Pursuing opportunities that don't align with organizational strengths
• Missing easy wins that could build momentum and support
• Underestimating change management requirements

How

Our analysis methodology is comprehensive yet efficient:

Discovery Process

We gather information through multiple channels:

• Executive interviews to understand strategic priorities
• Team surveys to assess readiness and concerns
• Technical assessments to evaluate infrastructure capabilities
• Process reviews to identify automation opportunities

Interactive Results Delivery

Findings are delivered through our client portal, which includes:

• Comprehensive assessment reports
• Prioritized opportunity matrix
• Capability gap analysis
• Agentic support that allows stakeholders to explore, analyze, and ask questions about results in real-time

This interactive approach ensures that insights are not just delivered, but truly understood and actionable.

Phase 3: Implement

Turn strategy into action through measurable programs and execution.

The Implement phase is where strategy becomes reality. We work alongside your teams to translate the Innovation North Star into concrete programs, clear objectives, and measurable outcomes.

What

Why

The Implementation phase delivers:

• Clear accountability for outcomes
• Measurable progress against defined metrics
• Organizational learning through structured execution
• Momentum that builds support for continued investment

How

Our implementation approach balances structure with flexibility:

Roadmap Development

We work collaboratively to create roadmaps that are:

• Realistic given organizational constraints
• Ambitious enough to drive meaningful change
• Flexible to adapt as conditions evolve
• Measurable with clear milestones

Performance Metrics

Success is measured across multiple dimensions:

Program Portal

All participants have access to a shared portal that provides:

• Real-time progress dashboards
• Document repositories
• Communication channels
• Metric tracking and reporting

Phase 4: Monitor

Sustain momentum through visibility, governance, and continuous improvement.

The Monitor phase ensures that your AI initiatives remain on track and continue delivering value over time. Through real-time visibility, structured governance, and continuous feedback loops, we help you sustain and accelerate your transformation.

What

Why

The Monitor phase prevents common late-stage failures:

• Drift from strategic objectives as execution pressures mount
• Hidden problems that compound over time
• Stakeholder disengagement due to lack of visibility
• Missed opportunities for optimization and improvement

How

Our monitoring approach combines technology and human judgment:

Real-Time Dashboard

The client platform provides:

• Live metrics updated in real-time
• Progress visualization against milestones
• Trend analysis to identify patterns
• Alert mechanisms for issues requiring attention

Agent-Assisted Insights

Our AI-powered interface allows stakeholders to:

• Ask natural language questions about program status
• Drill into specific metrics or time periods
• Generate custom reports on demand
• Receive proactive insights and recommendations

Governance Cadence

We establish a structured review rhythm:

• Weekly operational reviews for execution teams
• Monthly progress reviews for program leadership
• Quarterly strategic reviews with the Innovation Council
• Annual planning sessions to refresh the North Star

Continuous Improvement

The Monitor phase feeds directly back into the Initiate phase, creating a continuous improvement cycle. Insights gathered during monitoring inform:

• Adjustments to the Innovation North Star
• New priorities for the Innovation Council
• Updated capability assessments
• Refined implementation approaches

This circular flow ensures your AI strategy remains dynamic and responsive to changing organizational needs and market conditions.

Ready to Begin?

The AI Strategy & Implementation Program is designed to meet your organization where you are and guide you toward sustained AI success.

Breaking Down AI

What is AI?

AI is one of those few words in computer science that actually has a pretty intuitive definition. Two letters:

So if you're using a computer to emulate some aspect of human intelligence, you're doing AI. An application that emulates some aspect of human intelligence is an AI application.

AGI - Artificial General Intelligence

Sometimes you'll show someone an application and they'll say "that's not really AI." What they usually mean is it's not impressive enough to qualify. What they really mean is that's not AGI — artificial general intelligence — which is harder to achieve than just AI.

The best way to understand general intelligence is to compare it against narrow intelligence.

Google Maps is the perfect example of an AI application that's just good at doing one thing — it helps you get from point A to point B. The Spotify recommendation algorithm just helps find you songs or build you playlists. But you couldn't use the Spotify algorithm to plan a route. You couldn't use Google Maps to pick a song. That's because these are narrow intelligences — they're just good at one thing.

On the other hand, a person like you or I is generally intelligent. We can solve lots of different types of problems.

People are so excited about AI right now because we've gotten so much closer to something that feels generally intelligent. ChatGPT can solve lots of different types of problems. Up until the last few years, AI technologies were fundamentally very narrow — hyper-suited to one particular problem. This is the first time we can say we're working on something that feels closer to general intelligence.

Super Intelligence

Over 130,000 signatures currently lie on the Statement on Super Intelligence — a signed petition from prominent AI researchers and public intellectuals stating that they are worried about the current progress in AI.

So what is super intelligence, and what are they so worried about?

We just talked about levels of intelligence: narrow intelligence that can do one thing, and general intelligence that can do lots of things. But there's actually a third level: super intelligence — AI that is better than every person at everything. Better at physics than Einstein. Better at chess than Magnus Carlsen. Better than every human at every cognitive task.

How could we end up with a super intelligence? It might look something like this.

Imagine a chart representing humanity's AI progress over time. There's a point on that chart that represents self-improving AI — AI that can reprogram itself and make itself smarter. At that point, we would have runaway progress toward super intelligence. And many people are worried that we're right before the point where this feedback loop starts.

Is super intelligence something we should be excited about or worried about? It's kind of a mixed bag. A super intelligence could lead to incredible technological and medical advancements that help a lot of people. But if it had its own motives, or fell into the wrong hands, it could be very dangerous. The good news is that the AI community is actively working on safety research, alignment, and governance to make sure we get there responsibly — and the fact that so many researchers signed that petition shows the field takes this seriously.

The Four AI Techniques

A lot of people think of AI as a single tool. But it's better to think of it as a toolbox — with different tools for different jobs.

A lot of AI projects fail because people use the wrong tool for the job. So let's open up our toolbox and see what we have.

We break AI down into four key techniques. In the next lessons, we'll go through each one in detail and understand how they work and when to use them.

"That's Not AI!"

Why define AI so broadly? You might wonder why we include simple techniques like rules-based systems alongside complex ones like agents. There are four good reasons:

What About Data Science?

You might be wondering — where does data science fit into all of this? The two fields overlap a lot, and people often use the terms interchangeably, but they're not the same thing.

AI is about getting computers to emulate human intelligence — making decisions, recognizing patterns, taking actions. Data science is about understanding data — exploring it, visualizing it, and drawing conclusions from it.

There's plenty of overlap — machine learning, for example, lives in both worlds. But data science also includes things like exploratory data analysis (EDA), causal inference, and experimental design, which aren't really AI techniques.

Knowledge Check

Rules-Based AI

Let's start with the simplest form of AI.

Meet Sally, a loan officer. Her process is super clear and consistent, which makes it perfect for automation.

It's not difficult to see that Sally's work could be done by a computer. It might look something like this:

That's a script — a small piece of code that captures explicit human logic.

Rules Engines

Another way we can get a computer to do this workflow is through a rules engine. A rules engine lets Sally (and other subject matter experts) edit the rules themselves without needing a programmer. This is called self-service.

Here's Sally's loan approval rules. Try playing with them — adjust the thresholds and see what happens:

The Power of Self-Service

Now let's say Sally wants to change these rules. Maybe she wants to change the income threshold from $50,000 to $40,000. She could cut a ticket with the dev team, but that might take a lot of time. With a self-service rules engine, Sally has the ability to change the rules herself — she can delete rules she doesn't need and add new ones.

Try it yourself — scroll back up to the rules engine and check the Self Service checkbox. You'll see delete buttons appear on each rule, plus a + New Rule button that lets you add your own. This is what self-service looks like in practice: no developers needed.

The value of self-service can be very high when there's a high volume of changes.

So Which Approach is Better?

We've talked about scripting and we've talked about a rules engine — and they're different beasts. So which is better? Let's go through it feature by feature:

Self-service is a definite no for scripting — Sally's not going to be writing computer code. But for the rules engine, it's easy to make it self-service.

Testing: Scripting gets a check — most dev teams have unit tests and ways to test for regressions. For rules engines, you need to make sure you build testing in, and sometimes people forget.

Versioning: What if Sally wants to go back to the rules from a couple weeks ago? That's easy with scripting — most code has versioning tools like Git. For a rules engine, the dev team might not think to build in versioning.

Approvals: What if Jim works with Sally, and Sally wants to approve Jim's changes before he pushes them in? On a dev team, approvals are easy. For the rules engine, that's something that needs to be built in.

Flexibility: Code is infinitely flexible — you can do anything you can imagine. But a rules engine is more limited. You could add more fields and the ability to loop over constructs, but at a certain point it'd be easier to just build a script.

So which is better? It really depends on the context. The value of self-service can be very high if there's a high volume of changes where it's actually worth building out testing, versioning, approvals, and making a very flexible rules engine. But again, it depends on the context.

Knowledge Check

Optimization

Optimization is much more than just an AI technique — it's actually a way of seeing the world. And once you learn to recognize optimization problems, you'll start seeing them everywhere.

I think a lot of people get lost in the details and the math, but they forget how simple the core idea actually is.

So what is optimization?

At its heart, optimization is really just about finding the best option.

And since this is an AI course, what we really care about is how to get a computer to find the best option.

To get a computer to find the best option, we need to be precise about what we mean by best and what we mean by options.

Let's start with a very simple example.

Imagine someone tells me: "Go to the store and get the largest red apple."

That's an optimization problem.

"Best" means largest. The "options" are all the red apples in the store.

Simple.

Shortest Path Problems

Of course, in real problems, the options aren't usually physical things like apples. They're often more abstract. To understand this, let's go through another example.

In this one, we're going to use optimization to find the best route for shipments.

Meet Judy, a logistics coordinator.

Can we think of finding the best route as finding the best option? Absolutely.

In this case, "best" means shortest distance. And the "options" are all the possible paths from the start city to the end city.

And since this a well defined optimization problem, a computer can do it for us.

Here's an example where the computer finds the shortest path from New York to Los Angeles.

This is essentially what Google Maps is doing.

But what's the computer actually doing here?

Conceptually, it's searching through a huge space of possible routes until it finds the shortest one.

Problems like this are called shortest path problems, and they show up everywhere.

Let's take a quick second to talk about terminology. The circles that represent cities are called nodes. The paths between them are called edges. Computer scientists call structures like these — with nodes and edges — a graph.

Graphs aren't just good for finding directions. Many game AIs — say, for chess — represent the game as a graph. Each game position is a node, every possible move is an edge, and the AI searches for the best sequence of moves on that graph.

Social networks can also be modeled as graphs. Have you ever played "Six Degrees of Kevin Bacon," where you find how closely related an actor is to Kevin Bacon? You can model this as a shortest path problem on a graph — the nodes are actors, and the edges are the movies they've starred in together.

So that's one type of optimization problem: searching through a graph.

Integer Programming Problems

But not all optimization problems are graph problems. Let's look at another one together.

Meet Jill, who runs a furniture factory.

Once again, this is an optimization problem. "Best" means maximum revenue, and the "options" are all the different production plans that fit within the resource limits.

For example, here are some different options and the revenue each one generates:

Here's what it looks like when we let a computer search through the space for us. The computer tries different combinations, checks which ones violate the constraints, and keeps track of the best one it's seen so far.

Problems like this are called integer programming problems. The name sounds fancy, but the idea is still the same: define what "best" means, define what the valid options are, and let the computer search.

The Four Pieces of Every Optimization Problem

If we want to impress applied mathematicians at parties, we need to use the terminology that they use.

It turns out that almost every optimization problem has the same four pieces.

First, we need to define an objective — what we're trying to improve. Next, we choose whether we're maximizing or minimizing that objective. These two things together define what we mean by "best". For example, in the apple problem, we define "best" as maximizing the objective of apple size.

We also break the options into two pieces. The search space is what we're looking through. And finally, the constraints are the rules or limits that need to be followed. So going back to the apple problem, the search space could be all the apples in the store, but the constraint is that the apple must be red.

Let's map that language onto the examples we've already seen.

In the apple example:

In the map example:

In the factory example:

Optimization Is Everywhere

Once you learn to see problems this way, you start seeing optimization problems everywhere.

For example:

This is why optimization is so powerful.

It takes messy real-world problems and turns them into precise questions that computers can answer.

The Limits of Optimization

But there's a catch.

Optimization also has real limits.

Computational Complexity: Sometimes it takes too long to find the right answer — so often we settle for "good enough." For example, Google Maps doesn't really find the best route. A lot of the time, there are just too many paths for it to check. Instead, it finds a good enough route quickly.

Model Complexity: Sometimes the real world is so complicated that making a realistic model is too hard. For example, in inventory management, accurately modeling every cost of inventory and the random nature of customer demand would be almost impossible. In practice, you have to make concessions and understand that your model is simpler than reality.

Unclear Objectives: Sometimes it's hard to define exactly what "best" means. Many real-world problems have multiple competing objectives, or objectives that can't be defined mathematically.

And this points to a fundamental limitation: optimization only works as well as the objective we give it.

Which raises an interesting question. What if, instead of carefully writing down the objective, we let the computer learn what "good" looked like from examples?

That's the next big AI technique we'll cover. It's called machine learning.

Knowledge Check

Machine Learning

Learning from Examples

At its core, machine learning is about teaching a computer to learn — and specifically, learning from examples.

To understand what that means, imagine a child who's never seen an alligator or a crocodile. You sit down with them and a stack of flashcards. On the front of each card is a photo — a long snout, bumpy scales, maybe a toothy grin. On the back is the answer: alligator or crocodile.

The child flips through card after card. At first, they're just guessing. But after enough cards, something clicks. They start noticing patterns — the wider, U-shaped snout usually means alligator. The narrow, V-shaped snout usually means crocodile. They didn't memorize a rulebook. They learned from examples.

Try it yourself — can you tell them apart?

Crocodile or Alligator? Card 1 of 5

Loading image…

Crocodile Alligator

Next →

🎉

Quiz Complete!

You got 0 out of 5 correct

Try Again

That's exactly what machine learning is. Instead of flashcards, we give a computer data. Instead of photos on the front, we give it inputs. And instead of answers on the back, we give it correct answers to learn from. We'll walk through a concrete example in just a moment — but first, let's name this idea.

This is the technique behind self-driving cars, recommendation systems, and conversational AI. Let's see it in action.

Our First Algorithm

Remember Sally, our rules-based loan officer? Let's meet her colleague Marcus, who has a problem we can help solve.

Marcus can't articulate his rules — but that's okay. We don't need him to spell out the rules. We just need examples. And Marcus has 15 years of loan decisions we can learn from.

We'll start by looking at a plot. The X axis shows annual income, and the Y axis shows credit score. We'll plot all of Marcus's past loans on the graph. Each point represents a past loan application — circles are loans that Marcus approved, and X marks are loans he denied.

Let's submit a new loan application. We do this by clicking on the graph. On submission of a new loan application, we find the three closest loans. If the majority of the neighbors are circles, we approve. If the majority are X marks, we deny. This algorithm does a fantastic job of imitating how Marcus makes decisions.

Notice what just happened. Marcus couldn't explain his rules, and yet we got a computer to make decisions just like him. We fed it examples, and it figured out the pattern on its own.

Nearest Neighbors

This technique is called Nearest Neighbors. When classifying a new data point, we take a vote of its 3 nearest neighbors — the closest existing data points.

Now, nearest neighbors is just one machine learning algorithm. There's actually a whole toolbox of machine learning algorithms, each with different strengths.

Features and Labels

Now that you've seen machine learning in action, let's give proper names to the different pieces. Remember the alligator flashcards from earlier? That metaphor is about to come in handy.

Think back to our flashcards. The front of the card — the photo of the reptile — is what we look at to make our guess. In machine learning, we call these the features. They're the inputs.

The back of the card — the answer, alligator or crocodile — is what we're trying to predict. We call this the label.

In Marcus's loan example:

What Machine Learning Can Do

Let's go through more examples to really understand features and labels.

If you're predicting the price of a house, we're lucky — the label (sale price) is easy to find. We just need to look up past sales.

The reason we use machine learning for fraud detection is because banks can't anticipate every type of fraud. Criminals are just very creative. But by studying thousands of past transactions labeled as fraud or legitimate, the computer can spot patterns that would be impossible for a human to write rules for.

Every time you click "mark as spam," you're contributing a label to a training data set somewhere. You're helping a machine learning algorithm have more examples to learn from.

Try to come up with your own examples! What are the features and labels for predicting the weather? What about recommending a song? Take a moment to think through a few.

What's Next?

Now that we have a good understanding of what machine learning looks like — how we can learn from examples to map features onto labels — the next question is: how do we know if the algorithm is any good?

That's what we'll explore in the next lesson.

Knowledge Check

Evaluating Machine Learning Models

How Good Is It?

A few weeks into our project, Marcus stops by to check on progress.

Good question! But how do we even answer that?

We can't just check if the algorithm memorized Marcus's past decisions — that would be useless for new applicants. We need to know if it actually learned the pattern.

Remember the child learning alligators and crocodiles from flashcards? Imagine setting a few cards aside before they start studying. After they've gone through the rest of the deck, you quiz them with the held-out cards. If they get those right, they actually learned — they didn't just memorize the cards they'd already seen.

That's exactly what we do in machine learning. Before training, we hold back some examples — maybe 20% of them. We train the algorithm on the remaining 80%, then check: can it correctly predict the labels for examples it's never seen?

This is called the train/test split, and it's one of the most important ideas in machine learning.

So we run our test. Out of 100 loan applications the algorithm had never seen, it agreed with Marcus on 85 of them. That's 85% accuracy.

But here's the thing — accuracy alone doesn't tell the whole story.

Imagine Marcus approves 90% of loan applications. A lazy algorithm could just say "approve everyone" and achieve 90% accuracy without learning anything useful. It would approve every risky borrower too!

What we really care about is: what kinds of mistakes is the model making?

These four outcomes tell us much more than accuracy alone. A bank might care deeply about false positives (approving bad loans costs money), while a hospital screening for disease might care more about false negatives (missing a sick patient could be fatal).

Marcus gets it. Different mistakes have different costs, and a good evaluation considers not just how many errors, but what kind.

What's Next?

Now that we know how to evaluate machine learning models, let's explore how these algorithms actually learn — and discover that training is secretly an optimization problem.

Knowledge Check

How Machines Learn

When Simple Isn't Enough

In the last lesson, we saw how nearest neighbors can learn from examples — and it works great on problems like the loan problem, where we just have two features. But what if we wanted to predict whether a photo contains a cat or a dog?

The features in this problem would be every pixel inside of the image. So in a sense, there are thousands and thousands of features. Nearest neighbors isn't going to work very well here. We need something more powerful.

Here's another example. Let's say we want to do autocomplete:

When you type "see you" and your phone suggests "tomorrow," that's predicting the most likely next word. Again, nearest neighbors won't cut it here.

For problems like these, we need neural networks — models that can learn very complex relationships.

Why "deep"? In a neural network, parameters are organized in layers. Data flows in one side, gets transformed by the first layer of parameters, then the next, then the next — each layer building on the patterns the previous one found. "Deep" just means lots of layers.

Here's how these concepts nest inside each other:

But how do neural networks actually work? How does a machine "learn" anything? To understand that, we're going to look at a much simpler problem. The core ideas are the same — just easier to see.

Predicting Home Prices

Meet Brett, a real estate appraiser with decades of experience.

Brett has noticed a pattern: bigger houses tend to cost more. But doing every appraisal manually is slow and expensive. Can we capture Brett's expertise in an algorithm?

Here's Brett's past appraisal data. Each dot is a house he appraised — the position shows its square footage and what it sold for.

Can you see the pattern? Larger houses (more square feet) tend to sell for higher prices. This is the kind of relationship Brett has learned to recognize through years of experience.

Capturing the Pattern

What if we could draw a line through this data that captures the pattern? A line that shows: "for any given square footage, here's roughly what the price should be."

This curve represents Brett's expertise turned into math. It captures the pattern: how price tends to change as houses get bigger.

Click anywhere on the graph to predict the price for any house size. That's the power of having a good model — instant predictions based on learned patterns!

Features and Labels

This is a machine learning problem, just like Marcus's loan decisions. But instead of predicting approve or deny, we're predicting a number.

The feature is the input we use to make the prediction — in this case, the square footage of a house.

The label is what we're trying to predict — the home price.

Finding the Right Curve

But here's the question: how did we find that curve in the first place?

The curve is defined by parameters — numbers that control its shape. Different parameter values give you different curves. Some curves fit the data well. Others... not so much.

Let's peek behind the curtain. Here you can see and adjust the parameters yourself:

Try this: Move the sliders around and watch how the curve changes shape. Some parameter combinations make the curve fit the data well. Others miss badly.

But how do we know which curve is "best"?

Measuring the Fit

We need a way to measure how well a curve fits the data. The answer is simple: look at how far each prediction is from the actual price.

Each red dashed line shows the error for one data point — how far off the curve's prediction is from the actual price.

Try to minimize the error by hand. Can you get it below $20K? It's harder than it looks!

Notice what you're doing: you're searching through different parameter combinations, looking for the one with the lowest error.

This Is an Optimization Problem!

Wait a minute. We're trying to find the best parameter values. The options are all the different curves we could draw. And we want to minimize the error.

Does that sound familiar?

So to teach a machine, we just need to search through the space of possible curves and find the one that minimizes the error. But how do we search efficiently?

Watching the Machine Learn

When you moved the sliders, you probably noticed something: small changes to the parameters cause small changes to the error. The error surface is smooth — not random.

This smoothness is the key. It means the computer can search efficiently. Click Train and watch it find the best curve automatically:

The algorithm found the best parameters in a fraction of a second. But how did it search so efficiently? It used a technique called gradient descent.

Gradient Descent: How It Works

Think of it like finding the bottom of a valley in thick fog. You can't see the whole landscape, but you can feel which way is downhill. Take a step downhill. Feel again. Step again. Eventually you reach the bottom.

That's what the computer does — it starts with random parameters, calculates the best small change to reduce error, makes that change, and repeats until the error stops improving.

Let's see this step by step. Click Step to perform one gradient descent step at a time:

Each time you click "Step," the algorithm:

1. Calculates the gradient — which direction to adjust each parameter
2. Makes a small adjustment in that direction
3. The error goes down (most of the time!)

Try clicking Step many times. Notice how the curve gradually fits the data better and the error decreases. When you clicked "Train" earlier, this is exactly what happened — just thousands of times faster.

Training and Inference

Now we can give names to what you just experienced.

When you clicked "Train" or "Step," that was training — the algorithm learning from Brett's data.

When you clicked on the graph to predict a house price, that was inference — using what it learned to answer a new question.

In practice:

• Training can take hours, days, or even weeks for large models
• Inference is typically instant — milliseconds or seconds

When you use ChatGPT, you're doing inference. OpenAI already did the (very expensive) training — today's frontier models cost an estimated $500 million or more to train.

Scaling Up

Brett's model has just 3 parameters (a, b, and c) and 1 feature (square footage).

Real neural networks are the same idea — just much, much bigger. Here's how parameter counts have grown:

Each jump is roughly 1,000 times more parameters than the last. Yet the core technique — gradient descent — remains the same.

Peeking Ahead: How LLMs Learn

We'll cover LLMs in much more depth later in the course, but now that we understand how machines learn, let's peek ahead at how all of this connects to the AI tools you're already using.

Remember the autocomplete example from the start of this lesson? That's actually how Large Language Models (LLMs) like ChatGPT are trained.

Here's what that looks like in practice — given some text, the LLM predicts the most likely next word:

We're going to spend a lot more time on LLMs in the next lesson, but while machine learning is fresh, let's understand what the training data actually looks like.

The idea is simple: take a sentence and turn it into a series of feature-label pairs where the features are the words so far and the label is the next word.

For example, take the sentence "The boy threw the ball." The training data looks like this:

That's it. Take billions of sentences from the internet, break each one into feature-label pairs like this, and train a neural network to predict the next word. That's how an LLM is built.

But just how much data and compute are we talking about? The numbers are staggering:

Training is enormously expensive — it takes months, thousands of specialized chips, and hundreds of millions of dollars. But once training is done, inference is nearly instant. When you type a message into ChatGPT and get a response in seconds, the hard part was already done. You're just using a model that someone already spent months training.

But the core process is identical:

1. Define a model with adjustable parameters
2. Feed it data
3. Use gradient descent to find the parameters that minimize error
4. Use the trained model for inference

The magic of modern AI isn't a fundamentally different technique — it's the same optimization process, just at an incredible scale.

What's Next?

We've seen how machine learning works under the hood: it's optimization. We define a model, feed it data, and let gradient descent find the best parameters.

But there's a problem. Both Marcus's loan algorithm and Brett's price predictor learned from labeled data — examples where we already knew the right answers. What happens when we want AI that can take action in the world, not just make predictions?

That's where agents come in. See you in the next lesson.

Knowledge Check

Agents

In this lesson, we'll explore agents — AI systems that don't just respond to you, but work with you. They perceive, decide, and act. They use tools. They pursue goals.

We'll build an agent step by step — and discover the two ingredients every agent is made from.

Let's start by meeting someone who desperately needs an agent — for the most critical of tasks.

Building a Menu Assistant

Meet Jamie

Let's build Jamie an agent that can help — one step at a time.

Step 1: Just Chat

Here's a basic AI assistant. Try telling it about Jamie's preferences, then ask for a recommendation:

The text you just typed is called a prompt — it's how you tell the AI what you want.

It works! But there's a problem — one you've probably experienced yourself if you've ever used ChatGPT or Claude. General-purpose assistants don't know anything about you.

Step 2: Remember Jamie's Preferences

What if we could build Jamie a custom agent that actually understands their needs? Let's give the AI instructions before the conversation starts? See the green text below — that's what the AI knows before you even start chatting:

This hidden message is called a system prompt — it sets up the AI's behavior before the conversation begins.

Much better! The AI already knows Jamie. But there's still a problem...

Step 3: Give It Access to the Menu

What if the AI could actually see the menu? That's where tools come in — they let the AI reach out and get information it doesn't have.

Now the AI can use the menu tool to look up real dishes and prices — try asking about Greg's Place or The Corner Spot!

Retrieval-Augmented Generation

What we just did has a name: Retrieval-Augmented Generation, or RAG. Let's break it down:

What is an Agent?

What we just built is an agent. And here's the key insight:

The magic isn't in any single piece. It's in the combination:

• Prompts alone = AI that can talk but can't do anything
• Tools alone = capabilities with no intelligence to use them
• Prompts + Tools = an agent that can understand what you need AND take action

This means if you want to build your own custom agent, you only need to think about two things: What should it know? (prompts) and What should it be able to do? (tools). That's it. The rest is just the LLM doing what it does best — understanding language and figuring out what to do next.

Building Your Own Agent

So you want to build an agent? Great! But here's something most tutorials won't tell you: the hardest part isn't the technical setup.

The hardest part is knowing what the agent should actually do — and whether it's doing it well.

After watching many teams build agents, I've noticed a pattern. The successful ones have three key people involved:

Why all three? Because the most common failure mode is building something technically impressive that nobody wants to use — or that confidently does the wrong thing.

• Without the Expert, your agent might give plausible-sounding but wrong advice
• Without the User, you'll build for a workflow that doesn't match reality
• Without the Builder, well... you won't have an agent at all

Wrapping Up

Let's bring it back to where we started:

That's exactly right. Every agent you use — whether it's helping you write code, plan a trip, or choose dinner — is built from the same ingredients:

1. Prompts that shape what it knows and how it behaves
2. Tools that let it reach out into the real world (like RAG for retrieving data)

The magic isn't in any single piece. It's in the combination — and in knowing when to use them.

In the next lesson, we'll look under the hood at how all of this actually works — from tokens and text prediction to why tools matter so much.

Knowledge Check

How Agents Work

In the last lesson, we built an agent for Jamie — combining prompts and tools to create a menu assistant. Now let's look under the hood at how all of this actually works.

How Does an LLM Work?

Under the hood, agents are powered by a Large Language Model (LLM). Let's break down what each word means:

So an LLM is just a text prediction machine — a very large one. You give it some text, it predicts what comes next.

Let's see what the model predicts:

This is a prediction problem. Given some text, the model predicts the most likely next word. That's all an LLM does — over and over again.

Generative AI

But what happens when we keep clicking? Try it — keep selecting tokens and watch a story emerge:

What Are Tokens?

Try this one:

Notice something odd? The model predicts E, then iff, then el — not "Eiffel" as a whole word. Why?

To understand this, we need to talk about what LLMs actually predict. Not words, not characters — something in between called tokens. But first, a question:

Characters are flexible but slow. Words are fast but rigid. The solution? Tokens — a middle ground.

When you see "tokens" in AI pricing or context limits, this is what they mean. Most modern AI models have context windows of 100,000+ tokens — roughly the length of a novel.

It's Probabilistic

Let's see what the model thinks:

You might get The Beatles, The Rolling Stones, The Who, or something else entirely. That's because LLMs are probabilistic — they don't deterministically pick the "best" answer. They sample from a distribution of likely next words.

This is why chatting with an AI can feel almost emergent — like it has a mind of its own. It's not following a script. Each time it generates a response, it's sampling a different path through possibility space. Ask the same question twice, get different answers. That's also why "regenerate response" is a feature in every AI chat app.

Temperature

But how much randomness? That's controlled by a setting called temperature.

Most AI apps use a temperature between 0 and 1. Want a factual answer? Turn it down. Want creative writing? Turn it up.

Stop Tokens

We've seen how an LLM generates text — predicting one token at a time. But how does it know when to stop? Without a stopping mechanism, the model would just keep predicting forever.

The answer is a special token called a stop token (sometimes called an end-of-text token). It's not a word or a punctuation mark — it's a signal that says "I'm done."

Try this — keep clicking tokens until the model finishes its thought:

Once the model has said everything it needs to say, the most likely next token becomes — the stop token. That's how the model signals "I'm finished generating."

From Completion to Chat

Here's the familiar chat interface from earlier:

Now here's what the model actually sees — the exact same conversation, but as plain text:

Same conversation. The chat interface is just a prettier way to build up that text. Every time you send a message, the system adds "User: [your message]" to the text, and the model predicts what comes after "Assistant:".

This progression — from autocomplete to conversation — mirrors how AI actually evolved. For years, language models were essentially "fancy autocomplete," predicting a word or two before losing coherence. Then researchers discovered that scaling up this same mechanism — more data, bigger models, better training techniques — produced surprisingly capable systems. By 2020, models could follow instructions given a few examples. By 2022, ChatGPT showed the world what happens when you wrap a next-word predictor in a chat interface and train it to be helpful. Add tools, and that same autocomplete becomes an agent that can browse the web, write code, and book flights. The core mechanism never changed — it's still just predicting the next token.

The Hallucination Problem

Have you heard the term hallucination in the context of AI? It's one of the most important concepts to understand.

The model is only predicting what sounds right — not what's true. Watch it try to do math:

Try it a few times. The model isn't calculating — it's guessing. It will confidently produce a number that looks plausible but is almost certainly wrong.

Why Do Hallucinations Happen?

Remember — an LLM is a text prediction machine. It doesn't "know" facts the way you do. It learned patterns from billions of pages of text, and when you ask it a question, it generates whatever sounds like a good answer.

That works surprisingly well for many things. But it completely breaks down when the answer requires precision — math, dates, citations, real-time data. The model has no way to verify what it's saying. It doesn't think "let me check if this is true." It just predicts the next token.

So if LLMs can't actually compute or verify facts, how do we fix this? The answer is surprisingly simple: give them tools. Just like we gave Jamie's agent access to a menu, we can give an LLM access to a calculator — and let it reach out for the right answer instead of guessing.

How Tools Work Under the Hood

Same question. But now the model can reach out for help — and get it right.

Remember Jamie's menu problem from the last lesson? That same pattern — where the agent looks up a menu before making a recommendation — is RAG (Retrieval-Augmented Generation) in action. The model generates [[menu: gregs-place]], the system retrieves the menu data, and the model uses it to generate an informed response.

Toggle "Show tool calls" to see when the agent reaches for its calculator — or hide them to see what the experience feels like to a normal user.

Now You Try It

Here's the key insight: you can use the exact same syntax. The system doesn't care who typed the tool call — it just pattern-matches on the text.

Try being the agent yourself — type [[calc: expression]] when you need to compute something:

Notice what happened:

• You typed a message with the tool call syntax
• The system recognized the pattern [[calc: ...]]
• It extracted the expression, computed it, and replied with the result
• You used that result to answer the user's question

Same rules. Same system. Whether you type the tool call or the model generates it, the system handles it identically.

Beyond Calculators

The calculator is simple, but the pattern works for any tool:

The model doesn't "know" how to check the weather. It just learned to output [[weather: Tokyo]] when it needs weather data — and trusts that the system will handle it.

Tools Transform What's Possible

This is why modern AI assistants have access to web search, code execution, calculators, and more. The LLM provides the reasoning and conversation — the tools provide the connection to reality.

Prompting vs Training

Now that we've seen how LLMs work under the hood, there's an important distinction to make. There are two very different ways to shape how an LLM behaves: training and prompting. And they're done by very different people.

Training is the same process we learned about in the deep learning lesson — adjusting millions of parameters through gradient descent until the LLM gets good at predicting the next token. That's done once, by the company that builds the model, and it takes weeks of compute time on thousands of GPUs.

Prompting is something unique to agents. It's the art of writing clear instructions — system prompts, user messages, tool descriptions — so the LLM behaves the way you need it to. And it's a skill that anyone can learn.

Here's the good news: you don't need to train an LLM to build a powerful agent. The hard part — training — has already been done for you. Your job is the prompting: figuring out what the agent should know, how it should behave, and what tools it needs.

Knowledge Check

AI Systems

We've now covered four AI techniques — rules-based systems, optimization, machine learning, and agents. Each one is powerful. But here's the thing: every single one of them is narrow.

Remember how we talked about narrow intelligence in the first lesson — AI that's just good at one thing? All four of these techniques are narrow. Even agents.

Agents feel general because you can talk to them about anything. But when you actually deploy an agent in a business, you give it a specific system prompt, specific tools, and point it at a specific job. You build a customer support agent, a food ordering agent, a code review agent. Each one does one thing well.

So if every technique is narrow, how do organizations actually solve complex real-world problems?

Stitching Techniques Together

The most powerful AI applications aren't built from a single technique. They're pipelines — sequences of steps where each step's output feeds into the next.

Let's revisit some familiar faces and see how their problems are actually solved in the real world.

Example 1: Loan Approval

Remember Sally, who approves loans with simple rules? And Marcus, who uses machine learning to spot patterns in data? In a real lending company, you wouldn't pick one or the other — you'd combine them.

Notice: the rules engine comes before the ML model. It handles the easy cases cheaply so the expensive model only processes applications that actually need it.

Example 2: Inventory Management

Remember Judy, who optimizes delivery routes? Her company also needs to decide how much inventory to stock. That problem requires two techniques working together.

Neither technique works alone. The ML model can predict demand, but it can't decide how much to order — that's an optimization problem. The optimizer can find the best order, but only if someone tells it how much demand to expect — that's a prediction problem.

Example 3: Customer Support

Here's a customer support system that stitches three techniques together:

Notice that a script calls the agent — not the other way around. The cheapest, fastest tool handles the bulk of requests. The powerful (and expensive) agent only kicks in when it's actually needed.

Simpler Is Often Better

Here's something that surprises a lot of people: you don't always need the fanciest technique.

This doesn't mean agents and ML are never the right choice. But always ask: could something simpler work here?

Agents and neural networks can often be replaced with rules engines or scripts — especially when the logic is well-understood and doesn't change much. And sometimes it makes sense for scripts to call agents only for the cases that truly need flexibility.

Where Does the Intelligence Come From?

Here's a question that doesn't get asked enough: if AI is "intelligent," where does that intelligence actually come from?

The answer depends on the technique — and it's either data, expertise, or both.

This has huge practical implications for what you need to invest in.

Machine learning needs data. More data, better data → better predictions. If you don't have enough labeled examples, ML won't work — no matter how fancy the algorithm. That's why data collection and data quality are so important.

Rules-based systems need expertise. Sally doesn't need 10,000 loan examples to do her job. She just needs her two rules. But someone has to know those rules — and that someone is a domain expert, not an algorithm.

Optimization needs mathematical expertise. Judy's delivery routes need someone who can define the objective, the search space, and the constraints. That requires understanding the math of the problem — not necessarily a big dataset.

Agents need both. Training the underlying LLM requires massive amounts of text data. But making an agent useful for a specific task — writing the right system prompt, choosing the right tools, evaluating whether it's doing a good job — requires deep expertise about the domain.

Putting It All Together

Here's the mental model to take away from this course:

Knowledge Check