TL;DR - you can immerse yourself in real-life* magic* Quidditch commentary in the style of Championship Manager 00/01 over at QuidditchPL.com

*insert Arthur C. Clarke quote here

Intro

Things are moving so fast. I planned to revisit the GPT portion of my OCR blog scanner when “edit mode” was released, but “edit mode” is now old news! I’ll just wait for multi-modal GPT-4 and get it to do the entire ML pipeline. Simple.

While I hit F5 waiting for this week’s breakthrough in AI, a question needs to be answered.

Can I (and should I) build Championship Manager 00/01 style text commentary for Quidditch?

Can I? GPT can seemly write anything you tell it to, so surely yes. Should I? With obligatory scope creep to add GPT-triggered crowd noise and stable diffusion-generated graphics—yes please!

Championship manager — A youth well spent

Back in 2000 there was still time for me to become player-manager of Hereford United.

I practised my free kicks in the garden until darkness (and the hedge) hid the ball. Then, by the flickering light of CRT I finessed my formations, scoured my scouts’ reports, and placated my players using the original cutting-edge tooling that was Championship Manager 00/01. (00/01 stands for the 2000/2001 football season)

Sadly I did not become the player-manager of my dreams. Hereford United ceased to exist in 2014. I’m sorry I could not save you :‘(

All the real-world football scouts out there (who probably maybe use football manager in their day job) would agree that the peak match engine for the series around the 00/01, 01/02 era. The engine goes by the name of CM3 if you’re counting generations of the game.

In CM3, you hear the chatter of the crowd, the blast of the referee’s whistle and the thud of boot on ball as the match kicks off.

Text commentary is presented line by line, coloured to match the strip of your team (with some debatable contrast for teams like Barcelona), and flashing aggressively when the ball hits the back of the net.

At times you’d watch the build-up of each move, immersed in the atmosphere while planning your next tactical move. You’d hold your breath as “He shoots…” is held on the screen for a lingering second before you add to the crowd’s roar. Your 85th-minute substitute, Andy Williams, is the one who breaks the deadlock with 2 minutes left on the clock. We can do it.

He shoots…

Perhaps on another day, you’re not merely rooting like a fan in the stands. No, your the best in the business for a reason. This match is barely even significant—just one step in the journey to managerial glory. You set the commentary to maximum speed. The line dart across the screen like Mathieu Manset passing defenders by. This time you only clench your fist in appreciation as the final whistle blows—a job well done. Onwards. Upwards.

On a serious note, CM3 is testament to how even the simplest of mediums can transmit the joy of sport. Why worry if it’s Pro-Evo or FIFA that has the best grass physics? Instead, you can read the same commentary cliche for the 100th time and enjoy it like it was the first. Perfection.

…the ball flies into Row Z! What a miss!!

QPL v5?

The original idea to build a Quidditch simulation game dates back to University. My lab partner was reading Harry Potter for the first time, and we had just finished a project where we used Python to simulate the behaviour of photonic quantum chips.

What do you get when you bring a photon of sleep deprivation and a photon Harry Potter together so that their evanescent fields overlap?

Yep, we decided we wanted to simulate some more stuff. Enter QPL: The Quidditch Premier League.

Aside: this is a very abridged origin story. In reality, we wanted to try and orchestrate a cup of tea with J K Rowling (this is pre any controversies). We imagined we could get her attention by building a Quidditch-themed simulation game that would somehow raise $$$ for charity and make the world a better place™️.

The Physics final exams finished a week before most other subjects. So, like any typical student who has just spent four years studying at all times of day, walking up hills in the snow to 9 am lectures etc etc… we spent the first week of absolute freedom indoors with the blinds closed working on an app.

That was v1.

In the years that followed, I revisited the project whenever I wanted to learn something new and there wasn’t a more tangible side project available.

Many of my early forays into UX and Design were prototyping and user-testing QPL. There was a JavaScript rewrite, followed by a ReasonML rewrite, followed by a Typescript rewrite.

And now, the year is 2023, and it’s time for me to pass over the reins to my successor: GPT-3.5. This will be the final rewrite, as I assume any future iterations will be handled by AI.

Over to you: AI

Okay, on to the build. We want to simulate a game of Quidditch. What does that look like?

• A set of match events
• Text commentary for each event
• Basic stats for each event (was there a goal? who scored it?)

From this, we can power a simple UI that

• Plays back the game, event by event
• Displays the text commentary a la Championship manager
• Updates a score counter whenever necessary (if you didn’t already know, you get 10 points for throwing the Quaffle through one of three hoops, and 150 points if your seeker catches the golden snitch)

How did we achieve this before magic / AI?

How did we do anything pre Attention Is All You Need?

The original Quidditch match engine, dating back to my uni days:

• Took in hand-crafted CSVs full of player names and stats (Birch has shooting ability of 0.8 and flare of 0.9 kind of thing)
• Crunched some numbers and outputted high-level events weighted by each teams’ players’ attributes. e.g. Home team scores a goal, snitch is sighted but not caught, etc.

The exact weights here were determined by:

• What made a good game (e.g. How many events should there be on average? How many of these should be goals?)
• What makes a good season? How often should an underdog beat the best team in the league?

The match engine then broke each event into several event steps. First, it determined which player was mostly likely to have scored the goal/ been hit by a bludger etc. Then, working backwards, it chose what would happen before this step. Did the chaser retain possession? receive the Quaffle via a pass? Who had the Quaffle before? Knowing that it could choose what happened before this step and so on.

Finally, lines of templated commentary was chosen at random for each event step line

Time for GPT-3.5

Post ChatGPT launch, having seen all the amazing things people have created with LLMs, the time was right—it had to be—to wheel out the Quidditch Premier League project and give it a go.

AI overkill?

Well, the non-AI approach is already pretty good! It produces results comparable with CM3.

However, it did have limitations. In the non-AI version, each commentary line is selected at random and has templates that could be filled by pretty much any player, within any team, within any event, within any match. So the text has to be generic and cannot be context aware. This means that, for example, the commentator can’t reference what happened earlier in the match, or start talking about what the manager said in a news conference last week. Very restrictive.

So the real opportunity presented by GPT-3 is to generate commentary that can see beyond a single match event.

Where are we right now?

As already acknowledged, things are moving fast. As of writing:

• GPT-3.5 has been released, and the chatCompletions API is in beta
• I’m on the GPT-4 and plugins waitlist 🙌

This means that, at least until I get access to GPT-4, we’re restricted to the GPT-3 limit of 4,096 tokens.

The Approach—let’s write some commentary!

You could definitely just ask GPT-3 to write the full commentary for a game of Quidditch, and it would happily manage a Rowling-esque 4000 tokens of sporting magic. However, we want this to be as much an ode to Championship Manager as it is to a fictional sport, so we should make sure we have enough data to power a UI:

• The commentary should be broken up into events
• Each event should have match stats (so we can update a score counter)
• And more importantly, the commentary should not be mere fiction; It should rooted in the cold hard bits that have passed from laptop to laptop since I left uni. I’m talking REAL LIFE DATA. LONG LIVE player-stats.csv!

What are our limitations? We know generative pre-trained transformer models aren’t great at counting, and player-stats.csv will eat up our token limit pretty quickly. So, to take a little bit of the load off GPT-3, I will re-use the event generation code from an older version of QPL.

The event generation code spits out a match that looks something like this:

type Match: {
  meta: { teams: { home: Team; away: Team } }
  events: Array<MatchEvent>
}

where (a simplified) MatchEvent looks like this:

type MatchEvent = {
  type:
    | "GOAL"
    | "WONDER_GOAL"
    | "MISS"
    | "TERRIBLE_MISS"
    | "GOAL_SAVING_BLUDGER" // ...
  data: { changes: MatchStats }
  steps: Array<{
    type:
      | "RETAIN_POSSESSION_PASS"
      | "RETAIN_POSSESSION_DRIBBLE"
      | "LOOSE_POSSESSION_BLUDGER" // ...
    teams: {
      attacking: {
        team: Team
        players: {
          inPossession: Player<PlayerRoles.Chaser>
          inPossessionInNextStep: Player<PlayerRoles.Chaser>
          // ...
        }
      }
      defending: {
        team: Team
        players: {
          defending: Player<PlayerRoles>
          // ...
        }
      }
    }
  }>
}

We can then have a chat with GPT-3.5. Something along the lines of:

• Here’s some pre-match context for you. Can you write an intro?
• Great, now here’s a summary of the first match event. It has these steps in which these players were mentioned. The score before was 0-0, and the score after is 10-0. Can you write commentary for this event?
• Thank you, so kind of you. Here’s the second event for the match. The score before was 10-0… the score after is 20-0…
• etc

Running out of tokens

As we’re chatting over the chatCompletions API, we need to send the full history of our conversation with each new prompt if we are to have any hope of producing context-aware commentary. 4096 tokens is about 3000 words. Once you’ve included the history of the chat and a load of prompts, this doesn’t get you very far.

To tackle this, I’ve thrown in two complementary pieces of prompt engineering.

First, we can buy back a few extra tokens by attempting to condense the prompt as much as possible. With some very hand-wavy A/B testing, we can remove anything that isn’t really contributing to the goal while keeping the good stuff.

Second, we can compress the messages we send back to chatCompletions by rolling up the history of the conversation. There are many different ways we could approach this. For now, I’ve removed all incidental messages and I’ve added the following context to the event prompt:

• The commentary from the previous event (in the hope that the assistant will attempt to keep some continuity going)
• A short summary of the match so far, generated by concatenating together the results of asking GPT to:

Plz summarise this move from a Quidditch match in a few a words as possible
(no more than 20 words)

This should be enough for the assistant to understand how each player has contributed to the match so far.

Personality

Again, there’s so much potential for endless prompt engineering here. Below is the system prompt, which assigns the assistant its rules of engagement

const SYSTEM_MESSAGE = `Imagine you are providing live radio commentary for a Quidditch game.
You are enthusiastic and quick to praise players of both teams.
You don't like to repeat yourself when describing the match.
You are talking as the move unfold so ALWAYS write in the PRESENT TENSE and don't change tense.`

I’m stressing that the assistant should write in the present tense as GPT was not getting the message.

Pre-match Context

The power of using an actual AI instead of just randomly selecting generic lines from a spreadsheet is that it should be context aware. We can start the conversation with the assistant by laying down some info for it to bare in mind

function getMatchContextMessagePrompt(match: Match) {
  return `Here's some context that you may optionally use in your live commentary:
- ${match.meta.teams.home.meta.nickName} are playing ${match.meta.teams.away.meta.nickName}
- The weather is rain
- This is clash between the two best teams in the league
- The manager of the ${match.meta.teams.home.meta.nickName} has been talking to the media about how they want more from their players`
}

Creating the commentary

You can see here I’m trying to anchor the commentary on the facts of the match, (e.g. the score) and provide up-to-the-minute context of what’s been going on:

function createEventCommentaryPrompt(
  {
    eventMatchContext: { matchStateAfter, matchStateBefore },
    chatContext: { matchSummary, previousEventContent },
  }: Context,
  event: MatchEvent
): string {
  return `Can you now describe this move as it unfolds in the present tense?

Here is a summary of the match so far:
${matchSummary}

Here is the previous move (which occurred about 3 minutes ago):
${previousEventContent}

For this upcoming move:
- The score before the move was ${getScore(matchStateBefore)}
- The score after the move will be ${getScore(matchStateAfter)}
- The move is a ${event.type}
- We are approximately ${matchStateAfter.eventCount * 3} minutes into the match

The steps that make up the move are summarised in this JSON array:
${JSON.stringify(
  event.steps.map(step => ({ "step type": step.type, ...getStepPlayers(step) }))
)}

Could you make sure that:
- you try to use less than 1000 tokens in your reply. You can skip out or summarise some of the steps that build up to the climax of the move
- you highlight the players' positive sporting attributes and their personalities
- you show excitement!
- keep it snappy and don't repeat yourself, each move should not take long to read
- start speaking in the present tense
- do not give away the type of move until the climax of the move
- you can finish with a comment on the move in the context of the match. This is the only time you can speak in the past tense`
}

👇 Sound on

☝️ 🧙‍♂️

It’s not perfect. I’ve seen it get confused about the score or get confused about the intent of an event, (e.g. the seekers aren’t supposed to catch the snitch on a SNITCH_MISS)

But it’s certainly capable!

Adding Audio

It wouldn’t be a real side project without a healthy dose of scope creep. While I’ve resisted the temptation to feed the commentary into a 3D graphics generator, I could not sully the CM3 experience by omitting crowd noise.

It probably would have been passable to check for events of type GOAL and trigger some cheering sounds at roughly the right time, but obviously, it’s a lot more fun asking GPT to figure out what sound effect to play and when.

function getEventSoundsPrompt(commentary) {
  return `Using the following typescript type:

type AudioEvents = {
    type: 'GOAL' | 'MISS' | 'SAVE' | 'INTERCEPTION' | 'TACKLE' | 'BLUDGER HIT' | 'FOUL' | 'CATCH' | 'WIN' | 'EXCITEMENT BUILD' | 'ROAR'; 
    quote: string;
    team: 'home' | 'away';
    /** How exciting was this move? (integer between 0-10) */ 
    excitement: number; 
}[]

Please return a JSON array of events from the following commentary.
Make sure there are no trailing commas in the JSON array!

${commentary}`
}

Does GPT listen when I say stick to the AudioEvents['type'] listed? Nope! Could it benefit from a few on-the-fly training examples? Maybe!

It’s still pretty impressive, though.

Team home | away is used to pan the sound effect to the left or right of the stadium and the excitement level to set the gain.

There are a grand total of four audio samples used. Thank you to everyone who uploads to the fantastic Freesound project!

The samples are sampled from samples of football games between Atletico Minero vs Juventude, and Feyenoord vs an unknown team. I don’t speak Portuguese or Dutch, so here’s hoping there isn’t any rude language in there!

• https://freesound.org/people/felix.blume/sounds/611744/
• https://freesound.org/people/huubjeroen/sounds/39733/

Wrap-up

It’s been super fun dusting off of an old project and sprinkling in some modern AI magic.

• Check out the latest match here QuidditchPL.com!
• The crowd sounds are quite relaxing, so you could always just leave it in the background.
• Enjoy your day!