Dobble of nothing: our month of MLOps with ZenML

Recently, as part of ZenML’s Month of MLOps competition, we built an AI to beat the card game Dobble; and not only did we beat the game, we won the competition! 🥳​.

In this blog, I take a look back on how our team did it. It’s been a significant year at Fuzzy Labs for many reasons, the biggest of which has been growing the team. After a summer hiring spree, we needed a project that gave our fresh team of talented MLOPs engineers the opportunity to really get to know one another, and of course, to take on a hard technical challenge.

In Dobble, the game we picked for the competition, players are presented with two cards. Each card has eight symbols, and one of those symbols will appear on both cards; the first player to call out the match wins. Labelling data and training a model for this task is already a challenge, but to make it even harder, this model is deployed to an edge device, specifically an NVIDIA Jetson Nano.

Team lead Jon Carlton explains the whole thing succinctly in the below video:

Q&A with the team

Our team — Chris, Jon, Misha, Oscar, and Shubham — worked on this over four weeks (in-between client work), first understanding the data, then building pipelines, and deploying the model to the Nvidia hardware.

Reflecting on the project, I asked the team five questions.

What was the most challenging part of this project?

Shubham: Deployment: the initial approach was to pull models from ZenML’s server. But we were unable to install ZenML on Jetson, so we had to abandon that approach. In the end we pulled the model from MLFlow.

‍Oscar: Definitely inference on the Jetson Nano.

‍Misha: Definitely the deployment. There's a lack of off-the-shelf tooling to pull models easily from ZenML. Nvidia's tooling is a bit confusing and not perfectly documented.

‍Jon: Getting inference working on the edge device - there isn’t much support for doing this kind of thing on the edge!

‍My take: there are two closely related challenges. First, getting the model on to the hardware (deployment), and second getting inference to work reliably (serving). The MLOPs tooling for these tasks is mature if you’re running on the cloud, but there’s lots of room for new tools to enter this space and improve the experience on the edge.

How did pipelines help you iterate on your approach?

Jon: We knew that once a pipeline was in place, e.g. the data pipeline, it could be relied on for producing consistent results. This enabled us to separate work out more easily, e.g., teams could pair on different steps of the same pipeline.

‍Shubham: Pipelines helped break down this complex project into small testable, isolated  steps. On top of this, all the configuration for all the pipelines was handled by 1 YAML configuration file, so we could quickly experiment with different parameters.

‍My take: Composability is a pillar of good engineering. By breaking things down into small, composable units (in this case the steps and pipelines), you can work on those things in parallel, maintain them easily, and test them with confidence.

What were the biggest lessons learned?

Jon: That the edge-based MLOps tooling needs expanding.

‍Oscar: I really liked the use of different pre-commit hooks suggested by Shubham. It definitely helped us to tidy up our code, and made it easier to understand.

‍Shubham: Nvidia documentation for Jetson Nano and TensorRT seemed lacking. We faced some incompatibilities with ONNX and TensorRT versions.

‍My take: a wide spread of answers for this one. Difficulties with tooling and documentation for the Jetson Nano was a common theme on the project, but the time constraints also prevented the team from deeply exploring this ecosystem, something we’ll be doing more of in the future. We’d love to hear about other people’s experiences on this platform.

A great deal of things were learnt by the team, and one stand out for me is in tooling to help keep the code clean. The team used a set of Git pre-commit hooks to run code quality checks, and this is something we’ll do on all our projects going forward.

Anything you would do differently next time?

Chris: More focus on inference early on, so we can better control where we spend our time later.

Jon: Spend more engineering time earlier on the more challenging aspects of the project: deployment and inference on the edge.

Misha: Tackling the more difficult bits earlier in "a thin slice" would have helped.

My take: towards the end of the project, the team encountered some challenges with running the model on the Jetson Nano, and a lot of that can be attributed to a lack of familiarity. Nobody can know everything; on any MLOps project, there will be things you already know how to do, and things you’re not really familiar with, which in this case was the Nvidia platform. Tackling the biggest unknowns early on is a good thing for any MLOps practitioner to take on-board.

What games should a Jetson Nano AI tackle next?

An exciting question for the team, but also for you, the reader! Let us know what you’d like to see us take on next.

Here’s what we came up with: Top Trumps, Jungle Speed, Duckie Town, Monopoly.

Conclusions and more Dobble on the edge

I’d like to say thanks once again to ZenML for running a fantastic competition. There were some great competitors, and we were both thrilled and humbled to be picked for first place.

Fuzzy Labs will be matching the team’s winnings with a donation of £750 to the Trussell Trust, as selected by the team. The Trussell Trust provides emergency food and support to people locked in poverty in the UK.

If you’d like to check out our code and try the model out yourself, here’s the Github repo. The team also wrote some blogs that go into detail about different aspects of the project:

• The data science: how our model works and how we trained it
• The pipeline: a deep dive into the ZenML pipeline that ties everything together, going from data input to model deployment.
• The edge deployment: how we play Dobble on Jetson Nano.