Turning cricket video into data: how a computer vision pipeline works
Short answer: a computer vision pipeline turns cricket video into data in stages. It detects players and the ball in every frame, tracks them across frames, reconstructs the ball's flight, reads the batter's pose, classifies the shot, and marks events like bounce and bat contact. The result is one structured row per delivery, which every analytic such as pitch maps, heat maps and shot distribution is computed from.
I built this end to end as CricketVision AI. Here is how each stage works and the problems that made it hard.
Why turn video into a table?
Cricket analysis is still largely manual. Analysts watch footage frame by frame to note deliveries, shots and field placements. The useful signal, meaning where the ball went, how the batter moved and what shot was played, is locked inside pixels, so nothing downstream can query it.
The goal is not a clever video overlay. It is a delivery table: one row per ball, with everything an analyst would otherwise write down by hand.
What are the stages?
| Stage | Technique | Output |
|---|---|---|
| Detection | Trained YOLO detector | Ball, batter, bowler, fielders, wicket-keeper, umpire and stumps in each frame |
| Tracking | ByteTrack / BoT-SORT | Persistent identities across frames, with cricket roles assigned by geometric rules |
| Ball trajectory | Kalman filter, gap interpolation | A continuous flight path, bounce point and speed estimates |
| Pose | COCO-17 keypoints | Joint angles and batting technique features |
| Shot classification | R(2+1)D-18 video action model | One of ten cricket shots |
| Events | Rules over the above | Delivery start, bounce, bat contact, shot, fielding, boundary, each with a confidence score |
Each stage can be tested and swapped on its own, and the whole pipeline runs behind a FastAPI service with a web dashboard, packaged with Docker.
Why is the ball so hard to track?
At broadcast resolution the ball is a handful of pixels, moving fast, and it disappears for whole stretches behind players, in motion blur or against the crowd. A general-purpose detector loses it constantly.
Two things fixed this:
- A dedicated ball detection pass, separate from the player detector.
- Kalman filtering plus gap interpolation, which estimates where the ball must be between sightings. That rebuilds a continuous path from an intermittent signal.
Why does classifying clips not mean classifying video?
A shot classifier can score well on neat, pre-cut clips and still struggle on continuous footage, where nobody has marked where a delivery starts and ends. I measured accuracy end to end on unsegmented video, not only clip by clip, because that is what the system faces in real use.
This is the most common trap in applied computer vision: the number from the benchmark is not the number you get in production. Evaluate on the input the system will actually receive.
How do you measure ball speed from video?
Carefully. Speed in pixels per second means nothing across different camera angles. Where the camera can be calibrated, a pitch homography maps pixels to real distances and gives km/h. Where it cannot, the pipeline reports uncalibrated units instead of inventing a number.
What can you do with the delivery table?
Once every delivery is a structured row, analysis becomes queries:
- Shot distribution for a batter against a type of delivery
- Pitch maps of where a bowler lands the ball
- Heat maps of where shots go
- Statistical tests on what works against whom
The same pattern works beyond cricket. Any sport, or any process captured on video, becomes analysable once each event is a row of data. The code is open source on GitHub.
Frequently asked questions
In stages: a detector such as YOLO finds players and the ball in each frame, a tracker keeps their identities across frames, the ball's flight is reconstructed with filtering, pose estimation reads the batter's body, a video model classifies the shot, and events such as bounce and bat contact are marked. The output is one structured row per delivery.
