Return Home

Real-Time Traffic Object Detection with TensorFlow.js

Project JavaScript TensorFlow.js Library

Overview

I recently decided I was going to give myself 10 days to study for and pass AI-900 before it gets retired June 30, 2026. I reached the computer vision section of study and thought it would be fun to try and build something lightweight in the browser. There are hundreds of different methods for models and object detection, but I wanted to make something that I could display here.

Going into this I had absolutely zero experience with object detection but I had 2 main goals:

  1. Use a live traffic feed as my data source (don't care where from)
  2. Run entirely client-side

Live Feed

Below is a TensorFlow.js object detection demonstration that runs fully in the browser. The HLS.js library enables a live HTTPS stream sourced from the Iowa Department of Transportation. That stream is then fed into the TensorFlow.js COCO-SSD (Common Objects in Context Single Shot MultiBox Detection) model, which runs lightweight object detection directly in the browser.

Click the "Load Live Feed" button to load required libraries and start viewing the live feed with object detection.

Status

Model Not loaded
Backend Pending
Stream Not connected
Inference FPS 0

Challenges

Unlike past projects I think I did a better job of staying within the original scope! A lot of research went into actually finding an open feed that didn't require an API or some kind of proxy service to get around CORS. On top of that, these lightweight object detection models are very input sensitive (they require a high quality video, at a similar angle to what they were trained on).

This meant even though I had multiple feeds that fit my criteria, I also need a camera angle that enabled the model to work properly. I ended up finding a feed from the Iowa DOT that sits directly over an intersection with a red light, pointed straight down onto the oncoming traffic below. This ended up being the best camera I could find that met the criteria.

I found myself compromising on the model choice as well. I had 3 options available:

  • lite_mobilenet_v2 - fastest but least accurate model
  • mobilenet_v1 - middle ground model where we sacrifice some accuracy for a noticeable boost in speed
  • mobilenet_v2 - the most accurate but also the slowest model

Even though mobilenet_v1 is the oldest of the models, it ended up being the best model for our use case. I saw a noticeable boost in accuracy over the lite version, and the mobilenet_v2 model was just too slow to be usable in a live feed scenario without a dedicated processor.

I also had to include an inference fallback based on device hardware; WebGL -> WASM -> CPU. WebGL is picked with the highest priority when a device has a GPU/hardware acceleration enabled, WASM is the fallback for browsers where WebGL is unavailable, and CPU last so the demo still runs even in constrained environments.

Finally, I had difficulty accounting for fullscreen. If a user tried to use the native fullscreen button, the object detection didn't follow. I had to make sure the fullscreen functionality applied to the video and canvas elements, and not just the video like the native button did.

Final Thoughts

I think there's something to be said about training data vs real world application, especially for computer vision tasks. This is a lightweight model limited to only 80 detectable object classes. That being said, these lightweight models tend to be very sensitive to input data, and aren't robust or powerful enough to handle all scenarios: a motorcycle might only be detected from the side view and won't be detected from the front or back, a semi might be misclassified up as a bus, a traffic cone might fit the profile of a person (true story, I had to remove the person class). To put it simply, object detection is only as robust and as accurate as the training and input data allow.

There are a number of things that could improve object detection accuracy. If the camera view was closer and street-side like the highly contentious Flock ALPR cameras, or I wired a backend to handle a heavier model and process inference, I could have gotten this working with much higher accuracy. But, constraints are constraints and I didn't want scope creep to take over the project.