Building Realtime Object Detection WebApp with Tensorflow.js and Angular

TENSORFLOW.JS

Tensorflow.js is an open-source library enabling us to define, train and run machine learning models in the browser, using Javascript. For more information about Tensorflow.js, check out its official webpage. I will use Tensorflow.js library in Angular to build a Web App which detects multiple objects on webcam video feed.

COCO-SSD MODEL

First, we have to select the pre-trained model which we are going to use for object detection. Tensorflow.js provides several pre-trained models for classification, pose estimation, speech recognition and object detection purposes. Check out all the Tensoflow.js pre-trained models for more information.

COCO-SSD model, which is a pre-trained object detection model that aims to localize and identify multiple objects in an image, is the one that we will use for object detection. Original ssd_mobilenet_v2_coco model size is 187.8 MB and can be downloaded from tensorflow model zoo. Compared to original model, Tensorflow.js version of the model is very lightweight and optimized for browser execution. The default object detection model for Tensorflow.js COCO-SSD is ‘lite_mobilenet_v2’ which is very very small in size, under 1MB, and fastest in inference speed. If you would like better classification accuracy you can use ‘mobilenet_v2’, in this case the size of the model increases to 75 MB which is not suitable for web-browser experience.

‘model.detect’ takes img or video inputs directly from HTML so you do not need to convert the inputs into tensor before using them. It returns an array of classes, probability scores as well as bounding box coordinates for the detected objects.

model.detect(
  img: tf.Tensor3D | ImageData | HTMLImageElement |
      HTMLCanvasElement | HTMLVideoElement, maxDetectionSize: number
)

[{
  bbox: [x, y, width, height],
  class: "person",
  score: 0.8380282521247864
}, {
  bbox: [x, y, width, height],
  class: "kite",
  score: 0.74644153267145157
}]

ANGULAR WEBAPP INITIALIZING

After we all clear about the model, it is time to use Angular command line interface to initialize the Angular web application.

npm install -g @angular/cli
ng new TFJS-ObjectDetection
cd TFJS-ObjectDetection

Then I will use nmp package manager to load tensorflow.js and COCO-SSD library.

TFJS-ObjectDetection npm install @tensorflow/tfjs --save
TFJS-ObjectDetection npm install @tensorflow-models/coco-ssd --save

It is all set now, so we can start coding. I will start with importing coco-ssd model in ‘app.component.ts’.

import { Component, OnInit } from '@angular/core';

//import COCO-SSD model as cocoSSD
import * as cocoSSD from '@tensorflow-models/coco-ssd';

STARTING WEBCAM FEED

Then I will start webcam feed with the following code.

webcam_init()
{
  this.video = <HTMLVideoElement> document.getElementById("vid");
  navigator.mediaDevices
  .getUserMedia({
  audio: false,
  video: {facingMode: "user",}
  })
  .then(stream => {
  this.video.srcObject = stream;
  this.video.onloadedmetadata = () => {
  this.video.play();};
  });
}

OBJECT DETECTION FUNCTIONS

We need another function to load the coco-ssd model which also call ‘detectFrame’ function to make the prediction using images from the webcam feed.

public async predictWithCocoModel()
{
  const model = await cocoSSD.load('lite_mobilenet_v2');
  this.detectFrame(this.video,model);
  
}

‘detectFrame’ function uses requestAnimationFrame to loop the prediction over and over by making sure that the video feed is as much smooth as possible.

detectFrame = (video, model) => {
  model.detect(video).then(predictions => {
  this.renderPredictions(predictions);
  requestAnimationFrame(() => {
  this.detectFrame(video, model);});
  });
}

RENDERING PREDICTIONS

Meantime, ‘renderPredictions’ function draw the bounding boxes and class names into the screen for detected objects.

renderPredictions = predictions => {

  const canvas = <HTMLCanvasElement> document.getElementById ("canvas");
  
  const ctx = canvas.getContext("2d");

  canvas.width  = 300;
  canvas.height = 300;
  ctx.clearRect(0, 0, ctx.canvas.width, ctx.canvas.height);

  // Fonts
  const font = "16px sans-serif";
  ctx.font = font;
  ctx.textBaseline = "top";
  ctx.drawImage(this.video,0, 0,300,300);

predictions.forEach(prediction => {

  const x = prediction.bbox[0];
  const y = prediction.bbox[1];
  const width = prediction.bbox[2];
  const height = prediction.bbox[3];

  // Bounding box
  ctx.strokeStyle = "#00FFFF";
  ctx.lineWidth = 2;
  ctx.strokeRect(x, y, width, height);

  // Label background
  ctx.fillStyle = "#00FFFF";
  const textWidth = ctx.measureText(prediction.class).width;
  const textHeight = parseInt(font, 10); // base 10
  ctx.fillRect(x, y, textWidth + 4, textHeight + 4);

});

predictions.forEach(prediction => {
  
  const x = prediction.bbox[0];
  const y = prediction.bbox[1];

  ctx.fillStyle = "#000000";
  ctx.fillText(prediction.class, x, y);});

};

All required funtions are ready now to perform the object detection on the browser. We just need to call ‘webcam_init’ and ‘predictWithCocoModel’ on ‘ngOnInit’ to initialize the app on start.

ngOnInit()
{
  this.webcam_init();
  this.predictWithCocoModel();
}

HTML ELEMENTS FOR OBJECT DETECTION

One last step remaining is to modify ‘app.component.html’ to include

<div style="text-align:center">
  <h1>Tensorflow.js Real Time Object Detection</h1>
  <video hidden id="vid" width="300" height="300"></video>
  <canvas id="canvas"></canvas>
</div>

FULL CODE

Visit my github repository for full code of this project.

DEMO WEBAPP

Visit the live demo application to see the codes in action. Application runs in Google Crome browser without any issue. If you use any other browser, make sure that the browser you use supports ‘requestAnimationFrame’ .

FINAL NOTES

Thank you for reading.

Reach out at [email protected], if you have any question.