Object Detection in Flutter Using TensorFlow Lite and YOLOv8: A Comprehensive Guide

Object detection is a vital technique in computer vision that enables the identification and localization of objects within images or video frames. Unlike image classification or image recognition, which assigns a category label to an entire image, object detection not only identifies the object but also draws bounding boxes around them to determine their exact location in the image. This article provides a detailed guide on implementing object detection in a Flutter application using TensorFlow Lite (TFLite) and the YOLO (You Only Look Once) v8 model, emphasizing practical examples to simplify the process.

Why YOLO for Object Detection?

YOLO stands out as a popular algorithm for object detection due to several key advantages:

• Speed: YOLO processes images in real-time using a single neural network, making it significantly faster than other object detection algorithms.
• Accuracy: Despite its speed, YOLO achieves high detection accuracy, effectively identifying objects in various scenarios.
• Generalization: YOLO generalizes well to new domains, offering robust performance in diverse real-world applications.
• Open-Source: The YOLO algorithm is available as open-source, allowing developers to modify and integrate it into their projects.

YOLO frames the object detection problem as a regression problem rather than a classification task, breaking down images into smaller units and associating probabilities with the detected objects using a single Convolutional Neural Network (CNN).

Setting Up the Flutter Environment for TensorFlow Lite Object Detection

Before integrating the YOLO model into your Flutter application, ensure that your development environment is correctly set up to handle TFLite.

1. Setting Up Flutter Environment and Project

Create a new Flutter project using the command:

flutter create flutter_yolo_object_detection

Update your android/app/build.gradle to include the following configurations:

android {
    aaptOptions {
        noCompress 'tflite'
        noCompress 'lite'
    }
    compileSdkVersion 34
    minSdkVersion 21
    targetSdkVersion 34
}

buildscript {
    ext.kotlin_version = '1.7.10' // Use Latest Version
    repositories {
        google()
        mavenCentral()
    }
}

These settings ensure that the .tflite model files are not compressed and adjust the Kotlin version and SDK versions compatible with TensorFlow Lite.

2. Adding Your YOLOv8 Model to the Project

The YOLO model must be converted to the TFLite format (.tflite) for deployment on mobile devices. If you wish to train a custom YOLO model, use Python libraries like PyTorch or TensorFlow to export the trained model in the TFLite format. For this article, we'll use a pre-trained YOLOv8 model.

• Create an assets directory in your Flutter project and place the .tflite model and labels file in it:
• assets/labels.txt - Contains the class names (labels) for the model.
• assets/yolov8n.tflite - The YOLOv8 TFLite model file.
• Add the assets to your pubspec.yaml:

flutter:
  assets:
    - assets/labels.txt
    - assets/yolov8n.tflite

3. Adding Required Packages

To integrate TFLite object detection into your Flutter project, include the necessary packages in your pubspec.yaml:

dependencies:
  flutter:
    sdk: flutter
  flutter_vision: ^latest_version
  camera: ^latest_version

• flutter_vision allows us to use TFLite models for computer vision tasks.
• camera provides access to the device’s camera for capturing images and video streams.

4. Creating the Object Detection Class

Create a new Flutter widget, YoloVideo, to handle video stream capture and object detection. Here’s how to get started:

1. Initialize the Camera and Vision: Set up variables to handle camera input and TensorFlow Lite model processing.

late CameraController controller;
late FlutterVision vision;
late List<Map<String, dynamic>> yoloResults;
CameraImage? cameraImage;
bool isLoaded = false;
bool isDetecting = false;
double confidenceThreshold = 0.5;

2. Initialize the Camera and Load the YOLO Model: In the init function, set up the camera and load the TFLite model.

@override
void initState() {
  super.initState();
  init();
}

init() async {
  camerass = await availableCameras();
  vision = FlutterVision();
  controller = CameraController(camerass[0], ResolutionPreset.high);
  await controller.initialize();
  await loadYoloModel();
  setState(() {
    isLoaded = true;
    isDetecting = false;
    yoloResults = [];
  });
}

Future<void> loadYoloModel() async {
  await vision.loadYoloModel(
    labels: 'assets/labels.txt',
    modelPath: 'assets/yolov8n.tflite',
    modelVersion: "yolov8",
    numThreads: 1,
    useGpu: true,
  );
  setState(() {
    isLoaded = true;
  });
}

3. Create the Real-time Object Detection Logic: Implement a method to process video frames and detect objects.

Future<void> yoloOnFrame(CameraImage cameraImage) async {
  final result = await vision.yoloOnFrame(
    bytesList: cameraImage.planes.map((plane) => plane.bytes).toList(),
    imageHeight: cameraImage.height,
    imageWidth: cameraImage.width,
    iouThreshold: 0.4,
    confThreshold: confidenceThreshold,
    classThreshold: 0.5,
  );
  if (result.isNotEmpty) {
    setState(() {
      yoloResults = result;
    });
  }
}

4. Create UI for Real-time Object Detection: Construct the widget tree, displaying the camera preview and bounding boxes around detected objects.

@override
Widget build(BuildContext context) {
  final Size size = MediaQuery.of(context).size;

  if (!isLoaded) {
    return const Scaffold(
      body: Center(
        child: Text("Model not loaded, waiting for it"),
      ),
    );
  }
  
  return Scaffold(
    body: Stack(
      fit: StackFit.expand,
      children: [
        AspectRatio(
          aspectRatio: controller.value.aspectRatio,
          child: CameraPreview(controller),
        ),
        ...displayBoxesAroundRecognizedObjects(size),
        Positioned(
          bottom: 75,
          width: MediaQuery.of(context).size.width,
          child: Container(
            height: 80,
            width: 80,
            decoration: BoxDecoration(
              shape: BoxShape.circle,
              border: Border.all(width: 5, color: Colors.white, style: BorderStyle.solid),
            ),
            child: isDetecting
                ? IconButton(
                    onPressed: stopDetection,
                    icon: const Icon(Icons.stop, color: Colors.red),
                    iconSize: 50,
                  )
                : IconButton(
                    onPressed: startDetection,
                    icon: const Icon(Icons.play_arrow, color: Colors.white),
                    iconSize: 50,
                  ),
          ),
        ),
      ],
    ),
  );
}

5. Display Bounding Boxes: Add a method to display bounding boxes around detected objects.

List<Widget> displayBoxesAroundRecognizedObjects(Size screen) {
  if (yoloResults.isEmpty) return [];
  
  double factorX = screen.width / (cameraImage?.height ?? 1);
  double factorY = screen.height / (cameraImage?.width ?? 1);
  Color colorPick = const Color.fromARGB(255, 50, 233, 30);

  return yoloResults.map((result) {
    double objectX = result["box"][0] * factorX;
    double objectY = result["box"][1] * factorY;
    double objectWidth = (result["box"][2] - result["box"][0]) * factorX;
    double objectHeight = (result["box"][3] - result["box"][1]) * factorY;

    return Positioned(
      left: objectX,
      top: objectY,
      width: objectWidth,
      height: objectHeight,
      child: Container(
        decoration: BoxDecoration(
          borderRadius: BorderRadius.all(Radius.circular(10.0)),
          border: Border.all(color: Colors.pink, width: 2.0),
        ),
        child: Text(
          "${result['tag']} ${(result['box'][4] * 100).toStringAsFixed(1)}%",
          style: TextStyle(
            background: Paint()..color = colorPick,
            color: const Color.fromARGB(255, 115, 0, 255),
            fontSize: 18.0,
          ),
        ),
      ),
    );
  }).toList();
}

6. Start and Stop Detection: Implement functions to start and stop the camera’s video stream and object detection.

Future<void> startDetection() async {
  setState(() {
    isDetecting = true;
  });
  if (controller.value.isStreamingImages) return;

  await controller.startImageStream((image) async {
    if (isDetecting) {
      cameraImage = image;
      yoloOnFrame(image);
    }
  });
}

Future<void> stopDetection() async {
  setState(() {
    isDetecting = false;
    yoloResults.clear();
  });
}

Running the Application

Finally, invoke the YoloVideo class in main.dart to start the application:

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  runApp(const MaterialApp(
    home: YoloVideo(),
  ));
}

Advanced Usage and Optimization Tips

• Custom Model Training: If you have specific objects you need to detect, train a custom YOLO model using the latest YOLOv8 framework in Python. Export the trained model in .tflite format for use in your Flutter app.
• Adjust Thresholds: Tweak confidenceThreshold and iouThreshold values to control detection sensitivity. This can improve detection accuracy based on the objects and the environment.
• Performance Optimization: Leverage GPU acceleration (useGpu: true) if supported by the device for faster inference times.
• Sound Notifications: You can enhance the user experience by integrating sound notifications (as shown in the speak() method) for detected objects using packages like flutter_tts.

Conclusion

Integrating YOLOv8 with TensorFlow Lite into a Flutter app enables real-time, high-accuracy object detection. This guide provides a step-by-step approach to set up, implement, and fine-tune the detection process, making it easier for developers to harness the power of deep learning on mobile devices. With further customization and optimization, you can tailor this solution to fit various use cases, such as security surveillance, inventory management, and augmented reality applications.