Mobile Face Detection with Google ML Kit

Face detector

My goal in this article is to show you how to build a face detector mobile app with Expo and Google ML Kit in less than 5 minutes.

Expo is a powerful framework equips with tons of ready-to-use components that allows you to build a cross-platform mobile application with ease. In this article, we are going to use components such as expo-camera to access mobile phone camera and a built-in face detector module expo-face-detector to detect faces from images. Expo-face-detector is based on Google ML Kit Vision API.

Before we can use Camera object, we need to import expo-permissions package and access camera permission with this line of code.

await Permissions.askAsync(Permissions.CAMERA);

Expo camera has built-in face detectors properties in order for us to configure its properties and callback methods.

<Camera
  style={{ flex: 1 }}
  type={this.props.cameraType}
  onFacesDetected={this.handleFacesDetected}
  onFaceDetectionError={this.handleFaceDetectionError}
  faceDetectorSettings={{
    mode: FaceDetector.Constants.Mode.fast,
    detectLandmarks: FaceDetector.Constants.Landmarks.all,
    runClassifications: FaceDetector.Constants.Classifications.all,
  }}
  ref={ref => {
    this.camera = ref;
  }}
>

onFacesDetected: set to the method to handle faces when detected. While detecting faces, face detector will call this method with an array of faces objects as parameter. My implementation as below.

handleFacesDetected = ({ faces }) => {
  if (faces.length === 1){ // detect a face
    this.setState({
      faceDetected: true, // variable state to hold face is detected
      faceBox: faces[0].bounds // variable to hold face location
    });
  } else { // no faces detected
    this.setState({faceDetected: false });
  }
}

In order to show face bounding box and highlight detected face, we are using React View with corresponding styles to draw a dashed rectangle in the location returned by Face Detector model.

<View
  style={{
    position: 'absolute',
    backgroundColor: 'transparent',
    flexDirection: 'row',
    width: this.state.faceBox ? this.state.faceBox.size.width : '100%',
    height: this.state.faceBox ? this.state.faceBox.size.height : '100%',
    top: this.state.faceBox ? this.state.faceBox.origin.y : '0',
    left: this.state.faceBox ? this.state.faceBox.origin.x : '0',
    borderColor: '#33FF33',
    borderWidth: 5,
    borderStyle: 'dashed',
    display: this.state.faceDetected && !this.state.pictureTaken ? 'flex' : 'none',
  }}>
</View>

To draw the rectangle, notice that we are setting top left width height from our faceBox state variable which has origin.x, origin.y and size.width, size.height.

Here is the interesting part of faceDetectorSettings property, which we can detect not just faces but smiling and open eye probability by setting runClassifications: FaceDetector.Constants.Mode.all. Note if you only need faces detection set it to none. Refer to the API documentation here. There is also detectLandmarks property which we can detect and return face landmarks such as eyes, nose, mouth. With a bit more code, we can build an app that can alert exhausted drivers when they are unintentionally closing their eyes for too long

Expo camera and built-in face detector function allow us to build a mobile face detection app with just a few lines of code. It is simple yet powerful, but you might ask how can I use my own vision model with the camera? In my next article, I will show you how to use other machine learning model like Google MobileNet and TensorFlow.js platform adapter for react native framework in which we will see how to use TensorCamera object. This object augments the Expo.Camera component with the ability to yield tensors representing the camera stream. Stay tuned.

Full source code for this article can be found here:

Please follow me if you find this article useful, it would motivate me to write more useful articles and helping others to learn. Thank you.