How to Implement Pose Estimation with ml5.js?

Last Updated : 08 Aug, 2024

Pose estimation is a technology that can detect human poses in images or videos by identifying key body points like the head, shoulders, elbows, and knees. It has many applications, such as in fitness apps, gaming, and human-computer interaction. In this article, we'll learn how to implement pose estimation using ml5.js, a user-friendly JavaScript library built on top of TensorFlow.js.

Pose estimation involves detecting and tracking the positions of key points on the human body. By recognizing these points, we can understand the pose and movements of a person. This technology is used in various fields, from healthcare to entertainment.

Approach

Create the HTML File.
Sets up the basic structure of the webpage.
Includes the ml5.js library and p5.js library for handling graphics.
Creates a <video> element to capture webcam video and a <canvas> element to draw the pose estimation results.
Apply these methods:
- setup(): Initializes the canvas and video capture. Sets up PoseNet and starts detecting poses.
- modelReady(): Logs a message when the PoseNet model is loaded.
- draw(): Continuously draws the video and detected poses on the canvas.
- drawKeypoints(): Drawkey points red circles on the key points of detected poses.
- drawSkeleton(): Draws red lines between the key pointskey points to form the skeleton of detected poses.

Example: This example shows the implementation of the above-explained approach.

HTML

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" 
          content="width=device-width, 
                   initial-scale=1.0">
    <title>Pose Estimation with ml5.js</title>
    <style>
        body {
            display: flex;
            flex-direction: column;
            align-items: center;
            justify-content: flex-start;
            height: 100vh;
            margin: 0;
            background-color: #f0f0f0;
        }

        h1 {
            margin: 20px 0;
            z-index: 1;
        }

        #video,
        #canvas {
            margin-top: 20px;
            z-index: 0;
        }
    </style>
</head>

<body>
    <h1>Pose Estimation with ml5.js</h1>
    <video id="video" width="640" 
           height="480"
           autoplay></video>
    <canvas id="canvas"
            width="640" 
            height="480"></canvas>
    <script src=
"https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.4.0/p5.js"></script>
    <script src=
"https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.4.0/addons/p5.dom.js"></script>
    <script src=
"https://unpkg.com/[email protected]/dist/ml5.min.js" 
            type="text/javascript"></script>
    <script src="script.js"></script>
</body>

</html>

JavaScript

let video;
let poseNet;
let poses = [];

function setup() {
    const canvas = createCanvas(640, 480);
    canvas.position((windowWidth - width) / 2,
    (windowHeight - height) / 2);
    video = createCapture(VIDEO);
    video.size(width, height);

    poseNet = ml5.poseNet(video, modelReady);
    poseNet.on('pose', function (results) {
        poses = results;
    });

    video.hide();
}

function modelReady() {
    console.log('PoseNet Model Loaded!');
}

function draw() {
    image(video, 0, 0, width, height);

    drawKeypoints();
    drawSkeleton();
}

function drawKeypoints() {
    for (let i = 0; i < poses.length; i++) {
        let pose = poses[i].pose;
        for (let j = 0; j < pose.keypoints.length; j++) {
            let keypoint = pose.keypoints[j];
            if (keypoint.score > 0.2) {
                fill(255, 0, 0);
                noStroke();
                ellipse(keypoint.position.x,
                keypoint.position.y, 10, 10);
            }
        }
    }
}

function drawSkeleton() {
    for (let i = 0; i < poses.length; i++) {
        let skeleton = poses[i].skeleton;
        for (let j = 0; j < skeleton.length; j++) {
            let partA = skeleton[j][0];
            let partB = skeleton[j][1];
            stroke(255, 0, 0);
            strokeWeight(2);
            line(partA.position.x, partA.position.y, 
            partB.position.x, partB.position.y);
        }
    }
}

Note: When prompted, allow access to your webcam.

Output:

Conclusion

By following this guide, you should be able to create a simple pose estimation application using ml5.js and p5.js. This application captures video from your webcam, detects human poses, and draws keypoints and skeletons on the video in real-time. Pose estimation can be used in various projects, from interactive art installations to fitness apps and beyond.