Gesture Controlled Virtual Mouse with ESP32-CAM & OpenCV and Python

 

 Klik Link: https://ayojurnalku.blogspot.com/2022/03/gesture-controlled-virtual-mouse-with.html

Overview

Ringkasan

Pada artikel ini, kami akan mengembangkan Gesture Controlled Virtual Mouse dengan ESP32-CAM & OpenCV. Modul Kamera ESP32 bersama dengan program Python dapat digunakan untuk mengontrol pelacakan mouse dan operasi klik secara wireless/nirkabel.

Untuk memulai, kita harus memiliki pengetahuan yang baik tentang Python, Pemrosesan Gambar, Sistem Embeding/Tertanam, serta Internet of Things. Pertama, kita akan memahami cara mengontrol pelacakan dan klik mouse, dan semua persyaratan yang diperlukan untuk menjalankan program python. Pertama-tama kami akan menguji seluruh skrip python dengan webcam atau kamera internal laptop.

Pada bagian kedua, kita akan menggunakan Modul ESP32-CAM dan menjalankan kode Python. Jadi, ESP32-CAM akan digunakan sebagai perangkat input sebagai pengganti kamera PC atau kamera eksternal lainnya.

In this article, we will develop Gesture Controlled Virtual Mouse with ESP32-CAM & OpenCV. The ESP32 Camera Module along with a Python program can be used to control the mouse tracking and clicking operations wirelessly.

To get started one must have sound knowledge of Python, Image processing, Embedded Systems as well as the Internet of Things. First, we will understand how to control the mouse tracking and clicking, and what all requirements are needed to run the python program. We will first test the whole python script with a webcam or internal camera of a laptop.

In the second part, we will use an ESP32-CAM Module and run the Python code. So, the ESP32-CAM will be used as an input device instead of a PC camera or any other external camera.

---

Materials 

Bahan-Bahan

Berikut ini adalah daftar Material untuk membuat Mouse Virtual yang Dikendalikan Gesture. Anda dapat membeli semua komponen ini dari toko online

The following is the list of Materials for making a Gesture Controlled Virtual Mouse. You can purchase all these components from onlineShop.

S.N.	Components	Description	Quantity
1	ESP32-CAM Board	AI-Thinker ESP32 Camera Module	1
2	USB Cable	5V Mini-USB Data Cable	1

---

Controlling Mouse Tracking & Clicks with PC Camera

Mengontrol Pelacakan & Klik Mouse dengan Kamera PC

Sebelum melompat ke bagian proyek, mari buat Mouse Virtual Terkendali Gerakan menggunakan teknologi pengenalan gambar PC.

Before jumping to the project part, lets build a Gesture Controlled Virtual Mouse using PC image recognition technology.

---

Installing Python & Required Libraries

Pemasangan Python dan Librari yang diperlukan

Agar streaming langsung video dapat dilihat di komputer kita, kita perlu menulis skrip Python yang memungkinkan kita untuk mengambil frame video. 

Langkah pertama adalah menginstal Python. Buka python.org dan unduh Python versi 3.7.8. Penting untuk mengunduh versi ini atau menurunkan versi ke versi ini atau beberapa perpustakaan tidak akan berfungsi.

For the live stream of video to be visible on our computer we need to write a Python script that will enable us to retrieve the frames of the video. The first step is to install Python. Go to python.org and download Python version 3.7.8. It is important to download this version or downgrade to this version or else a few libraries won’t work.

Setelah selesai download dan install Python, kemudian buka Command Prompt lalu ketik perintah berikut:

1	python --version

Maka outputnya harus menampilkan berikut:

version 3.7.8.

Once downloaded and installed go to command prompt and run the commands below:

1	python --version

The output should be as below, that is version 3.7.8.

Sekarang kita harus menginstal beberapa perpustakaan. Untuk itu jalankan perintah berikut di bawah ini satu demi satu hingga semua library terinstal.

pip install numpy

pip install opencv-python

pip install autopy

pip install mediapipe

Jika versi python yang Anda instal sudah benar, maka menginstal ini tidak akan menjadi masalah.

Now we have to install a few libraries. For that run the following commands below one after another until all the libraries are installed.

pip install numpy

pip install opencv-python

pip install autopy

pip install mediapipe

If the python version you installed is correct then installing these won’t be an issue.

---

Source Code/Program

Kode Sumber/Program

Sekarang buat folder dan di dalam folder itu buat file python baru, beri nama file tersebut sebagai track_hand.py .

Sekarang salin dan tempel kode di bawah ini, dan simpan.

Now create a folder and inside that folder create a new python file, name the file as track_hand.py .

Now copy and paste the code below, and save it.

import cv2

import mediapipe as mp

import time

import math

import numpy as np

class handDetector():

    def __init__(self, mode=False, maxHands=1, modelComplexity=1, detectionCon=0.5, trackCon=0.5):

        self.mode = mode

        self.maxHands = maxHands

        self.modelComplex = modelComplexity

        self.detectionCon = detectionCon

        self.trackCon = trackCon

        self.mpHands = mp.solutions.hands

        self.hands = self.mpHands.Hands(self.mode, self.maxHands, self.modelComplex,

                                        self.detectionCon, self.trackCon)

        self.mpDraw = mp.solutions.drawing_utils

        self.tipIds = [4, 8, 12, 16, 20]

    def findHands(self, img, draw=True):

        imgRGB = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

        self.results = self.hands.process(imgRGB)

        # print(results.multi_hand_landmarks)

        if self.results.multi_hand_landmarks:

            for handLms in self.results.multi_hand_landmarks:

                if draw:

                    self.mpDraw.draw_landmarks(img, handLms,

                                               self.mpHands.HAND_CONNECTIONS)

        return img

    def findPosition(self, img, handNo=0, draw=True):

        xList = []

        yList = []

        bbox = []

        self.lmList = []

        if self.results.multi_hand_landmarks:

            myHand = self.results.multi_hand_landmarks[handNo]

            for id, lm in enumerate(myHand.landmark):

                # print(id, lm)

                h, w, c = img.shape

                cx, cy = int(lm.x * w), int(lm.y * h)

                xList.append(cx)

                yList.append(cy)

                # print(id, cx, cy)

                self.lmList.append([id, cx, cy])

                if draw:

                    cv2.circle(img, (cx, cy), 5, (255, 0, 255), cv2.FILLED)

            xmin, xmax = min(xList), max(xList)

            ymin, ymax = min(yList), max(yList)

            bbox = xmin, ymin, xmax, ymax

            if draw:

                cv2.rectangle(img, (xmin - 20, ymin - 20), (xmax + 20, ymax + 20),

                              (0, 255, 0), 2)

        return self.lmList, bbox

    def fingersUp(self):

        fingers = []

        # Thumb

        #print(self.lmList)

        #print(self.tipIds)

        

        if self.lmList[self.tipIds[0]][1] > self.lmList[self.tipIds[0] - 1][1]:

            fingers.append(1)

        else:

            fingers.append(0)

        # Fingers

        for id in range(1, 5):

            if self.lmList[self.tipIds[id]][2] < self.lmList[self.tipIds[id] - 2][2]:

                fingers.append(1)

            else:

                fingers.append(0)

        # totalFingers = fingers.count(1)

        return fingers

    def findDistance(self, p1, p2, img, draw=True,r=15, t=3):

        x1, y1 = self.lmList[p1][1:]

        x2, y2 = self.lmList[p2][1:]

        cx, cy = (x1 + x2) // 2, (y1 + y2) // 2

        if draw:

            cv2.line(img, (x1, y1), (x2, y2), (255, 0, 255), t)

            cv2.circle(img, (x1, y1), r, (255, 0, 255), cv2.FILLED)

            cv2.circle(img, (x2, y2), r, (255, 0, 255), cv2.FILLED)

            cv2.circle(img, (cx, cy), r, (0, 0, 255), cv2.FILLED)

        length = math.hypot(x2 - x1, y2 - y1)

        return length, img, [x1, y1, x2, y2, cx, cy]

def main():

    pTime = 0

    cTime = 0

    cap = cv2.VideoCapture(0)

    detector = handDetector()

    while True:

        

        success, img = cap.read()

        

        img = detector.findHands(img)

        lmList, bbox = detector.findPosition(img)

        if len(lmList) != 0:

            print(lmList[4])

        cTime = time.time()

        fps = 1 / (cTime - pTime)

        pTime = cTime

        fingers = detector.fingersUp()

        cv2.putText(img, str(int(fps)), (10, 70), cv2.FONT_HERSHEY_PLAIN, 3,

                    (255, 0, 255), 3)

        cv2.imshow("Image", img)

        cv2.waitKey(1)

if __name__ == "__main__":

    main()

Sekarang di dalam folder yang sama buat file python lain sebagai final.py.

Sekarang salin dan tempel kode di bawah ini, dan simpan. Namun sebelum menyimpan pastikan Anda mengubah beberapa atribut:

**Atribut wCam dan hCam harus diubah sesuai, ini adalah lebar dan tinggi webcam Anda. **

Now inside the same folder create another python file as final.py.

Now copy and paste the code below, and save it. But before saving make sure you change a few attributes:
** wCam and hCam attributes should be changed accordingly, these are the width and height of your webcam. ** 

 

import numpy as np

import track_hand as htm

import time

import autopy

import cv2

wCam, hCam = 1280, 720

frameR = 100

smoothening = 7

pTime = 0

plocX, plocY = 0, 0

clocX, clocY = 0, 0

cap = cv2.VideoCapture(0)

cap.set(3, wCam)

cap.set(4, hCam)

detector = htm.handDetector(maxHands=1)

wScr, hScr = autopy.screen.size()

# print(wScr, hScr)

while True:

    # 1. Find hand Landmarks

    fingers=[0,0,0,0,0]

    success, img = cap.read()

    img = detector.findHands(img)

    lmList, bbox = detector.findPosition(img)

    # 2. Get the tip of the index and middle fingers

    if len(lmList) != 0:

        x1, y1 = lmList[8][1:]

        x2, y2 = lmList[12][1:]

        # print(x1, y1, x2, y2)

    

    # 3. Check which fingers are up

        fingers = detector.fingersUp()

    # print(fingers)

    cv2.rectangle(img, (frameR, frameR), (wCam - frameR, hCam - frameR),

    (255, 0, 255), 2)

    # 4. Only Index Finger : Moving Mode

    if fingers[1] == 1 and fingers[2] == 0:

        # 5. Convert Coordinates

        x3 = np.interp(x1, (frameR, wCam - frameR), (0, wScr))

        y3 = np.interp(y1, (frameR, hCam - frameR), (0, hScr))

        # 6. Smoothen Values

        clocX = plocX + (x3 - plocX) / smoothening

        clocY = plocY + (y3 - plocY) / smoothening

    

        # 7. Move Mouse

        autopy.mouse.move(wScr - clocX, clocY)

        cv2.circle(img, (x1, y1), 15, (255, 0, 255), cv2.FILLED)

        plocX, plocY = clocX, clocY

        

    # 8. Both Index and middle fingers are up : Clicking Mode

    if fingers[1] == 1 and fingers[2] == 1:

        # 9. Find distance between fingers

        length, img, lineInfo = detector.findDistance(8, 12, img)

        print(length)

        # 10. Click mouse if distance short

        if length < 40:

            cv2.circle(img, (lineInfo[4], lineInfo[5]),

            15, (0, 255, 0), cv2.FILLED)

            autopy.mouse.click()

    

    # 11. Frame Rate

    cTime = time.time()

    fps = 1 / (cTime - pTime)

    pTime = cTime

    cv2.putText(img, str(int(fps)), (20, 50), cv2.FONT_HERSHEY_PLAIN, 3,

    (255, 0, 0), 3)

    # 12. Display

    cv2.imshow("Image", img)

    cv2.waitKey(1)

 

---

Testing

Pengujian

Sekarang jalankan kode di atas, sesuatu yang mirip dengan di bawah ini harus terlihat.

Gambar harus melacak semua tangan dan jari.

Sekarang saat Anda menggerakkan tangan di dalam kotak pembatas merah muda, kursor bergerak. Untuk mengklik, menaikkan angka tengah, dan meletakkannya, tempat kursor berada, mengklik akan terjadi.

Selamat setengah dari pekerjaan selesai sekarang. Mari pindah ke bagian lain yaitu perangkat atau bagian yang disematkan.

Now run the above code, something similar to below must be visible.

The image should be tracking all the whole hand and fingers.

Now as you move the hand inside the pink bounding box the cursor moves. To click, raise the middle figure, and put it down, the place where the cursor is, clicking will take place.

Congratulations half the work is done now. Let’s move to another part that is the device or embedded part.

---

ESP32 CAM Module

Modul CAM ESP32

Modul Kamera Berbasis ESP32 yang dikembangkan oleh AI-Thinker. Kontroler didasarkan pada CPU 32-bit & memiliki gabungan Wi-Fi + Bluetooth/BLE Chip. Ini memiliki SRAM 520 KB built-in dengan PSRAM 4M eksternal. Pin GPIO-nya memiliki dukungan seperti UART, SPI, I2C, PWM, ADC, dan DAC.

Modul tersebut digabungkan dengan Modul Kamera OV2640 yang memiliki Resolusi Kamera tertinggi hingga 1600 × 1200.

Kamera terhubung ke ESP32 CAM Board menggunakan konektor berlapis emas 24 pin. Papan mendukung Kartu SD hingga 4GB. Kartu SD menyimpan gambar yang diambil.

Untuk mempelajari secara detail tentang Modul Kamera ESP32, Anda dapat merujuk ke pravious Tutorial Memulai ini.

The ESP32 Based Camera Module developed by AI-Thinker. The controller is based on a 32-bit CPU & has a combined Wi-Fi + Bluetooth/BLE Chip. It has a built-in 520 KB SRAM with an external 4M PSRAM. Its GPIO Pins have support like UART, SPI, I2C, PWM, ADC, and DAC.

The module combines with the OV2640 Camera Module which has the highest Camera Resolution up to 1600 × 1200. The camera connects to the ESP32 CAM Board using a 24 pins gold plated connector. The board supports an SD Card of up to 4GB. The SD Card stores capture images.

To learn in detail about the ESP32 Camera Module you can refer to our previous Getting Started Tutorial.

---

ESP32-CAM FTDI Connection

The board doesn’t have a programmer chip. So In order to program this board, you can use any type of USB-to-TTL Module. There are so many FTDI Module available based on CP2102 or CP2104 Chip or any other chip.

Make a following connection between FTDI Module and ESP32 CAM module.

ESP32-CAM	FTDI Programmer
GND	GND
5V	VCC
U0R	TX
U0T	RX
GPIO0	GND

Hubungkan Pin 5V & GND ESP32 ke 5V & GND Modul FTDI. Demikian pula, sambungkan Rx ke UOT dan Tx ke Pin UOR.

Dan yang paling penting, Anda harus menyingkat Pin IO0 dan GND secara bersamaan. Ini untuk menempatkan perangkat dalam mode pemrograman. Setelah pemrograman selesai, Anda dapat menghapusnya.

Connect the 5V & GND Pin of ESP32 to 5V & GND of Friday Module. Similarly, connect the Rx to UOT and Tx to UOR Pin. And the most important thing, you need to short the IO0 and GND Pin together. This is to put the device in programming mode. Once programming is done you can remove it.

---

Installing ESP32CAM Library 

Menginstal Librari ESP32CAM

   

Di sini kita tidak akan menggunakan contoh server web ESP umum, melainkan proses streaming lainnya. Oleh karena itu kita perlu menambahkan library ESPCAM lainnya. Pustaka esp32cam menyediakan API berorientasi objek untuk menggunakan kamera OV2640 di ESP32

mikrokontroler. Ini adalah pembungkus perpustakaan esp32-camera.

Pergi ke berikut iniTautan Githubdan unduh pustaka zip seperti pada gambar Setelah diunduh tambahkan pustaka zip ini ke Folder Arduino Libray. 

Untuk melakukannya ikuti langkah-langkah berikut: 

Open Arduino -> Sketch -> Include Library -> Add .ZIP Library… -> Navigate to downloaded zip file -> add

Buka Arduino -> Sketsa -> Sertakan Perpustakaan -> Tambahkan Perpustakaan .ZIP… -> Arahkan ke file zip yang diunduh -> tambahkan 

  

Here we will not use the general ESP webserver example rather another streaming process. Therefore we need to add another ESPCAM library. The esp32cam library provides an object oriented API to use OV2640 camera on ESP32 microcontroller. It is a wrapper of esp32-camera library.

Go to the following Github Link and download the zip library as in the image

Once downloaded add this zip library to Arduino Libray Folder. To do so follow the following steps:
Open Arduino -> Sketch -> Include Library -> Add .ZIP Library… -> Navigate to downloaded zip file -> add

---

Source Code/Program for ESP32 CAM Module

Kode Sumber/Program untuk Modul CAM ESP32

Kode sumber/program ESP32 CAM Gesture Controlled Mouse dapat ditemukan di Contoh Perpustakaan. 

Jadi buka 

Files -> Examples -> esp32cam -> WifiCam.

Sebelum Mengunggah kode, Anda harus membuat sedikit perubahan pada kode. Ubah variabel SSID dan password dan sesuai dengan jaringan WiFi Anda.

Sekarang kompilasi dan unggah ke ESP32 CAM Board. Tetapi selama mengunggah, Anda harus mengikuti beberapa langkah setiap saat.

Pastikan pin IO0 korsleting dengan ground saat Anda telah menekan tombol unggah.

Jika Anda melihat titik dan garis saat mengunggah, tekan tombol reset segera. Setelah kode diunggah, lepaskan korsleting pin I01 dengan Ground dan tekan tombol reset sekali lagi.

Jika output Serial monitor masih tidak ada maka tekan tombol reset kembali.

Sekarang Anda dapat melihat output yang sama seperti pada gambar di bawah ini. 

The source code/program ESP32 CAM Gesture Controlled Mouse can be found in Library Example. So go to Files -> Examples -> esp32cam -> WifiCam.

Before Uploading the code you have to make a small change to the code. Change the SSID and password variable and in accordance with your WiFi network.

Now compile and upload it to the ESP32 CAM Board. But during uploading, you have to follow few steps every time.

• Make sure the IO0 pin is shorted with the ground when you have pressed the upload button.
• If you see the dots and dashes while uploading press the reset button immediately
• Once the code is uploaded, remove the I01 pin shorting with Ground and press the reset button once again.
• If the output is the Serial monitor is still not there then press the reset button again.

Now you can see a similar output as in the image below.

So setting up ESP32-CAM part is done now. The ESP32-CAM is transmitting the live video, make sure that you copy this IP address displayed.

---

Python Code + Gesture Controlled Virtual Mouse with ESP32-CAM

Lets move to the final part of the Gesture Controlled Virtual Mouse with ESP32-CAM project. So, we again come back to our final.py code and do some changes by or simply paste the code given.

 

import numpy as np

import track_hand as htm

import time

import autopy

import cv2

 

url="http://192.168.1.61/cam-hi.jpg"

wCam, hCam = 800, 600

frameR = 100

smoothening = 7

pTime = 0

plocX, plocY = 0, 0

clocX, clocY = 0, 0

'''cap = cv2.VideoCapture(0)

cap.set(3, wCam)

cap.set(4, hCam)'''

 

detector = htm.handDetector(maxHands=1)

wScr, hScr = autopy.screen.size()

while True:

    # 1. Find hand Landmarks

    fingers=[0,0,0,0,0]

    #success, img = cap.read()

    

    img_resp=urllib.request.urlopen(url)

    imgnp=np.array(bytearray(img_resp.read()),dtype=np.uint8)

    img=cv2.imdecode(imgnp,-1)

    

    img = detector.findHands(img)

    lmList, bbox = detector.findPosition(img)

    

    # 2. Get the tip of the index and middle fingers

    if len(lmList) != 0:

        x1, y1 = lmList[8][1:]

        x2, y2 = lmList[12][1:]

        # print(x1, y1, x2, y2)

    

    # 3. Check which fingers are up

        fingers = detector.fingersUp()

        

    cv2.rectangle(img, (frameR, frameR), (wCam - frameR, hCam - frameR),

    (255, 0, 255), 2)

    # 4. Only Index Finger : Moving Mode

    if fingers[1] == 1 and fingers[2] == 0:

        # 5. Convert Coordinates

        x3 = np.interp(x1, (frameR, wCam - frameR), (0, wScr))

        y3 = np.interp(y1, (frameR, hCam - frameR), (0, hScr))

        # 6. Smoothen Values

        clocX = plocX + (x3 - plocX) / smoothening

        clocY = plocY + (y3 - plocY) / smoothening

    

        # 7. Move Mouse

        autopy.mouse.move(wScr - clocX, clocY)

        cv2.circle(img, (x1, y1), 15, (255, 0, 255), cv2.FILLED)

        plocX, plocY = clocX, clocY

        

    # 8. Both Index and middle fingers are up : Clicking Mode

    if fingers[1] == 1 and fingers[2] == 1:

        # 9. Find distance between fingers

        length, img, lineInfo = detector.findDistance(8, 12, img)

        print(length)

        # 10. Click mouse if distance short

        if length < 40:

            cv2.circle(img, (lineInfo[4], lineInfo[5]),

            15, (0, 255, 0), cv2.FILLED)

            autopy.mouse.click()

    

    # 11. Frame Rate

    cTime = time.time()

    fps = 1 / (cTime - pTime)

    pTime = cTime

    cv2.putText(img, str(int(fps)), (20, 50), cv2.FONT_HERSHEY_PLAIN, 3,

    (255, 0, 0), 3)

    # 12. Display

    cv2.imshow("Image", img)

    cv2.waitKey(1)

 

Di sini, di kode di atas, pastikan Anda memperbarui variabel URL Anda, sesuai dengan IP yang ditampilkan di Arduino IDE

Pemantau serial. Juga, perbarui variabel wCam dan hCam sesuai dengan resolusi yang ditampilkan.

Sekarang jalankan kode aliran Nirkabel ESP32Cam dengan pelacakan mouse harus terlihat dan memungkinkan.

Dengan demikian, Mouse Virtual Terkendali Gerakan Virtual nirkabel kami dengan ESP32-CAM & OpenCV dibuat.

Here in the above code make sure you update your URL variable, according to the IP displayed on the Arduino IDE Serial monitor. Also, update the wCam and hCam variables according to the resolution being displayed.

Now run the code the Wireless stream of ESP32Cam with tracking of the mouse must be visible and possible.

Thus, our wireless Virtual Gesture Controlled Virtual Mouse with ESP32-CAM & OpenCV is made.

---

Video Tutorial & Guide

Gesture Recognition & Tracking with ESP32 Camera & OpenCV || Make Gesture Controlled Mouse

Watch this video.