{"id":877,"date":"2022-02-07T08:36:03","date_gmt":"2022-02-07T07:36:03","guid":{"rendered":"https:\/\/med.upc.edu\/team4-2021\/?page_id=877"},"modified":"2022-05-23T13:07:12","modified_gmt":"2022-05-23T12:07:12","slug":"3d-mapping-and-real-time-object-detection","status":"publish","type":"page","link":"https:\/\/med.upc.edu\/team4-2021\/3d-mapping-and-real-time-object-detection\/","title":{"rendered":"3D mapping and real-time object detection"},"content":{"rendered":"\n<p class=\"has-text-align-center has-black-color has-pale-cyan-blue-background-color has-text-color has-background has-large-font-size wp-block-paragraph\"><strong>INTRODUCTION<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Applications and Technologies for Unmanned Aircfrat Systems (PAYLOAD)<\/p>\n\n\n\n<p class=\"has-black-color has-pale-cyan-blue-background-color has-text-color has-background has-medium-font-size wp-block-paragraph\"><strong>OUR GOAL<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The idea is to connect and install a camera on the Raspberry and mount the Raspberry on the drone platform. In this way, images can be taken during the flight and the drone can take decisions during the flight based on these images.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"has-black-color has-pale-cyan-blue-background-color has-text-color has-background wp-block-paragraph\"><strong>In the next project our goal will cover the following topics:<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">1.-Setting up the RPi camera module.<br>2.-Accessing the RPi camera using Python and OpenCV.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">3.-Sending live video from the RPi to the computer.<\/p>\n\n\n\n<p class=\"has-black-color has-pale-cyan-blue-background-color has-text-color has-background wp-block-paragraph\"><strong>The steps will be:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Connecting the camera<\/strong><\/li><li><strong>Enabling the camera<\/strong><\/li><li><strong>Test out the camera module<\/strong><\/li><li><strong>Installing OpenCV<\/strong><\/li><li><strong>Capturing a single image from the camera.<\/strong><\/li><li><strong>Step and problem found during the process.<\/strong><\/li><\/ul>\n\n\n\n<p class=\"has-black-color has-luminous-vivid-amber-background-color has-text-color has-background wp-block-paragraph\"><strong>We will try to explain the process step by step and comment on the problems encountered and how we have solved them.<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Some PC have had to reinstall the GUI graphic interface for Ubuntu.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Setup WiFi: cmd.exe and ipconfig. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\"><strong>Connecting the camera.<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We just follow the steps in the PDF file, make sure to attach properly  the flexible ribbon cable.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Squeeze black tab<\/strong>!!!!<\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img fetchpriority=\"high\" decoding=\"async\" src=\"https:\/\/med.upc.edu\/team4-2021\/wp-content\/uploads\/sites\/5\/2022\/02\/20220218_17165215582-767x1024.jpg\" alt=\"\" class=\"wp-image-936\" width=\"341\" height=\"455\" srcset=\"https:\/\/med.upc.edu\/team4-2021\/wp-content\/uploads\/sites\/5\/2022\/02\/20220218_17165215582-767x1024.jpg 767w, https:\/\/med.upc.edu\/team4-2021\/wp-content\/uploads\/sites\/5\/2022\/02\/20220218_17165215582-225x300.jpg 225w, https:\/\/med.upc.edu\/team4-2021\/wp-content\/uploads\/sites\/5\/2022\/02\/20220218_17165215582-768x1025.jpg 768w, https:\/\/med.upc.edu\/team4-2021\/wp-content\/uploads\/sites\/5\/2022\/02\/20220218_17165215582-1151x1536.jpg 1151w, https:\/\/med.upc.edu\/team4-2021\/wp-content\/uploads\/sites\/5\/2022\/02\/20220218_17165215582.jpg 1403w\" sizes=\"(max-width: 341px) 100vw, 341px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Open Ubuntu and terminal, and install the libraries:<\/p>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\"><strong>Enabling the camera<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>File for editing<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">s<em>udo nano \/boot\/firmware\/config.txt<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Important to add !!!!<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>start_x=1<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Reboot the system<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Updating OS:<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>sudo apt update<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>sudo apt upgrade -y<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Install libraries:<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">sudo apt install libraspberrypi-bin libraspberypi0&#8230;&#8230;&#8230;..etc<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Open terminal an execute <\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Test out the camera module:<\/strong><\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">We need to check that the RPi camera is working properly. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">&#8220;<em>raspistill -o output.jpg<\/em>&#8221; is the command line tool for capturing photographs with the RPi camera.<\/p>\n\n\n\n<p class=\"has-black-color has-pale-pink-background-color has-text-color has-background has-medium-font-size wp-block-paragraph\">Troubleshooting<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">#pip3 is not installed in my Ubuntu<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">search in  google:  python3-pip<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/www.odoo.com\/es_ES\/forum\/ayuda-1\/how-to-install-pip-in-python-3-on-ubuntu-18-04-167715\" data-type=\"URL\" data-id=\"https:\/\/www.odoo.com\/es_ES\/forum\/ayuda-1\/how-to-install-pip-in-python-3-on-ubuntu-18-04-167715\">www.odoo.com\/es_ES\/forum\/ayuda-1\/how-to-install-pip-in-python-3-on-ubuntu-18-04-167715<\/a><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt install python3-pip<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\"><strong>Install the packages to acces the RPi camera using Python and OpenCv<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Installing OpenCv<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">sudo apt install libgl11-mesa-glx<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">sudo apt install python3-pip python3-dev<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Install the library for images<\/strong>:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">sudo pip3 install -U opencv-contrib-python <\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Check the installation<\/strong>:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em> python -c &#8220;import cv2&#8221;<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Now, we try the camera:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">unplug and plug the camera again and write the code:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">import cv2<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">#open the camera for video capturing<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>cap = cv2. VideoCapture (0)&#8230;..etc<\/strong><\/p>\n\n\n\n<p class=\"has-black-color has-pale-pink-background-color has-text-color has-background has-medium-font-size wp-block-paragraph\">Troubleshooting<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The camera is not working, we tried to check the code with my camera web and it works, we tried to reconnect the cables, the flexible ribbon,etc but it doesnt work.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We gonna try with the other camera provided by UPC, Intel RealSense D435.<\/p>\n\n\n\n<p class=\"has-black-color has-pale-pink-background-color has-text-color has-background has-medium-font-size wp-block-paragraph\">Troubleshooting<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We have tested the codes to see that the camera and the libraries are working, and have been executed correctly. In the final part, on having to send live video to the station, we have followed the steps of obtaining the IP and configuring the code. Also, we have found some conversion from OpenCv to the &#8220;station&#8221; to modify the video. I.E, pose the image in grayscale. Note that we have tried to transform the image to the raspberryPi and send it directly in grayscale, but Netgear didnt accept  the name of dimensions (2 expected 3), which must be because Opencv does not take care of the channel in the black and white images. Affecting a new dummy dimension with numpy doesn&#8217;t work either, which seems to be quite limited in this aspect of NetGear.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>NOTE!!!  If I try the Intelreal camera, try both USB 3, which has more bandwidth.<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Install the packages script\/path-realsense.ubuntu.lts as administrator<\/p>\n\n\n\n<p class=\"has-black-color has-pale-pink-background-color has-text-color has-background has-medium-font-size wp-block-paragraph\">Troubleshooting<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The packages do not appear for the version of Ubuntu 21, only till the 18th version.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We realize that the packages are not relevant, they are simply the software associated with the camera (the SDK). The problem was that the VideoCapture index was incorrect, since there are several devices that can fetch the camera. If I have index -1, OpenCv selects a camera automatically, which was the problem. When changing to an index 2, it works!!!!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"has-black-color has-pale-cyan-blue-background-color has-text-color has-background wp-block-paragraph\">     <strong>   Script GROUND STATION<\/strong><\/p>\n\n\n\n<h1 class=\"has-medium-font-size wp-block-heading\" id=\"import-required-libraries\">#import required libraries<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">from vidgear.gears import NetGear<br>import cv2<\/p>\n\n\n\n<h1 class=\"has-medium-font-size wp-block-heading\" id=\"define-tweak-flags\">#define tweak flags<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">options = {&#8220;flag&#8221; : 0, &#8220;copy&#8221; : False, &#8220;track&#8221; : False}<\/p>\n\n\n\n<h1 class=\"has-medium-font-size wp-block-heading\" id=\"define-netgear-client-at-fiven-ip-adrress-and-define-parameters-77-home-ip-address\">#Define Netgear Client at given IP address and define the parameters,(&#8230; .77) home IP address.<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">client = NetGear(<br>address = &#8220;192.168.1.77&#8221;,<br>port = &#8220;5454&#8221;,<br>protocol = &#8220;tcp&#8221;,<br>pattern = 0,<br>receive_mode = True,<br>logging = True,<br>**options)<\/p>\n\n\n\n<h1 class=\"has-medium-font-size wp-block-heading\" id=\"loop-over\">#loop over<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">while True:<br>frame = client.recv()<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>if frame is None:\n    break\ngray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)\ncv2.imshow(\"Output Frame\", frame)\n\nkey = cv2.waitKey(1) &amp; 0xFF\nif key == ord(\"q\"):\n    break<\/code><\/pre>\n\n\n\n<p class=\"has-medium-font-size wp-block-paragraph\"><strong>#close output window<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">cv2.destroyAllWindows()<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>#safely close client<\/strong><br>client.close()<\/p>\n\n\n\n<p class=\"has-black-color has-pale-cyan-blue-background-color has-text-color has-background wp-block-paragraph\">Script Dron time face detection and video using RPi camera<\/p>\n\n\n\n<p class=\"has-black-color has-pale-cyan-blue-background-color has-text-color has-background wp-block-paragraph\">dron_simulator_cameraface.py is the name of the file<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong> #import required libraries<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">from vidgear .gears import NetGear<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">impor cv2&#8230;&#8230;&#8230;&#8230;&#8230;&#8230;&#8230;&#8230;.etc the same code as provided in the PDF file.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">from vidgear.gears import NetGear\nimport cv2\nimport numpy as np\n\nstream = cv2.VideoCapture(4)\n#define tweak flags\n\noptions = {\"flag\" : 0, \"copy\" : False, \"track\" : False}\n\n#define netgear server at given IP address and define parameters\n\nserver = NetGear(\n\taddress = \"192.168.1.77\",\n\tport = \"5454\",\n\tprotocol = \"tcp\",\n\tpattern = 0,\n\tlogging = True,\n\t**options\n)\n\n#load default  face cascade classifier\nface_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')\n\n\nwhile True:\n\ttry:\n\t\t(grabbed, frame)  = stream.read()\n\t\tif not grabbed:\n\t\t\tbreak\n\t\t\t\n\t\t# Process the frame\n\t\tgray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)\n\t\tfaces = face_cascade.detectMultiScale(gray, 1.3, 5)\n\t\tfor (x,y,w,h) in faces:\n\t\t\tframe = cv2.rectangle(frame, (x,y), (x+w, y+h), (255,0,0), 2)\n\t\t\t\n\t\t#send frame\n\t\tserver.send(frame)\n\t\t\n\texcept KeyboardInterrupt:\n\t\tbreak\n#safely close video\nstream.release()\n\n#safely close server\nserver.close()<\/pre>\n\n\n\n<p class=\"has-text-align-center has-black-color has-vivid-red-background-color has-text-color has-background has-medium-font-size wp-block-paragraph\"><strong>RESULTS    <\/strong>   <\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Capturing a single image from the camera.<\/strong>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         <\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" width=\"1024\" height=\"576\" src=\"https:\/\/med.upc.edu\/team4-2021\/wp-content\/uploads\/sites\/5\/2022\/02\/Captura-10-1024x576.png\" alt=\"\" class=\"wp-image-941\" srcset=\"https:\/\/med.upc.edu\/team4-2021\/wp-content\/uploads\/sites\/5\/2022\/02\/Captura-10-1024x576.png 1024w, https:\/\/med.upc.edu\/team4-2021\/wp-content\/uploads\/sites\/5\/2022\/02\/Captura-10-300x169.png 300w, https:\/\/med.upc.edu\/team4-2021\/wp-content\/uploads\/sites\/5\/2022\/02\/Captura-10-768x432.png 768w, https:\/\/med.upc.edu\/team4-2021\/wp-content\/uploads\/sites\/5\/2022\/02\/Captura-10-1536x864.png 1536w, https:\/\/med.upc.edu\/team4-2021\/wp-content\/uploads\/sites\/5\/2022\/02\/Captura-10.png 1920w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption>Sending an image from the Rpi to the ground station using NetGears. The processing step of facial recognition is done in the RPi due to NetGears limitations.<\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>INTRODUCTION Applications and Technologies for Unmanned Aircfrat Systems (PAYLOAD) OUR GOAL The idea is to connect and install a camera on the Raspberry and mount the Raspberry on the drone platform. In this way, images can be taken during the flight and the drone can take decisions during the flight based on these images. In [&hellip;]<\/p>\n","protected":false},"author":6,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-877","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/med.upc.edu\/team4-2021\/wp-json\/wp\/v2\/pages\/877","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/med.upc.edu\/team4-2021\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/med.upc.edu\/team4-2021\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/med.upc.edu\/team4-2021\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/med.upc.edu\/team4-2021\/wp-json\/wp\/v2\/comments?post=877"}],"version-history":[{"count":31,"href":"https:\/\/med.upc.edu\/team4-2021\/wp-json\/wp\/v2\/pages\/877\/revisions"}],"predecessor-version":[{"id":1102,"href":"https:\/\/med.upc.edu\/team4-2021\/wp-json\/wp\/v2\/pages\/877\/revisions\/1102"}],"wp:attachment":[{"href":"https:\/\/med.upc.edu\/team4-2021\/wp-json\/wp\/v2\/media?parent=877"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}