RGB Depth Camera in Robotics: Starting with the Realsense R435 SDK

Installing the RealSense SDK

sudo apt-key adv --keyserver keys.gnupg.net --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE || sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDEsudo add-apt-repository "deb https://librealsense.intel.com/Debian/apt-repo focal main" -u
apt-get update
> sudo apt-get install librealsense2-dkms librealsense2-utils librealsense2-dev
> dmesg[  245.117583] usb 2-1: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[ 245.138470] usb 2-1: New USB device found, idVendor=8086, idProduct=0b07, bcdDevice=50.ce
[ 245.138485] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 245.138499] usb 2-1: Product: Intel(R) RealSense(TM) Depth Camera 435
[ 245.138511] usb 2-1: Manufacturer: Intel(R) RealSense(TM) Depth Camera 435
[ 245.138523] usb 2-1: SerialNumber: 020223022670
[ 245.178153] uvcvideo: Unknown video format 00000050-0000-0010-8000-00aa00389b71
[ 245.178354] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 435 (8086:0b07)
[ 245.182825] input: Intel(R) RealSense(TM) Depth Ca as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/input/input0
[ 245.183250] uvcvideo: Unknown video format 36315752-1a66-a242-9065-d01814a8ef8a
[ 245.183265] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 435 (8086:0b07)
[ 245.187161] usbcore: registered new interface driver uvcvideo
[ 245.187169] USB Video Class driver (1.1.1)
[ 250.017248] usb 2-1: USB disconnect, device number 2
[ 250.297627] usb 2-1: new SuperSpeed Gen 1 USB device number 3 using xhci_hcd
[ 250.322570] usb 2-1: New USB device found, idVendor=8086, idProduct=0b07, bcdDevice=50.ce
[ 250.322588] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 250.322603] usb 2-1: Product: Intel(R) RealSense(TM) Depth Camera 435
[ 250.322617] usb 2-1: Manufacturer: Intel(R) RealSense(TM) Depth Camera 435
[ 250.322630] usb 2-1: SerialNumber: 020223022670
[ 250.328686] uvcvideo: Unknown video format 00000050-0000-0010-8000-00aa00389b71
[ 250.328907] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 435 (8086:0b07)
[ 250.335195] input: Intel(R) RealSense(TM) Depth Ca as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/input/input1
[ 250.336253] uvcvideo: Unknown video format 36315752-1a66-a242-9065-d01814a8ef8a
[ 250.336268] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 435 (8086:0b07)
rs-enumerate-devices 
Device info:
Name : Intel RealSense D435
Serial Number : 018322070534
Firmware Version : 05.12.14.50
Recommended Firmware Version : 05.12.15.50
Physical Port : 2-1-5
Debug Op Code : 15
Advanced Mode : YES
Product Id : 0B07
Camera Locked : YES
Usb Type Descriptor : 3.2
Product Line : D400
Asic Serial Number : 020223022670
Firmware Update Id : 020223022670

Additional System Configuration

export PYTHONPATH=$PYTHONPATH:/usr/local/lib
python3 -m pip install opencv-python numpy
import pyrealsense2.pyrealsense2 as rs

Step-By-Step Example for Image Processing

import pyrealsense2.pyrealsense2 as rs
import numpy as np
import cv2
config = rs.config()
pipeline = rs.pipeline()
pipeline_wrapper = rs.pipeline_wrapper(pipeline)
pipeline_profile = config.resolve(pipeline_wrapper)
device = pipeline_profile.get_device()
device_product_line = str(device.get_info(rs.camera_info.product_line))
found_rgb = False
for s in device.sensors:
if s.get_info(rs.camera_info.name) == "RGB Camera":
found_rgb = True
break
if not found_rgb:
print("The demo requires Depth camera with Color sensor")
exit(0)
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)if device_product_line == "L500":
config.enable_stream(rs.stream.color, 960, 540, rs.format.bgr8, 30)
else:
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)
pipeline.start(config)
aligned_image = false
while not aligned_image:
# Wait for a coherent pair of frames: depth and color
frames = pipeline.wait_for_frames()
depth_frame = frames.get_depth_frame()
color_frame = frames.get_color_frame()
if not depth_frame or not color_frame:
continue
else:
aligned_image = true
depth_image = np.asanyarray(depth_frame.get_data())
color_image = np.asanyarray(color_frame.get_data())
# Apply colormap on depth image (image must be converted to 8-bit per pixel first)
depth_colormap = cv2.applyColorMap(
cv2.convertScaleAbs(depth_image, alpha=0.03), cv2.COLORMAP_JET
)
depth_colormap_dim = depth_colormap.shape
color_colormap_dim = color_image.shape
if depth_colormap_dim != color_colormap_dim:
resized_color_image = cv2.resize(
color_image,
dsize=(depth_colormap_dim[1], depth_colormap_dim[0]),
interpolation=cv2.INTER_AREA,
)
images = np.hstack((resized_color_image, depth_colormap))
else:
images = np.hstack((color_image, depth_colormap))
print(images)
print(dir(images))
cv2.imwrite("test.png", images)
cv2.namedWindow("Live Stream", cv2.WINDOW_FULLSCREEN)
cv2.imshow("Live Stream", images)
cv2.waitKey(1)
pipeline.stop()
pipeline.stop()

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sebastian

Sebastian

IT Project Manager & Developer