deestream-测雷达RGBD数据

       deepstreeam SDK包含很多sample, 其中deepstream-3d-depth-camera可以用来测RGBD数据，这个例子的详细说明可以参考readme，主要用来从Intel realsense camera(DS435)读取RGBD数据，然后做处理显示, pipeline有两种,一种是source+render, 一种是source+points/color转pointcloud+render.

       Intel  realsense深度相机有linux版的SDK，详见，基于此SDK, deepstream-3d-depth-camera可以基于此SDK采集相机实时流。

stairs-D435.bag是一个单个文件，还需要用SDK将录像解析为depth和color数据，解析代码如下，主要思路是用sdk解析文件，然后存个20帧，要注意的是color数据是rgb的，而deepstream-3d-depth-camera用的是rgba数据。

#include <librealsense2/rs.hpp> // Include RealSense Cross Platform API

#include <fstream>              // File IO
#include <iostream>             // Terminal IO
#include <sstream>              // Stringstreams

 
float get_depth_scale(rs2::device dev)
{
    // Go over the device's sensors
    for (rs2::sensor& sensor : dev.query_sensors())
    {
        // Check if the sensor if a depth sensor
        if (rs2::depth_sensor dpt = sensor.as<rs2::depth_sensor>())
        {
            return dpt.get_depth_scale();
        }
    }
    throw std::runtime_error("Device does not have a depth sensor");
}

// This sample captures 30 frames and writes the last frame to disk.
// It can be useful for debugging an embedded system with no display.
int main(int argc, char * argv[]) 
{
    // Declare depth colorizer for pretty visualization of depth data
    rs2::colorizer color_map;

    // Declare RealSense pipeline, encapsulating the actual device and sensors
    rs2::pipeline pipe;
    // Start streaming with default recommended configuration
    //pipe.start();
    rs2::config cfg;
    std::string file = "stairs-D435.bag";
    cfg.enable_device_from_file(file);
    rs2::stream_profile depthStream;
    rs2::stream_profile colorStream;
    rs2::pipeline_profile profile =  pipe.start(cfg); // Load from file
    float depth_scale = get_depth_scale(profile.get_device());
    if (1) {
        //printf  depthStream parameters
        std::cout << "printf  depthStream parameters" << std::endl;
        depthStream = profile.get_stream(RS2_STREAM_DEPTH);
        auto stream = depthStream.as<rs2::video_stream_profile>();
        auto intr = stream.get_intrinsics();
        std::cout << "_depthIntrinsics.width:" << intr.width << std::endl
            << "_depthIntrinsics.height:" << intr.height << std::endl
            << "_depthIntrinsics.centerX:" << intr.ppx << std::endl
            << "_depthIntrinsics.centerY:" << intr.ppy << std::endl
            << "_depthIntrinsics.fx:" << intr.fx << std::endl
            << "_depthIntrinsics.fy:" << intr.fy << std::endl;
    }
    if (1) {
        //printf  colorStream parameters
        std::cout << "printf  depth parameters" << std::endl;
        colorStream = profile.get_stream(RS2_STREAM_COLOR);
        auto stream = colorStream.as<rs2::video_stream_profile>();
        auto intr = stream.get_intrinsics();
        std::cout << "_depthIntrinsics.width:" << intr.width << std::endl
            << "_colorIntrinsics.height:" << intr.height << std::endl
            << "_colorIntrinsics.centerX:" << intr.ppx << std::endl
            << "_colorIntrinsics.centerY:" << intr.ppy << std::endl
            << "_colorIntrinsics.fx:" << intr.fx << std::endl
            << "_colorIntrinsics.fy:" << intr.fy << std::endl;
    }

    int width, height;
    int len; 
    char* pData;
    int lastNum = 0;
    int firstNum = 0;
    bool bSave = true;
    int frameNum = 0;
    while (true)
    {
        // Block program until frames arrive
        rs2::frameset data;
        if (pipe.poll_for_frames(&data))
        {
             std::cout << data.get_frame_number() << "\n";
            std::cout << data.get_frame_timestamp_domain() << "\n";
            std::cout << "timestamp:" << data.get_timestamp() << "\n";
 
            //std::cout << data.get_profile() << "\n";
            // handle frame
            // Try to get a frame of a depth image
            auto depthFrame = data.get_depth_frame();
            // Get the depth frame's dimensions
            width = depthFrame.get_width();
            height = depthFrame.get_height();
            pData = (char*)depthFrame.get_data();
            len = depthFrame.get_data_size();
            printf("d w:%d,h:%d,len:%d\n", width, height, len);
            if (bSave) {  //save
                std::ofstream depthFile("depth.raw", std::ios::binary | std::ios::app);
                depthFile.write(pData, len);
                depthFile.close();
            }
            // Query the distance from the camera to the object in the center of the image
            float dist_to_center = depthFrame.get_distance(width / 2, height / 2);
            // Print the distance
            std::cout << "The camera is facing an object " << dist_to_center << " meters away \n";
            // Try to get a frame of a color image
            // Get the color frame's dimensions
            auto colorFrame = data.get_color_frame();
            width = colorFrame.get_width();
            height = colorFrame.get_height();
            pData = (char*)colorFrame.get_data();
            len = colorFrame.get_data_size();
            printf("v w:%d,h:%d\n", width, height);
            if (bSave) {
                std::ofstream colorFile("color.raw", std::ios::binary | std::ios::app);
                for (int i = 0; i < len / 3; i++) {
                    colorFile.write(pData + i * 3, 3);
                    char c = 0;
                    colorFile.write(&c, 1);
                }
                colorFile.close();
            }
            frameNum++;
            if (frameNum == 20) {
                break;
            }
        } 
    }

     return EXIT_SUCCESS;
}
 

程序打印的参数，也是deepstream-3d-depth-camera配置需要的，这些都是深度相机的固定参数，解释详见realsense的SDK。

printf  depthStream parameters
_depthIntrinsics.width:1280
_depthIntrinsics.height:720
_depthIntrinsics.centerX:636.179
_depthIntrinsics.centerY:368.436
_depthIntrinsics.fx:636.073
_depthIntrinsics.fy:636.073
printf  depth parameters
_depthIntrinsics.width:1920
_colorIntrinsics.height:1080
_colorIntrinsics.centerX:973.773
_colorIntrinsics.centerY:529.999
_colorIntrinsics.fx:1367.57
_colorIntrinsics.fy:1363.07

测试source  + render

$ ./deepstream-3d-depth-camera  -c ds_3d_realsense_depth_capture_render.yaml

测试source + points/color转pointcloud + render 

$ ./deepstream-3d-depth-camera  -c ds_3d_realsense_depth_to_point_cloud.yaml

deepstream问题，可以发到deepstream论坛，有专人回复