The Working Sensor Fusion Code

[LucasEby]

This commit contains the Sensor Fusion Code which fuses 3D bounding box detections of YOLO + LIDAR and PointPillars. The Sensor Fusion code has been integrated into the rest of Perception code. A readme file contains instructions needed to run the code.

Attached are two RVIZ visualizations demonstrating the Sensor Fusion code. The Green bounding boxes are generated by the YOLO + LIDAR node, the blue bounding boxes are generated by PointPillars, and the Red bounding boxes are the fusion of both detections.

My main contributions in this PR:
I wrote some rosbag processing code that would extract train and test image and lidar data into KITTI supported file types
Code organization: Moved helper functions multiple files used into utility files. Cleaned up library imports.
Documentation: Documented complicated sections of code and created a README.md file which has instructions for Sensor Fusion set up and testing, Docker set up instructions, and Sensor Fusion visualization instructions
I created a PointPillars node which subscribed to image and lidar data (subscribed to both so we only run it on lidar data that has a corresponding image pair inside of our specified time threshold to make it easier to match the data with the YOLO publisher). I implemented PointPillars inside of this node and fixed a bug that caused the detections to be ~30 meters off in each direction (this wasn’t documented but it required reflectivity values that were normalized between 0 and 1 instead of intensity values inside of the input tensor).
I created a docker container to integrate PointPillars into the GEMstack code base. The docker container automatically runs the PointPillars node on start up. I additionally created a bash script which will check whether the required files exist and are located in the correct locations before setting up the docker container (to make the set up process easier).
I created the initial YOLO node which detected pedestrians and published an array of 3D bounding boxes. It is similar to PedestrianDetector and ConeDetector but is used with the sensor fusion code. Instead of integrating the YOLO node inside of . By having both the YOLO and PointPillars nodes publish their bounding box arrays with the timestamps from the lidar header, I was able to take advantage of a library to automatically pair up the messages. This decreased the complexity of CombinedDetector3D
I added Bird’s Eye View Intersection over Union code into CombinedDetector3D. This paired up the YOLO and PointPillars bounding boxes in the horizontal direction and uses the YOLO bounding boxes for the z center and dimension. I found when visualizing the results that this resulted in bounding boxes that were fused better.
I wrote code which averaged the yaw orientations of the fused bounding boxes.
I added publishers which were used to visualize the YOLO, PointPillars, and CombinedDetector3D bounding boxes. Updated the readme file with the visualization instructions.
Added id tracking to the CombinedDetector3D
I integrated the Sensor Fusion code in with the rest of the perception code.
I uploaded a file (GEMstack/knowledge/detection/befusion_instructions.md) which contains the terminal commands used to set up the BEVFusion instances for Cuda 11.1 and 11.3. I felt that it was more appropriate to upload the steps in this manner rather than as the separate bash scripts that were actually used when trying to figure out the package management issues.

[sonarqubecloud]

Quality Gate failed

Failed conditions
11 Security Hotspots
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

[lukasdumasius]

This is our last pull to merge our latest sensor fusion setup into main.

• Implemented PointPillars ROS node and visualization (GEMstack/onboard/perception/sensorFusion/pointpillars_ros/scripts/)
  • Latest combined node in GEMstack/onboard/perception/combined_detection.py
• 3d model comparison metrics (Matching bounding boxes, IoU, mAP)
• Late Fusion Logic
• Setup a large ec2 training instance, AWS was able to find a reservation for p4 instance with 8 A100s for a few days.
• Tested BEVFusion, PointPainting, PointPillars, YOLOv11, YOLOv8
• Debugged some CUDA, Pytorch, mmdet libraries for the models we tested.
• Documentation for sensor fusion functions

We're pushing PointPillars combined with YOLO+Lidar late bbox fusion for simplicity and extensibility. We already see performance increase over baseline YOLO+Lidar, and we can build on some of the current SoTA 3d detectors if we keep working on this.