RoboEvaluate The Performance Of Your LiDARs And Multi-Sensor Systems
Integrating LiDAR scanners into self-driving vehicles is undoubtedly the most discussed topic in the field of intelligent driving in 2021. It’s a uniform understanding of the automotive industry that the excellent perception capability of 3D LiDAR sensor will make autonomous driving and smart transportation safer and more comfortable.
However, the questions are how much the added high-resolution LiDAR can help to improve the perception capability of the whole intelligent perception system? Will it meet the expectation of the industry? How exactly is the overall performance of the multi-sensor fusion system of 3D laser LiDAR, camera and millimeter-wave radar?
The best way to figure out the above questions is to evaluate the system’s performance.
In the strict development system of the automotive industry, the process of testing and verifying performance is as important as the R&D and design process. This process of objectively testing and evaluating the performance of the perception sensor system based on the project requirements and goals can clearly point out the development direction at each key phase and keep the project in the state of “Know How To Do”.
LiDAR’s reliability and AI algorithm detection performance are higher than that of cameras and millimeter-wave radars. The standards and tools used in the evaluation of cameras and millimeter-wave radars do not meet the requirements of evaluating LiDAR.
Therefore, a set of high-level evaluation standards and tools is required for the evaluation of 3D laser LiDAR and perception sensor fusion systems.
Where the evaluation system can be compared to an exam, the ground truth data will be the “reference answer” to the evaluation result of the perception system.
Therefore, the accuracy of ground truth data must be significantly higher than the results given by the device under test (DuT) in all aspects including detection performance and geometric precision.
The ground truth data, usually stored in the PB-Level, includes dynamic information such as obstacle type, speed, and location, and static information such as lane lines and road boundaries.
During the constant and rapid development iteration, the requirements for ground truth data are always renewing along with the testing-and-R&D upgrading cycles. Therefore, the efficiency of ground truth data generation must keep up with the development requirements.
How to obtain high-quality PB-level ground truth data？
As mentioned above, data labeling quality and data generation efficiency are the two key aspects when considering ground truth data.
The traditional manual labeling method takes an extremely long time and a huge cost to label the targets frame-by-frame. It takes three years for staff to process the amount of data collected in just one day, where the labeling rate is about 1:1000 (data collected in one day takes about 1,000 days to label). The efficiency of manual labeling is too low to meet the frequently updated testing requirements.
In terms of labeling capabilities, the manual labeling method has significant defects: due to the frame-by-frame static data labeling method, the motion speed and acceleration of dynamic objects cannot be labeled. Nevertheless, the shape of objects cannot be completely reconstructed in a single frame data, (such as vehicles in front are usually marked as an L-shape in the single frame of point cloud), therefore, the accuracy of the size of bounding boxes cannot be guaranteed.
The RS-reference 2.1 can efficiently evaluate the performance of the LiDAR and multi-sensor fusion system.
The RS-Reference was launched to market in 2016 when the automotive-grade MEMS solid-state LiDAR RS-LiDAR-M1 project was established. The RS-Reference has been used and recognized by many global top OEMs and Tier1s.
The RS-Reference provides a set of intelligent ground truth generation and evaluation solution, which can output detection performance and geometric error indicators with a labeling efficiency close to 1:1. This is significantly more accurate than real-time perception, manual labeling and traditional labeling tools.
Ø High-performance and high-maturity data collection sensor system
The RS-Reference system includes the RoboSense 128-beam LiDAR RS-Ruby, Leopard camera, Continental 408 millimeter-wave radar, GI-6695 RTK, and has added two RoboSense RS-Bpearl LiDAR for near-field blind spots detection in the 2.1 version. The whole system can achieve full coverage and perception redundancy, reliability and accuracy of original environment data.
Ø Detachable roof-mounted deployment requires no vehicle body modification:
The RS-Reference system can adapt to different vehicle sizes, and does not occupy the sensor installation position of the DuTs. It can directly evaluate the intelligent driving system that is consistent with the sensor sets of commercial vehicles.
Ø Highly mature perception algorithm and dedicated offline processing mechanism
The algorithm is the key for smart labeling to outperform manual labeling, it is responsible for the extraction of ground truth data. The RS-Reference system uses a customized and dedicated offline perception algorithm, with over 13 years’ accumulated experience of LiDAR sensing algorithm development.
The RS-Reference performs a “whole process tracking and identification” for each obstacle data, and extracts all ground truth data from each frame. It can keep up with the speed and acceleration labeling, and accurately delineate the size of the bounding box with comprehensive shape and size information. The system is also able to accurately differentiate obstacles that are in close proximity in complex scenes.
Ø Full-stack evaluation tool chain
The evaluation process of sensor perception performance can be roughly divided into four major steps: data collection, offline processing to generate ground truth data, DuT evaluation and analysis, and DuT optimization and upgrade. The evaluation process is usually repeated for multiple times until a satisfactory perception solution is obtained.
The full-stack tool chain of the RoboSense RS-Reference includes data collection tools, sensor calibration tools, visualization tools, manual correction tools, evaluation tools, etc. The 2.1 version has upgraded the data management platform and has added the scene semantic labeling function that serves every step of the evaluation process.
Ø Evaluate each sensor in the multi-sensor fusion system
The RS-Reference can not only evaluate the fusion results of intelligent driving perception fusion system, but also provide dedicated evaluation tool modules based on the characteristics of different types of sensors such as high performance LiDAR, millimeter-wave radar, and camera. Moreover, customized tool modules can be developed according to customers’ requirements to carry out in-depth analysis of the performance of perception system.
Next, in the 2.2 version of the RS-Reference, we will also release the evaluation tool modules for road signs, lane line categories, scene semantic recognition and other projects.
Ø Huge potential in extended applications of RS-Reference
Support for planning and controlling algorithm development: with the help of RS-Reference, developers of intelligent driving systems can build a massive driving scenes database to support the development of the core algorithm of the “planning and controlling” system in parallel with the development of the perception system. This way, to a large extent, has overcome the constraints of sequential development of “perception→planning→control”, and can greatly accelerate the project development process.
Empowering simulation: The massive ground truth data of real scenes generated by the RS-Reference can be used to construct simulation scenes. In this way, not only the simulation scenes can get a reliable source of data, but also the authenticity of the data is greatly improved.
Evaluation of roadside perception systems: Based on the powerful ground truth data generation capability and evaluation tool chain, the RS-Reference can also evaluate the perception performance of roadside sensing system of the Vehicle-to-Infrastructure (V2I) projects, to advance the development and deployment of smart transportation projects.
Ø High Recognition from global top OEMs and Tier1
Since its launch, the RS-Reference has been used by many top OEMs and Tier1s globally to solve sensing problems of many smart driving projects, and has won high recognition and a large number of orders from customers.
Many OEMs and Tier1 began to use RS-Reference at the early stage of the development of the intelligent driving perception system project, and gradually formed a large-scale test fleet
About RoboSense
Founded in 2014, RoboSense (Suteng Innovation Technology Co., Ltd.) is the leading provider of Smart LiDAR Sensor Systems incorporating LiDAR sensors, AI algorithms and IC chipsets, that transform conventional 3D LiDAR sensors to full data analysis and comprehension systems. The company’s mission is to possess outstanding hardware and artificial intelligence capabilities to provide smart solutions that enable robots (including vehicles) to have perception capability more superior to humans. http://robosense.ai/