In the past, driving performance was the main factor determining the sales performance of a car model. Powerful engines, dynamic handling, and low fuel consumption are the selling points that car manufacturers use to attract customers. But times are changing, and consumers are increasingly paying attention to features and comfort similar to smartphones when choosing new cars. In addition, the increasing safety requirements of regulatory agencies are driving car manufacturers to pay more attention to interior design.
The use of 3D data is a major feature that enables future interior design and sets car manufacturers apart. The simpler and more economical 3D data collection method is to use a Time of Flight (ToF) camera. This type of camera modulates infrared light to illuminate the area of interest, then detects reflected light and compares it with the transmitted signal. This will produce a 2D Grayscale. At the same time, the distance information of each pixel is obtained from the phase difference, so as to obtain the 3D information of the scene.
2D grayscale images and 3D depth information are obtained from infrared light
This method is commonly referred to as Indirect Time of Flight (iToF). Unlike traditional cameras, iToF can provide highly reliable 3D data under all ambient light conditions (such as nighttime, strong sunlight, and frequently alternating light and shadow on roads) without being affected by shadows and overexposure. Compared to other depth sensing technologies, iToF also has other advantages: the ToF camera module design is simple and robust, without the need for any mechanical baseline; Camera calibration speed is fast, making it easy and reliable for large-scale production; The computational load of the application processor is relatively small.
Monitoring Drivers
Placing the ToF sensor in the A-pillar, steering wheel, or instrument cluster can be used for the Driver Monitoring System (DMS).
ToF based DMS is a more cost-effective option. In this system, the same ToF camera used for DMS functionality can be used for secure and anti fraud facial verification, just like many smartphones and smart door locks. This algorithm uses 2D infrared images for facial recognition and uses 3D data to avoid any type of deception attack. If there is no 3D data, a simple photo is enough to deceive facial recognition algorithms; And 3D data can immediately prevent such attempts.
But why do we need 3D facial authentication on cars? As consumers become more accustomed to the convenience provided by smart phones, smart watches, Smart TV and other smart devices, they expect cars to bring more and more convenience. This means seamless access to private data and cloud services, such as streaming music, no matter where you are, even in a smart car.
And these new services also require identity verification, such as easy payment when charging electric vehicles at public charging stations. When authentication is required for a certain operation, especially for sensitive transactions, security must be provided, which can be achieved through 3D data. But there are also some less sensitive facial recognition applications. Taking personalization as an example, it enables the car to detect which driver just entered the vehicle and load appropriate settings.
Security and Convenience Applications
Once a car is on the road, 3D data can also be used to more accurately determine the position of the driver's eyes and head. This means that the projection of the Head-up display will also be more adaptable to the driver's situation.
But DMS is not the only purpose of ToF cameras. The wide field ToF camera near the rearview mirror enables a complete Occupant Monitoring System (OMS), providing support for all necessary tasks during autonomous driving, paving the way for hand held steering wheel detection and autonomous driving takeover behavior. When the vehicle takes over driving, the driver can significantly change their posture or even lie down. In order to ensure the safety of the driver, the vehicle safety system must adapt to the new posture of the driver. For example, intelligent airbags or seat belts need to know the precise position of the occupants. The ToF camera provides such information through accurate 3D body models and real-time body tracking. So, even if the driver enjoys more "freedom" in L3 and L4 autonomous driving, their passive safety is still guaranteed.
The application of 3D sensing technology goes far beyond tracking motion, and it can also easily achieve passenger detection and classification. This provides accurate size and weight estimates, which can replace existing seat weight measurement systems. Similarly, ToF also supports a Child Presence Detection (CPD) system, which is gradually becoming a mandatory requirement.
ToF also offers new convenience applications. OMS's ToF sensor can track the position of the passenger's spine, providing options such as recommending a healthy seat position, activating massage seats, or providing fitness prompts, in order to take a break and relax the back during prolonged rides.
The ToF camera can also achieve gesture control, allowing passengers to interact with the large screen from different seat positions. At the same time, the vehicle can distinguish between driver and passenger inputs. However, its tracking is not limited to hand movements, but can also be extended to the entire body and movements. This makes intelligent and intuitive interior lighting possible, such as illuminating only the current area of interest. This demonstrates the enormous advantages of ToF technology. Although some applications can also be implemented through other (and possibly even cheaper) sensors, a single ToF sensor can provide multiple different functions simultaneously. As mentioned above, ToF can achieve new applications, optimize existing systems in the car, and even eliminate other sensors to reduce overall costs.
The first batch of vehicles using ToF sensors have been put on the road
BMW is a pioneer in using ToF sensors for gesture control; Mercedes is also utilizing this technology to provide convenience features. In China, many new electric vehicle startups are relying on ToF sensors for DMS and facial recognition. This technology has been applied in AITO models, and other major automakers are expected to launch more models using this technology next year. Therefore, this technology has penetrated into the interior of cars.
How 3D cameras prevent facial recognition fraud
Facial recognition algorithms are quite mature in this regard. Especially in smartphones, this technology is now widely used. But research (such as the German magazine Computer Bild Q1/20) shows that many smartphones are easily deceived: 20 out of 25 smartphones can be unlocked with a simple printed photo. Only smartphones equipped with 3D depth sensors can distinguish between real faces and photos. This makes ToF an ideal technology for adding safe facial authentication to car driver monitoring systems.
Infineon collaborated with Jungo Connectivity to showcase a driver monitoring system that combines powerful anti fraud facial authentication capabilities: the system is based on Jungo's artificial intelligence in car sensing solution CoDriver and Infineon's REAL3 ™ ToF image sensor.
ToF also brings new possibilities to the exterior of the vehicle. As mentioned earlier, facial authentication can be used to unlock vehicles. For this purpose, we installed the ToF module in the B-pillar.
Electric doors that can automatically open and close require sensors to ensure that there are no obstacles blocking the way. For this, ToF sensors can provide the required high-precision data. For automatic parking, the ToF sensor can provide accurate data on the position and size of objects such as curbs and pillars.
Of course, the ToF camera module is a complex system: it must be customized for each vehicle and application, and each module must be calibrated. The laser used must receive electrical and mechanical protection to ensure the eye safety of passengers at all times. Data processing cannot be underestimated. In automotive applications, the ToF module, like all other components, must meet specific standards. For example, intelligent airbag applications need to meet Functional safety requirements including ISO 26262.
The world's first ToF sensor certified by ISO 26262
Infineon is the first company to provide high-resolution automotive grade ToF image sensors that meet the functional safety requirements of ISO 26262 ASIL-B. ISO 26262 ASIL-B standard defines the latest technical level in the field of Functional safety. Any components in the vehicle that may affect passenger safety must comply with this standard.
Among devices that pass this standard, Infineon REAL3 ™ The IRS 2877A (S) ToF image sensor is one of them. It is based on the fifth generation pixel technology and has undergone practical testing on smartphones (IRS 2877C). With this new sensor, the lens size has been significantly reduced again, with a pixel circle of only 4mm (0.25 inches), making the lens as small as a smartphone camera. At the same time, it also has up to 640 × A system resolution of 480 pixels (VGA).
This is a highly integrated solution, with each pixel having backlight suppression function (so the image sensor is completely immune to sunlight interference), integrated safety circuits to protect the human eye, and optimized power supply design, which helps achieve efficient (size and cost) ToF camera module design. The entire device is packaged using a standard optical spherical pin grid array (BGA), suitable for standard welding processes.
summary
3D deep data makes everything unique: ToF solutions demonstrate that compliance and innovation are not contradictory. Only one sensor can meet the relevant regulatory standards of the European Union and the functional requirements of NCAP DMS (including head tracking, closed eye detection, and gaze area segmentation), while adding new functions and services through secure 3D facial authentication. This has opened up an era of seamless connectivity. In addition, the ToF based passenger monitoring system makes it possible to envision the driver becoming a passenger, for example, when the vehicle is autonomous, the driver can change their posture or even lie down.
Infineon has a strong ecosystem of partners, including module manufacturers, first tier customers, and application software experts. The company also has good ToF sensor performance in consumer, industrial, and IoT applications. Infineon, together with its development partner pmdtechnologies, provides a complete solution that includes complex depth algorithms, deep data artifact detection and correction, deep software support, and advanced, fast, and extensively validated calibration and testing settings.
Infineon is the first company to provide a high-resolution vehicle gauge level ToF image sensor that meets the requirements of ISO 26262 ASIL-B Functional safety.
Infineon's ToF image sensor is the world's first 3D ToF image sensor certified by the ISO 26262 ASIL-B standard, which can deal with current and future Functional safety applications.
Infineon is also a one-stop service provider in the field of in car sensors, providing other sensors such as millimeter wave radar and silicon microphones in addition to ToF sensors.