汽车英文文献

A high speed tri-vision system for automotive applicationsMarc Anthony Azzopardi & Ivan Grech & Jacques LeconteAbstractPurpose Cameras are excellent ways of non-invasively monitoring the interior and exterior of vehicles. In particular, high speed stereovision and multivision systems are important for transport applications such as driver eye tracking or collision avoidance. This paper addresses the synchronisation problem which arises when multivision camera systems are used to capture the high speed motion common in such applications. MethodsAn experimental, high-speed tri-vision camera system intended for real-time driver eye-blink and saccade measurement was designed, developed, implemented and tested using prototype, ultra-high dynamic range, automotive- grade image sensors specifically developed by E2V (formerly Atmel) Grenoble SA as part of the European FP6 project SENSATION (advanced sensor development for attention stress, vigilance and sleep/wakefulness monitoring). Results The developed system can sustain frame rates of 59.8 Hz at the full stereovision resolution of 1280 480 but this can reach 750 Hz when a 10 k pixel Region of Interest (ROI) is used, with a maximum global shutte speed of 1/48000 s and a shutter efficiency of 99.7%. The data can be reliably transmitted uncompressed over standard copper Camera-Link cables over 5 metres. The synchronisation error between the left and right stereo images is less than 100 ps and this has been verified both electrically and optically. Synchronisation is auto- matically established at boot-up and maintained during resolution changes. A third camera in the set can be configured independently. The dynamic range of the 10bit sensors exceeds 123 dB with a spectral sensitivity extending well into the infra-red range. Conclusion The system was subjected to a comprehensive testing protocol, which confirms that the salient require- ments for the driver monitoring application are adequately met and in some respects, exceeded. The synchronization technique presented may also benefit several other auto- motive stereovision applications including near and far- field obstacle detection and collision avoidance, road condition monitoring and others.KeywordsSynchronisation . High-speed automotive multivision . Active safety . Driver monitoring . Sensors1 IntroductionOver the coming years, one of the areas of greatest research and development potential will be that of automotive sensor systems and telematics 1, 2. In particular, there is a steeply growing interest in the utilisation of multiple cameras within vehicles to augment vehicle Human-Machine Interfacing (HMI) for safety, comfort and security.For external monitoring applications, cameras are emerging as viable alternatives to systems such Radio, Sound and Light/Laser Detection and Ranging (RADAR, SODAR, LADAR/LIDAR). The latter are typically rather costly and either have poor lateral resolution or require mechanical moving parts.For vehicle cabin applications, cameras outshine other techniques with their ability to collect large amounts of information in a highly unobtrusive way. Moreover, cameras can be used to satisfy several applications at once by re-processing the same vision data in multiple ways, thereby reducing the total number of sensors required to achieve equivalent functionality. However, automotive vision still faces several open challenges in terms of optoelectronic-performance, size, reliability, power con- sumption, sensitivity, multi-camera synchronisation, inter- facing and cost.In this paper, several of these problems are addressed. As an example, driver head localisation, point of gaze detection and eye blink rate measurement is considered for which the design of a dash-board-mountable automotive stereovision camera system is presented. This was developed as part of a large FP6 Integrated Project - SENSATION (Advanced Sensor Development for Attention, Stress, Vigilance and Sleep/Wakefulness Monitoring). The overarching goal of extendable to multivision systems 58.The camera system is built around a matched set of prototype, ultra-high dynamic range, automotive-grade, image sensors specifically developed and fabricated by E2V Grenoble SA for this application. The sensor which is a novelty in its own right, is the AT76C410ABA CMOS monochrome automotive image sensor. This sensor imple- ments a global shutter to allow distortion-free capture of fast motion. It also incorporates an on- chipMulti-ROI feature with up to eight Regions Of Interest (ROI) with pre- programming facility and allows fast switching from one image to another. In this way, several real-time parallel imaging processing tasks can be carried out with one sensor. Each ROI is independently programmableon-the-flywith respect to integration time, gain, sub-sampling/binning, position, width and height.A fairly comprehensive series of“bench tests”were conducted in order to test the validity of the new concepts and to initially verify the reliability of the system across various typical automotive operating conditions. Additional rigorous testing would of course be needed to guarantee a mean time before failure (MTBF) and to demonstrate the efficacy of the proposed design techniques over statistically significant production quantities.2 Application backgroundThe set of conceivable automotive camera applications is an ever-growing list with some market research reports claiming over 10 cameras will be required per vehicle 9. The incomplete list includes occupant detection, occupant classification, driver recognition, driver vigilance and drowsiness monitoring 10, road surface condition moni- toring, intersection assistance 11, lane-departure warning 12, blind spot warning, surround view, collision warning, mitigation or avoidance, headlamp control, accident record-ing, vehicle security, parking assistance, traffic sign detection 13, adaptive cruise control and night/synthetic vision (Fig. 1).2.1 Cost considerationsThe automotive sector is a very cost-sensitive one and the monetary cost per subsystem remains an outstanding issue which could very well be the biggest hurdle in the way of full deployment of automotive vision. The supply-chain industry has been actively addressing the cost dilemma by introducing Field Programmable Gate Array (FPGA) vision processing and by moving towards inexpensive image sensors based on Complementary Metal Oxide Semiconductor (CMOS) technology 14. Much has been borrowed from other very large embedded vision markets which are also highly cost-sensitive: These are mobile telephony and portable computing. However, automotive vision pushes the bar substantially higher in terms of performance requirements. The much wider dynamic range, higher speed, global shuttering, and excellent infra-red sensitivity are just a few of the characteristics that set most automotive vision applications apart. This added complex- ity increases cost. However, as the production volume picks up, unit cost is expected to drop quite dramatically by leveraging on the excellent economies of scale afforded by the CMOS manufacturing process.Some groups have been actively developing and pro- moting ways of reducing the number of cameras required per vehicle. Some of these methods try to combine disparate applications to re-use the same cameras. Other techniques (and products) have emerged that trade-off some accuracy and reliability to enable the use of monocular vision in scenarios which traditionally required two or more cameras 10, 15, 16. Distance estimation for 3D obstacle localisation is one such example. Such tactics will serve well to contain cost in the interim. However, it is expected that the cost of the imaging devices will eventually drop to a level where it will no longer be the determining factor in the overall cost of automotive vision systems. At this point, we argue that Fig. 1Some automotive vision applicationsreliability, performance and accuracy consid- erations will again reach the forefront.In this paper the cost issue is addressed, but in a different way. Rather than discarding stereo- and multi-vision altogether, a low-cost (but still high-performance) technique for synchronously combining multiple cameras is pre- sented. Cabling requirements are likewise shared, resulting in a reduction in the corresponding cost and cable harness weight savings.2.2 The role of high speed visionA number of automotive vision applications require high frame-rate video capture. External applications involving high relative motion such as traffic sign, oncoming traffic or obstacle detection are obvious candidates. The need for high speed vision is perhaps less obvious in the interior of a vehicle. However, some driver monitoring applications can get quite demanding in this respect. Eye-blink and saccade measurement, for instance, is one of the techniques that may be employed to measure a drivers state of vigilance and to detect the onset of sleep 10, 16. It so happens that these are also some of the fastest of all human motion and accurate rate of change measurements may require frame rates running up to several hundred hertz. Other applica- tions such as occupant detection and classification can be accommodated with much lower frame rates but then the same cameras may occasionally be required to capture high speed motion for visual-servoing such as when modulating airbag release or seatbelt tensioning during a crash situation.2.3 A continued case for stereovision/multivisionSeveral of the applications mentioned, stand to benefit from the use of stereovision or multivision sets of cameras operating in tandem. This may be necessary to extend the field of view or to increase diversity and ruggedness and also to allow accurate stereoscopic depth estimation 11. Then, of course, multivision is indeed one of the most effective ways of counteracting optical occlusions.Monocular methods have established a clear role (alongside stereoscopy) but they rely on assumptions that may not always be true or consistently valid. Assumptions such as uniform parallel road marking, continuity of road texture, and operational vehicle head or tail lights are somewhat utopian and real world variability serves to diminish reliability. Often, what is easily achievable with stereoscopy can prove to be substantially complex with monocular approaches 17. The converse may also be true, because stereovision depends on the ability to unambigu- ously find corresponding features in multiple views. Stereovision additionally brings a few challenges of its own, such as the need for a large baseline camera separation, sensitivity to relative camera positioning and sensitivity to inter-camera synchronisation.Not surprisingly, it has indeed been shown that better performance (than any single method) can be obtained by combining the strengths of both techniques 18, 19. As the cost issue fades away, monovision and multivision should therefore be viewed as complimentary rather than competing techniques. This is nothing but yet another example of how vision data can be processed and interpreted in multiple ways to improve reliability and obtain additional information.In this paper, the benefit of combining stereo and monocular methods is demonstrated at the hardware level. A tri-vision camera is presented that utilises a synchronized stereovision pair of cameras for 3D head localisation and orientation measurement. Using this information, a third monocular high-speed camera can then be accurately controlled to rapidly track both eyes of the driver using the multi-ROI feature. Such a system greatly economises on bandwidth by limiting the high speed capture to very small and specific regions of interest. This compares favourably to the alternative method of running a stereovision system at high frame rate and at full resolution.2.4 The importance for high synchronisationOne of the basic tenets of multivision systems is the accurate temporal correspondence between frames captured by the different cameras in the set. Even a slight frequency or phase difference between the image sampling processes of the cameras would lead to difficulties during transmis- sion and post processing. Proper operation usually rests on the ability to achieve synchronised, low latency video capture between cameras in the same multivision set. Moreover, this requirement extends to the video transport mechanism which must also ensure synchronous delivery to the central processing hubs. The need for synchronization depends on the speed of the motion to be captured rather than the actual frame rate employed, but in general, applications which require high speed vision will often also require high synchronisation.Interestingly, even preliminary road testing of automo- tive vision systems reveals another sticky problem camera vibration. This is a problem that has already been faced many years ago by the first optical systems to enter mainstream vehicle use 20The optical tracking mechanisms used in car-entertainment CDROM/DVD drives are severely affected by automotive vibration and fairly complex (and fairly expensive) schemes are required to mitigate these effects 21. The inevitable vibration essentially converts nearly all mobile application scenarios into high speed vision problems because even low amplitude camera motion translates into significant image motion. The problem gets worse as the subject distance and/or optical focal length increases.Mounting the cameras more rigidly helps by reducing the vibration amplitude, but it also automatically increases the vibration frequency which negates some of the gain. Active cancellation of vibration is no new topic 22; however, this usually comes at a disproportionate cost. Thus, while high frame rates may not be important in all situations, short aperture times and high synchronization remain critically important to circumvent the vibration problem.A small numerical example quickly puts the problem into perspective. Consider a forward looking camera for in- lane obstacle monitoring based on a inch, 1024512 image sensor array with an active area of 5.72.9 mm behind a 28 mm (focal length) lens. If such a system is subjected to a modest 10 mrad amplitude, sinusoidal, angular vibration at 100 Hz, simple geometric optics implies a peak pixel shift rate of around 32,000 pixels/sec.Thus, if the error in correspondence between left and right stereo frames is to be limited to a vertical shift comparable to one pixel, a stereovision system would require a frame synchronisation accuracy which is better than 30 microseconds. Then on the road, the levels of vibration can get significantly worse and this does not yet take into account the additional high speed motion that may be present in the field of view. In summary, synchronization is a problem that has been largely overlooked and will become more important as the industry and consumer performance expectations increase. In this paper, a synchronisation technique based on matched cameras sharing a single clock is presented. The system affords a very high degree of synchronisation in fact, much higher than is actually demanded by the driver monitoring application. Synchronisation difficulties arising during initialisation and camera mode changes are also addressed in this paper using a novel frozen-clock programming technique.2.5 High bandwidth interconnect and processingAutomotive vision faces another formidable challenge bandwidth. Having several cameras running at high frame rates and at high resolutions quickly pushes such applica- tions into the multi GBit/s domain. This poses new pressures on a sector that is still barely warming up to multi-MBit/s interface speeds. New automotive video interface standards will be required, and while it makes sense to base these on existing and proven interconnects, it may be argued that a completely new standard is needed to properly address the requirements of this peculiar market. The stage is set for a standards-war and in fact, one is currently brewing which should eventually see the evolu- tion of a veritable Automotive Video Bus. Such a bus faces a tall order which includes: low cable cost, low interface cost, low specific weight, multi-GBit/s sustained throughput, multiplex-ability, preservation of synchronisation, high integrity, excellent electromagnetic compatibility (EMC) characteristics, low latency, low jitter, and a minimum 5 m cable span without repeaters 23.There is of course a second repercussion of such high bandwidths. Impressive data rates necessitate equally impressive computational power in order to perform all the associated video processing in real-time. This is fairly problematic considering the limited capabilities of most automotive embedded processors, but this is changing with the entry of FPGAs into the automotive market 2325. Aside from offering substantial (and sufficient) in-line processing power, FPGAs also serve to reduce cost by combining most of the interface glue-logic into a single chip. Then, FPGAs have the added appeal of re- configurability which allows aftermarket updates through simple firmware changesthough this raises several security concerns 25.3 Video interfacesA survey of currently available interface standards reveals that none of the present offerings are ideally suited to faithfully transport high speed, high resolution, synchron- ised stereovideo over appreciable distances. The following is a comparative discussion of the merits and shortcomings of the various interfaces.3.1 Bandwidth considerationsThe Interface throughput is the major concern since high resolutions are desirable and the required frame rates can reach into the high hundreds per second. At a moderate 200 frames per second, a 10 bit per pixel, greyscale, 640480 2, stereovision system generates video at 1.229 GBit/s. Even 15367682 at 12 bit is not at all farfetched for certain applications and this touches 5.662 GBit/s which is impossible to accommodate on most current interfaces. Evidently, the interface is a bottleneck that needs to be addressed.For our driver monitoring application, 60 Hz is sufficient for accurate head localisation. However 200 Hz or more is desirable for fast eye-saccade and eye-blink capture. Running the entire system at 200 Hz at full resolution is therefore wasteful. By using a trinocular system, the frame rate of the stereovision pair can be set to 60 Hz, while a third monocular camera tracks the eyes alone at 200 Hz using a pair of 10,000 pixel ROIs. This way, assuming 10bit, the bandwidth requirements are reduced to a more manageable (369+40) MBit/s. The information collected using the stereovision system guides the ROI placement for the third camera.Hence, for this application, the strict requirement is for an interface that can sustain 409 MBit/s of throughput. However, in view of the possibility of other vision applications and future resolution improvements, the design should aim for an interface which should be able to handle a significantly higher bandwidth.3.2 Latency and jitter considerationsThroughput alone does not fully describe the problem. Low system latency is another aspect that cannot be neglected. Practically all of the automotive vision applications mentioned, depend on real-time low latency access to the processed output from the vision information. The driver vigilance application is no exception but other even more demanding applications come to mind. At 90 km/h a vehicle covers 25 m every second. A single second of lag in a high speed obstacle detection situation can make the difference between avoiding an accident and reacting too late. The problem with latency is that it all adds up. There is latency at the sensor, transmission latency, processing laten