Danil Prokhorov (Ed.) ComputationalIntelligenceinAutomotiveApplications Studies in Computational Intelligence, Volume 132 Editor-in-chief Prof. Janusz Kacprzyk Systems Research Institute Polish Academy of Sciences ul. Newelska 6 01-447 Warsaw Poland E-mail: kacprzyk@ibspan.waw.pl Furthervolumesofthisseriescanbefoundonour homepage: springer.com Vol. 111. David Elmakias (Ed.) New Computational Methods in Power System Reliability, 2008 ISBN 978-3-540-77810-3 Vol. 112. Edgar N. Sanchez, Alma Y. Alan´ıs and Alexander G. Loukianov Discrete-Time High Order Neural Control: Trained with Kalman Filtering, 2008 ISBN 978-3-540-78288-9 Vol. 113. Gemma Bel-Enguix, M. Dolores Jimenez-Lopez and Carlos Mart´ın-Vide (Eds.) NewDevelopmentsinFormalLanguagesandApplications, 2008 ISBN 978-3-540-78290-2 Vol. 114. Christian Blum, Maria Jos´e Blesa Aguilera, Andrea Roli and Michael Sampels (Eds.) Hybrid Metaheuristics, 2008 ISBN 978-3-540-78294-0 Vol. 115. John Fulcher and Lakhmi C. Jain (Eds.) Computational Intelligence: A Compendium, 2008 ISBN 978-3-540-78292-6 Vol. 116. Ying Liu, Aixin Sun, Han Tong Loh, Wen Feng Lu and Ee-Peng Lim (Eds.) Ad va nces o f Computational Intelligence in Industrial Systems, 2008 ISBN 978-3-540-78296-4 Vol. 117. Da Ruan, Frank Hardeman and Klaas van der Meer (Eds.) IntelligentDecisionandPolicyMakingSupportSystems, 2008 ISBN 978-3-540-78306-0 Vol. 118. Tsau Young Lin, Ying Xie, Anita Wasilewska and Churn-Jung Liau (Eds.) Data Mining: Foundations and Practice, 2008 ISBN 978-3-540-78487-6 Vol. 119. Slawomir Wiak, Andrzej Krawczyk and Ivo Dolezel (Eds.) Intelligent Computer Techniques in Applied Electromagnetics, 2008 ISBN 978-3-540-78489-0 Vol. 120. George A. Tsihrintzis and Lakhmi C. Jain (Eds.) Multimedia Interactive Services in Intelligent Environments, 2008 ISBN 978-3-540-78491-3 Vol. 121. Nadia Nedjah, Leandro dos Santos Coelho and Luiza de Macedo Mourelle (Eds.) Quantum Inspired Intelligent Systems, 2008 ISBN 978-3-540-78531-6 Vol. 122. Tomasz G. Smolinski, Mariofanna G. Milanova and Aboul-Ella Hassanien (Eds.) Applications of Computational Intelligence in Biology, 2008 ISBN 978-3-540-78533-0 Vol. 123. Shuichi Iwata, Yukio Ohsawa, Shusaku Tsumoto, Ning Zhong, Yong Shi and Lorenzo Magnani (Eds.) Comm unications and Discov eries from Mul tidisciplinary Data, 2008 ISBN 978-3-540-78732-7 Vol. 124. Ricardo Zavala Yoe (Ed.) ModellingandControlofDynamicalSystems:Numerical ImplementationinaBehavioralFramework,2008 ISBN 978-3-540-78734-1 Vol. 125. Larry Bull, Bernad´o-Mansilla Ester and John Holmes (Eds.) Learning Classifier Systems in Data Mining, 2008 ISBN 978-3-540-78978-9 Vol. 126. Oleg Okun and Giorgio Valentini (Eds.) SupervisedandUnsupervisedEnsembleMethods and their Applications, 2008 ISBN 978-3-540-78980-2 Vol. 127. R´egie Gras, Einoshin Suzuki, Fabrice Guillet and Filippo Spagnolo (Eds.) Statistical Implicative Analysis, 2008 ISBN 978-3-540-78982-6 Vol. 128. Fatos Xhafa and Ajith Abraham (Eds.) Metaheuristics for Scheduling in Industrial and Ma nufacturing Applications, 2008 ISBN 978-3-540-78984-0 Vol. 129. Natalio Krasnogor, Giuseppe Nicosia, Mario Pavone and David Pelta (Eds.) Nature Inspired Cooperative Strategies for Optimization (NICSO 2007), 2008 ISBN 978-3-540-78986-4 Vol. 130. Richi Nayak, N. Ichalkaranje and Lakhmi C. Jain (Eds.) Evolution of Web in Artificial Intelligence E nvironments, 2008 ISBN 978-3-540-79139-3 Vol. 131. Roger Lee and Haeng-Kon Kim (Eds.) Compu ter and Information Science, 2008 ISBN 978-3-540-79186-7 Vol. 132. Danil Prokhorov (Ed.) Computational Intelligence in Automotive Applica tions, 2008 ISBN 978-3-540-79256-7 Danil Prokhorov (Ed.) Computa tional In telligence in Automotive Applications With 157 Figures and 48 Tables 123 Danil Prokhorov ToyotaTechnicalCenter-ADivision of Toyota Motor Engineering and Manufacturing (TEMA) Ann Arbor, MI 48105 USA dvprokhorov@gmail.com ISBN 978-3-540-79256-7 e-ISBN 978-3-540-79257-4 Studies in Computational Intelligence ISSN 1860-949X Library of Congress Control Number: 2008925554 c  2008 Springer-Verlag Berlin Heidelberg This work is subject to copyright. All rights are reserved, whether the whole orpart of the material is con- cerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publica- tionorpartsthereofispermittedonlyundertheprovisionsoftheGermanCopyrightLawofSeptember 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Cover design: Deblik, Berlin, Germany Printed on acid-free paper 987654321 springer.com Computational Intelligence in Automotive Applications Preface What is computational intelligence (CI)? Traditionally, CI is understood as a collection of methods from the fields of neural networks (NN), fuzzy logic and evolutionary computation. Various definitions and opinions exist, but what belongs to CI is still being debated; see, e.g., [1–3]. More recently there has been a proposal to define the CI not in terms of the tools but in terms of challenging problems to be solved [4]. With this edited volume I have made an attempt to give a representative sample of contemporary CI activities in automotive applications to illustrate the state of the art. While CI research and achievements in some specialized fields described (see, e.g., [5, 6]), this is the first volume of its kind dedicated to automotive technology. As if reflecting the general lack of consensus on what constitutes the field of CI, this volume illustrates automotive applications of not only neural and fuzzy computations 1 which are considered to be the “standard” CI topics, but also others, such as decision trees, graphical models, Support Vector Machines (SVM), multi-agent systems, etc. This book is neither an introductory text, nor a comprehensive overview of all CI research in this area. Hopefully, as a broad and representative sample of CI activities in automotive applications, it will be worth reading for both professionals and students. When the details appear insufficient, the reader is encouraged to consult other relevant sources provided by the chapter authors. The chapter “Learning-based driver workload estimation” discusses research on estimation of driver cognitive workload and proposes a new methodology to design driver workload estimation systems. The methodology is based on decision-tree learning. It derives optimized models to assess the time-varying work- load level from data which include not only measurements from various sensors but also subjective workload level ratings. The chapter “Visual monitoring of driver inattention” introduces a prototype computer vision system for real-time detection of driver fatigue. The system includes an image acquisition module with an infrared illuminator, pupil detection and tracking module, and algorithms for detecting appropriate visual behaviors and monitoring six parameters which may characterize the fatigue level of a driver. To increase effectiveness of monitoring, a fuzzy classifier is implemented to fuse all these parameters into a single gauge of driver inattentiveness. The system tested on real data from different drivers operates with high accuracy and robustly at night. The chapter “Understanding driving activity using ensemble methods” complements the chapter “Visual monitoring of driver inattention” by discussing whether driver inattention can be detected without eye and head tracking systems. Instead of limiting themselves to working with just a few signals from preselected sensors, the authors chose to operate on hundreds of signals reflecting the real-time environment both outside and inside the vehicle. The discovery of relationships in the data useful for driver activity classification, as 1 Another “standard” CI topic called evolutionary computation (EC) is not represented in this volume in the form of a separate chapter, although some EC elements are mentioned or referenced throughout the book. Relevant publications on EC for automotive applications are available (e.g., [7]), but unfortunately were not available as contributors of this volume. VIII Preface well as ranking signals in terms of their importance for classification, is entrusted to an approach called random forest, which turned out to be more effective than either hidden Markov models or SVM. The chapter “Computer vision and machine learning for enhancing pedestrian safety” overviews methods for pedestrian detection, which use information from on-board and infrastructure based-sensors. Many of the discussed methods are sufficiently generic to be useful for object detection, classification and motion prediction in general. The chapter “Application of graphical models in the automotive industry” describes briefly how graphical models, such as Bayesian and Markov networks, are used at Volkswagen and Daimler. Production planning at Volkswagen and demand prediction benefit significantly from the graphical model based system developed. Another data mining system is developed for Daimler to help assessing the quality of vehicles and identifying causes of troubles when the vehicles have already spent some time in service. It should be noted that other automotive companies are also pursuing data mining research (see, e.g., [8]). The chapter “Extraction of maximum support rules for the root cause analysis” discusses extraction of rules from manufacturing data for root cause analysis and process optimization. An alternative approach to traditional methods of root cause analysis is proposed. This new approach employs branch-and-bound princi- ples, and it associates process parameters with results of measurements, which is helpful in the identification of the main drivers for quality variations of an automotive manufacturing process. The chapter “Neural networks in automotive applications” provides an overview of neural network tech- nology, concentrating on three main roles of neural networks: models, virtual or soft sensors and controllers. Training of NN is also discussed, followed by a simple example illustrating the importance of recurrent NN. The chapter “On learning machines for engine control” deals with modeling for control of turbocharged spark ignition engines with variable camshaft timing. Two examples are considered: (1) estimation of the in-cylinder air mass in which open loop neural estimators are combined with a dynamic polytopic observer, and (2) modeling an in-cylinder residual gas fraction by a linear programming support vector regression method. The authors argue that models based on first principles (“white boxes”) and neural or other “black box” models must be combined and utilized in the “grey box” approach to obtain results which are not just superior to any alternatives but are also more acceptable to automotive engineers. The chapter “Recurrent neural networks for AFR estimation and control in spark ignition automotive engines” complements the chapter “On learning machines for engine control” by discussing specifics of the air-fuel ratio (AFR) control. Recurrent NN are trained off-line and employed as both the AFR virtual sensor and the inverse model controller. The authors also provide a comparison with a conventional control strategy on a real engine. The chapter “Intelligent vehicle power management: An overview” presents four case studies: a conven- tional vehicle power controller and three different approaches for a parallel HEV power controller. They include controllers based on dynamic programming and neural networks, and fuzzy logic controllers, one of which incorporates predictions of driving environments and driving patterns. The chapter “Integrated diagnostic process for automotive systems” provides an overview of model-based and data-driven diagnostic methods applicable to complex systems. Selected methods are applied to three automotive examples, one of them being a hardware-in-the-loop system, in which the methods are put to work together to solve diagnostic and prognostic problems. It should be noted that integration of different approaches is an important theme for automotive research spanning the entire product life cycle (see, e.g., [9]). The chapter “Automotive manufacturing: intelligent resistance welding” introduces a real-time control system for resistance spot welding. The control system is built on the basis of neural networks and fuzzy logic. It includes a learning vector quantization NN for assessing the quality of weld nuggets and a fuzzy logic process controller. Experimental results indicate substantial quality improvement over a conventional controller. The chapter “Intelligent control of mobility systems” (ICMS) overviews projects of the ICMS Program at the National Institute of Standards and Technology (NIST). The program provides architecture, interface and data standards, performance test methods and infrastructure technology available to the manufacturing industry and government agencies in developing and applying intelligent control technology to mobility systems. A common theme among these projects is autonomy and the four dimensional/real-time control Preface IX systems (4D/RCS) control architecture for intelligent systems proposed and developed in the NIST Intelligent Systems Division. Unlike the book’s index, each chapter has its own bibliography for the convenience of the reader, with little overlap among references of different chapters. This volume highlights important challenges facing CI in the automotive domain. Better vehicle diag- nostics/vehicle system safety, improved control of vehicular systems and manufacturing processes to save resources and minimize impact on the environment, better driver state monitoring, improved safety of pedes- trians, making vehicles more intelligent on the road – these are important directions where the CI technology can and should make the impact. All of these are consistent with the Toyota vision [10]: Toyota’s vision is to balance “Zeronize” and “Maximize”. “Zeronize” symbolizes the vision and philosophy of our persistent efforts in minimizing negative aspects vehicles have such as environmental impact, traffic congestion and traffic accidents, while “Maximize” symbolizes the vision and philosophy of our persistent efforts in maximizing the positive aspects vehicles have such as fun, delight, excitement and comfort, that people seek in automobiles. I am very thankful to all the contributors of this edited volume for their willingness to participate in this project, their patience and valuable time. I am also grateful to Prof. Janusz Kacprzyk, the Springer Series Editor, for his encouragement to organize and edit this volume, as well as Thomas Ditzinger, the Springer production editor for his support of this project. Ann Arbor-Canton, MI, USA, Danil V. Prokhorov January 2008 References 1. http://en.wikipedia.org/wiki/Computational intelligence. 2. J.C. Bezdek, “What is computational intelligence?” In Zurada, Marks and Robinson (Eds.), Computational Intelligence: Imitating Life, pp. 1–12, IEEE Press, New York, 1994. 3. R.J. Marks II, “Intelligence: Computational Versus Artificial,” IEEE Transactions on Neural Networks, 4(5), 737–739, September, 1993. 4. W. Duch, “What is computational intelligence and what could it become?” In W. Duch and J. Mandziuk (Eds.), Challenges for Computational Intelligence,Vol.63ofStudies in Computational Intelligence (J. Kacprzyk Series Editor), Springer, Berlin Heidelberg New York, 2007. The chapter is available on-line at http://cogprints.org/5358/. 5. Intelligent Control Systems Using Computational Intelligence Techniques (IEE Control Series). Edited by Antonio Ruano, IEE, 2005. 6. R. Begg, Daniel T.H. Lai, M. Palaniswami. Computational Intelligence in Biomedical Engineering. CRC Press, Taylor & Francis Books, Boca Raton, Florida, 2007. 7. Marco Laumanns and Nando Laumanns, “Evolutionary Multiobjective Design in Automotive Development,” Applied Intelligence, 23, 55–70, 2005. 8. T.A. Montgomery, “Text Mining on a Budget: Reduce, Reuse, Recycle,” Michigan Leadership Summit on Business Intelligence and Advanced Analytics, Troy, MI, March 8, 2007. Presentation is available on-line at http://www.cmurc.com/bi-PreviousEvents.htm. 9. P. Struss and C. Price, “Model-Based Systems in the Automotive Industry,” AI Magazine, Vol. 24, No. 4, pp. 17–34, AAAI, Menlo Park, Winter 2003. 10. Toyota ITS vision, http://www.toyota.co.jp/en/tech/its/vision/. Contents Learning-Based Driver Workload Estimation Yilu Zhang, Yuri Owechko, and Jing Zhang 1 1 Background 1 2 Existing PracticeandItsChallenges 3 3 The ProposedApproach:Learning-BasedDWE 4 3.1 Learning-BasedDWEDesignProcess 4 3.2 Benefitsof Learning-BasedDWE 5 4 ExperimentalData 6 5 ExperimentalProcess 8 6 ExperimentalResults 10 6.1 Driver-IndependentTraining 11 6.2 Driver-Dependent Training 13 6.3 Feature Combination 14 7 ConclusionsandFuture Work 15 References 16 Visual Monitoring of Driver Inattention Luis M. Bergasa, Jes´us Nuevo, Miguel A. Sotelo, Rafael Barea, and Elena Lopez 19 1 Introduction 19 2 PreviousWork 20 3 SystemArchitecture 21 3.1 ImageAcquisitionSystem 22 3.2 Pupil DetectionandTracking 24 3.3 VisualBehaviors 26 3.4 Driver Monitoring 28 4 ExperimentalResults 30 4.1 TestSequences 30 4.2 ParameterMeasurementforOne oftheTestSequences 30 4.3 ParameterPerformance 31 5 Discussion 33 6 ConclusionsandFuture Work 35 References 36 XII Contents Understanding Driving Activity Using Ensemble Methods Kari Torkkola, Mike Gardner, Chris Schreiner, Keshu Zhang, Bob Leivian, Harry Zhang, and John Summers 39 1 Introduction 39 2 Modeling NaturalisticDriving 40 3 DatabaseCreation 41 3.1 ExperimentDesign 41 3.2 Annotation oftheDatabase 42 4 Driving Data Classification 43 4.1 DecisionTrees 44 4.2 RandomForests 45 4.3 RandomForestsforDriving ManeuverDetection 46 5 SensorSelectionUsing RandomForests 47 5.1 SensorSelectionResults 48 5.2 SensorSelectionDiscussion 50 6 Driver Inattention Detection Through Intelligent Analysis of Readily Available Sensors . . . . . . . . . . 50 6.1 Driver Inattention 50 6.2 Inattention DataProcessing 53 7 Conclusion 56 References 57 Computer Vision and Machine Learning for Enhancing Pedestrian Safety Tarak Gandhi and Mohan Manubhai Trivedi 59 1 Introduction 59 2 FrameworkforPedestrianProtectionSystem 60 3 TechniquesinPedestrianDetection 61 3.1 Candidate Generation 61 3.2 Candidate Validation 66 4 Infrastructure BasedSystems 71 4.1 Background Subtraction and Shadow Suppression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 4.2 RobustMulti-Camera Detection and Tracking 72 4.3 Analysisof ObjectActions andInteractions 72 5 PedestrianPathPrediction 72 6 Conclusionand Future Directions 75 References 76 Application of Graphical Models in the Automotive Industry Matthias Steinbrecher, Frank R¨ugheimer, and Rudolf Kruse 79 1 Introduction 79 2 Graphical Models 80 2.1 BayesianNetworks 80 2.2 MarkovNetworks 80 3 ProductionPlanningatVolkswagenGroup 80 3.1 Data Description and Model Induction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 3.2 Operationson the Model 82 3.3 Application 83 4 VehicleDataMining atDaimler AG 83 4.1 Data Description and Model Induction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.2 ModelVisualization 84 4.3 Application 85 5 Conclusion 88 References 88 [...]... 3.2 An Intelligent Controller Built Using DP Optimization and Neural Networks 3.3 Intelligent Vehicle Power Management Incorporating Knowledge About Driving Situations 4 Intelligent Systems for Predicting Driving Patterns 4.1 Features Characterizing Driving Patterns 4.2 A Multi-Class Intelligent... terminology, we use “workload” interchangeably with “cognitive workload” in this chapter Y Zhang et al.: Learning-Based Driver Workload Estimation, Studies in Computational Intelligence (SCI) 132, 1–17 (2008) c Springer-Verlag Berlin Heidelberg 2008 www.springerlink.com 2 Y Zhang et al Driver Vehicle Environment Sensors for: gaze position, pupil diameter, vehicle speed, steering angle, lateral acceleration,... eyes-offroad time On the other hand, engaging in thinking (the so-called minds-off-road phenomenon) is difficult to detect Since the cognitive workload level is internal to the driver, it can only be inferred based on the information that is observable In this chapter, we report some of our research results on driver’s cognitive workload estimation.1 After the discussion of the existing practices, we propose a new... 213 213 215 216 Automotive Manufacturing: Intelligent Resistance Welding Mahmoud El-Banna, Dimitar Filev, and Ratna Babu Chinnam 1 Introduction 2 Resistance Spot Welding: Background 3 Online Nugget Quality Evaluation Using Linear Vector Quantization... analysis and modeling The manual DWE design process is illustrated in Fig 2 Research along this line has achieved encouraging success The well-known existing models include the steering entropy [8] and the SEEV model [9] However, there are yet difficulties in developing a robust cognitive workload estimator for practical applications, the reasons of which are discussed below First, the existing data analysis... avoid overloading the driver Residual capacity sensitive: The residual capacity of the driver refers to the amount of spare capacity of the driver while he/she is performing the primary task of maneuvering the vehicle, and, if applicable, engaging in secondary tasks such as drinking a cup of coffee or operating an IVIS If the residual capacity is high, the driver is typically doing well in vehicle control... continuous and residual-capacity sensitive The challenge, however, lies in reliably acquiring and interpreting the physiological data sets, in addition to user acceptance issues Among the four workload assessment methods, primary-task performance measures and physiological measures (obtained by non-intrusive sensors) fulfill the above-discussed DWE requirements and are generally appropriate for DWE applications. .. Signal preprocessing DWE Workload index Fig 1 The working process of a driver workload estimation system A practical DWE system fulfills the following three requirements in order to identify driver’s cognitive status while the driver is engaged in naturalistic driving practice • • • Continuously measurable: A DWE system has to be able to continuously measure workload while the driver is driving the vehicle... typically doing well in vehicle control and may be able to be engaging in even more secondary activities Residual capacity is the primary interest for DWE Highly non-intrusive: A DWE system should not interfere with the driver by any means Before the discussion of existing DWE methodologies in the next section, it is helpful to give a brief introduction of cognitive workload assessment methods There exist... required to perform the driving task If the driver and the experimenter establish clear mutual understanding of the rating scale, the subjective measure can be very reliable Examples of popular subjective workload rating index are NASA Task Load Index (TLX) [3] and Subjective Workload Assessment Technique (SWAT) [4] Physiological measures include brain activities such as event-related potential (ERP) and Electroencephalogram . Networks in Automotive Applications Danil Prokhorov 10 1 1 Models 10 1 2 VirtualSensors 10 3 3 Controllers 10 6 4 TrainingNN 11 1 5 RNN: AMotivatingExample 11 6 6 VerificationandValidation (V &V) 11 8 References. 11 8 References 11 9 On Learning Machines for Engine Control G´erard Bloch, Fabien Lauer, and Guillaume Colin 12 5 1 Introduction 12 5 1. 1 CommonFeaturesin EngineControl 12 5 1. 2 NeuralNetworksinEngineControl 12 6 1. 3. Danil Prokhorov (Ed.) ComputationalIntelligenceinAutomotiveApplications Studies in Computational Intelligence, Volume 13 2 Editor -in- chief Prof. Janusz Kacprzyk Systems Research Institute Polish