forSatelliteNavigationLessonsLearned(ThusFar)fromt

Click to edit Master title style,Click to edit Master text styles,Second level,Third level,Fourth level,Fifth level,24 September 2009,*,Providing Integrity for Satellite Navigation: Lessons Learned (Thus Far) from the Financial Collapse of 2008 2009,ION GNSS 2009Savannah, GA.,24 September 2009,Sam Pullen,Stanford University,Overview and Motivation,My interest in this subject comes from my background in Probabilistic Risk Assessment (PRA), which formed the basis for my Ph.D. dissertation (Stanford, 1996).,Optimal satellite design,Optimal design of GPS integrity augmentations,Studying and understanding failures of the past are the key to improving risk assessment.,Motivation for Hurricane Katrina presentation from ION GNSS 2008,Since SatNav augmentations have been demonstrated to be safe (with substantial margin), the benefit of this work for SatNav is improving risk-assessment for future systems and upgrades.,Find means to reduce margin against “unknown unknowns”,Origins: The Changing Debt Market,Yield (%) on 10-Year Treasury Bonds: 1964 - 2009,Year,Yield (%),Low Treasury yields created demand for higher-yielding investments.,Source,:,A Simplified Picture of the Financial Market Collapse,“Wine Glass Pyramid” Overview of Collateralized Debt Market,Source,: Paddy Hirsch, “Crisis Explainer: Uncorking CDOs,”,. publicradio.org/subjects/economics_finance/financial_crisis/uncorking_cdos.shtml,Also see,: Jonathan Jarvis, “The Crisis of Credit Visualized,”,mortgage payments,overflows when full,1,st,Tranche (AAA, 3%),2,nd,Tranche (AA, 5%),3,rd,Tranche (BBB, 7%),4,th,Tranche (Unrated, 10%),can further insure via credit default swaps,The Efficient Market Hypothesis (EMH),Origins (“random walk”) go back to 1900, but codified and popularized by the Eugene Fama (“Chicago school” of economists, 1960s).,Expresses the concept that todays prices reflect all available information, properly (“rationally”) judged.,“The (market) price is right”, the foundation of quantitative economics,Traditional linear analysis with Markov state transitions,Gaussian (or log-Normal) market-state transition probabilities are assumed (definition of random walk).,Despite limited supporting evidence, EMH became widely accepted (and exploited),because of its academic and mathematical elegance/convenience,.,Non-Gaussian Stock Market Behavior,P DeGrauwe,et al, “How Abnormal was the Stock Market in October 2008?”,Euro Intelligence, 11 Nov. 2008.,s,chg,= 1.032%,An Honest Explanation from a Leading Master of Quantitative Finance,In finance we often assume that equity returns are normally distributed., We find ourselves using the normal distribution quite naturally for many financial processes.,As often with mathematical laws there is the legal small print, in this case the conditions under which the Central Limit Theorem applies. Of course, financial data may not satisfy all of these, or indeed, any.,In particular, it turns out that if you try to fit equity returns data with non-normal distributions you often find that the best distribution is one that has infinite variance. Not only does it complicate the nice mathematics of normal distributions and the Central Limit Theorem, it also results in infinite volatility.,This is appealing to those who want to produce the best models of financial reality but does rather spoil many decades of financial theory and practice based on volatility as a measure of risk for example.,Paul Wilmott,Frequently Asked Questions in Quantitative Finance,(Wiley, 2007), pp. 33 35:,One Disastrous Outcome:,The Copula Model for Credit Risk Correlation,Before 2000, debt markets were much more conservative due to the complexity of modeling default risk and the small data base of major loan defaults.,David Li of JP Morgan RiskMetrics group “removed” this difficulty by assuming a Gaussian copula formulation with a single correlation parameter,g,derived from comparative market prices.,Justified by EMH since market prices confer “best” knowledge,This approach led to dramatic growth in the credit-derivatives market until its fatal flaws were revealed by housing market crash of 2007-08.,Nationwide (correlated) loan defaults were not captured by model.,See Felix Salmon in,Wired,Magazine (March 2009),Wilmott Actually Predicted this Disaster Years Beforehand,Abstract (conclusion),Unfortunately, as the mathematics of finance reaches higher levels so the level of common sense seems to drop.,There have been some well-publicized cases of large losses sustained by companies because of their lack of understanding of financial instruments. In this article we look at the history of financial modelling, the current state of the subject and possible future directions.,It is clear that a major rethink is desperately required if the world is to avoid a mathematician-led market meltdown.,Paul Wilmott, “The Use, Misuse, and Abuse of Mathematics in Finance,”,Philosph Trans: Math, Phys and Eng Sci, Vol. 358, No. 1765 (Jan. 2000), pp. 63-73.,The Financial Modelers Manifesto,(in response to the crisis),Our experience in the financial arena has taught us to be very humble in applying mathematics to markets, and to be extremely wary of ambitious theories, which are in the end trying to model human behavior.,We like simplicity, but we like to remember that it is our models that are simple, not the world,The greatest danger is the age-old sin of idolatry. Financial markets are alive but a model, however beautiful, is an artifice. No matter how hard you try, you will not be able to breathe life into it. To confuse the model with the world is to embrace a future disaster driven by the belief that humans obey mathematical rules.,MODELERS OF ALL MARKETS, UNITE! You have nothing to lose but your illusions.,The Modelers Hippocratic Oath, I will remember that I didnt make the world, and it doesnt satisfy my equations., Though I will use models boldly to estimate value, I will not be overly impressed by mathematics., I will never sacrifice reality for elegance without explaining why I have done so., Nor will I give the people who use my model false comfort about its accuracy. Instead, I will make explicit its assumptions and oversights., I understand that my work may have enormous effects on society and the economy, many of them beyond my comprehension.,Excerpted from Derman and Wilmott,The Financial Modelers Manifesto,(Jan. 2009):,Key Lessons Applicable to Engineering Risk Modeling,Precise modeling of the unknown is not possible.,Therefore, probabilistic models built upon uncertainty are preferred to deterministic ones.,Simplified risk models may be justified for specific threats, but model limitations must be given as much weight as the results.,Study assumptions carefully and “carry them forward” with the results so that they are not “lost in time”.,Avoid “falling in love” with models keep a critical mind.,In particular, avoid extrapolating from a flawed model into the realm of absurdity.,Initial errors may be tolerable until exploited too aggressively by a follow-up model.,As with other engineering risk analyses, SatNav integrity shares features with financial models:,Deterministic equations,: cause-and-effect behavior is assumed known; uncertainty introduced by random variables with known distributions.,The,SatNav,solution worst-case modeling,: where significant uncertainty cannot be removed, “worst-case” simplifications are derived to bound unknown reality.,Gaussian distribution,: most random perturbations are modeled as Gaussian (or Gaussian variations),The,SatNav,solution,: theory and data are combined to determine “inflation” factors such that the unknown “true” distribution is bounded at sufficiently low probabilities.,Specific Lessons for SatNav Integrity,Summary,The financial crisis illustrates the perils of risk assess-ment based upon hubris and over-simplified models.,This experience provides many useful lessons:,Probabilistic models are better when uncertainty is large.,When using deterministic models:,Emphasize assumptions when presenting results.,Avoid over-extrapolating from results.,Remain open to new threats and threat model changes.,SatNav integrity models are deterministic but apply multiple levels of caution against uncertainty.,Care applied to insure proper use of Gaussian distribution,“Worst-case” error mitigation theoretically bounds all remaining uncertainty but often impacts user performance.,Backup Slides follow,My Background in Risk Assessment,Research on Probabilistic Risk Analysis (PRA) formed the core of my Ph.D. dissertation (1996).,PRA in my thesis was applied to optimal design of satellites and GPS integrity augmentations (RAIM, WAAS).,Since then, my work has focused on optimal design and verification of GPS augmentation systems,with pre-designed (and highly-constrained) architectures,.,Focus on GBAS (LAAS) ionosphere and ephemeris threats,Focus on optimal diagnosis and isolation of detected faults,Risk analysis failures outside of GNSS reinforce basic PRA principles and provide important lessons:,Hurricane Katrina (ION GNSS 2008),Recent financial crisis (this paper ),Opening Thoughts,The experience of being proved disastrously wrong is salutary.,No economist should be denied it, and none are.,John Kenneth Galbraith (,early 1980s,),Also by Galbraith:,The only function of economic forecasting is to make astrology look respectable.,Sources,: J. A. Smith,The Idea Brokers: Think Tanks and the Rise of the New Policy Elite,(1993).,Overview,Serious flaws in mathematical modeling directly contributed to the financial collapse of 2008,-,2009.,The caution and consideration applied to integrity assurance for satellite navigation stands in sharp contrast to the hubris of the financial community.,However, elements of the faulty financial models cited above exist in most traditional forms of risk analysis.,This briefing examines what lessons, if any, can be learned that are relevant to risk assessment in general and SatNav integrity analysis in particular.,A Serious Outcome:,Value-at-Risk (VaR) Modeling,Devised by JP Morgan in the 1990s to provide standardized trade and company-wide risk modeling,uses traditional linear-Gaussian statistics,Key selling point,: a single “Value at Risk” output,Represents a lower confidence bound, for a given percentile (95,th,or 99,th,) and duration, on the amount that could be lost.,VaR results available to managers in near-real time.,Ease of use and simplicity of results led to massive over-dependence and abuse.,Uncertainty parameters set using limited historical data.,Used as basis for capital requirements led to insufficient reserves when financial crisis hit.,See Joe Nocera in,NYT,Magazine,(4 January 2009),“Misleading Information” Error Criterion,Normalized Vertical Position Error (no. of sigmas),10,10,10,10,10,10,-10,-8,-6,-4,-2,0,Probability Density,VPL,(for this fault state),K,FFMD,K,fault,P,FFMD,VAL,(,for this flight operation),Nominal Error plus 4,s,Bias Fault,-6,-4,-2,0,2,4,6,8,10,12,14,16,Gaussian dist. model,Vertical Alert Limit,Vertical Protection Level,Model Definition vs. Reality,Level of Hazard,Error Size (meters),VAL,Typical Model,“Real-World” Model,In practice, maximum error bounds (VPL, VAL) are very conservative, actual “hazard level” at or just above VAL is low (if not zero).,V,AL +,d,is not materially more hazardous than VAL ,d,.,Resulting Region of “Most Threatening” Error (Snapshot View),Source: T. Zaugg,Proc. ION 58,th,Annual Meeting,June 2002.,= E,r,/ MERR,(normalized range error),Range Error = MERR (range-domain bound),Error giving max. hazard probability is much lower than MERR!,Dealing with the Unknown in GBAS:,Ionosphere Anomaly Threat Modeling,Boundaries of resulting “threat model” for LAAS in CONUS,Based upon most severe anomalies observed in CONUS since 1999,Source: S. Datta-Barua, J. Lee,et al, “Ionospheric Threat Parameterization for Local Area GPS-Based Aircraft Landing Systems,” Submitted to,AIAA J. of Aircraft,(August 2009, under review).,0,10,20,30,40,50,60,70,80,90,0,50,100,150,200,250,300,350,400,450,Elevation deg,Slope mm/km,L1-L2 and L1 CMC,L1 CMC,L1 CMC (low-elev),Worst-case gradients along this upper bound.,Dealing with the Unknown in GBAS:,Ionosphere Anomaly Threat Modeling,Boundaries of resulting “threat model” for LAAS in CONUS,Based upon most severe anomalies observed in CONUS since 1999,Source: S. Datta-Barua, J. Lee,et al, “Ionospheric Threat Parameterization for Local Area GPS-Based Aircraft Landing Systems,” Submitted to,AIAA J. of Aircraft,(August 2009, under review).,L1-L2 and L1 CMC,L1 CMC,L1 CMC (low-elev),Worst-case gradients along this upper bound.,0,100,200,300,400,500,600,700,800,0,50,100,150,200,250,300,350,400,450,Ground Speed m/s,Slope mm/km,high,elevation,low elevation,Iono. Anomaly “Wedge” Model Geometry,Simplified Ionosphere Front Model:,a,linear ramp,defined by constant,speed,slope,and,width,Front Speed,200 m/s,Airplane Speed, 70 m/s,(synthetic baseline due to smoothing 14 km,),Front Width,25,km,GBAS Ground Station,Front,S,lope,425 mm/km,LGF IPP Speed 200 m/s,Max. 6 km at DH,Impact of Ionosphere Anomaly Model and Worst-Case Error,0,5,10,15,20,25,30,35,40,45,0,0.02,0.04,0.06,0.08,0.1,0.12,0.14,User Vertical Position Error (meters),PDF,Worst-case error, or “MIEV”, is,41 m,RTCA-24 Constellation; All-in-view, all 1-SV-out, and all 2-SV-out subsets included; 2 satellites impacted simultaneously by ionosphere anomaly,Most errors ( 75%) are exactly zero due to detection/exclusion, but all zero errors have been removed from the histogram.,28.8-meter tolerable limit (CAT I PA),Vast majority of non-zero errors are well below tolerable limit.,Parameter Inflation (“geometry screening”) added to remove geometries with unsafe errors, but many good geometries are removed as well, significant availability loss.,Features of Worst-Case Mitigation,Theoretically, mitigating worst-case errors covers all threat scenarios,and removes unquantifiable risk due to “unknown unknowns.”,Worst-case mitigation almost always protects integrity with substantial margin (as in iono. anomaly example).,Key limitation,: “worst case” model remains dependent on un-provable assumptions.,Ongoing vigilance needed to monitor validity of key assumptions.,On the “cost” side, the difficulty of mitigating worst-case scenarios stresses the resulting system:,User benefits may be significantly degraded.,Loss of availability may have unforeseen safety implications.,How Much do Modeling Issues Matter?,Unlike absurdities in financial modeling, assumptions made in GNSS risk modeling are “hubris-free” and are almost always conservative.,Furthermore, in theory, focus on and mitigation of worst-case anomalies also covers all other threats.,Unfortunately, “worst-case” anomalies are, by definition, difficult to counter and require extensive hardware/software/personnel resources.,As a result, risk mitigation may become mis-focused, and sources of risk that do not easily fit the above models may get “assumed away”.,Similarities to Hurricane Katrina Lessons,A key lesson from Katrina is the importance of maintaining,flexibility,and,adaptability,in risk-mitigation systems.,New Orleans hurricane “threat model” did not change as better information became available.,Lengthy political battles prevented improvements to hurricane defense system to address worsening threat understanding and flaws in levees discovered over time.,The most obvious similarity here is the consistent refusal of mainstream financial economics to consider the obvious violations of EMH and their implications.,Even today, anecdotal evidence suggests that mainstream financial economists are mostly “sticking with their story ”,