PythonQuants conference - QuantUniversity presentation - Stress Testing in the Cloud
5 likes1,911 views
The document discusses the scaling of Python models for model verification and stress testing in the financial sector, emphasizing the importance of model risk management and regulatory compliance. It highlights the need for robust stress testing frameworks, challenges in deploying and scaling models, and the growing adoption of Python among financial institutions due to its quant-friendly packages. Additionally, it presents case studies on value-at-risk and conditional value-at-risk methodologies implemented in Python, MATLAB, and R.
![MODELVALIDATION
“Model risk is the potential for adverse consequences from
decisions based on incorrect or misused model outputs and
reports. “ [1]
“Model validation is the set of processes and activities
intended to verify that models are performing as expected, in
line with their design objectives and business uses. ” [1]
Ref:
[1] . Supervisory Letter SR 11-7 on guidance on Model Risk
Model Risk and Validation Defined](https://image.slidesharecdn.com/pythonquantsconference-quantuniversitypresentation-150502082506-conversion-gate02/85/PythonQuants-conference-QuantUniversity-presentation-Stress-Testing-in-the-Cloud-12-320.jpg)
PythonQuants conference - QuantUniversity presentation - Stress Testing in the Cloud
- 1. SCALING PYTHON MODELS FOR MODEL VERIFICATION AND STRESS TESTING ON THE CLOUD Information, data and drawings embodied in this presentation are strictly confidential and are supplied on the understanding that they will be held confidentially and not disclosed to third parties without the prior written consent of QuantUniversity LLC. Sri Krishnamurthy, CFA Founder and CEO QuantUniversity LLC. www.QuantUniversity.com FOR PYTHON QUANTS CONFERENCE, MAY 01 2015, NEW YORK
- 2. AGENDA About Model Risk Analytics Stress Testing and Model Risk: A Brief Introduction Why Python ? The Model Risk Analytics Solution 1 2 3 4 6 Agenda Case Study 5 Challenges in deploying and scaling models
- 3. MODEL RISK MANAGEMENT & STRESS TESTING FOR FINANCIAL INSTITUTIONS - ADVISORY SERVICES - PLATFORM TO STRESS TESTING AND MODEL RISK ASSESSMENT - TRAINING AND AUDITS
- 4. STRESS TESTING AND MODEL RISK– A BRIEF INTRODUCTION
- 6. STRESSTESTINGANDMODELRISKMANAGEMENT Regulatory efforts The Comprehensive Capital Analysis and Review (CCAR) is an annual exercise by the Federal Reserve to assess whether the largest bank holding companies operating in the United States have sufficient capital to continue operations throughout times of economic and financial stress and that they have robust, forward-looking capital- planning processes that account for their unique risks. Dodd-Frank Act stress testing (DFAST)--a complementary exercise to CCAR--is a forward-looking component conducted by the Federal Reserve and financial companies supervised by the Federal Reserve to help assess whether institutions have sufficient capital to absorb losses and support operations during adverse economic conditions. http://www.federalreserve.gov/bankinforeg/stress-tests-capital- planning.htm What’s driving Stress Testing and Model Risk Management efforts?
- 7. STRESSTESTINGANDMODELRISKMANAGEMENT Regulatory efforts SR 11-7 says “Banks benefit from conducting model stress testing to check performance over a wide range of inputs and parameter values, including extreme values, to verify that the model is robust” In fact, SR14-03 explicitly calls for all models used for Dodd- Frank Act Company-Run Stress Tests must fall under the purview of Model Risk Management. In addition SR12-07 calls for incorporating validation or other type of independent review of the stress testing framework to ensure the integrity of stress testing processes and results. What’s driving Stress Testing and Model Risk Management efforts?
- 8. STRESSTESTINGANDMODELRISKMANAGEMENT Regulatory efforts EIOPA: Europe-wide stress test for the insurance sector What’s driving Stress Testing and Model Risk Management efforts? https://eiopa.europa.eu/Publications/Surveys/eiopa-14- 215_stress_test_2014_specifications.pdf
- 9. STRESSTESTINGANDMODELRISKMANAGEMENT Model verification: • Migration of code from an older code-base/ from another language • Regression Testing Model Deployment: • Most models are stress tested prior to deployment Model Replication: • Models that replicate other models or functionality/strategy Model Back testing: • Test how a model performs under different conditions with historical data What’s driving Stress Testing and Model Risk Management efforts?
- 10. STRESSTEST&SCENARIOTEST Stress Tests and Scenario Tests Figure courtesy: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf
- 11. DEFINITIONS Definitions 1. Scenarios : “A scenario is a possible future environment, either at a point in time or over a period of time.” “Considers the impact of a combination of events“ 2. Sensitivity Analysis: “A sensitivity is the effect of a set of alternative assumptions regarding a future environment. “ 3. Stress Testing: • Analysis of the impact of single extreme events (or risk factors) • Extreme but plausible Ref: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf
- 12. MODELVALIDATION “Model risk is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports. “ [1] “Model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses. ” [1] Ref: [1] . Supervisory Letter SR 11-7 on guidance on Model Risk Model Risk and Validation Defined
- 13. WHY PYTHON?
- 14. PYTHONADOPTIONISGROWING Python gaining popularity and adoption Image courtesy: http://venturebeat.com/2014/02/03/the-most-popular-coding-languages-for-2014-are/ Python enjoyed 21 percent growth year-over-year based on mentions in Dice Job postings (http://news.dice.com/2014/09/18/python- snakes-up-list-tech-skills) SEC even proposed filing a Python program for ABS in 2010 https://www.sec.gov/rules/proposed/2010/33-9117.pdf
- 15. INSTITUTIONALADOPTIONOFPYTHONGROWING Large Institutions adopting Python Institutions like Deutsche Börse, JP Morgan and Bank of America adopting Python (http://www.slideshare.net/PyData/python-in- the-financial-industry-felix-fernandez )
- 16. PYTHONBECOMINGQUANTFRIENDLY Quant Focused Packages growing Packages Quants Love • NumPy • SciPy • Matplotlib • Pyomo • Pandas • scikit-learn • And more…. Distributions: • Anaconda • WinPython • PyCharm Ipython Notebook makes development a pleasure! Packages to support testing • Unittest • Pytest • Nose • doctest
- 17. CHALLENGES IN STRESS TESTING AND MODEL VERIFICATION
- 18. CHARACTERISTICSOFSTRESSANDSCENARIOTESTING Characteristics of Stress and Scenario testing 1. Difficult to build parametric models – Simulation driven approach necessary 2. Parameter space can explode easily 3. Tests independent of each other (Embarrassingly parallel) 4. Complete test-coverage – Useless 5. Human intervention required 6. Tests to be designed and customized for the companies needs considering portfolios, organization structure and regulatory obligations Ref: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf
- 19. MODELRISKMITIGATIONTHROUGHSYSTEMATICSTRESSTESTING Model Risk Mitigation through systematic stress testing 1. Model Implementation • Does it actually work for all intended use cases? 2. Model parameter testing • Number of parameters • How many Scenarios 3. Model Applicability 4. Model Benchmarking against Reference Implementation • Python vs MATLAB 5. Model Migration (version) • Regression Testing v1.0 to v2.0 6. Model Use case validation • Can we use the results to make decisions?
- 20. CHALLENGESWHENSCALINGANDDEPLOYINGMODELS Deploying and Scaling Python models still challenging 1. Package management • How to ensure packages and versions used to develop models are standardized? • Virtualenv? Docker? 2. How do you replicate and benchmark results? 3. Design choices for scaling: • Should you use concurrency ? • Should you parallelize code? • Do you leverage a GPU ? (https://wiki.python.org/moin/ParallelProcessing ) 4. What SLA can you support?
- 22. LEVERAGETECHNOLOGYTOSCALEANALYTICS Leverage technology to scale analytics 1.64 bit systems : Addressable space ~ 8TB 2.Multi-core processors : Explicit and Implicit Multi-threading 3.Parallel and Distributed Computing : Leverage commodity/Specialized hardware to scale problems 4.General-purpose computing on graphics processing units : Use graphics cards to scale your algorithms 5.Cloud Computing Ref: Gaining the Technology Edge: http://www.quantuniversity.com/w5.html
- 23. ADDRESSINGSTRESSTESTINGCHALLENGES How can we address challenges in Stress testing? 1. Leverage hardware for scaling 1. Enable developers to plan for stress tests 2. Enable developers to scale without having to rewrite code 2. Simplify deployment Enable developers to run their code 1. Locally 2. On Enterprise Servers 3. Cloud
- 24. ADDRESSINGSTRESSTESTINGCHALLENGES How can we address challenges in Stress testing? 3. Facilitate exhaustive and robust testing without having to get lost in details 1. Enable testers to specify their tests and configurations 2. Enable benchmarking and comparisons with other models 4. Facilitate deployment of models built using different frameworks including legacy code 1. Provide a choice of models (which could be built in different languages) 2. Enable benchmarking and comparisons with models built using other languages
- 26. MODELRISKANALYTICSSOLUTION Model Risk Analytics Solution • Distributed System to support Model Verification, Validation, Stress and Back Testing • Supports Python, MATLAB, R • Architecture leverages Tornado, MongoDB, Pyomo • Package management • Using Custom Client • Using Docker (Different container for different apps) • Both Synchronous and Asynchronous modes • Built on the Rackspace cloud but can work on any cloud/internal infrastructure (private or public)
- 27. CASE STUDY
- 28. Value-at-Risk & Conditional Value-at-Risk • VaR : The predicted maximum loss of a portfolio with a specified probability level (e,g., 95%) over a certain period of time (e.g. one day) • CVaR (Expected Shortfall) : The expected value of the loss given that the loss exceeds VaR Ref: Optimization Methods in Finance by Gerard Cornuejols, Reha Tutuncu, Cambridge University Press Image courtesy: http://www.imes.boj.or.jp/english/publication/mes/2002/me20-1-3.pdf )
- 29. How to Implement VaR and CVaR? Methodology: • Historical • Variance-Covariance method • Monte-Carlo simulations All 3 models are implemented in: • Python • MATLAB • R
- 30. Methods to compute VaR and CVaR • Historical method Image Courtesy: http://www.investopedia.com/articles/04/092904.asp 1. Compute Daily Returns and sort them in ascending order 2. For a given confidence level (α, e.g. 95%) , find VaRα (X) such that: P(X<= VaR α(X)) = α 3. Compute CVaR by taking the average loss of the tail
- 31. Methods to compute VaR and CVaR • Variance-Covariance Method Image Courtesy: http://www.investopedia.com/articles/04/092904.asp 1. Compute Daily Returns and fit a Normal distribution to obtain mean and Standard Deviation (µ & σ) 2. For a given confidence level (α, e.g. 95%) , find VaRα (X) such that: P(X<= VaR α(X)) = α Example 95% => -1.65* σ 3. Compute CVaR by taking the average loss of the tail (See Yamai and Yoshiba http://www.imes.boj.or.jp/english/public ation/mes/2002/me20-1-3.pdf )
- 32. Methods to compute VaR and CVaR • Monte-Carlo Simulations Image Courtesy: http://www.investopedia.com/articles/04/092904.asp 1. Compute Daily Returns and fit a Normal distribution to obtain mean and Standard Deviation (µ & σ) 2. Run n Monte-Carlo simulations with random numbers drawn from a normal distribution described by (µ & σ) 3. For a given confidence level (α, e.g. 95%) , find VaRα (X) such that: P(X<= VaR α(X)) = α 4. Compute CVaR by taking the average loss of the tail
- 33. Model Summary VaR Model Historical Method Python R MATLAB Variance- Covariance Method Python R MATLAB Monte-Carlo Simulations Python R MATLAB Given: 1. Historical Daily price time series for a specified time period 2. Constituents of a 3-asset long-only portfolio Compute: VaR and CVaR
- 34. Model Verification criteria 1. Model Benchmarking • Python vs R vs MATLAB 2. Parameter sweeps • Different Confidence Intervals (90, 95, 99) 3. Model Convergence • How many simulations needed ? (100, 500, 1000) 4. How do different methods compare? • Historical vs Variance-Covariance vs Monte-Carlo methods
- 37. WHEREHAVEWEUSEDTHISAPPROACH? Example 1: External Model Verification • Large bank, no formal model validation team • Model to estimate future cash flows factoring defaults and many other parameters • 20+ parameters, 100+ assets • Impossible to manually stress test the model thoroughly Solution : • Identified important parameters and ran more than 10000 tests automatically • Identified multiple issues where model failed especially when handling edge cases
- 38. WHEREHAVEWEUSEDTHISAPPROACH? Example 2: Energy Forecasting and Risk Management • Energy company with more than 100,000 customers • Model to estimate future energy usage based on historical usage, temperature forecasts etc. • 5+ parameters, 100K+ assets • Data-Driven model to help source wholesale energy and to hedge exposures Solution : • Working on building 100K+ models that will be clustered later • Need to run the model monthly factoring changing portfolio characteristics and new market information
- 39. FEATURES Coming soon! ModelRisk Model Definition Language • To specify model parameters ModelRisk Engine Optimization to leverage infrastructure in the cloud • Resource constraints: • Budget & Time constraints • Priority queues and jobs • Dynamic scaling and load balancing ModelRisk BackTest to enable back testing of your strategies on the cloud
- 40. MODELRISKANALYTICSBETAPROGRAM If you are interested in being a part of the Beta program: If you are interested in knowing more about the Model Risk Analytics system Or Want to be part of the Beta program, email: [email protected] to be added to the interest list Model Risk Analytics Beta program
- 41. MODELRISKMANAGEMENTCOURSE Coming soon: A 2-day course in Model Risk Management with hands-on case studies Email: [email protected] to be added to the interest list Stress Testig and Model Risk Management for Quants
- 42. REFERENCES Quantifying Model Risk Wilmott Magazine January 2014 The Decalogue Wilmott Magazine July 2014 To get copies of these articles, go to: http://www.quantuniversity.com/fitchSeminar.html OR Email: [email protected] The Role of Model Verification in Model Risk Management Oct 2014
- 43. Thank you! Sri Krishnamurthy, CFA, CAP Founder and CEO QuantUniversity LLC. srikrishnamurthy www.QuantUniversity.com Contact Information, data and drawings embodied in this presentation are strictly a property of QuantUniversity LLC. and shall not be distributed or used in any other publication without the prior written consent of QuantUniversity LLC.
- 44. • Founder of QuantUniversity LLC. • Advisory and Consultancy for Model Risk • Prior Experience at MathWorks, Citigroup and Endeca and 25+ financial services customers • Regular Columnist for the Wilmott Magazine • Author of forthcoming book “Financial Modeling: A case study approach” published by Wiley • Charted Financial Analyst and Certified Analytics Professional • Teaches Analytics in the Babson College MBA program and at Northeastern University Sri Krishnamurthy Founder and CEO SPEAKERBIO