Machine Learning for National Economic Accounts Jeff Chen

  • Slides: 31
Download presentation
Machine Learning for National Economic Accounts Jeff Chen, Abe Dunn, Kyle Hood, Alex Driessen

Machine Learning for National Economic Accounts Jeff Chen, Abe Dunn, Kyle Hood, Alex Driessen and Andrea Batch

Motivation End of Quarter Advance Estimate When we’d like it to be available Second

Motivation End of Quarter Advance Estimate When we’d like it to be available Second Estimate When source data are available 2

Motivation End of Quarter Advance Estimate Second Estimate Short-term prediction using machine learning (for

Motivation End of Quarter Advance Estimate Second Estimate Short-term prediction using machine learning (for services sector estimates) Traditional Data Alternative Data 3

Possibilities: ML for National Economic Accounts 2 1 3 § Identify which modeling considerations

Possibilities: ML for National Economic Accounts 2 1 3 § Identify which modeling considerations (e. g. algorithm, data, feature selection) are associated with accuracy gains by PCE services component. § Construct ‘hurricane tracks’ for projected quarterly economic growth to help build consensus that a predicted growth is likely. M 1 vs. M 2 § Develop a simple framework for evaluating tradeoffs in terms of revision reductions relative to current methods. 4

Hurdles: There are more variables than records. Issue Solution Traditional statistical methods have trouble

Hurdles: There are more variables than records. Issue Solution Traditional statistical methods have trouble with k > n Id Y X 1 X 2 X 3 X 4 X 5 . . . x 999 Many ML methods can efficiently sift through inputs that maximize predictive accuracy. Id 1 1 2 2 3 3 . . 29 29 Which variables to choose? ! Y X 1 3 X 2 X 3 X 4 X 5 . . . x 999 1 2 Ranked 5

Hurdles: Small samples call for different strategies. Issue Typical goal of prediction is to

Hurdles: Small samples call for different strategies. Issue Typical goal of prediction is to crown a definitive winner among all tested models. Solution For national accounts, the ideal is to find a general set of approaches that will consistently yield accuracy gains. M 1 M 2 Algorithm 3 Algorithm 1 0 RMSE (While M 1 is better than M 2, in small samples there is effectively no difference) 0 RMSE (If M 1 and M 2 are derived from the same algorithm but with different inputs, we can form a strategy around a class of algorithm) 6

Hurdles: Predictions must beat current methods. Absolute accuracy of a model is important, but

Hurdles: Predictions must beat current methods. Absolute accuracy of a model is important, but it needs to be contextualized in terms of national economic accounts. HOME GUEST 7

Approach (Part 1): A Prediction Horse Race Evaluate Absolute Performance Prediction Horse Race 1

Approach (Part 1): A Prediction Horse Race Evaluate Absolute Performance Prediction Horse Race 1 2 Identify Best Relative Reductions 3 Predict the Quarterly Services Survey (QSS). 8

Step 1: A Prediction Horse Race 9

Step 1: A Prediction Horse Race 9

Step 1: Data in Horse Race Draw on a broad range of potential source

Step 1: Data in Horse Race Draw on a broad range of potential source data to compare traditional sources and alternative sources. Quarterly Services Survey U. S. Census Bureau 188 industry series n = 31 quarters Source data for significant proportion of PCE Services Credit Card Transactions First Data – Palantir/ Fed Board Revised Series 192 industries Lagged QSS U. S. Census Bureau 188 industry codes lagged for t-4 to t-1 Search Queries Google Trends 230 associated searches Current Employment Survey BLS 140 industries Consumer Price Index BLS 600+ indexes 10 10/28/2020

Step 1: Variable Selection Procedures in Horse Race Cherry Picking Kitchen Sink Include only

Step 1: Variable Selection Procedures in Horse Race Cherry Picking Kitchen Sink Include only conceptually similar variables. All-in. 25 data set combinations 11 10/28/2020

Step 1: Algorithms in Horse Race 4 Q Moving Average Ridge Regression Extreme Gradient

Step 1: Algorithms in Horse Race 4 Q Moving Average Ridge Regression Extreme Gradient Boosting Stepwise Regression CART Support Vector Machines LASSO Regression Random Forest Multi-Adaptive Regression Splines 12 10/28/2020

Step 1: Algorithms in Horse Race 4 Q Moving Average Ridge Regression Extreme Gradient

Step 1: Algorithms in Horse Race 4 Q Moving Average Ridge Regression Extreme Gradient Boosting Type of Method Univariate Stepwise Regression LASSO Regression CART Random Forest Support Vector Machines Multivariate Regression Non-Linear or Non-Parametric Multi-Adaptive Regression Splines 13 10/28/2020

Step 1: Algorithms in Horse Race 4 Q Moving Average Ridge Regression Extreme Gradient

Step 1: Algorithms in Horse Race 4 Q Moving Average Ridge Regression Extreme Gradient Boosting Interpretation Linear Interpretation Stepwise Regression LASSO Regression CART Random Forest Support Vector Machines Other Interpretation None Multi-Adaptive Regression Splines 14 10/28/2020

Step 1: Algorithms in Horse Race 4 Q Moving Average Ridge Regression Extreme Gradient

Step 1: Algorithms in Horse Race 4 Q Moving Average Ridge Regression Extreme Gradient Boosting Single or Ensemble (many in one) Single Stepwise Regression LASSO Regression CART Random Forest Support Vector Machines Ensemble Multi-Adaptive Regression Splines 15 10/28/2020

Methods: A Prediction Horse Race Test Train For Later Iterations 1 Iteration For Later

Methods: A Prediction Horse Race Test Train For Later Iterations 1 Iteration For Later Iterations Test Train 2 3 4 5 6 7 t-5 t-4 t-3 t-2 t-1 t t+1 t+2 t+3 t+4 t+5 Time 16

Methods: A Prediction Horse Race 886, 608 models were trained, based on the combinations

Methods: A Prediction Horse Race 886, 608 models were trained, based on the combinations of industry x data sets x algorithm x variable selection x time period 17

Prediction tracks show persistent a growth pattern is considering many different modeling scenarios. 18

Prediction tracks show persistent a growth pattern is considering many different modeling scenarios. 18 10/28/2020

Some algorithms are more flexible in accounting for different ways of integrating information. NAICS

Some algorithms are more flexible in accounting for different ways of integrating information. NAICS 6211: Physician Offices 19 10/28/2020

Approach (Part 2): A Prediction Horse Race Evaluate Absolute Performance Prediction Horse Race 1

Approach (Part 2): A Prediction Horse Race Evaluate Absolute Performance Prediction Horse Race 1 2 Identify Best Relative Reductions 3 Measure what generally leads to an accuracy increase in the QSS 20

Step 2: Average Absolute Accuracy Estimate a fixed-effects regression to parse out the average

Step 2: Average Absolute Accuracy Estimate a fixed-effects regression to parse out the average accuracy gain associated with each algorithm, data set, etc. 21

Results: Average RMSE Improvement (Algorithms) 0. 56 0. 43 0. 16 0. 00 -0.

Results: Average RMSE Improvement (Algorithms) 0. 56 0. 43 0. 16 0. 00 -0. 04 -0. 25 -0. 68 -1. 48 -2. 15 Random Forest XGBoost LASSO Stepwise Regression Ridge SVM Decision Trees MARS Moving Average 22 10/28/2020

Results: Average RMSE Improvement (Data) 0. 97 0. 81 0. 39 0. 00 BLS

Results: Average RMSE Improvement (Data) 0. 97 0. 81 0. 39 0. 00 BLS CES Dependent Lags First Data BLS CPI Google Trends 23 10/28/2020

Methods: A Prediction Horse Race Evaluate Absolute Performance Prediction Horse Race 1 2 Identify

Methods: A Prediction Horse Race Evaluate Absolute Performance Prediction Horse Race 1 2 Identify Best Relative Reductions 3 Convert QSS into PCE and find sure-fire improvements compared with current 24

Step 3: Calculate Average Dollar Reduction in Revisions 1 Convert QSS into predictions of

Step 3: Calculate Average Dollar Reduction in Revisions 1 Convert QSS into predictions of PCE services components 2 Calculate on average revision if prediction is used 3 Calculate on revision reduction relative to current methods 25

Physician Services: High Chance of Revision Reduction 26 10/28/2020

Physician Services: High Chance of Revision Reduction 26 10/28/2020

Physician Services: High Chance of Revision Reduction 27 10/28/2020

Physician Services: High Chance of Revision Reduction 27 10/28/2020

vs. Stepwise regression is 25% less likely to yield a revision reduction to

vs. Stepwise regression is 25% less likely to yield a revision reduction to physician services when compared with the best method. 28 10/28/2020

Non Profit Hospitals: Less Useful Result 29 10/28/2020

Non Profit Hospitals: Less Useful Result 29 10/28/2020

Next Steps Construct a “moneyball” set of algorithms that yield marked wins for the

Next Steps Construct a “moneyball” set of algorithms that yield marked wins for the home team. 30

Jeffrey. Chen@bea. gov

Jeffrey. Chen@bea. gov

National accounts Environmental accounts Economywide Material Flow AccountsNational accounts Environmental accounts Economywide Material Flow Accounts
National accounts Environmental accounts Economywide Material Flow AccountsNational accounts Environmental accounts Economywide Material Flow Accounts
Machine Learning Machine Learning Machine learningstatistical learning artificialMachine Learning Machine Learning Machine learningstatistical learning artificial
Discussion of Machine Learning National Economic Accounts PatrickDiscussion of Machine Learning National Economic Accounts Patrick
MACHINE LEARNING 102 Jeff Heaton Jeff Heaton DataMACHINE LEARNING 102 Jeff Heaton Jeff Heaton Data
DEPARTMENTAL ACCOUNTS Introduction Departmental accounts are accounts relatingDEPARTMENTAL ACCOUNTS Introduction Departmental accounts are accounts relating
Company Accounts Final Accounts 1 Final Accounts OFCompany Accounts Final Accounts 1 Final Accounts OF
What is Machine Learning Machine learning defined MachineWhat is Machine Learning Machine learning defined Machine
Machine Learning Spark machine learning Spark Mllib MachineMachine Learning Spark machine learning Spark Mllib Machine
What is Machine Learning Machine learning defined MachineWhat is Machine Learning Machine learning defined Machine
NATIONAL ACCOUNTS STATISTICS Quarterly Annual National Accounts InNATIONAL ACCOUNTS STATISTICS Quarterly Annual National Accounts In
Machine Learning Introduction Jeff Howbert Introduction to MachineMachine Learning Introduction Jeff Howbert Introduction to Machine
Machine Learning Data Jeff Howbert Introduction to MachineMachine Learning Data Jeff Howbert Introduction to Machine
Machine Learnin Andrew Ng Machine Learning summary MachineMachine Learnin Andrew Ng Machine Learning summary Machine
Online Learning Yiling Chen Machine Learning Use pastOnline Learning Yiling Chen Machine Learning Use past
Agenda item 4 UNECE National Accounts Financial AccountsAgenda item 4 UNECE National Accounts Financial Accounts
National Accounts Key Accounts Public Sector COVID19 StandardsNational Accounts Key Accounts Public Sector COVID19 Standards
Blint Murai HCSO National Accounts department Production AccountsBlint Murai HCSO National Accounts department Production Accounts
Australian National Accounts State Accounts States of AustraliaAustralian National Accounts State Accounts States of Australia
Incorporating Labour accounts in National accounts Presentation ofIncorporating Labour accounts in National accounts Presentation of
Economic Growth Vs Economic Development Economic growth EconomicEconomic Growth Vs Economic Development Economic growth Economic
Module 5 Accounts Receivable and Accounts Payable LearningModule 5 Accounts Receivable and Accounts Payable Learning
machine learning Supervised Learning Unsupervised Learning Reinforcement Learningmachine learning Supervised Learning Unsupervised Learning Reinforcement Learning
Machine Learning Photo CMU Machine Learning Department protestsMachine Learning Photo CMU Machine Learning Department protests