Macroeconomic Predictions using Payments Data and Machine Learning^††thanks: The opinions expressed herein are those of the authors and do not necessarily reflect those of the Bank of Canada. We thank Poclaire Kenmogne for his excellent assistance on this project. We also thank Alessandro Bitetto, Dalibor Stevanovic, Ingomar Krohn, Jonathan Chiu, Narayan Bulusu, Nicolas Woloszko, Pierre-Yves Yanni, Segun Bewaji, and Zhentong Lu for their comments. In addition, we thank discussants and participants of the following conferences for their comments and suggestions: IAAE Annual Conference (2021), 7th RCEA Time Series Workshop, Big Data and Machine Learning in Finance Conference (2021), 24th Central Bank Macroeconomic Modelling Workshop, Advanced Analytics: New Methods and Applications for Macroeconomic Policy Conference (2021), Time Series in Finance Workshop at ICAIF (2021), Conference on Non-traditional Data, Machine Learning and Natural Language Processing in Macroeconomics (2021), and the seminar at the Bank of Canada. This paper is accepted to present at the AEA 2023 Annual Meeting. ${}^{†}$Corresponding author: ajit.ndesai@gmail.com

In the past, driven by technological advancements, some payment instruments from the ACSS were discontinued or merged into others, and several new payment instruments were created. For example, starting in 2012, a new stream was created to process the Government of Canada’s direct deposit payments. This addition caused a sudden drop in the value and volume of payments in the AFT credit stream, where they were originally processed (see Appendix A for specifics on changes in multiple ACSS streams over time). To overcome the effects of such sudden changes and to get a better representation of payment flow, we merged several streams belonging to similar categories and settled related payments. Also, to overcome the effects of consumers’ payment choices, i.e., when they switch payment method, because, for nowcasting, we are interested in capturing whether spending (or earning) has slowed (or stopped). We include the sum of all payment instruments in the ACSS “Allstream" as a separate series. This should help develop an overall picture from the ACSS and mitigate the effects of unused streams (see Table 1 footnotes for specifics on each adjustment performed).

After these adjustments, we are left with seven streams from the ACSS (which comprise transactions settled in all ACSS payment instruments) and two streams from the LVTS, which are listed in Table 1 along with a short description. For nowcasting, we use both the monthly gross dollar amount, i.e., value, and number of transactions, i.e., volume, settled in the payment instruments; this yields a total of 18 series.

Like other macroeconomic time series, payments data have a strong seasonal component. We adjust all series (both value and volume) for seasonality using the X-13 ARIMA tool [70]. Note that recursive seasonal adjustments are performed in real time using the data available up to the nowcasting horizon at each time step. Year-over-year (YOY) growth rates of the seasonality-adjusted payments series are used to predict the similarly adjusted YOY growth rates of macroeconomic indicators. Using growth rates (instead of levels) helps to induce (approximate) stationarity in both the target and predictors.

Our dataset does not include some payment instruments not settled through the ACSS or LVTS, such as credit card and e-transfer payments.⁸⁸8In 2019, credit card payments accounted for 6.2% of the value and 31.1% of the volume of total retail payments in Canada. Similarly, e-transfers accounted for 1.5% of the value and 2.5% of the volume ([60]. However, [40] conclude that credit card payment data for Canada do not add significant value in nowcasting GDP and retail sales.⁹⁹9Note that in [40], the authors use a short sample size in their analysis of credit card data. The results could differ for a larger sample size. Further, our dataset does not include on-us transactions where both sender and receiver have an account with the same financial institution; such transactions do not need to be settled in a payment system. However, their shares are small and may not materially influence our analysis.¹⁰¹⁰10On-us payments amount to roughly 20% more than those settled in the ACSS. The value of on-us transactions differs by stream, for instance, in encoded paper, it is about 25%, and in POS payments, it is 16% [60].

The crux of the nowcasting problem is that most official estimates of macro indicators are released with a substantial delay. For instance, in Canada, GDP, RTS, and WTS are released with a delay of six to eight weeks. In addition, they undergo multiple revisions, sometimes years later, highlighting the uncertainty of their measurement. Moreover, during a rapid crisis such as COVID-19, macroeconomic predictions are difficult because of the large and unprecedented economic impact. This can undermine the use of lagged data for nowcasting. Therefore, it is valuable to use more timely available information, in this case, payments systems data.

Payments data capture numerous types of transactions from both sides of macroeconomic accounts. For example, consumer income and expenditure, business-to-business payments, and Government of Canada spending. This variety, timeliness, and lack of sampling and measurement error in the payments dataset make it a rich economic information source.

For nowcasting exercises, we use Canada’s monthly GDP, RTS, and WTS at the latest available vintages (i.e., after revisions) and real-time vintages (i.e., first release) as target variables.¹¹¹¹11Latest vintages of seasonally adjusted monthly GDP, RTS, and WTS are obtained from Statistics Canada Tables 36-10-0434-01, 20-10-0008-01, and 20-10-0074-01, respectively. Similarly, historical releases of GDP, RTS, and WTS are obtained from Tables 36-10-0491-01, 20-10-0054-01, and 20-10-0019-01, respectively. We select these indicators because GDP is crucial for policymakers, and since we are using payments data, we think payments data have value in predicting RTS and WTS. All these indicators are released in Canada with a substantial lag and are available monthly for all historical releases. This variation allows us to test the robustness of our models.

YOY growth rates of the latest monthly GDP are plotted with encoded paper and AFT credit values in Figure 1 (top). Similarly, RTS’s YOY growth rates are plotted with POS payments and shared ABM values in Figure 1 (middle). The YOY growth rates of WTS are plotted with corporate payments and LVTS-T2 values in Figure 1 (bottom). To get a sense of the importance of payments data during a crisis, we highlight the growth rates of all variables during the 2008 GFC period (in gray) and the COVID-19 period (in blue).

During the GFC period, the decline and rebound in these payment streams’ growth rates go hand-in-hand with macroeconomic indicators. Similarly, during the COVID-19 shock, we observe a sudden drop in most payment streams and in the macro variables. For instance, GDP and encoded paper, RTS and POS payments, along with WTS and corporate payments show similar movement during both crisis periods. This is a good indication of the economic value associated with these payment streams during such times.

During the COVID-19 period, however, we observe a complicated relationship between the macro indicators and some payment streams. This may be the result of the difference in the nature of the two crises. For instance, the value of payments through the AFT credit stream (which includes GDD payments) did not drop significantly at the onset of the COVID-19 shock. On the contrary, starting in April 2020, the value of payments processed through the AFT credit stream increased due to the flow of government social payments to those directly affected by the pandemic (Figure 1, top). Similarly, we note that the value of payments through the LVTS-T2 stream surged significantly at the onset of COVID-19, showing an opposite behavior to that of macro indicators during the same period. Such behavior is not seen during the GFC period, where both WTS and LVTS-T2 growth rates drop (Figure 1, bottom).¹²¹²12Similar behavior is observed in LVTS-T1, where payment value rose dramatically at the onset of COVID-19 due to extraordinary measures taken by the Bank of Canada under its quantitative easing policy [10]. Such complex behavior could be challenging to capture using traditional linear models.

To exploit non-traditional and large-scale data sources, researchers have recently begun utilizing ML models for economic nowcasting [62, 56, 26]. ML models have been shown to handle wide- and large-scale data efficiently and can manage collinearity. Further, they have been demonstrated to capture nonlinear interactions between the predictors and the target [25, 71, 31, 32, 20].

We use some of the recently popularized parametric and non-parametric ML approaches, such as elastic net [72], support vector machines [65], random forest [19], gradient boosting [37], and feedforward artificial neural networks [12]. For each model, there are many variations proposed in the literature. However, we focus on the simpler version of each model to test their feasibility for macroeconomic nowcasting in Canada. In the remainder of this section, we provide a high-level description of these models. For further details on them, refer to Appendix B and various textbooks [38, 20, 45, 12].

The elastic net (ENT) is a regularized linear regression model. Here, the objective is similar to that of the OLS with the addition of $L_1$ and $L_2$ penalties depending on how large the sum of the parameters $w$ can become.¹³¹³13A regression model that uses only the $L_1$ penalty is a Lasso regression, and a model that uses only the $L_2$ penalty is a Ridge regression [45, 72]. In an ENT regression, the combination of $L_1$ and $L_2$ penalties allows for learning a sparse model while encouraging grouping effects, stabilizing regularization paths, and removing limitations on the number of selected variables [72].

Support vector regression (SVR) is based on support vector machines, where the task is to find a hyperplane that separates the entire training dataset into, for example, two groups, by using a small subset of training points called support vectors. In SVR, the main objective is to determine a decision boundary at a distance from the support vectors such that the data points closest to the hyperplane are within that boundary line. This gives us the flexibility to define how much error is acceptable in our model [22, 65].

Another popular approach is random forest regression (RFR), a decision tree-based ensemble learning method built using a forest of many regression trees. This is a non-parametric approach that addresses the multicollinearity problem slightly differently from parametric approaches such as ENT. Random forest is a bagging (bootstrap aggregation) approach, that is, each tree is independently built from a subset of the training dataset. Each sample randomly selects a subset of features from the available set of features, helping in decorrelation. The final prediction is performed by averaging the predictions of all regression trees [19, 53].

Similar to RFR, gradient boosting regression (GBR) is a tree-based, non-parametric ensemble learning approach. However, unlike RFR, GBR is based on a boosting in which a sequence of weak learners (decision trees) are built upon a repeatedly modified version of the training dataset. In this approach, the base learners are sequentially improved by repeatedly applying the same base learner with the target’s residuals as the outcome of interest [37].

The feedforward artificial neural network (ANN) with hidden layers consists of multiple layers of artificial neurons sandwiched between input and output layers. In this approach, the data always move forward through the network layers. The weighted sum of the first layers is typically passed through a nonlinear activation function resulting in a nonlinear function of the inputs. Then the outputs are sent to the next layer, and the process continues until the last layer. Once we obtain the final output from the network, we measure how good that output is compared to the target’s actual value using an objective function, for example, mean squared error. Given these results, we go back and adjust the weights and biases of the network. Typically we need a large training dataset to achieve good performance using ANN [12, 42].

Note that there are many advanced versions of both tree-based methods, such as LightGBM [49], and deep ANN-based methods, such as long short-term memory (LSTM) [47] proposed in the literature. However, to efficiently utilize them for prediction often requires a large training sample. Since our dataset is small (about 200 sample points), these models do not perform better than the models used in this paper.

Overfitting is commonly attributed to the use of ML models—especially nonlinear models. ML models have many parameters that can be optimized to improve prediction accuracy (commonly known as hyperparameter tuning). Therefore, it is straightforward to tune the model to perform well on a specific dataset, for example, an in-sample training set. However, such models generally fail to perform well when applied to unseen data [45].

This problem can be alleviated using $k$-fold cross-validation techniques. In the standard approach, the training sample is randomly split into $k$-folds, then for each iteration, the $k-1$ folds are used for in-sample training, and the $k^{th}$ fold is used for out-of-sample testing [45]. Such procedure effectively avoids overfitting. However, random splitting of the training sample breaks the order of series (autocorrelation) and could lead to the use of future data points for past predictions, giving an unfair advantage to the model. For these reasons, it may not practical to use it in the same way in nowcasting models [13].

This challenge can be mitigated using an expanding window approach for cross-validation (Figure 2, top). Here, the end portion of the training set, often called a validation set, is kept aside for model tuning and cross-validation. For each iteration of the expanding window, the training sample is increased by one period and the model prediction is performed on the next period from the validation sets [13]. Consequently, the model parameters can be chosen based on model performance on the validation sets. See Appendix C for additional details.

Such approach is useful for nowcasting during normal economic periods. However, in cases where the test sample includes an economic crisis but the validation sample does not, traditional expanding window cross-validation could be challenging because (a) the distribution of test and validation samples are quite different and (b) the model is predominantly tuned for normal periods and therefore may not perform well for out-of-sample crisis periods.

We implement a slightly altered version of the expanding window approach tailored to macroeconomic nowcasting models  (Figure 2, bottom). We randomly sample $n$-points (one for each iteration) from the validation superset (highlighted in gray), beginning just before the GFC and continuing to the end of the training set, and use them as a validation sample. For each iteration of expanding window validation, only data points that come before the chosen point are used for training—preserving the order of data and temporal dependency between observations (see Figure 3).

In the current exercises, since we have the COVID-19 period in the test set, using a random sampling strategy leads us to include a few sample points from the GFC in each validation subset. Therefore, the proposed approach helps make the distribution of validation and test sets somewhat similar (see Figure 14 in Appendix C) and assists in selecting a model that can perform well for both normal and crisis periods. Also, our approach removes the restriction on the number of validation sets we can sample that could be binding in traditional cross-validation approaches. However, in our approach, due to random sampling, some observations may be selected more than once in the validation subsets and some may never be selected. This could lead to overfitting if too many validation sets are sampled.

Further, instead of using all payment streams in each model or manually selecting a few payment streams for the given macro indicator, we use a data-driven approach for feature selection. We treat the number of payment steams $p$ as similar to a model parameter and use the expanding window cross-validation approach to optimally select the best $p$ streams for each target variable based on their performance on the training and validation sets. Further details on cross-validation and model selection are discussed in Appendix D.

Another problem commonly attributed to the use of ML model is the loss of interpretability due to their complex nature. However, interpretability is essential for many applications—including macroeconomic prediction [58, 6, 20].

Some ML models employed here, such as ENT, SVR, and the simple tree-based learning models, are inherently interpretable, but only to a certain extent [72, 19]. Tree-based ensemble approaches such as GBR and RFR can also be interpreted—to some extent—using impurity or permutation-based feature importance methods [19, 57]. However, they are mostly used for global (entire sample) interpretations. Also, each of those methods has different interpretability approaches, making them difficult to compare. To address these challenges, we use the Shapley value-based model agnostic approach SHAP developed in [54]. The SHAP can be used for both the local (each prediction instance) and global interpretations. Moreover, the SHAP can be used for dependence plots which could be valuable to get further insights.

In SHAP, the Shapley value method from coalitional game theory [63, 59] is used to fairly distribute the “payout” (= the prediction) among the “players” (= the predictors) [55]. In nowcasting, SHAP can be used to fairly distribute the ML model prediction among the set of predictors $X_t$ at each time horizon $t$ for local model interpretation. Further, using Shapley values for each instance $t$, we can compute the global interpretation of the ML models in the form of feature importance for the entire training or testing datasets.

[54] propose two approaches based on the type of underlying process used to compute the Shapley values: (1) KernelSHAP, a kernel-based estimation approach, which can be used for many ML models, such as ENT, ANN, and tree-based models, and (2) TreeSHAP, a computationally efficient approach for Shapley value estimation used only for tree-based ML models, such as decision trees, random forests, and gradient boosting models.

SHAP approaches are reliable because they are developed based on the Shapley value, which has game-theoretical foundations. However, the time required to estimate Shapley values could increase exponentially with the number of predictors. This is not a significant concern for our application because we have comparatively fewer predictors and smaller sample sizes. The KernelSHAP method also suffers from collinearity in the predictors, which could represent a concern for our work, given that several predictors are correlated. These problems can be mitigated to an extent using TreeSHAP, but only for tree-based models. Another challenge with SHAP approaches is that it is possible to create intentionally misleading interpretations to hide the bias. Also, in some cases, the outcomes can easily misinterpret and lead to ambiguous conclusions. Therefore, SHAP should be used with caution [2, 64, 57].¹⁴¹⁴14Note that the Shapley value-based interpretation approach does not provide causal inference or any optimal statistical criterion. They only explain the marginal contribution of each feature to the difference between the actual prediction and the mean prediction given the set of predictors. The Shapley value-based approaches could also be used to select the best subset of predictors. However, in the context of model interpretation, the focus is only on computing the marginal contribution of all predictors involved in the prediction exercise [54, 57]. Further details on Shapley values and SHAP with a test example are given in Appendix E.

As a benchmark (or base case), we employ a linear regression model using OLS. Here, we use the first available lagged target variable along with the latest available CPI, unemployment, CFSI,¹⁵¹⁵15CFSI is computed using data from the following seven market segments: equity market, Government of Canada bond market, foreign exchange market, money market, bank loans market, corporate bonds market, and housing market. and CBCI.¹⁶¹⁶16CBCI is based on the Conference Board’s survey of Canadian households, which provides a measure of consumer optimism on current economic conditions. CFSI and CBCI are available immediately after period-end and carry comprehensive and useful information about the macro indicators. Along with CPI and UNE (available with a one-to-two week delay), these predictors form a strong benchmark to assess information gain using payments systems data.

In the main case of interest, along with the predictors specified in the base case, we use the payments data listed in Table 1. Here, we first use DFM to assess the marginal contribution of payments data when used in a sophisticated econometric model. Next, we test the usefulness of the various ML models discussed above in Section 3. Finally, we compare the performance of the ML models with the benchmark case and DFM. We use RMSE as the key performance indicator for out-of-sample model evaluation.

For all cases, using a procedure similar to  [41, 40], we perform nowcasting at three monthly time horizons, extending from the start of the month of interest ($t$) until the month before the official release ($t+2$). As we advance in time, we include new predictors when they become available. For example, GDP nowcasting at time horizon $t$, that is, on the first day of the month of interest, we use the latest available benchmark variables and the monthly aggregated payments data available at $t-1$. Model $F$ can be specified as¹⁷¹⁷17GDP is released with a two-month lag, CPI and UNE are released with a one-to-two week lag, and CFSI, CBCC, and payments data are available the day after the end of the period.

Similarly, at the next nowcasting horizons $t+1$ and $t+2$, using the latest available predictors, the models can be specified as¹⁸¹⁸18At the $t+2$ nowcasting horizon (on the first day of the month in which the target month’s macro indicators are released), we have $t+1$ months of payments data. However, we do not include this because we are interested mainly in assessing the usefulness of $t$ month’s payment data to predict $t$ month’s macro variables. Also, note that the left-hand sides in Equations 4, 5, and 6 represent the same month’s target but they are estimated at different time horizons.

We train the nowcasting models using the expanding window approach as schematically outlined in Figure 3. First, we divide the dataset into two subsets: a training set (in-sample) for model training and a testing set (out-of-sample) for predictions. The OLS and DFM models are directly trained on the training set and used for predictions on the test set. The ML models, which require extensive hyperparameter tuning and cross-validation, are trained using the following procedure:

1. From the training sample, we select two dates covering the wider range of training data as a validation superset (Figure 2) and randomly choose a set of $n$ sample points as a validation set (where $n=24$ points, it is the same size as the test sample).¹⁹¹⁹19We choose a start date just before the GFC period and an end date just before the test set, then select $n$ random data points between these two dates as a validation set. This helps to include a few data points from the crisis period in each fold of the validation subset, at the same time avoiding use of a large cross-validation sample.

2. Thereafter, for each sample date in the validation subset, we select all the sample points before that date for training and use the sample date for prediction (Figure 3). In this way, we maintain temporal dependency and avoid using future data for predictions in the past.

3. Next, for each model, we specify the grid for selected hyperparameters. Then, for each value of a specified parameter, we iterate over the validation subset and compute RMSE.

4. Steps 2 and 3 are repeated $k$ times for the same set of hyperparameters but with a different validation subset randomly sampled from the validation superset ($k=5$ fold cross-validation).

5. Next, we select the best set of model parameters, i.e., the parameters with the lowest average validation RMSE (averaged over $k$ folds) as the final model.

6. Finally, the chosen model is used for predictions on the test set by utilizing the standard expanding window approach over the training and test set (Figure 3).

We now discuss the Shapley value-based interpretation of ML model predictions using the SHAP. We primarily focus on nowcasting GDP at time horizon $t+1$ using the tuned GBR model. However, we discuss a few key results for nowcasting RTS and WTS using a GBR, at the end of this section.

For demonstration, we use the entire sample (Mar 2005 to Dec 2020) for training. In Figure 4, we plot the SHAP global feature importance obtained by averaging the absolute Shapley values for each predictor across the training set (in-sample). This plot shows, on average, how much each feature influences the model prediction. These features are ranked according to their average influence (from high to low). For example, in the case of in-sample training data, both encoded paper value and GDP lag have a strong influence. However, the encoded paper stream is the strongest predictor (on average, it changes the GDP growth rate by $0.6$ points). This is followed by the unemployment lag feature, LVTS-T2 value, and the sum of all the ACSS streams (Allstreams value).²³²³23For the tree-based models like GBR, we can also use impurity or permutation-based global feature importance [19]. In our case, the permutation approach gives similar results as SHAP for the top three major predictors and matches eight out of the top ten highest contributors but slightly in a different order. Moreover, all three approaches rank the same three predictors in the top five list, and the Encode paper stream remains the most prominent feature in all approaches (see Figure 17).

In Figure 5, we show the global feature importance plot for the COVID-19 period with high negative growth rates (Mar to Dec 2020). During this crisis period, the influence of the encoded paper stream increases substantially along with the Allstream, AFT debit, and POS payments. GDP lag, the second most important feature for the entire training sample, loses its prediction power during the COVID-19 crisis period. A similar contribution from several payment streams is observed during the GFC period. These results suggest that, although lagged macro indicators influence GDP growth rates during normal periods, they do not add much value during crisis periods due to the delayed signal. During such periods, payments data become much more valuable and contribute well to macroeconomic prediction.

Next, using the SHAP “force” plots, we compute local feature importance, that is, the usefulness of each feature for a chosen sample point in the training set. Such insights could be important for nowcasting exercises because, during each step of the expanding window approach (i.e., when we advance by one month), the force plots could provide additional insights into each month’s predictions by highlighting marginal contributions of individual predictors.

For instance, in Figure 6, we plot the Shapley values as forces for Feb (top), Mar (middle), and Apr (bottom) 2020. Here, each Shapley value is an arrow that forces an increase (higher in red) or decrease (lower in blue) in the prediction from the baseline (the average of all predictions). The length of these arrows indicates the magnitude of the Shapley value for that feature. These forces balance at the model prediction of that instance shown as $f\left(x\right)$. For Feb 2020, just before the pandemic began affecting Canada’s economy, most payment predictors are positive (red) and pushing GDP growth higher. However, for Mar 2020 (the first month of the COVID-19 shock), most payment streams show a negative signal (blue). Similarly, for Apr 2020, all predictors show strong negative contributions, pushing the model prediction to the lowest value.

Figure 7 shows the force plots for each instance in the entire training sample rotated and stacked together vertically. We observe red clusters of predictors with positive values (most predictors contributing positively to GDP prediction) during positive economic growth periods and blue clusters with negative signals during crisis periods, such as the GFC and COVID-19 shock. Such clustered signals could prove valuable in tracking crises in real time.

In Figure 8, we show dependence plots for encoded paper value (left) and Allstream value (right). These plots capture the relationship between the feature values on the x-axis and the corresponding Shapley values on the y-axis. Observe that the small and negative values of encoded paper growth rates provide higher contributions in Shapley values compared to the positive growth rates. However, both positive and negative growth rates of Allstreams value contribute similarly (or symmetrically). The encoded paper plot (left) suggests that the contribution of this stream, in terms of Shapley values, is small and linear during periods of normal growth. However, during periods of strong negative or positive growth ($>=|10|$), the contribution of this stream is asymmetrical and nonlinear.

Similar behavior is observed in nowcasting models for RTS and WTS using payments data and GBR model. In Figure 9, we show the dependence plots of RTS with POS payments value (left) and WTS with Allstream value (right). We also show how these payment streams are influenced by CFSI. These plots suggest that at high-stress levels, that is, at high values of CFSI (shown in red) and negative values of payment growth rates, the signal from these payment streams is strong and their contribution is high. However, for low levels of stress (blue) and high positive payment growth rates, the contributions from these streams are positive but small. This confirms the asymmetrical and potentially nonlinear relationship between these streams and the corresponding macro variables.

Finally, in Figure 10, we plot the SHAP global feature importance for the entire training set at time horizon $t+1$ for RTS (left) and WTS (right), respectively. These plots suggest, in the case of RTS, that the Allstream and POS payments values highly influence the model prediction. This makes sense, given that POS payments are commonly used for retail trade. In the case of WTS, the Allstream value has its strongest impact on the model prediction along with the encoded paper and corporate payments value streams, highlighting the importance of these streams in predicting WTS. These plots suggest that, in general, aggregated payments streams (Allstream and encoded paper) are crucial predictors in nowcasting GDP, RTS, and WTS.