ARMA Cell: A Modular and Effective Approach for Neural Autoregressive Modeling

In this work, we propose a new type of RNN cell (cf. Figure 1) that can be seen as a natural connection between the classical time series school of thoughts and DL approaches. To analyze how the ARMA modeling philosophy can improve neural network predictions, we

• embed ARMA models in a neural network cell, which has various advantages over classical approaches (see Section 4);

• further exemplify how this proposal can be extended to convolutional approaches to model tensor-variate time series such as image sequences (Section 4.3);

• demonstrate through various numerical experiments that our model is on par with or even outperforms both its classical time series pendant as well as the LSTM cell in various settings; architectures ranging from shallow linear to deep non-linear time series models;

• provide a fully-tested, modular and easy-to-use TensorFlow (Abadi et al., 2016) implementation with a high-level syntax almost identical to existing RNN cells to foster its usage and systematic comparisons. It is available at https://github.com/phschiele/armacell.

The goal of this paper is further to make practitioners aware of an alternative to commonly used RNN cells, highlight that short-term recurrence can be sufficient in various time series applications, and that a simpler parameterized lag structures can even outperform data-driven forget mechanisms.

We start by discussing related literature in the following section. A short mathematical background is given in Section 3, followed by our proposed modeling approach in Section 4. We investigate practical aspects of our method in Section 5 and summarize all ideas and results in Section 6.

Autoregressive integrated moving average (ARIMA) models (see, e.g.,  Shumway and Stoffer, 2000) are a general class of linear models for forecasting a time series. They are characterized by a linear function of lags of the dependent variable (response) and lags of the forecasting errors. Boosted by the seminal paper of Box et al. (2015) and further popularized by their simplicity yet practical efficacy, ARIMA models have been extensively studied in statistics, econometrics, and related fields. A special case of ARIMA models are autoregressive moving average (ARMA) models, which are not “integrated”, i.e., no differencing steps are required to obtain a stationary mean function. To overcome the linear restrictions of the AR(I)MA model and to account for non-linear patterns observed in real-world problems, several classes of non-linear models have been introduced. Examples include the bilinear model of Granger and Andersen (1978); Rao (1981), the threshold autoregressive model by Tong and Lim (2009), the smooth transition autoregressive model by Chan and Tong (1986), or the Markov switching autoregressive model by Hamilton (2010). Although these approaches have advantages over the linear methods, these models are usually developed to capture very specific non-linear patterns, and hence their generalization is limited. Many common extensions of AR(I)MA models such as seasonal ARIMA (SARIMA) or ARIMA with exogenous variables (ARIMAX) exist (Box et al., 2015). We here only focus on the basic ARMA model, but the proposed approaches in this paper can be easily extended to include the respective peculiarities of real-world time series such as non-stationarity, seasonality, and (non-)linear exogenous variables. Several authors have recognized analogies between ARMA models and RNNs (Connor et al., 1991, 1994; Saxén, 1997), which we will discuss in the following.

When modeling sequential data such as time series, RNNs allow previous states to influence the current one, which presents a natural extension of classical perceptron-type connections (Rumelhart et al., 1986). When considering a feedforward neural network with a single hidden layer, such cyclical connections can be established by concatenating the output of the previous time step to the inputs of the next, yielding an Elman network (Elman, 1990). Similarly, a Jordan network is obtained by concatenating the previous hidden layer to the subsequent input (Jordan, 1986). An often-cited shortcoming of these so-called “simple recurrent networks” is their inferiority when learning long-term dependencies, which can be challenging due to vanishing or exploding gradients (e.g.,   Goodfellow et al., 2016). A variety of methods have since been developed to compensate for this shortcoming, including modeling multiple time scales simultaneously, adding leaky connections or allowing for longer delays within the recurrent connections. Most prominent are gated recurrent units (GRU;  Cho et al., 2014) and long short-term memory cells (LSTM; Hochreiter and Schmidhuber, 1997). LSTM cells introduce self-loops that allow the gradient to flow without vanishing even for long durations using an input and a forget gate. GRU cells are similar to LSTM cells but only use a single gating unit to simultaneously control the forgetting and updating mechanism. Both methods have been shown to effectively tackle problems associated with long-term dependencies.

A natural approach to allow for both linear autoregression and flexible non-linearity as provided by RNNs is to combine the two modeling techniques. Various hybrid approaches have been proposed in the past. One of the most common ways to combine the two paradigms is to fit a (seasonal) AR(I)MA model to the time series and subsequently train a neural network to explain the remaining variance in the residuals of the first stage mode (Aslanargun et al., 2007; Fathi, 2019; Tseng et al., 2002; Zhang, 2003). Other approaches specify a (time-delayed) neural network using the information from a preceding linear model by detrending and deseasonalizing in accordance with the first stage model (Taskaya-Temizel and Ahmad, 2005; Zhang and Qi, 2005).

State space models (SSMs) represent another popular model class in time series analysis (Aoki, 1990). The main advantages of SSMs are their generalized form and possible applications on complex, non-linear time series. A combination of SSMs and RNNs was proposed by Rivals and Personnaz (1996) as well as Suykens et al. (1995) using a network with a state layer between two hidden layers. This yields the so-called state space neural network (Amoura et al., 2011; Zamarreño and Vega, 1998). A more general representation of neural networks as a dynamical system is proposed by Hauser et al. (2019) and a deep combination of SSMs and RNNs was further suggested by Rangapuram et al. (2018). While it is possible to represent ARMA models using SSMs, existing neural network SSM approaches in the literature do not aim for a general and modular building block, but propose specific networks using a fixed architecture.used for the mapping from features to SSM parameters.

Error Correction Models (ECMs) are commonly used to forecast cointegrated time series, i.e., time series that have a common long-term stochastic trend. For such time series, the information contained in the levels of the data is lost during a differencing step, making ECMs more suitable compared to ARIMA models. Mvubu et al. (2020) introduced a neural variant of ECMs, the Error Correction Neural Networks.

While some hybrid methods such as the combination of graph neural networks with an LSTM network (Smyl, 2020) have been shown to excel in time series forecasts, the efficacy of hybrid approaches combining AR(I)MA models and RNNs is often not clear (Taskaya-Temizel and Ahmad, 2005; Terui and Van Dijk, 2002). Khashei and Bijari (2011) propose a two-stage procedure that is guaranteed to improve the performance of the two single models, but as for all other existing hybrid approaches their technique requires fitting two separate models and hence cannot be combined or extended in a straightforward manner.

For spatio-temporal sequence forecasting problems, an extension to fully-connected RNN cells are RNN cells that apply convolutional operations to the spatially distributed information of multiple time series. As for time series applications, popular approaches use long-memory mechanisms, e.g., convolutional GRU adaptations (Tian et al., 2019) or the ConvLSTM (Shi et al., 2015) as an extension of the LSTM cell for spatio-temporal data.

The ARMA($p,q$) model (Box et al., 2015) for $p,q\in N_0$ is defined as

where $x_t$ represents the variable of interest defined for $t\in Z$ and is observed at time points $t=1,\dots ,T$, $T\in N$¹¹1It is common to define a time series for time points $t=1,\dots ,T$ to describe its current value and recent history, while time series dynamics are assumed to originate from time points prior to $t=1$, hence $t\in Z$. $\alpha$, ${\beta }_1,\dots ,{\beta }_p$, ${\gamma }_1,\dots ,{\gamma }_q$ are real valued parameters and ${\epsilon }_t\stackrel{iid}{\sim }F\left({\sigma }^2\right)$ is an independent and identically distributed (iid) stochastic process with pre-specified distribution $F$ and variance parameter ${\sigma }^2>0$. By setting $q=0$ or $p=0$, the ARMA model comprises the special cases of a pure autoregressive (AR) and a pure moving average (MA) model, respectively. The class of ARMA models is, in turn, a special case of ARIMA models, where differencing steps are applied to obtain a stationary mean function before fitting the ARMA model. As stationarity is also a fundamental assumption for RNNs to justify parameter sharing (Goodfellow et al., 2016), we focus on the class of ARMA models in this work, i.e., assume that differencing has already been applied to the data. series is characterized by a constant mean and variance and a time invariant autocorrelation structure. ${\alpha }_0,\dots ,{\alpha }_p$ and ${\beta }_1,\dots ,{\beta }_q$ are model parameters and $p$ and $q$ characterize the number of lags of the dependent variable and the forecasting errors included in the model, respectively.

The univariate ARMA model can be generalized to a multivariate version – the vector autoregressive moving average (VARMA) model – by adapting the principles of the ARMA model for multivariate time series. The VARMA($p,q$) model (Tiao and Box, 1981) for $p,q\in N_0$ is defined as

where $x_t,t\in Z$ represents a vector of time series observed at time points $t=1,\dots ,T$. $B_i$ and ${\Gamma }_j$ are time-invariant $(k\times k)$-matrices, where $k\in N$ represents the number of individual time series. ${\epsilon }_t$ is a $k$-dimensional iid stochastic process with pre-specified $k$-dimensional distribution $F\left(\Omega \right)$ and covariance matrix $\Omega$. By setting $q=0$, the VARMA model comprises the special cases of a pure autoregressive (VAR) model, which is the most common VARMA model used in applications. Similar to the ARMA model being a special case of the ARIMA model class, the VARMA model is a special case of the VARIMA model class, representing only stationary time series.

For (multivariate) time series, we compare a shallow and a deep variant of the ARMA cell against the respective (V)ARMA model and neural models. For the latter, we consider LSTM, GRU, and Simple RNN cells, again each in a their shallow and deep variants. Hyperparameter optimization is done using a grid search with predefined parameter spaces for the number of units for all network layers and lags for ARMA-type models. All other hyperparameters of network layers are kept fixed with defaults that do not favor one or the other method. For moving images, we compare ConvARMA against a naïve approach of repeating the last image and a ConvLSTM network with parameter specifications that are defined as similar as possible to the one of ConvARMA. Further details on the specification of the architectures can be found in the Supplementary Material B.4.

We compare time series predictions using the root mean squared error (RMSE) for uni- and multivariate time series forecasts, and the cross-entropy loss for next frame video predictions. We provide further performance measures for our comparisons in the Supplementary Material D.