Modelling and Evaluating Restricted ESNs

Abstract

We investigate various methods of combining Echo State Networks (ESNs), including a method that we dub Restricted ESNs. We provide a notation for describing Restricted ESNs, and use it to benchmark a standard ESN against restricted ones. We investigate two methods to keep the weight matrix density consistent when comparing a Restricted ESN to a standard one, which we call “overall consistency” and “patch consistency”. We benchmark restricted ESNs on NARMA10 and the sunspot prediction benchmark, and find that restricted ESNs perform similarly to standard ones. We present some application scenarios in which restricted ESNs may offer advantages over standard ESNs.

Appendices

A Calculating $D_W$ for the Overall-Consistent Case

Given an ESN with N nodes and an average density $0 \le D \le 1$, we wish to restrict that ESN to have n subreservoirs of equal size; we assume n divides N. We set the density within the subreservoirs, $D_W$, to be greater than the density outside the subreservoirs by a factor of f, that is, $D_W = f D_B$.

In a restricted ESN with n subreservoirs, each of size N/n, there are n regions in the edge matrix ${\textbf {W}}$ of size $({N}/{n})^2$ with density $D_W$, and a further $n^2-n$ regions also of size $({N}/{n})^2$ with density $D_B$.

Hence the average density D of such a restricted ESN is:

$$\begin{aligned} D = \frac{n D_W + (n^2 - n) D_B}{n^2} \end{aligned}$$

(6)

Substituting $D_W = f D_B$, and rearranging to get an expression for $D_B$ in terms of D, we get:

$$\begin{aligned} D_B = \frac{Dn}{f + n - 1} \end{aligned}$$

(7)

Once $D_B$ is known, we also have $D_W$ from $D_W = f D_B$.

B Optimising f

In order to find the best possible restricted ESN within our constraints, we optimise over the parameter f. However, we must somehow limit our search space.

In the restricted ESN, we want $D_B$ to be strictly less than $D_W$ (less dense connections than subreservoirs); therefore, $f > 1$.

To find an upper bound, we assume that every subreservoir is connected to every other subreservoir, that is, every connection weight matrix ${\textbf {B}}_{i,j}$ has at least one entry. This requires $D_B \ge (n/N)^2$. (In the experiments, the weight matrices are generated probabilistically, so when close to this density limit, it may be the case that there is not an edge between all subreservoirs.)

Rearranging Eq. 7 gives:

$$\begin{aligned} f = \frac{D n}{D_B} - n + 1 \end{aligned}$$

(8)

The lower limit on $D_B$ gives an upper limit on f:

$$\begin{aligned} f \le \frac{N^2 D}{n} - n + 1 \end{aligned}$$

(9)

We also have an upper limit on the derived density, $D_W \le 1$ (equality implies there are no zero elements in the relevant weight matrix). Substituting for $D_W$ in Eq. 7 gives:

$$\begin{aligned} \frac{fDn}{f + n - 1} = D_W \le 1 \end{aligned}$$

(10)

Rearranging gives another upper limit on f:

$$\begin{aligned} f \le \frac{n-1}{Dn-1} \end{aligned}$$

(11)

Hence we have the upper and lower bounds on f:

$$\begin{aligned} 1 < f \le \text{ min } \left( \frac{N^2 D}{n} - n + 1, \frac{n-1}{Dn-1} \right) \end{aligned}$$

(12)