1 Introduction

Wireless communications are developing very fast in order to adapt with the increasing demand for high-speed and reliable data transmission over a wide coverage area. Using relays for coverage extension has become popular in various wireless networks such as ad hoc, sensor and cellular mobile. Physical-layer network coding (PNC) is an efficient solution to increase link throughput as well as system performance of the two-way relay communication networks [1,2,3]. Among various PNC schemes, the channel quantization-based one, abbreviated as CQ-PNC [3], is a potential candidate since it can achieve near optimal performance while requiring low computational complexity.

1.1 Related works

Meanwhile, multiple-input multiple-output (MIMO) technology, including the vertical bell-labs layer space-time (V-BLAST) [4] and the Alamouti’ space-time block code (STBC) [5], is a widely accepted solution for various wireless point-to-point communication systems. Combination of MIMO and PNC for the two-way relay networks has received increasing attention recently [6,7,8,9,10,11,12]. In [6], the authors proposed a two-way relay network model in which all nodes have two antennas to transmit Alamouti codes. At the relay node thanks to appropriate signal processing, it is possible to split the system into two 2x1 Alamouti transmit diversity ones, thereby achieving the order-2 diversity. Furthermore, a beamforming method was also used at terminal nodes in order to further increase the system performance. The transmit beamformer, however, needs to have knowledge of CSI at the two terminal nodes. In [7], the author applied the concept of MIMO to a two-way relay network. Although Alamouti STBC was used for transmission during the broadcast (BC) phase from the relay to the terminal nodes, only a single antenna was used for the multiple access (MA) phase for the inverse links from the two terminal nodes to the relay. This is in fact a single-input multiple-output (SIMO) relay system. Moreover, network coding (NC), i.e. decoding and re-encoding, rather than PNC was used in the system. The work in [8] considered an MIMO two-way relay system in which all nodes use the Alamouti STBC for transmission. NC based on ML estimation was proposed for obtaining the transmit symbol at the relay node. This system can improve the spectral efficiency as well as the BER performance. However, it requires more complexity for ML estimation. In a recent work [9], the authors proposed to combine the Alamouti STBC with PNC for an MIMO two-way relay network in which channel coding was also used for error correction. In this system, precoding was used at the terminal nodes in order to reduce the detection complexity at the relay. However, channel state information (CSI) of the forward channels is required at the two terminal nodes. In our two recent researches [10, 11], we have proposed MIMO-PNC schemes which use linear estimation in combination with mapping decision based on maximum likelihood and selective combining for PNC mapping. The work in [10] combines PNC with MIMO spatial division multiplexing (MIMO-SDM) while [11] considers the combination of STBC with PNC. Channel quantization was not used in these works and thus, performance could be further improved. The last work [12] integrates channel quantization into the MIMO-SDM system. However, since SDM was used in order to increase the spectrum efficiency, error performance was not optimized due to the multiplexing-diversity trade-off.

To achieve higher diversity, relay selection [11, 13] and transmit antenna selection [14] were also proposed for MIMO cooperative communication systems. These algorithms rely on the channel gains to select the best intermediate node to serve as the relay. The works in [11, 13] proposed three relay selection algorithms based on the maximum channel gains, the maximum harmonic mean of the channel gains, and the detection mean square error (MSE). Performance evaluations revealed that the MSE-based algorithm provides the best error performance. One disadvantage of this algorithm is that it requires inversion of the channel matrix and thus is of high complexity. The transmit antenna selection proposed in [14] achieves significant improvement in the symbol error rate (SER) performance but the implementation of the algorithm requires the transmitter to have full knowledge of the channel state information (CSI), which is not always possible in realistic situations.

1.2 Contributions of the paper

Motivated by the above open problems, this paper proposes a combined CQ-PNC and STBC system for the two-hop two-way relay system. Our contributions can be summarized as follows. First, based on the concept of CQ-PNC [3] we derive a PNC mapping rule for two pairs of network coded symbols. Second, we propose an adaptive mapping scheme based on minimum residual interference power to reduce the effect of quantization error. This adaptive scheme differs from those of [3, 12] in that we propose the minimum residual interference power as a metric for adaptation while both the previous works use the quantization step. Finally, a relay selection method based on minimum residual interference power is proposed as a solution to increase further the system performance. Simulation results confirm the performance improvement of the proposed scheme.

The rest of paper is organized as follows. The system model of the considered system is presented in Sect. 2. The proposed STBC-CQ-PNC scheme is introduced in Sect. 3 followed by the relay selection base on minimum residual interference power in Sect. 4. Computational complexity is analyzed and compared with related schemes in Sect. 5. Performance evaluation is given in Sect. 6 and, finally, conclusions are summarized in Sect. 7.

2 System model

Figure 1 illustrates a typical two-hop two-way relay network (TWRN) in which all nodes have two antennas and use the Alamouti STBC for transmission. Terminal nodes \(\mathrm {N}_k,~(k = 1,2)\) exchange their messages via the relay \(\mathrm {R}\). It is assumed that all nodes operate in the half-duplex mode. All channels are affected by flat quasi-static Rayleigh fading.

Fig. 1
figure 1

Model of bidirectional STBC-CQ-PNC system

In this scheme, the data exchange between the two terminal nodes consists of two phases, namely, the MA and BC. Each phase includes two consecutive time slots denoted by \(t_1 \), \(t_2 \). Since data transmission during the BC phase is similar to that in a single-hop transmission system, we limit ourselves only to the MA operations.

The transmit signal matrix \({{\mathbf {X}}}_k\) by the terminal node k (\(k = 1,2\)) using the Alamouti STBC during two consecutive time slots is given by

$$\begin{aligned} \begin{array}{l} {{\mathbf {X}}}_k = \left[ { \begin{array}{*{20}c} {x_1^{(k)} } &{} { - x_2^{(k)*} } \\ {x_2^{(k)} } &{} {x_1^{(k)*} } \\ \end{array}} \right] , \end{array} \end{aligned}$$
(1)

where \(x^*\) denotes the complex conjugation of x. The channel matrices between \(\mathrm {N}_1\), \(\mathrm {R}\) and \(\mathrm {N}_2\), \(\mathrm {R}\) are, respectively, denoted by \({{\mathbf {H}}}_1 \) and \({{\mathbf {H}}}_2 \) as follows:

$$\begin{aligned} {{\mathbf {H}}}_1 = \left[ { \begin{array}{*{20}c} {h_{11}^{(1)} } &{}\quad {h_{12}^{(1)} } \\ {h_{21}^{(1)} } &{}\quad {h_{22}^{(1)} } \\ \end{array}} \right] ,\quad {{\mathbf {H}}}_2 = \left[ { \begin{array}{*{20}c} {h_{11}^{(2)} } &{}\quad {h_{12}^{(2)} } \\ {h_{21}^{(2)} } &{}\quad {h_{22}^{(2)} } \\ \end{array}} \right] , \end{aligned}$$
(2)

where \(h_{uv}^{(k)} \) represents the channel gain between the v-th antenna \((v = 1,2)\) of the k-th terminal node and the u-th antenna \((u = 1,2)\) of the relay node. Each element \(h_{uv}^{(k)} \) is modeled by a complex Gaussian variable with mean 0 and variance 1, i.e. \(h_{uv}^{(k)} \sim {{\mathcal {N}}}_c \left( {0,1} \right) .\)

During time slot \(t_1\), the received signals at the two antennas of the relay node are given by

$$\begin{aligned} y_{1,t_1 }&= \frac{1}{{\sqrt{2} }}\left( h_{11}^{(1)} x_1^{(1)} + h_{12}^{(1)} x_2^{(1)} + h_{11}^{(2)} x_1^{(2)} + h_{12}^{(2)} x_2^{(2)} \right) + z_{1,t_1 } \end{aligned}$$
(3)
$$\begin{aligned} y_{2,t_1 }&= \frac{1}{{\sqrt{2} }}\left( h_{21}^{(1)} x_1^{(1)} + h_{22}^{(1)} x_2^{(1)} + h_{21}^{(2)} x_1^{(2)} + h_{22}^{(2)} x_2^{(2)} \right) + z_{2,t_1 }. \end{aligned}$$
(4)

Similarly, the received signals during time slot \(t_2\) are given by

$$\begin{aligned} y_{1,t_2 }&= \frac{1}{{\sqrt{2} }}\left( h_{12}^{(1)} x_1^{(1)*} - h_{11}^{(1)} x_2^{(1)*} + h_{12}^{(2)} x_1^{(2)*} - h_{11}^{(1)} x_2^{(2)*} \right) + z_{1,t_2 } \end{aligned}$$
(5)
$$\begin{aligned} y_{2,t_2 }&= \frac{1}{{\sqrt{2} }}\left( h_{22}^{(1)} x_1^{(1)*} - h_{21}^{(1)} x_2^{(1)*} + h_{22}^{(2)} x_1^{(2)*} - h_{21}^{(1)} x_2^{(2)*} \right) + z_{2,t_2 }, \end{aligned}$$
(6)

where the fraction \(\frac{1}{\sqrt{2}}\) is the transmit power normalization factor used by the terminal nodes. \(z_{{{u,t}}_1 }\) and \(z_{{{u,t}}_2 }\) denote the additive white Gaussian noise (AWGN) at the u-th antenna branch of the relay node at time slot \(t_1\) and \(t_2\), respectively. These noise elements has zero mean and variance \(\sigma _n^2 \), i.e. \(z_{{{u,t}}_1 } ,\,\,z_{{{u,t}}_2 } \sim {{\mathcal {N}}}_c \left( {0,\,\sigma _n^2 } \right) \).

Rearranging Eqs. (3), (4), (5) and (6) in the vector form, we have the system equation given by

$$\begin{aligned} {\mathbf {y = Hx + z}}, \end{aligned}$$
(7)

where

$$\begin{aligned} {{\mathbf {H}}} \buildrel \Delta \over = \frac{1}{{\sqrt{2} }}\left[ { \begin{array}{*{20}c} {h_{11}^{(1)} } &{} {h_{12}^{(1)} } &{} {h_{11}^{(2)} } &{} {h_{12}^{(2)} } \\ {h_{12}^{(1)*} } &{} {- h_{11}^{(1)*} } &{} {h_{12}^{(2)*} } &{} {- h_{11}^{(1)*} } \\ {h_{21}^{(1)} } &{} {h_{22}^{(1)} } &{} {h_{21}^{(2)} } &{} {h_{22}^{(2)} } \\ {h_{22}^{(1)*} } &{} {- h_{21}^{(1)*} } &{} {h_{22}^{(2)*} } &{} {- h_{21}^{(2)*} } \\ \end{array}} \right] , \end{aligned}$$

\({{\mathbf {x}}} = \big [x^{(1)}_1,x^{(1)}_2,x^{(2)}_1,x^{(2)}_2\big ]^T\), \({{\mathbf {y}}} = \left[ y_{1,t_1 }, y_{1,t_2 }^*, y_{2,t_1 }, y_{2,t_2 }^*\right] ^T\), and \({{\mathbf {z}}} = \left[ z_{1,t_1 }, z_{1,t_2 }^*, z_{2,t_1 }, z_{2,t_2 }^*\right] ^T\).

Using the QR decomposition the channel matrix \({{\mathbf {H}}}\) can be factored as:

$$\begin{aligned} {\mathbf {H}} ={\mathbf {Q}}{{\mathbf {R}}}, \end{aligned}$$
(8)

where \({{\mathbf {Q}}}\) is an unitary matrix with \({\mathbf {QQ}}^H = {{\mathbf {I}}}_4\) and \({{\mathbf {R}}} \in {{\mathbb {C}}^{4 \times 4}}\) is an upper triangular matrix. It is noted that

$$\begin{aligned} {\mathbf {R = Q}}^H {{\mathbf {H}}} = \left[ { \begin{array}{*{20}c} {r_{11} } &{}\quad 0 &{}\quad {r_{13} } &{}\quad {r_{14} } \\ 0 &{}\quad { - r_{11}^* } &{}\quad {r_{14}^* } &{}\quad { - r_{13}^* } \\ 0 &{}\quad 0 &{}\quad {r_{33} } &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad 0 &{}\quad { - r_{33}^* } \\ \end{array}} \right] . \end{aligned}$$

Multiplying both sides of (7) by \({{\mathbf {Q}}}^H\), we have

$$\begin{aligned} {{\mathbf {Q}}}^H {{\mathbf {y}}} = {\mathbf {Rx}} + {{\mathbf {Q}}}^H {{\mathbf {z}}}. \end{aligned}$$
(9)

Defining \({\mathbf {w}} \triangleq [w_1 ,\,w_2 ,\,w_3 ,\,w_4 ]^T = {{\mathbf {Q}}}^H {{\mathbf {y}}}\) and \({{\mathbf {u}}} \triangleq [u_1 ,\,u_2 ,\,u_3 ,\,u_4]^T = {{\mathbf {Q}}}^H {{\mathbf {z}}}\) we can rewrite Eq. (9) as follows:

$$\begin{aligned} {{\mathbf {w}}} = {\mathbf {Rx}} + {{\mathbf {u}}}. \end{aligned}$$
(10)

To perform PNC the relay needs to estimate the network coded symbols \(x^{(1)}_1\oplus x^{(2)}_2\) and \(x^{(1)}_2\oplus x^{(2)}_1\) from Eq. (10). In the following sections, we will use \(\overline{x^{(1)}_1\oplus x^{(2)}_2}\) and \(\overline{x^{(1)}_2\oplus x^{(2)}_1}\) to denote the estimates of \(x^{(1)}_1\oplus x^{(2)}_2\) and \(x^{(1)}_2\oplus x^{(2)}_1\), respectively. Here, \(a \oplus b \triangleq {\mathop {\mathrm{Re}}\nolimits } (a){\mathop {\mathrm{Re}}\nolimits } (b) + j{\mathop {\mathrm{Im}}\nolimits } (a){\mathop {\mathrm{Im}}\nolimits } (b)\) , where \({\mathop {\mathrm{Re}}\nolimits } (a),\,\,{\mathop {\mathrm{Im}}\nolimits } (a)\) represents the real and imaginary part of a, respectively. It is worth noting that the two encoded symbols at the relay can be generated from either pair of transmit symbols from the two terminal nodes. However, antenna and symbol indexes should be appropriately changed accordingly.

3 Proposed STBC-CQ-PNC scheme

In this section, based on the idea of CQ-PNC in [3] we will develop a simple STBC-CQ-PNC scheme. Although the idea of using channel quantization to obtain the PNC symbols at the relay is similar to [3], our derivations of the network coded symbols are supplemented due to increased noise and interference components in the STBC-CQ-PNC system. Our development is further enhanced by an adaptive STBC-CQ-PNC scheme based on minimum residual interference power.

3.1 STBC-CQ-PNC mapping rule

We begin our presentation by expanding (10) as follows:

$$\begin{aligned} w_1&= r_{11} x_1^{(1)} + r_{13} x_1^{(2)} + r_{14} x_2^{(2)} + u_1, \end{aligned}$$
(11)
$$\begin{aligned} w_2&= - r_{11}^* x_2^{(1)} + r_{14}^* x_1^{(2)} - r_{13}^* x_2^{(2)} + u_2, \end{aligned}$$
(12)
$$\begin{aligned} w_3&= r_{33} x_1^{\left( 2 \right) } + u_3, \end{aligned}$$
(13)
$$\begin{aligned} w_4&= - r_{33}^* x_2^{\left( 2 \right) } + u_4. \end{aligned}$$
(14)

Because \(x_1^{(1)} = x_1^{(1)} \,\oplus \, x_2^{(2)} \,\oplus \, x_2^{(2)}\) and \(x_2^{(2)} = x_1^{(1)}\, \oplus \, x_2^{(2)} \,\oplus \, x_1^{(1)}\), the relay can encode the pair of symbols \(x_1^{(1)}\) and \(x_2^{(2)}\) to obtain the network coded symbol \(\overline{x_1^{(1)} \oplus x_2^{(2)} } \) as follows. The method to obtain the other symbol \(\overline{x_2^{(1)} \oplus x_1^{(2)} } \) is similar and thus omitted to simplify our presentation.

Mapping rule for the STBC-CQ-PNC involves the following four steps:

Step 1: Soft estimations of \(x_1^{(2)}\) and \(x_2^{(2)}\).

From the Eqs. (13) and (14), soft estimates of \(x_1^{(2)}\) and \(x_2^{(2)}\) are given by [15] :

$$\begin{aligned} \begin{array}{l} {\hat{x}}_1^{(2)} = \mathrm {tanh} \left( {\left( {r_{33}^* w_3 } \right) /\sigma _n^2 } \right) \\ {\hat{x}}_2^{(2)} = \mathrm {tanh} \left( {\left( { - r_{33} w_4 } \right) /\sigma _n^2 } \right) . \end{array} \end{aligned}$$
(15)

Step 2: Channel quantization.

From Eq. (11), we can see that if \(x_1^{(1)}\) and \(x_2^{(2)}\) are considered the desired signals then \(x_1^{(2)}\) is the interference. Dividing both sides of (11) by the quantization step \(r_{11}\) gives us the quantized value

$$\begin{aligned} c_0 = \frac{{w_1 }}{{r_{11} }} = x_1^{(1)} + \frac{{r_{14} }}{{r_{11} }}x_2^{(2)} + \frac{{r_{13} }}{{r_{11} }}x_1^{(2)} + \frac{{u_1 }}{{r_{11} }}. \end{aligned}$$
(16)

Equation (16) can be rewritten as follows:

$$\begin{aligned} c_0 = x_1^{(1)} + Lx_2^{(2)} + lx_2^{(2)} + \frac{{r_{13} }}{{r_{11} }}\delta _{x_1^{(2)} } + \frac{{u_1 }}{{r_{11} }}, \end{aligned}$$
(17)

where \(\delta _{x_1^{(2)} } \triangleq x_1^{(2)} - {\hat{x}}_1^{(2)}\) is the interference resulted from estimation error of \(x_1^{(2)}\). The values of \(L,\,\,l\) are given by

$$\begin{aligned}&L = L_r + jL_i = \mathrm {round}\left( {\frac{{r_{14} }}{{r_{11} }}} \right) = \mathop {\,\mathrm {arg}\,\mathrm {min} }\limits _{v \in \left( {Z + jZ} \right) } \left\| {\frac{{r_{14} }}{{r_{11} }} - v} \right\| \nonumber \\&l = l_r + jl_i = \frac{{r_{14} }}{{r_{11} }} - L, \end{aligned}$$
(18)

in which \(j^2 = - 1\) and \(\mathrm {round}\left( \cdot \right) \) denotes the rounding operator to a nearest complex integer.

Step 3: Removing residual interference.

For soft network coding based on QR decomposition (referred shortly as Soft QR-NC), the estimate of \({\hat{x}}_2^{(2)}\) in Eq. (15) is used to cancel all \(x_2^{(2)}\) in Eq. (17) to estimate \(x_1^{(1)}\). If \(x_2^{(2)}\) can be estimated perfectly, it does not affect the estimate \(x_1^{(1)}\). Unfortunately, this assumption is difficult to achieve and using an imperfect estimate of \(x_2^{(2)}\) to estimate \(x_1^{(1)}\) can result in an error. To avoid this case, we propose to use the estimate of \(x_2^{(2)}\) to cancel only the residual interference, i.e. \(lx_2^{(2)}\) from Eq. (17). This solution allows reduction in the incurred noise, leading to improved error performance as compared with the traditional Soft QR-NC. After removing residual interference, Eq. (17) becomes

$$\begin{aligned}&c_1 \triangleq x_1^{(1)} + Lx_2^{(2)} + l\left( x_2^{(2)} - {\hat{x}}_2^{(2)} \right) + \frac{{r_{13} }}{{r_{11} }}\delta _{x_1^{(2)} } + \frac{{u_1 }}{{r_{11} }}\nonumber \\&\quad = x_1^{(1)} + Lx_2^{(2)} + l\delta _{x_2^{(2)} } + \frac{{r_{13} }}{{r_{11} }}\delta _{x_1^{(2)} } + \frac{{u_1 }}{{r_{11} }}, \end{aligned}$$
(19)

where \(\delta _{x_2^{(2)} } = x_2^{(2)} - {\hat{x}}_2^{(2)}\) is the interference resulted from estimation error of \(x_2^{(2)}\). Define \(x_m^{(k)} \triangleq 2s_m^{(k)} - (1 + j)\), \((m = 1,2)\) where \(x_m^{(k)} \in \left\{ { \pm 1\,\, \pm j} \right\} \) is the QPSK modulated symbol and \(s_m^{(k)} \in \left\{ {0,\,1,\,j,\,1 + j} \right\} \) is the un-modulated symbol of \(x_m^{(k)} \) at terminal node \(\mathrm {N}_k\). Now similar to [3] \(x_m^{(k)} \) in Eq. (19) is transformed back to its un-modulated symbol as follows:

$$\begin{aligned} {{{\tilde{c}}}_1}&= \frac{1}{2}\left[ {{u_1}\, + \left( {1 + j} \right) \,\, + L\left( {1 + j} \right) } \right] \nonumber \\&=\frac{1}{2}\left[ {\left( {x_1^{(1)} + 1\, + j} \right) + L\left( {x_2^{(2)} + 1 + j} \right) } \right] \nonumber \\&\quad + \frac{l}{2}{\delta _{x_2^{(2)}}} + \frac{{{r_{13}}}}{{2{r_{11}}}}{\delta _{x_1^{(2)}}} + \frac{{{u_1}}}{{2{r_{11}}}}\nonumber \\&= s_1^{(1)} + Ls_2^{(2)} + \frac{l}{2}{\delta _{x_2^{(2)}}} + \frac{{{r_{13}}}}{{2{r_{11}}}}{\delta _{x_1^{(2)}}} + \frac{{{u_1}}}{{2{r_{11}}}}. \end{aligned}$$
(20)

The hard decision of \(s_1^{(1)} + Ls_2^{(2)}\) can be then obtained as:

$$\begin{aligned} {\hat{c}}_1 = \mathrm {round}\left( {{\tilde{c}}_1 } \right) = s_1^{(1)} + Ls_2^{(2)} + e = s_{1,2} + e, \end{aligned}$$
(21)

where \(e = \mathrm {round}\left( {\frac{l}{2}\delta _{x_2^{(2)} } + \frac{{r_{13} }}{{2r_{11} }}\delta _{x_1^{(2)} } + \frac{{u_1 }}{{2r_{11} }}} \right) \) is the detection error due to rounding operation and \(s_{1,2} \triangleq s_1^{(1)} + Ls_2^{(2)} \) is the desired part which needs to be estimated.

Step 4: PNC mapping.

As the hard decision \(\hat{c}_1\) is obtained from \(L_r\) and \(L_i\), the network coded symbol is given for the following 5 cases:

- Case 1: If \(L_r\) is odd and \(L_i\) is even.

In this case, it can be shown that

$$\begin{aligned}&s_{1,2} \,\bmod \,2\nonumber \\&\quad = \left[ {\left( {s_{1,r}^{(1)} + L_r s_{2,r}^{(2)} - L_i s_{2,i}^{(2)} } \right) \,\bmod \,2} \right] \nonumber \\&\qquad + j\left[ {\left( {s_{1,i}^{(1)} + L_r s_{2,i}^{(2)} + L_i s_{2,r}^{(2)} } \right) \,\bmod \,2} \right] \nonumber \\&\quad = \left[ {\left( {s_{1,r}^{(1)} + s_{2,r}^{(2)} } \right) \,\bmod \,2} \right] + j\left[ {\left( {s_{1,i}^{(1)} + s_{2,i}^{(2)} } \right) \,\bmod \,2} \right] \nonumber \\&\quad =s_1^{(1)} \oplus \,s_2^{(2)}, \end{aligned}$$
(22)

where \(s_{k,r}^{(k)} ,\,s_{k,i}^{(k)}\) represents the real and imaginary parts of \(s_t^{(k)} \,\left( {t = 1,2} \right) \), respectively. Because \(x_1^{(1)} \oplus \,x_2^{(2)} = 2( {s_1^{(1)} \oplus \,s_2^{(2)} } ) - ( {1 + j} )\), we can obtain the estimate of the PNC encoded symbol using the decision rule in [3] as follows:

$$\begin{aligned} \overline{x_1^{(1)} \oplus \,x_2^{(2)} } = 2\left( {{\hat{c}}_1 \,\,\bmod \,\,2} \right) - \left( {1 + j} \right) . \end{aligned}$$
(23)

- Case 2: If \(L_r\) is even and \(L_i\) is odd.

Using [3], we can show that

$$\begin{aligned} s_{1,2} \,\bmod \,2&= \left[ {\left( {s_{1,r}^{(1)} + L_r s_{2,r}^{(2)} - L_i s_{2,i}^{(2)} } \right) \,\bmod \,2} \right] \nonumber \\&\quad + j\left[ {\left( {s_{1,i}^{(1)} + L_r s_{2,i}^{(2)} + L_i s_{2,r}^{(2)} } \right) \,\bmod \,2} \right] \nonumber \\&= \left[ {\left( {s_{1,r}^{(1)} + s_{2,i}^{(2)} } \right) \,\bmod \,2} \right] \nonumber \\&\quad + j\left[ {\left( {s_{1,i}^{(1)} + s_{2,r}^{(2)} } \right) \,\bmod \,2} \right] \nonumber \\&= s_1^{(1)} \oplus \,\left( {js_2^{(2)} } \right) . \end{aligned}$$
(24)

Because \(x_1^{(1)} \oplus jx_2^{(2)} = 2\left( {s_1^{(1)} \oplus js_2^{(2)} }\right) - \left( {1 + j} \right) \), we can estimate the PNC encoded symbol as follows:

$$\begin{aligned} \overline{x_1^{(1)} \oplus jx_2^{(2)} } = 2\left( {{\hat{c}}_1 \,\bmod \,2} \right) - \left( {1 + j} \right) . \end{aligned}$$
(25)

- Case 3: If both \(L_r\) and \(L_i\) are even or odd and \(\left| {L_r } \right| > 1\,\) or \(\left| {L_i } \right| > 1\,\).

It is easy to realize that \(\left| L \right| \ge 2\) so the received signal power of \(x_1^{(1)}\) is less than that of the \(x_2^{(2)}\). Therefore, the decision rules for this case are given by

$$\begin{aligned} {\hat{x}}_2^{(2)}&= \mathrm {sign}\left( {\frac{{{\mathop {\mathrm{c}}\nolimits } _1 }}{L}} \right) \end{aligned}$$
(26)
$$\begin{aligned} {\hat{x}}_1^{(1)}&= \mathrm {sign}\left( {{\mathop {\mathrm{c}}\nolimits } _0 - \left( {L + l} \right) {\hat{x}}_2^{(2)} } \right) , \end{aligned}$$
(27)

where

$$\begin{aligned} \mathrm {sign}(x) = \left\{ { \begin{array}{*{20}{c}} {\,1\,\,\,\mathrm{{if}}\,\,x \ge 0}\\ { - 1\,\,\,\mathrm{{if}}\,\,x < 0}\,\,. \end{array}} \right. \end{aligned}$$

Finally, we can obtain the PNC encoded symbol as follows: \(\overline{x_1^{(1)} \oplus \,x_2^{(2)} } = {\hat{x}}_1^{(1)} \oplus {\hat{x}}_2^{(2)}\).

- Case 4: If \(\left| {L_r } \right| = \,\,\left| {L_i } \right| = 1\).

In this case, since \(x_1^{(1)} + \left( {L_r + jL_i } \right) x_2^{(2)}\) cannot be mapped into \( {x_1^{(1)} \oplus \,x_2^{(2)} }\) in the QPSK constellation, we need to map it into a 5-QAM constellation \(\big ( - j3/\sqrt{55\,}, \pm 16/\sqrt{165} - j3\sqrt{55} , \pm 8/\sqrt{165} + j\sqrt{5/11} \big )\) as proposed in [3]. Detailed mapping rule for this special constellation can be found in [3, 16].

- Case 5: If \(L_r = L_i = 0\).

From (11), it is clear that \(\left| {r_{11} }\right| \geqq \left| {r_{14} } \right| \), so network coding can be performed using the following steps:

1. Hard decisions of \(x_1^{(1)}\) and \(x_2^{(1)}\) in Eq. (11):

$$\begin{aligned} {\hat{x}}_1^{(1)}&= \mathrm {sign}\Big ({\frac{{w_1 - r_{13} {\hat{x}}_1^{(2)} - r_{14} {\hat{x}}_2^{(2)} }}{{r_{11} }}} \Big ) \end{aligned}$$
(28)
$$\begin{aligned} {\hat{x}}_2^{(1)}&= \mathrm {sign}\Big ({\frac{{w_2 - r_{14}^* {\hat{x}}_1^{(2)} + r_{13}^* {\hat{x}}_2^{(2)} }}{{ - r_{11}^* }}} \Big ). \end{aligned}$$
(29)

2. Cancellations of \(x_1^{(1)}\) and \(x_2^{(1)}\) in Eq. (11):

$$\begin{aligned} w_1'&= r_{13} x_1^{(2)} + r_{14} x_2^{(2)} + \left( r_{11} x_1^{(1)} - r_{11} {\hat{x}}_1^{(1)} \right) + u_1 \end{aligned}$$
(30)
$$\begin{aligned} w_2'&= r_{14}^* x_1^{(2)} - r_{13}^* x_2^{(2)} - \left( r_{11}^* x_2^{(1)} - r_{11}^* {\hat{x}}_2^{(1)} \right) + u_2. \end{aligned}$$
(31)

3. Hard decisions of \(x_1^{(2)}\) and \(x_2^{(2)}\)

Since the expressions \({\mathop {w}\nolimits } _1'\) in (30) and \({\mathop {w}\nolimits } _2'\) in (31) are equivalent with those of a point-to-point Alamouti STBC system, we can use maximal-ratio combining (MRC) to combine the signals in Eqs. (30), (31) and (13) to obtain better estimates of \({\hat{x}}_2^{(2)}\) and \({\hat{x}}_1^{(2)}\) as follows:

$$\begin{aligned} {\hat{x}}_2^{(2)}&= \mathrm {sign}\left( {\frac{{r_{14}^* w_1' - r_{13} w_2' - r_{33} w_4 }}{{\left| {r_{13} } \right| ^2 + \left| {r_{14} } \right| ^2 + \left| {r_{33} } \right| ^2 }}} \right) , \end{aligned}$$
(32)
$$\begin{aligned} {\hat{x}}_1^{(2)}&= \mathrm {sign}\left( {\frac{{r_{13}^* w_1' + r_{14} w_2' + r_{33}^* w_3 }}{{\left| {r_{13} } \right| ^2 + \left| {r_{14} } \right| ^2 + \left| {r_{33} } \right| ^2 }}} \right) . \end{aligned}$$
(33)

Finally, from (28), (29), (32) and (33), we can obtain the PNC encoded symbols as follows: \(\overline{x_1^{(1)} \oplus \,x_2^{(2)} } = {\hat{x}}_1^{(1)} \oplus {\hat{x}}_2^{(2)}\) and \(\overline{x_2^{(1)} \oplus \,x_1^{(2)} } = {\hat{x}}_2^{(1)} \oplus {\hat{x}}_1^{(2)}\).

Note that although there is a case in which two types of modulation mappings, i.e. QPSK and 5-QAM, may be used by the system it is still possible for the two terminal nodes to switch to the correct mapping for demodulation. To be clear, let us take the value \(L = {L_r} + j{L_i}\) for explanation. Since the channel quantization based on (18) is used it, the receiver knows five possibilities which may occur for this value. However, there is only one case with \(\left| {{L_i}} \right| = \left| {{L_r}} \right| = 1\) in which the system needs to switch from QPSK to 5-QAM constellation. Under the assumption that the channel is symmetric (the forward and backward channels are identical) and that the receiver has knowledge of the CSI using a suitable channel estimator, each terminal node can certainly estimate the value \(L = {L_r} + j{L_i}\). In case \(\left| {{L_i}} \right| = \left| {{L_r}} \right| = 1\), the two terminal nodes will switch between the two modulation constellations.

3.2 The adaptive STBC-CQ-PNC

From the expression of \({\mathop {w}\nolimits } _1\) in (11) we can see that if \(r_{11} x_1^{(1)} + r_{14} x_2^{(2)}\) or \((r_{11} x_1^{(1)} + r_{13} x_1^{(2)})\) is the desired signal, then \(r_{13} x_1^{(2)} + u_1\) or \(r_{14} x_2^{(2)} + u_1\) is interference plus noise component. Moreover, since the \(r_{13}\) and \(r_{14}\) are symmetric in the expression of \({\mathop {w}\nolimits } _1\) and \({\mathop {w}\nolimits } _2\) in (11), at least one suitable pair of PNC symbols from \(\mathrm{{\{ }}x_1^{(1)} \oplus x_2^{(2)} ,\,\,x_2^{(1)} \oplus x_1^{(2)} \mathrm{{\} }}\) and \(\mathrm{{\{ }}x_1^{(1)} \oplus x_1^{(2)} ,\,\,x_2^{(1)} \oplus x_2^{(2)} \mathrm{{\} }}\) should be selected to minimize the undesired interference.

Given the pair \(\big \{x_1^{(1)} \oplus x_2^{(2)} ,\,\,x_2^{(1)} \oplus x_1^{(2)} \big \}\) is selected, it can be seen from (19) that the residual interference is given by: \(l\delta _{x_2^{(2)} } + \frac{{r_{13} }}{{r_{11} }}\delta _{x_1^{(2)} } + \frac{{u_1 }}{{r_{11} }}\). The average power of the interference can be given approximately by [3]

$$\begin{aligned} P_1&\approx E\left\{ {\left| l \right| ^2 } \right\} \frac{{\sigma _n^2 }}{{\sigma _n^2 + r_{33}^2 }} + \frac{{\left| {r_{13} } \right| ^2 }}{{r_{11}^2 }}\frac{{\sigma _n^2 }}{{\sigma _n^2 + r_{33}^2 }} + \frac{{\sigma _n^2 }}{{r_{11}^2 }}\nonumber \\&\approx \frac{{17}}{{44}}\frac{{\sigma _n^2 }}{{\sigma _n^2 + r_{33}^2 }} + \frac{{\left| {r_{13} } \right| ^2 }}{{r_{11}^2 }}\frac{{\sigma _n^2 }}{{\sigma _n^2 + r_{33}^2 }} + \frac{{\sigma _n^2 }}{{r_{11}^2 }}. \end{aligned}$$
(34)

Similarly, when the pair \(\big \{x_1^{(1)} \oplus x_1^{(2)} ,\,\,x_2^{(1)} \oplus x_2^{(2)} \big \}\) is selected, the residual interference is given by \(l\delta _{x_1^{(2)} } + \frac{{r_{14} }}{{r_{11} }}\delta _{x_2^{(2)} } + \frac{{u_1 }}{{r_{11} }}\) and its power is approximated by

$$\begin{aligned} P_2&\approx E\left\{ {\left| l \right| ^2 } \right\} \frac{{\sigma _n^2 }}{{\sigma _n^2 + r_{33}^2 }} + \frac{{\left| {r_{14} } \right| ^2 }}{{r_{11}^2 }}\frac{{\sigma _n^2 }}{{\sigma _n^2 + r_{33}^2 }} + \frac{{\sigma _n^2 }}{{r_{11}^2 }}\nonumber \\&\approx \frac{{17}}{{44}}\frac{{\sigma _n^2 }}{{\sigma _n^2 + r_{33}^2 }} + \frac{{\left| {r_{14} } \right| ^2 }}{{r_{11}^2 }}\frac{{\sigma _n^2 }}{{\sigma _n^2 + r_{33}^2 }} + \frac{{\sigma _n^2 }}{{r_{11}^2 }}. \end{aligned}$$
(35)

It can be seen that \(\mathrm {min} \{ P_1 ,\,P_2 \mathrm{{\} }}\) is equivalent with \(\mathrm {min} \{ \left| {r_{13} } \right| ,\left| {r_{14} } \right| \mathrm{{\} }}\). Therefore, we propose an adaptive detection rule that satisfies the minimum power \(P_{\mathrm {min} } = \mathrm {min} \left\{ {P_1 ,P_2 } \right\} \) as follows

Select:

$$\begin{aligned} \left\{ { \begin{array}{*{20}c} {\mathrm{{\{ }}x_1^{(1)} \oplus x_2^{(2)} ,x_2^{(1)} \oplus x_1^{(2)} \mathrm{{\} }}} \\ \\ {\mathrm{{\{ }}x_1^{(1)} \oplus x_1^{(2)} ,x_2^{(1)} \oplus x_2^{(2)} \mathrm{{\} }}} \\ \end{array}} \right. \,\, \begin{array}{*{20}c} {\text {when}\,\,\left| {r_{13} } \right| \le \left| {r_{14} } \right| } \\ \\ {\text {when}\,\,\left| {r_{13} } \right| > \left| {r_{14} } \right| }. \\ \end{array} \end{aligned}$$
(36)

4 Proposed relay selection based on minimum residual interference power

In this section, we aim to propose a simple relay selection algorithm based on the minimum residual interference power in order to improve the system performance. We assume that there are D potential relays randomly distributed between the two terminal nodes in the network. The network needs to select the best one in terms of minimizing the interference power to serve as the relay.

From Eqs. (19) and (34), we can see that the estimate of the PNC encoded symbol \(\overline{x_1^{(1)} \oplus \,x_2^{(2)} }\) that is mapped from \(x_1^{(1)} + \left( {L_r + jL_i } \right) x_2^{(2)}\) can be better obtained if the residual interference power \(P_1\) is minimized. Based on the channel matrix \({\mathbf {H}}^{(d)}\) (as defined in (7)), each potential relay can calculate the residual interference power associated with its link and use a distributed relay selection algorithm to find the optimal relay.

The relay selection can be now expressed for the simple STBC-CQ-PNC system as follows:

$$\begin{aligned} P_1^{(d)}&\approx \frac{{17}}{{44}}\frac{{\sigma _n^2 }}{{\sigma _n^2 + r_{33}^{(d)2} }} + \frac{{\left| {r_{13}^{(d)} } \right| ^2 }}{{r_{11}^2 }}\frac{{\sigma _n^2 }}{{\sigma _n^2 + r_{33}^{(d)2} }} + \frac{{\sigma _n^2 }}{{r_{11}^{(d)2}}} \end{aligned}$$
(37)
$$\begin{aligned} {\hat{d}}&= \mathop {\,\mathrm {arg}\,\,\mathrm {min} }\limits _{d = 1,2,\ldots ,D} \left( P_1^{(d)}\right) . \end{aligned}$$
(38)

For the adaptive STBC-CQ-PNC system, the relay selection can be expressed as:

$$\begin{aligned}&P_1^{(d)} \approx \frac{{17}}{{44}}\frac{{\sigma _n^2 }}{{\sigma _n^2 + r_{33}^{(d)2} }} + \frac{{\left| {r_{13}^{(d)} } \right| ^2 }}{{r_{11}^2 }}\frac{{\sigma _n^2 }}{{\sigma _n^2 + r_{33}^{(d)2} }} + \frac{{\sigma _n^2 }}{{r_{11}^{(d)2} }} \end{aligned}$$
(39)
$$\begin{aligned}&P_2^{(d)} \approx \frac{{17}}{{44}}\frac{{\sigma _n^2 }}{{\sigma _n^2 + r_{33}^{(d)2} }} + \frac{{\left| {r_{14}^{(d)} } \right| ^2 }}{{r_{11}^2 }}\frac{{\sigma _n^2 }}{{\sigma _n^2 + r_{33}^{(d)2} }} + \frac{{\sigma _n^2 }}{{r_{11}^{(d)2} }} \end{aligned}$$
(40)
$$\begin{aligned}&{\hat{d}} = \mathop {\,\mathrm {arg}\,\,\mathrm {min} }\limits _{d = 1,2,.,D} \left( P_1^{(d)} ,P_2^{(d)}\right) . \end{aligned}$$
(41)

5 Computation complexity analysis

In this section, we estimate the computational complexity of the proposed STBC-CQ-STBC to compare with other related schemes. Since the main processing is done at the relay, we will focus on the complexity associated with this node. The computation complexity is estimated and convert to flops (FLoating-point OPerations). According to [17], in the calculation a complex multiplication requires 6 flops, a complex division 11 flops, both a complex addition and a complex subtraction 2 flops, a complex and real multiplication 2 flops, a complex and real addition 1 flop and finally both a multiplication and an addition of real numbers only 1 flop.

According to [17], the QR decomposition of the channel matrix \({{\mathbf {H}}} \in {\mathbb {C}^{M \times M}}\) needs \((8{M^3} + 6{M^2})\) flops. In the case \({\mathbf {H}}\in {\mathbb {C}}^{4 \times 4}\), this computation complexity is equivalent to 608 flops. Also, computational complexity required for calculations of Eqs. (9) (10) is equivalent to 120 flops and that of (15) to (21) is 50 flops. The remaining calculations equal to 30 flops. As a result, the total computational complexity required by the simple or adaptive STBC-CQ-PNC is 808 flops for each.

6 Performance evaluation

6.1 SER performance

In this section, we investigate the symbol error rate (SER) performance of the proposed STBC-CQ-PNC system. In our simulations, all nodes are equipped with two antennas and use the Alamouti STBC with the QPSK modulation for transmission. The channel is assumed flat slowly-varying Rayleigh fading.

Figure 2 compares the SER performance of the proposed scheme with other related MIMO-STBC schemes including the conventional soft network coding based on QR decomposition (Soft QR-NC), the MIMO-STBC-PNC [11] and the Alamouti two-way relaying scheme in [6]. The proposed two STBC-CQ-PNC schemes are clearly shown to outperform other schemes. Specifically, at \(\mathrm {SER}=10^{-3}\), the adaptive STBC-CQ-PNC scheme obtains an SNR gain of approximately 5.7 dB as compared with the scheme in [6] and 3.8 dB with the soft QR-NC. Compared with MIMO-STBC-PNC in [11], the proposed simple and adaptive STBC-CQ-PNC scheme achieves about 3.5 and 5.0 dB gain, respectively. The performance improvement compared with the MIMO-STBC-PNC in [11] can be attributed to the advantage of channel quantization. Since the MIMO-STBC-PNC scheme in [11] uses linear detection to estimate the sum-and-difference of the pair of transmitted symbols and then use either log-likelihood or selective combination to map them to the PNC encoded symbols, it can be considered a specific case of the STBC-CQ-PNC when \(L=1\) or \(L=-1\). Whereas, in the proposed scheme L is adaptively selected based on the channel coefficients as described in Sect. 3.2.

Fig. 2
figure 2

Performance comparison of the proposed STBC-CQ-PNC scheme with other related MIMO-STBC schemes

Fig. 3
figure 3

Performance comparison of the proposed STBC-CQ-PNC with the CQ-PNC in [3]

Figure 3 presents the SER performance of the proposed STBC-CQ-PNC schemes versus that of the simple CQ-PNC and the adaptive CQ-PNC in [3]. The proposed schemes are shown to achieve some improvement in SNR gain. Specifically, at \(\mathrm {SER}=10^{-3}\) the simple scheme achieves, respectively, about 3.0 and 2.5 dB gain as compared with the simple CQ-PNC and adaptive CQ-PNC in [3] while the adaptive scheme has better performance about 1.0 dB gain than the simple one. More SNR gain can be obtained at lower SER. It is noted that both our proposed adaptive and simple schemes use only a quantization step \(r_{11}\), while the CQ-PNC scheme in [3] uses two adaptive quantization steps \(r_{11}\) and \(\frac{{r_{11} }}{{1 + j}}\).

Fig. 4
figure 4

SER performance of the simple STBC-CQ-PNC system with relay selection

Fig. 5
figure 5

SER performance of the adaptive STBC-CQ-PNC system with relay selection

Table 1 Computational complexity comparison

Figures 4 and 5 illustrate the effectiveness of the proposed relay selection algorithm based on minimum residual interference power. We can see from the figures that the proposed relay selection algorithm improves the SER performance of both the schemes significantly as the number of relays increases. It is also clear that the proposed relay selection algorithm based on residual interference is more efficient than the MSE-based one in [11, 13]. At \(\mathrm {SER} = {10^{-4}}\), the performance gap between the MSE-based and the proposed algorithms is about 5 dB in SNR for both the simple and adaptive STBC-CQ-PNC systems.

6.2 Computational complexity comparison

The computational complexity due to processing at the relay node of all related schemes is tabulated in Table 1. Compared with the previous MIMO-STBC-PNC scheme in [11], our proposed schemes require significantly less complexity. The number of estimated flops is only one third that of the previous scheme. The proposed scheme needs approximately the same number of flops of the scheme in [6] and the soft QR-NC. However, in comparison with the two CQ-PNC schemes in [3], the proposed scheme requires four times more complexity.

7 Conclusions

In this paper, we have proposed an enhanced CQ-PNC scheme for two-way relay STBC systems in which all nodes employ multiple antennas. Thanks to the use of STBC the proposed scheme achieves significant performance improvement due to the inherent diversity gain. We have also proposed an adaptive scheme and relay selection algorithm based on the minimum residual interference power. The simulation results show that our proposed scheme outperforms all related schemes in terms of SER performance but at the cost of additional complexity.