从零构建递归神经网络：仅用NumPy实现

在现代深度学习框架如TensorFlow和PyTorch的帮助下，递归神经网络（RNN）的实现变得更加容易。然而，对于初学者而言，掌握其基本原理和内部机制是至关重要的。本文将详细介绍如何仅使用NumPy库从头开始构建一个RNN，并应用于自然语言处理任务。

### 初始化参数
与传统神经网络不同，RNN具有三个权重参数：输入权重、内部状态权重和输出权重。这些参数需要初始化为随机值。此外，还需要设置词嵌入维度和输出维度。例如，假设词嵌入维度为100，输出维度为80（表示词汇表中的唯一词向量总数）。代码如下：

```python
hidden_dim = 100
output_dim = 80 # 总词汇表中唯一的词数
input_weights = np.random.uniform(0, 1, (hidden_dim, hidden_dim))
internal_state_weights = np.random.uniform(0, 1, (hidden_dim, hidden_dim))
output_weights = np.random.uniform(0, 1, (output_dim, hidden_dim))
```

变量`prev_memory`表示前一时间步的内部状态，其他参数如学习率、序列长度和截断反向传播的时间戳数也需初始化。

### 前向传播
考虑一个简单的句子“I like to play.”，其中每个单词映射到词汇表中的索引。为了展示从输入到输出的过程，我们先随机初始化每个单词的词嵌入。

```python
input_string = [2, 45, 10, 65] # 单词索引列表
embeddings = [] # 存储每个单词的嵌入向量
for i in range(len(input_string)):
x = np.random.randn(hidden_dim, 1)
embeddings.append(x)
```

RNN单元接受输入后，输出下一个最可能出现的单词。训练时，给定第t+1个词作为输出，将第t个词作为输入。计算损失函数所需的输出格式为独热编码（One-Hot）矢量。

### RNN的黑箱计算
有了权重参数和输入输出，接下来进行前向传播的计算。关键公式包括输入权重乘以输入向量、内部状态权重乘以前一层的激活，以及使用tanh激活函数。

```python
def tanh_activation(Z):
return np.tanh(Z)

def softmax_activation(Z):
e_x = np.exp(Z - np.max(Z))
return e_x / e_x.sum(axis=0)

def rnn_forward(input_embedding, input_weights, internal_state_weights, prev_memory, output_weights):
W_frd = np.dot(internal_state_weights, prev_memory)
U_frd = np.dot(input_weights, input_embedding)
sum_s = W_frd + U_frd
ht_activated = tanh_activation(sum_s)
yt_unactivated = np.dot(output_weights, ht_activated)
yt_activated = softmax_activation(yt_unactivated)
return ht_activated, yt_activated
```

### 计算损失函数
损失函数采用交叉熵损失函数，计算公式如下：

```python
def calculate_loss(output_mapper, predicted_output):
total_loss = 0
for y, y_ in zip(output_mapper.values(), predicted_output):
loss = -sum(y[i] * np.log2(y_[i]) for i in range(len(y)))
loss /= float(len(y))
total_loss += loss
return total_loss / float(len(predicted_output))
```

### 反向传播
反向传播通过链式法则计算梯度。对于RNN，需要计算三个梯度：输入权重、内部状态权重和输出权重的梯度。

```python
def delta_cross_entropy(predicted_output, original_t_output):
grad = predicted_output.copy()
for i, l in enumerate(original_t_output):
if l == 1:
grad[i] -= 1
return grad

# 梯度计算函数
def multiplication_backward(weights, x, dz):
gradient_weight = np.dot(dz, x.T)
chain_gradient = np.dot(weights.T, dz)
return gradient_weight, chain_gradient

def add_backward(x1, x2, dz):
dx1 = dz * np.ones_like(x1)
dx2 = dz * np.ones_like(x2)
return dx1, dx2

def tanh_activation_backward(x, top_diff):
output = np.tanh(x)
return (1.0 - np.square(output)) * top_diff

# 单个时间戳的反向传播
def single_backprop(X, input_weights, internal_state_weights, output_weights, ht_activated, dLo, forward_params_t, diff_s, prev_s):
W_frd = forward_params_t[0][0]
U_frd = forward_params_t[0][1]
ht_unactivated = forward_params_t[0][2]
yt_unactivated = forward_params_t[0][3]
dV, dsv = multiplication_backward(output_weights, ht_activated, dLo)
ds = np.add(dsv, diff_s)
dadd = tanh_activation_backward(ht_unactivated, ds)
dmulw, dmulu = add_backward(U_frd, W_frd, dadd)
dW, dprev_s = multiplication_backward(internal_state_weights, prev_s, dmulw)
dU, dx = multiplication_backward(input_weights, X, dmulu)
return dprev_s, dU, dW, dV

# 截断反向传播
def rnn_backprop(embeddings, memory, output_t, dU, dV, dW, bptt_truncate, input_weights, output_weights, internal_state_weights):
T = len(embeddings)
for t in range(T-1, -1, -1):
prev_s_t = np.zeros((hidden_dim, 1))
diff_s = np.zeros((hidden_dim, 1))
predictiOns= memory[f"yt{t}"]
ht_activated = memory[f"ht{t}"]
forward_params_t = memory[f"params{t}"]
dLo = delta_cross_entropy(predictions, output_t[t])
dprev_s, dU_t, dW_t, dV_t = single_backprop(embeddings[t], input_weights, internal_state_weights, output_weights, ht_activated, dLo, forward_params_t, diff_s, prev_s_t)
dU += dU_t
dW += dW_t
dV += dV_t
if t > 0:
for i in range(t-1, max(-1, t-bptt_truncate), -1):
forward_params_t = memory[f"params{i}"]
ht_activated = memory[f"ht{i}"]
prev_s_i = np.zeros((hidden_dim, 1)) if i == 0 else memory[f"ht{prev}"]
dprev_s, dU_i, dW_i, dV_i = single_backprop(embeddings[t], input_weights, internal_state_weights, output_weights, ht_activated, dLo, forward_params_t, dprev_s, prev_s_i)
dU += dU_i
dW += dW_i
dV += dV_i
return dU, dW, dV
```

### 权重更新
使用批量梯度下降法更新权重。

```python
def gd_step(learning_rate, dU, dW, dV, input_weights, internal_state_weights, output_weights):
input_weights -= learning_rate * dU
internal_state_weights -= learning_rate * dW
output_weights -= learning_rate * dV
return input_weights, internal_state_weights, output_weights
```

### 训练过程
完成所有步骤后，可以开始训练神经网络。训练过程中，可以选择静态或动态调整学习率。

```python
def train(T, embeddings, output_t, output_mapper, input_weights, internal_state_weights, output_weights, dU, dW, dV, prev_memory, learning_rate=0.001, nepoch=100, evaluate_loss_after=2):
losses = []
for epoch in range(nepoch):
if epoch % evaluate_loss_after == 0:
output_string, memory = full_forward_prop(T, embeddings, input_weights, internal_state_weights, prev_memory, output_weights)
loss = calculate_loss(output_mapper, output_string)
losses.append(loss)
print(f"{datetime.now().strftime('%Y-%m-%d %H:%M:%S')}: Loss after epoch={epoch}: {loss}")
dU, dW, dV = rnn_backprop(embeddings, memory, output_t, dU, dV, dW, bptt_truncate, input_weights, output_weights, internal_state_weights)
input_weights, internal_state_weights, output_weights = gd_step(learning_rate, dU, dW, dV, input_weights, internal_state_weights, output_weights)
return losses

losses = train(T, embeddings, output_t, output_mapper, input_weights, internal_state_weights, output_weights, dU, dW, dV, prev_memory, learning_rate=0.0001, nepoch=10, evaluate_loss_after=2)
```

恭喜！您已经成功从零构建了一个递归神经网络。接下来，可以进一步探索LSTM和GRU等更高级的架构。