这是机器学习中的神经网络第七讲的作业，有一些不太容易理解的部分
-
How many bits of information can be modeled by the hidden state (at some specific time) of a Hidden Markov Model with 16 hidden units?
- 4
- 2
- 16
- > 16
-
This question is about speech recognition. To accurately recognize what phoneme is being spoken at a particular time, one needs to know the sound data from 100ms before that time to 100ms after that time, i.e. a total of 200ms of sound data. Which of the following setups have access to enough sound data to recognize what phoneme was being spoken 100ms into the past?
- A Recurrent Neural Network (RNN) with 30ms of input <u>Correct RNN can reliably memorize long term history due to its temporal connections between hidden states.</u>
- A Recurrent Neural Network (RNN) with 200ms of input <u>Correct</u>
- A feed forward Neural Network with 30ms of input
- A feed forward Neural Network with 200ms of input <u>Correct A feed forward Neural Network can be used only in case it is given all the input it needs for the recognition task because it doesn't store the input history.</u>
-
The figure below shows a Recurrent Neural Network (RNN) with one input unit x, one logistic hidden unit h, and one linear output unity. The RNN is unrolled in time for T=0,1, and 2.img
The network parameters are: $$W_{xh}$$=0.5 , $$W_{hh}$$=−1.0 , $$W_{hy}$$=−0.7 , $$h_{bias}$$=−1.0, and $$y_{bias}$$=0.0. Remember, σ(k)=11+exp(−k).If the input x takes the values 9,4,−2 at time steps 0,1,2 respectively, what is the value of the hidden state h at T=2? Give your answer with at least two digits after the decimal point.
Correct Response: 0.075
Solve:
-
The figure below shows a Recurrent Neural Network (RNN) with one input unit x, one logistic hidden unit h, and one linear output unity. The RNN is unrolled in time for T=0,1, and 2.img
The network parameters are: $$W_{xh}$$=−0.1 , $$W_{hh}$$=0.5 , $$W_{hy}$$=0.25 , $$h_{bias}$$=0.4, and $$y_{bias}$$=0.0. If the input x takes the values 18,9,−8 at time steps 0,1,2 respectively, the hidden unit values will be 0.2,0.4,0.8 and the output unit values will be 0.05,0.1,0.2 (you can check these values as an exercise). A variable z is defined as the total input to the hidden unit before the logistic nonlinearity.If we are using the squared loss, with targets t0,t1,t2, then the sequence of calculations required to compute the total error $$E$$ is as follows:
...
If the target output values are t0=0.1,t1=−0.1,t2=−0.2 and the squared error loss is used, what is the value of the error derivative just before the hidden unit nonlinearity at T=1 (i.e. ∂E∂z1)? Write your answer up to at least the fourth decimal place.
Correct Response: 0.01392
Solve:
-
Consider a Recurrent Neural Network with one input unit, one logistic hidden unit, and one linear output unit. This RNN is for modeling sequences oflength 4 only, and the output unit exists only at the last time step, i.e. T=3. This diagram shows the RNN unrolled in time:img
Suppose that the model has learned the following parameter values:$W_{xh}=1$ $W_{hh}=−2$ $W_{hy}=1$All biases are 0For one specific training case, the input is 1 at T=0 and 0 at T=1, T=2, and T=3. The target output (at T=3) is 0.5, and we're using the squared errorloss function.We're interested in the gradient for wxh, i.e. ∂E∂wxh. Because it's only at T=0 that the input is not zero, and it's only at T=3 that there's an output, theerror needs to be backpropagated from T=3 to T=0, and that's the kind of situations where RNN's often get either vanishing gradients or explodinggradients. Which of those two occurs in this situation? <u>You can either do the calculations, and find the answer that way, or you can find the answer with more thinking and less math, by thinking about theslope ∂y∂z of the logistic function, and what role that plays in the backpropagation process.</u>
Vanishing gradient Correct
-
Exploding gradient
Solve:
- T = 0
$$
Z_0 = W_{xh}X_0
$$$$
h_0 = \sigma(Z_0)
$$$$
\frac{\partial h_0}{W_{xh}} = \frac{\partial h_0}{\partial Z_0} * \frac{\partial Z_0}{W_{xh}} = \sigma(Z_0)(1 - \sigma(Z_0))X_0
$$- T = 1
$$
Z_1 = W_{xh}X_1 + W_{hh}h_0
$$$$
h_1 = \sigma(Z_1)
$$
$$
\frac{\partial h_1}{W_{xh}} = \frac{\partial h_1}{\partial Z_1} * \frac{\partial Z_1}{W_{xh}} = \sigma(Z_1)(1 - \sigma(Z_1))(X_1+W_{hh}\frac{\partial h_0}{W_{xh}})
$$
- T = 2
$$
\frac {\partial h_2}{W_{xh}} = \frac {\partial h_2} {\partial Z_2} * \frac {\partial Z_2} {W_{xh}} = \sigma(Z_2)(1-\sigma(Z_2))(X_2 + W_{hh}\frac{\partial h_1}{W_{xh}})
$$
- T = 3
$$
\frac{\partial h_3}{W_{xh}} = \frac{\partial h_3}{\partial Z_3} * \frac{\partial Z_3}{W_{xh}} = \sigma(Z_3)(1 - \sigma(Z_3))(X_3+W_{hh}\frac{\partial h_3}{W_{xh}})
$$
$$
E = \frac {1} {2} (t-y)^2, y = W_{xy} * h_3
$$
$$
\frac{\partial E}{\partial W_{xh}} =\frac{\partial E}{\partial y} * \frac{\partial y}{\partial h_3} * \frac{\partial h_3}{\partial W_{xh}} = (y-t)W_{xy}*\frac{\partial h_3}{W_{xh}}
$$
主要的部分是$Z_3$的导数，也就是Sigmoid的导数，也就是在两极会趋向平缓，也就是梯度消失。此外，看这几项都是连乘的小数，很容易变成0，看一下第4题就知道了。
-
Consider the following Recurrent Neural Network (RNN):img
As you can see, the RNN has two input units, two hidden units, and one output unit.For this question, every arrow denotes the effect of a variable at time t on a variable at time t+1.Which feed forward Neural Network is equivalent to this network unrolled in time?
- img
-
imgCorrect
- img
The Second answer is correct
