Yoshua Bengio, et al.:A Neural Probabilistic Language Model

Link: word2vec 中的数学原理详解（三）背景知识

In this paper, the author introduce how to use deep learning neural network to predict the word in sentence and compare it with the traditional n-gram method. The goal of nlp is to learn the probabilistic model to describe how words and sentences are generated. We want to know that in a document, what word will come next after a specific sequence of words. This can help computer better understand the human language.

The traditional method is called n-gram, which model the human language under the naive bayes assumption. The probability of a specific T sequence words are model as the product of consecutive bayes conditional probability. The probability is approximate to the frequency of the sequence of word in a corpora. However, this need a huge amount of count. If the T is very large, the count procedure will cost a lost. So a modified n-gram is proposed to reduce this. When consider bayes conditional probability, we only consider (n-1) word before the word we going to predict. Usually, n is chosen to 3. The drawbacks of this n-gram method are: (1) it don’t consider context farther than 3. (2) it don’t consider the potential relationship under the words. In nature language, context words can up to 10 or more. Also, latent semantic is very prevail among words. When counting word sequence in the corpora, we cannot ignore similar sentence but different words. The traditional way is use word vector to represent each word. But this is not effective and when corpora is very large, might introduce the curse of dimensionality, which means the vector to represent each word has too big dimension and not computable. In this paper, this two issues are solved by using distributed word feature vector and the neural network.

The distributed word feature vector provide a vector space to contain all the words in the corpora. It has m features (elements) which is specific by user and depended on the size of the corpora. Usually, the m is 60, which means each word can be described by using 60 parameters. And between similar meaning words, the vector degree is small. So when consider n words sentence, we concreate the each words vector together to form a n*m sentence vector. For similar sentences, the sentences’ vectors should have small degree. This is how we map the nature word to the input data of the neural network.

The basic idea of machine learning or the neural network is: first model the problem we want to solve. The model is control by a set of parameters. Then find the loss function or the likelihood function which describe how far is the prediction from the distribution's expectation. Then driven this evaluation function with the training data and the corresponding labels. So now the problem turn out to be to find the parameter make the evaluation function have the max/min value. Finally, use gradient descent or ascent to find a set of parameter satisfied this as the model parameters. Then we can use this model to predict in our problem.

In this paper, they use neural network to model this problem. They use a five layer recurrent neural network. The first layer is the input layer which do nothing just input (n-1) nature language words into the net. This also use the modified bayes model. The second layer is a hidden layer and also called word feature layer in the paper. This layer convert each input nature word to the m-feature vector space. Each words is map to a m-dimension vector C. Notice, this C is not pre-defined. It is also considered as a set of parameter in the net. We need to train the model to get C for each word. The next is a hidden layer, which map the concatenation of the (n-1)’s vector C to a row vector x and then use (d+Hx) to model this sentence vector. D is the layer bias which has the dimension of the neural number h in this layer. H is the weight of this hidden layer. The next layer is the output layer. This layer basic apply hyperbolic tangent to the output of the last layer and weighted with U, which is a |V|*h weight matrix. Also have a output bias of b which is |V| dimension. And a direct connection weight W which I will discuss at last. So the output of the output layer is a non-normalized probabilistic vector in respect to the |V| words in the corpora. The output function is: y = b + Wx + U tanh(d + Hx). Before we pick the final prediction, we need to normalized the probabilistic with the softmax function at the softmax layer and pick the largest p as our prediction. This is how to predict with neural network and also called forward phase.

After we define our model, we need to tranning the label data to get all of our parameters. We have PARAM=|V|(1+nm+h)+h(1+(n-1)m) free parameter need to specific. Now, let’s define the likelihood function to describe how confident we are about our prediction: L = 1/T∑log f(w_t, w_t-1...w_t-n ; theta) + R(theta). The method to find the theta make the L get max value is by using gradient ascent. However, at here, if we directly use gradient ascent, this will turn out to be a PARAM dimension problem which is hard to solve when PARAM is large. So we divide the model into layers and each time only train the parameters within a layer. This is how neural nets reduce the computation of traditional machine learning problems. BTW, R(theta) is the weight decay penalty which restrict the gradient ascent searching area after every epoch and don’t make it go too far and cannot converge.

To train this kind of net, we still have a very large computation. So the author come out with the parameter-parallel processing to train and predict the model. Suppose we have M CPU and |V| words in the corpora. So the output will be a |V| dimensional vector. We instead of divide the input data into subsets, they divide the output into subsets and every CPU take charge of |V|/M output units. The forward phase is basicly like what I describe above but each CPU calculate |V|/M outputs and merge together then softmax. What is most interesting is how they train the model with the parameter parallel processing.

The training process is also called Backward/Update phase. They train the model from the back layer to the front layer and each time only train parameters within one layer. The learning rate (step) is e. First, the train the output layer. The parameters in the output layers are (b, W, U) and I still ignore the direct connection weight W. So the (b, U) can be represented by using a. Then for each CPU, use gradient ascent to find the a which can make the L has the maximum value. For b, since it is a constant bias, we compute partial derivative of the L to y_i to update b and start from the initial b. Y_i represent the input variable to the L in this layer. Eventually, we can get a b make L largest. The do the similar gradient ascent to U, but multiply with a. After all M CPU computation is done, we get M set of  a=(b, U) as every parameter’s best solution. This layer parameter training has finished. The only information will pass to the next training layer, hidden layer with (d, H), is the partial derivative of L to a. In this hidden layer, we have h neurals so we use the back-propagation to get the partial derivative of L to o. O is (d+Hx) represent the input variables in this layer to the L function. Then still use gradient ascent to get d and H for each neural. Also update the partial derivative of L to x and pass to the next layer’s training. Then in the word feature layer, use gradient ascent to get C for each input word. Then we get distributed representation vector of (n-1) input word. We should training the whole corpora to get the C for each word. And when prediction, we can use them as the word feature layer’s parameter. Also, they use a method to avoid the exp in softmax is too much to compute.

The W is called direct connection weight, which means ignore the hidden layer and directly map the input C to the output layer and then use softmax to normalize. This might reduce the iteration converge times but not improve the model’s performance.

From this paper, I learn how the neural net are used in nlp problems and have a better understanding of the neural nets. I am very interested in this topic and will study further inside.

Tomas Mikolov, et al.:Distributed Representations of Words and Phrases and their Compositionality

Link: 深度学习word2vec笔记之算法篇, 文本深度表示模型Word2Vec, Vector Representations of Words - TensorFlow

In this paper, the author introduces new model to analysis the nature language called word2vec. The word2vec has two model: CBOW and the Skip-gram model. CBOW is aim at predicting the next word based on the context. The Skip-gram model is aim at predicting a word around the target word. They are all based on the Huffman tree which can reduce the overall computation a lot. The paper also introduce the technique called subsampling and negative sampling which can reduce computation and the phrase based model.

The Skip-gram can be understanded as a 3 layer neural net. The input layer is just word. The hidden layer is map the word to the word vector. The output layer is basically a Huffman tree. All the words in the corpora are at the leaf and the node is only a word vector which can be understanded as a classifier of the word. Each word confront a node will be put into either left or the right path. This can roughly distinguish the word. And after pass through all the level in the Huffman tree, finally arrive at the specific word as final predict. The Skip-gram doesn’t consider the sequence of the word in a sentence, only consider the distance toward the target word. First, the author get the likelihood function. When training the parameters, the goal is to maximum the likelihood function. The likelihood function consider the word within c radius. Then the traditional method is model the p(W_0|W_i) by using the softmax and compute the degree between the two word vector. However, this is found not practical when corpora is large, so the author introduce the method below.

The first method is using the Hierarchical softmax. This make the output layer look like a tree instead of a vector with |V| elements. The tree is obtained during the training process and the most frequently use words are put at the leftmost leaf. You can also put at right most leaf, here word2vec use leftmost. So then the p(W_0|W_i) can be understand as the possibility of from the root to the word W_0. At each node, the input W_i can compute the degree between the W_i and the node vector. If the path from the root to the W_0 require the walk to goto the right child node, then the possibility is set negative. Because that the most frequently used words are located leftmost. This is kind of penalty on the possibility. The degrees are map by the sigmoid function. By using this way, the computation amount is only proportional to the log(|V|). This reduce the computation a lot.

Another way to reduce it is using the so-called negative sampling. This use the technique called Noise contrastive Estimation. This method assume that a good model should be able to differentiate data from noise by the mean of the logistic regression. So other than the computation of teh degree between W_0 and W_i, this also randomly generate k noise word vectors which is not belong to this corpora. The degree between the noise vector and the target word should be very large, so that the sigmoid function can map this penalty to almost 0. By using this method, each prediction’s computation only consume (k+1).

Also the paper talk about the subsampling which can drop some frequent use but less meaningful word such as “a” and “the”. In the the subsampling, every word has a chance to be dropped and it is related to the frequence of the word. Finally the author talk about the phrase model. They think phrase model sometimes is more reliable than the word model. Since some phrase combination might be rare to see but the phrase itself is frequently used. They evaluated two words’ possibility of form a phrase using the basic count method. Here they only consider the bigram case.

One of the most interesting thing in the paper is the word vector calculation. They find that two set of word with same kind of relationship can be mapped to another by using a similar transfer function. Such that “Madrid”-”Spain”+”France”~”Paris”. This indicate that the distance between same relationship set of word might be same. This reveal an interesting way of studying nlp.