Paper Reference: word2vec Parameter Learning Explained

1. One-word context Model

In our setting, the vocabulary size is $V$, and the hidden layer size is $N$.

The input $x$ is a one-hot representation vector, which means for a given input context word, only one out of $V$ units, $\{x_1,\cdots,x_V\}$, will be 1, and all other units are 0. for example,


The weight between the input layer and the output layer can be represented by a $V \times N$ matrix $W$. Each row of $W$ is the $N$-dimension vector representation $v_w$ of the associated word of the input layer.

Given a context (a word), assuming $x_k=1$ and $x_{k’}=0$ for $k’\neq k$ then


which is just copying the $k$-th row of $W$ to $h$. $v_{w_I}$ is the vector representation of the input word $w_I$. This implies that the link (activation) function of the hidden layer units is simply linear (i.e., directly passing its weighted sum of inputs to the next layer).

From the hidden layer to the output layer, there is a different weight matrix $W’=\{w’_{ij}\}$, which is a $N \times V$ matrix. Using these weights, we can compute a score $u_j$ for each word in the vocabulary,

\[ u_j={v’_{w_j}}^T \cdot h \]

where $v’_{w_j}$ is the $j$-th column of the matrix $W’$. Then we can use the softmax classification model to obtain the posterior distribution of the words, which is a multinomial distribution.


where $y_j$ is the output of the $j$-th unit in the output layer.

Finally, we obtain:

\[p(w_j | w_I) = y_j = \frac{\exp( {v’_{w_o}}^T v_{w_I})}{\sum_{j’=1}^V{\exp( {v’_{w’_j}}^T v_{w_I})}}\]

Note that $v_w$ and $v’_w$ are two representations of the word $w$. $v_w$ comes from rows of $W$, which is the input $\to$ hidden weight matrix, and $v’_w$ comes from columns of $W’$, which is the hidden $\to$ output matrix. In subsequent analysis, we call $v_w$ as the “input vector”, and $v’_w$ as the “output vector” of the word w.

1.2. Cost Function

Let’s derive the weight update equation for this model. Although the actual computation is impractical, we still derivate the update equation to gain insights on the original model without tricks.

The training objective is to maximize the conditional probability of observing the actual output word $w_o$ (denote its corresponding index in the output layer as $j*$) given the input context word $w_I$ with regard to the weights.

\[ \max p(w_o|w_I)=\max y_{j*}\]

which is equivlant to minimize the negative-log probability, where :

\[ E=-\log p(w_o|w_I)=u_{j*}-\log \sum\limits_{j’=1}^V{\exp(u_{j’})}\]

$j*$ is the index of the actual output word in the output layer. Note that, this loss function can be understood as a special case of the cross-entropy measurement between two probabilistic distributions, which has been talked about in previous post:  Negative log-likelihood function.

1.3. Update weight from output $\to$ hidden layer

Let’s derive the update equation of the weights between hidden and output layer $W’_{N\times V}$.

Take the derivative of $E$ with regard to $j$-th unit’s net input $u_j$:

\[ \frac{\partial E}{\partial u_j}= y_j-t_j := e_j \]


\[t_j = \left\{\begin{aligned}1, j=j* \\ 0, j \neq j* \end{aligned}\right. \]

i.e., $t_j$ will only be 1 when the $j$-th is the actual output word, otherwise, $t_j=0$. Note that, this derivative tis simply the prediction error $e_j$ of the output layer.

Next we take the derivative on $w’_{ij}$ to obtain the gradient on the hidden $\to$ output weights $W’_{N \times V}$.

\[  \frac{\partial E}{\partial w’_{ij}}=  \frac{\partial E}{\partial u_j} \cdot  \frac{\partial u_j}{\partial w’_{ij} }= e_j\cdot h_i  \]

Therefore, with SGD, we can obtain the weight update equation for the hidden $\to$ output weight:

\[ w’_{ij} = w’_{ij} – \alpha \cdot e_j \cdot h_i\]

or vector repesentation:

\[ v’_{w_j}=v’_{w_j} – \alpha\cdot e_j \cdot h ~~~~~~j=1,2,\cdots,V\]

where $\alpha$ is the learning rate, $e_j = y_j – t_j$ and $h_i$ is the $i$-th unit in the hidden layer; $v’_{w_j}$ is the output vector of $w_j$.

Note that this update equation implies that we Have To Go Through Every Word In The Vocabulary, check its output probability $y_j$, and compare $y_j$ with its expected output $t_j$ (either 0 or 1).

If $y_j \ge t_j$ (“overestimating”), then we subtract a proportion of the hidden vector $h$ (i.e., $v_{w_I}$) from $v’_{w_o}$, thus making $v’_{w_o}$ far away from $v_{w_I}$; If $y_j \le t_j$ (“underestimating”), we add some $h$ to $v’_{w_o}$, thus making $v’_{w_o}$ closer to $v_{w_I}$. if $y_j \approx t_j$, then according to the update equation, little change will be made to the weights. Note, $v_w$ (input vector) and $v’_w$ (output vector) are two different vector representations of the word $w$.

1.4. Update weight from hidden $\to$ input layer

Having obtained the update equations for $W’$, we now move on to $W$. We take the derivative of $E$ on the output of the hidden layer, obteaining:

\[\frac{\partial E}{\partial h_i}=\sum\limits_{j=1}^V\frac{\partial E}{\partial u_j}\cdot\frac{\partial u_j}{\partial h_i}=\sum\limits_{j=1}^V{e_j\cdot w’_{ij}}:=EH_i\]

where $h_i$ is the output of the $i$-th unit of the hidden layer; $u_j$ is the net input of the $j$-thunit in the output layer; $e_j=y_j-t_j$ is the prediction error of the $j$-th word in the output layer. $EH$, a $N$-dim vector, is the sum of the output vectors of all words in the vocabulary, weighted by their prediction error.

Next, we should take the derivative of $E$ on $W$. First, we recall that the input layer value performs a linear computation to form the hidden layer:

\[h_j=\sum\limits_{k=1}^V{x_k}\cdot w_{ki}\]

Then, we  obtain:

\[\frac{\partial E}{\partial w_{ki}}=\frac{\partial E}{\partial h_i} \cdot \frac{\partial h_i}{\partial w_{ki}}=EH_i \cdot x_k\]

\[\frac{\partial E}{\partial W}=x\cdot EH\]

from which, we obtain a $V\times N$ matrix. since only one component of $x$ is non-zero, only one row of $\frac{\partial E}{\partial W}$ is non-zero, and the value of that row is $EH$, a $N$-dim vector.

We obtain the update equation of $W$ as :

\[v_{w_I}=v_{w_I}-\alpha\cdot EH\]

where, $v_{w_I}$ is a row of $W$, the “input vector”of the only context word, and is the only row of $W$ whose gradient is non-zero. All the other rows of $W$ remains unchanged in this iteration.

Vector $EH$ is the sum of output vectors of all words in the vocabulary weighted by their prediction error $e_j=y_j-t_j$, which is adding a portion of every output vector in  the vocabulary to the input vector of the context word.

As we iteratively update the model parameters by going through context-target word pairs generated from a training corpus, the e ects on the vectors will accumulate. We can imagine that the output vector of a word w is “dragged” back-and-forth by the input vectors of $w$'s co-occurring neighbors, as if there are physical strings between the vector of w and the vectors of its neighbors. Similarly, an input vector can also be considered as being dragged by many output vectors. This interpretation can remind us of gravity, or force-directed graph layout. The equilibrium length of each imaginary string is related to the strength of cooccurrence between the associated pair of words, as well as the learning rate. After many iterations, the relative positions of the input and output vectors will eventually stabilize.

2. Multi-word context Model

Now, we move on to the model with a multi-word context setting.

When computing the hidden layer output, the CBOW model computes the mean value of the inputs:

\[h=\frac{1}{C}W\dot (x_1+x_2+\cdots+x_C)=\frac{1}{C}\cdot (v_{w_1}+v_{w_2}+\cdots+v_{w_C})\]

where $C$ is the number of words in the context, $w_1,\cdots,w_C$ are the words in the context, and $v_w$ is the input vector of a word $w$. The loss function is:

\[E = - \log p(w_o | w_{I,1},\cdots,w_{I,C}) \]


2.2. Update weight from output $\to$ hidden layer

The update equation for the hidden $\to$ output weights stay the same as that for the one-word-context model.

\[ v’_{w_j}=v’_{w_j} – \alpha\cdot e_j \cdot h ~~~~~~j=1,2,\cdots,V\]

Note that we need to apply this to every element of the hidden!output weight matrix for each training instance.

2.3 Update weight from hidden $\to$ input layer

The update equation for input $to$ hidden weights is similar to 1.4, except that now we need to apply the following equation for every word $w_{I,c}$ in the context:

\[ v_{w_{I,c}}=v_{w_{I,c}} –\frac{1}{C}\cdot  \alpha \cdot EH \]

where $w_{w_{I,c}}$ is the input vector of the $c$-th word in the onput context; $\alpha$ is the learning rate; and $EH=\frac{\partial E}{\partial h_i}$.

CBOW Model Formula Deduction的更多相关文章

  1. RBM Formula Deduction

    Energy based Model the probability distribution (softmax function): \[p(x)=\frac{\exp(-E(x))}{\sum\l ...

  2. Logistic Regression - Formula Deduction

    Sigmoid Function \[ \sigma(z)=\frac{1}{1+e^{(-z)}} \] feature: axial symmetry: \[ \sigma(z)+ \sigma( ...

  3. word2vec——高效word特征提取

    继上次分享了经典统计语言模型,最近公众号中有很多做NLP朋友问到了关于word2vec的相关内容, 本文就在这里整理一下做以分享. 本文分为 概括word2vec 相关工作 模型结构 Count-ba ...

  4. 基于pytorch实现word2vec

    一.介绍 word2vec是Google于2013年推出的开源的获取词向量word2vec的工具包.它包括了一组用于word embedding的模型,这些模型通常都是用浅层(两层)神经网络训练词向量 ...

  5. Spark之导出PMML文件(Python)

    PMML,全称预言模型标记语言(Predictive Model Markup Language),利用XML描述和存储数据挖掘模型,是一个已经被W3C所接受的标准.PMML是一种基于XML的语言,用 ...

  6. ISLR系列:(2)分类 Logistic Regression & LDA & QDA & KNN

       Classification 此博文是 An Introduction to Statistical Learning with Applications in R 的系列读书笔记,作为本人的一 ...

  7. cw2vec理论及其实现

    导读 本文对AAAI 2018(Association for the Advancement of Artificial Intelligence 2018)高分录用的一篇中文词向量论文(cw2ve ...

  8. 通过Visualizing Representations来理解Deep Learning、Neural network、以及输入样本自身的高维空间结构

    catalogue . 引言 . Neural Networks Transform Space - 神经网络内部的空间结构 . Understand the data itself by visua ...

  9. [IR] Word Embeddings

    From: Ref: ...


  1. Linux shell相关

    1 一些常见的问题及解决方法 <1> ssh登录不显示用户名跟路径 可能原因: ssh登录的用户没有配置shell 对应解决方案:在/etc/passwd文件对应用户那一行末尾添加/bin ...

  2. Xamarin Android中引用Jar包的方法

    新建一个Java Bingdings Library 将Jar包复制,或使用添加已存在的文件,到Jars文件夹中 确认属性中的“生成操作” 如果有类型转换不正确,请修改Transforms文件夹中的相 ...

  3. Device Tree(三):代码分析【转】

    转自: Device Tree(三):代码分析 作者:linuxer 发布于:201 ...

  4. vim中如何替换

    1) 文件内全部替换:   :%s#abc#123#g (如文件内有#,可用/替换,:%s/abc/123/g)   --注:把abc替换成123   (或者: %s/str1/str2/g 用str ...

  5. MySQL——索引与优化 写在前面:索引对查询的速度有着至关重要的影响,理解索引也是进行数据库性能调 ...

  6. windows下bat批处理实现守护进程(有日志)

    开发部的一个核心程序总是会自己宕机,然后需要手工去起,而这个服务的安全级别又很高,只有我可以操作,搞得我晚上老没法睡,昨晚实在受不了了,想起以前在hp-ux下写的shell守护进程,这回搞个windo ...

  7. python 实现单链表

    #! /usr/bin/env python ### ### Linked List python implementation ### ### @reference Data Structures ...

  8. 【PHP】PHP获得第一章

    一,PHP上部和下部壳体敏感 1)所有的用户定义的函数.类和keyword敏感. 例如以下结果输出一致: echo  "hello world" Echo  "hello ...

  9. Codeforces Round #271 (Div. 2) E题 Pillars(线段树维护DP)

    题目地址: 第一次遇到这样的用线段树来维护DP的题目.ASC中也遇到过,当时也非常自然的想到了线段树维护DP,可是 ...

  10. mybatis-java-依赖注入

    第一种 setXXX形式的注入 我们的所有dao都会继承自定义的BaseDao,因此在BaseDao中完成对所有dao的注入 在DaoUtil中增加方法injectDao()来注入具体的dao pac ...