Word2Vec word embedding tutorial in Python and TensorFlow

gensim word embedding softmax trainer

In coming tutorials on this blog I will be dealing with how to create deep learning models that predict text sequences.  However, before we get to that point we have to understand some key Natural Language Processing (NLP) ideas.  One of the key ideas in NLP is how we can efficiently convert words into numeric vectors which can then be “fed into” various machine learning models to perform predictions.  The current key technique to do this is called “Word2Vec” and this is what will be covered in this tutorial.  After discussing the relevant background material, we will be implementing Word2Vec embedding using TensorFlow (which makes our lives a lot easier).  To get up to speed in TensorFlow, check out my TensorFlow tutorial. Also, if you prefer Keras – check out my Word2Vec Keras tutorial.

Why do we need Word2Vec?

If we want to feed words into machine learning models, unless we are using tree based methods, we need to convert the words into some set of numeric vectors.  A straight-forward way of doing this would be to use a “one-hot” method of converting the word into a sparse representation with only one element of the vector set to 1, the rest being zero.  This is the same method we use for classification tasks – see this tutorial.

So, for the sentence “the cat sat on the mat” we would have the following vector representation:

the \\
cat \\
sat \\
on \\
the \\
mat \\
1 & 0 & 0 & 0 & 0 \\
0 & 1 & 0 & 0 & 0 \\
0 & 0 & 1 & 0 & 0 \\
0 & 0 & 0 & 1 & 0 \\
1 & 0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 & 1

Here we have transformed a six word sentence into a 6×5 matrix, with the 5 being the size of the vocabulary (“the” is repeated).  In practical applications, however, we will want machine and deep learning models to learn from gigantic vocabularies i.e. 10,000 words plus.  You can begin to see the efficiency issue of using “one hot” representations of the words – the input layer into any neural network attempting to model such a vocabulary would have to be at least 10,000 nodes.  Not only that, this method strips away any local context of the words – in other words, it strips away information about words which commonly appear close together in sentences (or between sentences).

For instance, we might expect to see “United” and “States” to appear close together, or “Soviet” and “Union”.  Or “food” and “eat”, and so on.  This method loses all such information, which, if we are trying to model natural language, is a large omission.  Therefore, we need an efficient representation of the text data which also conserves information about local word context.  This is where the Word2Vec methodology comes in.

The Word2Vec methodology

As mentioned previously, there is two components to the Word2Vec methodology.  The first is the mapping of a high dimensional one-hot style representation of words to a lower dimensional vector. This might involve transforming a 10,000 columned matrix into a 300 columned matrix, for instance. This process is called word embedding.  The second goal is to do this while still maintaining word context and therefore, to some extent, meaning. One approach to achieving these two goals in the Word2Vec methodology is by taking an input word and then attempting to estimate the probability of other words appearing close to that word.  This is called the skip-gram approach.  The alternative method, called Continuous Bag Of Words (CBOW), does the opposite – it takes some context words as input and tries to find the single word that has the highest probability of fitting that context.  In this tutorial, we will concentrate on the skip-gram method.

What’s a gram?  A gram is a group of words, where n is the gram window size.  So for the sentence “The cat sat on the mat”, a 3-gram representation of this sentence would be “The cat sat”, “cat sat on”, “sat on the”, “on the mat”.  The “skip” part refers to the number of times an input word is repeated in the data-set with different context words (more on this later).  These grams are fed into the Word2Vec context prediction system. For instance, assume the input word is “cat” – the Word2Vec tries to predict the context (“the”, “sat”) from this supplied input word.  The Word2Vec system will move through all the supplied grams and input words and attempt to learn appropriate mapping vectors (embeddings) which produce high probabilities for the right context given the input words.

What is this Word2Vec prediction system?  Nothing other than a neural network.

The softmax Word2Vec method

Consider the diagram below – in this case we’ll assume the sentence “The cat sat on the mat” is part of a much larger text database, with a very large vocabulary – say 10,000 words in length.  We want to reduce this to a 300 length embedding.

Word2Vec softmax trainer

A Word2Vec softmax trainer

With respect to the diagram above, if we take the word “cat” it will be one of the words in the 10,000 word vocabulary.  Therefore we can represent it as a 10,000 length one-hot vector.  We then interface this input vector to a 300 node hidden layer (if you need to scrub up on neural networks, see this tutorial).  The weights connecting this layer will be our new word vectors – more on this soon.  The activations of the nodes in this hidden layer are simply linear summations of the weighted inputs (i.e. no non-linear activation, like a sigmoid or tanh, is applied).  These nodes are then fed into a softmax output layer.  During training, we want to change the weights of this neural network so that words surrounding “cat” have a higher probability in the softmax output layer.  So, for instance, if our text data set has a lot of Dr Seuss books, we would want our network to assign large probabilities to words like “the”, “sat” and “on” (given lots of sentences like “the cat sat on the mat”).

By training this network, we would be creating a 10,000 x 300 weight matrix connecting the 10,000 length input with the 300 node hidden layer.  Each row in this matrix corresponds to a word in our 10,000 word vocabulary – so we have effectively reduced 10,000 length one-hot vector representations of our words to 300 length vectors.  The weight matrix essentially becomes a look-up or encoding table of our words.  Not only that, but these weight values contain context information due to the way we’ve trained our network.  Once we’ve trained the network, we abandon the softmax layer and just use the 10,000 x 300 weight matrix as our word embedding lookup table.

What does this look like in code?

The softmax Word2Vec method in TensorFlow

As with any machine learning problem, there are two components – the first is getting all the data into a usable format, and the next is actually performing the training, validation and testing.  First I’ll go through how the data can be gathered into a usable format, then we’ll talk about the TensorFlow graph of the model.  Note that the code that I will be going through can be found in its entirety at this site’s Github repository.  In this case, the code is mostly based on the TensorFlow Word2Vec tutorial here with some personal changes.

Preparing the text data

The previously mentioned TensorFlow tutorial has a few functions that take a text database and transform it so that we can extract input words and their associated grams in mini-batches for training the Word2Vec system / embeddings (if you’re not sure what “mini-batch” means, check out this tutorial).  I’ll briefly talk about each of these functions in turn:

[pastacode lang=”python” manual=”def%20maybe_download(filename%2C%20url%2C%20expected_bytes)%3A%0A%20%20%20%20%22%22%22Download%20a%20file%20if%20not%20present%2C%20and%20make%20sure%20it’s%20the%20right%20size.%22%22%22%0A%20%20%20%20if%20not%20os.path.exists(filename)%3A%0A%20%20%20%20%20%20%20%20filename%2C%20_%20%3D%20urllib.request.urlretrieve(url%20%2B%20filename%2C%20filename)%0A%20%20%20%20statinfo%20%3D%20os.stat(filename)%0A%20%20%20%20if%20statinfo.st_size%20%3D%3D%20expected_bytes%3A%0A%20%20%20%20%20%20%20%20print(‘Found%20and%20verified’%2C%20filename)%0A%20%20%20%20else%3A%0A%20%20%20%20%20%20%20%20print(statinfo.st_size)%0A%20%20%20%20%20%20%20%20raise%20Exception(%0A%20%20%20%20%20%20%20%20%20%20%20%20’Failed%20to%20verify%20’%20%2B%20filename%20%2B%20′.%20Can%20you%20get%20to%20it%20with%20a%20browser%3F’)%0A%20%20%20%20return%20filename” message=”” highlight=”” provider=”manual”/]

This function checks to see if the filename already has been downloaded from the supplied url.  If not, it uses the urllib.request Python module which retrieves a file from the given url argument, and downloads the file into the local code directory.  If the file already exists (i.e. os.path.exists(filename) returns true), then the function does not try to download the file again.  Next, the function checks the size of the file and makes sure it lines up with the expected file size, expected_bytes.  If all is well, it returns the filename object which can be used to extract the data from.  To call the function with the data-set we are using in this example, we execute the following code:

[pastacode lang=”python” manual=”url%20%3D%20’http%3A%2F%2Fmattmahoney.net%2Fdc%2F’%0Afilename%20%3D%20maybe_download(‘text8.zip’%2C%20url%2C%2031344016)” message=”” highlight=”” provider=”manual”/]

The next thing we have to do is take the filename object, which points to the downloaded file, and extract the data using the Python zipfile module.

[pastacode lang=”python” manual=”%23%20Read%20the%20data%20into%20a%20list%20of%20strings.%0Adef%20read_data(filename)%3A%0A%20%20%20%20%22%22%22Extract%20the%20first%20file%20enclosed%20in%20a%20zip%20file%20as%20a%20list%20of%20words.%22%22%22%0A%20%20%20%20with%20zipfile.ZipFile(filename)%20as%20f%3A%0A%20%20%20%20%20%20%20%20data%20%3D%20tf.compat.as_str(f.read(f.namelist()%5B0%5D)).split()%0A%20%20%20%20return%20data” message=”” highlight=”” provider=”manual”/]

Using zipfile.ZipFile() to extract the zipped file, we can then use the reader functionality found in this zipfile module.  First, the namelist() function retrieves all the members of the archive – in this case there is only one member, so we access this using the zero index.  Then we use the read() function which reads all the text in the file and pass this through the TensorFlow function as_str which ensures that the text is created as a string data-type.  Finally, we use split() function to create a list with all the words in the text file, separated by white-space characters.  We can see some of the output here:

[pastacode lang=”python” manual=”vocabulary%20%3D%20read_data(filename)%0Aprint(vocabulary%5B%3A7%5D)%0A%5B’anarchism’%2C%20’originated’%2C%20’as’%2C%20’a’%2C%20’term’%2C%20’of’%2C%20’abuse’%5D” message=”” highlight=”” provider=”manual”/]

As you can observe, the returned vocabulary data contains a list of plain English words, ordered as they are in the sentences of the original extracted text file.  Now that we have all the words extracted in a list, we have to do some further processing to enable us to create our skip-gram batch data.  These further steps are:

  1. Extract the top 10,000 most common words to include in our embedding vector
  2. Gather together all the unique words and index them with a unique integer value – this is what is required to create an equivalent one-hot type input for the word.  We’ll use a dictionary to do this
  3. Loop through every word in the dataset (vocabulary variable) and assign it to the unique integer word identified, created in Step 2 above.  This will allow easy lookup / processing of the word data stream

The function which performs all this magic is shown below:

[pastacode lang=”python” manual=”def%20build_dataset(words%2C%20n_words)%3A%0A%20%20%20%20%22%22%22Process%20raw%20inputs%20into%20a%20dataset.%22%22%22%0A%20%20%20%20count%20%3D%20%5B%5B’UNK’%2C%20-1%5D%5D%0A%20%20%20%20count.extend(collections.Counter(words).most_common(n_words%20-%201))%0A%20%20%20%20dictionary%20%3D%20dict()%0A%20%20%20%20for%20word%2C%20_%20in%20count%3A%0A%20%20%20%20%20%20%20%20dictionary%5Bword%5D%20%3D%20len(dictionary)%0A%20%20%20%20data%20%3D%20list()%0A%20%20%20%20unk_count%20%3D%200%0A%20%20%20%20for%20word%20in%20words%3A%0A%20%20%20%20%20%20%20%20if%20word%20in%20dictionary%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20index%20%3D%20dictionary%5Bword%5D%0A%20%20%20%20%20%20%20%20else%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20index%20%3D%200%20%20%23%20dictionary%5B’UNK’%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20unk_count%20%2B%3D%201%0A%20%20%20%20%20%20%20%20data.append(index)%0A%20%20%20%20count%5B0%5D%5B1%5D%20%3D%20unk_count%0A%20%20%20%20reversed_dictionary%20%3D%20dict(zip(dictionary.values()%2C%20dictionary.keys()))%0A%20%20%20%20return%20data%2C%20count%2C%20dictionary%2C%20reversed_dictionary” message=”” highlight=”” provider=”manual”/]

The first step is setting up a “counter” list, which will store the number of times a word is found within the data-set.  Because we are restricting our vocabulary to only 10,000 words, any words not within the top 10,000 most common words will be marked with an “UNK” designation, standing for “unknown”.  The initialized count list is then extended, using the Python collections module and the Counter() class and the associated most_common() function.  These count the number of words in the given argument (words) and then returns the most common words in a list format.

The next part of this function creates a dictionary, called dictionary which is populated by keys corresponding to each unique word.  The value assigned to each unique word key is simply an increasing integer count of the size of the dictionary.  So, for instance, the most common word will receive the value 1, the second most common the value 2, the third most common word the value 3, and so on (the integer 0 is assigned to the ‘UNK’ words).   This step creates a unique integer value for each word within the vocabulary – accomplishing the second step of the process which was defined above.

Next, the function loops through each word in our full words data set – the data set which was output from the read_data() function.  A list called data is created, which will be the same length as words but instead of being a list of individual words, it will instead be a list of integers – with each word now being represented by the unique integer that was assigned to this word in dictionary.  So, for the first sentence of our data-set [‘anarchism’, ‘originated’, ‘as’, ‘a’, ‘term’, ‘of’, ‘abuse’], now looks like this in the data variable: [5242, 3083, 12, 6, 195, 2, 3136].  This part of the function addresses step 3 in the list above.

Finally, the function creates a dictionary called reverse_dictionary that allows us to look up a word based on its unique integer identifier, rather than looking up the identifier based on the word i.e. the original dictionary.  

The final aspect of setting up our data is now to create a data set comprising of our input words and associated grams, which can be used to train our Word2Vec embedding system.  The code to do this is:

[pastacode lang=”python” manual=”data_index%20%3D%200%0A%23%20generate%20batch%20data%0Adef%20generate_batch(data%2C%20batch_size%2C%20num_skips%2C%20skip_window)%3A%0A%20%20%20%20global%20data_index%0A%20%20%20%20assert%20batch_size%20%25%20num_skips%20%3D%3D%200%0A%20%20%20%20assert%20num_skips%20%3C%3D%202%20*%20skip_window%0A%20%20%20%20batch%20%3D%20np.ndarray(shape%3D(batch_size)%2C%20dtype%3Dnp.int32)%0A%20%20%20%20context%20%3D%20np.ndarray(shape%3D(batch_size%2C%201)%2C%20dtype%3Dnp.int32)%0A%20%20%20%20span%20%3D%202%20*%20skip_window%20%2B%201%20%20%23%20%5B%20skip_window%20input_word%20skip_window%20%5D%0A%20%20%20%20buffer%20%3D%20collections.deque(maxlen%3Dspan)%0A%20%20%20%20for%20_%20in%20range(span)%3A%0A%20%20%20%20%20%20%20%20buffer.append(data%5Bdata_index%5D)%0A%20%20%20%20%20%20%20%20data_index%20%3D%20(data_index%20%2B%201)%20%25%20len(data)%0A%20%20%20%20for%20i%20in%20range(batch_size%20%2F%2F%20num_skips)%3A%0A%20%20%20%20%20%20%20%20target%20%3D%20skip_window%20%20%23%20input%20word%20at%20the%20center%20of%20the%20buffer%0A%20%20%20%20%20%20%20%20targets_to_avoid%20%3D%20%5Bskip_window%5D%0A%20%20%20%20%20%20%20%20for%20j%20in%20range(num_skips)%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20while%20target%20in%20targets_to_avoid%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20target%20%3D%20random.randint(0%2C%20span%20-%201)%0A%20%20%20%20%20%20%20%20%20%20%20%20targets_to_avoid.append(target)%0A%20%20%20%20%20%20%20%20%20%20%20%20batch%5Bi%20*%20num_skips%20%2B%20j%5D%20%3D%20buffer%5Bskip_window%5D%20%20%23%20this%20is%20the%20input%20word%0A%20%20%20%20%20%20%20%20%20%20%20%20context%5Bi%20*%20num_skips%20%2B%20j%2C%200%5D%20%3D%20buffer%5Btarget%5D%20%20%23%20these%20are%20the%20context%20words%0A%20%20%20%20%20%20%20%20buffer.append(data%5Bdata_index%5D)%0A%20%20%20%20%20%20%20%20data_index%20%3D%20(data_index%20%2B%201)%20%25%20len(data)%0A%20%20%20%20%23%20Backtrack%20a%20little%20bit%20to%20avoid%20skipping%20words%20in%20the%20end%20of%20a%20batch%0A%20%20%20%20data_index%20%3D%20(data_index%20%2B%20len(data)%20-%20span)%20%25%20len(data)%0A%20%20%20%20return%20batch%2C%20context” message=”” highlight=”” provider=”manual”/]

This function will generate mini-batches to use during our training (again, see here for information on mini-batch training).  These batches will consist of input words (stored in batch) and random associated context words within the gram as the labels to predict (stored in context).  For instance, in the 5-gram “the cat sat on the”, the input word will be center word i.e. “sat” and the context words that will be predicted will be drawn randomly from the remaining words of the gram: [‘the’, ‘cat’, ‘on’, ‘the’].  In this function, the number of words drawn randomly from the surrounding context is defined by the argument num_skips.  The size of the window of context words to draw from around the input word is defined in the argument skip_window – in the example above (“the cat sat on the”), we have a skip window width of 2 around the input word “sat”.

In the function above, first the batch and label outputs are defined as variables of size batch_size.  Then the span size is defined, which is basically the size of the word list that the input word and context samples will be drawn from.  In the example sub-sentence above “the cat sat on the”, the span is 5 = 2 x skip window + 1.  After this a buffer is created:

[pastacode lang=”python” manual=”buffer%20%3D%20collections.deque(maxlen%3Dspan)%0Afor%20_%20in%20range(span)%3A%0A%20%20%20%20buffer.append(data%5Bdata_index%5D)%0A%20%20%20%20data_index%20%3D%20(data_index%20%2B%201)%20%25%20len(data)” message=”” highlight=”” provider=”manual”/]

This buffer will hold a maximum of span elements and will be a kind of moving window of words that samples are drawn from.  Whenever a new word index is added to the buffer, the left most element will drop out of the buffer to allow room for the new word index being added.  The position of the buffer in the input text stream is stored in a global variable data_index which is incremented each time a new word is added to the buffer.  If it gets to the end of the text stream, the “% len(data)” component of the index update will basically reset the count back to zero.

The code below fills out the batch and context variables: 

[pastacode lang=”python” manual=”for%20i%20in%20range(batch_size%20%2F%2F%20num_skips)%3A%0A%20%20%20%20target%20%3D%20skip_window%20%20%23%20input%20word%20at%20the%20center%20of%20the%20buffer%0A%20%20%20%20targets_to_avoid%20%3D%20%5Bskip_window%5D%0A%20%20%20%20for%20j%20in%20range(num_skips)%3A%0A%20%20%20%20%20%20%20%20while%20target%20in%20targets_to_avoid%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20target%20%3D%20random.randint(0%2C%20span%20-%201)%0A%20%20%20%20%20%20%20%20targets_to_avoid.append(target)%0A%20%20%20%20%20%20%20%20batch%5Bi%20*%20num_skips%20%2B%20j%5D%20%3D%20buffer%5Bskip_window%5D%20%20%23%20this%20is%20the%20input%20word%0A%20%20%20%20%20%20%20%20context%5Bi%20*%20num_skips%20%2B%20j%2C%200%5D%20%3D%20buffer%5Btarget%5D%20%20%23%20these%20are%20the%20context%20words%0A%20%20%20%20buffer.append(data%5Bdata_index%5D)%0A%20%20%20%20data_index%20%3D%20(data_index%20%2B%201)%20%25%20len(data)” message=”” highlight=”” provider=”manual”/]

The first “target” word selected is the word at the center of the span of words and is therefore the input word.  Then other words are randomly selected from the span of words, making sure that the input word is not selected as part of the context, and each context word is unique.  The batch variable will feature repeated input words (buffer[skip_window]) which are matched with each context word in context.

The batch and context variables are then returned – and now we have a means of drawing batches of data from the data set.  We are now in a position to create our Word2Vec training code in TensorFlow.  However, before we get to that, we’ll first create a validation data-set that we can use to test how our model is doing.  We do that by measuring the vectors closest together in vector-space, and make sure these words indeed are similar using our knowledge of English.  This will be discussed more in the next section.  However, for now, the code below shows how to grab some random validation words from the most common words in our vocabulary:

[pastacode lang=”python” manual=”%23%20We%20pick%20a%20random%20validation%20set%20to%20sample%20nearest%20neighbors.%20Here%20we%20limit%20the%0A%23%20validation%20samples%20to%20the%20words%20that%20have%20a%20low%20numeric%20ID%2C%20which%20by%0A%23%20construction%20are%20also%20the%20most%20frequent.%0Avalid_size%20%3D%2016%20%20%20%20%20%23%20Random%20set%20of%20words%20to%20evaluate%20similarity%20on.%0Avalid_window%20%3D%20100%20%20%23%20Only%20pick%20dev%20samples%20in%20the%20head%20of%20the%20distribution.%0Avalid_examples%20%3D%20np.random.choice(valid_window%2C%20valid_size%2C%20replace%3DFalse)” message=”” highlight=”” provider=”manual”/]

The code above randomly chooses 16 integers from 0-100 – this corresponds to the integer indexes of the most common 100 words in our text data.  These will be the words we examine to assess how our learning is progressing in associating related words together in the vector-space.  Now, onto creating the TensorFlow model.

Creating the TensorFlow model

For a refresher on TensorFlow, check out this tutorial.  Below I will step through the process of creating our Word2Vec word embeddings in TensorFlow.  What does this involve?  Simply, we need to setup the neural network which I previously presented, with a word embedding matrix acting as the hidden layer and an output softmax layer in TensorFlow.  By training this model, we’ll be learning the best word embedding matrix and therefore we’ll be learning a reduced, context maintaining, mapping of words to vectors.

The first thing to do is set-up some variables which we’ll use later on in the code – the purposes of these variables will become clear as we progress:

[pastacode lang=”python” manual=”batch_size%20%3D%20128%0Aembedding_size%20%3D%20128%20%20%23%20Dimension%20of%20the%20embedding%20vector.%0Askip_window%20%3D%201%20%20%20%20%20%20%20%23%20How%20many%20words%20to%20consider%20left%20and%20right.%0Anum_skips%20%3D%202%20%20%20%20%20%20%20%20%20%23%20How%20many%20times%20to%20reuse%20an%20input%20to%20generate%20a%20context.” message=”” highlight=”” provider=”manual”/]

Next we setup some TensorFlow placeholders that will hold our input words (their integer indexes) and context words which we are trying to predict.  We also need to create a constant to hold our validation set indexes in TensorFlow:

[pastacode lang=”python” manual=”train_inputs%20%3D%20tf.placeholder(tf.int32%2C%20shape%3D%5Bbatch_size%5D)%0Atrain_labels%20%3D%20tf.placeholder(tf.int32%2C%20shape%3D%5Bbatch_size%2C%201%5D)%0Avalid_dataset%20%3D%20tf.constant(valid_examples%2C%20dtype%3Dtf.int32)” message=”” highlight=”” provider=”manual”/]

Next, we need to setup the embedding matrix variable / tensor – this is straight-forward using the TensorFlow embedding_lookup() function, which I’ll explain shortly:

[pastacode lang=”python” manual=”%23%20Look%20up%20embeddings%20for%20inputs.%0Aembeddings%20%3D%20tf.Variable(%0A%20%20%20%20tf.random_uniform(%5Bvocabulary_size%2C%20embedding_size%5D%2C%20-1.0%2C%201.0))%0Aembed%20%3D%20tf.nn.embedding_lookup(embeddings%2C%20train_inputs)” message=”” highlight=”” provider=”manual”/]

The first step in the code above is to create the embeddings variable, which is effectively the weights of the connections to the linear hidden layer.  We initialize the variable with a random uniform distribution between -1.0 to 1.0.  The size of this variable is (vocabulary_size, embedding_size) – the vocabulary_size is the 10,000 words that we have used to setup our data in the previous section.  This is basically our one-hot vector input, where the only element with a value of “1” is the current input word, all the other values are set to “0”.  The second dimension, embedding_size, is our hidden layer size, and is the length of our new, smaller, representation of our words.  We can also think of this tensor as a big lookup table – the rows are each word in our vocabulary, and the columns are our new vector representation of each of these words.  Here’s a simplified example (using dummy values), where vocabulary_size=7 and embedding_size=3:

\begin{array}{c|c c c}
anarchism & 0.5 & 0.1 & -0.1\\
originated & -0.5 & 0.3 & 0.9 \\
as & 0.3 & -0.5 & -0.3 \\
a & 0.7 & 0.2 & -0.3\\
term & 0.8 & 0.1 & -0.1 \\
of & 0.4 & -0.6 & -0.1 \\
abuse & 0.7 & 0.1 & -0.4

As can be observed, “anarchism” (which would actually be represented by a unique integer or one-hot vector) is now expressed as [0.5, 0.1, -0.1].  We can “look up” anarchism by finding its integer index and searching the rows of embeddings to find the embedding vector: [0.5, 0.1, -0.1].

The next line in the code involves the tf.nn.embedding_lookup() function, which is a useful helper function in TensorFlow for this type of task.  Here’s how it works – it takes an input vector of integer indexes – in this case our train_input tensor of training input words, and “looks up” these indexes in the supplied embeddings tensor.  Therefore, this command will return the current embedding vector for each of the supplied input words in the training batch.  The full embedding tensor will be optimized during the training process.

Next we have to create some weights and bias values to connect the output softmax layer, and perform the appropriate multiplication and addition.  This looks like:

[pastacode lang=”python” manual=”%23%20Construct%20the%20variables%20for%20the%20softmax%0Aweights%20%3D%20tf.Variable(tf.truncated_normal(%5Bvocabulary_size%2C%20embedding_size%5D%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20stddev%3D1.0%20%2F%20math.sqrt(embedding_size)))%0Abiases%20%3D%20tf.Variable(tf.zeros(%5Bvocabulary_size%5D))%0Ahidden_out%20%3D%20tf.matmul(embed%2C%20tf.transpose(weights))%20%2B%20biases” message=”” highlight=”” provider=”manual”/]

The weight variable, as it is connecting the hidden layer and the output layer, is of size (out_layer_size, hidden_layer_size) = (vocabulary_size, embedding_size).  The biases, as usual, will only be single dimensional and the size of the output layer.  We then multiply the embedded variable (embed) by the weights and add the bias.  Now we are ready to create a softmax operation and we will use cross entropy loss to optimize the weights, biases and embeddings of the model.  To do this easily, we will use the TensorFlow function softmax_cross_entropy_with_logits().  However, to use this function we first have to convert the context words / integer indices into one-hot vectors.  The code below performs both of these steps, and also adds a gradient descent optimization operation:

[pastacode lang=”python” manual=”%23%20convert%20train_context%20to%20a%20one-hot%20format%0Atrain_one_hot%20%3D%20tf.one_hot(train_context%2C%20vocabulary_size)%0Across_entropy%20%3D%20tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits%3Dhidden_out%2C%20%0A%20%20%20%20labels%3Dtrain_one_hot))%0A%23%20Construct%20the%20SGD%20optimizer%20using%20a%20learning%20rate%20of%201.0.%0Aoptimizer%20%3D%20tf.train.GradientDescentOptimizer(1.0).minimize(cross_entropy)” message=”” highlight=”” provider=”manual”/]

Next, we need to perform our similarity assessments to check on how the model is performing as it trains.  To determine which words are similar to each other, we need to perform some sort of operation that measures the “distances” between the various word embedding vectors for the different words.  In this case, we will use the cosine similarity measure of distance between vectors.  It is defined as:

$$similarity = cos(\theta) = \frac{\textbf{A}\cdot\textbf{B}}{\parallel\textbf{A}\parallel_2 \parallel \textbf{B} \parallel_2}$$

Here the bolded A and B are the two vectors that we are measuring the similarity between.  The double parallel lines with the 2 subscript ($\parallel\textbf{A}\parallel_2$) refers to the L2 norm of the vector.  To get the L2 norm of a vector, you square every dimension of the vector (in this case n=300, the width of our embedding vector), sum up the squared elements then take the square root of the product i.e.:

$$\sqrt{\sum_{i=1}^n A_{i}^2}$$

The best way to calculate the cosine similarity in TensorFlow is to normalize each vector like so:


Then we can simply multiply these normalized vectors together to get the cosine similarity.  We will multiply the validation vectors/words that were discussed earlier with all of the words in our embedding vector, then we can sort in descending order to get those words most similar to our validation words.

First, we calculate the L2 norm of each vector using the tf.square(), tf.reduce_sum() and tf.sqrt() functions to calculate the square, summation and square root of the norm, respectively:

[pastacode lang=”python” manual=”%23%20Compute%20the%20cosine%20similarity%20between%20minibatch%20examples%20and%20all%20embeddings.%0Anorm%20%3D%20tf.sqrt(tf.reduce_sum(tf.square(embeddings)%2C%201%2C%20keep_dims%3DTrue))%0Anormalized_embeddings%20%3D%20embeddings%20%2F%20norm” message=”” highlight=”” provider=”manual”/]

Now we can look up our validation words / vectors using the tf.nn.embedding_lookup() that we discussed earlier:

[pastacode lang=”python” manual=”valid_embeddings%20%3D%20tf.nn.embedding_lookup(%0A%20%20%20%20%20%20normalized_embeddings%2C%20valid_dataset)” message=”” highlight=”” provider=”manual”/]

As before, we are supplying a list of integers (that correspond to our validation vocabulary words) to the embedding_lookup() function, which looks up these rows in the normalized_embeddings tensor, and returns the subset of validation normalized embeddings.  Now that we have the normalized validation tensor, valid_embeddings, we can multiply this by the full normalized vocabulary (normalized_embedding) to finalize our similarity calculation:

[pastacode lang=”python” manual=”similarity%20%3D%20tf.matmul(%0A%20%20%20%20%20%20valid_embeddings%2C%20normalized_embeddings%2C%20transpose_b%3DTrue)” message=”” highlight=”” provider=”manual”/]

This operation will return a (validation_size, vocabulary_size) sized tensor, where each row refers to one of our validation words and the columns refer to the similarity between the validation word and all the other words in the vocabulary.

Running the TensorFlow model

The code below initializes the variables and feeds in each data batch to the training loop, printing the average loss every 2000 iterations.  If this code doesn’t make sense to you, check out my TensorFlow tutorial.

[pastacode lang=”python” manual=”with%20tf.Session(graph%3Dgraph)%20as%20session%3A%0A%20%20%23%20We%20must%20initialize%20all%20variables%20before%20we%20use%20them.%0A%20%20init.run()%0A%20%20print(‘Initialized’)%0A%0A%20%20average_loss%20%3D%200%0A%20%20for%20step%20in%20range(num_steps)%3A%0A%20%20%20%20batch_inputs%2C%20batch_context%20%3D%20generate_batch(data%2C%0A%20%20%20%20%20%20%20%20batch_size%2C%20num_skips%2C%20skip_window)%0A%20%20%20%20feed_dict%20%3D%20%7Btrain_inputs%3A%20batch_inputs%2C%20train_context%3A%20batch_context%7D%0A%0A%20%20%20%20%23%20We%20perform%20one%20update%20step%20by%20evaluating%20the%20optimizer%20op%20(including%20it%0A%20%20%20%20%23%20in%20the%20list%20of%20returned%20values%20for%20session.run()%0A%20%20%20%20_%2C%20loss_val%20%3D%20session.run(%5Boptimizer%2C%20cross_entropy%5D%2C%20feed_dict%3Dfeed_dict)%0A%20%20%20%20average_loss%20%2B%3D%20loss_val%0A%0A%20%20%20%20if%20step%20%25%202000%20%3D%3D%200%3A%0A%20%20%20%20%20%20if%20step%20%3E%200%3A%0A%20%20%20%20%20%20%20%20average_loss%20%2F%3D%202000%0A%20%20%20%20%20%20%23%20The%20average%20loss%20is%20an%20estimate%20of%20the%20loss%20over%20the%20last%202000%20batches.%0A%20%20%20%20%20%20print(‘Average%20loss%20at%20step%20’%2C%20step%2C%20’%3A%20’%2C%20average_loss)%0A%20%20%20%20%20%20average_loss%20%3D%200″ message=”” highlight=”” provider=”manual”/]

Next, we want to print out the words which are most similar to our validation words – we do this by calling the similarity operation we defined above and sorting the results (note, this is only performed every 10,000 iterations as it is computationally expensive):

[pastacode lang=”python” manual=”%23%20Note%20that%20this%20is%20expensive%20(~20%25%20slowdown%20if%20computed%20every%20500%20steps)%0Aif%20step%20%25%2010000%20%3D%3D%200%3A%0A%20%20%20%20sim%20%3D%20similarity.eval()%0A%20%20%20%20for%20i%20in%20range(valid_size)%3A%0A%20%20%20%20%20%20%20%20valid_word%20%3D%20reverse_dictionary%5Bvalid_examples%5Bi%5D%5D%0A%20%20%20%20%20%20%20%20top_k%20%3D%208%20%20%23%20number%20of%20nearest%20neighbors%0A%20%20%20%20%20%20%20%20nearest%20%3D%20(-sim%5Bi%2C%20%3A%5D).argsort()%5B1%3Atop_k%20%2B%201%5D%0A%20%20%20%20%20%20%20%20log_str%20%3D%20’Nearest%20to%20%25s%3A’%20%25%20valid_word%0A%20%20%20%20%20%20%20%20for%20k%20in%20range(top_k)%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20close_word%20%3D%20reverse_dictionary%5Bnearest%5Bk%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20log_str%20%3D%20’%25s%20%25s%2C’%20%25%20(log_str%2C%20close_word)%0A%20%20%20%20%20%20%20%20print(log_str)” message=”” highlight=”” provider=”manual”/]

This function first evaluates the similarity operation, which returns an array of cosine similarity values for each of the validation words.  Then we iterate through each of the validation words, taking the top 8 closest words by using argsort() on the negative of the similarity to arrange the values in descending order.  The code then prints out these 8 closest words so we can monitor how the embedding process is performing.

Finally, after all the training iterations are finished, we can assign the final embeddings to a separate tensor for use later (most likely in some sort of other deep learning or machine learning process):

[pastacode lang=”python” manual=”final_embeddings%20%3D%20normalized_embeddings.eval()” message=”” highlight=”” provider=”manual”/]

So now we’re done – or are we?  The code for this softmax method of Word2Vec is on this site’s Github repository – you could try running it, but I wouldn’t recommend it.  Why?  Because it is seriously slow.

Speeding things up – the “true” Word2Vec method

The fact is, performing softmax evaluations and updating the weights over a 10,000 word output/vocabulary is really slow.  Why’s that?  Consider the softmax definition:

$$P(y = j \mid x) = \frac{e^{x^T w_j}}{\sum_{k=1}^K e^{x^T w_k}}$$

In the context of what we are working on, the softmax function will predict what words have the highest probability of being in the context of the input word.  To determine that probability however, the denominator of the softmax function has to evaluate all the possible context words in the vocabulary.  Therefore, we need 300 x 10,000 = 3M weights, all of which need to be trained for the softmax output.  This slows things down.

There is an alternative, faster scheme called Noise Contrastive Estimation (NCE).  Instead of taking the probability of the context word compared to all of the possible context words in the vocabulary, this method randomly samples 2-20 possible context words and evaluates the probability only from these.  I won’t go into the nitty gritty details here, but suffice to say that this method has been shown to perform well and drastically speeds up the training process.

TensorFlow has helped us out here, and has supplied an NCE loss function that we can use called tf.nn.nce_loss() which we can supply weight and bias variables to.  Using this function, the time to perform 100 training iterations reduced from 25 seconds with the softmax method to less than 1 second using the NCE method.  An awesome improvement!  We replace the softmax lines with the following in our code:

[pastacode lang=”python” manual=”%23%20Construct%20the%20variables%20for%20the%20NCE%20loss%0Ance_weights%20%3D%20tf.Variable(%0A%20%20%20%20%20%20%20%20tf.truncated_normal(%5Bvocabulary_size%2C%20embedding_size%5D%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20stddev%3D1.0%20%2F%20math.sqrt(embedding_size)))%0Ance_biases%20%3D%20tf.Variable(tf.zeros(%5Bvocabulary_size%5D))%0A%0Ance_loss%20%3D%20tf.reduce_mean(%0A%20%20%20%20%20%20%20%20tf.nn.nce_loss(weights%3Dnce_weights%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20biases%3Dnce_biases%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20labels%3Dtrain_context%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20inputs%3Dembed%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20num_sampled%3Dnum_sampled%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20num_classes%3Dvocabulary_size))%0A%0Aoptimizer%20%3D%20tf.train.GradientDescentOptimizer(1.0).minimize(nce_loss)” message=”” highlight=”” provider=”manual”/]

Now we are good to run the code.  You can get the full code here.  As discussed, every 10,000 iterations the code outputs the validation words and the words that the Word2Vec system deems are similar.  Below, you can see the improvement for some selected validation words between the random initialization and at the 50,000 iteration mark:

At the beginning:

Nearest to nine: heterosexual, scholarly, scandal, serves, humor, realized, cave, himself

Nearest to this: contains, alter, numerous, harmonica, nickname, ghana, bogart, marxist

After 10,000 iterations:

Nearest to nine: zero, one, and, coke, in, UNK, the, jpg

Nearest to this: the, a, UNK, killing, meter, afghanistan, ada, indiana

Finally after 50,000 iterations:

Nearest to nine: eight, one, zero, seven, six, two, five, three

Nearest to this: that, the, a, UNK, one, it, he, an

By examining the outputs above, we can first see that the word “nine” becomes increasingly associated with other number words (“eight”, “one”, “seven” etc.).  This makes sense. The word “this”, which acts as a pronoun and definite article in sentences, becomes associated with other pronouns (“he”, “it”) and other definite articles (“the”, “that”, etc.) the more iterations we run.

In summary then, we have learnt how to use the Word2Vec methodology to reduce large one-hot word vectors to much reduced word embedding vectors which preserve the context and meaning of the original words.  These word embedding vectors can then be used as a more efficient and effective input to deep learning techniques which aim to model natural language.  These techniques, such as recurrent neural networks, will be the subject of future posts.