Utilities for creating Word2Vec vectors for Dbpedia Entities via a Wikipedia Dump.
Within the release of Word2Vec the Google team released vectors for freebase entities trained on the Wikipedia. These vectors are useful for a variety of tasks.
This Tool will allow you to generate those vectors. Instead of
mids entities will be addressed via
DbpediaIds which correspond to wikipedia article's titles.
Vectors are generated for (i) words appearing inside wikipedia (ii) vectors for topics i.e:
You can download via torrent one of the prebuilt word2vec models:
Using a prebuilt model
- Get python 2.7
- Install gensim:
pip install gensim
- uncompress downloaded model:
tar -xvf model.tar.gz
- Load model in gensim:
from gensim.models import Word2Vec model = Word2Vec.load("path/to/word2vec/en.model") model.similarity('woman', 'man')
The automated Script set up and runs everything on Ubuntu 14.04. For other Platforms check
Going the long way
sudo sh prepare.sh <Locale> PathToOutputFolder. i.e:
sudo sh prepare.sh es_ES /mnt/data/will work on the spanish wikipedia
sudo sh prepare.sh en_US /mnt/data/will work on the english wikipedia
sudo sh prepare.sh da_DA /mnt/data/will work on the danish wikipedia
- Download the latest wikipedia dump for the given language
- Clean the dump, stem it and tokenize it
- Create a
outputFolder, this corpus can be fed to any word2vec tool to generate vectors.
Once you get
wiki2vec.sh pathToCorpus pathToOutputFile <MIN_WORD_COUNT> <VECTOR_SIZE> <WINDOW_SIZE>
this will install all requiered dependencies for Gensim and build word2vec vectors.
wiki2vec.sh corpus output/model.w2c 50 500 10
- Discards words below 50 counts, generate vectors of size 500, and the window size for building the counts of each occurence is 10 words.
prepare.sh script installs:
- Java 7
- Apache Spark
wiki2vec.sh script installs:
Going the long way
- Get sbt
- make sure
JAVA_HOMEis pointing to Java 7
Wikipedia dumps are stored in xml format. This is a difficult format to process in parallel because the xml file has to be streamed getting the articles on the go. A Readable wikipedia Dump is a transformation of the dump such that it is easy to pipeline into tools such as Spark or Hadoop.
Every line in a readable wikipedia dump follows the format:
org.idio.wikipedia.dumps.ReadableWiki gets a
multistreaming-xml.bz2wikipedia dump and outputs a readable wikipedia.
- path to wikipedia dump
- path to output readable wikipedia i.e:
java -Xmx10G -Xms10G -cp org.idio.wikipedia.dumps.ReadableWiki wiki2vec-assembly-1.0.jar path-to-wiki-dump/eswiki-20150105-pages-articles-multistream.xml.bz2 pathTo/output/ReadableWikipedia
Creates a Tokenized corpus which can be fed into tools such as Gensim to create Word2Vec vectors for Dbpedia entities.
- Every Wikipedia link to an article within wiki is replaced by :
if an article's text contains:
[[ Barack Obama | B.O ]] is the president of [[USA]]
is transformed into:
DbpediaID/Barack_Obama B.O is the president of DbpediaID/USA
- Articles are tokenized (At the moment in a very naive way)
Getting a Word2Vec Corpus
- Make sure you got a
- Download Spark : http://d3kbcqa49mib13.cloudfront.net/spark-1.2.0-bin-hadoop2.4.tgz
- In your Spark folder do:
bin/spark-submit --master local[*] --executor-memory 1g --class "org.idio.wikipedia.word2vec.Word2VecCorpus" target/scala-2.10/wiki2vec-assembly-1.0.jar /PathToYourReadableWiki/readableWiki.lines /Path/To/RedirectsFile /PathToOut/Word2vecReadyWikipediaCorpus
- Feed your corpus to a word2vec tool
By default the word2vec corpus is always stemmed. If you don't want that to happen:
If using the automated scripts..
pass None as an extra argument
sudo sh prepare.sh es_ES /mnt/data/ None will work on the spanish wikipedia and won't stem words
If you are manually running the tools:
Pass None as an extra argument when calling spark
bin/spark-submit --class "org.idio.wikipedia.word2vec.Word2VecCorpus" target/scala-2.10/wiki2vec-assembly-1.0.jar /PathToYourReadableWiki/readableWiki.lines /Path/To/RedirectsFile /PathToOut/Word2vecReadyWikipediaCorpus None
- DeepLearning4j: Feb 2014, Gets stuck in infinite loops on a big corpus
- Spark's word2vec: Feb 2014,
number of dimensions*
vocabulary sizehas to be less than a certain value otherwise an exception is thrown. issue
- Remove hard coded spark params
- Handle Wikipedia Redirections
- Intra Article co-reference resolution