ArXiv analysis

Run online variational LDA on all the abstracts from the arXiv. The implementation is based on Matt Hoffman's GPL licensed code.

Usage

You'll need a mongod instance running on the port given by the environment variable MONGO_PORT and a redis-server instance running on the port given by the REDIS_PORT environment variable.

The code depends on the Python packages: numpy, scipy, requests, pymongo and redis.

• mkdir abstracts
• ./analysis.py scrape abstracts — scrapes all the metadata from the arXiv OAI interface and saves the raw XML responses as abstracts/raw-*.xml. This takes a long time because of the arXiv's flow control policies. It took me approximately 6 hours.
• ./analysis.py parse abstracts/raw-*.xml — parses the raw responses and saves the abstracts to a MongoDB database called arxiv in the collection called abstracts.
• ./analysis.py build-vocab — counts all the words in the corpus removing anything with less than 3 characters and removing any stop words.
• ./analysis.py get-vocab 100 5000 > vocab.txt — lists the vocabulary skipping the first 100 most popular words and keeping 5000 words total.
• ./analysis.py run vocab.txt — runs online variational LDA by randomly selecting articles from the database. The topic distributions are stored in the lambda-*.txt files. This will run forever so just kill it whenever you feel like it.
• ./analysis.py vocab.txt lambda-100.txt — list the topics and their most common words at step 100.