Assignment 6 — detecting topics.

• DueMar 26 by 12:00pm
• Submittinga text entry box or a file upload

In this assignment, you will perform topic modeling of all of Wikipedia.

Step 1

Complete the gensim tutorial: https://radimrehurek.com/gensim/tutorial.html (Links to an external site.).

As part of the tutorial, you will use Latent Dirichlet Allocation to create a topic model of Wikipedia.

Step 2

Download a dump of Wikipedia. I recommend using this torrent:

http://itorrents.org/torrent/D567CE8E2EC4792A99197FB61DEAEBD70ADD97C0.torrent (Links to an external site.)

You can also download it directly from a mirror. The file you’re looking for is enwiki-20170820-pages-articles.xml.bz2. A more recent dump is okay, just make sure that it’s the English Wikipedia and that you’re getting the bzipped XML of all article pages (-pages-articles.xml.bz2).

NOTE: You may also use the “Simple English Wikipedia” (simplewiki) instead of the full English Wikipedia. The benefit is that it’s ~100 times smaller and so will be much quicker to process. The downside is that it is 100x smaller and thus not quite as good an example of large-scale topic modeling of the kind you might want to do in the future.

Step 3

Run LDA on all of Wikipedia, using 20 topics.

Step 4

View the words that are weighted heavily in each topic and give each topic a name.

Step 5

Visit 3 random Wikipedia pages by following this link three times:

http://en.wikipedia.org/wiki/Special:Randompage (Links to an external site.)

For each page, copy all the text into a text file. Then use the model you have trained to determine that document’s topics.

Step 6

Write 1 page that includes the names of the topics from Step 4, links to the three articles from Step 5, and the names of the topics assigned to those 3 articles. Include at least 1 paragraph describing the results and anything important or interesting you notice.