July 25, 2014

# What are the most popular keywords of CRAN packages?

A large proportion of R's power should be attributed to the enormous amount of extension packages. Many packages are published to CRAN.

These packages cover a wide range of fields. In this post, I'll show you how to use R to scrap the titles of all CRAN packages from the web page and find out which keywords are the most popular.

To minimize the efforts, we try best to avoid reinventing the wheels and get some answer as quickly as possible. We only use existing packages to do all the work.

Here is our toolbox that is useful in this task:

First, we equip our R environment with these tools.

library(rvest)
library(rlist)
library(pipeR)


url <- "http://cran.r-project.org/web/packages/available_packages_by_date.html"
page <- html(url)


Now page is a parsed HTML document object that is well structured and is ready to query. Note that we need to get the texts in the third column of the table. Here we use XPath to locate the information we want. Or you can use CSS selector to do the same work.

The following code are written in fluent style with pipeline.

words <- page %>>%
html_node("//tr//td[3]//text()", xpath = TRUE) %>>%
# select the 3rd column
list.map( # map each node to ...
# 1. get the trimmed text in the XML node
XML::xmlValue(.) %>>%
# 2. split the text by non-word-letters
strsplit("[^a-zA-Z]") %>>%
# 3. put everything together in vector
unlist(use.names = FALSE) %>>%
# 4. lower all words
tolower %>>%
# 5. filter words with more than 3 letters to be meaningful
list.filter(nchar(.) > 3L)) %>>%
# put everything in a large character vector
unlist %>>%
# create a table of word count
table %>>%
# sort the table descending
sort(decreasing = TRUE) %>>%
# take out the first 100 elements
# print out the results
print

          data       analysis         models           with      functions
864            718            484            404            371
package     regression     estimation          model          based
336            308            273            249            238
using          tools           from       bayesian         linear
235            225            194            173            169
methods           time      interface   multivariate    statistical
169            168            160            133            124
test    generalized     clustering          tests         series
114            112            105            105            104
inference     statistics         random   distribution      selection
101            101            100             97             96
modeling        spatial      algorithm       multiple     simulation
89             89             87             87             82
mixed         method     likelihood  distributions      modelling
81             78             77             76             73
network           sets classification        mixture       sampling
72             70             68             67             64
effects         robust         sparse       survival       variable
63             63             60             60             60
high        fitting           gene       function   optimization
58             57             57             56             56
graphical        testing       networks          files  nonparametric
55             55             54             52             52
plots         sample    dimensional        genetic          multi
52             52             51             51             51
utilities  visualization implementation        density         matrix
51             51             50             49             49
hierarchical          lasso       learning         markov    correlation
48             48             48             48             47
dynamic           plot     prediction       censored           meta
47             47             47             46             46
45             45             45             44             44
binary         design          least         normal         system
44             44             43             43             43
fast     functional          point       analyses     confidence
42             42             42             41             41
experiments       graphics        objects     population        process
41             41             41             41             41


The work is done, in 12 lines, in only a little more than 2 seconds!

If you want to know more about these packages, please visit their project pages. Hope you can do more amazing things in your work.