Quantitative thoughts

Few weeks ago GitHub announced, that its timeline data is available on bigquery for analysis. Moreover, it offers prizes for the best visualization of the data. Despite my art skills and minimal chances to win beauty contest, I decided to crunch GitHub data and run data analysis.

After initial trial of bigquery service, I found hard to know, what price, if any, I’m going to pay for the service. Hence, I pulled the data (6.5 GB) from bigquery on my machine and further I used my machine for analysis. Bash scripts have been used to clean up and extract necessary data, R for data analysis and visualization and C++ for text extraction.

GitHub dataset is one table, where each row consist of information about repository (i.e. path, date of creation, name, description, programming language, number of forks/watchers and etc.) and action, which was done by user (i.e. username, location, timestamp and etc.).

As a result, we can check how GitHub users actions are spread over time during the day. The X axis on the graph below is labeled with the hours of the day (GMT) and the Y axis represent median values of the actions for each hour. From it, we can make a deduction, that highest load for GitHub can be expected between 15:00 and 17:00 GMT and lowest to be expected between 05:00 and 07:00 GMT. The color of the line indicates how busy was the day based on quantiles: green are calm days (20% of days), blue – normal days (50% quantile) and red are busy days (80% quantile). I should to mention, that auto-correlation or serial correlation is high (70% for following hour), which means, that busy hours tend to be followed by busy hours and calm hours tend to be followed by calm hours. Moreover, busy days tend happen after busy days.

Second graph below shows median of actions divided by weekdays. There is not big surprise – weekends are more slow than weekdays, nevertheless the programmers are slightly less productive on Mondays and Fridays.

The analysis of creation of new repository shows, that the pattern of busy or calm hours remains over the years. This can be attributed to the fact, that majority of the users comes from North America and Europe.
Another hypothesis can be drawn from this information, that number of creation of the new repositories grow exponentially. However, I mind you, that the graph below is biased – most likely, GitHub users update recent projects, consequently more recent projects appeared on timeline. Even though, 2009-2011 years show exponential grow.
The X axis of the graph below is labeled with the hour of the day, the Y axis – log of median values of new repositories.

Following graph shows the number of forks per project (the X axis, log scale) versus number of watchers (the Y axis, log scale). As expected, there is linear correlation between forks and watchers. Even so there is something interesting about outliers, which are below bottom line – the projects, where number of watchers is low, but number of forks is high. These are anomalies and worth to check.

The next thing to do is to look at the repository description. Let’s group the repositories by programming language and count most dominant words in the description. The graph below has C++ word cloud on the left and Java – right . C++ projects are about library, game, simple(?), engine, Arduino. Java is dominated by android, plugin, server, minecraft, spring, maven.

Ruby (left) vs Python(right ):

“Surprise”, “surprise” – R projects (left) are largely about data analysis, however “machine” word, which corresponds to Machine learning is very tiny. Shell (right) is dominated by configuration, managing, git(?).

GitHub dataset includes location field. Unfortunately, the users can enter whatever they want – country, city or leave it empty. Nevertheless, I was able to extract good chunk of actions, where location field has meaningful value.  The video below shows country based users activity, where dark red corresponds to high activity and light red – minor. Only 30 most active countries are included, the rest are grey.
The same pattern persist over the days – activity in Asia increases around midnight, Europe wakes up around 8:00 or 9:00, where America starts around 15:00. Who said, that hackers and programmers work at night?

What else can be done with GitHub dataset? Most repositories have description field, which can be used to find similar projects by implementing tf-idf method. I tried that method and the results are satisfying.

Most of the graphs shown above are reproducible (except word clouds) and the code can be found on GitHub.

There are plenty of data on internet, however it is raw data. Think for a second about public surveillance cameras - useful to check the traffic on the route or busy place, but anything else? What if you want to know how many cars are on the route? How many car were yesterday at the same time? Given so many cars on the route, how much polluted air in the area?
While working on the road map for data dive event, I started to wonder, how feasible is to use data of public surveillance cameras. So I quickly built a pilot project and now I would like to share my experience.

First step – data acquisition. At beginning I was thinking to plug my smartphone somewhere and collect data of the busy route.  Nevertheless, I quickly found surveillance cameras in Vilnius and started to collect images. Run a search and I’m sure, that you will find them in your city:

Here is bash script, which I use to collect images:

#you need full path for crontab
cd /home/git/carCount/img
a=`date +%s`
b=${a}_4.jpg
wget -O $b -q "http://www.sviesoforai.lt/map/camera.aspx?size=full&image=K7742-1.jpg&rnd=0.15417794161476195"

Data preparation. After while you will have enough data to train your machine (for beginning more than 30 images should be O.K.).
How do we train the algorithm? The goal is to identify the cars in a given image. That means, that we have to provide the examples of positive images (clear image of the cars) and negative images (no car, parts of the car and etc.). Important note – we don’t feed whole image, but we cut a chosen image with sliding window (100×100 in my case). 4 examples of positive images:

Meanwhile, it is worth converting each image to portable grey format PGM. For this specific task, we can sacrifice information about the color of the car – it won’t improve prediction. Besides, PGM images can be loaded into R and easily transformed into matrix. Here is bash script, which converts jpg to pgm and slices each image:

#remove image duplicates
find . -maxdepth 1 -type f -exec md5sum {} \;  >test.txt
awk 'a[$1]++ {gsub(/^\*/,"",$2); print "rm ", $2}' test.txt |sh
rm test.txt
 
#convert jpg
 
if [ -d "out" ]; then
        rm -r out
fi
mkdir out
for k in $(ls *.jpg); do convert $k out/$k.pgm; done
 
cd out
mkdir slide
for filename in $(ls *.pgm);
 do 
 
w=`convert $filename -print "%w" /dev/null`
h=`convert $filename -print "%h" /dev/null`
let "ww= $w/100"
let "hh= $h/100"
for((y=150;y<=250;y+=50))
do
for((i=100;i<=400;i+=50))
do
echo "slide/$i.$filename"
let "h_slide=$i"
convert $filename -crop 100x100+$i+$y slide/$y.$i.$filename
done
done
done

Training, predicting, cross validation. Now is time to open R, load 100×100 images from “train/out/slide” directory and train the algorithm. Important note – each image is a matrix, however you have to feed a matrix of all images to learning algorithm (support vector machine in my case). What you have to do is to “unroll” each image matrix into a vector, get 1X10000 vector and build a new matrix, where each row is an image.
Once training is done, load unseen data from “crossval/out/slide” directory and check “result/” directory, where you will find  images of the cars. R script, which does all above:

setwd('/home/git/carCount/')
 
######read positives############
files=list.files('test/pos/')
pos=matrix(nrow=NROW(files),ncol=100*100)
 
for(i in 1:NROW(files))
{
  gray_file=read.pnm(paste('test/pos/',files[i],sep=''))
  pos[i,]=c(gray_file@grey)
}
outcome=vector(length=NROW(files))
outcome[which(outcome!=1)]=1
 
########read negatives#############
files=list.files('test/neg/')
neg=matrix(nrow=NROW(files),ncol=100*100)
 
for(i in 1:NROW(files))
{
  gray_file=read.pnm(paste('test/neg/',files[i],sep=''))
  neg[i,]=c(gray_file@grey)
}
tmp=vector(length=NROW(files))
tmp[which(tmp!=0)]=0
outcome=c(outcome,tmp)
forecast=svm(rbind(pos,neg),outcome)
cross_val=pos[84:90,]
pred=predict(forecast,cross_val,decision.values=TRUE)
 
##########################unseen data######################
files=list.files('crossval/out/slide/')
cross=matrix(nrow=NROW(files),ncol=100*100)
 
for(i in 1:NROW(files))
{
  gray_file=read.pnm(paste('crossval/out/slide/',files[i],sep=''))
  cross[i,]=c(gray_file@grey)
}
pred=predict(forest,cross,decision.values=TRUE)
 
###############copy positives into result directory###############
dir.create('result')
file.copy(paste('crossval/out/slide/',files[which(as.double(pred)>0.6)],sep=''),'result/')

Classified as positive by algorithm:

Classified as negative by algorithm:

Conclusion. It is truly amazing how well algorithm is able to separate wheat from the chaff without additional tuning. Mind you, my impression is biased after so many fails with financial data, which is noisy and good predictions are scarce.
Nevertheless, this project is far away for ideal – it doesn’t take into account weather condition, traffic jams, perspective view, movements of the camera and etc. But I leave this fun for data-dive event.

Fork the code: https://github.com/kafka399/carCount/