and Me

It’s been a month since I have been contributing on My project is to import endpoints used by bundler to install gems to  Did you know that

Fetching gem metadata from

is kind of a lie? It is actually talking to and that is a sinatra app (bundler-api) different from rails app of didn’t have the infrastructure to handle all the requests coming from all the Rubiest in the world running bundle install  that is why  dependency endpoint used by bundler had to be served from a different app. Now with the help of Ruby Together we have the resources, so we would really like it if we don’t have two applications to maintain. It is so not fun when people can’t find new version releases.

My project got a jump start because #1225 was merged. I only had to make a few changes so that the response matched the one of bundler-api. In the process, I removed some of the things we weren’t using anymore: rubyforgers and version_histories tables and downloads column from rubygems table. While the former two had not been used for a while, the downloads column had gone out of service fairly recently. The way tracks downloads of gems is rather interesting. If you ask me, how I would track number of downloads, I would probably just say that I would keep a field count for it and increment it every time someone downloaded a gem. Given that bundle-api handles 5-7k requests per minute, that won’t be a good idea. I know the request stats because Nick (@qrush)  gave me access to New Relic app of rubygems ❤ Thanks Nick! Data is beautiful, indeed.

Screenshot from 2016-05-28 22-59-36

What does is that, it updates downloads in bulk. Gems are served over Fastly CDNs, and process the log files generated by them every minute (For details checkout: Simplifying our stack). There is more… We track downloads by version. So, how would you find out all the downloads of a gem? I would probably say that I would sum the downloads of  all the versions of a gem with an activerecord query. What does is that, during the bulk update it also updates the version_id: 0 row with:

increment(count, rubygem_id: rubygem_id, version_id: 0)

Now, we don’t have to sum over 709 versions of caboose-cms and it is simple fetch of a row. Following the same pattern, it keeps track of the total count of all the gem downloads with:

increment(total_count, rubygem_id: 0, version_id: 0)

My contribution related to all this was that now you won’t see show all versions link below downloads stats on gem show page.

Screenshot from 2016-05-28 22-00-45

I guess I haven’t made any significant change so far. David (@dwradcliffe) has suggested that I pick up metadata migration. That has me excited! I will be changing the way people build gems.. at least I will be writing the code that will make it happen. We also have plans of adding features to our search system. We are using elasticsearch on, I am pretty sure it can do much more than basic search we have right now.

Lastly, I would suggest that you should checkout upsert. I got it know about it through Arthur (@arthurnn), SQL seems to be his thing. It was introduced only in pg 9.5 and I think it’s really cool. I can’t wait until we use it on

Advertisements and Me

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s