Featured

Rhodecode Load Balancing

By Aaron // Information Technology

Here at Darkwind Media, we use Rhodecode (rhodecode.org) as a Mercurial/Git repository browser and manager. Recently, we have noticed drastic slowdowns, taking several seconds to load web pages and several hours to clone some of our larger repos over a local gigabit connection. After some Googling, we found a few mentions of problems similar to ours, but no real solutions. We stumbled on a Google Groups post that described our very problem, and posed a potential solution here. User Chris Mennie posted his results:

With a single server and single thread, the test took on average 370 seconds (3.66 seconds each repo).

With a single server and 10 threads, the test took on average 995 seconds (9.85 seconds each repo).

With a 10 servers and 10 threads (balanced over servers), the test took on average 99 seconds (0.98 seconds each repo).

The results posted seemed very promising. We decided to do some testing and see to what extent increasing Paster instances would benefit us.

The first step to take would be to set up the additional configs that the Paster instances would reference. We decided to run an initial test of five total paster instances, so we created five paster configs; in our case they were named “production<1-5>.ini”. For each instance created, the corresponding config file needed to be modified to host itself on a different port. For each new Paster config, the port and instance_id attributes will need to be incremented (ex. production2.ini would have them set to 8081 and 2, respectively).

Once we created the five instance configs, Rhodecode needed to be started and informed of the new Paster configs. We replaced our existing init.d script with one that used a fixed array to launch the new configs:

 

After restarting Rhodecode with the new init.d script, we were able to access the Paster instances on ports 8080-8084, and could view the new Paster instances as processes on our server. However, we did not want to have to randomly pick which instance a repository should point to, so we decided to use Apache load balancing.

For our case, we modified our existing configuration that created a virtual host on port 80. To enable load balancing, a balancer must be added to the Proxy, balancer addresses must be added and a balancing method must be selected. Our config looks like this:

This will balance all incoming traffic to our Rhodecode server among the five Paster ports corresponding to the instances after a restart of Apache. Now that we’ve shared the configuration tweaks we made to make Rhodecode load balancing possible, I’d like to share some testing results.

First, let’s go over our current configuration and testing environment. Darkwind Media’s Rhodecode server exists on a 12 core Xeon-based machine that currently handles a majority of our internal web and network-related services. The real issue came from our Teamcity Server which has nearly 500 build configurations that pull every few minutes. We have the polling set high as our build are triggered by commits, allowing us to have very up-to-date builds. Unfortunately, this means that the Paster instances are often taken up by this polling (read: Mercurial pulling). For my test, I decided to run a timed clone on a single large repository (total file size after clone was around 16 gigabytes). I decided to run the test using Mercurial (our DVCS of choice) using the uncompressed option as my machine was connected via gigabit LAN to the Rhodecode server.

I ran six total tests; 1, 3, 5, 7,8, and 10 instances of Paster. Below is a table and graph of my results:

Results_tablechart_1

 

The optimal number of Paster instances for us would be between 5 and 9 instances. Since we have added the additional paster instances, our Mercurial interactions with the remote repositories have become much quicker. Clones that used to take over an hour now take a few minutes. We hope that this information can help someone else, so please feel free to let us know of any inaccuracies, problems, or successes at aaron@darkwindmedia.com

Thanks,

–Aaron