A mainstream QuadCore 64bits server with 4GB Ram. Nothing fancy at all, just what you can purchase with about $500.
100 desktop computers like the ones your team can use, all of them running Linux. They're quite heterogeneous: from single core 5 years old machines to newer 4 cores, from 1.5 Gb RAM to 4Gb.
Server and clients are connected by a simple 100Mbps network.
We tested with a variety of different repositories, from really small ones to larger ones.
The one I'm describing today is just a small one (and I can tell you results only get worse for the slow SCMs with more data...):
In order to automate all the client machines we used PNUnit, you know, the extension we made to NUnit for "parallel" testing. Quite useful for load testing.
Test scenario 1 - working on trunk
A really simple scenario every developer is familiar with: just commit changes to the main branch.
Every client will do the following:
Test scenario 1 - working on trunk - results
Ok, how our beautiful friends behave under really heavy load? Considering we tested with Subversion and Perforce, 2 of the most used version controls on the market, we expected high scalability... :)
We used SVN 1.5.7, Perforce 2009.2 64bits and Plastic SCM 3.0.
All results are using a Windows server and Linux clients, except for Subversion: we run the SVN server on Linux (dual boot server machine) because on Windows it couldn't handle more than 30 concurrent clients without consistently crashing (out of memory, 4Gb!!! and gone).
We run the same test described above with 1 client, 10, 20, 50 and 100. Check the results here:
The two old irons doesn't scale that well at all, uh? ;-)
Plastic is using a SQL Server backend and it seems it can handle the load much better than the others, even doing trunk development.
Test scenario 2 - working on branches
The second scenario tries to reproduce a "branch per task" pattern, something we strongly recommend with Plastic.
The scenario is as follows:
Test scenario 2 - working on branches - results
We always say most of the version control systems out there are not ready to handle branching, and we always hear people asking why.
Ok, a picture is worth a thousand words.
If you miss some data point in one of the version control systems compared is not because of a mistake, the reason is that the server simply starts locking too much, rejecting clients and making the test fail (even considering that the test is able to handle retries if it gets rejection errors).
I'll be sharing the data regarding the Plastic server running on Linux in the coming weeks. We used MySQL on Linux and while it is slightly slower, it still consistently beats all competitors.