[cvsnt] Stress Tests results for CVSNT/CVS/Subversion

[Rahul Bhargava]

FYI,

Just wanted to share some stress test results with the CVSNT community.

We recently undertook an extensive stress test exercise with the objective
of understanding how CVSNT, CVS, Subversion servers behave under high 
client load.

We created a bank of client machines  (Windows 2k3, Linux 2.6.x) to 
generate client
load. The workload that each client iterated through was the usual 
cvs/cvsnt/subversion command set
that a development organization would see - import, update, checkout, 
log , diff, tag, rtag, status etc
Each client would repeatedly execute the same workload with or without a 
wait time.

With 50 clients pounding on a CVSNT server (2.5.03) running on a Windows 
2003 Server machine
with 1GB RAM, 2GB SWAP, 2xPentium 4 CPUs (2.8GHz Dell server class 
machine), we saw that after
about 15 minutes of stress the CVSNT Lock Daemon service would freeze, 
the CPUs would be maxed out at
100%. When the freeze happened, almost always the command `rtag' would 
be the one running. We would see
several clients trying to `rtag' the same module leading up to the 
freeze. Sometimes add/commit would trigger similar issue. The clients 
were running the same CVSNT version also (2.5.03). Shutting down the lock
daemon service immediately brought the CPUs back to idling state.

We tried the same stress run on a Linux 2.6.5, 2 CPU machine. The 
clients were running on Win/Linux
and the CVSNT 2.5.03 server was running on the Linux box. Similar 
results - the server would go on for
15 mins - 2 hours before hanging the Linux machine. The CPUs would be 
maxed out and the only way
out would be to reboot the Linux box.

Next we tried the same experiment with CVS (1.11.21) and we could run 
the stress for days without any issue.
The CPU usage would be fairly low with and we didn't see any freezes or 
hangs.

Similar experience with the latest Subversion release (1.3.0) - we could 
run the stress for days without any problems. The CPU consumption was a 
lot higher than vanilla CVS. Subversion `svnserve' processes
would consume around 80% of the CPUs when the stress was on. Other than 
that, checkouts were noticeably
slower with subversion as the number of revisions grew.