This morning I discovered that our StarTeam server was being uncooperative – refusing to accept check-ins because of a lack of available storage space.
Users were receiving this message:
A quick look at the appropriate network share revealed plenty of space – while the hives were configured to use up to 99% of the available space, the share was only 86% full.
Weirdness.
Delving into the log file, I found this message (formatted for this post):
The storage limit threshold has been exceeded for mount point \<server>\StarTeam. The storage limit threshold was set to 99% (861,692 MB) of the total disk space. The space utilized on this volume is 872,989 MB. The total disk space on this volume is 1,023,994 MB.
If you look into the math, you’ll see that something is wrong – 99% of total disk space works out at 1,013,754MB, much higher than the threshold in use.
It turns out that the StarTeam server only calculates the threshold when it starts up, and had therefore “missed” the change we made last week when our helpful Systems Administrator added an extra 150GB of space to the share.
What can we learn?
If you’re a StarTeam administrator, remember to restart your server after adding space to the volume housing your hives.
For the rest of us, the developers, this is a reminder to be careful how we write our checks.
The StarTeam server is interested in knowing how much free space is available, but instead of checking free space, it checks to see how much has been used. Instead of checking of free space has fallen too low, it checks if space used exceeds a pre-calculated threshold.
By checking a proxy, rather than the actual metric of interest, the server failed to handle an unanticipated situation.
Comments
blog comments powered by Disqus