This morning I discovered that our StarTeam server was being uncooperative – refusing to accept check-ins because of a lack of available storage space.

Users were receiving this message:

A quick look at the appropriate network share revealed plenty of space – while the hives were configured to use up to 99% of the available space, the share was only 86% full.

Weirdness.

Delving into the log file, I found this message (formatted for this post):

The storage limit threshold has been exceeded for mount point \<server>\StarTeam. The storage limit threshold was set to 99% (861,692 MB) of the total disk space. The space utilized on this volume is 872,989 MB. The total disk space on this volume is 1,023,994 MB.

If you look into the math, you’ll see that something is wrong – 99% of total disk space works out at 1,013,754MB, much higher than the threshold in use.

It turns out that the StarTeam server only calculates the threshold when it starts up, and had therefore “missed” the change we made last week when our helpful Systems Administrator added an extra 150GB of space to the share.

What can we learn?

If you’re a StarTeam administrator, remember to restart your server after adding space to the volume housing your hives.

For the rest of us, the developers, this is a reminder to be careful how we write our checks.

The StarTeam server is interested in knowing how much free space is available, but instead of checking free space, it checks to see how much has been used. Instead of checking of free space has fallen too low, it checks if space used exceeds a pre-calculated threshold.

By checking a proxy, rather than the actual metric of interest, the server failed to handle an unanticipated situation.

Comments

blog comments powered by Disqus
Next Post
Scheduling Downtime  19 Nov 2008
Prior Post
Small Fonts  11 Nov 2008
Related Posts
Using Constructors  27 Feb 2023
An Inconvenient API  18 Feb 2023
Method Archetypes  11 Sep 2022
A bash puzzle, solved  02 Jul 2022
A bash puzzle  25 Jun 2022
Improve your troubleshooting by aggregating errors  11 Jun 2022
Improve your troubleshooting by wrapping errors  28 May 2022
Keep your promises  14 May 2022
When are you done?  18 Apr 2022
Fixing GitHub Authentication  28 Nov 2021
Archives
November 2008
2008