If there is one thing we have a lot of here at Therap, it’s data. Whenever possible and appropriate we try and make decisions based on it.
One of these choices that we have been working on recently is when to take the system down for updates.
If you don’t work third shift and you don’t work weekends, you may think that Therap basically never goes down. Unfortunately life isn’t that simple. There are a range of reasons that we may need to take the system down (usually for 15 minutes to half an hour) to post an update to the application or bring up new equipment. Thankfully the list of reasons that we might have to do this is actually shriking as we develop new tools that allow us to do rolling updates which require no downtime.
If you think about our last major release, (Therap 2012.0) which we released on January 15th. Well, we din’t have to release 2012.0.1 until yesterday! If you go back and look at our previous releases (please don’t look too hard at Therap 9.0) you will see that in the days and weeks following there we numerous releases.
Despite this significant progress and all the steps we are continuing to make to improve things, it doesn’t make it much fun if you are the one who is working when we take the system down. In particular, if you are regularly scheduled to be working around 4:25am eastern on Sundays each week, the chances of you being impacted by downtime increase enormously.
There is no perfect time to bring the system down as people are always logged in. Add in the complications of Time Zones (from Eastern, to Pacific, to Alaska, to Hawaii, and even adding Europe) and picking the right time is not easy. Interestingly the peak times for data entry and data retreival are different, peaks (and troughs) are also different for different modules. We have looked at all of this (on a minute to minute level) and tried to come up with the best timing we can.
There are also going to be times where we can’t use the optimal slot for a number of reasons, but whenever we do take the system down, we do it knowing that we are causing invonvenience, but also that we are doing something to make the system better and more stable.
Whenever it is possible we try to give ample notice of this downtime, but when something goes wrong we don’t always get much notice ourselves.
So, if you are one of those folks who works weekend night shifts, please accept my apologies and know that we are always doing our best to get you back to work as soon as we can.
:: Justin ::