XtremeCamera

That’s when it starts. Panic. Out of nowhere something, or someone, is about to ruin your day, disrupt your business, and otherwise mess with your head. Of course, at this point I’m not going to put blame on anyone or anything…not yet. After all, perhaps a major pipeline in New Jersey is out. Maybe my cable company (Comcast) is having a problem. At this time of the day the maybe’s are running wild because You.Just.Don’t.Know…

I popped up another browser window and went straight for Apple.com. It came up in a near instant. Not good. I then went through each and every morning stop in my daily routine…and every site came up in the browser, smoothly, quickly, without a hint of a hiccup. Not good. Not good at all.

I tried XtremeCamera again. Still nothing… Only thing to do now is call our hosting company. Well, that’s not exactly true… first I send out emails and SMS to the rest of the crew to alert them about the situation. THEN I call CI Host… and it rang and rang and rang…. And then went dead.

I did this for about a half hour before I finally got an answer. The empty sounding machine on the other end began routing me through the system until I wound up at the Collocation Tech Support Department… I was number 19 in the queue. For the next 45 minutes I went from 19 to 1… then an actual human being, sounding like he was a meth addict talking to the police, informed me “due to severe storms our power grid went out. We’re working on the problem.”

I asked if he had an ETA for getting the system back. (A system, by the way, that houses more than 100,000 domains in the Bedford Texas facility) He told me 8AM Central Time… so I waited. 8AM came and went…nothing. My blood pressure at this point was screaming at me to pop another attenenol. I waited till 8:45 Central time and called back… this time I was put through in just under 45 minutes.

Same spiel, different ETA. “Noon” he told, meaning 1PM my time. “This is nuts” I said. “Tell me about it” was his reply. Well, nothing to do but live with it I thought to myself… so I went about my day knowing that members and visitors to XtremeCamera were being subjected to that ugly “Server not responding” message in their own browser windows.

At the XtremeCamera offices in New Hampshire and Texas (Arlington) there are always a million things on the official To-Do list. Yesterday was no different. The page(s) you are reading this on is one of our planned upgrades, and the Arlington office was to spend the day putting the finishing touches on it. In New Hampshire it was all about the new Video features, something that will roll out before the end of April. The video feature still need much work in the CSS (cascading style sheets) area, and that’s MY area. Well, not all mine; our Texas resident genius is the actual CSS master.

Fortunately all the computers run local server software so that any changes to the code, the design, graphics, etc, can be seen locally as if it were on our servers at CI Host in order that we can bring them online without having to make adjustments. The ability to work on the CSS of the video gallery was easy enough, the only thing getting in the way were my nerves and this nonstop feeling that members were going to think we’ve closed up shop and gone home…

I couldn’t work, so I decided to get out of the office and run some errands that were scheduled for the weekend. Thinking this would ease my mind and provide a few more hours of intense Xbox 360 game play I was up and out the door in within minutes of making the decision to do so.

I was out and about when Noon-central came and went. In the parking lot of Shaws grocery store I pulled out my iPhone, touched the XtremeCamera icon on the homescreen (a beautiful icon it is!) and waited… still nothing. I called the office, “Is it up yet?” I asked. “Nope” came the disgusted reply.

Sitting in the parking lot I called CI Host again… Twenty minutes later I’m talking to Tech Support and they tell me that it is now 5PM Central before the servers will be online. I was about to ask him if he knew how badly this was going to affect our business (we do have other sites on our server that we do business on) then I stopped myself. “Of course he knew!” I said to myself. “How many people are giving this guy grief right now?” So I thanked him for the information and he gave me a URL where I could keep track of the status of our servers. I finished up my errands and went home. This day wasn’t going to be spent on CSS or anything else. I wasn’t feeling the job…

I’m sure everyone here knows the feeling of being completely helpless. This is what I felt, and it was terrible. If only I was there, I thought, I could help them. Which of course, wasn’t true, and that made matters worse. I decided the only thing to do was to try and make sure this couldn’t happen again.

When I got home I began surfing the net for ways I could prevent this in the future. Maybe set up a server in another part of the country and switch the DNS if our Texas servers couldn’t be reached. But even today’s fast DNS switching it could still take 8 hours or more to for the DNS to propagate through the networks so that isn’t the answer… unless we knew for sure that we were going to be down for days, not hours.

In the end I decided that, really, nothing much could be done except to alert every member through an email blast or SMS should this happen again. At least the members would be made aware, and they wouldn’t worry about it. Any community is a fragile system, and an online community is even more so. Leave your members in the dark and they will walk away, so this much, at least, will be done.

6PM EDT comes around, my nerves are now completely shot, and I make another attempt to reach XtremeCamera. Still nothing. I log onto the network status URL the Tech Support Team gave me and I see now that the ETA is “Unknown”…. Great, just great…

2AM EDT and I wake up in a cold sweat. I reach for my computer, try the site, still nothing. I sneak out of bed, grab the phone and call. I’m on hold for 35 minutes; reach #1 in the queue and then the line goes dead. I go back to bed resigned to the mercy of CI Host, and I’m hoping that when I wake up the server will be alive and kicking.

5:12AM I wake up, do the same thing. Still nothing. I place yet another call to CI Host and this time make it through. I’m told that all is well, but every Collocation server must be rebooted manually and they have not yet reached our rack. The servers we own, at their mercy. Not a good feeling. “Hang in there a while longer” I’m told.

6:15AM Nothing

6:30AM Nothing

7AM Nothing

7:37AM We’re back!

I send out a system wide message that our members receive when they log in and tell them what happened. I am relieved.
In an email from our PHP guru I am told that our new Homepage is done, 100% and will be uploaded when, or rather, if, the server comes back up.

This is the XtremeCamera Blog, a part of our new “Daily Snapshot”… Here you can get an instant feel for what’s happening inside at any time. This is the heartbeat of our community… And in our next blog post we’ll tell you all about it. But for now, letting you know what happened yesterday and what it feels like when the lights go out, was just a bit more important.

Today is Friday. Welcome to XtremeCamera, where everyday is a new adventure. Yep.

John Manzione
Admin