The calm after...

In our last episode, I was whanging on about the frequency of tornado alarms. After the devastation in Alabama and Georgia, I retract evything I said. Much better to be warned and it blow over, than to not be warned and have the worst happen.

The storms might be over here in the mod-south, but the floods are just getting going. One of my coworkers is currently evacuating the apartment he moved into last weekend, due to the Wolf river knocking on his door. I took some pictures this morning on the way in to work, and I'll post them once I get 'em developed.

Wolf! Wolf!

Shelby county has a wonderful tornado alert system, with sirens that cover the whole county.  The only problem is, whenever theres a tornado warning anywhere in the county, they ALL go off.  So, the actual inclement weather could be at the far end of the county from where I happen to be at the time, and to paraphrase a line from the movie Armegeddon, it's a big ass county. Also, in stark contrast to the years that go by without a tornado warning in the east side of the state, Memphis is in Tennessee's version of tornado alley, with at least 4 alarms in the last 5 days.  As a result, I've gotten not just blase, but downright dismissive of the alarms.  Not that I don't check the radar, but since I've started this post, we've had 2 different warnings...one for the northern county, and one for the extreme southeast part of the county.  Here in the middle, it's just been a bit of wind and a lot of rain.  I only wish that we could get a bit more specific in the areas of the county where the alarms are going off...but that would entail that the county's emergency management team be fully staffed and competent.

While on this subject, I'm gonna again harp on the local broadcasters.  Granted, when the weather service puts out the tornado warnings, they have to report them.  And given how freakin' big the viewing area is here (flat, remember?) a warning 3 - 4 counties away justifies breaking into regularly scheduled programming to spread the word. But do they have to sound like their covering the end of the world?

And, for something completely different...

One of my co-workers turned in his 2 week notice yesterday.  Anyone know of a good IT support tech looking for work?

Memphis in May

Every year, the city of Memphis puts on a month-long celebration during May.  It's starts with a 3 day music festival (which coincides with monsoon like rains every year), and goes on from there.  There's a bar-b-que competition that gets a lot of coverage on the Food Network, and it all wraps up with a symphony concert on the riverfront.

I avoid the music festival (even though there's always somebody playing that I'd love to see...I'm just getting too damned old to put up with huge crowds and rain), and the bbq competition is a tease...unless you get tickets to the tasting booths, you don't get any food.  However, I love the very concept of a city with the problems that Memphis has being able to take a month to tell the world how proud they are of this place, and how much they want to share it with everyone.

This comment will probably shock the hell out of a lot of people who know me.  I wasn't thrilled with the thought of moving here.  I prefer to live where there's at least big hills, if not mountains, and I've yet to find the bluff that provides the city's nickname. Four years ago Memphis seemed like a war zone, with a heinous murder and violent crime rates. The public schools were failing, with a huge drop-out rate. The mayor at the time was a joke, who used race as the main reason to be re-elected and over-paid cronies to supposedly run the city. But, I figured that I could put up with anything for a while, and this is where the jobs were.

Fast forward to today.  The crime rate is down, and Mayor Willie is just a bad memory. Memphis weathered the recession/depression better than most cities, and now we even have new businesses moving in, providing good jobs. The clinic I work for grows on you, and I've been fortunate to have been able to increase my skills and experience with new systems and technology. The schools still are terrible, but they're working on consolidating the city schools with the county, hoping that will help.  It's still Kansas like in geography, but you tend to not notice too much when the trees are leafed out.

Maybe this isn't such a bad place, after all....

The devilish details...

Ok, so yesterday we left our intrepid band with an empty data store at one location, and a bit better than a 1/2 TB data store just aching to be copied at the other.  Now, I might have a boatload of bandwidth between offices, but it's occasionally nice to leave room for other things...email, internet access, other applications.  You know the stuff that pays the bills.  So, how did I get the data copied, you ask?

At this point I could regale you with tales of black magic...summoned storage demons (or is that daemons?), pentagrams on the floor of the server room, and the obligatory critter sacrifice.  Instead, I'll talk of something even more mysterious, and rare.  A freakin' vendor app that works exactly as promised.

The SAN manufacturer (there was originally a cutsie, insider description of this manufacturer in this parenthetical aside.  Bag that, it's Dell EqualLogic) has a replication utility built into their iSCSI SANs.  You set up the 2 devices (one with the data, the other the target), and tell it to replicate once.  You can throttle the bandwidth usage down, so it leaves enough room for other apps, and it merrily goes about it's business.  I started this replication yesterday at around 1pm, and it wrapped up today at 8am.  I promoted the replica to a volume, and connected the vm to it via a software initiator. A touch of work by the PAC vendor, and we were back in business. Voila! As nifty a piece of magic as any seen in any grimoire, and a lot less messy than offering up the goat.

Good job, kid! Don't get cocky!

Having been in IT for as long as I have, you'd have thought that I'd known better to mention hubris, good jobs, and things turning out ok in public.  The gods do love a proud man...the better to grind him down.  First the good news...at least my attendance at the storage forum will be covered by the company putting it on.  With that, I'll go if I have to cover the rest on my dime.  After all, it's only a 13 hour drive!

And then the other shoe fell.

The company I work for is like any for-profit organization...if money can be saved by cutting corners, then of course you cut them.  When we virtualized our 3 remote clinics, we set them up with a single ESX host, and a local SAN.  The SANs that we installed were designed to be fully redundant.  In other words, dual controllers, power supplies, etc.  Ok, we got the dual power supplies but went with only one controller.  After all, we were only 1/2 filling the drive bay, so one should be fine, right?

I'm sure those of you in the know are cringing by now, because of course one of the remote clinic's SAN lost it's controller last Thursday night. Now, normally you'd think that if you could get a replacement (I had a spare installed at one of the remote sites), and you can move the flash memory card to said replacement, you should be able to power everything back up and be right back in business. At least, that's what we were thinking when I headed out Friday morning to get the spare and install it in the failed SAN.

Ok, so the spare is installed, and the SAN restarted.  Oops, no LAN links on the controller.  So, lets find a serial cable and hope the serial port on the non-ESX server in the rack works.  The cable was easy, the serial port decided that whatever program had control of it, it wasn't willing to share.  So, grab a pc and get connected via Putty.  By now, I'm with a level 2 tech, who gets in and drops bombshell number one of the day (at least from tech support)...any time that a single controller SAN fails, it REQUIRES L2 support to get it back on line.  Then, bombshell number 2...the read/write cache is unrecoverable, so any read/writes that didn't make it to the SAN when the crash happened is lost.  Telling me that the only way to get the controller on-line is to dump the cache, he then requested that I ok the dump.  Kinda like the commander of the firing squad requesting that you give the order to fire, eh?

Never one to hesitate when confronted with recalcitrant hardware, I had 'em pull the trigger, dump the cache and glory be, we were back on line, with LAN connections and all. The VMs mounted up, and everything was just peachy.  Oh, except for the corrupt 1/2 terrabyte of storage on our XRay image store VM.  Thus begins the tech support shuffle, with the SAN techs suggesting one thing, and the PACs (thats xray software, to you non-healthcare techies) techs wanting to go the checkdisk path.

No joy on any suggestions from the SAN techs, and chkdsk failed miserably. Those of you wondering whyinhell we weren't backing up this data, the original design was to replicate every image to an archive server, which we'd been assured that we could recover a site from with not too much problem.  So, lets get the xray machines aimed at a different server (we have 100mbps connections between our offices...gotta love a metro fibre ring!), and then restore the data that got trashed..  Now, this little redirect chore requires that the company that supports the xray equipment make the networking change, on site and in person, using a hardware security dongle, and they assure us that there would be a tech on-site first thing Monday morning.

Monday morning, and no tech.  A quick call to the xray company, and we get forwarded to the tech, who was just leaving St. Louis, with an expected ETA  of mid-afternoon. So, the boss borrows a security dongle from the IT director at another orthopaedic clinic in town who uses the same company (they paid large green for the training and the dongle) and a coworker went down and redirected the machines to send their images to our main office, which got our doctors at that location back in business.  Hurrah for us!

For about 45 minutes, that is.  That's how long of a reprise we had bfore getting a call that another site had it's PACs software stop working altogether.  So, all afternoon was spent troubleshooting that issue(me working on the hardware, and interfacing with the PACs techs).  We had docs at that site calling the CEO, wanting to know what IT was doing, and why wasn't it fixed yet, dammit!

I found zero wrong with the server/SAN/network/workstations.  That's what the PACs techs said about their software.  So, they fell back to the (usually right) IT solution...reboot everything.  The server, the ESXi host, the SAN. I grouse, but agree, and plan on doing that at 9pm last night.  Finally, I leave the day from hell behind me, and head for home, for a brief respite before the massive reboot.

At about 5:30, I get an email from work.  I call to get details, and it seems that I don't have to reboot a damn thing.  Since we still have the PACs security dongle, the co-worker who had redirected the images from the SAN failure site had decided to do the same at the site of the new failure.  Once there, he discovered that during the full upgrade on that system a month or so ago, the company that did the upgrade had ignored the IP addresses they'd been given for that site, and had instead directed the images to the only other site that hasn't had any major problems during all of this Charlie-Fox.  He directed the images to the correct server, and strangely enough, everything started working. Unrelated work by our PACs company had actually turned off the failed sites ability to pull images from their server.  That's what triggered the afternoon outage.

So, how was YOUR Monday?

Still wondering about the large data recovery?  Tune in tomorrow....

Hubris, anyone?

Ok, so there is a downside to the old self horn-tooting. No, nothing bad happened to my ESXi installs.  As a matter of fact, the auto-update configuration I set up is working wonderfully.  I hope to have our main office updated this coming weekend.  I may even upgrade during the week...something that I can do, as we have a 3 host HA setup, and so I have enough server power to keep us up and working  while upgrading each server.  No, I instead refer to the presentation opportunity at the storage forum in June.  The guy at VMware still wants me to present , but pretty well cut me loose on getting the trip covered by either them or the storage manufacturer.  Of course, since he'd included 2 contacts at the storage company in all of his email, I just hit them up to at least cover the forum for me.  I would think that getting that done shouldn't be a problem, as it's not like they'd be out that much income from allowing me to attend for free.  The rub is getting the invitation, then being told that it'd all be on my dime, if I can't get any slack from the hosting company.  This is the professional equivalent of having to work on the computers of friends and relatives for free...after all, I'm in computers, I should be glad to have the opportunity.

I've decided to modify the name of this blog, and add "and computers" to the title.  After all, cats are supposed to have been considered the familiars of witches, and I've always claimed that computers only work due to demonic influence.  Anyone who has spent an all-nighter trying to figure out why a server upgrade hasn't worked, only to have it miraculously (and seemingly on it's own) start working just as the sun is coming up, and you were about to restore the original configuration knows exactly what I'm talking about.

When you're good, you're good

I managed to get 3 VMware ESX 4.0 servers upgraded to ESXi 4.1, update 1, and all VMs back on-line in 6 hours today.  Of course about an hour and a half of that was driving between sites, so it was really only 4.5 hours.  For those of you in the biz, I was upgrading IBM 3650 M2 servers which take for EVER to reboot, so I think that I did pretty good.  Occasionally ya gotta toot your own horn...otherwise it never gets tooted.

For the last 3 weeks, I've been smelling what seemed to be hot antifreeze when I got out of my car.  I looked, but I never really saw any leaks.  Today, the car made a damned good attempt to overheat.  Fortunately, I was only a few miles from one of the offices I was upgrading, so I managed to limp in without actually overheating.  I let it cool off for about an hour, then added about a gallon or so of water, both in the radiator and the overflow tank.  I guess this means that it's time to again try out the extended warranty I purchased when I bought my car.

I've decided that some of the posts to this blog might end up a bit technical, and will not be coming with explanations of the processes and procedures, except to the level that an IT pro will understand. This is for a number of reasons. One, it's nice to be able to talk the talk without having to provide a layman's level explanation, and two, I might be able to help one of my peers with issues that they may be having.

All that being said, I did get a bit of good news this week.  I've worked with some of the VMWare engineers directly, as my company was the SMB beta tester for VSphere 4. This has led to me being asked to do part of a presentation at Dell EqualLogic's forum in June. This in turn might lead to a repeat at VMWorld in August, which would be just fantastic. Who knew that being egotistical, argumentative, and talkative could be good things?

April in Memphis

A week ago the weather in Memphis was perfect...lows in the mid-40's, highs in the 60's and low humidity.  A few days ago it got down into the 30's at night, and I actually had the heat on for a bit.  Today it hit 82, with the humidity pushing 60%, and is supposed to be 88 tomorrow. I guess it's a good thing that I plan on working most of the day...our offices are cooled (or heated, as needed) on the weekends, and that means I can stay comfortable on the companies dime, not mine.  The cats can find comfort where they can...for all that they're stuck in a fur coat, I'v not seen that they suffer too badly, and will even hang out in a handy  patch of sun when I'm in shorts and sitting in front of a fan.

I think the thing that really gets me is that early April is supposed to be when you can open up the house, and be comfortable, maybe even be thinking of putting a bit of glass back in those holes.  Instead, I'm not only thinking of closing everything back up, but turning the AC on.  It's times like this that I really do miss the pacific northwest.  Nah....I just be bitching about the rain if I was there!

Midweek Doldrums

I've worked in IT on and off for pushing 3 decades.  The first 15 years was usually as a side thought...most of the companies I worked for had only a few desktop PCs, and no need for a full time IT staff.  So, like a lot of us, I made my bones a bit at a time. However, even back then it was always assumed that any work that required a computer to be off line would be done off hours. After finally giving up the day job for full time IT work, this expectation of working nights, weekends and holidays only got more entrenched.

What brought on this train of thought is that for most folks, Wednesday is the halfway point in their week.  For most of us in IT, it's the day that we'd better have whatever we've got to do over the weekend planned out, so that we get as much free time as possible. When you work for a large company, you have all sorts of change management processes that ensure that any and all after hours work is well planned out, with failback plans, and full testing procedures. When it's a smaller company, not so much. Any change management process is, for the most part, self imposed and self regulated.  I try to make sure that I have all of my ducks in a row, but sometimes shit happens.  I was in the office last weekend preparing to upgrade some servers, when I discovered that the test machines I had used to verify the process lacked one piece of hardware that the production servers have...mainly a dedicated card to connect to our storage array. So, back to the drawing board, and another weekend (this one coming up, in fact) where I get to see more of our server room than I might have wished.

The storm that blew through on Monday missed both the clinic and my apartment, but wreaked havoc on a lot of other places here.  The usual housed with trees in the living room, busted windows, and one car that an uprooted tree had pancaked.  I think the best was a house that was pretty much destroyed by a tree falling into it, that had a "for sale - foreclosure" sign in front.  Just hope they'd kept up their homeowners insurance!

Lunching to to sounds of tornado sirens

Today we had a massive cold front come through Memphis, which gave our local tv stations the opportunity to cover the accompanying thunderstorms as if they were the apocalypse. Granted, we had a tornado warning in place for a while, which means we were all grouped together in the back hallway of the clinic, waiting for the all clear.  But I've never seen such ludicrous, overblown behavior by meteorologists in my life.  I'll be very surprised if they don't preempt all regularly schedule programming tonight, just so that they can say that they were protecting the residents of their viewing area.

I was worried about the effect the deluge was going to have on the tomatoes I planted this past weekend.  I seem to remember that last year the planter I used had a bad habit of just filling up with water, and not really draining.  But, when I finally got home from work, everything looked fine.  Go figure.

Ever since moving here, I've noticed that every time it rains with a little more intensity than a mist, most of the traffic lights on the only direct route between work and home go out.  Now we're talking about a divided 4 lane highway that carries a ton of traffic, it being the only major north/south route for at least 5 miles .  So, when the traffic lights are out, it's easier going the 5+ miles out of the way to get home. I've always been able to cope with long drives much better than long waits in traffic.

The boys were unimpressed with the storm, or at least didn't bother to tell me if they had any issues with it.  Seeing how storms don't normally keep food from the bowls, the cats tend to consider them entertainment, once they decide the loud noises can't get them.

Sunday morning

Most Sunday mornings start fairly late for me...especially if I indulged in a wee dram or two the previous evening. As time has progressed, however, I'm finding that I begrudge the restful, wasted time spent lounging in bed. Especially given that this time is also spent warding off the cats, both of whom are reminding me that the food doesn't pour itself into the bowl. Mikey is usually content to thump down beside me, using my back to ricochet into a comfortable position, there to wait for me to get up. Julian, however,believes in a much more "paws on" type of approach. His favorite method of getting your attention is the tried and true "cat snot sneeze to the face", which is just as disgusting as it sounds. He is the only cat I know that can sneeze on demand.  If that fails, he thinks that a whack in the face with a (sometimes) soft paw will do the trick. This might work on Gayle, but the only result he gets from this is an attempt at reminding him which of us is at the top of the food chain around here.

This morning I poured a cup of coffee, and decided that a taste of the home cured bacon Gayle brought down from Kentucky would be just the thing.  I had set it out to thaw, it having been in the freezer.  So I opened the ziplock bag, and discovered that it really was slab bacon...a chunk about 3 inches thick.  No worries, I whipped out a knife and start cutting some rashers.  The first one was even, not too thick, and looked like something any store would have been proud to display.  By the time I got to number 4, it was looking like something that came out on the losing side of a bar room knife fight.  How in the hell did Gayle get the last batch cut so well?

Anyway, into the electric skillet they went.  I busied myself frying up a couple of eggs to go with, and a slice of toast (can't let any of the good stuff get away...you just gotta sop it up!).  The bacon's frying, the cats are creeping into the kitchen like ninjas, hoping a: I don't see them, and b: I drop something for them to eat. I cloud up and rain all over these plans as I detect them, as cats underfoot don't go well with hot skillets, in my opinion. Finally, the bacon's out of the pan, the eggs are done (with just a touch of lace...damn griddle got too hot), and I'm plopping down at the table to enjoy the fruits of my labor.  The cats are lined up, strategically placed to catch anything that might hit the floor, and I tuck in.

When dealing with homemade, salt cured bacon, a word of advice.  Rinse the salt off the bacon prior to cooking.  If you don't, then what you have is the seasoning and flavoring ingredient for about 10 pots of green beans.  This morning's delicious looking breakfast treat was the salt equivalent of the first time I ever made salsa, and didn't know that you were supposed to remove the seeds from the hot peppers prior to dumping them into the sauce. Ah, well, I'm sure that Gayle had told me about this...who knew I was supposed to listen and remember?

I gamely plowed my way through two slices before giving it up.  Both the boys indicated that they would be more than happy to take the remainder off my hands, but I don't think that it would be any better for them than it was for me. so they're off sulking right now.  Me, I'm getting ready to do some laundry, waiting for my blood pressure to subside a bit prior to heading out to update the servers at a couple of our clinics.

Will the ghosts of dead plants ever seek revenge?

For the first time in my adult life, I went out and bought a couple of tomato plants, and plopped them into the planter on the patio. Now, the deathwatch begins.

The surprising part of all this is that it's normally Gayle who has such good intentions in the spring, spending hours of time readying the planters, picking out just the correct plants (or seeds, as she did this year in Kentucky) and tending them with loving care.

Life being what it is, this loving care normally lasts for the first 2 weeks.  After that, the poor plants are left on their own, to survive or not as weather and roaming critters dictate.  Much the same can be said about indoor plants. t's not that Gayle is irresponsible, but rather that something else always seems to come up.  I've notice that this is evidently something taught to artists and crafters, in that there's always something else that comes up.   Now, I'm just as guilty of foliage homicide as Gayle, when it comes to the indoor plants.  In this, Mikey and I are in agreement...if the plant wants water, it can go to the sink and get it itself.  Of course, Mikey is a full accomplice in planticide...he has yet to meet one that he doesn't think is delicious.  He'll go to any extreme to get to a plant to nibble on, especially if the plant in question is fresh cut flowers. When I object to this behavior, I get the "but...they're FLOWERS!  Of course I'm gonna eat 'em!" argument, which is actually what Mikey considers an exceptionally well thought out argument. He's a pretty big cat, but I aways hear the voice of the yeti from the old Bugs Bunny cartoons riffing on Lenny.."duh, gee George.  Which way did the little bunny rabbit go, George?"

Jules, the more intellegent of the two, is fully cognizant of the fact that he's a carnivore, and much prefers raw meat to any plant.  He's pretty damned smug about it, too.

From all this, I'm sure you've figured out that, for plants, our door should have "abandon all hope, ye who enter here". So, back to the original thought...what would a haunting by a plant be like?  Hmm...maybe that explains the horrendous mildew on my shower curtain!

Taking the plunge

I decided to start a blog in response to my wifes having moved to Kentucky to take care of her elderly parents. This means that I'm living solo in Memphis, with 2 cats for company. I dont get out much, not really having integrated into Memphis society. As a result, and being afflicted with a genetic predisposition to talk a lot, I find myself having detailed interactions with the feline roommates. Now, anyone who has ever lived with a cat knows that they have the uncanny ability to look like they are not only listening, but understand every nuance of the conversation. Unfortunately, they also don't look like they they think much of what I have to say. If you follow along for this ride, you'll get to know both Julian and Mikey all too well.

This morning I got up bright and early and went to a large urban park here in Memphis called Shelby Farms. A few months ago I bought a used 35mm camera, and I wanted to go try it out. Having walked way too far, given my abysmal physical condition, i managed to get some pretty good shots (for a total neophyte, that is) including some pretty good ones of the buffalo herd at the park.

I also ended up with a swollen left foot, which I'm treating with generic ibuprofen and a heating pad. Both of the cats have both expressed a complete lack of sympathy, simultaneous with requests to share the heating pad. I say, sympathy first, sharing therapeutic devices second. They, as usual, replied with their usual snide comments, which I ignored with dignity.