One of the biggest threats we see on an ongoing basis to the overall integrity of a Healthcare Information System is erosion of the initial technology specification or design. Erosion is fresh in my mind having accidentally scheduled a brief hiatus in the Caribbean just prior to the arrival of the hurricane, superstorm, and media event also known as “Sandy”. I had the dubious honor of enjoying her side effects in the southeastern Caribbean as well as following her home along the US coastline in an airplane. Three hundred miles southeast of Sandy’s outer arms, here is what her wind-driven surf did – subtracting a good 10 yards of beach from my home away from home.
Weird things sometimes lodge themselves in our brains. Now that Hollywood has taken “the Avengers” from nerd-dom to the mainstream, I don’t mind admitting that in “The Defenders” – another Marvel Comics supergroup from the 70’s, a frequent comic foil who accompanied the arrival of archfoe Galactus was a greek-chorus-like character who repeatedly droned: “Entropy, entropy, all winds down!” My high school physics teacher would have approved. The “erosion” manifestation of entropy is particularly insidious, because everything seems fine – until all of a sudden it is not.
Erosion manifests itself most frequently in the HCIS as a change management challenge. The system was built for 1300 network connections but it seems to “handle” 1400 pretty well. Suddenly at 1425 user response times go to heck in the proverbial handbasket. The host farm was designed for 40 virtual machines plus a fixed amount of vMotion headroom and spare capacity, but there was a need to create a few “temporary” servers that became permanent. What happens then when the EMR tries to vMotion to what had previously been reserve capacity? The archive segment grew beyond all expectation, so we had to steal spindles from the production storage pool without much thinking about what that did to peak IOPS (input/output per second) load. That night, IDR is mysteriously slower. The Radiology department spun up a new modality in a hurry and said they would backfill the IT infrastructure later. All of a sudden, admission screens are flipping in 4 seconds instead of 1. I think a lot of our readers have seen this movie before. You begin to wonder which chamber has the bullet in it!
I would simply offer that the design you believed in on day one was probably a pretty good design. If within your organization you are developing the ability to bill back IT services to departments, or at least to projects, then maybe you have a chance to defend the integrity of the original design with service change orders. Another potential way of managing this situation in advance is offering multi-year TCO (total cost of ownership) models with the original system purchase that specify lifecycle additions both programmatically and in response to currently unforeseen changes. Even if your IT advisory committee or finance board is inclined to invest in long cycles, building change management into your plans at least offers awareness of the importance in sustaining IT thresholds over time. See you again soon.
Jim Fitzgerald is EVP and CTO of Park Place International. He wanted to be a musician, but his fingers were too short. Replies of any kind to our blog entries at PPI are welcomed. Particularly insightful or funny comments might find themselves gifted with a random prize.