We've been servicing a particular health care practice for over 20 years, and I was initially excited when I learned they were moving out of their 25+ year-old dumb terminal server used for patient records and billing. But for those in health care considering any hosted solution--proceed with caution and leave no stone unturned.
Over a year ago I completed forms from the customer's new shareholder, which is a public company. Afterwards, I was asked to attend any meetings and I agreed to do so, but since that time, I never heard a word again until right after Christmas--the customer called in and needed emergency assistance on site to facilitate implementation of the "new system." Here’s a glimpse of what happened.
The public company's staff, consultant and IT contractors failed to contact the existing vendors supporting the customer. A site assessment that consists of filling in blank pages on a form isn't adequate--it doesn't really qualify as a site assessment--and for those that think this is sufficient, don’t bet your paycheck on it. A site survey or assessment means gaining an understanding of the customer, network and other relevant information, especially for those that don’t like walking through minefields later.
A few weeks prior to the attempted implementation, my customer and her staff complained about the poor training they had received on the new system, and how unsure they were. Once the contractors showed up for implementation, my customer’s staff repeatedly warned the contractors about not plugging network cables into receptacles that look like network connections. There's a good reason why the jack inserts and patch cords are color coded, but the contractor staff ignored the warnings and remarked to the customer, "These are just network connections."
So as these guys ripped and roared, things began to fail and not work as expected. Each user station has two "network connections;" only one is actually a network connection, and the other is a pass-through to a hard-wired serial port on the back of the dumb terminal server. The dumb terminal server is what the hosted provider proposed to replace, with the condition that the users could still access the dumb terminal server from their new workstations until the migration of data was tested, verified and completed during the coming months.
My customer was clear with the message that the old dumb terminal system must be maintained and reconnected to the new PCs that were replacing the dumb terminals. Nevertheless, someone overlooked the need to maintain the old system and the additional network drops for access to the network and Internet by dumb terminal users (now PC users), along with the network drops needed for each of the new network printers. The printers that were delivered would not work with the dumb terminal server and they were not network printers.
The new PCs are connected to two network cables--one for the network/Internet and the other for the dumb terminal server using an ICC DB9/RJ45 adapter (retails under $2) that I had to pick up along the way to the customer site, since the contractor didn’t know how to connect the PCs' serial port. I asked the contractor if they knew how to assemble the adapters and they said yes but didn't have time, so my customer, a business manager, asked me how. So I spent a few minutes with her and off she went to assemble adapters while I started to work on damage control.
The customer originally requested network printers and instead they were given standalone desktop printers and told to "have Matt order 18 USB print servers" on New Years Eve. I did get “emphatic” and explained to my customer that this was not in her best interest, and that she should have these guys make it right by ordering network printers. I ordered optional power supplies and daughter cards for the remote sites with ADTRAN 3448s (router/switch/firewall) and some 3Com PoE network jacks in lieu of wiring, since there was no way we could get the amount of drops installed in time, thus the cheaper and faster alternative to implement the optional PoE and 3Com network jacks.
In a few hours' time, these guys had managed to go through the corporate office and knock users out of service. Why would you show up on December 31st at 2 pm and then tell the customer you only have two hours to work, then leave? Scanners for patient records (drivers license and insurance card) were missing and the remote sites were ignored. The contractor was prompt and short and at 4 pm they told the customer, "we're finished and don’t have the time to give to every user." You can imagine how my customer felt, and during the next several days she used the words "dump, dumped and dumping."
After disconnecting what was connected to the wrong ports, we also discovered that not all the new delivered PCs were the same. We ended up rebuilding three using the recovery disks we could find in a box, and continued by downloading drivers from the manufacturer's website. No drivers could be found with the original disks. We worked the next couple of days and did what we could until the principals of the customer returned to inform the contractor, consultant and corporate office that they needed to make things right. The IT contractor did return for two days, having to rebuild other new PCs that they previously delivered, citing that the Windows updates were corrupted.
Prior to this fiasco, we had spent two years focused on building a solid network infrastructure and installing a server for existing backup needs along with minimal voice applications that account for 5% server utilization. We anticipated then that thin clients would replace most of the dumb terminals and that some PCs would remain and that the server would be used for the new implementation. The forms I completed the year earlier included the server and details about the converged network. We also included the recommended migration plans for this customer. The parties involved failed to realize that PCs weren’t needed, thin clients were preferred for most dumb terminal users and that the purpose of the server was to make life easier for everyone with the "new system." Driving from site to site, I also discovered new routers, installed in every location, sitting on the floor and connected to a service receptacle with no power or circuit protection, or battery backup. I learned that these routers were intended for the MPLS network (the "new system") for my customer. I asked the contractor, "How do you plan to introduce MPLS into the converged VoIP network?"
After waiting through the long silence that followed, the project manager replied that he’d get back to me: "You see, the new system is on the other end of the MPLS network." The next day the contractor spent time adding shortcuts using a Citrix client running in the native mode to connect the customer PCs to the "new system." Of course these failed because the contractor didn’t know that the security policies in place required user login to access the Internet. The Citrix client would connect but on the return side would get blocked by the firewall since there was no initial authentication by the user. Several months prior, the customer asked us to set up a policy to prevent certain users from accessing anything on the web save a handful of "authorized" sites. These users were unable to access the remote system until we removed the policy.
Without an effective plan and involvement of all parties, your chances for success are greatly diminished. The corporate management team failed to realize and understand this customer’s daily operation and how to effectively migrate a 25+-year-old system into the fold of the hosted solution. This cutover required maintaining the old system and essentially providing dual connectivity to the workstations by way of using a serial port on the new PCs to reconnect back to the old system using structured wiring and an adapter. Then, the details were completely overlooked regarding which printers to use that would work on the old system along with the best configuration to maintain or improve productivity of these workers. The assumption that the customer had enough network drops caused more frustration. Migration to the server was never considered. Then, overlooking the existing network running VoIP and the fact that the existing router could easily connect to MPLS and offer the migration to MPLS using existing equipment is the kind of mistake that creates reaction.
My customer was in reaction mode and this always costs more money while creating a perception that IT doesn’t understand the customer needs. End user companies cannot afford these kinds of mistakes. Whatever work and resources were spent on extending MPLS to all these sites amounted to wasted money, time and effort. The effect of a missed cutover, poor execution that obviously lacked any plan and I’m guessing a touch of arrogance or simply an attitude of not caring about the customer’s requirement of keeping the old dumb terminal server alive added fuel to the fire. Adding to the anxiety was executive management and partners in the practices (doctors) nodding their heads in agreement that they didn’t really know what they were agreeing to and clearly failing to understand the clear line of responsibilities of all involved, thus the breakdown in executive communications.
For those considering implementing hosted health care services or hosted services in other verticals let me offer a few simple suggestions.
Executive
Executives, don't stick your foot in your mouths, and be very clear that you are communicating what you want to have happen. Put the responsible people in charge and get out of the way. Don't interrupt the process. Doctors, you can apply this message to yourselves. Don't go ballistic purchasing cheap gear for a WiFi solution needed in an enterprise environment, and don't buy PDAs, notebook or tablet PCs before selecting the software that you want to use for your practice. This is called putting the cart before the horse. Collaborate on IT/Telecom with other doctors and in similar practices and find out what works and doesn’t work for them and why. Then, approach your IT team and/or contractors with a higher confidence of knowing or having an idea of what you want, and try, hard as it may be, to not interrupt the process.
Managerial
Discovery process: Is key in learning all you can about the customer, the site and all the wares and conditions of the site(s). This is how you avoid stepping on landmines or falling into crevices with no return. The time spent in discovery and planning will pay off later. This means you put skin in the game, show up and spend time on the sites--paper trails have a way of failing. Look, Ask, Find, Go, Get, Do are the key operatives.
Planning process: isn't as easy as filling out forms and pushing paperwork around. After discovering--and this often means touching--what's onsite, then find out how it’s all interconnected so that you can draft a plan to successfully implement change.
Documentation: The old ways, methods and procedures don’t go away, especially when you must run two systems (the old and the new) for a period of time before completing the migration. Migration path is simply where are you going and how will you get there. Never leave it up to the “experts” and instead learn all you can and make the experts explain it to you in such a way that you can visualize how things will work. Leave nothing to chance.
Communications: keep all parties involved, and this often means enlisting the help of the vendor(s), whether you are replacing them or not.
Technical
Hardware under-buying and over-buying are both common. Right-buy to enhance user productivity and before buying new hardware, trial it first with proposed configurations, not mock ups.
What is the backup or failover plan when things go awry? Have one.
Financial
Hosted or not, it's easy to get lured into a less than desirable model because it’s cheaper. The hardware must be configured in such a way as to offer the same or better experience and improve prior conditions.
Control
Hosted anything often means changing the span of control, and it doesn't mean you are powerless. You are still the customer and you still write the check. In my customer's case, big corporate daddy that bought into their company is writing checks too. It's precarious but still manageable since the business is about remaining profitable. Yes, in case you're wondering they have written some pretty fat checks to my company and will be writing some more before this is all over. There is much more work that wasn’t discussed and now timelines are extended. My customer knows now never to believe that "you just don't plug it in" because it's virtual or hosted.
Timing
Timing is everything. It’s true in networks and in business. Don't flirt with poor timing because it will hurt your business. Never be afraid to say, "Let's wait for a better time."
In going forward, we sat down for a brief meeting with the IT contractor and customer. Their CTO is now involved and getting a grasp on understanding my customer's configuration, network and needs before proceeding with MPLS. I think we were able to at least provide damage control and maintain costs within reason. The IT contractor offered some redemption by sending a worker bee onsite to remedy the numerous ailments of the new PCs. We are supporting the corporate folks, the IT contractor, consultant and customer. While I may never learn why or how all these details were overlooked, I do think the outlook is positive and I remain hopeful that the customer will benefit at least from the experience.
There are many truths and lessons in this story. For one--listen to your Mother. My Mom says I need to be more "emphatic." She’s right. I didn't get "emphatic" when I learned my customer's training experience was less than expected or when the customer's explanation prior to the last minute cutover didn't jibe with what should be happening. My older sister and I have had a few email exchanges about this too. She's a retired MD and was entertaining the idea about going back to work to keep busy. I invited her to translate physician IT needs to the IT/ITC folks and to intercede for doctors. What a great piece of glue to have in this case! I also taunted her with a generalization that 30-50% of her retirement would be up in smoke anyway (thanks to Woe Street), then why not work for me? You can guess how far I got with this.
Anyway, the project manager stated in later meetings that "he would forward 'best practices' over to me for implementing our server," which had my customer rolling her eyes. My feelings were different but I kept my mouth shut and didn’t get emphatic and just smiled. These bozos have handed me a golden check because they didn’t do their jobs. As it is, the big daddy public corporate shareholder is footing the bill. You see there is justice in this world. Sometimes it smacks you over the head but you can't complain. Their money is just as good as my customer’s, only it feels better coming from "big daddy."
Over the next several days I caught a bug from my daughter and was laid up feeling miserable. My customer called and said, "we need you to be at this meeting and know you’re sick." Of course I attended and it was brutally long. Here’s what I learned.
There were three CEOs in attendance along with managing partners from my customer and their business manager, the IT contractor’s PM and a couple of staff members. One CEO is tasked with the hosted solution for all the member practices, another CEO was representing another practice and all the practice’s board, and then the CEO of the IT firm was represented.
The communications around the table started with the COO of the hosted provider making the assessment that all parties were at fault and that the purpose of the meeting was to make things right. After listening a bit to the contract terms and then by what was intended, it was very clear that intentions and communications didn’t match. My customer was under the impression that the IT contractor would handle the migration and the contract stated that the contractor would basically deliver (cash and carry) as we had observed on implementation. The "IT guy" for the customer was expected to carry out the tasks and ensure that the "new system" was working and the users were prepared. The "IT guy" being my company had no prior knowledge of what the new system was until the day of implementation. (New PCs with a shortcut on the desktop to connect to a webserver). Of course my customer asked, "Who met with Matt?" and got a blank stare. After another hour or so of back and forth they all concluded that we needed to talk about the existing network.
My turn to be emphatic came up. I gave an overview of the past and drew a picture on the board and explained as simplistically as I could muster that we must migrate the network over to MPLS and to do so meant we needed to have a thorough understanding of the network. This was deemed Project Number 1. Project Number 2 is to assess the user needs, determine PC or thin clients for each user and then implement the changes, with Project Number 3 being to migrate the user PCs/Thin Clients over to the in house server.
Asked, "How long did I think this will take for Project Number 1?" I replied, "4-6 weeks, even longer." That wasn't what the CEOs and COO wanted to hear. Why so long? I got a turn at being emphatic again. After I delivered a one-minute lecture on what a real network assessment is I started into the customer's operations. We won't disrupt their business with experiments and any work including testing, patching or anything service affecting unless it is accomplished after hours and on weekends. Anything changed is backed up with next day onsite support during open hours. Their operation is revenue producing.
Next, I let them know that we've already made an investment in routers, and those existing routers must be reused and that the "free" routers that come with MPLS aren't acceptable. For one, the remote sites are using integrated routers with switch, PoE, firewall and backup DSL, and this will continue.
Several years ago I wrote a piece for BCR, and the same things are true here:
* Communications is a meeting of the minds,
* Setting customer expectations is paramount to the sale itself,
* Planning, planning, planning and even more planning is essential because it pays off in the backend.
Since this meeting I've met with the contractor employee orchestrating the migration and afterwards we've exchanged emails. There is a solid plan in place and specific action items to occur before moving the customer over MPLS.
Here’s the short list:
MPLS Implementation Plan
1. Use existing routers due to each site's requirements
2. Migrate all site connectivity off of existing Verizon Point-to-Point lines to MPLS connectivity for internal communications.
a. Setup QoS service to match dscp 46 for all Voice over IP packets (Need Verizon to setup QoS on MPLS)
b. Setup automated failover of Internet access through local DSL connections at each site to provide redundancy for Web Apps: Payroll, FTP, Centricity, credit card processing, and Telvox
3. Headquarters Site
a. Setup DHCP Services on router
b. Setup remote access VPN connectivity on router
c. Setup backup remote access VPN by Forwarding Ports (IPsec) through MPLS network to device
4. Provide list of Web Content Filtering
5. Outstanding issues before going live:
a. Customer to purchase VPN clients
b. Remote site #2 needs T1.5/MPLS extended to office
c. 20Mbps connection serving hosted customers must be installed first
d. Verizon setup of QoS on their Router for Voice (dscp 46) Type Packets
Now, it's been four weeks since our meeting with all the executives, IT contractor and customer employees and partners. The plan is being worked, but not until the provider has a 20 Mbps pipe instead of a single T1 pipe that is carrying their other smaller customers. One of my customer’s T1 spans is still waiting to get extended, tested and verified in a remote office. QoS must be implemented on the entire hosted network and once all the ducks are lined up, the hosted guys and their IT contractor must meet with us onsite and have techs on the other sites to test the new router configurations after hours. Then, all services and applications must be tested including the interoffice traffic--voice, fax, file sharing, and dictation/transcription. Once all these tests are made satisfactorily, we either stay live or go into the crunch mode to find out and fix what's wrong with the new configurations. The backup configs and point to point T1s will remain as the fail-safe to revert back should the issues prove too time consuming to resolve. Now, just imagine what would have happened had the contractor just plugged in the new routers and walked away at 4 pm on New Year’s Eve.
The implementation plan seems pretty straightforward, but getting to that plan required some work on part of the hosted provider. The executives on all sides including the customer didn’t effectively communicate and made erroneous assumptions.
I'm still excited about this project and hope that it continues to move forward. It's not my intention to portray anyone in a negative light, but just because hosted solutions can work it doesn't mean they will work in all situations. Getting them in successfully requires just as much planning and communications with all parties involved as do premise based solutions.
As it turns out, another 30-60 days is needed by Verizon to add QoS to the network, extend service to one of the remote offices and to ensure that adequate bandwidth (20 Mbps) is installed for the customers of the hosted provider sharing the same network.
My customer has also agreed to let my IT contractor come in, assess the local PCs, server and network to put together a plan to establish a domain controller, dictation/transcription file sharing across the network and to identify candidates for thin clients. A lot can happen in 30-60 days and I’m betting it will be positive now that we have synchronized everyone's thinking.
