A single point of failure, or SPOF, is that one hardware or software gotcha that will bring a network down -- and if that happens on your watch, then you best start looking for a new job. Keeping the network up and running at all times is an imperative given the always-on, always-available expectations of doing business today.
To protect against SPOF, I've seen one CTO decide to run point-to-point fiber between closets rather than using a hub-and-spoke configuration from all closets back to the computer room. His method cost more but enabled easier recovery from a failed switch stack or fiber component. This is typical when trying to alleviate SPOF concerns: The easier you find the solution, the more it costs.
Seeking Out Vulnerabilities
The example I mention came up recently when my team was standing up a new firewall for an MPLS network. We were working with a high-end firewall, redundant links, and a hot standby chassis.
First we evaluated traffic from the firm's existing setup, gathering data to assess the connections and types of traffic passing through the firewalls. Because we'd be consolidating all Internet traffic through the firm's headquarters location, we needed to be sure we had rightsized the firewall and Internet links. We also needed to take growth into account, and make sure the firewall not only would support current but also future traffic volume.
We discussed known and potential vulnerabilities. For example, because the firm is in a hurricane zone, we asked: "What if the roof of headquarters blows off? Doesn't it make sense to relocate the server room from an upper to a lower floor?" That was a valid concern, but no closet on the lower floors could accommodate a server room. Additionally, space on those floors outside the closets was at a premium. Moving the server room was not a practical option.
We also considered whether we should use a fiber or copper connection for the carrier handoff. By spending $100 more and investing in the fiber connection, we could ensure that power issues would not traverse from the carrier to the customer premises equipment or vice versa -- plus fiber provides a better connection than copper.
For the fiber gear, we decided to use generic fiber transceivers instead of the switch manufacturer's transceivers. They proved to be just as good, but priced significantly lower at $99 compared to $299. We also decided to use uplink kits from this same lower-priced vendor -- but first we connected the firewall to a managed LAN switch using one of its uplink kits. It worked perfectly, and so we could use lower-priced gear here, too.
Ensuring Resiliency
All Internet traffic for this new MPLS network would route through one firewall. As such, the company would save in the subscription fees for intrusion protection services and deployment costs of numerous smaller firewalls and redundant links at all connected sites. However, we needed to ensure resiliency.
To do so, we mirrored the ports on the primary firewall but connected those ports for the secondary firewall to a different switch in the same stack. That way, should the switch serving the primary firewall fail, then traffic could still pass through the secondary firewall.
Alternatively, we could have deployed higher-end, high-availability switches with redundant power supplies, links, fans and chassis to avoid disruption in any switch stack. Again, cost factored into the decision. When the business weighed deployment cost against the risk of disruption, it opted to go with the less expensive option.
As for those switch stacks on each floor, deciding between hub and spoke or short fiber links between closets came down to the best choice for each situation. By adding a fiber cable from the fourth floor to the first floor, we could provide ease of recovery from a switch failure. With fiber patch cables, we could use spare fiber pairs as pass-through ports from one floor to the server room.
Lastly, we discovered that servers weren't using spare power supply. In a simple update, we added the appropriate power cord to each server and used the spare power supply to provide standby power. This was another low-end decision that could potentially have high-end benefits.
I recall custom homebuilder telling me that if he threw in Cat 5 cabling, brass fixtures and other upgrades then homeowners would no longer be able to afford the homes he built. The analogy works when considering SPOF protections, too. Avoiding one or multiple SPOF can be costly or cost effective. You have to apply good judgment and weigh costs against risk to figure out which is which along every step of the way.
Follow Matt Brunk on Twitter and Google+!
