Private cloud installations are an increasingly common component of enterprises' data center strategies. They complement traditional data center infrastructure and public cloud services (such as Amazon Web Services, Google Compute Engine, and Microsoft Azure).
Private cloud installations are essentially a DIY version of public cloud services. To build a private cloud, an enterprise purchases and then (on its own or through third-party providers) installs, hosts and operates dedicated equipment, network facilities and software designed to provide internal business units and customers with highly elastic, virtualized and pooled compute, network and storage resources; metered resource usage, and automated self-service provisioning capabilities. These are some of the essential benefits offered by public clouds, and they come without many of the risks some fear when moving to public cloud vendors' multi-tenant, shared infrastructures.
Attracted by the agility, operational flexibility, efficiencies and scalability of public cloud computing, wary enterprises resist a full leap to public cloud computing and turn to private clouds instead. The reasons often cited for this decision include:
• IT Security requirements that are incompatible with public cloud;
• Regulatory restrictions associated with certain types of data/applications;
• Custom requirements that do not map well to public cloud services;
• Control over service and security management that is unavailable to public cloud customers; and
• General nervousness associated with exposing sensitive company or customer data to public cloud services.
Some enterprises build their own private cloud environments. Many, however, increasingly turn to system integrators and other IT service providers to install, host and manage dedicated private cloud infrastructures--blurring the lines between public and private cloud.
When private clouds are built using third-party services, the resulting equipment acquisition, software licensing, hosting and professional services arrangements can be complex. Success depends on thoughtful planning. This article examines some of the important deal points enterprises should consider when developing and executing successful sourcing and vendor negotiation strategies for hosted private cloud transactions.
MANAGE YOUR EXPECTATIONS
Private cloud arrangements will not, and should not be expected to, replicate the flexibility, economic efficiencies, and (seemingly) limitless scalability of public cloud services. Public cloud computing is fundamentally a utility service that can rapidly and flexibly absorb increases and decreases in the required workload, with corresponding increases and decreases in metered service charges.
A private cloud is a purpose-built, virtual environment dedicated to a particular customer. It is not a utility service. In a hosted private cloud, the provider purchases and deploys dedicated equipment and software on your behalf. If your workloads reduce, then the service provider will still expect to be reimbursed (with margin on top) for the outlay. Unlike in a public cloud scenario, the charges in a hosted public cloud arrangement may not actually reduce in accordance with decreasing workloads; indeed, the charges may be based on the maximum workload you will need.
Enterprise customers often assume that service providers can provide private cloud instances on a far more flexible basis than is actually feasible. Private cloud services are not "productized" in the same manner as many services. Service providers do not routinely re-deploy or reuse infrastructure for different customers. Private cloud deals (including scope and pricing) are assembled on a semi-custom basis--often drawing from a list of equipment and software supported by the provider). The resulting hosted environment is bespoke to each individual customer instance, as is the pricing.
The dedicated nature of hosted private clouds means that the provider expects to recoup all of its outlay from each customer. The workload and cost flexibility that public cloud services offer are, therefore, incredibly difficult to replicate in private cloud deals.
Nevertheless, private cloud arrangements fulfill certain needs that public cloud services cannot. Understanding (and being sympathetic to) the areas where service providers have limited flexibility and, equally, knowing where the service providers can and should be flexible, will enable you to negotiate a best-in-class private cloud deal.
COMPETITION IS CRUCIAL
The most important factor in achieving a best-in-class deal is creating competition for your business. And the best way to drive competition is via a competitive procurement process involving multiple bidders.
Most of the deal and contract concepts discussed in this article should be tackled very early in the competitive process (e.g., by including them in a Request for Proposal document issued to multiple prospective service providers). Negotiating preferential positions while you still have multiple providers fighting for your business, rather than waiting until you've selected your preferred service provider and are close to the end of the negotiation process, will help you achieve the best possible negotiated outcomes.
PRICING AND COMMITMENT GUIDANCE
A key foundation of any best in class deal is competitive pricing.
Pricing for private cloud deals is highly customized, but typically covers the following pricing elements:
• Amortized capital costs (e.g., for the purchase of necessary hardware and software licenses, plus installation);
• Recurring costs for space, power and HVAC;
• Hardware and software maintenance;
• Managed services (ongoing monitoring and management of the private cloud infrastructure);
• Bandwidth/connectivity costs; and
• One-time costs for the upfront implementation and for service changes during the life of the contract.
Service providers often bundle cost components together (e.g., rolling capital costs and space, power and HVAC costs) into a bundled recurring charge that applies on a per server/appliance basis. Understanding the relationship between the underlying costs and the service provider's pricing construct will help you to negotiate improved pricing and enable you to more accurately compare pricing across multiple service providers' proposals.
A general pricing best practice is to keep the pricing structure aligned with the underlying cost base incurred by the service provider. As an example, you may be tempted to ask a service provider to offer pricing for private cloud services that more closely resembles public cloud pricing (e.g., per user, per instance, or per workload as appropriate for the use case). However, these pricing constructs are rarely aligned with the service provider's underlying costs and could create scenarios where the service provider is unable to recover its capital costs.
To eliminate the profit risk of these scenarios, service providers simply increase the offered pricing--resulting in uncompetitive pricing. Competitive pricing is always closely aligned to the service providers' underlying cost basis reflected in the core pricing elements referenced above.
Next Page: Principles for negotiations
Negotiate all charges for service changes up front
Providers will almost always try to levy additional one-time fees for changes to the private cloud infrastructure (e.g., to modify software configurations). One-time charges also typically apply when adding services or new resources, or increasing capacity of existing resources after the initial implementation. And don't be surprised to see higher charges for changes performed on-site, rather than remotely, even though the infrastructure is hosted at the service provider's facility.
The starting point for all such charges can be very high, and thus it is crucial to negotiate preferential rates rather than discover how high the charges are the first time you request a change. It is also worth negotiating fixed prices against a fixed menu of changes by type, instead of (or in addition to) hourly rates.
Avoid Overpaying for the Supplier's Upfront Capital Outlay
From the customer's perspective, it is perfectly reasonable to assume that the charges for the services will decrease once the service provider has recouped its upfront outlay on software, hardware and installation. Yet service providers often object to the concept, and certainly will not proactively offer pricing that declines after the upfront investment has been fully recovered through service charges and one-time charges.
It is, therefore, critical to lock in the appropriate price adjustment by setting in the pricing schedule the date or event that triggers the charge or rate reduction, and the actual amount of that reduction. Avoid hollow commitments to renegotiate the pricing later in the contract term, at which point you'll have limited leverage to negotiate a fair price adjustment.
Manage lock-in and Commitments.
Commitments and vendor lock-in arise in various ways in private cloud deals. Overall spend and volume commitments are readily identifiable and negotiable. You must, however, tackle them early in your vendor negotiations; don't wait to address them at the time you are drafting or marking up the contract.
Commitments can arise in other ways, too. In-service requirements are particularly constraining. In-service commitments occur when individual service components, once installed/activated, cannot be de-installed or deactivated until a certain period of time has elapsed. If the customer de-installs or deactivates a service before its minimum in-service period is satisfied, the customer is nevertheless obligated to pay the service charges for the remainder of the in-service period, even if it does not use the service.
For charges associated with the service provider's initial capital outlay, it is reasonable to accept a commitment to pay the associated charges for the time period over which the service provider has amortized the capital costs. But for most other types of services charges, minimum in-service periods should not apply (e.g., for management services, maintenance, space/power/HVAC charges, bandwidth/connectivity charges).
Bundled charges are also a source of concern. If amortized capital costs are bundled with managed services charges and space/power/HVAC charges, the financial penalties for not fulfilling the minimum in-service period should not be 100% of the bundled charges. It should, instead, be a lesser percentage of the bundled charge related specifically to the provider's upfront capital outlay.
It seems obvious, but including an express right to de-install services is a money saver. Service providers often structure contracts in ways that commit customers to retain (or at least pay for) service components for the entirety of the contract term. There may be no standard mechanisms or documented processes to de-install service components. This is another form of hidden lock-in/commitment.
Negotiating the notice period for de-installations also helps contain costs and align the customer's total charges with its resource utilization. Ideally, charges for service components will cease as soon as you provide the de-install order (regardless of when the service provider gets around to physically de-installing the service component), but service providers typically ask for 30 to 60 days' notice before charges cease. This long delay is antithetical to the benefit of cloud computing, where most of the resources ought to be provisioned and decommissioned using automated tools. Any particularly long de-installation dates ought to be limited to physical hardware removal.
Service providers typically spread their upfront capital cost outlay across the entire initial contract term in hosted private cloud deals. They do this in order to keep recurring charges as low as possible. This creates an added complexity when service volumes are expected to grow over the course of the contract. The time-in-service commitment for elements added part way through the contract term will extend beyond the end of the initial contract term. This deceases your flexibility and your ability to efficiently switch providers at the end of the term. Developing and negotiating options that properly tackle contract term-extending in-service commitments should be front and center during the commercial negotiations.
Minimize the service provider's rights to change the prices
In general, the provider should agree to fix the rates and charges (in the form of stabilized unit rates or fixed monthly recurring charges) for the in-scope services for the term of your contract. Appropriate and reasonable exceptions to the "fixed" pricing may relate to labor costs incurred for services provided on a time and material basis, but even this is highly negotiable.
Power-related charges represent a common, yet difficult, exception to stable private cloud pricing. Similar to traditional co-location deals, providers hosting private clouds usually attempt to reserve the right to increase their charges if their electricity costs increase. It is generally unrealistic to completely eliminate such rights. A better course is to focus on managing the impact:
• Any price increases should be in direct proportion to the increase in energy prices, and should only affect the proportion of individual charge elements that are related to the cost of power.
• Prices should go down as well as up.
• Prices should only increase in relation to a material change in supply costs.
• Service provider should only be allowed to increase the prices periodically (e.g., no more than once per year).
• There should be some burden of proof on the service provider that its costs have increased, and by how much.
• Although it does not typically convert into specific pricing parameters, customers should require their providers to disclose the power usage effectiveness ("PUE") at the data center where the private cloud is hosted before inking the deal, and then require the providers to regularly report PUE. Comparing this figure to industry standard can help inform your power pricing negotiations.
Nail Down Trial Period Pricing and Exit Costs
Most private cloud pilots or trial periods will still require the service provider to fund significant upfront costs, particularly on hardware and software. It is unrealistic to expect to be able to exit a contract after a pilot or trial period at no liability/cost, particularly if the service provider is not at fault (e.g., if you conclude that the workloads planned to be implemented in the private cloud environment are simply not suitable for a virtualized environment). Pilots and trial periods can be negotiated, but expect there to be shared risk and a shared cost burden.
Next Page: Fine tuning scope and performance
FINE TUNING SCOPE AND PERFORMANCE
Carefully document the scope of the services
For any outsourced or managed services, it is essential to carefully document the services to be performed and responsibilities to be fulfilled by the provider. Lack of clarity on the demarcation points between in-scope (no additional charge) and out-of-scope (additional charges apply) work can frustrate an otherwise good deal. Avoid misunderstandings and additional costs later by clearly defining at the start of the process, for instance:
• Will additional charges apply for routine changes to the services or are such changes included in the managed services fees?
• What end user support, if any, is included?
• Who is responsible for capacity management?
• Where is the dividing line between the provider's management responsibility and the applications that you will manage and install?
It is also useful to negotiate specific rights regarding the ability to grow the service, specifying an upper service volume/capacity that the service provider commits to make available if requested by the customer. Negotiating room to grow within the primary data center is particularly important where the private cloud environment is located in a single data center, not spread over multiple data centers.
Negotiate robust service level guarantees
Service providers' standard service level guarantees are notoriously weak. They are designed to look good based on a cursory review, and to avoid liability for meaningful service credits under almost any circumstances. Experienced customers negotiate custom service level guarantees that are aligned with the importance of the private cloud services to their business, and that suitably incents the providers to do all they can to maintain the availability and operability of the services. Key service level requirements to seek include:
• Overall availability of the private cloud services. Overall availability of the services, rather than just availability service levels for individual service components, is important. For example, if users cannot access any services whatsoever because of a cross connect issue, then a service credit based on a percentage of a small monthly cross connect charge is unlikely to be satisfactory. Availability of individual components, or time to restore individual components, should be a complementary service level to overall availability.
• Initial implementation guarantees. Service credits must be payable in the event that the implementation of the services is not completed within the agreed timeframes.
• Time to complete changes or add new resources. There should be guaranteed timeframes within which the service provider will complete standard service changes, upgrades or service additions (e.g., adding users or incremental infrastructure components). For true private clouds, many routine resource provisioning requests should be automated, and occur rapidly.
• Service availability during maintenance periods. As a basic principle, the service provider's maintenance activities should never mean the services are unavailable. A robust private cloud service will have sufficient redundancy that maintenance can be always performed without actually taking down the production environment.
COMMAND, CONTROL AND ACCOUNTABILITY
Maintain Control
In private clouds, enterprise customers lose most of the pay-as-you-go, flexible cost advantage available in the public cloud context. Inevitably (as discussed above), building a private cloud, or having one built for you, means you must pay for the licenses, hardware and implementation services used to stand up the environment, whether in the form of upfront purchase with capital, non-recurring implementation charges and/or fixed-duration recurring charges.
In exchange for this cost disadvantage, private cloud customers receive dedicated environments, which means they should have considerably greater control. In fact, greater control is often the principal driver for selecting private instead of public clouds.
An important element of control is transparency into critical IT service management processes. In private clouds, the customer's change and security management requirements should not presumptively yield to the provider's standard processes.
In public clouds, customers are beholden to the provider's pace of change and upgrade cycles, and must accept much less visibility and participation in service management of the provider's multi-tenant, shared infrastructure. Since private cloud customers pay for the privilege, they get to dictate critical aspects of change management. For example, you should prohibit your provider from introducing material and adverse changes into the private cloud environment; require all testing and upgrades to be coordinated with you; and preserve the right to reject upgrades or changes you may find undesirable.
Other important examples of hosted cloud contract controls include:
• Your provider should not be allowed to relocate data centers or move the environment within the selected data center without your consent;
• Because private clouds are ordinarily purpose-built for customers and there are usually considerable financial commitments to get them up and running, you should obtain a commitment from your service provider to support the environment for the term; any provisions that give the provider the right to discontinue service (except in the case of your uncured material breach) puts at risk your private cloud investment, and should be removed;
• You should have full access to perform vulnerability and penetration testing, and to conduct audits of your private cloud environments for security, data privacy and regulatory compliance purposes; and
• The contract should draw clear lines of responsibility for, and use of, the test environment, and document policy-enforced workflows for moving items from test to the production private cloud environment.
Fairly Allocate Responsibility and Liability for Security Incidents
A third-party provider that operates, hosts, and manages private clouds for an enterprise customer has actual control over the physical--and much of the logical--security of the environment. Importantly, however, the customer also bears responsibility for good security practices with respect to its access, and use of the private cloud, including (ordinarily) management of the applications themselves. Although it is often the most critical factor in moving to private clouds, clearly describing in the contract the allocation of responsibility for security management between provider and customer is often overlooked.
Drawing these lines in the contract documents in advance of work commencing can be difficult, but doing so is essential to effective compliance management. It allows the parties to expose any potential misunderstandings, and then identify and close the gaps before entering into contract.
Working out the allocation of operational responsibility is hard work for the technical and security-focused deal teams, but it is not usually fraught with controversy. Getting providers to actually accept financial responsibility for damages caused by their failures to perform security and privacy-related obligations assigned to them can, however, be very difficult. Providers typically look to exempt themselves from any liability for loss, corruption, or improper disclosure of customer data and applications, even when caused by the provider's contractual breach, negligence or misconduct. Enterprise customers, on the other hand, expect these sorts of liabilities to be uncapped. Both providers and customers are willing to walk away from transactions over the issue.
There is no single solution to this dilemma, and your risk management team ought to have a clear playbook on the degree of risk your company is willing to accept (with fallback and bottom line positions), ideally before commencing negotiations with any specific provider. As part of the playbook, you should, at the very least, only do business with providers that are willing to accept financial responsibility for losses resulting from their security failures.
To close deals with risk-focused but otherwise reasonable suppliers, you also have to be comfortable setting maximum exposure limits for your suppliers, subject to historically uncomfortably-narrow exceptions (e.g., uncapped liability for fraud, willful misconduct, violations of law, and certain types of remediation costs, with other types of damages capped at an agreed upon multiple of the contract's annual value).
CONTINUITY AND BUSINESS DISRUPTION
Private clouds are often selected in lieu of public clouds when sensitive, highly regulated or other critical customer data is involved. This ordinarily means the applications destined for private clouds serve mission-critical or competitively significant business processes. For these critical applications, designing the environment for high availability operations and tight security is the best protection. In hosted private cloud transactions, threats to business continuity are also key contractual considerations.
Some suppliers like to reserve liberal rights to suspend services when they, in their sole discretion, determine suspension is appropriate. Suspension rights of this sort should be identified, and removed. In a private cloud context, a supplier's right to suspend the customer's access to, or full use of, its private cloud environment should only arise when the customer fails to cure a material breach of contract or the customer engages in malicious conduct that materially threatens security, stability or normal operation of the vendor's infrastructure or the services it provides to other customers.
It is important to note that contractual suspension rights are distinct from security incident management. To respond to and remediate security incidents, suppliers may need to temporarily restrict access until the situation is resolved.
Provider termination rights, like suspension rights, ought to be very narrowly constructed. Termination should only be permitted for material breach of the negotiated terms and conditions where that breach remains uncured after an appropriate cure period has lapsed.
Non-payment by customers as a termination event can be particularly tricky. Providers rarely accept materiality as a precondition on non-payment related terminations. Customers, partly in recognition that provider billing practices are notoriously error prone, want some leeway for payment delays. The typical compromise allows providers to terminate for any non-payment, subject to prescriptive notice periods and the customer's right to withhold payment of any amount it disputes in good faith.
When the contract expires or is terminated for any reason (including the customer's breach), suppliers should be contractually bound to render migration assistance services that allow the customer to transfer the private cloud environment and related services to another provider or bring them in-house in an orderly fashion; migration services include continued provision of the in-scope services for a defined period (e.g., 9 to 12 months depending on the complexity of the environment), along with additional handoff activities (e.g., documenting the elements and configuration of the environment, transferring software licenses and hardware assets using a pre-negotiated pricing formula).
When the termination is the result of the customer's non-payment of undisputed amounts, it is reasonable for the provider to condition the migration services on the customer's pre-payment of a portion of the service charges to be provided over the migration period.
PRIVATE CLOUD SERVICES ARE GROWING, AND HERE FOR SOME TIME
Ultimately, as public cloud services mature and evolve, enterprise customers will become more comfortable adopting these services for sensitive and mission-critical services and data. We expect that to happen over a relatively long horizon. In the meantime, enterprises will continue to deploy private cloud installations over the next several years to gain the advantages of cloud computing innovations.
Hosted private cloud services represent a significant proportion of private cloud installations, and are expected to lead enterprises to greater adoption of public cloud services. Many vendors offering both private and public cloud services are highly motivated to win hosted private cloud deals to strongly position themselves for future public cloud services. The cloud providers' eagerness to sell cloud services, along with fierce competition amongst private and public cloud providers, allow enterprise customers to negotiate best-in-class transactions and obtain market leading services, by understanding the most important negotiable points, and proactively applying the rapidly evolving cloud deal best practices.
Ben Fox is a Managing Director of TechCaliber Consulting, LLC, a global IT and telecom consultancy that advises the world's largest companies on strategies for reducing their costs for telecom and IT products and services. Marc Lindsey is a partner in Levine, Blaszak, Block & Boothby, LLP, the leading firm representing large users in the negotiation of information and communications technology-related transactions.
Ben and Marc will be further discussing the negotiation of private cloud deals in a free webinar on April 8. For further details and to register please visit https://www3.gotomeeting.com/register/836193702.
Ben and Marc will be further discussing the negotiation of private cloud deals in a free webinar on April 8. For further details and to register please visit https://www3.gotomeeting.com/register/836193702.
