Benchmarking becoming obsolete?

Today I was asked to give my opinion on the proposition “when outsourcing is benchmarking required to prevent a failure” by an editor of an IT magazine. My first thought was: of course it is, as including a benchmark clause in a contract was standard practice when I drafted a contract. But then I thought again and come to a less back-and-white conclusion.

Benchmarking before or during the contract period is common good and big business for companies like Gartner, Alsbridge and Compass, but given the increased complexity and tailoring of the solutions provided I wonder whether their usage is not restricted to a limited set of services. Services which soon get so common that many just buy them out of the cloud. Let me explain.

Many organizations outsourcing for the first time did not know what to expect from a supplier and often also have no clue about the cost of a workspace or server. They just knew the total IT budget and had a target to reduce it by 5% before next year. The benchmark company helped these companies in two way: allocate the aggregated budget to individual services and provide the average cost in the market for that same service. That way the organization knew whether there would be a positive business case.

Since than many things have changed. For starters have most organizations learned that selecting the cheapest suppliers is not always the best decision. The damage caused by crappy services and the invoices send for additional work ensure many business cases end up in the archive, together with the responsible IT manager. Both not to be seen again. As a result know companies that outsource for a second or third time what to expect and go for a more balanced contract. And as they have the price sheet of their existing supplier, it is easy to calculate any benefit by choosing a competing supplier.

At the same time are many standard IT services commoditizing to an extent that it is as easy as going to a grocery store. Server capacity, storage and a continuously increasing list of applications can be bought out of the cloud on a pay per use basis. How much price transparency do you want?

That leaves the company specific solutions and technology. Solutions which key business value are for example improving flexibility or speed-to-market of the business. How are you going to benchmark those qualities? How do you ensure you use the same definitions per company? Everybody can easily agree on a definition of ‘availability’ or ‘response time’, but defining ‘flexibility’ is already more complex (e.g. volume, service mix, time). The same applies to the cost. How do you create a peer group if there are maybe one or two companies which have similar services? In other words, the validity of the outcome of a benchmark deteriorates fast when complexity and tailoring increase.

As a result I believe that the relevance of benchmarking will decrease over time with a) companies leveraging experience from past deals, b) standard services becoming so commoditized that benchmarking does not add any value, c) IT services retained by companies become to specialized to benchmark as the peer group is too thin.

No shares in benchmark companies for me I guess ;-)