The Socha-Gelbmann Electronic Discovery Survey: Time for a Change
The time has come for us, the creators of the Socha-Gelbmann Electronic Discovery Survey provider rankings, to kill our rankings. We intend to replace those rankings with an improved way to assess providers and their capabilities.
Why are we killing the rankings? We believe our survey rankings have reached the point where they no longer serve their original purpose. When they are announced, we are told, they can affect the share prices of publically held companies. They can have an impact on the ability of providers to obtain financing. They can be a key factor, sometimes the most important factor, in determining which provider is selected to take on a project or deliver a software program.
No set of generalized rankings should have this type of influence. Not ours. Not anyone’s.
A Bit of History
When we issued our first set of rankings six years ago, we felt they helped guide consumers entering an inchoate market. We offered the rankings as indicators of which providers appeared to be rising to the top. In the first year, as some may recall, we published a single set of rankings — a list of the top five electronic discovery providers.
Over time, we expanded the scope of the rankings. With the third survey, we extended the overall rankings from the top five to the top 20, divided into groupings of 1-5, 6-10 and 11-20.
The following year, we expanded the rankings even more. We drew distinctions between services and software providers. We offered service provider rankings based on assessments of their experience, their capacity, and the views of law firms and the corporations from whom we gathered data. We added software provider rankings based on law firm and corporate responses as well. We also added rankings for services and software providers by the Electronic Discovery Research Model stage.
This year we added a set of software provider rankings that looked at the usage of those providers’ software.
We always have sought to portray the service and software providers as objectively and as fairly as possible. This year, for example, we examined more than 350 separate categories of information. For each category, we gathered together all that data we had on that topic. We quantified the data, converting it from text to numbers where necessary. We developed something akin to a distribution curve and from that prepared a rating scale.
We recognize that perfection is never achievable, which is why we have always presented the rankings with a cautionary warning. Last year, we cautioned that consumers “should not choose a provider based solely on these lists, or solely on the basis of any other top-X list.” This year, we stressed that “anyone who makes buying decisions primarily on these rankings is a fool.”
Time for a Change
As noted above, we are told that the rankings have become so influential that now they affect share prices and the ability to get funding. We are hearing, as well, that consumers are using them as a substitute for the work they should be doing themselves — analyzing consumer needs and assessing what and whose services and software might meet those needs.
As we stated above, this is wrong. So much for the effectiveness of warnings.
Where Do We Go From Here?
For the next round of our survey, we intend to discontinue the rankings in their current form. We hope to replace them with a system that will more effectively allow potential consumers of electronic discovery services and software match up what they perceive to be their needs with what the providers have to offer.
As we have done with previous surveys, we will work on improving our survey to deliver the most value to our subscribers. We will retool how we present providers and their capabilities. We will continue to publish a high level view of key industry trends, market size and provider and their capabilities in Law Technology News.
We welcome your suggestions — let us know what would be more useful than the rankings. You can send your thoughts to us at firstname.lastname@example.org.