Selection of advanced TAR or CAR-based type search and review software is not an easy decision.
I had a chance recently to study the private manuscript of Professor Doug Oard on search design, Information Retrieval for E-Discovery. He is one of, if not the top independent expert on search in the world today. His manuscript written with William Webber is a survey that will be used to teach the next generation of information retrieval engineers, the ones who will help design the legal search software of the future.
Doug was also kind enough to try to explain key portions of this manuscript to me. The end result is a rather complex and lengthy blog entitled The Many Types of Legal Search Software in the CAR Market Today. Here is my takeaway list of twenty-five plus factors you should consider to evaluate the strength of search software. They should help you to evaluate the overall quality, effectiveness, and efficiency of the software on the market today.
- Search of content.
- Search of contextual metadata.
- Search of descriptive metadata.
- Search of behavioral metadata.
- Classifier design.
- Active query formulation features (building blocks).
- Annotated example training features (pearl growing).
- Random training abilities.
- Human judgment training features.
- Software uncertainty selected training.
- Balance in your classifier system between complexity and intuitiveness.
- Deduplication, near deduplication, and similarity searches.
- Concept searches, what exactly are they doing and how.
- Clustering features.
- Email thread and families.
- Quality control features.
- Quality assurance features.
- Confidentiality protection features, including privilege logging and redacting.
- Recommended method(s) of use (level four).
- Ease of operation.
- Clarity and efficiency of user interface.
- Customizability for particular licensees, projects, users.
- Response time (each entry and length of search processing).
- Reporting capacities and usefulness.
There are many other criteria, I know, such as information security, more specific user interface issues, reviewer and production issues, reviewer monitoring and interface, online access, etc. Software selection should also carefully consider the software support and training issues. Project management support is also critical, some say just as important as the software itself, especially for novice users of the search software.