Legal Technology News - E-Discovery and Compliance Blog

Predictive/Computer-aided Coding Feed

March 12, 2013

EDI-Oracle Study Update

It has been a few weeks since our last update. We have extended the time to respond for the participants. While some thought the extension was unfair to those that have submitted on or before the due date, Pallab, the professors, and I all want as many data points as possible in the evaluation. That said, we are taking note of those teams that submitted early.

Again, we should remind all participants that time is one of the factors in the subsequent evaluation by Oracle. Therefore, a quicker turnaround is preferable and no participant should withhold submitting results unless the additional time is required to complete the review. While there is no penalty for submitting after the first deadline, Oracle will look favorably upon those that finish quickly.

Also, if there were tasks required by the protocol that your firm would not have normally undertaken in a real-life review (categorizing privilege for NR docs), please identify that task as a break-out on your model invoices with an explanation. You may submit your model invoices at you leisure, but no later than May 1.



March 04, 2013

Search Software: 25 Things to Consider

4-levels-pyramid Selection of advanced TAR or CAR-based type search and review software is not an easy decision.

I had a chance recently to study the private manuscript of Professor Doug Oard on search design, Information Retrieval for E-Discovery. He is one of, if not the top independent expert on search in the world today. His manuscript written with William Webber is a survey that will be used to teach the next generation of information retrieval engineers, the ones who will help design the legal search software of the future.

Doug was also kind enough to try to explain key portions of this manuscript to me. The end result is a rather complex and lengthy blog entitled The Many Types of Legal Search Software in the CAR Market Today. Here is my takeaway list of twenty-five plus factors you should consider to evaluate the strength of search software. They should help you to evaluate the overall quality, effectiveness, and efficiency of the software on the market today.

  1. Search of content.
  2. Search of contextual metadata.
  3. Search of descriptive metadata.
  4. Search of behavioral metadata.
  5. Classifier design.
  6. Active query formulation features (building blocks).
  7. Annotated example training features (pearl growing).
  8. Random training abilities.
  9. Human judgment training features.
  10. Software uncertainty selected training.
  11. Balance in your classifier system between complexity and intuitiveness.
  12. Deduplication, near deduplication, and similarity searches.
  13. Concept searches, what exactly are they doing and how.
  14. Clustering features.
  15. Email thread and families.
  16. Quality control features.
  17. Quality assurance features.
  18. Confidentiality protection features, including privilege logging and redacting.
  19. Recommended method(s) of use (level four).
  20. Ease of operation.
  21. Clarity and efficiency of user interface.
  22. Customizability for particular licensees, projects, users.
  23. Response time (each entry and length of search processing).
  24. Reporting capacities and usefulness.
  25. Expense.

There are many other criteria, I know, such as information security, more specific user interface issues, reviewer and production issues, reviewer monitoring and interface, online access, etc. Software selection should also carefully consider the software support and training issues. Project management support is also critical, some say just as important as the software itself, especially for novice users of the search software.

January 15, 2013

Audio Recording of EDI-Oracle Study Training Session

Just a quick update to answer questions about Friday's training session. EDI will release the audio recording some time later this evening. Please remember, that the recording is subject to commitments outlined in the NDA, Agreement, and Confidentiality Agreement with Oracle and its outside counsel.

January 14, 2013

1.6 Million Documents Released: EDI-Oracle Study Updates

File_cabinet_tower400As many of you know, EDI has been working on its second computer-assisted review (CAR) study, comparing the results of an actual document review in a real litigation to CAR providers.

After almost two years of planning, last week the EDI Oracle Study team released the entire dataset of over 1.6 million docs to the first wave of participants. Top performers stand to gain Oracle’s e-discovery business. A link to the press release about the data is attached to this post.

Download EDI Oracle Study Press Release Jan 8

On Friday, January 11, 2013, the team had its first training call with outside counsel. Over 60 representatives from the participants dialed in. The goal of the training was to familiarize participants with the legal issues surrounding the matter. The original complaint was filed under seal — but we are working on what we can discuss in public.

Prior to and during the call participants submitted dozens of questions about the matter — mostly focused on the data load and technical issues. The EDI and Oracle team are working diligently on what we thought the technical team would be able to answer by today, but it is looking like we will respond to technical questions on Wednesday 1/16/2013.

I will post weekly with updates on the status of the study here on EDD Update. I will Tweet from @patrickoot and post on LinkedIn when I make the updates.

Image by Redjar

December 13, 2012

EDRM Publishes CARRM Framework

EDRM published Wednesday its Computer Assisted Review Reference Model (CARRM) framework diagram, which can be downloaded here.

EDRM's CARRM represents the joint efforts of computer-assisted review providers — Automony, an HP Company; Daegis; Exterro; Falcon Discovery; FTI Consulting; kCura; KPMG; Kroll Ontrack; NightOwl Discovery; and Recommind — as well as leaders from Bowman & Brooke; DLA Piper (U.S.); Littler Mendelson; and Quarles & Brady. Click here for a complete list of EDRM Search participants.

Follow us at EDRM's CARRM section as we add an explanation of the model, descriptions of each stage in the model, and more.

November 09, 2012

Judge Peck's Decision Not to Recuse Himself Upheld


U.S. District Judge Andrew Carter of the Southern District of New York upheld Magistrate Judge Peck's decision not to recuse himself from the Da Silva Moore, et al. v. Publicis Groupe SA and MSLGroup litigation, Evan Koblentz reports. Plaintiffs had asserted Peck had conflicting interests in the use of predictive coding software due to public statements and appearances favoring the review method.

Read the article on LTN online.

Image by Monica Bay

November 01, 2012

Throwing a Wrench in the Document Review Machine

Binary_suit_400Computers against humans — is this the debate being waged around predictive coding to determine the future of document review? if this is the debate, is it what legal practitioners should really be focusing on in e-discovery?

In a recent New York Law Journal article, Steve Green and Mark Yacano of Hudson Legal question two sources widely cited to support the use of technology-assisted review. One source is Laura Grossman and Gordon Cormack's "Technology-Assisted Review Can (and Does) Yield More Accurate Results Than Exhaustive Manual Review, With Much Lower Effort" from the Richmond Journal of Law & Technology; the other is the 2009 Text Retrieval Conference (TREC) Legal Track Interactive Task study.

Continue reading "Throwing a Wrench in the Document Review Machine" »

October 29, 2012

Del. Chancery Court Judge Orders Predictive Coding

Welcome_to_delaware400There is a new predictive coding case in Delaware Court of Chancery that could have a big impact on the attitude of corporate America to technology-assisted review. EORHB, Inc., et al v. HOA Holdings, LLC, C.A. No. 7409-VCL (Del. Ch. Oct. 15, 2012). This case involves a complex multimillion-dollar commercial indemnity dispute involving the sale of Hooters, a well-known restaurant, famous for its chicken wings, beer, and other things.

The judge hearing the dispute, Vice Chancellor J. Travis Laster, is also well-known, but for something completely different, namely surprise rulings. This case adds to that reputation. He completely surprised the lawyers in this case with a surprise bench ruling that they all use predictive coding and share a common vendor. Here are his exact words:

Continue reading "Del. Chancery Court Judge Orders Predictive Coding" »

FTI Consulting, Huron Legal Offer Predictive Coding

October is proving to be a busy month for predictive coding in the e-discovery industry. On the heels of Daegis and Exterro announcing PC offerings, two more e-discovery vendors have entered the fray.

Fti_consultinglogoInternational business and legal advisory firm FTI Consulting, Inc., announced the launch of a new Predictive Discovery service by its technology practice group, reports Sean Doherty. The new service promises to reduce review time and cost as well as unite people and processes with predictive coding software — familiar language to anyone watching technology-assisted review products. Doherty spoke with Joe Looby, senior managing director in FTI's technology practice, to see what sets FTI's Predictive Discovery apart from other offerings.

Continue reading "FTI Consulting, Huron Legal Offer Predictive Coding" »

October 17, 2012

Daegis, Exterro Add Predictive Coding

Two more e-discovery vendors hopped on the predictive coding bandwagon this week, which may have some speculating at what point the technology will be omnipresent in the industry.

Daegis400Roseville, Calif.-based Daegis Inc., a company that hosts its software for law firms and corporate legal, announced Acumen, its machine learning technology that will be available as a standard feature for subscribing clients and also for customers on a per-gigabyte model, reports Evan Koblentz. Acumen takes what Doug Stewart, vice president of technology at Daegis, describes as a "process-driven" approach.

Continue reading "Daegis, Exterro Add Predictive Coding" »

September 24, 2012

Predictive Coding Vendors Duel for 'Dummies'

E-book_on_shelf400Legal professionals seeking a predictive coding primer have two free competing guides to choose from, both with the same title, "Predictive Coding for Dummies." The dueling "for Dummies" are being distributed by e-discovery vendors Symantec (Clearwell) and Recommind.

So which one's authentic? The answer, it appears, is ... both.

Continue reading "Predictive Coding Vendors Duel for 'Dummies'" »

August 31, 2012

Resistance Is Not Futile

IMG_3150I'm home with a broken ankle today and so was able to watch the stock market reaction to the release of Federal Reserve Chairman Ben Bernanke's speech on fiscal policy. Tens of thousands of brokers and investors were eagerly awaiting this speech to try to forecast the direction of the stock market. Many were ready to immediately execute trades based on the content of this writing.

I watched CNBC and the market reaction to the speech. The market almost immediately dropped 100 points, just minutes after the speech was released to the public and before he had even begun reading his speech. No human could possibly have read his speech that fast. At best they could have skimmed a few paragraphs. Yet millions, perhaps billions of dollars changed hands as the market plummeted. Then, as if by magic, in the next hour the market climbs back up that 100 points lost, and goes even higher. What the hell happened? Bernanke did not depart from his published speech.

A CNBC reporter on the floor of the New York Stock Exchange (I think it was Bob Pasini) provided a very interesting explanation, one which I predict will not be again repeated and may even be hushed up. I will not forget it because it made perfect sense to me and has implications for legal search in e-discovery.

Read the full commentary on LTN online.

Image: courtesy of Ralph Losey

August 29, 2012

EDI, Oracle Team to Put TAR to the Test

Oot_patrick2_400The Electronic Discovery Institute is ready to undertake an academic study of technology-assisted review and has chosen the study's chief scientists, Stanford University's Peter Glynn and Gerd Infanger, writes Monica Bay.

In an announcement Tuesday night, Patrick Oot, special counsel for e-discovery at the SEC and co-founder of EDI, said the study would build on research by The National Institute of Standards and Technology (NIST)'s Text Retrieval Conference (TREC) Legal Track and EDI's original study on document review. Oracle's litigation team will partner in the study, which will run tests on actual data from a closed matter.

Will the technological tools fare better than human reviewers? Turn to LTN online for the full scoop.

Image by Monica Bay

August 22, 2012

'Kleen' Comes Clean From Predictive Coding

Code_road_400Plaintiffs in Kleen Products v. Packaging Corp. had sought to force defendants to employ predictive coding technology to ensure the accuracy of document production. Now, in the filing of a stipulation of the parties and an order signed by Judge Nan Nolan, plaintiffs have agreed that keywords can be used to search for documents relevant in electronically stored information collected before October 1, 2013, writes Sean Doherty.

Read the full story on the pages of LTN online.

Image by

August 17, 2012

Predictive Coding Watch: 'In Re: Actos'

Past_present_future400Predictive coding is being put to the test in court again — this time in the Western District of Louisiana. According to a post by Greg Buckles on the eDiscovery Journal, a case management order by Judge Rebecca Doherty frames a "Search Methodology Proof of Concept" to examine the viability of Equivio's predictive coding tool Relevance for review and production in the matter — the defendant's e-discovery provider Epiq Systems uses Equivio Relevance.

The case is In Re: Actos (Pioglitazone) Products Liability Litigation, a multidistrict litigation consolidating 11 civil actions for pretrial proceedings. The plaintiffs allege that Actos, a prescription drug for the treatment of diabetes type 2, increases users' risk of developing bladder cancer — a risk the plaintiffs allege defendants concealed and failed to adequately warn consumers about.

Continue reading "Predictive Coding Watch: 'In Re: Actos'" »

June 27, 2012

FJC Pocket Judge Update

Fjc_logo128The Federal Judicial Center -- the research and education agency of the federal judicial system -- has published the second edition of its influential booklet, Managing Discovery of Electronic Information: A Pocket Guide for Judges, by Barbara Rothstein, Ronald Hedges, and Elizabeth Wiggins. The 48-page publication updates the previous 2007 edition, and can be downloaded free from the center's website. It covers a range of topics, from explaining the difference between conventional paper discovery and electronically stored information, to providing tips on a judge's role. The booklet also includes a five-page glossary, mostly derived (with permission) from The Sedona Conference Glossary: E-Discovery & Digital Information Management (3d ed. 2010).

While reaction has been largely positive, there are some rumblings of discontent about the new publication. Read the full story here.

What do you think about the update?

Image: Federal Judicial Center.

June 20, 2012

Peck to Participate in U.K. Panel on Predictive Coding

Peckpanel51712Predictive coding (aka technology-assisted review or computer-aided coding) crosses the Atlantic to London for a June 27 panel about technology's role in cutting e-discovery, or e-disclosure, costs.

The panel, sponsored by Epiq Systems, will feature Magistrate Judge Andrew Peck of the Southern District of New York, (left) who recently penned an order refusing to recuse himself from the controversial litigation, Da Silva Moore. Plaintiffs had challenged his neutrality, partly because he advocates predictive coding on industry-sponsored panels. Joining him in the debate will be Senior Master Steven Whitaker of the Queen's Bench Division, whose judgment in Goodale v The Ministry of Justice is regarded as a landmark U.K. decision about e-disclosure. Laura Kibbe, Epiq's managing director of document review and expert services , will moderate.

The event will include arguments about the defensibility of technology-assisted review, the effect on law firms' fees when TAR is used -- and how law firms can expect to be penalized in costs if they fail to match up to the expectations of the court, Epiq advises.

The event is open to legal professionals at law firms and corporations. Register for the event on Epiq's website or contact Epiq at

Press release.

Image: Monica Bay

June 06, 2012

New Hire at DOAR, Jackson Lewis Turns to Kroll Ontrack

Laptop_code400An industry consultant and a law firm involved in the brouhaha around predicitive coding in Monique da Silva Moore, et al. v. Publicis Groupe SA and MSLGroup made moves in e-discovery this week.

DOAR Litigation Consulting, a plaintiffs consultant in Da Silva Moore involved in the drafting of its ESI protocol, announced the addition of Gerard Britton as a managing director of its discovery consulting practice. "The fact that DOAR is advising clients as experts at the highest levels on cutting edge legal issues is what most appealed to me," says Britton of his new position.

Jackson Lewis, who represent defendant Publicis Groupe SA in Da Silva Moore, selected Kroll Ontrack as its preferred provider for e-discovery, which gives the law firm Kroll's discovery and document review, workflow processes, early case assessment tools, and review technology. In exchange Jackson Lewis will recommend Kroll to its clients. "[The] relationship with Kroll Ontrack will enable Jackson Lewis to focus on its core competency of practicing labor law, resolving complex disputes, and mitigating risk to organizations," says Losey.

Image by

May 23, 2012

The 24/7 Andrew Peck Show

Greetings from LegalTech West Coast in relatively-smog free and sunny Los Angeles. Day one (Tuesday May 22) was full of energy and standing-room only presentations with, as usual, tons of e-discovery in the seminar rooms (and on the busy exhibit hall floor). It's not too late to stop in for Day 2 (today, May 23); it's at the downtown Westin Bonaventure and one day tickets are available if you want some education, demos, and CLE credits.

Topic A among the L.A. EDD crowd, as expected, is the closely-watched Da Silva Moore case, where U.S. District Court Magistrate Judge Andrew Peck has ordered the use of computer-assisted coding (aka predictive coding). Law Technology News is one of those closely-watching, our most recent story is here.

NelsonOn the blawg-o-sphere, attorney Sharon Nelson, (left) president of Sensei Enterprises, has been monitoring the DSM sideshow in her ride the lightening blog. See her latest post, "Da Silva Moore: EDD's Version of Keeping Up with the Kardashians," is  here.

Peck-5-17-12-bPeck and his Second District colleague Lisa Smith gave a terrific program Thursday evening at the New York City chapter of Women in Discovery -- demonstrating his trademark wit and substantive knowledge. See "Federal Judges Preside Over Women in E-Discovery Meeting."

But Tuesday's LTWC keynote address reminded us that Peck isn't a one-trick pony judge doin' nuttin' but e-discovery 24/7/365. (He did win LTN's 2011 Champion of Technology Award). Peck's turf also has included overseeing another matter that kept lawyers Keynote1chewing their fingernails -- to say nothing of anybody with a dollar in a bank. Kevin Genirs (left). In his keynote address yesterday, "2008 v. 2012: Lessons from the Lehman Brothers," described how Peck oversaw the frantic, sleep-deprived, mega-lawyered transactions under excruciating deadlines that ultimately resulted in the sale of Lehman Brothers ("for the price of our building") to Barclays, praising Peck for recognizing the global impact of the sale on the stability of financial markets, and Peck's non-stop efforts to get the deal consummated. Genirs, who had been general counsel, investment banking, at Lehman's, was one of the 10,000 Lehman workers who had new jobs with Barclays when the sun rose the next day.

Later this morning, check out for my colleague Michael Roach's  report on "Under Fire: Defending and Challenging Technology-Assisted Review," which featured Irell's Tom Werner, O'Melveny's Jeffrey Flower, Oracle's Pallab Chakraborty, and moderator Andrea Gibson, from Kroll Ontrack (the sponsor of the Litigation Technology Track." Yes, DSM and Peck were front-and-center in the discussion!

Meanwhile, you can keep up with all the substance and occasional drama about Peck on our website's home page, and its E-Discovery/Compliance "channel" -- and right here at EDD Update. Onward to Day 2! 

Update 5/24: Oooops. Who knew there were two Judge Pecks in the Southern District of New York! Actually, many -- and Silicon Valley consultant Mark Michels, a member of LTN's editorial advisory board, was the first to advise me that I goofed. It wasn't Andrew Peck, it was Judge James Peck, of the U.S. Bankruptcy Court. But you get my point.

P.S. Here's Michael's story 

Photos: Monica Bay, except for Sharon Nelson (courtesy of ride the lightening)

LegalTech West Coast: Day One

As LegalTech West Coast rolled into Los Angeles Tuesday at the Westin Bonaventure, it's a good opportunity to bring your e-discovery knowledge up to date from exhibits and educational sessions.

MikeprogramIn a track sponsored by Kroll Ontrack Tuesday, "Litigation Technology: Trends, Technology & Innovations," two panels piqued my interest. The first, titled "A 'Stormy' Subject ... Exploring Cloud-Based E-Discovery," was moderated by Wayne Wong, (top right) managing consultant at Kroll, and features Scott Sachs, (left) e-discovery attorney for Atkinson Andelson, and Adam Sand, (front) associate general counsel for and former general counsel of ZL Technologies, an e-discovery and compliance software provider. The 10:30 a.m. discussion examined both sides of e-discovery in the cloud: the issues involved in collecting and preserving data stored in cloud-based models as well as using software as a service-based e-discovery tools versus on-premise software.

With Da Silva Moore, Global Aerospace, and Kleen Products bringing predictive coding into the courtroom, "Under Fire: Defending and Challenging Technology-Assisted Review" at 1 p.m. is topical. It was moderated by Andrea Gibson, product director of Kroll Ontrack, with speakers Tom Werner, associate at Irell & Manella, Jeffrey Fowler, partner at O'Melveny & Myers, and Pallab Chakraborty, director of e-discovery at Oracle. This is the year predictive coding stormed into the courtroom and the technology continues to stir controversy. Even proponents of the technology at law firms and discovery service providers I've spoken to have said they'd limit the technology to first-pass review or even culling the opposing side's production -- but not yet for full review. The panel addresses this, as well as other technologies that assist the review process, watch for my report on

The track's third panel at 3:30 p.m. was "Exploring Hot E-Discovery Trends: FRCP Amendments, Social Media, and Emerging Case Law."

Photo: Monica Bay

May 17, 2012

More News on Plaintiffs' Effort to Recuse Peck

Red_clip_papers_400In the latest round of Monique da Silva Moore v. Publicis Groupe and MSLGroup, Magistrate Judge Andrew Peck denied an amicus brief in support of the plaintiffs' motion for his recusal, Sean Dohery writes.

The brief was filed by Richard E. Flamm, a Berkeley, Calif.-based attorney and author of Judicial Disqualification: Recusal and Disqualification of Judges retained by the plaintiffs as an expert witness. Peck denied the motion to file the brief, writing "an alleged amicus brief that is paid for by plaintiffs is hardly necessary or appropriate." Read the full story on LTN online.

For more on the plaintiffs' recusal efforts, read their reply memorandum supporting the motion for recusal (.pdf) and its supporting exhibits (.pdf), filed May 10.

Image by Monica Bay

May 15, 2012

TREC on Hiatus for 2012

Hexadecimal_counting400With a new data set not yet ready and its 2011 results nearly two months late and counting, TREC Legal Track's annual document review project for testing new systems has been canceled for 2012, reports Evan Koblentz. In related news, Recommind confirmed that it's been asked to leave the project for prematurely disclosing the company's 2011 results.

Part of the U.S. Text Retrieval Conference, TREC Legal Track is used by researchers from academic and corporate sectors to test the precision and recall of newly developed technologies. Read the full story on LTN online.

Image by Tilman Piesk

May 10, 2012

'Da Silva Moore' Plaintiffs' Discovery Objections Continue

Business_man_code400From the not-exactly-surprising department: Plaintiffs in the now-famous Monique da Silva Moore, et al. v. Publicis Groupe SA and MSLGroup case are objecting to Magistrate Judge Peck's denial of their request to stay discovery until plaintiffs' motions and objections have been resolved. Our technology editor, Sean Doherty, posted the full story at Law Technology News.

Image by

May 04, 2012

'Da Silva Moore' Defendants Oppose Motion to Recuse

Binary_glasses_man400The defendants in Monique da Silva Moore, et al. v. Publicis Group SA, et al. (Case No. 11-CV-1279), on April 30 filed a formal opposition to the plaintiffs' motion to recuse or disqualify Magistrate Judge Andrew Peck. The defendants ("MSL") have also requested that the court award attorneys' fees and costs incurred in filing their motion.

Read the full article, "Defendants in 'Da Silva Moore' Oppose Motion to Recuse Judge," on LTN online.

Image by

May 02, 2012

Predictive Coding Takes: the Court and the Market

PredictiveResponses to Andrew Carters' recent affirmation of Judge Peck's decision authorizing the use of predictive coding in da Silva Moore continue to roll in.

For more opinions from legal practitioners and industry veterans on what this all means for legal technology, read Evan Koblentz' article for LTN online, "Take Two: Reactions to 'Da Silva Moore' Predictive Coding Order."

In the midst of all the prognostications about machine learning and the discovery process, New York-based litigation support provider Empire Discovery announced its adoption of OrcaTec's Document Decision Suite, highlighting its predictive coding capabilities. OrcaTec's technology is being put to use in the source of last week's other judge's order involving predictive coding, Global Aerospace.

Read about the OrcaTec deal here.

Image by

April 24, 2012

Va. Judge Orders Predictive Coding

In what could be a milestone moment, a state judge in Virginia yesterday ordered that defendants can use predictive coding, despite plaintiff's objections that the technology is not as effective as purely human review.

In Global Aerospace Inc., et al, v. Landow Aviation, L.P. dba Dulles Jet Center, et al, Loudoun County Circuit Judge James Chamblin wrote:

Judge"Having heard argument with regard to the Motion of Landow Aviation ... it is hereby ordered Defendants shall be allowed to proceed with the use of predictive coding for purposes of processing and production of electronically stored information."

Chamblin, in allowing 60 days for processing with "production to follow as soon as practicable and in no more than 60 days," continued:

"This is without prejudice to a receiving party raising with the court an issue as to the completeness of the contents of the production or the ongoing use of predictive coding."

There are many questions to ponder. Will a county court order have any impact on state or federal courts? What about impacting the current predictive coding controversies in the da Silva Moore and Kleen Products cases? Which software companies are involved? Can the plaintiffs take any further appeal action?

Update, 6:20 PM: The full story is now posted on Law Technology News.

Image by

Sign Up for the E-Discovery and Compliance Newsletter

An Affiliate of the Network

From the Newswire

Sign up to receive Legal Blog Watch by email
View a Sample

Contact EDD Update

Subscribe to this blog's feed

RSS Feed: LTN Podcast

Monica Bay's Law Technology Now Podcasts are also available as an RSS feed.

Go to RSS Subscribe page

March 2013

Sun Mon Tue Wed Thu Fri Sat
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30

Blog Directory - Blogged