Over-inclusive vs. Under-inclusive

These self-explanatory terms are applicable in several contexts within the legal domain. With respect to e-discovery search, they refer to search practices that either; A) do not return all the documents that may be relevant to the particular search (under-inclusive), or B) return more than just the documents that are relevant to the particular search (over-inclusive). While it is generally accepted that all search methodologies will have a certain level of each, too much of either is a recipe for disaster. Judge Paul Grimm’s opinion last year in the Victor Stanley case provides an interesting perspective on the over-inclusive – under-inclusive issue, while also providing useful insight on how to enhance the defensibility of search in general.

Keyword Search

Thanks to our friends at Google and Yahoo!, most of us are more or less familiar with how keyword search works. Simply type in a word or phrase, and the search engine returns the documents that contain that word or phrase. – no rocket science there. Of course, even keyword search has its complexities. While Boolean operators (AND, OR, NOT) are relatively straightforward, “wildcard” or “star” searching (e.g., “infra*” returns any document containing a word that begins with those characters), stemming (e.g., “jog” with stemming returns any documents containing “jog”, “jogs”, “jogger”, “jogged”, “jogging”) and “fuzzy” searching (returns any document containing various spellings and/or misspellings of a search term), have made keyword search a lot more susceptible to some of the issues that make search such a controversial and scrutinized subject within the e-discovery space.

Concept Search

Concept search, which has been getting a lot of buzz in the industry lately, tries to overcome the under-inclusive issue that is perceived to be inherent in traditional keyword search. The theory is that by simply looking for specific words, keyword search will miss documents regarding the same subject matter but containing synonyms of the keywords rather than the keywords themselves. For example, one document may refer to an employee as having been “fired”, while another may refer to the employee as having been “terminated.” While these documents clearly address the same topic, it’s easy to see how keyword search could miss one or the other. That said, the flip side of the under-inclusive coin is over-inclusiveness, which can lead to many other problems, including skyrocketing costs.

Concept Categorization

Concept categorization, also known as concept or document clustering, gained mainstream notoriety in the e-discovery space with the success of Attenex and Stratify. What these tools and others like them do is automatically bundle documents of similar subject matter together, which subsequently increases reviewer productivity. While this notion certainly has merit, at some point the industry began to think of concept categorization as a search process. Concept categorization looks at each document, applies some logic (e.g, Bayesian), and based on that logic makes a determination on which documents have similar subject matter. Concept categorization does not take in a search term and retrieve documents based on that parameter.

“Black-Box”

Traditionally search companies like the aforementioned Google and Yahoo! guard their search algorithms as extremely confidential trade secrets. This approach also applies to enterprise search companies, many of who have had their technology adopted by the e-discovery industry. The problem is that secrecy is not a good thing when is comes to the question of how evidence was found. Imagine a homicide proceeding where the prosecution answers the question of how they found the murder weapon with, “Sorry Judge, it’s a secret.” Needless to say, that probably wouldn’t sit well with most judges. Unfortunately, that’s essentially what attorneys that use “black-box” search tools have to say when the same question is posed regarding documents in an e-discovery case. Of course, there are ways to mitigate this issue, and consulting with opposing counsel and agreeing on search methodology before hand is the key component for doing so.

“Transparent”

The idea behind transparent search is to overcome the exact issue presented by black-box search algorithms. By bringing the operations that search engines typically do behind the scenes to the forefront, and, in some cases, allowing users to tweak those operations, transparent search is supposedly better suited for e-discovery and the defensibility requirement inherent in any litigation. Take for example the stemming capabilities offered by many keyword search tools. In a typical black-box search tool, a stemming enabled search for the term “jog” might automatically return any documents containing the words “jog”, “jogs”, “jogger”, “jogged”, and “jogging.” In a black-box solution, these stem variants would be largely unknown and the user would not have any ability to tweak them. Transparent search on the other hand, would allow users to adjust the stem variants as necessary, therefore limiting the search to only the desired terms and providing full “transparency” into the process. While the benefits of transparent search are clear, the technology is quite new and has yet to be proven in the mainstream e-discovery space. Furthermore, transparent search creates more complexity and additional areas of debate among opposing counsels. Time will tell if the promise of transparent search truly comes to fruition.

E-Discovery search is a complex and ever evolving topic. Like so much of the technology in the industry, e-discovery has had to make due with search tools that were not designed for use in litigation or a legal setting, which has led to lessons being learned the hard way. However, the development of e-discovery specific search technology, together with more refined search methodologies, the future is bright, and search promises to play an increasingly important role in e-discovery.

As one could logically surmise, the primary advantage of web-enabling an existing desktop application using Citrix is that doing so allows geographically dispersed access to the application without having the redevelop the software as a web application. The cost-benefit here is an assessment of whether the redevelopment costs outweigh the added efficiencies a true web application would bring to bear. Sometimes — depending on many other variables that I’d rather not get in to here — the answer is yes. Conversely, very often the answer is no, which is why so many enterprises have undertaken development projects to modernize their applications and convert them to web-based.

In a hosted ESI review context, the primary advantage of web-enabling desktop applications using Citrix becomes a moot point. The users of these applications (GCs and Law Firms) are not the owners of the technology, and therefore the cost-benefit analysis I previously mentioned is not applicable. In a hosted/on-demand/software as a service model, users’ only concern is having a tool that facilitates maximum efficiency at a minimum cost, and a desktop application that has been web-enabled via Citrix certainly doesn’t fit this bill. I’m not trying to bash Citrix here — it certainly has its “sweet spot” — I just want to be clear that in a hosted ESI review context, this practice is down right laughable.

So what makes desktop applications that are web-enabled via Citrix less efficient than true web applications?

1. Higher Cost: Citrix (and similar terminal service technology) require costly licenses for the software to be deployed on both the server and all client machines. In addition, more hardware is required to support the same number of users/data.
2. Steeper Learning Curve: A desktop application can employ a multitude of user interface/navigation constructs. True web-applications on the other hand, all have similar navigation constructs that have become second nature for most users.
3. Worse Performance: Because web-applications have been designed and developed to work over the web, they perform better than desktop applications in that environment (faster response times, page loads/screen refreshes, data transfers, etc).
4. Decreased Integrations: Web applications are built on protocols and standards, making them relatively easy to integrate with other web applications. Desktop applications were never meant to integrate in this way, therefore they can not.

With the evolution of Web 2.0, web applications are becoming much more functional and responsive, and have enhanced the overall user experience. My guess is that for hosted ESI review, web-enabled desktop applications will eventually be phased out completely, which will increase the overall efficiency of e-discovery. Let’s hope our client community demands that this occur sooner rather than later.

My issue with the term “native review” is that it implies these tools are displaying documents in their native application, for example, xyz.doc displayed in MS Word. Of course, that is not the case, and to build a review platform that actually launched the native application of every document that might exist in a collection is completely outside the realm of feasibility. “Native” review tools still display documents through a viewer, just as TIFFs are displayed. However, “native” review viewers are more sophisticated in that they are able to render hundreds of document formats, and make them appear very similar to the way they would be displayed in their native application. In some instances (for MS Office documents for example), these native viewers even have hooks into the native applications that help refine the rendering, furthering the “native” review myth.

To take things a step further, virtually all “native” review tools use some variant of Oracle’s (formerly Stellent’s) Outside In viewer technology. Whether it be Avantstar’s Quick View Plus (used by Ringtail), or Daeja System’s ViewONE Pro (used by iCONECTnXT and LitScope), or some other flavor, it’s all Outside In behind the scenes. This is why all native review tools can claim virtually identical capabilities in terms of the number of file formats their application can render (which is up to over 400 at this point). Of course, Outside In is much more than just a viewer, but I’ll save further expansion on that topic for a later post.

Finally, let me reiterate that I am in no way questioning the benefits of “native” review — I am a huge proponent of this practice and believe any vendor recommending TIFF (or any other uniform image format) reviews is doing their client a disservice. I just think it’s important that both vendors and client’s alike understand what “native review” really means, which will (hopefully) help us all get on the same page.

If I were on the client side of the fence, I think it’d be a HUGE struggle to sift through the various vendors who pitch their services to me on a daily basis. The sheer number of e-discovery vendors — a number that increases daily — is one thing, but perhaps a bigger challenge is the overwhelming variance among companies offering e-discovery services. To illustrate this point, let’s look at the top 20 e-discovery service providers based on the 2008 Socha-Gelbmann Survey. Companies within this group include:

• Two of the “Big Four” accounting firms
• A business advisory consulting firm
• A computer forensics software company
• Several pure e-discovery ASPs
• A company that has evolved from paper-based litigation support services

Keep in mind this variance is only among the top 20 providers, and there is an estimated 500 providers in the space. So how exactly does one select from this list? On the law firm side, many have turned to a formal RFP/Response process. This is probably long overdue, but unfortunately many of the RFPs that I’ve seen are poorly structured/written and therefore have limited value. On the corporate side, many turn to independent consultants — of which there seems to be an endless number these days — to help vet/qualify/disqualify providers.

Any way you slice it, picking the the right provider is a daunting task, and I for one do not envy those charged with doing so.

For me, the timing of this read was somewhat ironic as just a couple days prior I had learned that a large project I had been tracking for a few weeks had been awarded to one of eDiscovery "big boys" (as per the norm on this blog I will not name names). My source on this deal told me that the vendor did not budge off their standard rate, which is known to be somewhere between $2000 - $2500 per GB (which includes cull down, extraction, clustering/concept search, and perhaps 3 months hosting). Learning that this particular law firm (and most definitely many more) would still allow one of their client to be charged $2000 - $2500 per GB for these services was equally, if not more baffling to me.

So what's the bigger joke, $2500 per GB or $4.30 per GB???