Please enjoy this blog authored by Amy Tom, Manager, eDiscovery Staffing, Morrison & Foerster.
Document Review Attorneys and Artificial Intelligence
Best Practices for Review
By Amy C. Tom, Esq.
If you have attorneys with varying levels of interest and experience in artificial intelligence, an expanding list of potential data sources, and a looming substantial completion deadline, you are in good company.
When planning an effective document review, here are some considerations and recommendations:
Leverage TAR
Consider the use of Technology Assisted Review (TAR). Test it on a subset of your potential data before the review starts for richness. If a large portion of the documents are not relevant, perhaps using AI makes sense. If you are new to Active Learning or a case team opts not to use it to drive the review, consider using it to identify coding inconsistencies (e.g., the TAR relevancy ranking is high but it was coded Not Relevant). Gauge the clarity of your review protocol and preemptively identify documents that hit on issues or are defined as key (e.g., Relativity aiR).
Staffing Your Review
The team that you recruit for your document review will greatly impact its quality. Proactively enlisting knowledgeable and experienced staffing vendors, Review Project Managers, and Document Review Attorneys results in fewer errors on First Pass Review, fewer hours making corrections on Second Pass Review, and lower costs.
Recruiters and Vendors
Characteristics of successful document review recruiters and vendors include: a proactive, collaborative, detailed, and transparent approach; established relationships with individual reviewers to minimize attrition; a robust repository of data and metrics from past reviews. Working with the same recruiter for all your document reviews is ideal, as your staffing requirements and preferences will be well established.
While a lower hourly cost may appear to provide the best value for your client, this may not be the case. Reviews priced below market rate may mean higher attrition and extra attorney hours spent onboarding replacement reviewers. Similarly, a vendor hosting your data may not necessarily be the most qualified vendor to staff your review. A vendor’s knowledge and experience for different review components may vary considerably.
Review Project Manager
Your Review Project Manager (PM) is integral to the effectiveness of the review. Ideally, the same Review PM will work on all your reviews. This way the Review PM can apply knowledge acquired on prior reviews to future ones to provide defensibility, ensure consistency, and expedite onboarding.
Successful Review PMs embody a proactive, problem-solving approach (e.g., rather than reporting that there is high attrition, providing several options to address the issue). Instead of simply providing metrics reports, the Review PM thoroughly analyzes them to identify and investigate any trends (e.g., if there is an unusually low or high review rate, determining why).
Document Reviewers
Attract the strongest reviewers. Identify reviewers with applicable subject matter knowledge and experience. For example, a reviewer who has worked on a government investigation may not have the technical background needed for a patent litigation review. Similarly, a strong First Pass Reviewer may also be well suited for the nuanced analysis required for privilege logging or may not. Reviewers who are actively engaged during the review (e.g., ask questions and escalate issues) are invaluable.
Financial incentives such as an above-market rate or more hours per week may assist you in retaining knowledgeable reviewers. Lower hourly rates can mean a less experienced team, more errors on First Pass Review, and more time billed for Second Pass Review.
Review Protocol and Training
A thoughtfully drafted and concise protocol will greatly facilitate an effective review. Circulate the draft protocol to the case team and the Review PM to elicit and incorporate feedback (e.g., revise unclear instructions to minimize confusion, streamline the coding pane). If you opt to have 20 issue codes, expect a slow rate of review, and adjust accordingly. During the Review Training, pull up examples of significant documents and explain why they would be tagged as “key.”
Engagement During Review
Respond to questions promptly so reviewers can course correct in real-time. Failure to do so means more documents being coded incorrectly on First Pass Review and impeding the efficacy of TAR. Ideally, Second Pass Review starts within a day of First Pass Review and runs concurrently during the entire review. Schedule check-in calls with the reviewers frequently at the beginning and periodically during the review to provide feedback and answer questions.
Establish thresholds that trigger Review PM action (e.g., # docs per hour, % overturn).
Enlist the Review PM to proactively monitor metrics and promptly resolve issues (e.g., unusually low or high review rate, precision across the team). Establish an escalation path if there are no improvements. Memorialize actions taken in daily reporting.
If team-wide tagging errors are identified on Second Pass Review (e.g., overly broad application of responsive tag), notify the Review PM to update the team. Conversely, if there are tagging inconsistencies on Second Pass Review, the Review PM will provide examples to the case team.
Cost Management
To stay within the prescribed review budget, include updated cost calculations in the daily reporting. Identify and address any change in scope that may impact the budget or deadlines.
Identify opportunities to decrease costs. For example, if a category of Not Relevant documents can easily be identified by file type (e.g., .bin) or subject matter (e.g., resumes), remove them from First Pass Review to decrease the number of reviewer hours. Another area where costs can be managed is via credits to the client when reviewers do not meet expectations or leave the project early.
Knowledge Retention
Establish a process for retaining relevant information and feedback. This may include: 1) potential improvements; 2) questions asked and answered during the review; 3) noting reviewers who performed well and reviewers who did not. Leveraging knowledge gained on every review will positively impact your management of future document reviews.
#200Level
#LitigationSupportoreDiscovery
#PracticeManagementandPracticeSupport
#ProfessionalDevelopment
#Blog