Please enjoy this blog post authored by Carolyn Bragiel, Advisor, Legal E-discovery, Cardinal Health.First off, let me say that I do not consider myself an expert in metrics, and I speak from my personal experience! As part of a corporate in-house eDiscovery team, we have always collected metrics on our piece of the puzzle. We joke that this is how we prove our value to the company. When my boss approached me about taking over the collection of vendor metrics I was excited for the opportunity, but also scared and intimidated by the task! The initial point of this article was to find out how metrics are used, what is collected, and why these metrics are important. The more I dove into discussions, the more I found there were more questions than answers! This turned into brainstorming sessions!
I collect metrics every month. I think a lot of us do. I started with a spreadsheet that I sent out to my vendors every month, then, at times, I had to pester them to get the data back. This led to developing a more automated system that sent out reminders, and the users could go into the program, enter data and the formulas created pretty charts and graphs. I received great feedback from users on the ease of use, as well as feedback on how to make it even better. Great! My life is easier, the user’s life is easier. It’s all good…or so I thought!
Then I started thinking about what happens with all the data I collect. Who really looks at it, how is it used, how could it be used? My intention with this collection was to help our in-house counsel see statistics on their matters, track the budget, make any adjustment needed, etc. I also thought it would be great information for budget forecasting, planning, and contract renewal. As I continue this journey, I find that the data is not used to its potential.
Now, I will confess that I can be a bit of a ‘numbers geek’ at times, and I do enjoy looking at and analyzing the data. I know not everyone has the same enthusiasm as I do. Some people glaze over and get that ‘deer in the headlights’ look. Others may never look at the data I collect. Pretty graphs and charts do help to get the point across. After all, presentation can help keep people’s attention! So, what happens next? Is it the data put out of mind? Is it a waste of time for my vendors and myself? I reached out to some in-house counsel, vendors and outside counsel to get some answers.
The consensus seems to be that in a perfect world, the data would be used as I intended. Having a baseline for budgeting and planning, contract negotiation with vendors, evaluating performance and so forth. I received several answers that data is collected because we are told to collect, or we feel like we must collect some sort of metrics. In this day and age, we are all busy, so are we just created more work with no true purpose? I know, personally, when I show the data, it can be an eye opener, and there is always positive feedback, but then…radio silence!
One vendor I spoke with said that his approach when working with clients is to work backwards. What result is desired? Once that is determined, then you figure out how to get there. He also feels that for in-house legal teams, the legal operations team should be heavily involved in the process or at least be interested in the results.
Another vendor uses various their metrics for staffing decisions on new matters or adding staff to an existing matter if it grows or is more complicated than originally thought. They also use metrics to help with review decisions such as using AI vs linear review. They might also use the metrics with new clients to show cost savings associated with culling data, time management and budget estimates. This vendor also tracks review metrics and uses those on a case-by-case basis to see if adjustments need to be made anywhere on the review team.
Everyone I spoke with agreed that data collected should include data volumes, collected volumes, processed and reviewed volumes, and produced volumes as well as hours spent on review, including first pass, privilege, redactions, and QC. I also collect data on consulting hours, project management time and technical time. I have the contracted rates for all items I collect built in as well and can track overall spend. I use the data reported when I do invoice review to make sure everything matches up. I also can look at data for vendors who may have several matters with us, and I can compare across vendors as well, and I can look at overall spends.
What can we do with all of this? When we look at contract renewal or go through an RFP to update our preferred providers, we can use the costs as a negotiation tool, and set expectations. We can evaluate performance of vendors overall or across matters, especially in the review space. Now, I am sure a lot of you are thinking that no two matters are the same, so how is this fair? You are correct, no two matters are the same, but over time, we can get average numbers that are useful. Hosting numbers, after the initial ingestion normally stay consistent. Review numbers are pretty fluid initially, but then settle in as the review proceeds. As I see review numbers slowing down and eventually stopping, this is my cue to follow up and see if we can nearline, or archive data to save on hosting costs, or even close out the matter. We can use our data for cost savings in other ways as well. We send huge volumes of data to vendors, they process the data, ingest it, cull it, etc. Can we be doing more upfront to reduce the data volumes sent?
Other thoughts, as I chatted with people, were using the data for process improvement, and ROI on process improvements. When we implement a new tool, we can track when it was put into play and track cost savings associated with this new process. I think this is a great idea, and plan to work in some type of reporting on this.
Speaking with outside counsel, I found that review metrics are important to them. They can see how the review is going, and where they may be able to tweak their process. I think the addition of financial metrics can also help keep them accountable, as far as their piece of the budget. The last thing I find important, and others seem to as well, is how to gain traction and interest from our stakeholders! I plan to come up with some use cases for the metrics collections to present to stakeholders to show them what information we can gain, and how it can be used.
In wrapping up, what started as curiosity on my part lead to great learning and insights. I plan to implement some new things in my analysis (ROI tracking, case uses, better presentation and reporting) and find better ways to engage stakeholders in the process and use of the metrics.
I would like to also thank the following for their time and insights:Matthew L. Smith, Vice President of Business Development, ConsilioBradley Johnston, Senior Counsel, eDiscovery, Cardinal HealthAmy E Sellars, Legal Chief of Staff and Director of Operations, Sun Power CorporationAcorn Legal Solutions - Luke Riddle, Tracey Oldenburg, Amy DeFelippis, Lia Majid#CorpLegalDepartment#CorporateLegalOperations#ProfessionalDevelopment#Firm#DataManagement