In 2004, the Roetzel & Andress executive management team approved a major desktop upgrade that included Microsoft Word 2003 and add-in tools.  Part of the anticipated return on investment (ROI) for the upgrade relied upon improvements in our attorney-to-secretary ratio, so our plan had to support the technology and training approach that could help accomplish that.

With more than 200 lawyers in ten offices across Ohio, Florida and Washington, the desktop upgrade was a significant investment for our firm.  Our goals were to improve our document production tools and create an ongoing educational program for using the tools in our new environment.  We also needed to evaluate the impact and value of the upgrade by measuring to what extent the technologies were being used and adopted.

The Measurement Challenge
Our desktop upgrade from Microsoft Office 97 to Office 2003 was a great opportunity for improving document production at Roetzel & Andress.  In the past, guidelines for document best practices did not exist, styles were rarely used and "cutting and pasting" from unstable documents consistently created problems for our users.

The upgrade included new capabilities for document automation, metadata removal and comparison to encourage the design of documents according to best practices.  It also included custom toolbars to simplify the firmwide application of numbering and styles and to overcome any document corruption or instability that might arise.

In preparation for our upgrade, we reviewed helpdesk call records to learn what challenges our users were facing and to help plan our training approach with the new tools we had selected.  After the upgrade, we wanted to be able to measure the adoption level of the new technology and its impact on productivity, as well as gain insight into the effectiveness of our training program.

Choosing Our Analysis Approach
Like most firms, doing more with less is a constant goal at Roetzel & Andress.  Our measurement initiative had to be cost-effective and repeatable.  After exploring several possibilities, we decided that a software application for analyzing documents against best practices was the best option for the project.  This would provide the foundation for creating our measurement capabilities. 

To execute the initiative, we assembled an internal team to work with the software vendor.  This team included our training coordinator and a group of our office administrators.

The measurement criteria we used were very important factors in our success.  We also focused extensive time and energy developing and refining our communication plan throughout the assessment process.  We developed the process and the document characteristics we wanted to measure and then focused on how we would report those findings to our end users.  The software vendor instructed us how to set up the system and create the rule sets we would use, transferring the knowledge we needed to maintain the system internally.

We created a document production dashboard with the key metrics we wanted to track.  The dashboard provided Roetzel & Andress management with utilization metrics that tracked four basic document characteristics:

• Percentage of template usage
• Percentage of styles usage
• Percentage of consistent formatting
• Percentage of correct numbering usage

The following are the measurement best practices we developed and how we arrived at them:

Measure both production and test documents.  One very important lesson we learned was the benefit of analyzing both production and test documents in order to gain a comprehensive view of document production skills.

Initially, our document analysis was done against a sample of production documents pulled from our document management system.  The results of that analysis seemed to indicate that many secretaries were not using the templates and lacked the skills to apply styles and numbering to documents:

• Template usage - 52 percent
• Style usage - 38 percent
• Formatting consistency - 45 percent
• Numbering application - 88 percent

As this analysis did not appear to provide an accurate assessment of our secretaries' document production skills, we also decided to develop a targeted skills assessment test document.

Focus on a representative skills assessment.  The first skills test document we created, an agreement with table of contents and page numbering, was given to a diverse test group that represented multiple practice areas.  The test group determined that the agreement was too corporate-focused and that a test document was needed that represented a work product example that any secretary would reasonably be expected to create.  We surveyed our secretaries about the types of documents they were creating and determined that a letter would be the best starting point for the test document.  We then created a sample letter with a letter template and styles, automatic numbering, direct formatting and a header/footer.

This collaborative approach to selecting the test document helped drive buy-in of the concept of skills assessment.  We didn't want to negatively affect morale by fielding a test document that was too difficult or that didn't reflect real-world responsibilities.  Also, using a letter for the test allowed us to generate positive, constructive feedback, not just a score.

The assessments showed that the secretaries actually did have the document production skills they needed, a key fact that would not have been realized had we limited our analysis to production documents.  This led us to focus on trying to understand how to bridge the gap between the secretaries' document skills and how they actually use those skills on our production documents.

Balance skills assessment with interviews.  Office administrators met with secretaries one-on-one to review and explain the scores from the test documents.  During these conversations, the office administrators learned that secretaries often worked pragmatically to prepare documents and meet client deadlines.  One way they shortened document preparation time was to take shortcuts on document styling.

This proved to be an invaluable step in the process, as our secretaries were able to ask questions and gain knowledge about firm best practices.  And we also gained significant insight about the day-to-day challenges that they faced with the new system.

The meetings provided several deeper insights into the scores:

• Style usage – We learned why styles often weren't being used even though it was clear that staff understood how to use them.  We commonly heard comments such as, "I don't bother restyling a document received from a client if I know that it's going back to the client."
• Numbering usage – The assessment found that the automatic numbering feature was often not used.  During the interviews, several secretaries commented that lawyers refer to certain paragraph numbers when they dictate.  If automatic numbering is used, then a secretary implementing the lawyer's dictation cannot be sure that the numbering in the document corresponds to the numbering identified by the lawyer during the dictation.
• Regional utilization variance – In some of our offices, the production documents scored relatively low compared with others.  Yet we knew that the secretaries, as assessed with the test document, were highly skilled.  We learned that the variance could be due to the type of law being practiced.  For example, real estate and trusts/estates are both heavily forms-based.  As a result, in those practice areas, the secretaries would often just go back to the form and create a new document if changes were made to an existing document by a client.

Results to Date
We now know much more about our document production strengths and weaknesses.  We've developed the in-house analytical capability to identify specific training needs and deliver the educational programs most likely to improve desktop upgrade utilization.

These targeted training programs create new professional development opportunities and enable our secretaries to learn new skills that can be used across practice areas.  This, in turn, helps us make progress on our objective to improve the attorney-to-secretary ratio using our work-team secretarial support model.

About our authors

Jeff Farkas is Chief Information Officer at Roetzel & Andress and is responsible for the planning, management and delivery of the firm's information and telecommunication systems.  Jeff has 13 years of experience in the legal environment and specializes in IT and project management.  He can be reached at jfarkas@ralaw.com.

Brian Hall is Vice President of Knowledge Partnership and Marketing at Microsystems, where he is responsible for ensuring that Microsystems' customers best use Knowledge Partnership to create and maintain high-performance document production environments.  Brian can be reached at brianh@microsystems.com.