[NOTE: The survey whose results are summarized here was conducted in January 2002. A copy of  the February 2002 issue containing the full survey report may be ordered here.

We surveyed professional development managers in 134 law offices in the U.S., U.K., and Canada, of whom 37 (28%) responded, 36 private law firms and 1 government agency. Organization size ranged from under 150 lawyers to over 1,000 and averaged 498 lawyers.

The law offices employing upward evaluation numbered 21 of the 37 respondents, all of them private law firms; in one of these, upward evaluation is one component of a broader 360-degree feedback program. Median longevity of these programs was 1-3 years (7 firms), and ranged from less than 1 year (6 firms) to over 10 years (2 firms).

Of the 16 law offices that do not have an upward evaluation program, 2 are actively exploring the possibility, 12 are not currently considering it, and 2 have had a program in the past but are not currently using it.

Size. In the survey group there was no apparently correlation between the size of the organization and the existence of an upward evaluation program, although the average size of those with programs (513 lawyers) was slightly larger than average size of those without them (478). However, some of the smallest firms (under 150 lawyers) in the survey had programs in place, while some very large firms did not.

Location. U.S. law firms headquartered in states west of the Mississippi (86%) and the Northeastern states (75%) were the most likely to have upward evaluation programs in place.

Subsequent sections of the report also note other correlations for two designated subcategories of respondents: 11 "Home-Grown Systems," whose programs were developed and are maintained without external assistance; and 6 "Successful Systems," so designated based on system longevity and a combined minimum score on 3 leading indicators of program effectiveness.

At the 21 firms with upward evaluation programs, the most common arrangement is for all lawyer subordinates below the level of partner to evaluate all partner supervisors. At 12 firms, however, supervisors below the level of partner are evaluated, including non-lawyer supervisors at 1 firm. At 3 firms partners submit evaluations, and at 2 firms non-lawyers evaluate their supervisors.

Supervisor Receptivity. "Supervisors’ receptivity to the evaluation feedback" was rated as "Generally very positive" by 37% of the firms with programs, and as "Mixed" by 53%.

Evaluator Participation. The median level of evaluator participation was 50-75% (at 37% of the firms). Of the firms that achieved a higher level of participation, 11% received evaluations from 90% or more of potential evaluators, while 26% received them from 75-90% of evaluators. The remaining 26% of the firms achieved under 50% participation.

Evaluation Quality. The predominant ratings on the quality of the evaluations received (on a 3-point rating scale of High/Moderate/Low) were:

• Accuracy: High (67%)
• Constructiveness: High (62%)
• Completeness: Moderate (43%)

Computer Support. Twelve (12) firms have automated one or more process steps.

Outside Consulting Support. Seven (7) firms have used outside consulting help for one or more steps in developing and managing their programs.

Most Significant Benefits. The 37 survey respondents, both those with and those without upward evaluation programs in place, were in substantial agreement on the benefits such a program does or could be expected to confer. The benefits the majority of respondents rated as significant were:

• Feedback that helps supervisors improve (95%)
• Subordinate satisfaction from having a say  (84%)
• Management information on the general quality of supervision and areas for improvement (70%)
• Management information on individual supervisors for allocation of rewards and remedies (65%)

Major Disadvantages. The most significant disadvantages perceived by the majority of all 37 respondents were:

• Subordinates’ fear of participating and/or of providing candid evaluations (92%)
• Lack of follow-through on improving supervisors' performance (65%)

Again, on the above two disadvantages both the law offices with programs in place and those without them were in substantial agreement. However, the two groups’ views differed remarkably on two other potential disadvantages. The majority of the law offices without systems rated the following two disadvantages as "major":

• Supervisors’ disinterest in or negative reaction to the feedback they receive (75%)
• Damage to working relationships from negative feedback (56%)

By contrast, the majority of firms with systems did not view these two items as major difficulties. However, more of the offices with systems (29%) than of those without (19%) saw "Time burden for participants in the process" as a major disadvantage.

III. Strategies for Success

At the heart of most firms’ efforts to reduce subordinates’ anxiety about evaluating their supervisors are various techniques for protecting evaluator anonymity – most commonly, (a) by not attributing evaluation ratings or comments, (b) by maximizing participation, (c) by summarizing and/or editing the comments provided to supervisors so as to eliminate any that might serve to identify the evaluator, and (d) by using an outside provider to collect the evaluations and compile the reports.

Firms’ strategy for ensuring supervisor follow-up on improving performance typically involve strong leadership communication and intervention from top management. There was, however, significant disagreement as to what form the intervention should take: whether tangible rewards and/or penalties are necessary, or whether the evaluation feedback should be used solely for developmental purposes.

We took a more in-depth look at the 6 "Successful Systems" and asked their managers some follow-up questions.

The most striking thing about these systems is how very different they are in organization size and in program characteristics. They represent a wide range of possible differences in terms of (a) who is evaluated by whom, (b) what form the evaluations take, and how they are collected and reported, (c) whether computer and/or consulting support is used, and (d) whether the evaluations are used for developmental purposes, to determine tangible rewards and/or penalties, or both.

The common elements we could identify in these 6 systems were:

• protection of evaluator anonymity,
• an institutional commitment that is backed up by leadership and resources, and
• highly qualified and competent people to set up and maintain the program.

Copyright © 2002, Evelyn Gaye Mara. All rights reserved. This document may be printed for personal use only. Any reproduction, retransmission, or republication of all or part of this material is expressly prohibited without prior written consent from the copyright holder. Contact maraeg@profdev.com for reprint permission.