Skip to content

Calibrating evaluations: how to prepare and run the meeting

The calibration meeting step by step: who takes part, what data each manager brings, how to read the distributions, how to document the changes and what research shows about calibration’s effects.

By the Pulsika team Performance Published 10 minutes read

The evaluations of two managers, A and B, are joined by lines to the same list of criteria: results, behaviours and evidence.

Calibrating evaluations is the meeting in which the managers of comparable teams discuss their proposed ratings before these are communicated to people, so that the same level means the same thing across the whole company. It does not split people into quotas and does not replace the direct manager’s judgement. The article describes the preparation (what data each manager brings), the meeting minute by minute, how to read the distributions on a worked example, how to document each change and what research shows about the effects of calibration, including the unwanted one.

Quick answer. A calibration meeting brings together the managers who evaluate comparable roles, with a facilitator from HR. Each manager comes with the proposed ratings and with evidence. The group compares the distributions, discusses the cases at the extremes and on the border between levels, changes a rating only on the basis of evidence and writes down the reason. The direct manager then communicates the result and can explain it.

Definitions: calibration, distribution, leniency, central tendency

Calibration is the group review of the evaluations proposed by several managers, so that the same standards are applied to everyone. A 2011 SHRM survey, carried out with APT Metrics among 510 members from organisations with more than 2,500 employees, found that 54% of respondents worked in organisations that held formal calibration sessions.

The distribution of ratings shows how many people each manager has on each level. Leniency is an evaluator’s tendency to give higher ratings than the results would justify; severity is the opposite. Central tendency is the clustering of ratings in the middle of the scale, often so that the evaluator avoids uncomfortable decisions. The signs of each error in the ratings are described in the article on errors in employee evaluation.

Forced distribution (also known as the “vitality curve” or “stack ranking”) imposes fixed percentages on the levels, for example 20% at the top, 70% in the middle, 10% at the bottom. Calibration can use a reference distribution as a check signal, but that is not the same thing as a mandatory quota.

What research shows about calibration

The study published by Demeré, Sedatole and Woods in Management Science (2019) presents itself as the first empirical evidence on the role of calibration committees and analyses the data of a large multinational organisation. The conclusions from its abstract:

  • the calibration committees changed relatively few ratings (the adjustment rate was 25%), and downward adjustments were more frequent and larger than upward ones;
  • they lowered the ratings of managers who rated above average and raised those of managers who rated below average;
  • they improved consistency between managers and reduced leniency, but accentuated central tendency;
  • they changed less often the ratings of people further from the committee members in the hierarchy, that is, they left the decision to those who knew the person better;
  • managers learned from the adjustments what the organisation expected.

The SHRM survey gives a picture of practice: 63% of respondents said ratings change “rarely” after calibration, 35% “regularly”. The reasons for the changes: inconsistent evaluations between managers (69%), new information about performance (45%), fitting a planned distribution (39%).

Remember. Calibration corrects the differences between managers, but it can push people towards the middle. When you discuss a top case or a weak one, ask explicitly: “Are we lowering this rating because the evidence calls for it, or because it is different from the others?”

Calibration, forced distribution or no calibration?

ApproachWhat it doesAdvantageMain risk
No calibrationEach manager evaluates aloneFast; the decision stays with whoever knows the personThe same level means different things in different teams
Calibration with a reference distributionThe group discusses the ratings; the distribution is only a signalCommon standards, explainable decisionsCentral tendency; the influence of those who speak more persuasively
Forced distributionFixed percentages on the levels, applied to every teamRemoves widespread leniencyPenalises good and small teams; encourages internal competition

For small teams, fixed quotas are almost impossible to apply fairly: in a team of five people, 10% means half a person. A company-wide reference distribution, used as a question and not as a rule, avoids the problem.

How to prepare the meeting

Who takes part

The participants are the managers who evaluate comparable roles (for example, all the store coordinators in a region), their manager and a facilitator from HR. The facilitator does not vote: they keep time, ask for evidence and record the decisions. Keep groups to at most 8 managers; beyond that, each case gets too little time.

What each manager brings

  • the proposed rating for each person, filled in before the meeting;
  • 2–3 pieces of evidence for each rating at the extremes and for the borderline cases: results, examples of behaviour, feedback from internal customers;
  • the context that changes the interpretation: long absences, role changes, indicators with missing data.

What HR prepares

HR prepares a table with the distribution of ratings by manager, the average and the percentage of people on the top levels. Next to them, it puts the unit’s result (for example, achievement of plan), so the group can see whether the ratings move in the same direction as the results.

A worked example: how to read the distributions

Hypothetical example. Fictitious data. Three store coordinators propose ratings on a scale from 1 to 5 (4 and 5 are the top levels). Team A has 8 people, B has 10, C has 7.

ManagerProposed ratingsAverageOn levels 4–5The unit’s achievement of plan
A5, 5, 4, 4, 4, 4, 4, 333 ÷ 8 = 4.137 out of 8 = 87.5%98%
B4, 3, 3, 3, 3, 3, 3, 3, 2, 229 ÷ 10 = 2.901 out of 10 = 10%112%
C5, 4, 4, 3, 3, 3, 224 ÷ 7 = 3.433 out of 7 = 42.9%104%
Total25 people86 ÷ 25 = 3.4411 out of 25 = 44%

The table does not say that A is wrong. It says where to start the discussion: A’s unit had the weakest result and the highest ratings, and B’s unit the best result and the lowest ratings. The group asks for the evidence for A’s five people rated 4 and for B’s seven people rated 3.

The outcome may be that the ratings stay: A’s team may be very good in a unit with low footfall. The distribution is a question, not a verdict. If the ratings change, each change has its reason written down.

How to run the meeting

  1. Opening (5 minutes). The facilitator recalls the scale, the level descriptions and the rule: a rating changes only on the basis of evidence.
  2. The distributions (10 minutes). The group looks at the table prepared by HR, without names, and notes where the differences between managers are not explained by the results.
  3. The extremes (20–30 minutes). Each manager presents, in at most 2 minutes, the people rated 5 and 1–2. The others may ask questions; they may not rate people they have not observed.
  4. The borderline cases (30–40 minutes). For people between two levels, the question is: “What would you need to have seen for the rating to be one level higher?” Compare with the level description, not with the colleague.
  5. The final check (10 minutes). Look at the distribution again after the changes and ask whether any category of people (part-time, new in the role, with long leave) has systematically gone down.
  6. Closing (5 minutes). The facilitator reads out the list of changes and the deadline by which the managers communicate the ratings.

Added up, the stages come to 80–100 minutes for a group of 25 people. Plan at most two hours; if the list is longer, split it into two meetings, so that the last cases do not get less attention than the first.

Practical tip. Let the direct manager have the last word when the group brings no new evidence, only impressions. In the study by Demeré and colleagues, the committee changed less often the ratings of people far from it in the hierarchy, leaving the decision to the manager who knew them better.

How to document the decisions

For each changed rating, keep a line in a simple register. The example below uses fictitious data.

The personProposed ratingFinal ratingThe reason (evidence)Who asked for the change
Consultant, team A43Service standards met, but the individual target achieved at 91%; the level 4 description requires exceeding the targetThe regional manager
Consultant, team B34Target exceeded three months in a row, two documented compliments from customers; the manager had applied a stricter standard than the descriptionThe HR facilitator

The register serves three purposes: the manager can explain the rating, the employee can get an answer if they challenge it, and at the next calibration the group can see whether the same kinds of cases come back.

Caution. Romania’s Labour Code (Codul muncii, Law no. 53/2003) requires the internal regulations to contain the criteria and procedures for the professional evaluation of employees (art. 242(i)). If ratings can be changed in a group meeting, the evaluation procedure should say so. Discussing people is also processing of personal data: bring only the information needed for the evaluation, in line with the data minimisation principle in GDPR Art. 5(1)(c). Check the wording with the company’s lawyer.

After the meeting: communication

The direct manager communicates the rating, not the committee. The explanation “that is what calibration decided” shifts the responsibility and lowers trust. If the rating went down, the manager says what evidence counted and what would have changed the result. For difficult conversations, see the evaluation conversation when results are poor.

Common mistakes

  • Mandatory quotas on small teams. Correct it: a company-wide reference distribution, used as a question.
  • Ratings changed on impressions. “He seemed weak to me at the March meeting” is not evidence. Correct it: the evidence rule, applied by the facilitator.
  • Comparing people instead of comparing with the level. Correct it: ask what the level description requires, not which of two people is better.
  • The drift towards the middle. Correct it: check at the end whether the extremes have disappeared without written reasons.
  • Calibration without prepared data. Correct it: the distributions and the units’ results reach the participants before the meeting.
  • No trace of the decisions. Correct it: the register of changes, with the reason for each one.

The evaluator’s individual errors (the halo effect, recency, contrast) are dealt with in the article on errors in employee evaluation.

Frequently asked questions

What is an evaluation calibration meeting?

A calibration meeting is one in which the managers who evaluate comparable roles present their proposed ratings and the evidence behind them, before the ratings are communicated. The aim is for the same level to mean the same thing in different teams. A facilitator from HR keeps time, asks for evidence and records each change with its reason.

Does calibration mean forced distribution?

No. Forced distribution imposes fixed percentages on the levels, regardless of the team’s actual results. Calibration compares the ratings with the level descriptions and with the evidence, and a reference distribution can serve only as a signal of where a question is worth asking. A very good team can rightly have more people on the top levels.

Who should take part?

The participants are the direct managers of the people evaluated, for comparable roles, their line manager and a facilitator from HR. The employees evaluated do not take part. Groups of more than eight managers become hard to run and are split. Each participant discusses only the people they have observed or about whom they have concrete evidence.

How often is calibration done?

Calibration is done once per evaluation cycle, before the ratings are communicated: yearly or every six months, depending on the company’s rhythm. Monthly scores calculated from indicators do not need calibration; there, the scorecard’s rules ensure consistency. Calibration is useful above all for the parts evaluated through the manager’s judgement.

What do I do if the group disagrees with the direct manager?

If the group brings new evidence, the rating is discussed again against the level description and may change, with the reason written down. If the group has only impressions, the direct manager keeps the rating. Unresolved cases go to the line manager, who decides and records the reason. Do not vote on people.

Does the employee find out that their rating was calibrated?

The employee should know in advance that the evaluation procedure includes a group review, and the procedure should be written into the internal regulations. The direct manager communicates the final rating and the evidence that counted. If the employee challenges the rating, the register of changes allows a concrete answer, without vague references to “the committee’s decision”.

Can calibration introduce other errors?

Yes. The research by Demeré, Sedatole and Woods found that, although it reduces leniency and makes ratings more consistent between managers, calibration accentuates central tendency. There can also be the influence of those who speak more persuasively, or a disadvantage for less visible people. At the end, the facilitator checks the extremes and the categories of people who have systematically gone down.

What to do next

Before the next evaluation cycle, prepare the table of distributions by manager with the units’ results alongside, and schedule a pilot meeting for a single group of comparable roles.

Was this article useful?

See how Pulsika works for your team

Try every module free or book a demo with our team. No card needed.

or let us set up a trial company with your structure