Tier list maker

A tier list is a ranking with the argument left in: things dropped into rows from best to worst, and the rows are the answer. If you want one right now, the free tier list maker here needs no account and nothing to install. The whole board is carried in the link, so you can send it to somebody or print it.

Made alone, a tier list settles nothing. It is one person's opinion arranged neatly, and the neatness is doing work the thinking has not done. The same list ranked by everybody in the room at the same moment is a different object.

Francis Galton collected 787 usable guesses at the weight of an ox at a West of England fat stock show, and published the result in 1907. Kenneth Wallis went back through Galton's own worksheets and found that the ox's dressed weight was 1,197 pounds and that the mean of all 787 guesses was 1,197 pounds. The crowd's error was zero. The middlemost guess, which is the statistic Galton chose to publish rather than the mean, was eleven pounds out once Wallis had corrected both figures.

Why one person's tier list settles nothing

A room's ranking beats the average person in it, and there is a line of arithmetic that says exactly how much by. Scott Page, in a deck hosted by the National Academies, calls it the diversity prediction theorem and writes it as crowd error equals average error minus diversity.

Read it backwards. A room that already agrees with itself adds nothing to what any one member already knew. The gap between people is not noise sitting around the answer. It is the thing paying for the accuracy. The theorem also promises less than the slogan does: the comparison is with the average person in the room, not with the best one.

That arithmetic needs a right answer to be right about, and most tier lists do not have one. Nobody is wrong about the best biscuit. What a room's board gives you instead is a map of where the room actually stands, drawn in about two minutes. That map is usually what you wanted out of the meeting in the first place.

When a room's ranking is worth more than yours, and when it is not

Two conditions have to hold, and naming them is more useful than the slogan. A third fact about groups then decides the format rather than the answer.

The first is independence. Lorenz, Rauhut, Schweitzer and Helbing had 144 people answer factual questions and then let some of them see what everyone else had said before answering again. Knowing the other estimates narrowed the range of answers far enough to undermine the effect, and it left people more confident without making them more accurate.

The second is that the room genuinely holds the information. Lightle, Kagel and Arkes ran groups through problems where the best answer was visible only if members pooled what each of them knew privately. The groups found 35% of those answers, against a little over 75% that was reachable on recall alone. The information was in the room. The room did not produce it.

That second condition is the one that fails in real meetings. When nobody present knows the answer, a board gives you a confident average of guesses. It reads as agreement and there is nothing behind it. Ask the people who know instead.

The third thing is not a condition but a reason to use a board at all. Talking is a slow way to collect what a room knows. Diehl and Stroebe ran four experiments on why brainstorming groups produce less than the same people working separately. The cause turned out to be mechanical: production blocking accounted for most of the loss. In a group discussion only one person can talk at a time. Everyone else is queueing.

So the design is fixed: everyone ranks the same list at the same moment, nobody sees anybody else's board while they do it, and the aggregate arrives in one piece. That is what the Tier List activity does: private phones, a count on the big screen, and one press that opens the board.

Ranking against scoring, and what each one hides

A tier list is a ranking that allows ties. That sounds like a technicality and it is the reason the format works at all. Putting nine things in a strict order is miserable. Putting the same nine into four rows takes a minute, and nobody has to decide between the two they rate equally.

Against a score, a ranking has one clean advantage. A score carries the scorer with it. Iramaneerat and Yudkowsky, studying standardised patients who rated medical students in a clinical skills assessment, list the standard failures by name. Leniency they define as a constant tendency of a rater to give out ratings higher than they should receive. Restriction of range is the clustering of ratings around one part of the scale.

A tier list is harder to distort that way, because it is read as an order. Every placement is made against the other things on the same board, so a kind ranker and a harsh one still put the same item first. What the format does not survive is compression. Somebody who tips most of the tray into one row has expressed no order at all. The aggregate cannot tell them apart from a person who genuinely thinks those items are equal.

What a ranking cannot tell you is level. S tier is the best of what you handed people, which in a bad list is still bad. The rows carry a rough sense of distance and only a rough one: they say these two are close and those two are not, and nothing about how far. Lettered rows will never tell you the room dislikes every option on it. If that matters, rename the rows so one of them means no, or run a live poll with one blunt option: would you keep any of these. That catches the case a tier list is blind to.

Ranking has its own failure mode too, and it is not small. Atsusaka and Kim built a way to measure how many people answer a ranking question at random rather than from preference, and in their American survey it was about 30% of respondents. Their word for what those answers do to a result is dilute. Your meeting is not a paid survey panel, though a meeting nobody chose to attend is closer to one than you would like. Watch for the case it names: a placement made without thinking still lands somewhere, and the board cannot tell it from a considered one.

How many things can go on a tier list before it stops working

Working memory is the first ceiling. Cowan's review puts the limit at 3 to 5 meaningful items held at once by young adults. That is not fatal here, because a tier list never asks anyone to hold the whole ordering in their head. The board is on the screen and every comparison is local.

The number people quote at this point is seven, from Miller's 1956 paper, and Miller would not thank them. He proposed seven as the span of absolute judgment on a single dimension, and he ended the paper by calling the recurrence of the number a pernicious, Pythagorean coincidence. It is a good essay and a bad rule.

Layout degrades a ranking as well as length does. In a companion paper on ranked ballots, Atsusaka names pattern ranking, where people rank by the geometry of the grid in front of them instead of by what they think. He finds it survives randomising the order of the options. A long list on a small screen is an invitation to rank by shape.

So, working numbers. Eight to twelve items is where a room stays honest and the whole round fits inside two minutes. Fifteen is fine if the items are one or two words each. Past twenty, the bottom of the tray gets whatever is left over, and you are measuring stamina. The Tier List setup step is described as "Up to forty things to rank, into two to six rows", and a ceiling is not a target.

Cut before you add. The item you are unsure about belongs in the next round, not this one.

What makes a good tier list prompt

A prompt is good when two people who disagree can both answer it without asking you what you meant. Six rules get it there.

Name the dimension. "Rank these" is not a prompt. Ranked by what: usefulness, frequency, cost, how much you would miss it, how badly it goes wrong. Say the word out loud before anybody starts.

Pick a dimension people can honestly differ on. If everyone in the room would produce the same board, you have run a quiz with no answer at the end. Skip it.

Keep the items the same kind of thing. A list mixing tools, rituals and people ranks nothing, because each person silently picks a different comparison.

Plant one item you expect to split the room. The divided row is the reason to run this, so put at least one candidate for it in the list on purpose.

Ask about what happened, not what will. "What actually helped you in your first month" gets you memory. "What will help new starters" gets you policy opinions the room has never tested. Deciding is the exception: a prioritisation is a choice rather than a prediction, and the rows should say so.

Never rank the people in the room. That one has its own section below, and it is the only hard no on the list.

The board underneath is a prompt built to these rules. The question is what actually helped you in your first month, the items are what an onboarding retro turns up, and fourteen people have ranked it privately. Watch the wiki.

Why S tier exists, and when to rename the rows

Tier lists came out of games. Wikipedia's entry says the concept "originated in video game culture where playable characters or other in-game elements are ranked by their tournament viability". The same entry has the rows typically ordered S, A, B, C, D, E, F. The odd part is the row above A.

The S grade is borrowed. EventHubs traces it to Japanese school systems and their grading, where a mark above A already existed. From there it went into fighting games and action games that grade a run, where an S rank is what an A cannot describe.

It survives because it does a job. On an A-to-F scale, A is the ceiling, and everything the room likes piles into it until the top row means "fine". Adding a row above the ceiling keeps A meaning very good and reserves the top for the rare thing. If your boards keep coming back with a stuffed top row, that is the fix.

Four rows is usually right, and the number of rows deserves one deliberate thought. Clustering in the middle of a scale is the named failure Iramaneerat and Yudkowsky call restriction of range, and an odd number of rows gives people a middle to sit in. Nobody has tested that on tier rows, so take it as the reason for a default rather than as a finding. S, A, B and C keeps the row above the ceiling and takes the middle away. The free tier list maker linked at the top opens at five rows, S to D, and the "How many tiers" box is where you drop it to four.

Rename the rows whenever the tier list is a decision rather than a game. Same activity, same two minutes, and the output goes straight into a plan. The Tier List page offers both: "Keep S/A/B/C if you want the shorthand the room already knows, or rename them to must-have, nice-to-have and no."

Named rows buy the thing the letters cannot show, which is level: if the whole list lands in no, that is an answer. Three of them put a middle back on the board, so add a fourth row called "not now" if you want the even scale the letters gave you.

Ready-made tier lists for work

Each of these is a prompt, and most come with a starting set of items. Swap in your own names, cut it to ten, and run it. The last few supply no items because the items are yours, and those are the ones worth a whole agenda item.

• What actually helped you in your first month. A buddy, shadowing calls, the week-one demo, the docs site, the wiki, team lunch, chat channels, recorded talks.
• Meeting formats, by whether they earn the hour. Daily stand-up, retrospective, all-hands, one-to-one, brainstorm, status update, workshop, incident review.
• Ways we communicate, by how well they work here. Email, chat message, a call, a comment on the doc, a hallway conversation, a recorded video, a meeting, a written proposal.
• Our internal tools, by how much you would miss them. Ticket tracker, docs site, chat, the code review tool, dashboards, the design tool, the expenses system, the intranet.
• Interview stages, by what they actually tell us. CV screen, phone screen, take-home exercise, live coding, system design, portfolio walkthrough, culture interview, reference call.
• Sources of interruption, by damage done. Unscheduled calls, chat pings, meeting invites, alerts, someone at your desk, email, phone notifications, being pulled onto something else.
• Documentation, by whether it is still true. Onboarding guide, runbooks, architecture notes, API reference, the wiki, meeting notes, comments in the code, the pinned message in the channel.
• Perks, by whether you would notice if it went. Learning budget, gym contribution, cycle scheme, team socials, the snack shelf, home office allowance, extra leave, the referral bonus.
• Our own rules, by how often we actually follow them. Review before merge, tests with every change, tickets first, deployment checklist, style guide, on-call runbook, change log, last retro's actions.
• Where work goes to die, by how badly it hurts. Waiting on review, waiting on a decision, waiting on another team, unclear ownership, scope changes, environment problems, meetings, half-finished migrations.
• Our incident process, by which step we are worst at. Detection, paging, the first update, the war room, mitigation, the all-clear, the write-up, the follow-up actions.
• Customer complaints, by which one we should fix next. Price, speed, a missing feature, a bug, onboarding confusion, support response time, billing, an integration that broke.
• Next quarter's candidates, by must-have and no. Put your six to ten roadmap items in, and rename the rows before you start.
• The current sprint board, by whether each ticket should be there. Use the real ticket titles, and cut it to a dozen if the board is longer than that. This one is uncomfortable, which is the point.
• Our recurring meetings, by whether they could have been a message. List them by name, and put in the ones you run as well as the ones you attend.

Three or four of these would carry a retrospective on their own. If that is the meeting you are in, retrospective games has that round and the formats around it.

Ready-made tier lists for a classroom

A tier list is a low-stakes way to ask every student for an opinion before anyone hears the loudest one. The ranking is the warm-up. The argument about the divided row is the lesson.

• Study techniques, by what worked for you. Rereading, flashcards, practice tests, summarising, teaching a friend, highlighting, spaced review, cramming.
• Sources, by whether you would cite them. A textbook, an encyclopaedia, a news report, a preprint, a peer-reviewed paper, a video essay, a forum answer, a chatbot answer.
• Inventions, by how much they changed daily life. The printing press, antibiotics, the wheel, refrigeration, the internet, vaccines, the plough, electric light.
• Feedback, by what you would rather receive. A grade only, written comments, a voice note, a one-to-one, peer review, a ticked rubric, a class-wide summary, no feedback at all.
• Maths topics, by how often you have met them outside a maths lesson. Percentages, algebra, geometry, probability, trigonometry, compound interest, statistics, calculus.
• Assessment formats, by how fairly they test what you know. Multiple choice, short answer, essay, open book, oral, coursework, practical, timed problem set.
• Punctuation marks, by how much work they do. Full stop, comma, semicolon, colon, question mark, apostrophe, brackets, ellipsis.
• Lab rules, by how bad the worst case is if you skip one. Goggles, tied hair, no food, labelled containers, waste disposal, fume cupboard, hand washing, tidy bench.
• Characters in the set text, by how much they deserve what happens to them. Use the real names. Expect a fight over the bottom row.
• Causes of the event we studied, by weight. List the five or six the syllabus names and let them argue about the order.
• Which of these helped you understand today's topic. The explanation, the diagram, the worked example, the practice questions, the video, the group work, the reading, the recap. Cut the ones that did not happen today.
• Excuses for late homework, by plausibility. The printer, the dog, the wifi, a lost book, the wrong homework, the bus, forgot it was due, it was in my other bag. It ends a term well.

Fun tier lists for a room that needs warming up

Run one of these first when the group is new to the format. Two minutes on biscuits teaches everybody how the board works, so the round that matters spends its two minutes on the question. Where a prompt below names no dimension, say the dimension out loud anyway: how much you like the thing.

• Office snacks. Crisps, chocolate biscuits, the fruit bowl, cereal bars, nuts, the birthday cake, whatever is left in the fridge, mints.
• Biscuits, by dunking performance. Digestive, rich tea, ginger nut, custard cream, shortbread, chocolate finger, jaffa cake, bourbon.
• Breakfasts. Toast, porridge, cereal, eggs, a pastry, fruit, last night's leftovers, nothing at all.
• Pizza toppings. Pepperoni, mushroom, pineapple, olives, extra cheese, chilli, anchovy, sweetcorn.
• Weather, by how you feel about it. Crisp and cold, drizzle, a heatwave, a thunderstorm, fog, first snow, grey and still, proper wind.
• Days off, by quality. New year, a spring bank holiday, the long summer weekend, a random Monday off, the winter break, a national one-off, your birthday off, a snow day.
• Ways to commute. Walk, cycle, bus, train, drive, scooter, a lift from a colleague, working from the kitchen table.
• Seats on a plane. Window, aisle, middle, exit row, bulkhead, front of the cabin, the back row, the one that does not recline.
• Types of chair. Office chair, dining chair, armchair, beanbag, folding chair, bar stool, sofa, the arm of the sofa.
• Sandwich fillings. Cheese, ham, egg, tuna, bacon, hummus, coronation chicken, peanut butter.
• Condiments. Ketchup, mayonnaise, mustard, brown sauce, hot sauce, vinegar, pesto, salad cream.
• Ways to leave a video call. The wave, the sudden drop, the honest goodbye, blaming the connection, the slow fade, the hard stop on the hour, the second meeting excuse, just leaving.
• Board game mechanics, by how much fun they are. Rolling dice, drafting cards, hidden roles, trading, area control, co-operation, worker placement, elimination. Only if the room plays them.
• Films everybody claims to like. Pick eight your group will argue about. The bottom row is the whole point.
• Sounds, by how much they bother you. Chewing, a ticking clock, notifications, someone else's music, a vacuum cleaner, rain on a window, typing, a phone face-down on a hard desk.

Do not use a tier list to rank people

Somebody suggests it sooner or later, usually cheerfully. The answer is no, and it is worth explaining rather than just refusing, because the reason is not squeamishness.

Ranking people is a real management practice with a real literature, and a paper that finds in its favour is still discouraging. Scullen, Bergey and Aiman-Smith simulated forced distribution rating systems, which they describe as systems that require firing a certain percentage of the workforce each year. Their model does find a gain in workforce potential. It also finds that most of it arrives in the first several years, and that the size of it is largely a function of how many people you fire.

That is the machinery a forced ranking of people needs before it does anything: a defined scale, a documented process, and consequences somebody signed off. A tier list in a meeting has none of it. What it has is the public ordering of people who will still be in the room afterwards.

Nothing in this paragraph is a finding, so read it as an argument. Making the ordering anonymous does not soften it. When the board says a named colleague belongs in the bottom row and nobody will say who put them there, you have manufactured a fact that nobody owns and nobody can answer. The ordering is the problem rather than the anonymity. One warm nomination of one person is a different object, and most likely to questions sets out where even that stops being safe.

Rank the work instead. "Which parts of our process are actually helping" answers the question the room was reaching for, and nobody has to be in the bottom row.

How to run a tier list in a live meeting

The ranking itself takes two minutes. Most of the decisions that make it land are made before anybody ranks anything, and the conversation afterwards runs as long as it is worth. In a room of four a split row is two people, so read the board as four opinions rather than as a signal.

Show the list before you explain the rows. People want to know what they are ranking. Reading eight items takes ten seconds, and it stops the first minute being questions.

Rank in silence, all at once. This is the independence condition from the Lorenz result, built into the screen. While people are ranking, the Tier List presenter shows the question and a count, under the line "Nobody sees anybody else's board until the reveal."

Reveal once, then stop talking. Give the room ten seconds of silence to read the board. If nobody speaks by then, ask whose own board looks least like the one on screen.

Go to the divided row first, not the top one. The board marks the most divided items with a "split" tag, and that is where the meeting is. An item can sit mid-table because everyone thinks it is middling, or because half the room put it top and half put it bottom, and those are two completely different problems.

Let people leave things unranked. It is on by default, and it is the honest setting. Forcing a placement on something nobody has an opinion about manufactures a signal you will later act on.

Discuss after the reveal, never before. Crouch and Mazur found that the number of students giving the right answer to a concept question rose substantially after discussion. The condition was that the initial share correct sat between 35% and 70%, and the improvement was largest around 50%. There is no right answer on a tier list, so that is a shape and not a result. It still points at the row worth the time, which is the one the board split down the middle.

Do not re-run the round to force agreement. A second pass after everyone has seen the board breaks the independence condition on purpose. You will get convergence. It is the Lorenz result again: more confidence, and no more accuracy.

When the room agrees on everything, say so and move on. A board with no split still answers the question, in two minutes, against the half hour you had set aside. The finding is that there was nothing to argue about.