The rules decide the game
In any competition, what you reward is what you get. A challenge that scores by program length will produce short programs. One that scores by speed will produce fast ones. Neither is better in the abstract, but each leads solvers toward a particular style, and often away from others.
That is why scoring deserves as much thought as the puzzle itself. A well-chosen metric produces solutions that feel elegant to the organizers. A careless one produces entries that technically win but that nobody admires.
Common scoring methods
Here are the measures that challenges most often use, with typical strengths and weaknesses.
- Length. Count characters, bytes, grid cells or instructions, and prefer the smallest. This is the heart of code golf. It is simple to measure and easy to understand, but it encourages obscure tricks and can make winning code unreadable.
- Running time or step count. Prefer the fastest program, or the one that takes the fewest steps. This rewards efficient algorithms but depends on hardware, interpreter and measurement noise unless the steps are counted by the interpreter itself.
- Memory use. Prefer the program using the least storage. Useful on small machines, but it can be hard to define for languages without a clear memory model.
- Correctness only. Any program that passes all tests ties. Fair and simple, but it provides no way to rank solutions.
- Votes or judging. People rate entries on cleverness, style or beauty. This allows qualities that numbers cannot capture, at the cost of subjectivity.
- Points per test. Partial credit for each passing case. This rewards progress, but needs well-designed tests.
Some contests combine several measures. Combining them calls for a stated rule about how: ranking by one metric and breaking ties with another is easy to explain, whereas blending numbers into a weighted sum is harder to justify and can hide odd incentives.
How rules steer solutions
Every metric has side effects. A few examples of how incentives play out:
- Length scoring and layout. When whitespace counts, solvers squeeze it out. In a grid language where layout is the program, this can force awkward shapes. If only non-space characters count, solvers may spread code across a huge grid instead. The choice of counting rule matters a great deal for size limits.
- Speed scoring and special cases. A program might be tuned to the exact test inputs and be slow on anything else.
- Correctness scoring and hard-coding. If the test inputs are few and public, a program can simply print the known answers. Hidden tests reduce this.
- Any metric and loopholes. Solvers are creative. If a rule has a gap, someone will use it, and then everyone feels they must too.
None of this means that contests are doomed. It means organizers should imagine the cleverest, least cooperative entry and ask whether they would be happy to see it win.
Principles of fairness
Fairness is partly about equal treatment and partly about predictability. A few principles help:
- Publish the rules in advance. Solvers should know exactly how entries are measured before they start. Changing rules midway is almost always unpopular.
- Define counting precisely. What is a character? Are line breaks counted? What about trailing whitespace? Is the count in bytes or in characters? Ambiguity here causes disputes.
- State the environment. Interpreter, version, flags and input handling should be fixed. A program that depends on unspecified behavior may work for one participant and fail for another.
- Use hidden tests. Keep some inputs private so that solutions must be general. Tell solvers that hidden tests exist.
- Make the check automatic where possible. A script applied to everyone removes the suspicion of favoritism.
- Decide tie-breaks ahead of time. Earliest submission, a secondary metric, or a shared win, but choose before there is a tie.
- Allow appeals for clear errors. If an entry is wrongly rejected because of a flaw in the checker, there should be a path to fix it.
Different languages, different footing
Challenges that accept multiple languages face a particular problem: languages differ so much that raw numbers are not comparable. A task that takes a few characters in a language with a built-in function may take hundreds in a minimalist one.
Common solutions are to score each language in its own category, or to restrict the challenge to a single language so that the competition is among people, not tools. Esoteric language contests often take the second path, which also suits the spirit of esolang challenges: the interest lies in what clever people can do under shared constraints.
Keeping the spirit
Not every contest needs strict machinery. Casual challenges among friends can run on a light touch: an agreed rule sheet, honest reporting, and shared enjoyment. The more money or reputation is at stake, the more careful the rules should be. Even in a friendly setting, stating the scoring in writing prevents misunderstandings.
After a contest ends, it can help to publish the solutions and discuss them. Seeing why the winner won, and where the metric nudged people, is a valuable lesson, and reading others’ solutions turns a competition into a learning exercise for everyone.
A checklist for organizers
Before you launch, ask yourself:
- Does the metric match what I actually want to reward?
- Could a trivial or degenerate solution win?
- Is every counting rule written down with an example?
- Can I run the checker on a sample entry and get a clear result?
- What will I do in the event of a tie or a dispute?
If you can answer all five confidently, you are in good shape. And if something goes wrong anyway, owning the mistake and adjusting for the next round is better than defending a flawed rule.