Code Readability Results
Spearman correlation between predicted scores and human ratings, computed separately for each dataset. Counts show evaluated samples / total samples.
Methods
Datasets
Independent Readability Interferences
Percentage of changed variants scored less readable than their matched original, reported separately for Java and Python. Counts show evaluated changed pairs.
Methods
Languages
Motivating example
A month-offset lookup adapted from Sakamoto’s compact day-of-week algorithm. All five developers rated it 1/5, yet four traditional models place it above the 74th percentile.
A lookup hidden in a string
class MonthOffsets {
static int offset(int month) {
int value = "-ilkDwlu2thk5".charAt(month);
return value % 7;
}
}
Relative readability
Posnett
86.5
Scalabrino
92.2
Dorn
76.4
Mi
74.5
DeepSeek V4 Pro
8.3
BTTF
1.9
Human
0.0
Model percentiles over the human-rated benchmark pool; lower is less readable. Human 1/5 is mapped to 0. DeepSeek averages five judgements.
Largest BTTF penalties and their source regions
Assignment-value surprisal-0.253
"-ilkDwlu2thk5".charAt(month)Literal-tail surprisal-0.24
"-ilkDwlu2thk5"Declaration-surprisal variation-0.073
int value = ...Identifier-onset surprisal-0.037
valueExplainability analysis
LLM explanations and BTTF feature contributions for 30 randomly sampled programs.
Datasets and samples
| Dataset | Samples | Evaluation | Download |
|---|---|---|---|
| MBJP | 17 | human-rated | JSON |
| Buse | 100 | human-rated | Official source |
| Scalabrino | 200 | human-rated | Official source |
| Dorn | 360 | human-rated | Official source |
| Schnappinger | 304 | human-rated | JSON |
| JetBrains | 119 | human-rated | JSON |
| Java transformations | 1400 | controlled | JSON |
| Python transformations | 1400 | controlled | JSON |