Explainability analysis
30 programs randomly sampled from the 1,100 human-rated samples; seed 42.
Score consistency across runs
Three runs per sample · 0–20 scale
| Model | Identical scores | Mean score range | Maximum score range |
|---|---|---|---|
| GPT-6.1 Sol | 14/30 (46.7%) | 0.53 | 1 |
| DeepSeek V4 Pro | 3/30 (10.0%) | 3.70 | 14 |
Sampled examples
Runs 1 / 2 / 3 |
Runs 1 / 2 / 3 |
||||
|---|---|---|---|---|---|
| E01 · Dorn/java/109 | Dorn | 17 / 16 / 16 | 16 / 15 / 14 | 0.642 | 0.922 |
| E02 · Buse/55 | Buse | 13 / 13 / 12 | 12 / 8 / 15 | 0.446 | 0.202 |
| E03 · JetBrains/i49 | JetBrains | 17 / 17 / 17 | 16 / 13 / 14 | 0.313 | 0.246 |
| E04 · JetBrains/e14 | JetBrains | 18 / 18 / 18 | 18 / 18 / 17 | 0.478 | 0.288 |
| E05 · Dorn/python/97 | Dorn | 15 / 15 / 16 | 17 / 15 / 16 | 0.618 | 0.947 |
| E06 · Dorn/java/166 | Dorn | 16 / 15 / 15 | 16 / 14 / 13 | 0.635 | 0.538 |
| E07 · Dorn/cuda/9 | Dorn | 15 / 16 / 16 | 16 / 14 / 12 | 0.271 | 0.483 |
| E08 · Dorn/cuda/61 | Dorn | 13 / 13 / 13 | 12 / 14 / 14 | 0.247 | 0.212 |
| E09 · Schnappinger/aoi/artofillusion.view.GLCanvasDrawer | Schnappinger | 15 / 15 / 15 | 14 / 16 / 14 | 0.148 | 0.023 |
| E10 · Buse/68 | Buse | 18 / 17 / 17 | 17 / 3 / 16 | 0.591 | 0.51 |
| E11 · Buse/64 | Buse | 16 / 16 / 17 | 4 / 14 / 13 | 0.469 | 0.596 |
| E12 · Dorn/cuda/73 | Dorn | 14 / 14 / 13 | 6 / 8 / 4 | 0.521 | 0.409 |
| E13 · Dorn/python/88 | Dorn | 18 / 17 / 17 | 12 / 2 / 10 | 0.86 | 0.944 |
| E14 · JetBrains/b39 | JetBrains | 18 / 18 / 18 | 18 / 17 / 18 | 0.575 | 0.987 |
| E15 · Schnappinger/junit4/junit.framework.TestCase | Schnappinger | 16 / 16 / 16 | 16 / 16 / 16 | 0.702 | 0.804 |
| E16 · Buse/58 | Buse | 18 / 18 / 18 | 14 / 16 / 15 | 0.466 | 0.101 |
| E17 · Dorn/python/51 | Dorn | 16 / 16 / 16 | 7 / 18 / 14 | 0.517 | 0.735 |
| E18 · Schnappinger/aoi/artofillusion.translators.TextureImageExporter | Schnappinger | 15 / 14 / 14 | 14 / 17 / 13 | 0.117 | 0.205 |
| E19 · Dorn/python/91 | Dorn | 15 / 14 / 14 | 8 / 6 / 12 | 0.551 | 0.652 |
| E20 · Schnappinger/argoUML/org.argouml.uml.diagram.deployment.ui.UMLDeploymentDiagram | Schnappinger | 14 / 15 / 15 | 14 / 14 / 14 | 0.465 | 0.119 |
| E21 · JetBrains/i56 | JetBrains | 18 / 19 / 19 | 18 / 18 / 17 | 0.578 | 0.246 |
| E22 · Buse/20 | Buse | 19 / 19 / 19 | 18 / 18 / 18 | 0.418 | 0.449 |
| E23 · Dorn/java/207 | Dorn | 15 / 15 / 16 | 18 / 17 / 16 | 0.62 | 0.983 |
| E24 · Schnappinger/aoi/artofillusion.view.TexturedVertexShader | Schnappinger | 15 / 15 / 15 | 16 / 14 / 15 | 0.222 | 0.073 |
| E25 · Scalabrio19 | Scalabrino | 19 / 19 / 19 | 18 / 19 / 18 | 0.67 | 0.734 |
| E26 · Dorn/java/199 | Dorn | 15 / 15 / 15 | 17 / 18 / 17 | 0.608 | 0.819 |
| E27 · Dorn/python/81 | Dorn | 17 / 16 / 17 | 14 / 14 / 7 | 0.289 | 0.379 |
| E28 · Scalabrio183 | Scalabrino | 16 / 17 / 16 | 16 / 14 / 16 | 0.532 | 0.455 |
| E29 · Dorn/cuda/71 | Dorn | 15 / 15 / 15 | 14 / 12 / 11 | 0.256 | 0.187 |
| E30 · Scalabrio83 | Scalabrino | 13 / 13 / 13 | 8 / 4 / 7 | 0.393 | 0.399 |