Coding Agent Benchmark

A small Java project for comparing coding agents. Its compact codebase and existing test suite provide a starting point for observing how an agent understands unfamiliar code, investigates bugs, and makes focused changes.

This project is intended as a benchmark and learning environment. It does not include a runnable application; start with the source code and unit tests.

The Example Project

SplitZipWriter writes files from input streams into ZIP archives, creating multiple parts based on a size threshold. Archives are named <baseName>_01.zip, <baseName>_02.zip, and so on. Each input file is written entirely into one archive; switching archives happens between files.

Other components:

• ZipResult collects the names and sizes of the generated archives.
• OutputStreamEnhancer allows the output stream to be wrapped with additional behavior.
• InputStreamEnhancer defines the corresponding interface for input streams.
• SplitZipWriterTest checks archive splitting and file contents.

Requirements

• JDK 21, matching the repository's .java-version
• A filesystem supporting POSIX file permissions, such as those typically used on Linux or macOS
• Internet access for the first build to download Gradle and dependencies

The Gradle Wrapper is included, so no separate Gradle installation is required. JAVA_HOME should point to the JDK you want to use.

Quick Start

Run all tests from the project directory:

./gradlew test

Run a full build, including tests:

./gradlew build

Force the tests to run again, even if Gradle considers them up to date:

./gradlew test --rerun-tasks

The HTML test report is available at build/reports/tests/test/index.html after the test run.

Warning: The tests delete and recreate the destination/ directory inside the project directory. Do not store your own files there.

Using This Project as a Benchmark

The baseline contains one failing unit test. Another test covering large amounts of data is disabled with @Disabled. A failing test run is therefore part of the benchmark's initial state.

Example prompt for a coding agent:

Inspect the project and run the unit tests. Find the cause of the failing test and fix it with the smallest reasonable change. Preserve the existing test expectations and do not disable any tests. Explain the cause, your change, and how you validated it.

For comparable runs:

1. Start each agent from the same commit in a separate checkout.
2. Use the same prompt, Java version, and access permissions.
3. Record the model, agent version, settings, runtime, and token usage where available.
4. Review the resulting diff and run the tests again.

Possible evaluation criteria include a correct root cause analysis, a small and understandable change, preserved test expectations, and successful validation. Passing tests alone do not replace reviewing the solution.

Project Structure

.
├── build.gradle                 # Dependencies and test configuration
├── settings.gradle              # Gradle project settings
├── gradlew / gradlew.bat         # Gradle Wrapper
├── AGENTS.md                    # Guidance for coding agents
└── src
    ├── main/java/illgner/ch
    │   ├── createzip            # ZIP creation and result data
    │   └── stream               # Stream enhancement interfaces
    └── test/java/illgner/ch
        └── createzip            # Unit tests for SplitZipWriter

Technology

Java, Gradle, JUnit Jupiter 6, Hamcrest, Apache Commons IO, and Apache Commons Lang.