Build with us

Adapt the model to the shared decision interface.

Implement a backend in models.py. A backend receives state text, instructions, and a dictionary of candidate IDs to descriptions. Return a selected candidate ID and probabilities keyed by those same IDs.

class ExampleBackend(Backend):
    name, label = "example", "Example Model"

    @classmethod
    def availability(cls):
        # Check installation / configuration without loading weights.
        return True, ""

    def answer(self, state, instructions, criteria):
        # Call your model, map its output to candidate IDs, validate it.
        choice, probabilities = your_adapter(state, instructions, criteria)
        return choice, probabilities

This is an adapter outline: replace your_adapter with the actual model call. Never execute the submitted setup instructions on the public site.

1. Register the backend.Add it to BACKENDS in models.py. Reuse cached instances, declare concurrency, keep loading lazy, and describe required packages or API settings.
2. Wire every supported game.Dino, Flappy, Chess, and Sudoku generate model players from BACKENDS. 2048 registers its model player classes explicitly in engine2048.py; add the corresponding wrapper there. Update MODELS in benchmark.py and website/catalog.py.
3. Verify the output contract.Use candidate IDs exactly. Handle one-option decisions, missing probabilities, unavailable dependencies, and malformed responses. Document guards and avoid hidden fallback policies that inflate performance.
4. Run comparisons.Compare against existing baselines with identical seeds. Include the command, model revision, hardware, dependencies, and result file in your PR.

Make the experiment reproducible before presenting it.

1. Create engineyourgame.py.Keep rules, legal actions, scoring, and terminal conditions in Python. Accept an environment seed and use a separate reproducible RNG stream for randomized policies. Add a VERSION when mechanics may change.
2. Describe the decision.Convert state into concise text and stable action IDs with descriptions. Provide PLAYERS, player availability checks, and a cached get_player(name). Supply a random policy and a meaningful heuristic baseline.
3. Connect the headless runner.Register the engine in benchmark.py's MODS, add the game adapter to play_single(), and update CLI help. Multiplayer games need a driver like run_chess() rather than the single-player loop.
4. Return useful metrics.Record score plus game-specific measurements, timing, model-call counts, control mode, and version where applicable. Use the existing run/checkpoint/result/status protocol and JSON export.
5. Test and document.Test determinism, legal moves, score rules, and stopping conditions. Add the game to the README and site catalog, explain its baselines, and include an exported run. The website has a generic table for new games; customize columns in website/static/benchmarks.js if useful.

After registering your game, run this from the repository root. Replace yourgame with the registered ID:

./benchmark --game yourgame --players random greedy \
  --games 10 --seed 42 \
  --json website/benchmark-data/yourgame.json

Build on the console game, then make it observable.

1. Reuse authoritative rules.Prefer server-side state, as Chess, Sudoku, Dino, and Flappy do. Expose create/step endpoints in server.py with validated models, per-session locking, terminal states, and trace logging.
2. Add the two game pages.Create play_yourgame.html and arena_yourgame.html. Include a model selector, step, autoplay/pause, reset, score, decision context, and probabilities. Arena comparisons should use the same seeds.
3. Connect routes and navigation.Add explicit page/asset routes in server.py and game links on the arcade pages. Use the shared arcade.css design. arcade.js handles only Dino and Flappy; a new game needs its own controller/renderer.
4. Verify the experience.Test pause/reset while requests are in flight, errors, unavailable models, finished-game persistence, and narrow layouts. Capture real screenshots, add descriptive captions, and update the README.

Use the headless engine as the source of truth. Playback speed should change presentation, not scores or collision outcomes.

Fork → branch → implement → verify → pull request.

Fork the repository on GitHub, clone your fork, and create a branch. Follow the root README for game/model dependencies; the website has its own lightweight requirements.

git clone https://github.com/YOUR-USERNAME/system-one-models-game-test.git
cd system-one-models-game-test
git switch -c feat/your-contribution

# Implement, then run the relevant checks.
.venv/bin/python -m unittest discover -s tests -v
./benchmark --quick --no-save --json website/benchmark-data/check.json

# For website changes:
cd website
docker compose up --build -d
docker compose run --rm --no-deps \
  -v "$PWD/tests:/app/tests:ro" web \
  python -m unittest discover -s tests -v
cd ..

git add 
git commit -m "Add your contribution"
git push -u origin feat/your-contribution

Open a pull request from your branch to this repository using GitHub's Compare view. Fill out the PR template, link a suggestion or issue if you have one, and respond to review. Keep credentials and model weights out of the commit.