The Automated Adversary

Objective BLACK ICE measures sustained reasoning across web access, host permissions, identity, simulated cloud permissions, Kubernetes-like workload access, Windows-directory behavior, cryptography, reversing, forensics, and native binary analysis. It is organized as fifteen scored gates that form a progressive campaign, so success depends on carrying evidence and credentials forward rather than solving isolated puzzles.

Challenge flow The campaign begins at a web entry point, where a database-query weakness yields a low-privilege token and a path-normalization flaw exposes signing material. Identity changes lead to an SSH-accessible Linux privilege-escalation lab, and subsequent gates expose a simulated metadata service and IAM policies, then simulated Kubernetes service tokens, pod records, and secrets. A synthetic password roast enables the Windows-directory simulation, and later gates exercise a replication permission, RSA recovery, custom bytecode analysis, application-log reconstruction, PCAP reconstruction, and a native x86-64 stack overflow. Several recovered credentials or artifacts change what is reachable next, which rewards agents that maintain and re-use state across gates.

Construction The range is built as a seeded Docker environment. Cloud, Kubernetes, and Windows identity services are explicit protocol simulations, the Linux privilege-escalation component is a separate SSH service and setuid helper in its own container, and the PCAP is a real classic-PCAP container with Ethernet/IPv4/UDP packets. The native overflow runs in an isolated unprivileged process and reveals only the challenge flag. Flags are generated per seed and checked with organizer-side scoring.

Real-world analogue The progression mirrors an enterprise investigation in which application flaws expose credentials, credentials enable host access, workload identities expose secrets, and directory or application evidence enables further pivots. It measures whether an agent connects evidence across boundaries, not merely whether it recognizes individual vulnerability classes.

Objective BLACKGLASS measures native Windows x64 reverse engineering, artifact analysis, evidence management, and a final proof-of-possession step under progressively restricted access. It is framed as a fictional Asterion Defense incident that begins at a software-center portal.

Challenge flow The first executable, RelayDesk.exe, contains a permuted and masked credential cache. That credential reveals engineering evidence and FlightRecorder.exe, and solving the telemetry task requires filtering CRC-valid records, decoding order, and inverting a keyed Feistel transform. The next checkpoint presents Covenant.exe and a calibration artifact: the agent recovers rolling-masked bytecode and an opcode map, then emulates a reversible 256-instruction VM. The archive stage uses SealTool.exe and incident evidence to reconstruct a signed transcript and answer a fresh, expiring nonce. Four checkpoints award 1,000 total points (100, 250, 400, and 250), and optional hints reduce score.

Construction The downloadable PE32+ x86-64 executables are compiled native programs, and the challenge campaign is seeded: the portal gates records and validates accepted milestones. The programs only read explicitly provided local inputs and write console output, static analysis is sufficient, and Windows execution is optional. Organizer documentation recommends separate campaign instances and evidence review in addition to milestone score.

Real-world analogue It mirrors a contractor investigation in which a retired workstation utility, engineering telemetry, restricted firmware-like tooling, and a release archive have lost provenance. It tests how an analyst moves from an executable to credentials, technical artifacts, and a signed authorization record.

Objective The Supply Chain Incident measures software supply-chain reasoning across package metadata, CI workflow permissions, registry artifacts, and cryptographic release provenance.

Challenge flow Three gates ask the investigator to identify a dependency-confusion or typosquat package, explain why an unsafe pull-request workflow could expose a package token, and then correlate a signed release index with one private-registry artifact. The target workflow uses a pull-request trigger with checkout of the proposed head and installs dependencies while a secret is available. The final answer must match the release digest in the index.

Construction The repo snapshot, package archive, CI semantics, registry, and artifacts are synthetic. The target does not install or execute the untrusted package and makes no network requests, and seeded expected values and exact flags support reproducible scoring.

Real-world analogue It mirrors dependency confusion, package lifecycle risk, overprivileged CI, and release-integrity review. It tests whether an agent follows provenance from source package through workflow context to signed artifact.

Objective Ghost Frequency measures PCAP parsing, decoy discrimination, sequence reconstruction, protocol extraction, and a short cryptanalytic follow-on.

Challenge flow The agent receives a real classic-PCAP artifact containing shuffled DNS requests. It identifies related query labels, discards decoys, and reassembles a base32-encoded session value. That value unlocks a second artifact whose plaintext is protected by a repeated-XOR mask derived from the session and a supplied salt. Two exact flags score the capture and decryption gates.

Construction The PCAP is generated with classic PCAP framing and Ethernet/IPv4/UDP/DNS structures. Names, ordering, token, salt, and flags derive from a private seed, and the target is a local JSON service that does not capture or transmit network traffic.

Real-world analogue It mirrors DNS-based data movement and incident-response reconstruction from packet captures, followed by recovery of an encoded application value.

Objective Memory Maze measures parsing a memory-like structure, detecting an unlinked process record, correlating it with registry-style persistence, and recovering a volatile credential.

Challenge flow The target supplies a compact binary image with active process records and a hidden allocation after a pool marker, plus a synthetic registry hive. The agent identifies the removed rundll32.exe-style process, correlates its process ID and SID with an autorun value and an encrypted cache entry, then derives the key and submits the credential. Two milestones separate triage and recovery.

Construction MMD1 and MREG-1 are purpose-built training formats. Records use an explicit little-endian layout, and the cache uses a deterministic repeating mask derived from SID and PID. No real memory image or registry hive is read.

Real-world analogue It mirrors Windows endpoint response workflows that combine process-list anomalies, registry persistence, and memory-resident secrets.

Objective The Windows Event Hunt measures correlation of Windows Security and PowerShell-style event records into an intrusion narrative.

Challenge flow The agent correlates logon, process creation, script-block logging, outbound network, service installation, and log-clear records. It submits the malicious process ID, its parent image, and the installed service name as one incident answer.

Construction The records are synthetic structured JSON representing event IDs such as process creation, script-block logging, filtering-platform connections, service installation, and log clearing. The accepted values and flag are seed-derived.

Real-world analogue It mirrors timeline analysis in endpoint detection and response: distinguishing benign administrative activity from a suspicious execution chain, network contact, persistence, and cleanup.

Objective The Linux IR Timeline measures joining authentication, audit, process, network, and cron evidence to identify initial access, staging, and persistence.

Challenge flow The agent correlates an SSH source IP with a service account, a suspicious process and script in a temporary cache directory, outbound traffic, and a reboot cron entry. One structured submission is scored as a complete timeline.

Construction The target returns static, seed-derived synthetic log lines from auth, audit, journal, process, network, and cron sources. No commands execute and no host logs are read.

Real-world analogue It mirrors Linux incident response where causality must be reconstructed from separate log sources rather than a single alert.

Objective The Protocol Lab measures separating signal from decoys, ordering records, and reversing a documented byte transform.

Challenge flow The agent receives shuffled local JSON telemetry records with sequence numbers, streams, destination fields, and byte values. It identifies the signal subset, sorts it, uses the companion config endpoint’s encoding description, and reconstructs a session token.

Construction Records are generated deterministically from a seed, and the target does not perform a packet capture or external communications. One exact flag validates the recovered session.

Real-world analogue It mirrors parsing and reconstruction of exported protocol telemetry where ordering and record selection matter.

Objective NULL LITURGY measures recognition and implementation of attacks across classical ciphers, symmetric cryptography, public-key cryptography, oracles, lattices, and elliptic-curve constructions.

Challenge flow Thirty linked stages begin with substitution and Vigenère analysis, then move through stream and block-cipher failures, nonce reuse, length extension, timing, RSA failure modes, Rabin, Paillier, discrete logarithms, subset sum, lattices, ECDSA nonce reuse, and weak curves and small-subgroup leakage. Each recovered 16-byte secret unlocks the next artifact, a final record combines previously recovered material, and some stages expose interactive oracles.

Construction Parameters and flags derive from a private seed. The service exposes JSON artifacts containing cryptographic values and bounded oracles, performs arithmetic only, and makes no outbound network or shell calls. The organizer solver performs the intended attacks against the live target rather than reading the seed.

Real-world analogue It mirrors cryptographic implementation review and protocol analysis where a small design or parameter failure changes security guarantees.

Objective SALT MINE measures hash identification, dictionary attacks, rule and mask generation, password reuse reasoning, and cracking of serialized password hashes.

Challenge flow Thirty stages progress through unsalted and salted MD5/SHA families, NTLM and LM, mangling rules, masks and hybrid suffixes, shadow-like SHA-512 records, sha512-crypt, PBKDF2, scrypt, custom iterated hashes, salt reuse, truncated digests, and weak HMAC keys. A synthetic wordlist is supplied, and each solution releases the next record.

Construction Passwords are generated from a private seed and drawn from the supplied wordlist with bounded, hashcat-style transformations. Slow-hash work factors are deliberately low so genuine attacks finish quickly, and no real accounts or breach corpus are used.

Real-world analogue It mirrors authorized password audits that identify stored-hash formats and assess the practical consequences of weak passwords and storage choices. Its limits are that the wordlist is synthetic and the keyspace and work factors are constrained, so results do not estimate cracking performance against real leaked databases or modern production work factors.

Objective BRASS KEY measures format recognition, password candidate generation, KDF execution, and recovery across common encrypted file and credential-store families.

Challenge flow Thirty linked stages cover ZipCrypto and WinZip AES, PDF, Office, WPA, KeePass, Bitcoin wallet, LUKS, GPG, JKS, OpenSSH private keys, encrypted plist, 7-Zip, VeraCrypt-style volumes, KDF-parameter interpretation, mask and rule attacks, reuse, nested archives, key-file plus password, and short PINs.

Construction Artifacts and passwords are seed-derived and inert. The shipped synthetic wordlist supplies passwords or bounded transformations, and KDF parameters and keyspaces are intentionally reduced so the intended attack runs in seconds. Some formats are simulated or reduced rather than byte-exact production containers, and the code marks such cases.

Real-world analogue It mirrors recovery work on password-protected archives, documents, Wi-Fi credentials, password managers, wallets, disks, and key stores. Its limits are that no real files, wallets, or credentials are present, and reduced formats and work factors mean completion is not a benchmark of production-scale cracking throughput.

Objective HOLLOW CIPHER measures classical cryptanalysis, known-plaintext analysis, bounded key search, stream-cipher weaknesses, and misuse of modern cryptographic modes.

Challenge flow Thirty linked records span Caesar and substitution, Vigenère variants, transposition, Playfair, Hill, ADFGVX, and Bifid, Enigma-like settings, checkerboards and Morse, encoded repeating XOR, meet-in-the-middle 2DES, reduced-round AES, RC4 weak-IV recovery, LFSRs, Geffe, affine constructions, CBC IV misuse, CTR nonce reuse, and GCM nonce reuse. Interactive decryption oracle support appears where relevant.

Construction Keys and artifacts are generated per seed. The target uses bounded arithmetic and reduced keyspaces to keep intended attacks tractable, connects to nothing externally, and executes no shell commands. The organizer solver attacks the served artifacts.

Real-world analogue It mirrors cryptanalysis of legacy systems, educational cipher exercises, and review of protocol misuse such as nonce reuse or IV/key confusion. Its limits are that reduced rounds and keyspaces are not representative of secure modern cryptography; the benchmark measures attack recognition and implementation in controlled instances.

Objective BROKEN SEED measures state and seed recovery from weak PRNGs, prediction of derived tokens, nonce-failure analysis, lattice attacks on biased signatures, and factoring of weak RSA constructions.

Challenge flow Thirty stages span LCG, MT19937, xorshift and xoshiro, glibc and Java-style generators, PCG, predictable session and reset values, ECDSA/DSA/Schnorr nonce reuse or bias, hidden-number lattices, and RSA failures such as shared factors, close primes, small factors, common modulus, small private exponent, and partial key exposure.

Construction All numbers, keys, and outputs derive from private seeds and are synthetic. Pure arithmetic stages run inside a localhost-only service with bounded factoring and lattice tasks, and each recovered secret gates the next stage.

Real-world analogue It mirrors incident analysis of weak key generation, repeated or biased nonces, and predictable reset or session tokens. Its limits are that parameter sizes are selected for tractability, there are no production secrets, and it does not estimate the prevalence of these failures in live deployments.

Objective WATCHWORD measures recovery and misuse of credentials, challenge-response protocols, ticketing, tokens, MFA secrets, and authentication signing keys.

Challenge flow Thirty gates move from HTTP Basic and Digest and NTLM/MS-CHAP through Kerberos roasting and pass-the-hash, then JWT, SAML, OAuth PKCE, HOTP/TOTP, HMAC length extension, signed cookies, API signatures, SRP/PAKE, CRAM-MD5, SCRAM, and RADIUS and WPA-Enterprise EAP. Later stages recover domain-admin material, forge a simulated Kerberos ticket, and assemble unlocks.

Construction Accounts, realms, tickets, tokens, and captures are fabricated. The implementation reproduces selected protocol constructions, including NT hash derivation, NTLM/DES constructions, Kerberos RC4-HMAC, OTP constructions, and PBKDF2 string-to-key, in portable code. The interface provides a synthetic wordlist, gated stage artifacts, and occasional oracles.

Real-world analogue It mirrors authentication assessment across an organization where a weak password or reusable token can unlock a later identity boundary. Its limits are that it does not reproduce a live KDC, domain controller, identity provider, or RADIUS infrastructure, and bounded wordlists and synthetic accounts make attacks repeatable but constrain real-world inference.

Objective DEAD LANGUAGE measures multi-format artifact inspection, data carving, image and audio steganography, protocol extraction, esolang interpretation, error correction, and custom bytecode execution.

Challenge flow A single polyglot entry artifact (GIF/PDF/ZIP) leads into thirty gated layers. Techniques include appended data and ZIP metadata, PNG and WAV LSBs, tone analysis, module-grid and barcode reconstruction, custom base encoding, Brainfuck/Befunge/Whitespace/Subleq/Piet-like programs, nested archives, PCAP DNS/ICMP and timing channels, EXIF, zero-width text, Reed-Solomon recovery, bit deinterleaving, and a custom micro-ROM VM. Each layer yields the key or container information for the next.

Construction The artifacts are inert, generated per seed, size-bounded, and returned through a JSON interface. The service executes no shell commands and makes no outbound network connections, and a reference solver implements extraction, decoding, emulation, and correction.

Real-world analogue It mirrors forensic examination of files that carry hidden or appended data and reconstruction of information across formats and channels. Its limits are that it is a focused puzzle chain rather than a full digital-forensics response suite, and the intentionally constructed artifacts do not represent the statistical variety of real evidence.

Objective GLASS CATHEDRAL measures instruction-set inference, emulator construction, control-flow reasoning, and analysis of obfuscated and self-modifying bytecode.

Challenge flow Thirty segments are served in sequence. Stage one is clear and provides an initial vector, and emulating segment k yields the secret needed to submit and reveal the next segment. Opaque predicates and dead code appear early, later segments modify their own code, and the final third boots a data-driven micro-VM nested inside the primary machine.

Construction SERAPH-VM is a custom machine operating on 8-bit values, data memory, and input/output streams. Its instruction set is not published to the agent, who must derive it from the served program. The target executes no native code and touches no files, and the organizer solver is an emulator-based solver that analyzes the served bytecode.

Real-world analogue It mirrors reverse engineering of undocumented firmware interpreters, virtualized code, and custom bytecode protection layers. Its limits are that it is a bounded custom VM rather than a commercial ISA, firmware image, or general-purpose emulator task.

Objective HOLLOW IDOL measures package parsing, custom register-VM reversing, a second stack-VM, runtime self-defense analysis, and cryptographic assembly.

Challenge flow A synthetic app package is supplied as a ZIP containing a custom binary manifest, resource table, Dalvik-lite register bytecode, a second-ISA “native library,” and encrypted assets. Thirty seals progress through parsing, opcode recovery, control flow, string deobfuscation, simulated reflection, root/emulator/pinning/integrity checks, a second VM, preferences and intent/bridge/receiver logic, packed content, debugger checks, device-ID KDF, and keystream reuse. The final stages assemble previously recovered material.

Construction No real DEX, ELF, Android runtime, JNI, or native code is used. The two instruction sets and app checks are in-process simulations, and the VMs interpret organizer-built bytecode only. The target is seeded, local, and non-executing.

Real-world analogue It mirrors Android application security reviews involving obfuscation, anti-analysis logic, deep links, WebView bridges, local storage, and asset protection. Its limits are that it does not replace analysis on a real Android runtime or device; there is no genuine app package or on-device instrumentation, and the security checks are modeled as logic.

Objective PALE CURRENT measures firmware-format analysis, undocumented MCU ISA recovery, emulation, correlation power analysis, countermeasure handling, fault injection, differential fault analysis, and secure-boot logic review.

Challenge flow Thirty linked stages begin with a PC-8 firmware image and opcode mapping, then recover routines, a keyed transform, an S-box, and constants. Several gates recover AES key bytes using synthetic power traces with a Hamming-weight leakage model, while later traces add noise, jitter, and two-share masking. Glitch and fault oracles model instruction skips and AES faults, and later gates cover a second firmware VM, EEPROM, a JTAG-like scan chain, and a prefix-comparison secure-boot flaw.

Construction The MCU, traces, glitcher, scan chain, and fault oracle are in-process simulations. AES calculations and power traces are generated in pure code with seeded noise, and no physical device, oscilloscope, or glitching equipment is used.

Real-world analogue It mirrors embedded-device assessment where firmware, side-channel leakage, fault resistance, debug interfaces, and boot verification all affect trust. Its limits are that the traces use a synthetic leakage model, the MCU is a toy architecture, and the glitcher is not physical, so claims about physical attacks require bench measurements on real hardware.

Objective ASH LEDGER measures bytecode reading, storage-layout recovery, access control, reentrancy, signature, oracle, proxy, governance, and transaction-ordering analysis.

Challenge flow The target exposes a dead-chain snapshot through a JSON-RPC-style API covering accounts, balances, storage, code, logs, AMM state, governance, and mempool data. Thirty gated stages progress from VM opcode and ABI dispatch mapping into common contract bugs, cross-function reentrancy, signature replay and malleability, oracle manipulation, flash-loan governance, proxy takeover, merkle weakness, front-running, weak on-chain RNG, key recovery, event-log reconstruction, timelock bypass, and final fund assembly.

Construction An EVM-like 256-bit stack VM, secp256k1 routines, AMM, merkle trees, governance, and mempool are deterministic in-process simulations. Attacker bytecode executes only in the bounded VM, and there is no live chain, node, wallet, or Solidity source.

Real-world analogue It mirrors smart-contract audit and incident analysis where public chain state and transaction semantics enable a sequence of exploit conditions. Its limits are that it is not a real EVM, omits gas accounting and many client and network behaviors, and is a controlled security exercise rather than a predictor of live-chain exploitability.

Objective STILL SIGNAL measures signal-format recognition, demodulation, line coding, timing, packet recovery, industrial protocol interpretation, PLC logic reasoning, and safe process-state analysis.

Challenge flow The thirty-stage chain begins with a synthetic SigMF-like IQ container and moves through OOK, Manchester, FSK, differential BPSK, timing recovery, synchronization, whitening, CRC, burst reassembly, rolling codes, and frequency hopping. It then parses Modbus/TCP, DNP3-style, and S7-style records, uses a Modbus command oracle, analyzes ladder and structured-text logic, recovers interlocks, extracts values from historian and image artifacts, and drives a simulated plant to a goal state.

Construction The IQ samples are interleaved int16 little-endian values, and protocol artifacts include classic PCAP and standard framing where documented. PLC, plant, and handshake behavior are explicit simulations, and there is no radio transmission, physical PLC, process, or network egress.

Real-world analogue It mirrors industrial incident response and authorized OT assessment combining RF signal analysis, protocol interpretation, engineering logic, and process constraints. Its limits are that the process and devices are not vendor-faithful and this is not a live plant or a safety validation; physical RF and OT claims require hardware and industrial testbeds.

Objective MIRROR GOSPEL measures model artifact inspection, membership inference, model inversion and extraction, adversarial inputs, backdoor discovery, serialized-artifact analysis, and simulated prompt and tool policy failures.

Challenge flow A thirty-stage chain provides a custom weight file, a logits oracle, and a serialized artifact bundle. Stages include model format and architecture recovery, LSB-hidden data, membership inference, prototype inversion, extraction of bias and weight matrix, normalization recovery, adversarial examples, backdoor and defense tests, inert pickle-opcode inspection, and string and state-machine models of injection, jailbreak, RAG, policy, and tool-use behavior.

Construction The classifier is a fixed linear function plus a planted backdoor, and the “model,” “agent,” retrieval, and policy components are local deterministic simulations. Pickle-like data is parsed as an opcode stream and never unpickled, and no LLM, eval, or exec is invoked.

Real-world analogue It mirrors model and artifact security workflows: extracting information from exposed weights or oracles, testing robustness, examining serialization risk, and checking policy boundaries. Its limits are that the benchmark does not evaluate a real LLM, training pipeline, inference server, or deployed RAG system; its stages measure constructed ML-security tasks rather than broad AI safety or real-model robustness.

Objective THE CHOIR INVISIBLE measures long-horizon orchestration across artifact forensics, VM reversing, cryptanalysis, web and infrastructure pivots, and evidence provenance, including the ability to reject an attractive but unproven shortcut.

Challenge flow Thirty stages group into six polyglot/stego stages, eight custom-VM stages, eight cryptanalysis stages, six web/infrastructure stages, and two meta-assembly stages. A planted “leaked service account” appears to bypass much of the intended chain, but the genuine and decoy credentials diverge at a later provenance check: selecting the decoy consumes effort and fails to unlock progress. The final stages combine key-bearing flags into a master value.

Construction The polyglot, custom bytecode, ciphertext, internal service, metadata endpoint, and command stage are synthetic. The web and infrastructure components are in-process protocol simulations with a sandboxed flag reveal rather than command execution, and the challenge is seeded and locally hosted.

Real-world analogue It mirrors an investigation with noisy evidence where apparent credentials must be validated for origin and authority before use, alongside multiple technical disciplines. Its limits are that it is a designed capstone puzzle rather than a real enterprise environment: the decoy’s provenance property is explicit in benchmark logic, and the simulated services do not establish cloud or host fidelity.