From YAML to Queue: DevOps Automation for Reproducible Benchmarks

• Jayantjaishwin Shanmugam Kavitha

  The University of Texas at Dallas

  Author

This paper reframes scientific benchmarking as "experiments-as-code," presenting a DevOps automation approach that codifies multi‑step runs, hyperparameter sweeps, and cross‑site portability in declarable runbooks executed against heterogeneous compute tiers. A pair of complementary Python‑based orchestration stacks demonstrates how containerized tasks, scheduler adapters, and templated workflows enable continuous, repeatable experiment execution from laptops to leadership‑class clusters and public cloud. The approach emphasizes Git‑driven configuration, immutable artifacts, and environment capture to strengthen provenance and FAIR‑aligned reproducibility, while policy‑aware submission integrates with batch systems and secure remote access patterns. Iteration‑first specifications (loops, arrays, and conditional stages) simplify large ensembles beyond DAG‑only models, and cost‑aware planning supports pragmatic cloud bursts without sacrificing portability. The result is a practical blueprint for applying CI/CD‑style discipline to versioned configurations, automated provisioning, template reuse, and consistent reporting to scientific benchmarking at scale. By aligning workflow definition, execution, and evidence collection with DevOps practices, the framework reduces operational toil, shortens feedback cycles for model and system tuning, and promotes shareable templates that accelerate onboarding and collaboration across research teams.

• PDF

Copyright (c) 2026 International Journal of Intelligent Systems and Data Science

This work is licensed under a Creative Commons Attribution 4.0 International License.