Software Engineer

About this role

Project details The project focuses on building an AI-powered platform that enables production-grade LLM agents for complex audit workflows. It combines intelligent agents, evaluation pipelines, and observability tooling to continuously improve quality, reliability, performance, and cost efficiency in production. We're looking for an experienced engineer with strong data engineering and AI systems expertise to build the infrastructure that evaluates, monitors, and optimizes AI agents at scale.

You'll work across backend services, data pipelines, analytical databases, and agent architecture, designing systems for evaluation, experimentation, logging, tracing, and debugging. Many of the challenges involve building reliable infrastructure around non-deterministic AI systems with no established blueprint to follow. You should be comfortable working in complex backend environments, making sound architectural decisions, and delivering scalable solutions that help production AI systems continuously improve.

Key responsibilities You'll help design, build, and operate the infrastructure that supports our production AI agents, with a strong emphasis on data platforms, evaluation, observability, and continuous optimization. Your responsibilities will include: Developing online and offline evaluation frameworks for LLM agents using benchmark datasets, ground-truth labels, human feedback, and experimental results. Implementing automated validation pipelines that verify changes to prompts, models, context, and agent behavior before deployment.

Investigating large-scale execution logs and agent traces to uncover quality issues, performance bottlenecks, reliability problems, and opportunities to reduce inference costs. Designing and maintaining data solutions built on analytical databases and column-oriented storage technologies such as BigQuery, ClickHouse, or comparable platforms. Building systems for long-term storage, replay, and analysis of production agent activity to support debugging and continuous improvement.

Creating observability capabilities, including monitoring, trace inspection, logging, dashboards, and debugging tools for AI services. Contributing directly to the backend platform and agent ecosystem by enhancing existing AI agents and developing new capabilities when required. Requirements Solid experience developing backend applications in Python or similar server-side technologies. Confident writing advanced SQL queries and working with large-scale datasets.