Remote
Member of Technical Staff, Evals Platform
About this role
ABOUT US: Fireworks is the platform for specialized intelligence, enabling companies to build, train, and serve AI models tailored to their own data, workflows, and products. Founded by the team behind PyTorch and backed by AMD, Atreides, Benchmark Capital, Index Ventures, Lightspeed, NVIDIA, Sequoia Capital, and TCV, Fireworks powers production AI with hundreds of state-of-the-art open models across text, image, embedding, audio, and multimodal workloads.
Today, Fireworks is a Series D company valued at $17.5 billion, bringing together an ambitious, collaborative team that's building the future of enterprise AI. We are seeking a Member of Technical Staff, Evals & Post-Training Product to help define how developers improve models on Fireworks. This role sits at a unique intersection of scalable system design, deep data science, and model quality. You will build the infrastructure and workflows that connect evaluation and post-training.
Our evaluation setup has grown past its original scope, and we need someone who can take it to the next stage, improving programmatic access and scale. You will work across backend systems, sandbox infrastructure, and user-facing surfaces to make it easier to author evals, understand results, and iterate quickly. KEY RESPONSIBILITIES - Scale Eval Infrastructure: Take ownership of our internal eval setup and evolve it for the future.
You will design systems to eliminate single-host coordination bottlenecks, resolve log-syncing latency, and build seamless programmatic access. - Benchmark Obsession & Reproduction: Act as a "metrics obsessive." Track state-of-the-art (SOTA) benchmarks, read the latest research papers, dig deep into data discrepancies, and insist on rigorously reproducing published results internally. - Pioneer New Benchmarks: Design and build entirely new benchmarks to measure model performance on complex, emerging, or domain-specific use cases.
- Own Fine-Tuning Product Experiences: Build and improve user-facing workflows for post-training, including fine-tuning experiences across SFT, RFT, and related model-improvement capabilities. - Work Closely With Users: Partner with customers and internal stakeholders to understand evaluation and fine-tuning needs, triage issues, and convert bespoke workflows into productized, reusable solutions. MINIMUM REQUIREMENTS - Experience: 1–7 years of software engineering or data science experience (We are hiring at multiple levels for this role).