Machine Learning Engineer — Training Optimization

Remote, Worldwide

About the Role

We’re looking for an ML Engineer focused on training optimization to help us scale and improve large-scale model training. You’ll work at the intersection of research and production, optimizing training pipelines for speed, stability, and cost—while collaborating closely with researchers pushing model architecture and capability forward.

This is a high-impact role with real ownership: your work directly affects how fast we can iterate, how large we can scale, and how efficiently we deploy new models.

What You’ll Do

Optimize large-scale model training pipelines (throughput, convergence, stability, and cost)

Improve distributed training strategies (data, model, and pipeline parallelism)

Tune optimizers, schedulers, batch sizing, and precision (bf16 / fp16 / fp8)

Reduce training time and compute cost via profiling, bottleneck analysis, and systems-level improvements

Collaborate with researchers on architecture-aware training strategies

Build and maintain robust training infrastructure (checkpointing, fault tolerance, reproducibility)

Evaluate and integrate new training techniques (e.g. gradient checkpointing, ZeRO, FSDP, custom kernels)

Own training...