Machine Learning Engineer — Training Optimization

This position is listed on behalf of a partner company, who manages all applications and next steps.

Our partner is looking for a Machine Learning Engineer — Training Optimization based in Germany.

This role offers the opportunity to improve the foundations behind large-scale AI model development and deployment.

You will work at the intersection of machine learning research, systems engineering, and production optimization.

The position focuses on making model training faster, more stable, and more cost-efficient through advanced engineering techniques.

You will optimize training pipelines, improve distributed systems, and collaborate with researchers to push model capabilities forward.

This is a high-impact opportunity for an engineer who enjoys solving complex performance challenges at scale.

You will have significant ownership in shaping training infrastructure, experimentation workflows, and the future of AI systems.

Accountabilities

As a Machine Learning Engineer focused on Training Optimization, you will improve the efficiency, scalability, and reliability of large-scale model training systems.

You will combine deep technical expertise with practical engineering execution to optimize how advanced AI models are developed.

• Optimize large-scale model training pipelines to improve throughput, convergence, stability, and overall computational efficiency.
• Improve distributed training approaches, including data parallelism, model parallelism, and pipeline parallelism strategies.
• Tune key training components such as optimizers, learning rate schedulers, batch sizes, and numerical precision methods including bf16, fp16, and fp8.
• Identify and resolve performance bottlenecks through profiling, system analysis, and infrastructure-level improvements.
• Collaborate closely with research teams to develop architecture-aware training strategies and improve model performance.
• Build and maintain reliable training infrastructure, including checkpointing systems, fault tolerance mechanisms, and reproducible workflows.
• Evaluate and integrate advanced training techniques such as gradient checkpointing, Ze RO, FSDP, and custom optimization solutions.
• Define, monitor, and improve training performance metrics to continuously enhance efficiency.
• Translate research concepts into production-ready systems and scalable engineering solutions.

Requirements

The ideal candidate is a machine learning engineer with strong experience in training large neural networks and optimizing complex AI systems.

You should be comfortable working across research and engineering environments while solving challenging scalability and performance problems.

• Strong experience training large-scale neural networks, including large language models or similarly complex architectures.
• Hands-on experience with machine learning training optimization, beyond simply using existing models.
• Strong understanding of backpropagation, optimization algorithms, training dynamics, and model convergence behavior.
• Experience with distributed machine learning training systems and large-scale computing environments.
• Proficiency with Py Torch and modern machine learning development workflows.
• Ability to work close to hardware constraints, including GPU performance, memory limitations, and networking considerations.
• Strong programming skills with the ability to transform research ideas into reliable production code.
• Experience with multi-node and multi-GPU training environments is highly preferred.
• Familiarity with frameworks and technologies such as Deep Speed, FSDP, Megatron, or custom training stacks is a plus.
• Experience optimizing workloads on NVIDIA or AMD GPU platforms is beneficial.
• Contributions to open-source machine learning infrastructure or research projects are valued.
• Exposure to alternative neural network architectures beyond Transformers is a plus.

Benefits

• Competitive compensation package with meaningful equity opportunities.
• Opportunity to work on cutting-edge AI models and large-scale training systems.
• High ownership role where your contributions directly influence technical direction and company growth.
• Collaboration with a small, highly technical team focused on engineering excellence and research innovation.
• Fast feedback loops and an environment that encourages experimentation and impact.
• Opportunity to solve complex machine learning infrastructure challenges at significant scale.
• Strong emphasis on technical quality, continuous learning, and advanced AI development.
• Ability to contribute to foundational systems shaping future AI capabilities.

How Jobgether works

We use an

AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements.

Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company.

The final decision and next steps (interviews, assessments) are managed by their internal team.

We appreciate your interest and wish you the best!

Why Apply Through Jobgether?

Data Privacy Notice

By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer.

This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR).

You may exercise your rights (access, rectification, erasure, objection) at any time.

#LI-CL1

We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses and identifying potential inconsistencies or verification signals in application materials based on available information.

These tools assist our recruitment team but do not replace human judgment.

Final hiring decisions are ultimately made by humans.

If you would like more information about how your data is processed, please contact us.