Harbor Visualiser
π
Browse Harbor task specs from HF, GitHub, or local
AI & ML interests
Aligning LLMs to be helpful, honest, harmless, and huggy (H4)
Recent Activity
View all activity
AdithyaSKΒ
updated a
Space about 8 hours ago
alvarobarttΒ
posted an update about 23 hours ago
Post
820
Latest
TL; DR MoEs can be misleading to reason about from active parameters alone, since each token only activates a subset of experts, while the serving setup still needs to account for the full resident memory footprint.
π§
ποΈ Dense models usually load and use most weights every forward pass, while MoEs load many experts but only route each token to a few of them
β‘ Active params isn't the same as memory footprint, especially for sparse architectures
π¦ Runtime memory is about what is used per request/token, while loading memory also includes the expert weights that need to be resident
π KV cache can still dominate depending on context length, batch size, and concurrency
π Expert Parallelism (EP) helps shard experts across accelerators when expert weights dominate
π Data Parallelism (DP) + EP is often a good fit for throughput-oriented MoE serving
Check the repository at https://github.com/alvarobartt/hf-mem
hf-mem release added a breakdown of Mixture-of-Experts (MoE) memory usage!TL; DR MoEs can be misleading to reason about from active parameters alone, since each token only activates a subset of experts, while the serving setup still needs to account for the full resident memory footprint.
π§
hf-mem now splits MoE memory into base model weights, routed experts, and KV cacheποΈ Dense models usually load and use most weights every forward pass, while MoEs load many experts but only route each token to a few of them
β‘ Active params isn't the same as memory footprint, especially for sparse architectures
π¦ Runtime memory is about what is used per request/token, while loading memory also includes the expert weights that need to be resident
π KV cache can still dominate depending on context length, batch size, and concurrency
π Expert Parallelism (EP) helps shard experts across accelerators when expert weights dominate
π Data Parallelism (DP) + EP is often a good fit for throughput-oriented MoE serving
Check the repository at https://github.com/alvarobartt/hf-mem
qgallouedecΒ
posted an update 9 days ago
Post
9918
Shipped hf-sandbox! π₯‘
π§ͺ Running an eval that executes model-generated C on a few thousand prompts? You probably don't want any of that on your laptop.
Just shipped hf-sandbox, a Modal-style sandbox API on top of Hugging Face Jobs. Spin up an isolated, ephemeral container, run untrusted code, get the result back. No Docker on your laptop, no infra to manage.
Just pip install hf-sandbox.
Early days (v0.1); feedback and issues very welcome:
π https://github.com/huggingface/hf-sandbox
π§ͺ Running an eval that executes model-generated C on a few thousand prompts? You probably don't want any of that on your laptop.
Just shipped hf-sandbox, a Modal-style sandbox API on top of Hugging Face Jobs. Spin up an isolated, ephemeral container, run untrusted code, get the result back. No Docker on your laptop, no infra to manage.
Just pip install hf-sandbox.
Early days (v0.1); feedback and issues very welcome:
π https://github.com/huggingface/hf-sandbox
- 1 reply
qgallouedecΒ
posted an update 11 days ago
Post
257
**TRL v1.4 is out π** Chunked NLL loss for SFT and a first-class **OpenReward** integration.
**Chunked NLL loss for SFT β drops peak VRAM by up to 14Γ**
Standard SFT materializes a full
Peak GPU memory, AdamW fp32:
- Qwen3-14B, 8ΓH100 FSDP2, 16k seq: 58.9 GB β 38.9 GB
- Qwen3-4B, 1ΓH100 80GB, 16k seq: OOM β 63.8 GB
- Qwen3-32B, 8ΓH100 FSDP2, 8k seq: OOM β 71.2 GB
End-to-end it's consistently as fast or faster than
Works with PEFT and VLMs out of the box.
**Open Reward Standard environment adapter**
The new
v1.4 also brings MFU helpers for dense + MoE models, GRPO support for Liger 0.8.0 (delta clipping + VESPO + KL bias correction), TΓΌlu 3's length-normalized DPO loss, four more training chat templates (Cohere, Cohere2, Gemma 3, Qwen3-2507), and a 5+ GB CUDA memory leak fix in activation offloading.
Full release notes: https://github.com/huggingface/trl/releases/tag/v1.4.0
**Chunked NLL loss for SFT β drops peak VRAM by up to 14Γ**
Standard SFT materializes a full
[batch Γ seq Γ vocab] logits tensor before computing cross-entropy, which dominates peak memory at long context lengths. The new loss_type="chunked_nll" path drops ignored-label tokens before the lm_head matmul and computes cross-entropy in checkpointed chunks of 256.Peak GPU memory, AdamW fp32:
- Qwen3-14B, 8ΓH100 FSDP2, 16k seq: 58.9 GB β 38.9 GB
- Qwen3-4B, 1ΓH100 80GB, 16k seq: OOM β 63.8 GB
- Qwen3-32B, 8ΓH100 FSDP2, 8k seq: OOM β 71.2 GB
End-to-end it's consistently as fast or faster than
nll, and unlocks sequence lengths that don't fit at all under the standard path.SFTConfig(loss_type="chunked_nll")Works with PEFT and VLMs out of the box.
**Open Reward Standard environment adapter**
The new
trl.experimental.openreward adapter plugs any environment speaking the [Open Reward Standard](https://openrewardstandard.io) protocol into any TRL trainer that takes an environment_factory. One string β a catalog name or a URL β wires the dataset, factory, and reward_func slots; tools are bound dynamically from JSON Schema, no per-env wrapper code:from trl import GRPOTrainer
from trl.experimental.openreward import OpenRewardSpec
spec = OpenRewardSpec("Eigent/SETA", num_tasks=64)
trainer = GRPOTrainer(
...,
train_dataset=spec.train_dataset,
environment_factory=spec.environment_factory,
reward_funcs=spec.reward_funcs,
)v1.4 also brings MFU helpers for dense + MoE models, GRPO support for Liger 0.8.0 (delta clipping + VESPO + KL bias correction), TΓΌlu 3's length-normalized DPO loss, four more training chat templates (Cohere, Cohere2, Gemma 3, Qwen3-2507), and a 5+ GB CUDA memory leak fix in activation offloading.
Full release notes: https://github.com/huggingface/trl/releases/tag/v1.4.0
sergiopaniegoΒ
posted an update 11 days ago
Post
1763
OpenEnv is growing fast in tutorials. If you're looking to get started with RL environments, check them out
> evaluate your agents using OpenEnv
> learn how rewards work via rubrics
> connect agents via MCP
> many moreeeee!
anything you think it's missing?
https://meta-pytorch.org/OpenEnv/tutorials/index.html
> evaluate your agents using OpenEnv
> learn how rewards work via rubrics
> connect agents via MCP
> many moreeeee!
anything you think it's missing?
https://meta-pytorch.org/OpenEnv/tutorials/index.html
sergiopaniegoΒ
posted an update 12 days ago
Post
800
OpenEnv already ships π’ with a ready-to-deploy RLM environment on free HF Spaces
Drop "Attention Is All You Need", write code that spawns parallel LLM calls β β correct answer, reward 1.0, in 4.2s
Run GRPO (TRL) β model learns to write that search strategy itself
test it yourself β sergiopaniego/repl-env
check out OpenEnv β https://github.com/meta-pytorch/OpenEnv
Drop "Attention Is All You Need", write code that spawns parallel LLM calls β β correct answer, reward 1.0, in 4.2s
Run GRPO (TRL) β model learns to write that search strategy itself
test it yourself β sergiopaniego/repl-env
check out OpenEnv β https://github.com/meta-pytorch/OpenEnv
qgallouedecΒ
posted an update 23 days ago
Post
8007
TRL v1.3 ships day-one training support for Qwen 3.6 π
The new Qwen 3.6 family (
Qwen/Qwen3.6-27B, Qwen/Qwen3.6-35B-A3B) reuses the Qwen3.5-MoE architecture but ships a slightly different chat template, so we updated the stack end-to-end: new training template with {% generation %} markers, tool-call response schema routing, tiny test models for the VLM matrix.SFT with assistant-only loss works out of the box:
from trl import SFTConfig, SFTTrainer
trainer = SFTTrainer(
model="Qwen/Qwen3.6-27B",
args=SFTConfig(assistant_only_loss=True),
train_dataset=dataset,
)
trainer.train()So does GRPO tool-calling β just hand
tools=[...] to GRPOTrainer.v1.3 also brings a new experimental TPO trainer (Triple Preference Optimization), speculative decoding in
trl vllm-serve (Qwen3 MTP / Eagle3 drafts), 12 more KTO β DPO alignment PRs (KTO promotion to stable is now in reach), three more {% generation %} chat templates (Gemma/Gemma 2, Phi-3, GLM-4-MoE), and a chunky SFT entropy bug fix.Full release notes: https://github.com/huggingface/trl/releases/tag/v1.3.0
qgallouedecΒ
posted an update about 1 month ago
Post
1987
TRL v1.2 introduces the SSDTrainer π
Simple Self-Distillation (SSD) from Apple's paper "Embarrassingly Simple Self-Distillation Improves Code Generation" is now available as an experimental trainer in TRL.
The recipe is as minimal as the name suggests: sample completions from the model itself at a training-time temperature, then fine-tune on those raw, unverified samples with plain cross-entropy. No reward model. No verifier. No teacher model. No reinforcement learning. Just prompts and the model.
v1.2 also ships expanded tool-calling support (LLaMA 3.1 / 3.2, DeepSeek-V3), another round of KTO β DPO alignment getting us closer to promoting KTO to stable, a big GRPO simplification for overlong tool results, deprecation of
Full release notes: https://github.com/huggingface/trl/releases/tag/v1.2.0
Simple Self-Distillation (SSD) from Apple's paper "Embarrassingly Simple Self-Distillation Improves Code Generation" is now available as an experimental trainer in TRL.
The recipe is as minimal as the name suggests: sample completions from the model itself at a training-time temperature, then fine-tune on those raw, unverified samples with plain cross-entropy. No reward model. No verifier. No teacher model. No reinforcement learning. Just prompts and the model.
from trl.experimental.ssd import SSDConfig, SSDTrainer
trainer = SSDTrainer(
model="Qwen/Qwen3-4B-Instruct",
args=SSDConfig(temperature=0.6, top_k=20, top_p=0.95),
train_dataset=dataset,
)
trainer.train()v1.2 also ships expanded tool-calling support (LLaMA 3.1 / 3.2, DeepSeek-V3), another round of KTO β DPO alignment getting us closer to promoting KTO to stable, a big GRPO simplification for overlong tool results, deprecation of
use_transformers_paged, and key fixes for VLM response parsing.Full release notes: https://github.com/huggingface/trl/releases/tag/v1.2.0
sergiopaniegoΒ
posted an update about 1 month ago
Post
1377
Earlier this month, Apple introduced Simple Self-Distillation: a fine-tuning method that improves models on coding tasks just by sampling from the model and training on its own outputs with plain cross-entropy
Andβ¦ it's already supported in TRL, built by Kashif Rasul. you can really feel the pace of development in the team π
Paper by Ruixiang ZHANG, He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, Yizhe Zhang at Apple π
How it works: the model generates completions at a training-time temperature (T_train) with top_k/top_p truncation, then fine-tunes on them with plain cross-entropy. no labels or verifier needed
You can try it right away with this ready-to-run example (Qwen3-4B on rStar-Coder):
https://github.com/huggingface/trl/blob/main/trl/experimental/ssd/ssd.py
or benchmark a checkpoint with the eval script:
https://github.com/huggingface/trl/blob/main/trl/experimental/ssd/ssd_eval.py
One neat insight from the paper: T_train and T_eval compose into an effective T_eff = T_train Γ T_eval, so a broad band of configs works well. even very noisy samples still help
Want to dig deeper?
Paper: Embarrassingly Simple Self-Distillation Improves Code Generation (2604.01193)
Trainer docs: https://huggingface.co/docs/trl/main/en/ssd_trainer
Andβ¦ it's already supported in TRL, built by Kashif Rasul. you can really feel the pace of development in the team π
Paper by Ruixiang ZHANG, He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, Yizhe Zhang at Apple π
How it works: the model generates completions at a training-time temperature (T_train) with top_k/top_p truncation, then fine-tunes on them with plain cross-entropy. no labels or verifier needed
You can try it right away with this ready-to-run example (Qwen3-4B on rStar-Coder):
https://github.com/huggingface/trl/blob/main/trl/experimental/ssd/ssd.py
or benchmark a checkpoint with the eval script:
https://github.com/huggingface/trl/blob/main/trl/experimental/ssd/ssd_eval.py
One neat insight from the paper: T_train and T_eval compose into an effective T_eff = T_train Γ T_eval, so a broad band of configs works well. even very noisy samples still help
Want to dig deeper?
Paper: Embarrassingly Simple Self-Distillation Improves Code Generation (2604.01193)
Trainer docs: https://huggingface.co/docs/trl/main/en/ssd_trainer
sergiopaniegoΒ
posted an update about 1 month ago
Post
470
Great experience yesterday at PyTorch Conf Europe in Paris π«π·
We (w/ @kashif ) talked about training LLMs through interaction, using trajectories across games, browsers, or simulators
Room was packed, a clear sign of interest in where RL post-training is heading.
sharing the slides! π€
https://drive.google.com/file/d/16k7YRnf5EJEo0XjXGlRJ_hVeLoFWKyNP/view?usp=sharing
We (w/ @kashif ) talked about training LLMs through interaction, using trajectories across games, browsers, or simulators
Room was packed, a clear sign of interest in where RL post-training is heading.
sharing the slides! π€
https://drive.google.com/file/d/16k7YRnf5EJEo0XjXGlRJ_hVeLoFWKyNP/view?usp=sharing
sergiopaniegoΒ
posted an update about 2 months ago
Post
2872
Gemma 4 π is here and itβs strong!
to celebrate, weβre rolling out in TRL:
> support for multimodal tool responses for environments (OpenEnv)
> an example to train it in CARLA for autonomous driving with image-based tool calls
go check it out ποΈποΈ
blog: https://huggingface.co/blog/gemma4
script: https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/carla_vlm_gemma.py
to celebrate, weβre rolling out in TRL:
> support for multimodal tool responses for environments (OpenEnv)
> an example to train it in CARLA for autonomous driving with image-based tool calls
go check it out ποΈποΈ
blog: https://huggingface.co/blog/gemma4
script: https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/carla_vlm_gemma.py
sergiopaniegoΒ
posted an update about 2 months ago
Post
2090
TRL is officially an adult π₯³
excited to announce TRL v1.0βοΈ
head to the blog to see how we got here and whatβs next for this post-training library, designed to keep pace with the field
https://huggingface.co/blog/trl-v1
excited to announce TRL v1.0βοΈ
head to the blog to see how we got here and whatβs next for this post-training library, designed to keep pace with the field
https://huggingface.co/blog/trl-v1
- 2 replies
qgallouedecΒ
posted an update about 2 months ago
Post
2434
TRL v1.0 is out!
Hugging Face's TRL library is downloaded 3 million times a month. Over 130k models trained with it are public on the Hub, and major projects like @unsloth and @axolotl-ai-co build directly on top of it. v1.0 is the moment we acknowledged that responsibility explicitly, with a real stability contract.
The field hasn't settled. Building stable software in a domain that keeps invalidating its own assumptions is the actual problem we're solving. The answer is a design that can absorb the next shift without breaking what people rely on.
What's in v1.0:
Deep Hugging Face integration, low infrastructure burden
What's next: asynchronous GRPO, better scaling support, and making training legible enough that agents can inspect and steer it.
Read more: hf.co/blog/trl-v1
Hugging Face's TRL library is downloaded 3 million times a month. Over 130k models trained with it are public on the Hub, and major projects like @unsloth and @axolotl-ai-co build directly on top of it. v1.0 is the moment we acknowledged that responsibility explicitly, with a real stability contract.
The field hasn't settled. Building stable software in a domain that keeps invalidating its own assumptions is the actual problem we're solving. The answer is a design that can absorb the next shift without breaking what people rely on.
What's in v1.0:
Deep Hugging Face integration, low infrastructure burden
What's next: asynchronous GRPO, better scaling support, and making training legible enough that agents can inspect and steer it.
pip install --upgrade trlRead more: hf.co/blog/trl-v1
sergiopaniegoΒ
posted an update 2 months ago
Post
832
ICYMI, great blog by @kashif and @stas on Ulysses Sequence Parallelism: train with million-token contexts
on 4ΓH100s: 12x longer sequences, 3.7x throughput
learn how to integrate it with Accelerate, Transformers, and TRL ‡οΈ
https://huggingface.co/blog/ulysses-sp
on 4ΓH100s: 12x longer sequences, 3.7x throughput
learn how to integrate it with Accelerate, Transformers, and TRL ‡οΈ
https://huggingface.co/blog/ulysses-sp
clefourrierΒ
authored a
paper 2 months ago
sergiopaniegoΒ
posted an update 2 months ago
Post
491
We just released a big blog surveying 16 OSS frameworks for async RL training of LLMs!
We're building a new async GRPO trainer for TRL and as first step, we needed to understand how the ecosystem solves this problem today.
The problem: in synchronous RL training, generation dominates wall-clock time. 32K-token rollouts on a 32B model take hours while training GPUs sit completely idle. With reasoning models and agentic RL making rollouts longer and more variable, this only gets worse.
The ecosystem converged on the same fix: separate inference + training onto different GPU pools, rollout buffer, and async weight sync.
We compared 16 frameworks across 7 axes: orchestration, buffer design, weight sync, staleness management, partial rollouts, LoRA, and MoE support.
This survey is step one. The async GRPO trainer for TRL is next!
https://huggingface.co/blog/async-rl-training-landscape
We're building a new async GRPO trainer for TRL and as first step, we needed to understand how the ecosystem solves this problem today.
The problem: in synchronous RL training, generation dominates wall-clock time. 32K-token rollouts on a 32B model take hours while training GPUs sit completely idle. With reasoning models and agentic RL making rollouts longer and more variable, this only gets worse.
The ecosystem converged on the same fix: separate inference + training onto different GPU pools, rollout buffer, and async weight sync.
We compared 16 frameworks across 7 axes: orchestration, buffer design, weight sync, staleness management, partial rollouts, LoRA, and MoE support.
This survey is step one. The async GRPO trainer for TRL is next!
https://huggingface.co/blog/async-rl-training-landscape
sergiopaniegoΒ
posted an update 2 months ago
Post
454
Nemotron 3 Super by @nvidia is here! NVIDIA's hybrid Mamba2/Transformer models are now natively supported in transformers (no trust_remote_code needed)
Fine-tune them with TRL in just a few lines of code. Notebook + script included to get started right away. goooo!
- Notebook: https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_nemotron_3.ipynb
- Script: https://github.com/huggingface/trl/blob/main/examples/scripts/sft_nemotron_3.py
- Collection with all the models: https://huggingface.co/collections/nvidia/nvidia-nemotron-v3
Fine-tune them with TRL in just a few lines of code. Notebook + script included to get started right away. goooo!
- Notebook: https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_nemotron_3.ipynb
- Script: https://github.com/huggingface/trl/blob/main/examples/scripts/sft_nemotron_3.py
- Collection with all the models: https://huggingface.co/collections/nvidia/nvidia-nemotron-v3
alvarobarttΒ
posted an update 2 months ago
Post
3729
Learn how to deploy Microsoft Research VibeVoice ASR on Microsoft Azure Foundry with Hugging Face to generate rich audio transcriptions with Who, When, and What! π₯
> π 60-minute single-pass processing, no chunking or stitching
> π€ Customized hotwords to guide recognition on domain-specific content
> π Rich transcription: joint ASR + diarization + timestamping in one pass
> π 50+ languages with automatic detection and code-switching support
> π€ Deployed on Microsoft Foundry via an OpenAI-compatible Chat Completions API
https://huggingface.co/docs/microsoft-azure/foundry/examples/deploy-vibevoice-asr
> π 60-minute single-pass processing, no chunking or stitching
> π€ Customized hotwords to guide recognition on domain-specific content
> π Rich transcription: joint ASR + diarization + timestamping in one pass
> π 50+ languages with automatic detection and code-switching support
> π€ Deployed on Microsoft Foundry via an OpenAI-compatible Chat Completions API
https://huggingface.co/docs/microsoft-azure/foundry/examples/deploy-vibevoice-asr
sergiopaniegoΒ
posted an update 3 months ago
Post
673
did you know you can train agentic models with RL deploying the environments on HF Spaces? π€
with TRL + OpenEnv, your training script connects to remote environments hosted as Spaces
want to train faster? β just add more Spaces (TRL handles the parallelization natively)
we used this to train a model to solve the trolley problem in CARLA. 2 HF Spaces running a full driving simulator, each on a T4 GPU
full write-up with code and results β https://huggingface.co/blog/sergiopaniego/bringing-carla-to-openenv-trl
with TRL + OpenEnv, your training script connects to remote environments hosted as Spaces
want to train faster? β just add more Spaces (TRL handles the parallelization natively)
we used this to train a model to solve the trolley problem in CARLA. 2 HF Spaces running a full driving simulator, each on a T4 GPU
full write-up with code and results β https://huggingface.co/blog/sergiopaniego/bringing-carla-to-openenv-trl