Upload PLAN_AND_PROGRESS.md with huggingface_hub

PLAN_AND_PROGRESS.md

@@ -101,68 +101,511 @@ We use a balanced mix of datasets to cover various audio qualities and transcrip
 
 ---
 
+## 📊 Training Data Analysis
+
+This section documents the composition and length distribution of our training data (from `RASMUS/canary-finnish-asr-data`, accessed 2026-02-26).
+
+### Dataset Summary
+
+| Dataset | Samples | Mean Duration | Max Duration | Total Hours |
+|---------|---------|--------------|-------------|-------------|
+| **Common Voice v24** | 9,086 | 4.5s | 10.5s | 11.2h |
+| **VoxPopuli** | 8,164 | 10.1s | 50.5s | 23.0h |
+| **CSS10** | 3,226 | 7.7s | 20.2s | 6.9h |
+| **FLEURS** | 2,704 | 11.7s | 43.2s | 8.8h |
+| **TOTAL** | **23,180** | **7.8s** | **50.5s** | **~50h** |
+
+### Duration Distribution (Training Set)
+
+```
+ 0–5s   : 33.3%  (7,725 samples)  ████████████████████████████████████████
+ 5–10s  : 43.7%  (10,139 samples) █████████████████████████████████████████████████████
+10–15s  : 15.0%  (3,473 samples)  ██████████████████
+15–20s  :  5.4%  (1,241 samples)  ██████
+20–30s  :  2.4%  (562 samples)    ███
+  >30s  :  0.2%  (40 samples)
+```
+
+**Key insight:** 77% of training samples are shorter than 10 seconds. The model has very little exposure to longer audio segments (only 0.2% are >30s). This has direct implications for long-form inference stability.
+
+### Evaluation Set Durations
+
+| Eval Set | Samples | Mean Duration | Max Duration |
+|----------|---------|--------------|-------------|
+| FLEURS   | 918     | 13.0s        | 33.7s       |
+| Common Voice | 1,554 | 5.1s       | 10.5s       |
+| CSS10    | 170     | 7.5s         | 10.2s       |
+| VoxPopuli | 430   | 10.6s        | 47.5s       |
+
+---
+
+## 🔢 Number Handling Analysis
+
+### Live Inference Results: Base vs Finetuned (2026-02-26)
+
+We ran both models on 5 FLEURS test samples to determine each model's number output style.
+
+| # | Scenario | Reference | Base Canary-v2 | Our Finetuned |
+|---|----------|-----------|----------------|---------------|
+| 1 | Spoken "sata" (hundred) | `yli sata vuotta` | `yli 100 vuotta` ❌ | `yli 100 vuotta` ❌ |
+| 2 | Spoken "seitsemäntoista" (17) | `surmaten seitsemäntoista henkeä` | `surmaten 17 henkeä` ❌ | `surmaten seitsemäntoista henkeä` ✅ |
+| 3 | Digits in reference (15, 2011, 2017) | `15 metriä... 2011... 2017` | Correct ✅ | Correct ✅ |
+| 4 | Abbreviation "jKr." (AD) | `400 jKr.` | `400 jälkeen Kristuksen` | `400 jälkeen Kristuksen` |
+| 5 | Range "25–30" (en-dash U+2013) | `25–30 vuodella` | `25-30 vuodella` (ASCII hyphen) | `25 ⁇ 30 vuodella` ❌ UNK token |
+
+**Key findings:**
+
+1. **Base model outputs digits.** When the speaker says "sata" (hundred) or "seitsemäntoista" (seventeen), the base Canary-v2 outputs `100` and `17`. This is NVIDIA's built-in text normalisation — Canary always outputs digit form for numbers.
+
+2. **Finetuning introduced inconsistency.** Our finetuning partially reversed this: for `seitsemäntoista` the finetuned model now outputs the written word (because FLEURS training transcripts used written-out numbers), but still outputs `100` for `sata`. This inconsistency is worse than either consistent policy.
+
+3. **En-dash produces a UNK token in the finetuned model.** The character `–` (U+2013 en-dash) in `25–30` causes the finetuned model to emit `⁇` (SentencePiece UNK). The base model degrades gracefully to an ASCII hyphen `25-30`. This is a regression introduced by finetuning — likely because the en-dash was absent or inconsistently encoded in our training data.
+
+4. **Abbreviations are expanded by both models.** `jKr.` → `jälkeen Kristuksen` in both — this is model behaviour, not a finetuning artifact.
+
+### Policy Decision
+**We want digit output** (not written-out Finnish number words). The base model's behaviour is correct here. The finetuned model regressed on consistency because our FLEURS training transcripts used written-out numbers.
+
+### Training Data Issues Found
+- Only **2.5% (578 / 23,180)** of training samples contain digit characters at all.
+- FLEURS transcripts use written-out numbers (`sata vuotta`) while VoxPopuli and Common Voice use digits. This gives the model conflicting signal.
+- En-dash (`–` U+2013) may be absent or mis-encoded in training manifests, causing UNK tokens at inference time.
+
+### Action Plan: Numbers & UNK Token
+
+#### Step 1 — Normalise training transcripts to digit form
+Run a pre-processing pass on `train_manifest.json` before the next training run:
+- Use the Python library `num2words` with locale `fi` to convert Finnish written-out numbers to digits: e.g. `sata` → `100`, `seitsemäntoista` → `17`.
+- OR (simpler / safer): replace the FLEURS transcripts in the manifest with their **raw reference texts which already have digits** (FLEURS provides both `raw_transcription` and `transcription` columns; currently we use `raw_transcription` which has written numbers).
+- Target: **all numeric quantities consistently in digit form** across all four datasets.
+
+#### Step 2 — Fix en-dash encoding (ROOT CAUSE CONFIRMED)
+
+**Confirmed via tokenizer inspection (2026-02-26):**
+
+```python
+m.tokenizer.text_to_ids("25–30")  # → [16053, 1125, 1128, 0, 1126, 1123]
+#                                              ↑ id 0 = UNK for the en-dash!
+m.tokenizer.text_to_ids("25-30")  # → [16053, 1125, 1128, 16107, 1126, 1123]
+#                                              ↑ ASCII hyphen tokenises correctly
+```
+
+- **En-dash `–` (U+2013) and em-dash `—` (U+2014) are NOT in the CanaryBPETokenizer vocabulary** (both map to UNK id 0).
+- Training data contains **85 entries with en-dash** (83 FLEURS, 2 Common Voice). During training, the en-dash in the TARGET text was encoded as UNK, so the model learned to produce UNK for the corresponding speech sounds.
+- **Fix: replace all `–` and `—` with ASCII hyphen `-` in all training transcripts** before the next training run. This is a one-line preprocessing step.
+
+```python
+# In manifest preprocessing:
+text = text.replace('\u2013', '-').replace('\u2014', '-')
+```
+
+#### Step 3 — Re-evaluate after normalisation
+After normalising transcripts, re-run the 5-sample live inference test to verify:
+- `sata vuotta` audio → model outputs `100 vuotta`
+- `seitsemäntoista` audio → model outputs `17`
+- `25–30` audio → model outputs `25-30` or `25–30` (no UNK)
+
+---
+
+## 🔈 Long-Form Audio: Root Cause Analysis
+
+Our test file `moo.wav` is **30 minutes** (1,800s) of continuous Finnish speech. This reveals a core gap vs. our finetuned Whisper model.
+
+### How Canary-v2 Handles Long Audio (Natively)
+- NVIDIA's Canary-v2 uses **dynamic chunking** with 1-second overlap between chunks.
+- This is automatically triggered for audio longer than **40 seconds**.
+- The model was pre-trained on a 1.7M-hour multilingual corpus with this chunking strategy baked in.
+
+### Our Current Approach (`inference_vad.py`)
+1. Silero VAD detects speech segments.
+2. Segments are merged into chunks up to `chunk_len` seconds (default: **15s**).
+3. Each chunk is transcribed **independently** — no shared context between chunks.
+
+### Root Causes of Degradation on Long-Form
+
+| Issue | Detail |
+|-------|--------|
+| **Training length mismatch** | 77% of fine-tuning data is <10s. Inference chunks at 15s are longer than nearly all training examples, creating distribution shift. |
+| **No cross-chunk context** | Each 15s chunk is transcribed in isolation. Canary's attention decoder has no memory of previous chunks, so topic/speaker continuity is lost at boundaries. |
+| **VAD vs. native chunking** | Our VAD-based approach differs from Canary's built-in dynamic chunking. The model was not fine-tuned with this chunking strategy. |
+| **Repetition / hallucination** | At chunk boundaries with silence or music, the decoder can loop. This is worsened when segments are near the edge of the model's training length distribution. |
+| **No overlap** | Without overlap between chunks, words at segment boundaries can be dropped or doubled. |
+
+### Comparison: Canary vs. Our Finetuned Whisper on Long-Form
+
+Whisper was explicitly designed and trained for long-form audio with:
+- Sliding window inference with overlap
+- Previous-chunk text as conditioning (prompt-based context)
+- Timestamps for alignment
+
+Canary's AED architecture does not use previous-chunk text as input, making long-form continuity fundamentally harder to achieve without careful chunk overlap and stitching.
+
+---
+
 ## 🚀 Progress & Results
 
-### Current Status: **Project Completed & Models Uploaded**
-We have successfully completed the end-to-end pipeline: finetuning, KenLM integration (5M samples), full benchmarking, and deployment to the Hugging Face Hub.
+### Current Status: **Model Released & Repository Consolidated**
+We have successfully completed the finetuning, KenLM integration, and repository consolidation phases. The model and its associated language models are now hosted on Hugging Face at `RASMUS/Finnish-ASR-Canary-v2`.
+
+- **Infrastructure:** Finetuned on **RTX 6000 PRO Blackwell** (96 GB VRAM) on Verda.com platform in Finland.
+- **Model Suite:** Acoustic model + 3 KenLM variants (1M, 2M, 5M sentences).
+- **Best Performance (with KenLM 5M):**
+    - **FLEURS:** 7.86% WER
+    - **Common Voice:** 4.70% WER
+    - **CSS10:** 7.07% WER
+    - **VoxPopuli:** 11.65% WER
+- **Deployment:** Integrated Silero VAD-based inference for robust long-form audio processing.
+
+### Next Steps:
+1.  **Long-form Tuning:** Reduce default `chunk_len` to 8–10s (closer to training distribution median) and add 0.5–1s overlap between chunks to reduce boundary artifacts.
+2.  **Data Quality Audit:** Fix 28 confirmed corrupted Common Voice entries where raw TSV metadata (client ID hashes, gender tags) was accidentally written into the `text` field. Audit VoxPopuli for missing capitalisation (all-lowercase transcripts despite `pnc: yes`).
+3.  **Number Handling:** Add Finnish-specific training data with numeric content. Consider TTS-synthesised samples covering phone numbers, years, statistics, and measurements (both digit and written-out forms paired).
+4.  **Long-form Training Data:** Incorporate longer audio segments: TTS synthetic long-form audio (`fbc_monolog_processed`, parliament data) into the training manifest to shift the duration distribution toward 15–30s.
+5.  **KenLM Refinement:** Re-train KenLM with high-quality punctuated text. Current LM trained on mixed-quality data.
+6.  **Advanced Evaluation:** Implement CER evaluation on non-normalised test sets to better capture punctuation/casing accuracy.
+7.  **Repetition Penalty:** Explore repetition penalty in decoding if chunk-level loops persist after chunk length tuning.
+8.  **Real-world Evaluation:** Benchmark on diverse long-form samples (podcasts, meetings, call-centre audio).
+
+---
+
+## 🗺️ Action Plan: Next Training Run
+
+This section details the concrete steps for the next finetuning iteration, based on the root-cause analysis above.
+
+### Priority 1 — Fix Training Data (before re-training)
+
+#### 1a. Normalise numbers to digit form (Gemini Flash)
+Finnish written-out numbers in FLEURS transcripts cause the finetuned model to output inconsistent number forms. We will use the Gemini Flash API to convert all training transcripts in a single batch pass:
+
+```python
+# Pseudocode — run once on train_manifest.json before next training
+import google.generativeai as genai
+import json
+
+genai.configure(api_key=GEMINI_API_KEY)
+model = genai.GenerativeModel("gemini-2.0-flash")
+
+SYSTEM_PROMPT = """You are a Finnish text normalizer.
+Convert any written-out Finnish numbers, ordinals, or number words in the text to digit form.
+Examples:
+  "yli sata vuotta" → "yli 100 vuotta"
+  "seitsemäntoista henkeä" → "17 henkeä"
+  "vuonna tuhat yhdeksänsataa" → "vuonna 1900"
+Keep all other text exactly as-is. Return only the modified text, nothing else."""
+
+entries = []
+with open('manifests/train_manifest.json') as f:
+    for line in f:
+        d = json.loads(line)
+        response = model.generate_content(f"{SYSTEM_PROMPT}\n\n{d['text']}")
+        d['text'] = response.text.strip()
+        entries.append(d)
+
+with open('manifests/train_manifest_normalised.json', 'w') as f:
+    for e in entries:
+        f.write(json.dumps(e, ensure_ascii=False) + '\n')
+```
+
+Cost estimate: 23,180 entries × ~50 tokens average = ~1.2M tokens. At Gemini Flash pricing (~$0.075/1M tokens input) ≈ **< $0.10 total**.
+
+#### 1b. Fix en-dash UNK token (confirmed root cause)
+The en-dash `–` (U+2013) is NOT in the tokenizer vocabulary — it maps to UNK (id 0). Replace it with ASCII hyphen before training:
+
+```python
+# Add to the manifest preprocessing step
+text = text.replace('\u2013', '-').replace('\u2014', '-')
+```
+
+This affects **85 entries** in `train_manifest.json` (83 FLEURS, 2 Common Voice).
+
+#### 1c. Fix 28 corrupted Common Voice entries
+Replace entries where the `text` field contains raw TSV metadata (tabs + client_id hashes). Strip everything after the first tab character.
+
+---
+
+### Priority 2 — Add Long-Form Training Data
+
+#### TTS Long-Form Dataset: `RASMUS/canary_asr_finetune_tts_long_data`
+
+| Property | Value |
+|----------|-------|
+| Size | 8.0 GB zip |
+| Format | FLAC audio + JSONL manifest |
+| Mean duration | **16.5s** (vs 7.8s in current data) |
+| Median duration | 15.9s |
+| Max duration | 25.0s |
+| Content | Finnish speech: lectures, podcasts, YouTube |
+| Segments >20s | ~25% |
+
+This dataset directly addresses the training length mismatch. Adding it will shift the duration distribution from a mean of 7.8s toward ~10–12s and significantly increase the proportion of 15–25s segments that match inference chunk lengths.
+
+**Integration plan:**
+```bash
+# Download the dataset
+curl -L -H "Authorization: Bearer ${HF_TOKEN}" \
+  "https://huggingface.co/datasets/RASMUS/canary_asr_finetune_tts_long_data/resolve/main/canary_dataset.zip" \
+  -o /workspace/data/tts_long_data.zip
 
-- **Hardware:** RTX 6000 Ada (96 GB VRAM)
-- **Acoustic Model:** Finetuned Canary-v2 (1B params)
-- **Language Model:** 6-gram KenLM (5M samples, token-aligned)
-- **Final Performance:** **7.51% Average WER** across four datasets (40.8% error reduction from baseline).
-- **Deployment:** [RASMUS/Finnish-ASR-Canary-v2](https://huggingface.co/RASMUS/Finnish-ASR-Canary-v2)
+# Extract
+unzip /workspace/data/tts_long_data.zip -d /workspace/data/tts_long_data/
+
+# Apply number normalisation and dash fix to canary_manifest.jsonl
+# then merge with existing train_manifest_normalised.json
+```
+
+After applying number normalisation and dash fixes to the new manifest, concatenate with the existing training set. Expected combined size: ~23,180 + N (estimate 5,000–20,000+ entries depending on total dataset size).
 
 ---
 
-## 🛠️ Implementation Details
-
-### 1. Framework Setup (NVIDIA NeMo)
-We use a specific commit of NVIDIA NeMo to ensure compatibility with Canary-v2.
-- **Repository:** `https://github.com/NVIDIA/NeMo.git`
-- **Commit:** `557177a18d`
-- **Installation:** Cloned into `/workspace/NeMo` and installed in editable mode (`pip install -e .[asr]`).
-- **Patches Applied:**
-  - Fixed `OneLogger` import issues in `nemo/lightning/callback_group.py`.
-  - Fixed Canary-v2 EOS assertion in `nemo/collections/common/prompts/canary2.py` for inference compatibility.
-
-### 2. Model Initialization & Conversion
-- **Base Model:** Downloaded `nvidia/canary-1b-v2` from HuggingFace using `scripts/init_canary_model.py`.
-- **Finetuned Model:** Downloaded the PyTorch Lightning checkpoint (`canary-finnish--epoch=11--val_wer=0.1258.ckpt`) from [RASMUS/Canary_finetune_trial](https://huggingface.co/RASMUS/Canary_finetune_trial/tree/main) and converted it to a portable `.nemo` file at `/workspace/models/canary-finetuned-finnish.nemo`.
-
-### 3. Data Ingestion
-- **ASR Data:** Downloaded from HuggingFace (`RASMUS/canary-finnish-asr-data`) using `huggingface-cli download`.
-- **Structure:**
-  - Manifests: `/workspace/Canary_finetune_trial/manifests/`
-  - Audio: `/workspace/data/audio/` (Extracted from tars).
-
-### 4. KenLM Language Model Integration
-To solve Finnish-specific challenges like long compound words, we integrated a token-aligned 6-gram KenLM model.
-- **Training Data:** 5 million lines of high-quality Finnish text filtered from Reddit, FinePDF, Wiki-Edu, and ASR transcripts.
-- **Data Integrity:** Removed 1,833 leaked sentences from the training corpus to ensure fair evaluation.
-- **Performance Optimization:** Converted the ARPA text models into binary `.nemo` LM bundles, reducing load time from 5+ minutes to ~7 seconds.
-
-### 5. Benchmarking the Evolution
-Our journey saw massive gains at every step. Here are the exact evaluation results (metrics are clean WER %) across our development cycle:
-
-| Dataset | Original Canary-v2 | Finetuned (Greedy) | **Finetuned + KenLM (5M, α=0.2)** | Improvement |
-| :--- | :---: | :---: | :---: | :---: |
-| **Common Voice** | 17.95% | 12.82% | **5.98%** | **-66.7%** |
-| **FLEURS** | 7.79% | 8.33% | **6.48%** | **-16.8%** |
-| **CSS10 (Audiobook)**| 17.07% | 12.19% | **11.85%** | **-30.6%** |
-| **VoxPopuli (Formal)**| 7.96% | 4.46% | **5.73%** | **-28.0%** |
-| **GLOBAL AVG** | 12.69% | 9.45% | **7.51%** | **-40.8%** |
+### Priority 3 — Inference Tuning (without re-training)
+
+Even before re-training, we can improve `moo.wav` performance by adjusting `inference_vad.py`:
+
+| Parameter | Current | Recommended |
+|-----------|---------|-------------|
+| `chunk_len` | 15s | 8–10s (match training median of 7.8s) |
+| chunk overlap | 0s | 0.5s (reduce boundary word drops) |
+| `alpha` (KenLM) | 0.2 | Try 0.1–0.15 (current may over-constrain decoder) |
+
+---
+
+## 🔄 Round 2: Data Pipeline & Splits
+
+This section documents the data preparation methodology for Round 2 finetuning, including all new eval sets, the TTS integration, and the final manifest composition.
+
+### Overview of Changes vs Round 1
+
+| Item | Round 1 | Round 2 |
+|------|---------|---------|
+| Base model | `canary-1b-v2.nemo` | `canary-1b-v2.nemo` (fresh start) |
+| Training samples | 23,180 | **28,858** |
+| Training hours | ~50h | **75.6h** |
+| Mean duration | 7.8s | **9.4s** |
+| Max duration allowed | 20.0s | **30.0s** |
+| Transcripts normalised | No | **Yes (digits, dashes fixed)** |
+| Eval sets | 4 | **6** |
+
+### Step 1 — Transcript Normalisation (`normalize_manifests.py`)
+
+All training transcripts were cleaned in two layers:
+
+**Deterministic fixes (no API call needed):**
+- En-dash `–` (U+2013) and em-dash `—` (U+2014) → ASCII hyphen `-` (fixes UNK token regression)
+- Corrupted Common Voice entries (raw TSV metadata in `text` field) → strip everything after first tab
+
+**Gemini 2.5 Flash API calls (2,586 of 23,180 entries needed conversion):**
+- Pre-filtered with a Finnish number-word regex so only entries that actually contain written numbers are sent to the API (cost: ~$0.62)
+- Written Finnish numbers converted to digit form: `sata vuotta` → `100 vuotta`, `seitsemäntoista` → `17`
+- Explicit DO NOT CONVERT rules: ordinals (`ensimmäinen`, `toinen`), superlative constructions (`yksi tärkeimmistä`), and `toinen` as "another/other"
+
+### Step 2 — TTS Long-Form Data Integration
+
+Downloaded `RASMUS/canary_asr_finetune_tts_long_data` (4.8 GB, 6,365 entries, mean 16.4s).
+
+Aligned to NeMo training format:
+- Path rewritten to relative style: `data/tts_long_data/audio/{filename}`
+- Fields mapped: `language` → `source_lang`/`target_lang`, `task: "transcription"` → `taskname: "asr"`, added `pnc: "yes"`
+- Same Gemini normalisation pass applied (888 entries converted)
+
+### Step 3 — Eval Set Construction (TTS Data)
+
+The 6,365 normalised TTS entries were split into train / eval / long-form-test:
+
+```
+All TTS entries (6,365)
+│
+├── Long-form pool (>20s): 1,501 entries
+│   ├── eval_long_form (sampled): 200 entries  ← random.seed(42) shuffle → first 200
+│   └── Returned to training pool: 1,301 entries
+│
+└── Medium pool (10–20s): 4,864 entries
+    ├── eval_tts (10% hold-out): 487 entries  ← stratified by duration bucket
+    └── tts_train: 4,377 entries
+```
+
+**Why eval_long_form = 200 entries?**
+The original 1,501 long-form entries (>20s) had a total duration of ~9.4 hours — far too long to run as a validation set every epoch. At batch_size=32 on a single GPU, each validation pass over 1,501 entries takes ~25 minutes, adding 2.5h per epoch. 200 entries (≈75 minutes of audio) provides a representative sample of the long-form distribution at reasonable cost: ~4 minutes of eval time per epoch.
+
+**eval_tts construction:**
+487 entries were held out from the 10–20s duration range (10% stratified sample). This tests the model's ability to handle medium-length audio and is separate from the original 4 eval sets.
+
+### Step 4 — Combined Training Manifest
+
+Final `train_manifest_combined.jsonl` composition:
+
+| Source | Entries | Notes |
+|--------|---------|-------|
+| Original train (normalised) | 23,180 | Digits + dash fix applied |
+| TTS train (10–20s) | 4,377 | Synthesised long-form speech |
+| Long-form overflow | 1,301 | >20s entries not selected for eval_long_form |
+| **Total** | **28,858** | Mean 9.4s, 75.6h |
+
+### Final Eval Sets (Round 2)
+
+| Set | File | Entries | Mean Duration | Purpose |
+|-----|------|---------|--------------|---------|
+| `eval_fleurs` | `eval_fleurs.json` | 918 | 13.0s | Primary benchmark (monitored for checkpointing) |
+| `eval_common_voice` | `eval_common_voice.json` | 1,554 | 5.1s | Crowdsourced quality |
+| `eval_css10` | `eval_css10.json` | 170 | 7.5s | Clean single-speaker |
+| `eval_voxpopuli` | `eval_voxpopuli.json` | 430 | 10.6s | Formal/parliament speech |
+| `eval_tts` | `eval_tts.jsonl` | 487 | 14.5s | Medium-length TTS (new) |
+| `eval_long_form` | `eval_long_form.jsonl` | **200** | 22.5s | Long-form >20s sample (new) |
+
+**Checkpoint monitoring:** `val_wer` tracks FLEURS (first validation set). All 6 WERs are logged independently to WandB.
+
+### Round 2 Training Config
+
+File: `configs/canary_finetune_finnish_v2.yaml`
+Key settings:
+- `init_from_nemo_model`: `/workspace/Finnish-ASR-Canary-v2/models/canary-1b-v2.nemo` (fresh start from base)
+- `max_duration`: 30.0s (up from 20.0s to include TTS segments up to 25s)
+- `max_steps`: 18,000 (scaled: 28,858 / 32 ≈ 902 steps/epoch × 20 epochs ≈ 18,040)
+- `lr`: 1e-5, `WarmupAnnealing`, 500 warmup steps
+- `precision`: bf16, single GPU, `strategy: auto`
+
+---
+
+## 🛠️ Workflow Status Details
+
+### 1. Data Preparation - DONE
+- [x] Identify and inventory all 4 datasets
+- [x] Create unified processing script (`scripts/prepare_all_manifests.py`)
+- [x] Run `scripts/prepare_all_manifests.py` on devcontainer
+- [x] Verify manifest sample counts and audio file integrity
+
+### 2. Configuration Setup - DONE
+- [x] Create Hydra training config (`configs/canary_finetune_finnish.yaml`)
+- [x] Configure multi-validation with 4 eval datasets
+- [x] Checkpoint monitors primary eval set (FLEURS) via `val_wer`
+- [x] All 4 eval WERs logged independently to WandB
+
+### 3. Training - DONE
+- [x] Run finetuning via `run_training.sh`
+- [x] Monitor per-dataset WER in WandB
+
+### 4. KenLM / NGPU-LM Language Model Integration - DONE
+- [x] Install KenLM tools (`install_beamsearch_decoders.sh`)
+- [x] Gather Finnish text (ASR transcripts + Wikipedia + mc4)
+- [x] Train 3 variants of KenLM (1M, 2M, 5M sentences)
+- [x] Evaluate with LM fusion on all 4 test sets
+
+### 5. Repository & Long-Form Inference - IN PROGRESS
+- [x] Consolidate README and model metadata for Hugging Face release
+- [x] Upload model checkpoints and KenLM bundles to HF Hub
+- [x] Implement Silero VAD-based chunking for long-form audio (`inference_vad.py`)
+- [x] Root-cause analysis of long-form degradation vs. Whisper (see above)
+- [ ] Reduce `chunk_len` to 8–10s and add chunk overlap (Current Focus)
+- [ ] Optimize `alpha` for stability on `moo.wav` (30 min test file)
+
+### 6. Data Quality & Advanced Evaluation - PARTIALLY DONE
+- [x] Fix 28 corrupted Common Voice manifest entries (raw TSV data in text field) — done in normalisation pass.
+- [x] Fix en-dash/em-dash UNK token regression — done in normalisation pass.
+- [ ] Audit VoxPopuli transcripts for all-lowercase entries (capitalisation missing).
+- [ ] Re-train KenLM with high-quality punctuated text.
+- [ ] Evaluate CER on non-normalized test sets.
+
+### 7. Number Normalisation & UNK Token Fix - DONE
+- [x] Replace en-dash `–` and em-dash `—` with ASCII hyphen `-` in all training manifests (85 train + 70 TTS entries fixed).
+- [x] Use Gemini 2.5 Flash to normalise written-out Finnish numbers to digit form (2,586 API calls across train + TTS).
+- [ ] Re-evaluate on the 5-sample number test set after Round 2 training to verify consistency.
+
+### 8. Long-Form Data Expansion - DONE
+- [x] Download `RASMUS/canary_asr_finetune_tts_long_data` (4.8 GB zip, 6,365 entries, mean 16.4s).
+- [x] Align TTS manifest to NeMo training format and integrate into combined training manifest.
+- [x] Round 2 training configured and ready to launch (see Round 2 section below).
+- [ ] Benchmark Round 2 model against Round 1 and finetuned Whisper on `moo.wav`.
+
+---
+
+## 🛠️ NeMo Environment Setup
+
+This section documents the exact steps to set up a working NeMo inference/training environment, including the fixes required for the `nvcr.io/nvidia/pytorch:25.01-py3` container.
+
+### Installation (from scratch on pytorch:25.01-py3 base image)
+
+```bash
+# 1. Clone the HF model repo (contains NeMo source with patches applied)
+#    Skip LFS to avoid downloading the 3.6 GB model during clone
+GIT_LFS_SKIP_SMUDGE=1 git clone \
+  "https://user:${HF_TOKEN}@huggingface.co/RASMUS/Finnish-ASR-Canary-v2" \
+  /workspace/Finnish-ASR-Canary-v2
+
+# 2. Install NeMo in editable mode from the patched source
+cd /workspace/Finnish-ASR-Canary-v2/NeMo
+pip install -e ".[asr]"
+
+# 3. Install pinned dependencies
+pip install 'fsspec==2024.12.0' 'numpy<2.0' 'librosa>=0.11.0' kaldialign wandb
+```
+
+### Required Compatibility Fixes
+
+The pytorch:25.01-py3 container ships with packages that conflict with NeMo 2.8.0rc0:
+
+```bash
+# Fix 1: Downgrade lightning to the version NeMo requires (<=2.4.0)
+# The container ships lightning 2.4.0 but pip may upgrade it — pin it back.
+pip install "lightning==2.4.0" "pytorch-lightning==2.4.0"
+
+# Fix 2: Remove incompatible torchvision
+# The container's torchvision (0.20.0a0) was built against torch 2.6.0a0 (the original
+# container torch), but NeMo's install upgrades torch to ~2.10. torchvision then fails
+# on import and blocks NeMo. ASR does not need torchvision.
+pip uninstall -y torchvision
+```
+
+### Downloading the Finetuned Model
+
+```bash
+# Download the finetuned acoustic model (3.6 GB)
+curl -L \
+  -H "Authorization: Bearer ${HF_TOKEN}" \
+  "https://huggingface.co/RASMUS/Finnish-ASR-Canary-v2/resolve/main/canary-finnish.nemo" \
+  -o /workspace/Finnish-ASR-Canary-v2/canary-finnish.nemo
+
+# KenLM models are also LFS — download the 5M variant (best WER):
+curl -L \
+  -H "Authorization: Bearer ${HF_TOKEN}" \
+  "https://huggingface.co/RASMUS/Finnish-ASR-Canary-v2/resolve/main/kenlm_5M.nemo" \
+  -o /workspace/Finnish-ASR-Canary-v2/kenlm_5M.nemo
+```
+
+### Quick Inference Smoke Test
+
+```python
+import warnings; warnings.filterwarnings('ignore')
+from nemo.collections.asr.models import EncDecMultiTaskModel
+
+model = EncDecMultiTaskModel.restore_from(
+    '/workspace/Finnish-ASR-Canary-v2/canary-finnish.nemo',
+    map_location='cuda'
+)
+model.eval()
+
+results = model.transcribe(
+    audio=['path/to/audio.wav'],
+    task='asr', source_lang='fi', target_lang='fi', pnc='yes'
+)
+print(results[0].text)
+```
+
+### Loading the Base Model (for comparison)
+
+```python
+# Downloads ~3.6 GB on first run, cached in ~/.cache/huggingface/
+model_base = EncDecMultiTaskModel.from_pretrained("nvidia/canary-1b-v2", map_location='cuda')
+```
 
 ---
 
 ## 📝 Progress Log
-- **2026-01-11:** Initial project setup with Chatterbox-TTS-10k + MCV test set.
+- **2026-01-11:** Initial project setup.
 - **2026-02-08:** Redesigned data pipeline for 4 real datasets (CSS10, FLEURS, VoxPopuli, Common Voice).
-- **2026-02-14:** 
-  - Environment: Cloned NeMo (`557177a18d`), applied patches.
-  - Models: Converted epoch 11 checkpoint to `.nemo` format.
-  - Data: Downloaded full ASR dataset from Hub.
-- **2026-02-15:**
-  - **KenLM Expansion:** Scaled from 500k to 5M training samples with high-quality filtering (FineWeb-Edu).
-  - **Optimization:** Converted ARPA to NeMo LM bundles for fast GPU loading.
-  - **Evaluation:** Completed full benchmarks on all four test sets.
-  - **Deployment:** Uploaded all final models and documentation to [RASMUS/Finnish-ASR-Canary-v2](https://huggingface.co/RASMUS/Finnish-ASR-Canary-v2).
+- **2026-02-10:** **Finetuning complete.** Epoch 11 reached `val_wer=0.1258` on FLEURS.
+- **2026-02-13:** Mermaid diagrams and project documentation for DS team.
+- **2026-02-18:** **KenLM benchmarks finished.** Consolidated repository structure. Applied NeMo patches for inference stability.
+- **2026-02-20:** **Model Released.** Release of `Finnish-ASR-Canary-v2` on HF. Implemented VAD-based inference pipeline. Currently tuning for long-form stability on `moo.wav` with various `alpha` settings (0.0 - 0.4 tested).
+- **2026-02-26:** **Root-cause analysis complete.** Investigated long-form gap vs. Whisper and number handling. Key findings: (1) 77% of training data is <10s, creating distribution shift at inference chunk lengths; (2) No cross-chunk context in Canary's AED architecture; (3) Only 2.5% of training samples contain digit characters — numbers are a known weak point; (4) 28 corrupted Common Voice entries found (TSV metadata in text field); (5) `moo.wav` test file confirmed as 30 minutes. Action plan: shorten chunk_len, add chunk overlap, fix data corruption, and plan a long-form training data expansion round.
+- **2026-02-26:** **Live number inference + tokenizer audit completed.** Ran base Canary-v2 vs. finetuned model on 5 FLEURS samples. Confirmed: (1) base model always outputs digits (`100`, `17`); (2) finetuned model regressed to mixed output — sometimes written words, sometimes digits — due to inconsistent training transcripts; (3) en-dash (`–`) produces UNK token `⁇` in finetuned model, base model degrades gracefully to ASCII hyphen. Policy decision: **standardise on digit output** and fix en-dash encoding in training manifests before next training run. NeMo environment setup documented (with fixes for `torchvision` and `lightning` version conflicts). TTS long-form dataset (`canary_asr_finetune_tts_long_data`, 8GB, mean 16.5s/segment) identified as key data source for next training run. Action plan for next run: (1) normalise numbers to digits via Gemini Flash API, (2) fix en-dash → ASCII hyphen, (3) fix 28 corrupted CV entries, (4) add TTS long-form data.
+- **2026-03-01:** **Round 2 data pipeline complete.** Ran `normalize_manifests.py`: 2,586 Gemini 2.5 Flash API calls (~$0.62), 1,137 number changes in train + 888 in TTS, 85 en-dash and 28 corrupted CV entries fixed. Downloaded and extracted TTS long-form dataset (6,365 entries, 4.8 GB). Split TTS data into train (4,377), eval_tts (487, mean 14.5s), and long-form pool (1,501 entries >20s). Sampled 200 entries into `eval_long_form.jsonl` (seed 42) and returned 1,301 to training, yielding `train_manifest_combined.jsonl` (28,858 entries, 75.6h). Round 2 training config created (`configs/canary_finetune_finnish_v2.yaml`). **Training ready to launch.**
+- **2026-03-01:** **Training crash diagnosed and fixed.** Round 2 training ran 505 steps then crashed with CUDA `vectorized_gather_kernel index out of bounds`. Root cause: entry 14857 in `train_manifest_combined.jsonl` contained 11,247 chars of Python code (Gemini normalization returned a code block instead of a transcript for `voxpopuli_005371.wav`). When tokenized with the canary2 prompt format, the sequence far exceeded the decoder's `max_sequence_length=1024`, causing position-embedding OOB. Additionally, 4 entries in `eval_common_voice.json` had TSV metadata contamination (same v1 issue, not previously caught in the v2 eval set). Both manifests fixed. Config rewritten from full-architecture spec to minimal v1-style format (`tokenizer: update_tokenizer: false`) using `speech_to_text_finetune.py` (which restores the full model from the `.nemo` file). Training re-launched. Manifests synced to `canary-finnish-asr-data` HuggingFace dataset repo.