VLA Data Efficiency Study on LIBERO

Overview

Vision-Language-Action (VLA) models like SmolVLA are pretrained on large internet-scale corpora and can zero-shot execute manipulation tasks via language instructions. In practice, teams fine-tune on task-specific demonstrations. A key open question is: how many demonstrations does a VLA actually need?

Collecting robot demos is expensive — if a VLA can achieve near-peak performance with 25% of the data, practitioners can cut collection costs by 4×. This study systematically evaluates SmolVLA fine-tuning across 5 data regimes on LIBERO-Spatial, comparing against a zero-shot baseline evaluated on 500 rollouts (50 per task × 10 tasks).

Q1What is the zero-shot baseline performance with no fine-tuning?

Q2How much does fine-tuning help on LIBERO-Spatial?

Q3What is the minimum demo count for >70% average success?

Q4Is improvement uniform across tasks, or highly task-dependent?

Q5Does the model overfit at very low data regimes (≤ 5%)?

Scaling Curve

Average task success rate across all 10 LIBERO-Spatial tasks vs. number of demonstration episodes used for fine-tuning.

Zero-shot Baseline

73.0%

SmolVLA with no fine-tuning
500 rollouts · 10 tasks

→ fine-tuning →

Worst (5% data, 21 eps)

52.4%

↓ 20.6 pp below zero-shot
Severe overfitting confirmed

…

Best (100% data, 432 eps)

70.2%

↓ 2.8 pp still below zero-shot
Fine-tuning never recovers

Key Findings

Four main insights from the data efficiency study.

⚠️

Finding 1

Fine-tuning Hurts at Low Data

Fine-tuning with only 5% of demos (~2 per task) drops average success from 73.0% → 52.4% — a 20.6 percentage-point regression. Noisy gradients from 2 demonstrations cause severe task forgetting.

−20.6 pp at 5% data

📈

Finding 2

Zero-shot Remains Unbeaten

Even with 100% of data (432 demos, ~43/task), fine-tuning achieves only 70.2% — still 2.8 pp below the zero-shot baseline. SmolVLA's internet-scale pretraining gives it a strong prior on LIBERO-Spatial.

73.0% zero-shot > 70.2% full fine-tune

📊

Finding 3

Recovery Curve After 10% Data

From 10% data onward, performance recovers steadily: 57.9% → 67.1% → 69.0% → 70.2%. Approximately 25% data (~11 demos/task) recovers to within 6 pp of zero-shot, suggesting a potential data-efficiency knee.

25% data → 67.1% (−5.9 pp gap)

🔥

Finding 4

Task 5 Persistently Fails

Task 5 ("on ramekin") is the hardest across all conditions: 36% zero-shot → crashes to 2.6% at 10% data, recovering to only 26.8% at full data. Spatial disambiguation may require fundamentally more data or architectural changes.

Task 5: 2.6%–29% across all fine-tuned runs

Per-Task Breakdown

Success rate (%) for each of the 10 LIBERO-Spatial tasks across all 6 conditions. Color indicates performance level.

<25% 25–50% 50–65% 65–80% ≥80%

Condition	T0 btw plate & ramekin	T1 next to ramekin	T2 table center	T3 on cookie box	T4 top drawer	T5 on ramekin ⚠️	T6 next to cookie box	T7 on stove	T8 next to plate	T9 wooden cabinet	Avg
Zero-shot 0 demos	62.0	94.0	78.0	76.0	72.0	36.0	84.0	80.0	78.0	70.0	73.0
5% data ~2 demos/task	86.4	64.2	80.6	81.4	31.0	7.2	26.8	51.8	82.8	12.0	52.4
10% data ~4 demos/task	49.2	45.6	76.6	77.0	59.2	2.6	87.6	65.6	75.0	40.8	57.9
25% data ~11 demos/task	86.2	68.2	85.0	76.2	65.0	29.0	83.4	61.8	52.0	64.4	67.1
50% data ~21 demos/task	78.2	81.2	88.8	84.6	65.8	17.0	82.8	69.6	62.4	59.2	69.0
100% data ~43 demos/task	79.8	80.4	84.0	82.8	52.2	26.8	82.0	73.0	73.2	68.0	70.2

Notable patterns: Task 1 (next to ramekin) starts strong at zero-shot (94%) and remains near the top across all conditions. Task 5 (on ramekin) is the outlier — it never exceeds 36% across any condition, suggesting precise spatial disambiguation may require more demonstrations or richer spatial representations than are available in this setup. High per-task variance at low data (5%–10%) confirms noisy gradient updates when fewer than 5 demos per task are available.

Experimental Setup

Model architecture, dataset, and evaluation protocol.

🤖

SmolVLA (500M)

lerobot/smolvla_base

Backbone VLM backbone + Action Expert

Fine-tune Expert + state projector (vision frozen)

Optimizer AdamW, lr=1e-4, cosine → 2.5e-6

Steps 30,000 (batch_size = 16)

Action chunk_size = 50 steps predicted at once

Seed 42, same hyperparameters across all conditions

🗄️

LIBERO-Spatial

lerobot/libero_spatial_image

Tasks 10 spatial manipulation tasks

Episodes 432 total (~43 per task)

Observations agentview RGB + wrist RGB + 8D proprio

Actions 7D (3D EEF pos Δ + 3D rot Δ + 1D gripper)

Eval 50 rollouts × 10 tasks = 500 per condition

Metric success_rate (%) per task, average across tasks

LIBERO-Spatial
432 episodes

→

Stratified
subset 5–100%

→

Fine-tune
SmolVLA 30k steps

→

Checkpoint
30,000 final

→

Evaluate
500 rollouts

→

Results
success_rate

LIBERO-Spatial Tasks

All 10 tasks share the same goal structure: pick up the black bowl from a spatially-described location and place it on the plate. Difficulty varies by spatial context.

T0

Between plate & ramekin

Zero-shot: 62%

T1

Next to ramekin

Zero-shot: 94% ★

T2

Table center

Zero-shot: 78%

T3

On cookie box

Zero-shot: 76%

T4

Top drawer

Zero-shot: 72%

T5

On ramekin ⚠️

Zero-shot: 36% — hardest

T6

Next to cookie box

Zero-shot: 84%

T7

On stove

Zero-shot: 80%

T8

Next to plate

Zero-shot: 78%

T9

On wooden cabinet

Zero-shot: 70%

Data Regime Details

Subsampling is stratified per task — each fraction takes equal proportions from every task to avoid class imbalance. All conditions use identical hyperparameters.

Zero-shot

No Fine-tuning 0 episodes · 0 demos/task

SmolVLA base model evaluated directly on LIBERO-Spatial. Establishes the pretraining prior baseline. The model leverages internet-scale VLM knowledge to understand spatial language instructions.

avg_success = 73.0% · range [36% – 94%] across tasks

5% Data

Minimal Data Regime ~21 episodes · ~2 demos/task

With only 2 demonstrations per task, gradient updates are highly noisy. The model experiences catastrophic forgetting on several tasks while overfitting on others. Worst-case degradation to 7.2% on Task 5.

avg_success = 52.4% · Δ = −20.6 pp vs zero-shot

10% Data

Low Data Regime ~43 episodes · ~4 demos/task

Recovery begins but remains below zero-shot. Task 5 crashes to 2.6% (worst across all conditions). Task 6 surprisingly reaches 87.6%, demonstrating high per-task variance at this regime.

avg_success = 57.9% · Δ = −15.1 pp vs zero-shot

25%–100% Data

Moderate to Full Data 108–432 episodes · 11–43 demos/task

From 25% data onward, average performance approaches but never reaches zero-shot. The curve flattens rapidly: 25% → 67.1%, 50% → 69.0%, 100% → 70.2%. Diminishing returns are visible — doubling from 50% to 100% yields only +1.2 pp.

frac25→frac50→frac100: 67.1% → 69.0% → 70.2%

Training Configuration

Parameter	Value	Notes
`model`	lerobot/smolvla_base	500M params, VLM + action expert
`steps`	30,000	Same across all data fractions
`batch_size`	16	—
`learning_rate`	1e-4	AdamW, cosine decay
`lr_min`	2.5e-6	Cosine decay floor
`chunk_size`	50	Action steps predicted per forward pass
`frozen`	vision encoder	Expert + state projector trained
`seed`	42	Reproducibility
`eval_rollouts`	500	50 eps × 10 tasks per condition
`save_freq`	30,000	Final checkpoint only
`subsampling`	stratified	Equal proportion per task

Tech Stack

LeRobot

Training + evaluation framework

SmolVLA

500M VLA base model

LIBERO

Benchmark suite (Spatial split)

MuJoCo

Physics simulation

PyTorch

Deep learning framework

HuggingFace

Model hub, dataset loading

Hydra

Config management

matplotlib

Results visualization