pico-JEPA v2

Abstract

pico-JEPA learns to represent videos without labels, on a machine you can keep under a desk. In this second version we test a simple idea: instead of training one model on all the data, train several small models — each on a different slice — and combine their predictions. We evaluate on 30 Kinetics-700 classes with a compact ~3.4M-parameter network trained on a single desktop GPU.

The main result: when the ensemble's members are diverse and their predictions are combined by a learned meta-model (stacking), the ensemble beats the single model by +5.10 percentage points, with formal statistical significance. But the advantage is not universal — two controlled experiments show it vanishes if the backbone is too small or if there are too few classes. Rather than a yes-or-no validation, our central contribution is a map of the conditions under which the collective wisdom of small models beats the single large one.

Key results

How the predictions are combined (4 heterogeneous submodels; single model: 6.80%). The verdict is supported when the bootstrap 95% confidence interval excludes zero.

When it works — and when it doesn't

Two controlled experiments mark the boundaries. Shrinking the backbone or narrowing the class space pushes the submodels toward the same solution, so the diversity that stacking exploits never appears — and the advantage disappears with it.

Controlled experiments (stacking). The ensemble's advantage needs a backbone with enough capacity and a wide class space.

Takeaway

For anyone working under tight compute: splitting your data across several small models can win, but not for free. You have to design the diversity (different heads), learn the combination (stacking rather than voting), and check that the problem has enough room — capacity and classes — for that diversity to exist at all.

Limitations

Four caveats bound these conclusions. The holdout is small (294 videos), so individual intervals are wide. Absolute accuracies are low (~12% over 30 classes, about 3.6× chance), reflecting an encoder pre-trained on only 3,000 videos — some 670× less data than the original V-JEPA. Ensemble diversity is confined to the classification head: all submodels share one encoder, because training several encoders exceeded the available hardware. And the results come from a single encoder training run, so initialisation variability is not captured by the reported intervals.

BibTeX

@inproceedings{rostagno2026picov2,
  title     = {pico-JEPA v2: ¿Cuándo Superan Muchos Modelos Pequeños a Uno Grande?},
  author    = {Rostagno, Adrián and Iparraguirre, Javier and Briatore, Roberto
               and González, Agustín and Aggio, Santiago},
  booktitle = {XXXII Congreso Argentino de Ciencias de la Computación (CACIC)},
  year      = {2026},
  address   = {Concepción del Uruguay, Entre Ríos, Argentina},
  month     = {October}
}