Gourab RoyHome

RESEARCH PROJECT · 2026

A lighter CLAD formulation for vision language procedure planning

I reproduced CLAD and used it as the starting point for a lighter formulation. Compact CLAD performs diffusion on the action plan while supplying the observations, task identity, and endpoint latents as context.

Read the code View model and artifacts Original CLAD paper
Datasets
NIV and CrossTask
Horizon
3 actions
Evaluation
5 inference seeds
Planner size
About 1.08M parameters

THE QUESTION

Can the diffusion state be made smaller?

CLAD combines task, action, and observation features into one noisy tensor. The visual feature alone has 1,536 values, and it is repeated at every point in a short plan.

I explored whether the model could retain endpoint conditioning while using a smaller noisy state. My version keeps the two stage structure and asks the denoiser to generate only the action grid.

Original CLADTask + action + observation

All three components are part of the noisy state at every step.

Compact CLADAction grid

Task, observations, and endpoint latents move into the condition encoder.

NIV97.0%less state width
CrossTask93.7%less state width
COIN68.8%less state width

METHOD

The pipeline I trained

  1. 01

    Learn the endpoints

    A normalized VAE learns representations of the start and goal action observation pairs. The decoder uses signed output MSE because these features are real valued.

  2. 02

    Build the context

    The condition encoder combines the start observation, goal observation, task class, and two VAE latents.

  3. 03

    Denoise the actions

    A small temporal U Net predicts the clean action grid. Training uses a cosine noise schedule and gives extra weight to the two endpoints. Inference uses ten DDIM steps.

RESULTS

Completed horizon 3 runs

Mean over seeds 0 through 4

DatasetAcc@1Acc@5SuccessmIoU1mIoU2
NIV53.7081.1633.6379.4562.66
CrossTask63.1184.9730.4451.0560.60
NIVEpoch 125

1,079,536 planner parameters
Best VAE validation ELBO: 597.9396

CrossTaskEpoch 80

1,087,721 planner parameters
Best VAE validation ELBO: 617.4352

READING THE RESULTS

What these numbers do and do not show

NIV has a validation caveat

The event class enriched official NIV test manifest was also used for validation and model selection. I treat the NIV figures as a record of the completed engineering experiment, not as an untouched test estimate.

The CrossTask values come from the saved five seed test run. I am not claiming a new state of the art result. The useful finding is narrower: the action focused formulation trains, produces usable plans, and reduces the diffusion state substantially.

I saved the checkpoints, configurations, logs, TensorBoard events, inference outputs, and SHA 256 manifest so the experiment can be inspected rather than taken on trust.

REPRODUCE

Code and evidence

ImplementationTraining, inference, tests, and documentationGitHub Model and experiment archiveCheckpoints, logs, outputs, and checksumsHugging Face CLAD paperThe method that this project reproduces and extendsarXiv
Run the test suitepython -m unittest discover -s tests -v