Gourab RoyHome

RESEARCH PROJECT · 2026

Adapting a compact vision-language model to remote sensing

TerraQ-VL applies staged visual-language alignment to aerial and satellite imagery. A connector first aligns frozen CLIP features with Qwen2.5-3B, then LoRA instruction tuning adapts the language model for captions and visual questions.

Read the codeModel weightsModel DOI
Training data
139,575 records
Training images
19,861 images
Stage 2 trainable
36.2M parameters
Held-out test
1,367 records

THE QUESTION

Can the same efficient recipe work for remote-sensing images?

Aerial and satellite images pose a different grounding problem from ordinary photographs. The model must connect overhead visual patterns with descriptions of land use, infrastructure, spatial relationships, and object counts.

TerraQ-VL tests whether a frozen vision encoder and mostly frozen language model can learn that connection through a small trainable connector followed by parameter-efficient instruction tuning.

METHOD

Alignment first, instruction tuning second

  1. 01

    Encode overhead imagery

    A frozen CLIP ViT L/14 converts each image into visual patch features. Keeping the vision tower fixed reduces the number of parameters that need training.

  2. 02

    Learn the visual-language bridge

    A trainable MLP connector maps the CLIP features into the embedding space of Qwen2.5-3B-Instruct. Stage 1 trains this connector while both base models remain frozen.

  3. 03

    Tune for captions and questions

    Stage 2 starts from the aligned connector and adds LoRA adapters to the language model. The connector and adapters are trained together while the vision encoder and base language model weights stay frozen.

DATA

A split that keeps images separate

Training uses VRSBench, a collection of aerial and satellite imagery with detailed captions and visual question-answer turns. The split is made by image, so every caption and question associated with an image stays on the same side of the train, validation, or test boundary.

SplitImagesRecords
Training19,861139,575
Validation2041,448
Held-out test1971,367

TRAINING RESULTS

Released checkpoints for both stages

Held-out validation loss, Stage 1

Stage 13,270 steps

Across three epochs.
Held-out validation loss: 1.793 → 1.209

Stage 22,180 steps

Instruction-tuning steps.
Connector and LoRA adapters trained together.

The release includes all Stage 1 and Stage 2 checkpoints, configurations, held-out loss curves, and predictions. Stage 2 predictions cover every record in the held-out test split.

READING THE PROJECT

Useful for research, with clear constraints

The released weights are non-commercial

The Qwen2.5-3B base model uses the Qwen Research License, and VRSBench is released under CC BY-NC 4.0. The resulting TerraQ-VL checkpoints are therefore intended for research and non-commercial use.

CLIP processes images at 224 by 224 pixels, which can discard small objects and fine spatial detail in large remote-sensing scenes. The released results document training and held-out predictions, but they should not be read as a broad comparison against every remote-sensing vision-language model.

RESOURCES

Code, checkpoints, and data

ImplementationTraining, inference, evaluation, and data-building scriptsGitHub TerraQ-VL checkpointsStage 1 and Stage 2 weights, curves, configurations, and predictionsHugging Face VRSBenchRemote-sensing captions and visual question-answer dataDataset Model DOIPersistent citation record for the released weightsDOI

The checkpoints require the repository code and base models. See the model card for exact loading instructions and license terms.