One Model for All Vision Tasks? SenseNova‑Vision Answers with Unified Multimodal Generation

Computer vision has long been a patchwork of isolated islands. Object detection spits out bounding boxes, segmentation produces masks, depth estimation gives grayscale maps, and 3D reconstruction deals with point clouds and camera parameters. Each task has its own output format, its own network heads, its own evaluation metrics. Trying to do multiple things with a single model usually means wrestling with incompatible output spaces.

SenseNova‑Vision tackles a straightforward question: can we squeeze the entire spectrum of computer vision tasks into a single generative framework? Much like GPT unified NLP through text generation, can vision tasks be covered by a single “generation” interface?

Their answer: yes. And you don’t need task‑specific prediction heads.

What Problem Does Unified Multimodal Generation Actually Solve?

Imagine you are building a vision system that must detect objects, segment their outlines, and estimate depth from a single image. The traditional path forces you to maintain three separate models, or at least bolt three different output heads onto one backbone. During training, each task’s supervision stays in its own silo—never interacting, never helping the others.

SenseNova‑Vision flips that. It maps every visual task output onto just two modalities: text and image.

• Structured information – bounding boxes, keypoints, OCR text, camera poses – goes to text generation.
• Dense spatial information – depth maps, normal maps, segmentation masks, 3D point maps – goes to image generation.

That’s it. No special output heads. No per‑task decoding logic. Everything shares the same generator – text via the LLM’s next‑token prediction, images via the diffusion model’s VAE latent space. Both losses are optimised side by side during training, with shared parameters.

You might dismiss this as a neat engineering trick. But at the training level, it changes the game. Supervision signals from different vision tasks now talk to each other in the same parameter space. Detection coordinates help segmentation understand spatial layout; segmentation masks help depth estimation recognise object boundaries. That cross‑task transfer is nearly impossible in traditional task‑specific architectures, where each task’s gradient path stays isolated.

My initial worry was whether image generation quality would hold up for dense prediction. Diffusion models are great at natural images, but depth and normal maps demand numeric precision – a few grayscale levels off can mean metres of depth error. The numbers suggest this is more controllable than I expected, and we will get to that.

Four Vision Families, One Generation Interface

SenseNova‑Vision covers four major task families. Each has a clear encoding convention under the unified framework.

Structured Visual Understanding – Text Generation for Detection, OCR, Keypoints

These tasks naturally speak text. A bounding box is four normalised coordinates; keypoints are named coordinate pairs; OCR is text plus boxes.

The paper defines a lightweight markup syntax – e.g., <p>person</p> for class labels, <bbox>[0.1, 0.2, 0.5, 0.8]</bbox> for boxes. The model learns to generate these marked‑up sequences during training, and a simple regex parser turns them back into standard detection results.

The real gain is that the same text‑generation engine handles detection, referring comprehension, point localisation, and GUI grounding. You just change the instruction: “Detect all persons” vs. “Find the red cup on the table” – both produce coordinate‑bearing text, differing only in the prompt and markup order.

Dense Geometric Prediction – Image Generation for Depth and Normals

Depth and normal maps share the same spatial layout as the input image, so image generation feels natural.

Depth is encoded as inverse depth mapped to 0–255 grayscale; normals go straight into RGB channels. The model generates what looks like a grayscale or colour image, but it actually carries geometric values. Decoding reverses the transform to recover metric depth or normal vectors.

One engineering detail worth noting: depth values are clipped to 0.1m–80m during training, with anything outside treated as invalid. For normals, images with more than 10% sky are filtered out to avoid large invalid regions confusing the model.

Segmentation – Mixed Text‑and‑Image Outputs

Segmentation is trickier because you need to tell the user what each mask corresponds to, not just output the mask itself.

For single‑target tasks (referring, reasoning, interactive segmentation), a binary mask image suffices – white foreground, black background, threshold at decode time.

For multi‑target tasks (semantic, panoptic, GCG segmentation), you need both an instance list and colour‑coded masks. The instance list is generated as text, with each instance assigned a distinct RGB colour; the mask image is generated with those colours filled in. Text and image are supervised separately – text via cross‑entropy, image via rectified‑flow loss – but they are optimised together within the same sample.

This design gives users flexible control over colour description. You can say “mark all pedestrians in red” or “use #FF0000 for pedestrians” – the model understands both and renders the corresponding colour mask. That flexibility comes from coupling text and image: colour semantics are understood in text space, while the actual RGB values are rendered in image space.

Multi‑View Visual Geometry – Multiple Inputs, Mixed Outputs

Multi‑view reconstruction and camera pose estimation require processing several input images at once, outputting per‑view 3D point maps and camera parameters.

Point maps are generated as images – three RGB channels encode XYZ world coordinates per pixel. Camera poses are text – quaternions for rotation, translation split into direction and scale, all quantised into special tokens.

Cross‑view alignment is demanding. Training samples randomly select up to 10 views per scene, and all point maps are aligned to the first frame’s coordinate system. For datasets with sparse depth (e.g., LiDAR scans), they use LingBot‑Depth to densify before converting to point clouds. The appendix gives precise filtering thresholds for each preprocessing step.

Where Does the Data Come From? Building a 50M‑Sample Corpus

Unified output formats are useless without paired data – input images plus matching text or image targets.

SenseNova‑Vision tackles this by building the SN‑VC corpus, converting public vision datasets into a uniform instruction‑response format. The corpus spans four task families with over 50 million samples (the open‑source subset SN‑VC‑50M holds 50M of those).

Conversion logic is conceptually simple but labour‑intensive. Each source dataset has its own annotation quirks; you need scripts to map bounding boxes, masks, depth maps, and camera parameters into the unified text/image target format. Every sample also gets a natural language instruction – and there are many paraphrased templates per task to prevent overfitting to fixed prompts.

For datasets with incomplete or low‑quality annotations, they use auxiliary data engines to fill gaps. MoGe‑2 densifies sparse LiDAR depth; Rex‑Omni pipelines augment detection and OCR. Those augmented samples are released as SN‑VC‑50M; the rest can be reproduced from public sources using conversion scripts.

One important detail: any image that appears in evaluation benchmarks is excluded from training to prevent data leakage.

Training Strategy – Learning Text and Image Generation Together

The base model is Bagel‑7B‑MoT, a UMM that natively supports both multimodal understanding and generation. SenseNova‑Vision fine‑tunes this checkpoint without changing the architecture – only the data mix and some hyperparameters differ.

Training data comes in two streams. The first is SN‑VC – vision task samples providing structured visual supervision. The second is general multimodal data – VQA, text‑to‑image, image‑to‑image – to keep the base model’s broad capabilities from degrading.

During training, both streams are mixed with a weighted sampler. Each mini‑batch may contain detection (text loss), depth estimation (image loss), VQA (text loss), and image editing (image loss) samples. The text and image losses are computed and accumulated in the same optimisation step, then backpropagated together.

This mixed strategy has a real risk: gradients from different tasks might conflict, causing one task to learn fast while another stalls or even regresses. The convergence curves (Figure 6 in the paper) show that depth and normals converge fastest, multi‑view reconstruction is slower, and dense detection is the slowest. That indicates clear differences in learning speed, but all metrics eventually reach acceptable levels without catastrophic forgetting.

High‑resolution input is another key factor. The original Bagel limited SigLIP2 resolution on the generation side, but segmentation and dense prediction are sensitive to spatial detail. SenseNova‑Vision raises the generation‑side SigLIP2 resolution to 980 pixels while keeping the understanding side unchanged. This noticeably improves detail retention in masks and depth maps.

How Well Does It Work? The Gap to Specialised Models Is Narrowing

Structured Visual Understanding – Multiple Leads

On COCO detection, SenseNova‑Vision hits 53.7 mAP, beating Youtu‑VL’s 47.1. On LVIS long‑tail and Dense200 small‑object detection, it also outperforms recent generalist vision models. OCR localisation on HierText and ICDAR15 is competitive, and referring comprehension on RefCOCO/+/g approaches specialised models like LISA and X‑SAM.

These results show that text generation can deliver structured outputs at a precision that rivals task‑specific architectures. Dense detection is especially telling – a single image may contain dozens of small objects, and the model must output dozens of coordinate‑bearing entries without omissions or drift. That level of spatial precision from text generation is no accident.

Dense Geometric Prediction – Closing In on Specialists

On five depth benchmarks (NYUv2, KITTI, ETH3D, ScanNet, DIODE), SenseNova‑Vision achieves δ1 accuracy above 95% on all, reaching 98.1% on NYUv2 – very close to Depth Anything V2’s 97.9%. Normal estimation mean error on NYUv2 is 14.4°, near MoGe‑2’s 14.7°.

Compared with other generation‑based methods like Marigold, DICEPTION, and Lotus‑2, SenseNova‑Vision leads across every depth benchmark. That suggests diffusion‑based dense prediction is viable, and the unified multimodal framework does not sacrifice single‑task accuracy for multitasking.

Segmentation – Reasoning Segmentation Stands Out

On panoptic and semantic segmentation, SenseNova‑Vision’s PQ and mIoU lag slightly behind X‑SAM – which benefits from strong mask priors like SAM or Mask2Former, while SenseNova‑Vision learns mask generation from scratch without such priors.

But on reasoning segmentation (ReasonSeg), SenseNova‑Vision’s gIoU hits 63.2, beating LISA’s 52.9 and X‑SAM’s 56.6. Reasoning segmentation demands understanding complex language (“the ball that can only be hit into the hole at last”) and generating the corresponding mask. This task stresses vision‑language alignment, and SenseNova‑Vision’s edge likely comes from joint training across detection, referring, and VQA data in one unified space – better cross‑modal alignment as a result.

Multi‑View Geometry – Still a Gap to Specialists

On multi‑view point cloud reconstruction and camera pose estimation, SenseNova‑Vision still trails dedicated geometric models like VGGT and Depth Anything 3 – F1 on 7Scenes reconstruction is around 60 vs. 70+ for specialists.

That gap is unsurprising. Multi‑view geometry involves feature matching, triangulation, and pose optimisation, which specialist models handle with carefully designed geometric modules (cross‑view attention, iterative refinement). SenseNova‑Vision uses a generic generative framework with no explicit geometric reasoning – learning cross‑view alignment purely from data. Achieving this level without inductive biases is already nontrivial.

Interesting Findings – Task Capabilities Transfer

Two experiments are worth highlighting.

First, referring‑style interactive segmentation. Standard interactive segmentation requires visual prompts – points, boxes, scribbles, or mask images. But in a text‑and‑image generation interface, a point can be expressed much more compactly as text: <p><point>[0.345, 0.678]</point></p> is far leaner than a full image with a red dot.

The catch: the model never saw this exact “text coordinate → mask” mapping during training. It did see coordinate text → bounding boxes (detection), referring text → masks (referring segmentation), and visual point prompts → masks (interactive segmentation). Yet it composes these pieces and generates a mask from a text coordinate. That implies the model learned a general “spatial location” representation in the unified text‑image space – knowledge from different tasks transfers naturally.

Second, free‑form language to masks. The user says “mark the text area for ‘coke’ in red” – the model generates the correct text‑region mask. Another user says “use hexadecimal colour codes to segment all visible letters” – the model adapts. These phrasings and formats are not in the training set, but the model understands different colour expressions (natural language vs. hex codes) and renders them correctly on the mask.

Both observations point to the same direction: when detection, segmentation, depth, and OCR are jointly trained in one generative space, the model acquires a more general “vision‑language‑space” representation. This representation can recombine knowledge from different tasks, producing behaviours not explicitly covered in the training protocols.

Limitations and Future Directions

A few caveats are worth noting.

First, multi‑view geometry still trails specialists. If your primary need is 3D reconstruction or SLAM, dedicated models remain the safer bet.

Second, training cost is non‑trivial. A 7B model on 50M samples for 50K steps demands serious compute – even without exact GPU hours, the token count and context length (32K per sample) put it beyond most individual developers.

Third, image generation resolution is capped – masks and depth outputs match the input resolution (up to 980px). For high‑res applications (e.g., 4K fine segmentation), VAE compression loss may affect detail. The paper does not explore scaling to higher resolutions.

Looking ahead, several directions seem promising: extending unified generation from images to video, using temporal supervision for stronger spatial understanding; stronger in‑context learning so users can define new vision tasks with few‑shot examples without retraining; and deeper integration with powerful LLMs to tackle complex visual reasoning.

There is a longer‑term vision too: if 2D and 3D perception can be absorbed into a single generative framework, this might evolve beyond a vision tool toward a world model – one that perceives space, understands physics, and predicts interactions. That remains speculative, but the direction is compelling.

Practical Summary / Action Checklist

If you are considering SenseNova‑Vision or a similar unified multimodal model for your vision tasks, here is a quick decision guide:

1. Detection and structured outputs – You can rely on it. Accuracy approaches or matches specialists. Ideal if you need multiple structured outputs (detection + attributes + OCR) from one forward pass.

2. Depth and normal estimation – It works well. The gap to specialised geometry models is acceptable for most use cases. You can replace a separate depth model without much loss.

3. Segmentation – Reasoning segmentation is a standout; regular segmentation is slightly behind specialists. If your scenario involves complex language instructions (“segment the person in red standing under the tree”), SenseNova‑Vision may actually outperform dedicated segmenters.

4. Multi‑view geometry – Not yet recommended. Stick with VGGT or Depth Anything 3 for now.

5. Language‑defined novel tasks – This is the most exciting angle. If your application needs users to define custom visual tasks in natural language (“highlight all blue circular objects”), this approach is worth exploring.

6. Deployment – The model is open‑source. You serve one model instead of multiple specialists, which reduces GPU memory and latency. 7B parameters fit on a single A100.

One‑Page Snapshot

Frequently Asked Questions

How does SenseNova‑Vision compare to SAM?
SAM is a dedicated segmentation model with a strong mask decoder prior – it outputs high‑quality masks but only segmentation. SenseNova‑Vision is a unified generator that handles detection, depth, pose, etc. – segmentation is one of many tasks. For pure segmentation quality, SAM still wins, but SenseNova‑Vision excels when you need language‑aware reasoning segmentation.

How much training data is required?
Over 50 million converted vision instruction samples, plus auxiliary multimodal data. The open‑source subset (SN‑VC‑50M) is directly downloadable; the rest can be reproduced via conversion scripts.

What about inference speed?
The paper does not report exact latency. Since text and image generation share the same Transformer, inference speed is similar to the Bagel base. Compared to cascading multiple specialists (detection → segmentation → depth), end‑to‑end latency should be lower because all tasks share one forward pass.

Can I use Chinese instructions?
The base model (Bagel) supports multiple languages, and SN‑VC includes some Chinese data (converted from Chinese datasets). So Chinese instructions work, though English likely generalises better due to the training distribution.

How does it handle tasks not in the training set?
This is where the unified framework shines. The model can combine capabilities from different tasks to respond to novel instructions – e.g., text‑coordinate‑to‑mask segmentation was never trained but emerged from detection and segmentation knowledge. Generalisation has limits, though – entirely new tasks still need training data.

What is the difference from GPT‑4V or other multimodal LLMs?
GPT‑4V is understanding‑only – it reads images and outputs text, but cannot generate image outputs (masks, depth). SenseNova‑Vision is a generative UMM – it can both understand and generate images, so it outputs dense visual predictions directly. They are complementary rather than competing.