WeChat Open-Sources WeMM-Embedding: A Detailed Overview of the Multimodal Vector Model and Benchmark Comparisons

Background: Why Multimodal Embedding Is Needed ​

Two basic concepts first: Embedding converts text, images, and other content into numeric vectors, enabling computers to compute "how similar two pieces of content are"; the closer the vectors, the more similar the content. Retrieval in search engines, matching in recommendation systems, and knowledge base retrieval all rely on embeddings at their core.

Traditional retrieval is "text searches text, image searches image," requiring extra conversion systems for cross-modal scenarios. Multimodal Embedding aims to encode text, images, video, and documents into a single vector space, providing a unified foundation for capabilities like "search video with a sentence," "find documents with an image," and "find similar products with a text description."

There are two mainstream approaches to multimodal embedding, each with pain points:

• Dual-tower CLIP-style architectures: Images and text are encoded separately and then aligned, leaving modality paths disconnected and making it difficult to directly model "image-text mixed" inputs and complex reasoning.
• Using Multimodal Large Language Models (MLLMs) directly as encoders: Insufficient fine-grained ranking capability, prone to false negatives on negative samples, and high computational overhead.

WeMM-Embedding is designed to address these issues.

Model Overview: One Family, Three Sizes ​

WeMM-Embedding stands for WeChat Multi-Modal Embedding, developed by WeChat Vision (Tencent), officially open-sourced in late August 2026 with code and model weights publicly released under the Apache 2.0 license.

All three versions support: text, images, video, visual documents (e.g., screenshots, PDF layout pages, receipts), and arbitrarily interleaved multimodal inputs (multiple images and videos mixed within a text segment). Audio input is not currently supported.

How Vectors Are Computed ​

The vector extraction method for WeMM-Embedding is straightforward: input is organized as "task instruction + ordered multimodal segments," with a dedicated <embedding> token appended at the end of the sequence. The output vector is taken from the last-layer hidden state at that token position, followed by L2 normalization.

Multiple <embedding> tokens can be placed. For instance, when processing "video + subtitle text," placing one after the video tokens and another at the sequence end yields both "video-only representation" and "video-plus-text fused representation" in a single forward pass—offering great flexibility.

Matryoshka: Adjustable Output Dimensions ​

WeMM-Embedding supports Matryoshka Representation Learning (MRL)—that is, training with high-dimensional vectors (e.g., 2048/4096 dims) first, and then at deployment, taking the first N dimensions and re-normalizing to obtain shorter, storage- and computation-efficient low-dimensional vectors.

Official technical report data shows: on MMEB-v2, the 2B model compressed to 256 dimensions retains 98.7% of the full-dimensional image and video retrieval performance. This allows developers to trade off between effectiveness, speed, and cost as needed—for high-volume, low-latency retrieval scenarios, low-dimensional vectors can significantly reduce storage and compute costs.

Technical Highlights: Native Multimodal Backbone + Two-Stage Training ​

Built on Qwen3.5 Native Multimodal Architecture ​

According to the Hugging Face model card and multiple media reports, WeMM-Embedding is built on the native multimodal architecture Qwen3.5 (Qwen3.5 is a natively visual-language model from the Tongyi team, fusing text and vision early, supporting multiple languages). Choosing a native multimodal backbone means the model already possesses "interleaved text-image understanding" capabilities from the pretraining stage, which aligns better with the goal of unified representation than "stitching two single-modal models together afterward."

Unified Pair Data Protocol ​

To unify diverse tasks such as classification, image-text retrieval, video grounding, and QA reasoning, the research team abstracts all tasks as measuring the relevance between "source instance vs. target candidate," constructing billions of multimodal training samples covering six data categories:

• Weakly supervised image-text/video pairs: Massive naturally aligned web data, providing coarse-grained cross-modal semantic priors.
• Fine-grained caption pairs: High-quality dense descriptions covering object attributes, spatial relationships, long-term causality, and dynamic events.
• Cross-modal retrieval pairs: Compositional retrieval, long-context retrieval, spatial grounding, agent/tool/GUI retrieval, etc.
• Classification pairs: Converting recognition tasks into "source instance ↔ category description" matching.
• Multimodal QA pairs: OCR, document chart parsing, spatiotemporal logical reasoning, etc.
• Graded relevance pairs: Multi-level relevance annotations for fine-grained ranking needs in search and recommendation.

Two-Stage Training and Cross-Scale Knowledge Distillation ​

Training proceeds in two stages:

1. Large-scale multimodal alignment: Trained on billions of paired data. Standard pairs use contrastive learning objectives; graded relevance pairs use ranking objectives; near-duplicate samples are masked.
2. Fine-tuning on curated data: Refined on curated data roughly one-tenth the size of the first stage. The curated set is constructed via semantic ID inverse-density resampling (targeting long-tail data), MLLM-based cleaning and rewriting, and modality-adaptive hard negative mining. It also introduces re-ranker supervision and embedding distillation from "same-family larger models."

According to the technical report and media coverage, the 2B and 4B models are distilled using the frozen 9B version as teacher, while the 9B is obtained by merging multiple complementary variants and expands visual input budgets via higher resolution inputs and denser video frame sampling—this is the key to "smaller models punching above their weight."

Evaluation Benchmarks: MMEB-v2 and MMEB-v3 ​

All scores below are from the Tencent technical report (arXiv:2608.24053), with higher scores being better. Please note: these are the model's own reported results, and some baselines are closed-source leaderboard submissions; interpretation requires context.

MMEB-v2 (78 datasets) ​

Image and video tasks use Hit@1, visual document tasks use NDCG@5:

† Closed-source leaderboard submissions, with no public model weights or inference APIs.

Two noteworthy points:

• 2B beats 8B: WeMM-Embedding-2B (77.9) surpasses the previous leading open-source 8B baseline Qwen3-VL-Embedding-8B (77.8). Note that Qwen3-VL-Embedding, also built on Qwen3-VL, was released in January 2026 and was previously the open-source SOTA on MMEB-v2.
• 9B tops the leaderboard: Overall average of 80.6, exceeding all public and commercial closed-source models (including DME-Medium's 78.4).

MMEB-v3 (190 tasks) ​

V3-All includes 78 MMEB-v2 tasks, 53 text tasks, 47 agent tasks, 11 audio tasks, and MCMR (multimodal compositional retrieval). Text tasks use NDCG@5; agent / MCMR / audio use Hit@1; unsupported tasks score zero:

The MMEB-v3 results need more cautious interpretation: WeMM-Embedding does not support audio, so its overall score is achieved while "opting out" of the audio track; conversely, audio-capable models like E5-Omni and LCO-Embedding-Omni are pulled down by audio tasks in V3-All. Looking only at Text / Agent / MCMR, WeMM-Embedding-9B ranks first in all three.

Reproducibility ​

To facilitate community reproduction, the WeChat team has released the complete evaluation code for MMEB-v3 (under the mmeb_v3_eval/ directory). It is based on the official VLM2Vec evaluation pipeline with minimal modifications: multi-node multi-GPU inference support, backbone adaptation for WeMM-Embedding preprocessing and batch inference, aligned data instructions, and 64-frame video sampling. This is critical for credibility—the evaluation process remains consistent with existing authoritative baselines rather than being built from scratch.

Deployment: Not a Demo, Already Running in WeChat Businesses ​

One of the biggest highlights of WeMM-Embedding is its real-world deployment. According to the technical report abstract and media coverage, the model family has been massively deployed in WeChat's Channels, Official Accounts, Moments, and e-commerce (as well as search scenarios like "Search Everything"), serving content recall, search, and recommendation.

• Achieved significant improvements on 26 internal WeChat task benchmarks.
• Completed 14 online A/B tests, all showing consistent improvements.
• This means: when you browse Channels, search Official Accounts, or view Moments recommendations, this embedding model may already be running behind the scenes.

"Production-ready is where true capability proves itself"—beyond benchmark numbers, this is the most noteworthy aspect of WeMM-Embedding.

Quick Start ​

Environment Setup ​

The official recommendation is to use transformers==5.2.0 to ensure preprocessing behavior matches training (newer versions may have preprocessing differences):

pip install torch transformers==5.2.0 "qwen-vl-utils[decord]==0.0.14" \
  "sentence-transformers>=5.7.0" "accelerate>=1.1.0"

Transformers Inference ​

python examples/transformers_inference.py \
  --model /path/to/WeMM-Embedding-2B \
  --image /path/to/image.jpg \
  --video /path/to/video.mp4 \
  --dimension 2048

The example generates separate embeddings for text, image, and video; omitting --dimension outputs the full dimension. --model also accepts Hugging Face model IDs (e.g., tencent/WeMM-Embedding-2B).

Sentence Transformers ​

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("tencent/WeMM-Embedding-9B", trust_remote_code=True)
embeddings = model.encode(["Nice weather today.", "It's so sunny outside!"])

Text, image, and video all go through encode(), with MRL dimensions specified by --dimension.

Online Service Deployment ​

The official repository has verified vLLM 0.27.0 and SGLang 0.5.9:

# vLLM
MODEL_PATH=/path/to/WeMM-Embedding-2B
vllm serve "$MODEL_PATH" \
  --runner pooling \
  --chat-template "$MODEL_PATH/embedding_chat_template.jinja"

# SGLang
MODEL_PATH=/path/to/WeMM-Embedding-2B
python scripts/patch_sglang_video.py
python -m sglang.launch_server \
  --model-path "$MODEL_PATH" \
  --is-embedding \
  --enable-precise-embedding-interpolation

The repository also provides one-click scripts scripts/serve_vllm.sh and scripts/serve_sglang.sh.

Limitations and Caveats ​

1. No audio support: The current version does not accept audio input, scoring 0 on MMEB-v3 audio tasks. Scenarios requiring unified audio-video retrieval are not suitable at this time.
2. Self-reported benchmark results: MMEB-v2/v3 scores are from the Tencent technical report—official self-reported numbers; third-party independent reproduction is worth waiting for.
3. Baseline comparison nuances: The MMEB-v2 table includes closed-source leaderboard submissions (DME-Small/Medium, marked with †), requiring caution in cross-comparison as their weights are not public.
4. Version sensitivity: The official recommendation is to lock transformers==5.2.0; regression validation is needed before upgrades. Community discussions already exist around integration with newer versions of transformers and sentence-transformers (Hugging Face model page discussions).
5. Open-source but commercially validated: Model weights and code are fully open-source under Apache 2.0, but its internal WeChat deployment data (e.g., 26 internal task benchmarks, 14 A/B tests) is only reported in the technical report and cannot be reproduced externally.

Data Sources and References ​

The facts in this article are based on official channels and cross-validated by multiple media outlets:

• Official GitHub repository (with Chinese and English README, evaluation code): github.com/Tencent/WeMM-Embedding
• Technical report (arXiv 2608.24053): arxiv.org/abs/2608.24053
• Hugging Face model collection: huggingface.co/collections/tencent/wemm-embedding
• Hugging Face paper page (with abstract and training details): huggingface.co/papers/2608.24053
• 智猩猩 / Tencent News "WeChat actually open-sourced its 'invisible AI'": news.qq.com
• 我爱计算机视觉 "2B beats 8B baseline, 9B tops MMEB-v2": sohu.com
• TechNode "WeChat Team open-sources WeMM-Embedding": sohu.com
• Baseline Qwen3-VL-Embedding technical report: Qwen3-VL-Embedding [file content end]