Visual search (search by image)

Customers often cannot name what they want ("that chair from the café"), and staff cannot find the right spare part or photo in a large archive by keywords. Visual search embeds images into a vector space so that a photo, crop or screenshot retrieves the most similar items. Typical benefits are better product discovery in e-commerce, faster asset reuse in marketing and media archives, and quicker part identification in service and maintenance.

Problem & value

• Today: catalogue search relies on text attributes that are incomplete, inconsistent or in the wrong language. Media and DAM teams rely on manual tags; service technicians page through catalogues. Duplicates and near-duplicates (re-uploads, cropped images) are hard to find.
• AI value: language-independent retrieval, "find similar", "shop the look", duplicate detection and combined image+text queries ("this dress, but in green"). It also reduces dependence on tag quality.
• Evidence: vendors publish conversion and engagement uplifts for visual search, but these are self-reported and shop-specific; measure with your own A/B test rather than relying on quoted figures. Pinterest published its production visual-search architecture early [5].

Variants

• E-commerce "search by photo" / shop-the-look: fashion, furniture, home and decor, DIY.
• Spare-part and technical catalogue lookup: photo of a worn part to SKU (see VIS-018 for labels/OCR).
• Digital asset management (DAM) and media archives: find similar shots, reuse licensed images, detect near-duplicates (see KNOW-016 for audio/video).
• Brand and IP protection: find counterfeits or unlicensed use of own images via reverse image search.
• Public sector / culture: museum and archive collections (e.g. Europeana-style open collections).
• Private use: reverse image search on photos (plant, product, place) with consumer tools.
• Excluded: searching for people by face is face recognition (VIS-016) and falls under strict rules; do not build it as a "visual search" feature (see Risks).

How it works

1. catalogue/asset images
2. (optional) detect & crop items (object detection / segmentation)
3. image embedding (CLIP/SigLIP-class or multimodal embedding API)
4. vector index (HNSW/IVF) + metadata store query photo (+ text)
5. same embedding (+ optional crop of the item of interest)
6. ANN search + metadata filters (stock, price, category, rights)
7. rerank (cross-modal reranker / VLM / business rules)
8. results UI with feedback signals (click, add-to-cart)

• Embedding model: a contrastively trained vision-language model maps images and text into one space, so both image queries and text queries work. Domain fine-tuning improves fine-grained matches (fabrics, parts).
• Detection step: on photos with several objects, detect and crop the item, otherwise the background dominates the embedding.
• Reranking: top-k is re-scored with attribute match (colour, material) or a VLM; filters ensure that only purchasable, rights-cleared items appear.
• Human-in-the-loop: merchandisers curate synonyms, pin or block results and review failed queries; DAM users confirm matches before reuse of licensed images.

Data preparation

Data needed: product, asset or archive images with identifiers and basic metadata. In an SMB these live in the shop system, the ERP or PIM, on file shares (often with names like IMG_0042.jpg), in supplier portals and sometimes only on paper catalogues.

Typical gaps: duplicates and near-duplicates, inconsistent image sizes and backgrounds, missing or wrong article numbers, variants without photos, outdated products, unclear image rights, and photos of people or licence plates in the background.

Preparation steps: 1. Export images and master data from ERP or shop (PREP-005). 2. Deduplicate and normalise sizes, formats and backgrounds with simple scripts (ImageMagick, Python). 3. Link each image to a stable article ID and add basic attributes (PREP-010). 4. Scan paper catalogues if needed (PREP-006). 5. Build 50 to 100 test queries with expected hits (PREP-018). 6. Check image rights and blur faces or plates where personal data appears (PREP-022).

Effort: typically a third of the project; embedding itself is automatic, but clean IDs and metadata decide result quality.

Implementation paths

Which path for whom

• SMB: Path A, because shop systems and marketplaces offer mature visual-search plugins at modest monthly prices and the process is standard.
• Enterprise: Path B, since catalogue search is a core conversion lever and embedding APIs integrate with the own PIM and search stack at scale.
• Developer: Path C, because open embedding models (e.g. CLIP variants) and vector databases run on modest hardware without per-query costs.
• Public: Path C, since archives and museum collections need sovereign hosting and open models avoid procurement and lock-in issues.

Path A — Buy (off-the-shelf; recommended for most SMBs)

• Visual search / discovery SaaS: Algolia (NeuralSearch, EU data centres), Bloomreach, Constructor, Klevu, Syte, ViSenze, Vue.ai, Lucidworks; EU-based: Empathy.co (ES), Findologic (DE), Doofinder (ES), most other vendors listed are non-EU, so check data location.
• Platform plugins: Shopify and Adobe Commerce search apps, Salesforce Commerce Cloud.
• DAM: Adobe Experience Manager Assets, Bynder (NL), Canto, Frontify, Cloudinary and Pimcore (AT) offer similar-image or AI-tagging search.
• Choose A when: you run a standard shop or DAM and want results in weeks without ML staff.

Path B — Build on cloud APIs

• Embedding APIs: Google Vertex AI multimodal embeddings (EU regions), Amazon Titan Multimodal Embeddings on Bedrock (Frankfurt/Ireland), Azure AI Vision multimodal embeddings, Cohere Embed v4 (multimodal), Jina AI embeddings (Berlin-founded, now part of Elastic; multimodal v4), Voyage multimodal embeddings.
• Vector stores: managed OpenSearch, Elasticsearch (kNN), MongoDB Atlas Vector Search, pgvector on managed Postgres, Qdrant Cloud (DE-founded, EU regions), Weaviate Cloud, Pinecone.
• Rerank / explain: a frontier VLM (Claude, Gemini, GPT, Mistral Pixtral class) checks top-10 candidates and writes an attribute explanation.
• Choose B when: you have a developer team and want control over ranking, filters and UX.

Path C — Self-host (open source)

• Models: OpenCLIP, SigLIP / SigLIP 2 (Google, open weights), Meta DINOv2 / DINOv3 for instance-level similarity, Jina CLIP v2, Nomic Embed Vision, ColPali-style models for page images.
• Detection/segmentation for cropping: YOLO family (check licence, e.g. AGPL for Ultralytics), RT-DETR, Grounding DINO, SAM 2.
• Index: Qdrant, Milvus, FAISS, pgvector, OpenSearch; serve models via Triton, vLLM or ONNX Runtime.
• Hardware: HW1 for query-time embedding of small models; HW2 (one 24 GB GPU) comfortably indexes millions of images in days and serves hundreds of queries per second with small ViT models.
• Choose C when: data must not leave your premises (archives, unreleased product images) or catalogue size makes API cost high.

Path D — Custom training

Fine-tune the embedding model with metric learning (triplet or contrastive loss) on your own catalogue when fine-grained distinctions matter (spare parts, textiles, art). Needs thousands of matched pairs (click logs, "same SKU, different photo", user-generated vs studio images). A lightweight adapter on a frozen backbone is often enough and cheaper than full training.

State of the art (2026)

• Foundation embeddings (SigLIP 2, DINOv3, multimodal API embeddings) deliver strong zero-shot retrieval; DINO-style self-supervised features are better at instance matching, CLIP/SigLIP-style at semantic matching and text queries. Many systems combine both.
• Composed retrieval (image + text modification) and multimodal RAG (retrieving images to ground VLM answers) are now standard options; late-interaction models improve document-image retrieval.
• Hybrid search (vector + BM25 + filters) with reranking beats pure vector search in commerce.
• Limits and failure modes: background and context bias; viewpoint and lighting gaps between user photos and studio shots; poor results for fine-grained look-alike parts; text in images is weakly captured (combine with OCR, VIS-018); embedding drift when the model is replaced requires re-indexing; bias towards popular items; adversarial or counterfeit imagery.

Shared resources & combinations

• Same stack (vlm + embeddings + vector-db): KNOW-005 semantic product search, KNOW-016 multimodal archive search, DOC-014 document parsing for RAG; one vector database and one embedding service can serve all of them.
• Upstream/downstream: VIS-017 visual product recognition, VIS-009 shelf monitoring (matching detected products to the catalogue), VIS-025 video tagging, VIS-030 anonymisation of user-uploaded photos before storage, VIS-018 OCR for text on parts.
• Hardware: the HW2 GPU box used for detection (VIS-001, VIS-009) also covers embedding jobs.

Effort, cost & KPIs

• Path A: days to a few weeks; SaaS fees usually scale with query or catalogue volume.
• Path B: 1–3 months incl. indexing pipeline, UI and evaluation; cost drivers are one-off embedding of the catalogue, vector storage and per-query embedding calls.
• Path C: 2–4 months; GPU for batch indexing, vector DB operations and monitoring effort.
• Path D: adds 1–3 months for data collection and training.
• KPIs: recall@k / NDCG on a labelled query set, zero-result rate, click-through and add-to-cart from visual search, conversion vs control, latency (p95), index freshness, share of queries needing fallback.

People & change

Who and when: management decides on scope; the department lead (e-commerce, sales or archive) owns relevance criteria; data owners of PIM and image libraries clean the catalogue first; IT integrates search; the DPO is consulted when user-uploaded photos are processed. Involve end users, such as sales and service staff, early with real search examples.

Works council: generally no, as the feature serves customers and does not evaluate employees (§87(1) no. 6 BetrVG). Exception: if internal use is logged per employee, for example in service desks, co-determination may apply. Inform the works council anyway under §90(1) no. 3 BetrVG if work processes change.

Data protection: customer uploads can contain faces, locations or documents; delete queries promptly, do not use them for training without a legal basis, and sign an AVV (Art. 28 GDPR) with the provider (GUIDE-005).

AI literacy: support staff with a short briefing on what similarity search can and cannot do (Art. 4 AI Act, GUIDE-006).

Pilot and communication: pilot on one product category and compare with text search (GUIDE-009).

Typical concerns: wrong hits and fear of replacing product experts. Keep experts in control of tuning and synonyms, and show a fallback to classic search.

Risks, regulation & compliance

• EU AI Act: minimal risk for product and asset search; AI literacy duty (Art. 4) applies. Art. 5 warning: a "search by image" feature that identifies people by face, or builds face databases through untargeted scraping of internet or CCTV images (Art. 5(1)(e)), is prohibited; real-time remote biometric identification and other biometric uses are restricted or high-risk (Annex III). Exclude person/face search from consumer visual search, filter or blur faces in indexes (VIS-030), and do not offer face-based reverse search tools [7].
• GDPR: user-uploaded photos can show people, homes, number plates or location metadata; define purpose, minimise retention (delete query images after search), strip EXIF, inform users in the privacy notice, sign a DPA with API providers, prefer EU regions. Archive images with identifiable people may involve special-category data (Art. 9) if faces reveal sensitive attributes by inference.
• Copyright and rights: indexing third-party images may infringe copyright or database rights; restrict results to rights-cleared assets; EU text-and-data-mining exceptions (DSM Directive Art. 3/4) matter when training own models.
• Other risks: biased or manipulated ranking, counterfeit promotion, DSA transparency duties for recommender parameters on large platforms.
• Mitigations: face/person exclusion, rights metadata filters, query logging with short retention, offline evaluation per category, merchandiser override.

Getting started

1. Define the query types (shop-the-look, part lookup, DAM) and build 200 real test queries with expected results.
2. Index a representative slice of the catalogue with two embedding models (one API, one open-weight) and compare recall@10.
3. Add detection/cropping and metadata filters; add a text+image hybrid query.
4. Launch behind an A/B test on one category; log clicks and zero-result queries.
5. Run a DPIA-style check for uploads (retention, EXIF, faces) before general rollout.

Sources

1. Google Cloud — Vertex AI multimodal embeddings API — https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-multimodal-embeddings
2. AWS — Amazon Titan Multimodal Embeddings (Bedrock) — https://docs.aws.amazon.com/bedrock/latest/userguide/titan-multiemb-models.html
3. SigLIP 2: Multilingual Vision-Language Encoders (paper) — https://arxiv.org/abs/2502.14786
4. DINOv3 (Meta AI, paper) — https://arxiv.org/abs/2508.10104
5. Pinterest Engineering — visual search at Pinterest (paper) — https://arxiv.org/abs/1505.07647
6. OpenCLIP project — https://github.com/mlfoundations/open_clip
7. Regulation (EU) 2024/1689 (AI Act), Art. 5 — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
8. Qdrant documentation — https://qdrant.tech/documentation/
9. Regulation (EU) 2016/679 (GDPR) — https://eur-lex.europa.eu/eli/reg/2016/679/oj