Why not just use CLIP for every image in your RAG corpus?

While using CLIP for image retrieval simplifies the ingestion pipeline by removing the need for an intermediate captioning step, it creates a significant operational hurdle. Because CLIP maps images and text into a shared latent space, the resulting similarity scores are not uniform across different modalities. Consequently, a single global threshold for similarity will likely misfire, leading to poor retrieval performance. Engineers must account for these modality-specific distribution shifts rather than relying on a one-size-fits-all approach.