CLIP

This post was migrated from Tistory. You can find the original here.

CLIP (Contrastive Language-Image Pre-Training)

https://github.com/openai/CLIP

CLIP (Contrastive Language-Image Pre-Training) is a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3.

(Can it really be “instructed in natural language”? That phrasing always struck me as a bit odd, but it’s the way OpenAI describes it, so I’ll leave it as is.)

The main goal of CLIP is to train the embeddings extracted from the image encoder and the text encoder to share the same embedding space. That way, the text “a cat wearing a hat” can end up with a representation vector similar to that of an actual photo of a cat wearing a hat.

That’s why CLIP uses image-text pairs rather than image-label pairs.

The image-text pairs were not mapped by hand. Instead, images and their associated natural language text were automatically extracted via web crawling, resulting in a massive dataset of 400 million image-text pairs.

Since a dataset made up of image-text pairs has no fixed labels, it can’t be trained as a classification problem. So CLIP instead trains the network to solve the problem of finding the correct pairings between N images and N pieces of text.

Pre-training techniques that make use of image-text pairs already existed before CLIP.

image captioning (transformer based) => a cat running while wearing a hat

image captioning (bag of words based) => hat, run, cat

However, these were less efficient than existing classification models, since they were larger in size and took longer to train and to run inference.

Unlike previous techniques, CLIP used contrastive learning as its method for connecting images and text.

Contrastive learning is a method of learning the relationship between positive and negative samples within the data. CLIP embeds the sentence that matches an image as the positive sample and sentences that don’t match as negative samples, and it is trained so that similarity with the positive text increases while similarity with negative text decreases.

References

http://dmqm.korea.ac.kr/activity/seminar/365

https://dhk1349.tistory.com/11

[[Paper Review] Learning Transferable Visual Models From Natural Language Supervision (CLIP)

Hello. The paper I’ll be reviewing today is Learning Transferable Visual Models From Natural Language Supervision, published by OpenAI, which is often just called CLIP. Personally, I feel this paper marks a turning point…

dhk1349.tistory.com](https://dhk1349.tistory.com/11)

https://inforience.net/2021/02/09/clip_visual-model_pre_training/

[(CLIP) Visual Model Pre-training Using Text Information

Click here » Hello, world !! (from ShadowEgo) In this post, I’ll introduce the CLIP model[1] recently announced by OpenAI. It reduces the dimensionality of data such as images and audio while converting it into a more meaningful form…

inforience.net](https://inforience.net/2021/02/09/clip_visual-model_pre_training/)