22b9d24a81
|
||
|---|---|---|
| .. | ||
| Clip.ipynb | ||
| README.md | ||
README.md
Multi-Modal Networks
Afta transformer models don show say dem fit solve NLP tasks well, people don dey use di same or similar architecture for computer vision tasks. Pipo dey show interest for how dem go fit build models wey go combine vision and natural language sabi. One of di try wey OpenAI do na CLIP and DALL.E.
Contrastive Image Pre-Training (CLIP)
Di main idea for CLIP na to fit compare text prompts wit image and check how di image match di prompt.
Picture from this blog post
Di model dey train wit images wey dem collect from Internet and di captions wey follow di images. For each batch, we go take N pairs of (image, text), and turn dem to vector representations I1,..., IN / T1, ..., TN. Di representations go then dey match together. Di loss function dey set to make di cosine similarity between vectors wey match one pair (e.g. Ii and Ti) high, and make di cosine similarity between di other pairs low. Na why dem dey call dis approach contrastive.
CLIP model/library dey available for OpenAI GitHub. Di approach dey explain for this blog post, and dem explain am well well for this paper.
Once dem don pre-train di model, we fit give am batch of images and batch of text prompts, and wetin e go return na tensor wit probabilities. CLIP fit dey use for plenty tasks:
Image Classification
Suppose we wan classify images between, say, cats, dogs and humans. For dis case, we fit give di model one image, and series of text prompts: "a picture of a cat", "a picture of a dog", "a picture of a human". For di vector wey get 3 probabilities we go just pick di index wey get di highest value.
Picture from this blog post
Text-Based Image Search
We fit also do di opposite. If we get collection of images, we fit pass di collection to di model, and one text prompt - e go give us di image wey match di prompt pass.
✍️ Example: Using CLIP for Image Classification and Image Search
Open di Clip.ipynb notebook to see how CLIP dey work.
Image Generation wit VQGAN+ CLIP
CLIP fit also dey use for image generation from text prompt. To do dis, we need generator model wey fit generate images based on vector input. One of di models wey fit do dis na VQGAN (Vector-Quantized GAN).
Di main ideas wey make VQGAN different from ordinary GAN na:
- E dey use autoregressive transformer architecture to generate sequence of visual parts wey dey rich wit context wey go form di image. Di visual parts na CNN dey learn am.
- E dey use sub-image discriminator wey dey check whether di parts of di image na "real" or "fake" (unlike di "all-or-nothing" approach for traditional GAN).
Learn more about VQGAN for di Taming Transformers website.
One big difference between VQGAN and traditional GAN na say di traditional GAN fit produce better image from any input vector, but VQGAN fit produce image wey no go make sense. So, we need to guide di image creation process well, and CLIP fit help us do dis.
To generate image wey match text prompt, we go start wit random encoding vector wey go pass through VQGAN to produce image. Then CLIP go dey use to produce loss function wey go show how di image match di text prompt. Di goal na to reduce dis loss, using back propagation to adjust di input vector parameters.
One better library wey dey implement VQGAN+CLIP na Pixray
 |
 |
 |
|---|---|---|
| Picture wey dem generate from prompt a closeup watercolor portrait of young male teacher of literature with a book | Picture wey dem generate from prompt a closeup oil portrait of young female teacher of computer science with a computer | Picture wey dem generate from prompt a closeup oil portrait of old male teacher of mathematics in front of blackboard |
Pictures from Artificial Teachers collection by Dmitry Soshnikov
DALL-E
DALL-E 1
DALL-E na version of GPT-3 wey dem train to generate images from prompts. E don train wit 12-billion parameters.
Unlike CLIP, DALL-E dey take both text and image as one stream of tokens for both images and text. So, from plenty prompts, you fit generate images based on di text.
DALL-E 2
Di main difference between DALL.E 1 and 2 na say e dey generate more realistic images and art.
Examples of image generation wit DALL-E:
 |
 |
 |
|---|---|---|
| Picture wey dem generate from prompt a closeup watercolor portrait of young male teacher of literature with a book | Picture wey dem generate from prompt a closeup oil portrait of young female teacher of computer science with a computer | Picture wey dem generate from prompt a closeup oil portrait of old male teacher of mathematics in front of blackboard |
References
- VQGAN Paper: Taming Transformers for High-Resolution Image Synthesis
- CLIP Paper: Learning Transferable Visual Models From Natural Language Supervision
Disclaimer:
Dis docu wey you dey see don use AI translation service Co-op Translator take translate am. Even though we dey try make sure say e correct, make you sabi say translation wey machine do fit get mistake or no too accurate. Di original docu for di language wey dem first write am na di main correct one. If na important information, e go better make professional human translator check am. We no go fit take blame for any misunderstanding or wrong interpretation wey fit happen because you use dis translation.