AI-For-Beginners/translations/en/lessons/4-ComputerVision/10-GANs

Generative Adversarial Networks

In the previous section, we explored generative models: models capable of creating new images similar to those in the training dataset. VAE was a good example of such a model.

Pre-lecture quiz

However, if we attempt to generate something truly meaningful, like a painting at a reasonable resolution, using VAE, we may find that training does not converge effectively. For this purpose, we need to learn about another architecture specifically designed for generative models—Generative Adversarial Networks, or GANs.

The core idea of a GAN is to have two neural networks trained in opposition to each other:

Image by Dmitry Soshnikov

✅ A quick glossary:

• Generator: A network that takes a random vector and produces an image as output.
• Discriminator: A network that evaluates whether an image is real (from the training dataset) or fake (generated by the generator). Essentially, it acts as an image classifier.

Discriminator

The discriminator's architecture is similar to a standard image classification network. In its simplest form, it can be a fully-connected classifier, but more commonly, it will be a convolutional network.

✅ A GAN that uses convolutional networks is called a DCGAN.

A CNN-based discriminator typically includes several convolutional and pooling layers (with decreasing spatial dimensions) followed by one or more fully-connected layers to extract a "feature vector" and a final binary classifier.

✅ "Pooling" refers to a technique that reduces the size of the image. "Pooling layers reduce the dimensions of data by combining the outputs of neuron clusters at one layer into a single neuron in the next layer." - source

Generator

The generator is slightly more complex. You can think of it as a reversed discriminator. Starting with a latent vector (instead of a feature vector), it uses a fully-connected layer to reshape the vector into the desired size/shape, followed by deconvolutions and upscaling. This is similar to the decoder part of an autoencoder.

✅ Since convolution layers are implemented as linear filters traversing the image, deconvolution is essentially similar to convolution and can be implemented using the same layer logic.

Image by Dmitry Soshnikov

Training the GAN

GANs are called adversarial because the generator and discriminator are constantly competing against each other. Through this competition, both networks improve, enabling the GAN to produce increasingly realistic images.

Training occurs in two stages:

• Training the discriminator: This step is straightforward. We generate a batch of images using the generator, label them as 0 (fake images), and take a batch of images from the input dataset labeled as 1 (real images). We calculate the discriminator loss and perform backpropagation.
• Training the generator: This step is more challenging because we don't directly know the expected output for the generator. We use the entire GAN network (generator followed by discriminator), feed it random vectors, and expect the output to be 1 (indicating real images). We freeze the discriminator's parameters (to prevent it from being trained during this step) and perform backpropagation.

During training, the losses for both the generator and discriminator typically oscillate rather than decrease steadily, reflecting the ongoing improvement of both networks.

✍️ Exercises: GANs

• GAN Notebook in TensorFlow/Keras
• GAN Notebook in PyTorch

Challenges in GAN Training

GANs are notoriously difficult to train. Common issues include:

• Mode Collapse: This occurs when the generator learns to produce a single successful image that consistently fools the discriminator, rather than generating a diverse set of images.
• Sensitivity to hyperparameters: GANs often fail to converge, but a sudden adjustment in hyperparameters, such as the learning rate, can lead to convergence.
• Maintaining a balance between the generator and discriminator: In many cases, the discriminator's loss can drop to zero quickly, making it impossible for the generator to improve further. To address this, you can try using different learning rates for the generator and discriminator or skip discriminator training when its loss is already very low.
• Training for high resolution: Similar to the challenges faced by autoencoders, reconstructing high-resolution images can lead to artifacts due to the complexity of convolutional layers. This issue is often addressed through progressive growing, where initial layers are trained on low-resolution images, and additional layers are gradually "unblocked" or added. Another approach involves adding extra connections between layers and training multiple resolutions simultaneously—see this Multi-Scale Gradient GANs paper for more details.

Style Transfer

GANs are excellent for generating artistic images. Another fascinating technique is style transfer, which takes one content image and redraws it in a different style using filters derived from a style image.

Here’s how it works:

• Start with a random noise image (or a content image, but for simplicity, we'll use random noise).
• The goal is to create an image that resembles both the content image and the style image. This is achieved using two loss functions:
  • Content loss: Calculated based on features extracted by the CNN at certain layers from the current image and the content image.
  • Style loss: Computed between the current image and the style image using Gram matrices (more details in the example notebook).
• To smooth the image and reduce noise, introduce Variation loss, which calculates the average distance between neighboring pixels.
• The main optimization loop adjusts the current image using gradient descent (or another optimization algorithm) to minimize the total loss, which is a weighted sum of all three losses.

✍️ Example: Style Transfer

Post-lecture quiz

Conclusion

In this lesson, you learned about GANs and their training process. You also explored the unique challenges associated with this type of neural network and strategies to address them.

🚀 Challenge

Try running the Style Transfer notebook using your own images.

Review & Self Study

For further reading, explore these resources on GANs:

• Marco Pasini, 10 Lessons I Learned Training GANs for one Year
• StyleGAN, a widely-used GAN architecture
• Creating Generative Art using GANs on Azure ML

Assignment

Revisit one of the two notebooks associated with this lesson and retrain the GAN using your own images. What can you create?

---