8.8 KiB
Introduction to Computer Vision
Computer Vision is a field aimed at enabling computers to achieve a high-level understanding of digital images. This definition is quite broad because understanding can encompass various tasks, such as identifying objects in an image (object detection), interpreting events (event detection), describing an image in text, or reconstructing a scene in 3D. There are also specialized tasks related to human images, such as estimating age and emotions, detecting and identifying faces, and estimating 3D poses, among others.
Pre-lecture quiz
One of the simplest tasks in computer vision is image classification.
Computer vision is often considered a branch of AI. Today, most computer vision tasks are solved using neural networks. In this section, we will explore a specific type of neural network used for computer vision, convolutional neural networks.
However, before passing an image to a neural network, it is often beneficial to use algorithmic techniques to enhance the image.
Several Python libraries are available for image processing:
- imageio can be used for reading and writing various image formats. It also supports ffmpeg, a useful tool for converting video frames into images.
- Pillow (also known as PIL) is more versatile and supports image manipulation tasks such as morphing, palette adjustments, and more.
- OpenCV is a powerful image processing library written in C++ and is considered the de facto standard for image processing. It has a convenient Python interface.
- dlib is a C++ library that implements many machine learning algorithms, including some for computer vision. It also has a Python interface and can be used for complex tasks like face and facial landmark detection.
OpenCV
OpenCV is widely regarded as the de facto standard for image processing. It includes numerous useful algorithms implemented in C++. OpenCV can also be accessed from Python.
A great resource for learning OpenCV is this Learn OpenCV course. In this curriculum, our goal is not to master OpenCV but to demonstrate some examples of its use and functionality.
Loading Images
Images in Python can be conveniently represented as NumPy arrays. For instance, grayscale images with dimensions of 320x200 pixels would be stored in a 200x320 array, while color images of the same size would have a shape of 200x320x3 (for 3 color channels). To load an image, you can use the following code:
import cv2
import matplotlib.pyplot as plt
im = cv2.imread('image.jpeg')
plt.imshow(im)
Traditionally, OpenCV uses BGR (Blue-Green-Red) encoding for color images, whereas other Python tools use the more common RGB (Red-Green-Blue) format. To ensure the image appears correctly, you need to convert it to the RGB color space, either by swapping dimensions in the NumPy array or by using an OpenCV function:
im = cv2.cvtColor(im,cv2.COLOR_BGR2RGB)
The same cvtColor function can be used for other color space transformations, such as converting an image to grayscale or to the HSV (Hue-Saturation-Value) color space.
OpenCV can also be used to load video frame-by-frame. An example is provided in the exercise OpenCV Notebook.
Image Processing
Before feeding an image to a neural network, you may want to apply several preprocessing steps. OpenCV offers a variety of functionalities, including:
- Resizing an image using
im = cv2.resize(im, (320,200),interpolation=cv2.INTER_LANCZOS) - Blurring an image using
im = cv2.medianBlur(im,3)orim = cv2.GaussianBlur(im, (3,3), 0) - Adjusting the brightness and contrast of an image through NumPy array manipulations, as described in this Stackoverflow note.
- Using thresholding by calling
cv2.threshold/cv2.adaptiveThresholdfunctions, which is often preferable to adjusting brightness or contrast. - Applying various transformations to the image:
- Affine transformations are useful for combining rotation, resizing, and skewing when you know the source and destination locations of three points in the image. Affine transformations preserve parallel lines.
- Perspective transformations are helpful when you know the source and destination positions of four points in the image. For example, if you take a picture of a rectangular document with a smartphone camera at an angle and want to create a rectangular image of the document itself.
- Understanding movement within an image using optical flow.
Examples of using Computer Vision
In our OpenCV Notebook, we provide examples of how computer vision can be applied to specific tasks:
- Preprocessing a photograph of a Braille book. This example demonstrates how thresholding, feature detection, perspective transformation, and NumPy manipulations can be used to isolate individual Braille symbols for further classification by a neural network.
 |
 |
 |
|---|
Image from OpenCV.ipynb
- Detecting motion in video using frame difference. If the camera is stationary, frames from the camera feed should be quite similar. Since frames are represented as arrays, subtracting the arrays of two consecutive frames will reveal pixel differences. These differences will be minimal for static frames and more pronounced when there is significant motion in the image.
Image from OpenCV.ipynb
-
Detecting motion using Optical Flow. Optical flow helps track how individual pixels move across video frames. There are two types of optical flow:
- Dense Optical Flow calculates a vector field showing the movement of each pixel.
- Sparse Optical Flow focuses on distinctive features in the image (e.g., edges) and tracks their movement across frames.
Image from OpenCV.ipynb
✍️ Example Notebooks: OpenCV try OpenCV in Action
Let's experiment with OpenCV by exploring OpenCV Notebook.
Conclusion
Sometimes, relatively complex tasks like motion detection or fingertip detection can be solved entirely through computer vision techniques. Therefore, understanding the basics of computer vision and the capabilities of libraries like OpenCV is highly beneficial.
🚀 Challenge
Watch this video from the AI show to learn about the Cortic Tigers project and how they developed a block-based solution to make computer vision tasks accessible via a robot. Research other similar projects that help introduce new learners to the field.
Post-lecture quiz
Review & Self Study
Learn more about optical flow in this excellent tutorial.
Assignment
In this lab, you will record a video with simple gestures, and your task will be to extract up/down/left/right movements using optical flow.
