Image Basics

This tutorial covers reading and writing images in Python, indexing into an image as a matrix, kernels and convolution, a few basic filters including Sobel gradients and the Laplacian, and resizing, rotation, brightness and perspective changes. It goes with the Image Retrieval 1 lecture (decks 2 and 3).

All the code is in one file, image_basics.py. Download it and the sample image, astronaut.png (a public-domain NASA photo), then run it:

$ pip install numpy pillow
$ python3 image_basics.py astronaut.png
shape: (512, 512) dtype: float32
wrote gray, box, gaussian, sharpen, sobel_x, sobel_y, sobel_mag, laplacian, log
wrote small_slice, small_lanczos, flip, rotate_nearest, rotate_pil, brighter, contrast, gamma, perspective

Image I/O

An image file is a compressed encoding (PNG, JPEG, ...). To work with the pixels, decode it with Pillow and turn it into a NumPy array:

import numpy as np
from PIL import Image

rgb = np.asarray(Image.open("astronaut.png"))
print(rgb.shape, rgb.dtype)   # (512, 512, 3) uint8

A color image is a 3-D array of shape (height, width, 3): one 8-bit value from 0 to 255 for each of red, green and blue. Most of the filters in this course work on a single grayscale channel, so convert first. Use float32 so arithmetic doesn't overflow:

def load_gray(path):
    """Read an image file and return a 2-D float32 array in [0, 255]."""
    return np.asarray(Image.open(path).convert("L"), dtype=np.float32)

To write an array back out, clip it to the 0–255 range and convert back to uint8:

def save(arr, path):
    """Clip an array to [0, 255] and write it as an 8-bit image."""
    Image.fromarray(np.clip(arr, 0, 255).astype(np.uint8)).save(path)

Pitfall: uint8 arithmetic wraps around silently. np.array([200], dtype=np.uint8) + 100 gives [44], not 300. Convert to float32 before filtering, and clip only when you save.

Indexing an image matrix

The origin is the top-left corner. The first index is the row (y, increasing downward) and the second is the column (x, increasing to the right). So im[i, j] is the pixel i rows down and j columns across, which is the reverse of the usual (x, y) order.

rgb[0, 0]        # top-left pixel: array([154, 147, 151], dtype=uint8)
rgb[0, 0, 0]     # its red value: 154

im = load_gray("astronaut.png")
im.shape         # (512, 512): (height, width)
im[100:105, 200:205]   # a 5 x 5 patch: rows 100-104, columns 200-204
[[ 60  43 130 154  32]
 [ 61  61  93  91  27]
 [ 72  61  45  37  38]
 [126  84  79  66  70]
 [157 157 166 141 120]]

Slicing returns a view, not a copy, so assigning to a slice changes the original image:

crop = im[:256, 256:]      # top-right quarter
im[:50, :50] = 0           # paint the top-left 50 x 50 block black
patch = im[10:20, 10:20].copy()   # use .copy() when you need an independent array

Kernels and convolution

A kernel is a small matrix of weights, usually 3×3 or 5×5 and always odd-sized so it has a center. Applying a kernel means sliding it over every pixel: each output pixel is the weighted sum of the input pixels under the kernel, with the kernel's center k[0, 0] on the pixel being computed.

out[i, j] = Σu Σv  k[u, v] · im[i + u, j + v]      for u = −a … a,  v = −b … b

for a (2a+1)×(2b+1) kernel. Strictly, this is cross-correlation; true convolution flips the kernel first. The two agree for symmetric kernels like blurs, and computer-vision libraries usually mean this formula when they say "filter".

Pixels near the border need neighbors that don't exist. A common fix is to pad the image by repeating its edge pixels (mode="edge"). Here is the formula written out directly:

def correlate_slow(im, k):
    """Apply kernel k to every pixel of im, one pixel at a time."""
    a, b = k.shape[0] // 2, k.shape[1] // 2
    padded = np.pad(im, ((a, a), (b, b)), mode="edge")
    out = np.zeros_like(im)
    for i in range(im.shape[0]):
        for j in range(im.shape[1]):
            total = 0.0
            for u in range(-a, a + 1):
                for v in range(-b, b + 1):
                    total += k[u + a, v + b] * padded[i + a + u, j + b + v]
            out[i, j] = total
    return out

That's four nested Python loops: about 2 seconds for one 512×512 image, which adds up fast across many images. Swap the loop order: for each kernel entry, shift the whole padded image by (u, v), multiply by the weight and add. Now NumPy does the per-pixel work and Python only loops over the 9 or 25 kernel entries. This version takes about 5 milliseconds, roughly 400× faster:

def correlate(im, k):
    """Same result as correlate_slow, but loops over the kernel, not the image."""
    a, b = k.shape[0] // 2, k.shape[1] // 2
    padded = np.pad(im, ((a, a), (b, b)), mode="edge")
    height, width = im.shape
    out = np.zeros_like(im)
    for u in range(-a, a + 1):
        for v in range(-b, b + 1):
            out += k[u + a, v + b] * padded[a + u:a + u + height,
                                            b + v:b + v + width]
    return out

Basic filters

Blurring: box and Gaussian

A box filter replaces each pixel with the plain average of its neighborhood. A Gaussian filter takes a weighted average that favors nearby pixels, which blurs more smoothly. Both kernels sum to 1, so overall brightness is unchanged.

BOX = np.ones((5, 5), dtype=np.float32) / 25

def gaussian_kernel(size, sigma):
    """Return a size x size Gaussian kernel whose entries sum to 1."""
    r = size // 2
    x = np.arange(-r, r + 1, dtype=np.float32)
    g = np.exp(-x**2 / (2 * sigma**2))
    k = np.outer(g, g)
    return k / k.sum()

blurred = correlate(im, gaussian_kernel(9, 2.0))

np.outer(g, g) builds the 2-D kernel from a 1-D one. A Gaussian is separable: blurring down and then across gives the same result as one 2-D pass.

Sharpening

The sharpen kernel adds to each pixel the difference between it and its four neighbors, exaggerating local contrast. Its weights also sum to 1.

0−10
−15−1
0−10
Grayscale astronaut photo
Grayscale input
Box-blurred photo
5×5 box blur
Gaussian-blurred photo
9×9 Gaussian, σ = 2
Sharpened photo
Sharpen

Sobel gradients

An edge is where brightness changes quickly, so finding edges means estimating the image gradient. A plain central difference [−1, 0, 1] responds to noise as much as to edges, so Sobel combines it with smoothing: smooth down with [1, 2, 1] and take the difference across.

−101
−202
−101
−1−2−1
000
121

SOBEL_X (left) and SOBEL_Y (right, its transpose).

SOBEL_X = np.array([[-1, 0, 1],
                    [-2, 0, 2],
                    [-1, 0, 1]], dtype=np.float32)
SOBEL_Y = SOBEL_X.T

gx = correlate(im, SOBEL_X)    # positive where it gets brighter to the right
gy = correlate(im, SOBEL_Y)    # positive where it gets brighter downward
magnitude = np.hypot(gx, gy)   # sqrt(gx**2 + gy**2): edge strength
direction = np.arctan2(gy, gx) # edge orientation, in radians

gx and gy can be negative, so to view them, shift zero to mid-gray (128 + gx / 8). gx picks out vertical edges and gy horizontal ones. The magnitude combines the two into one edge map.

Horizontal Sobel response
gx
Vertical Sobel response
gy
Sobel gradient magnitude
Magnitude

Gradients are the building block for the rest of the lecture: Harris corners are built from sums of gx and gy products, and gradient histograms bin direction weighted by magnitude.

Laplacian

Sobel estimates the first derivative: how fast brightness is changing. The Laplacian estimates the second derivative, summed over both directions: how fast that change is itself changing. Its kernel compares each pixel with its four neighbors:

010
1−41
010
LAPLACIAN = np.array([[0, 1, 0],
                      [1, -4, 1],
                      [0, 1, 0]], dtype=np.float32)

lap = correlate(im, LAPLACIAN)

The weights sum to 0, so flat regions and smooth ramps give 0. Across an edge, the response swings positive on one side and negative on the other, and the edge itself sits at the zero crossing between them. Unlike Sobel, it is a single kernel with no direction: it treats edges at every angle the same way.

The sharpen kernel above is just identity − LAPLACIAN: sharpening subtracts the Laplacian from the image, which pushes each side of an edge further apart.

Second derivatives amplify noise even more than first derivatives, so in practice you smooth first. Blurring with a Gaussian and then applying the Laplacian is called the Laplacian of Gaussian (LoG). The larger sigma is, the coarser the structures it responds to, which makes LoG a classic detector for blobs of a given size.

log = correlate(correlate(im, gaussian_kernel(9, 2.0)), LAPLACIAN)
Laplacian response
Laplacian (mid-gray = 0)
Laplacian of Gaussian response
Laplacian of Gaussian, σ = 2

The plain Laplacian picks up fine texture and noise everywhere; the LoG version keeps only the larger outlines.

Resizing and transformations

Retrieval has to cope with the same scene photographed at a different size, angle or exposure. Each of those is a transformation you can apply yourself, which is also how you test whether a detector or descriptor is robust to it. These examples use Pillow on the grayscale image, pil = Image.fromarray(im.astype(np.uint8)), alongside NumPy versions where they're short.

Resizing

Shrinking an image by keeping every 4th pixel is one slice, im[::4, ::4], but it throws away the 15 pixels in between instead of averaging them. Fine detail that repeats faster than the new pixel spacing turns into jagged or speckled patterns. This is called aliasing. The fix is to blur first, for example with a Gaussian, and then sample. A good resampling filter like Lanczos does both in one step:

small = im[::4, ::4]                                   # fast, but aliases
small = pil.resize((128, 128), Image.Resampling.LANCZOS)   # smooths, then samples
big = pil.resize((1024, 1024), Image.Resampling.BICUBIC)   # enlarging: interpolate

The size argument is (width, height), the reverse of NumPy's (height, width) shape. When enlarging, NEAREST repeats pixels as blocks, while BILINEAR and BICUBIC interpolate between them.

Enlarged corner of the image shrunk by slicing, with jagged stars
im[::4, ::4], top-left corner enlarged 6×
Enlarged corner of the image shrunk with Lanczos, smoother
Lanczos, same region

Flipping and rotating

Flips and quarter turns only move pixels, so they are exact: np.fliplr(im), np.flipud(im) and np.rot90(im). Any other angle lands output pixels between input pixels, which needs inverse mapping: loop over the output pixels and ask which input location each came from. Going the other way, pushing each input pixel forward, leaves holes in the output.

def rotate_nearest(im, degrees):
    """Rotate counterclockwise about the center, by inverse mapping:
    for each output pixel, look up the input pixel it came from."""
    height, width = im.shape
    cy, cx = (height - 1) / 2, (width - 1) / 2
    theta = np.deg2rad(degrees)
    rows, cols = np.indices(im.shape)
    y, x = rows - cy, cols - cx                   # output, relative to center
    src_x = np.cos(theta) * x - np.sin(theta) * y + cx
    src_y = np.sin(theta) * x + np.cos(theta) * y + cy
    src_i, src_j = np.round(src_y).astype(int), np.round(src_x).astype(int)
    inside = (src_i >= 0) & (src_i < height) & (src_j >= 0) & (src_j < width)
    out = np.zeros_like(im)                       # black where nothing maps
    out[inside] = im[src_i[inside], src_j[inside]]
    return out

Rounding to the nearest input pixel gives the same result as pil.rotate(30, resample=Image.Resampling.NEAREST). BILINEAR instead blends the four input pixels around each source location, which avoids jagged edges. Corners that map from outside the input come out black. Pass expand=True to enlarge the canvas so nothing is cut off.

Photo flipped left to right
np.fliplr(im)
Photo rotated 30 degrees counterclockwise
rotate_nearest(im, 30)
Photo rotated 30 degrees with bilinear interpolation
pil.rotate(30, BILINEAR)

Brightness, contrast and gamma

These change pixel values but not positions, so they are just arithmetic on the array:

def adjust(im, contrast=1.0, brightness=0.0, gamma=1.0):
    """Scale contrast around mid-gray, add brightness, then apply gamma."""
    out = np.clip(contrast * (im - 128) + 128 + brightness, 0, 255)
    return 255 * (out / 255) ** gamma

Clipping at 0 and 255 loses information: two pixels that both clip to 255 can no longer be told apart. Brightness and contrast changes keep the order of pixel values, except where clipping happens. That is why BRIEF, which only compares pairs of pixels, isn't affected by them.

Brighter photo
brightness=60
Higher-contrast photo
contrast=1.6
Photo with lifted shadows
gamma=0.5

Perspective

Photographing a flat surface from an angle makes a rectangle look like a general four-sided shape. Parallel lines meet, and far parts shrink. The map from output (x, y) to input (X, Y) is a perspective transform (a homography) with 8 unknowns:

X = (a·x + b·y + c) / (g·x + h·y + 1)        Y = (d·x + e·y + f) / (g·x + h·y + 1)

Each point correspondence gives two linear equations, so 4 points pin down all 8 unknowns. Multiply both sides by the denominator and move the unknowns to one side:

def perspective_coeffs(dst, src):
    """Solve for the 8 coefficients (a..h) of the perspective map taking
    each output point (x, y) in dst to its input point (X, Y) in src:
    X = (a x + b y + c) / (g x + h y + 1),  Y = (d x + e y + f) / (g x + h y + 1)."""
    rows, rhs = [], []
    for (x, y), (big_x, big_y) in zip(dst, src):
        rows.append([x, y, 1, 0, 0, 0, -big_x * x, -big_x * y])
        rows.append([0, 0, 0, x, y, 1, -big_y * x, -big_y * y])
        rhs += [big_x, big_y]
    return np.linalg.solve(np.array(rows, dtype=float), np.array(rhs, dtype=float))


corners = [(0, 0), (511, 0), (511, 511), (0, 511)]   # input corners, (x, y)
tilted = [(80, 60), (511, 0), (511, 511), (80, 451)] # where they should land
coeffs = perspective_coeffs(tilted, corners)
warped = pil.transform(pil.size, Image.Transform.PERSPECTIVE, coeffs,
                       Image.Resampling.BILINEAR)

Pillow, like rotate_nearest, maps each output point back to the input, so the coefficients go from tilted to corners. Squeezing the left edge makes the photo look turned away from the camera. With g = h = 0 the denominator is 1 and the map is affine, which covers scaling, rotation, shear and translation. OpenCV's cv2.getPerspectiveTransform solves the same system. Estimating these coefficients from matched keypoints is the core of geometric verification in the next lecture.

Photo warped so its left edge is shorter, as if turned away
Left edge squeezed from 512 to 392 pixels

Using a library

Write it yourself once to understand it; in real code, use a library. These give the same results as correlate above:

from scipy import ndimage
out = ndimage.correlate(im, SOBEL_X, mode="nearest")

import cv2   # pip install opencv-python
out = cv2.filter2D(im, -1, SOBEL_X, borderType=cv2.BORDER_REPLICATE)
gx = cv2.Sobel(im, cv2.CV_32F, 1, 0, ksize=3, borderType=cv2.BORDER_REPLICATE)
blurred = cv2.GaussianBlur(im, (9, 9), 2.0, borderType=cv2.BORDER_REPLICATE)
lap = cv2.Laplacian(im, cv2.CV_32F, ksize=1, borderType=cv2.BORDER_REPLICATE)
lap = ndimage.laplace(im, mode="nearest")

The border options make the edge pixels match our mode="edge" padding; library defaults differ, which only changes a 1–4 pixel band around the border. Also note that OpenCV's cv2.imread returns color channels in BGR order, not RGB.

Try it

  1. Blur with Gaussians of increasing sigma before running Sobel. What happens to the fine edges in the hair and the flag?
  2. Check that a 9×9 Gaussian applied as one 2-D kernel matches a 9×1 pass followed by a 1×9 pass. Which is faster, and why?
  3. Find the zero crossings of the LoG image: pixels where the sign differs from the pixel to the right or below. Compare the result with the Sobel magnitude.
  4. Run Sobel on small_slice and on small_lanczos. Which edge map looks more like a shrunken version of the full-size one?
  5. Threshold magnitude to get a black-and-white edge map. How does the threshold you choose change the result?