Chapter 5 - Image Filtering
Slides 1 - 63
In this chapter, we’re introducing the concept of Image Filtering. Filters can be applied on 2D Image data either for various applications. We can broadly differenciate low-pass filters smooth images (retrain low-frequenciy components) and high-pass filters (retain contours / edges, e.g. high frequencies).
Low-pass filters
Low pass filters are typically applied to reduce noise in images. Noise can be seen as random artifacts in an image. For example, salt & pepper noise describes the random occurrence of black / white pixels in the image, while gaussian noise is a random increase/decrease in each pixel’s color value, following a gaussian distribution.
Figure 1: Different noise types. source
Low-pass filters assume that the pixel variation in the image should be lower than perceived, e.g. pixels should have a color value close to their neighbours. Low-pass filters therefore replace each pixels value with an average of the values in the pixels neighbourhood. The neighbours can either be weighted based on their distance to the center pixel, or equally.
Moving a filter over all possible positions in an image is called a convolution, the filter is called a Kernel or Mask and denoted H. When convoluting a filter over an image, we flip the kernel by 180° before performing at each position before computing the weighted average between the filter values and the pixel. If we do not flip the kernel, we speak of a cross-correlation instead of a convolution. For symmetric filters like “Gaussian Filter” or “Median Filter”, a convolution and a cross-correlation will of course produce the same results.
Figure 2: Convolution vs. Cross-corelation formula. source
In the following example, a smoothing averaging filter is applied to a 1D signal, bringing the neighbouring values significantly closer together and reducing outliers.
Figure 3: World Coordinates -> Pixel Coordinates. source
The same principle can be applied to 2D Data like images. In the following example, a 2D Filter with size 1/1 averages a neighbourhood of 9 pixels, overwriting the central pixel with an equally weighted average of all 9 pixels. Such a filter is called a “box” filter.
Figure 4: Illustration of a box filter over a black image with a white square. source
The output is - of course - a blurred version of the very same image.
Figure 5: Output of a smoothed image using a box-filter. source
While the box filter smoothens an image quite well, it produces horizontal and vertical aftifacts since the filter itself has sharp edges. These artifacts are also called “aliasing” and is caused by the high frequency components of the box filter. A better way to smooth an image is with a gaussian filter, a filter implementing the 2D gaussian function. For perfect results, take a large gaussian filters with smooth edges, e.g. low standard derivation that ensures that the outermost values of the filter are close to 1 while preserving a smooth derivative.
Figure 6: Gaussian filter visualization. source
Figure 7: Gaussian Filter comparison. source
When we apply any filter on an image, the question remains how to deal with the image boundary. Since - in most cases - we don’t want the resulting image to be smaller than the input image, we have to simulate additional boundary pixels. There are different strategies with varying results, like zero-padding (surrounding black pixels), wrap-around (repeating the image), copy-edge (always use outermost pixel values) or reflect accross edge (mirroring around edge, gives best results).
Non-linear Low-pass filters
Gaussian filters or box-filters do not denoise salt & pepper noise since they get influenced by outliers by a high degree. That’s where median filters come into play. They can not be interpreted as a classical convolution filter like a Gaussian filter, it rather takes the median pixel value from the neighbourhood. The median filter is therefore much less influenced by strong noise, while he also preserves edges much better than the linear smoothing filters.
Figure 8: Median Filter removing Salt & Pepper Noise. source
Another such filter is the billateral filter. It acts like a median filter with preserving edges even more by adapting the kernel locally to the intensitiy profile of the underlaying image. They only average pixels with similar brightness: Pixels that fall below a brightness difference compared to the center pixel.
Figure 9: Bilateral filer with mask. source
The extend to which neighbouring pixels have to be similar to the central pixel is controlled via a factor sigma.
High-pass filters
High-pass filters are mainly used for edge detection since react to sharp change in pixel intensity. Edges are sharp changes in an image functions intensity value. Applying the first derivative on an image would leave us with an image where sharp edges are shown.
Figure 10: Image derivative detecting edges. source
We therefore construct a filter that acts like a derivative by approximating the image derivative dI(x,y) / dx ~ I(x+1, y) - I(x,y) and dI(x,y) / dy ~ I(x, y+1) - I(x,y). So we essentially compare each pixel to its direct neighbour and take the difference as an output.
Figure 11: Partial derivative filter. source
More advanced filters are larger in size and therefore produce less artifacts. The sobel-filter is an example for a larger derivative filter:
Figure 12: Prewitt & Sobel filter. source
The direction of the edge can be determined by calculating the pixel regions gradient, so the diretion of fastest intensity change. The gradient direction is given by angle = arctan2(dI/dx, dI/dy), so the two dimensional arcus tangens of the image derivative values. The edge strenght is given by the gradients magnitude: *strength = sqrt((dI/dx)^2 + (dI/dy)^2).
A big problem for high-pass filters is gaussian noise: there will always be a steep difference between two neighbouring pixels, caused by normal gaussian noise produced by the image sensor. It is therefore best practice to softly filter the image first with a gaussian filter before applying a high-pass filter.
In the following graphic, we see the original image I, the kernel H, the resulting image when H is applied IH as well as the derrived image d(IH)/dx
Figure 13: Process steps for edge detection. source
A way better approach is to directly include the smoothing in the filter itself, giving us the filter dH/dx as seen in the following image:
Figure 13: Gaussian smoothing within a derivative filter. source
This is called a “derivative of gaussian” filter: it multiplies a normal gaussian filters with a high-pass 2x1 derivative filter.
Figure 14: Difference of Gaussians filter. source
Since we deal with two partial derivatives, we’d need to filter the image twice. A solution to this is given by the “Laplacian of Gaussian”-Filter, which finds the derivative in all directions simultaniously. It is constructed by subtracting a smaller radius gaussian Filter from a large radius gaussian filter.
Figure 15: Laplacian of Gaussian. source
Canny edge detection
Canny edge detection uses partial gaussian derivative filters to find all corners in an image. It then sets all pixelsvalues to 0 that fall under a given threshold. Finally, Canny takes the local maximum of any corner along the gradient direction, e.g. it only takes the peak of a wide edge.
Figure 16: Canny edge detection. source
Overview summary
Let me quickly illustrate the main differences of Smoothing and Derivaitve filters.
Smoothing filters always contain positive filter values that sum to 1 to preserve the overall brightness of constant regions. They are constructed to remove high-frequency components.
In contrast, derivative filters have two regions with opposite signs to get a high response in regions of high contrast. Their components sum to 1 to create no response on images with constant color. They are created to highlight high-frequency, not to remove them.