Segment background using Computer Vision

Intro

Hello all! Did you check out my previous blog post on tracking stationary objects? It talks about how to subtract the background from an image and get the object in the foreground by absolute difference between two frames. Today we will see a different way to deal with backgrounds. Rather than calling this method a type of background subtraction, we'll call it a way of identify the background in the scene so that it can be segmented out of the frame.

That sounds interesting, doesn't it?

"But", you may ask, "why are we doing this? What is so special about it?" Well, let me give you a glimpse of the importance of background segmentation. In computer vision, we process images, so that we can observe the behavior of the objects in the image. It could be anything from face recognition to traffic patterns to security. Every image we take to process is made up of background and foreground. Most of the time it is very useful to separate out this background in order to concentrate on the foreground. For example, the background could be a traffic signal while the foreground could be the vehicles passing by. We can come up with tons of examples where getting rid of the background could be really helpful in observing the patterns in the foreground.

In many cases we get an image of a stationary background which can be used for subtraction or segmentation from other frames of the same scene. But we do not always get lucky. Sometimes the background scene is moving or there are shadows all over. In those cases maybe taking an absolute difference between two frames will not be sufficient. So it's not always so simple, is it? That's why we are going to look at new way of identifying the background of the scene.

Method

OpenCV has few implementations of Background Segmentation. We will be looking at one of those. The class is called BackgroundSubtractorMOG2. It is a Gaussian Mixture-based Background Segmentation Algorithm. This algorithm takes the background pixels and assigns a Gaussian Distribution to each one. The weight of this distribution is the amount of time the colors stay in the scene. Basically, the algorithm tries to identify the background by the information from the Gaussian mixture. The idea is that the longer the color stays the higher the probability that it is a part of the background. The gaussian distribution helps this method to adapt to variance in illumination. This class supports parallel computing.

You can use this method in Python like so:

cv2.createBackgroundSubtractorMOG2(
    int history = 500,
    double  varThreshold = 16,
    bool    detectShadows = true
)

Parameters are:

• history : Length of history. (Default=500)

• varThreshold : This value determines whether a pixel is well described by its background model. (Default=16)

• detectShadows: Determines whether shadows are given importance in the scene. Detects shadows.(Default=True)

createBackgroundSubtractorMOG2 creates a Background object of class BackgroundSubtractorMOG2. When applied to the first image provided to this object, it starts creating a background model. As frames are fed to this object it continues updating the background. The varThreshold parameter helps set the threshold at which a pixel is determined to be in the background. The value does not affect the background model update. This value is a distance threshold between the pixel and background model. The higher the threshold, the more certainty is required. We use it to indicate that we just want those pixels which are well described by the background model.

This class works very well for video capture of scenes. As an example we will take 3 separate frames and look at the output of applying BackgroundSubtractorMOG2.

Basic Example

background_object = cv2.createBackgroundSubtractorMOG2()

image1 = cv2.imread('image1.JPG')
foreground_mask1 = background_object.apply(image1)

image2 = cv2.imread('image2.JPG')
foreground_mask2 = background_object.apply(image2)

image3 = cv2.imread('images3.JPG')
forground_mask3 = background_object.apply(image3)

We first create an object of class BackgroundSubtractorMOG2. We then read images and then apply this our BackgroundSubtractorMOG2 instance to the image. This segments the background in the image based on the background model so far. Once that is done, it then updates the background model to have a stronger model on which to determin which pixels belong to background. Here we are using the default parameters.

Here are each of the original images with the images after background segmentation to obtain the foreground.

Set of original images:

Set of background segmented images using default parameters:

In the above processed images we see that:

• Foreground image 1: Since it is the first fed image, the background model is blank, hence it has detected the whole image as foreground. Grey scale determines any new pixel introduced.

• Foreground image 2: Now we have a background model formed with first image. Hence it detects a new object, here the yellow cube as foreground.

• Foreground image 3: Now the background model is updated with yellow object in background. Hence detects new foreground, here the red cube.

Let's experiment by playing with the parameters, history, varThreshold and detectShadows.

history

background_object = cv2.createBackgroundSubtractorMOG2(history=1)

Here I've changed the history factor from it's default,500, to1. It shows that only the previous frame, i.e. 1 frame affects the current frame. We can clearly see a difference in the background segmentation in the third frame as compared to the third frame of the basic example. This factor greatly helps when the motion capture has several frames:

varThreshold

background_object = cv2.createBackgroundSubtractorMOG2(varThreshold=1000)

Here I provide a varThreshold to determine how well a pixel is defined by the background model. By setting a high value the algorithem will only detect new pixels which are extremely different from the background model. This helps us detect newly introduced objects at the cost of missing any objects that are similar to the background.

detectShadows

background_object = cv2.createBackgroundSubtractorMOG2(detectShadows=0)

Here I've disabled shadow detection via the detectShadows parameter. This makes the resultant detection array binary, instead of scalars. Why so? The scalars represent the varying contrast in the image. The shadows are marked in gray scale depending upon how strong they are. It decreases the speed of computation, but is helpful when separating the objects from the shadows. Here, since the default value of varThreshold is 500, we don't see a varying contrast for shadows when detectShadows = true. But we see that when the shadows are not detected, the foreground just says whether a pixel belongs to background or foreground, hence binary image. Note that the first blank space is the white image with the foreground detected:

All together

background_object = cv2.createBackgroundSubtractorMOG2(history=2, varThreshold=1000, detectShadows=0)

Here's a composite of each of the changes above in the same set of images. Well, we can say that history = 2makes a background model with last two frames, varThreshold = 1000applies a high threshold on the segmented image and detectShadows = 0segments the background without giving importance to shadows, hence giving us a very decent foreground detection by segmenting the background. That's pretty neat, isn't it? Note that the first blank space is the white image with foreground detected:

Conclusion

Isn't this cool? You can play with the parameters and see how it works for your images. This will help you in discarding the background so that you can work on the foreground to analyze its pattern.

I hope you found this useful. Try it out, play with other types of preprocessing to make it more effective. I'd glad to hear how it works for you. Please share any tips to make this better.