How to Extract the Foreground of an Image Using OpenCV Python?

OpenCV is a powerful library for computer vision and image processing. It offers a vast range of functions to manipulate and analyze different types of image data with ease. In this article, we’ll look at how to extract the foreground of an image using OpenCV in Python.

Image extraction is one of the fundamental components of image processing. It involves isolating the relevant part of the image from the background. The extracted part can be edited, analyzed, or used for further processing. Foreground extraction refers to the separation of the object of interest from the rest of the image.

The Concept of Foreground Extraction

The process of foreground extraction involves defining a mask or region of interest (ROI) that highlights the object of interest. The mask can be defined manually or using algorithms that automatically detect the object of interest. Once the mask is defined, it can be used to extract the foreground.

The Approach of Foreground Extraction in OpenCV

OpenCV Python provides a function called grabCut that is used for foreground extraction. The function takes an image and a mask as input and returns a new mask with the foreground pixels marked. The foreground pixels are represented by a label of 1, while the background pixels are represented by a label of 0.

To perform foreground extraction, we need to follow these steps:

1. Load the image
2. Define the ROI mask
3. Define the background and foreground model
4. Run the grabCut function and get the result
5. Create the final mask from the result
6. Apply the mask to the original image to get the foreground

Step-by-Step Guide to Foreground Extraction

Let’s look at each step of the foreground extraction process in detail.

Step 1: Load the Image

The first step is to load the image from the disk using the cv2.imread() function. The function takes the path of the image file as the input and returns the image data in the form of a NumPy array. Here’s the code:

import cv2
import numpy as np

# Load the image
img = cv2.imread('image.jpg')

Step 2: Define the ROI Mask

The next step is to define the ROI mask that highlights the object of interest. There are different ways of defining the mask. We can use a rectangular ROI, a polygon-shaped ROI, or a mask image. In this example, we’ll use a rectangular ROI that covers the object in the image.

# Define the ROI mask
rect = (50, 50, 200, 200)    # (x, y, w, h)
mask = np.zeros(img.shape[:2], np.uint8)
cv2.rectangle(mask, rect, 255, -1)

Here, we define a rectangle with coordinates (50, 50) and dimensions (200, 200) using the rect variable. We then create a blank image of the same size as the input image using the np.zeros() function. The cv2.rectangle() function is used to draw the rectangle on the mask with a thickness of -1, which fills the rectangle with white color.

Step 3: Define the Background and Foreground Model

The grabCut function requires an initial background and foreground model to start the segmentation process. We define the background and foreground models using the cv2.grabCut() function.

# Define the background and foreground model
bgdModel = np.zeros((1, 65), np.float64)
fgdModel = np.zeros((1, 65), np.float64)

# Run the grabCut function
masked_img = cv2.grabCut(img, mask, rect, bgdModel, fgdModel, 5, cv2.GC_INIT_WITH_RECT)

The bgdModel and fgdModel variables are used to store the Gaussian mixture models (GMMs) learned by the algorithm to represent the background and foreground. We initialize them as arrays of zeros with a shape of (1,65).

In the cv2.grabCut() function, we pass the input image (img), the ROI mask (mask), the rectangle defining the ROI (rect), the background and foreground models (bgdModel and fgdModel), and the number of iterations (5). The cv2.GC_INIT_WITH_RECT flag tells the function that we want to initialize the GMMs with the rectangular ROI.

The function returns a new mask with the foreground pixels labeled as 1, the background pixels labeled as 0, and the uncertain pixels labeled as 2.

Step 4: Run the grabCut function and Get the Result

Now that we have defined the models and providedthe input, we can run the cv2.grabCut() function to get the result. The function modifies the input mask and returns a new mask that separates the object from the background. Here’s the code:

# Run the grabCut function
masked_img = cv2.grabCut(img, mask, rect, bgdModel, fgdModel, 5, cv2.GC_INIT_WITH_RECT)

# Extract the result mask
mask2 = np.where((masked_img==2)|(masked_img==0), 0, 1).astype('uint8')

Here, we create a new mask called mask2 by thresholding the output of the cv2.grabCut() function. We use the np.where() function to set the uncertain and background pixels to 0 and the foreground pixels to 1. We also convert the mask to the uint8 data type for compatibility with other OpenCV functions.

Step 5: Create the Final Mask from the Result

The next step is to create the final mask that we can use to extract the foreground. We do this by multiplying the original mask with the result mask.

# Create the final mask
final_mask = mask2*255

Here, we multiply the mask2 with 255 to convert the values from 0 and 1 to 0 and 255. This creates a binary mask that highlights the object of interest.

Step 6: Apply the Mask to the Original Image to Get the Foreground

The final step is to apply the mask to the original image to get the foreground.

# Apply the mask to the original image
foreground = cv2.bitwise_and(img, img, mask=final_mask)

Here, we use the cv2.bitwise_and() function to apply the mask to the original image (img). The function sets all pixels outside the mask to 0 and retains the pixel values inside the mask to get the foreground.

Full Code Example

Here’s the full code example that puts all the above steps together:

import cv2
import numpy as np

# Load the image
img = cv2.imread('image.jpg')

# Define the ROI mask
rect = (50, 50, 200, 200)
mask = np.zeros(img.shape[:2], np.uint8)
cv2.rectangle(mask, rect, 255, -1)

# Define the background and foreground model
bgdModel = np.zeros((1, 65), np.float64)
fgdModel = np.zeros((1, 65), np.float64)

# Run the grabCut function
masked_img = cv2.grabCut(img, mask, rect, bgdModel, fgdModel, 5, cv2.GC_INIT_WITH_RECT)

# Extract the result mask
mask2 = np.where((masked_img==2)|(masked_img==0), 0, 1).astype('uint8')

# Create the final mask
final_mask = mask2*255

# Apply the mask to the original image
foreground = cv2.bitwise_and(img, img, mask=final_mask)

# Display the result
cv2.imshow('Original Image', img)
cv2.imshow('Foreground Mask', final_mask)
cv2.imshow('Foreground', foreground)
cv2.waitKey(0)
cv2.destroyAllWindows()

Conclusion

Extracting the foreground of an image is a critical component of image processing. OpenCV Python offers a straightforward approach to this task using the grabCut function. By defining a mask or region of interest, we can extract the foreground of an image and use it for further processing or analysis.

In this article, we learned how to extract the foreground of an image using OpenCV Python. We covered the concept of foreground extraction, the approach of foreground extraction in OpenCV, and provided a step-by-step guide to the process. We also shared a full code example that you can use to test the method yourself.