In computer vision tasks, we need to crop a rotated rectangle from the original image sometimes, for example, to crop a rotated text box. In this post, I would like to introduce how to do this in OpenCV.

If you search in the internet about cropping rotated rectangle, there are several answers in the Stack Overflow which suggest using minAreaRect() to find the minimum bounding rectangle, rotating the original image and finally cropping the rectangle from the image. You can find these questions here and here. While some of the answers work, they only work in certain conditions. But if the rotated rectangle is near the edge of the original image, some part of the cropped rectangle is cutted out in the output.

The imperfect way

Take the following image with rotated text as an example,

The corner points1 are (in top left, top right, bottom right, bottom left order):

[64, 49], [122, 11], [391, 326], [308, 373]

If you crop the rectangle using the following script (based on this answer):

import cv2
import numpy as np


def main():
    img = cv2.imread("big_vertical_text.jpg")
    cnt = np.array([
            [[64, 49]],
            [[122, 11]],
            [[391, 326]],
            [[308, 373]]
        ])
    # find the exact rectangle enclosing the text area
    # rect is a tuple consisting of 3 elements: the first element is the center
    # of the rectangle, the second element is the width, height, and the
    # third element is the detected rotation angle.
    # Example output: ((227.5, 187.50003051757812),
    # (94.57575225830078, 417.98736572265625), -36.982906341552734)
    rect = cv2.minAreaRect(cnt)
    print("rect: {}".format(rect))

    box = cv2.boxPoints(rect)
    box = np.int0(box)

    # print("bounding box: {}".format(box))
    cv2.drawContours(img, [box], 0, (0, 0, 255), 2)

    # img_crop will the cropped rectangle, img_rot is the rotated image
    img_crop, img_rot = crop_rect(img, rect)
    cv2.imwrite("cropped_img.jpg", img_crop)
    cv2.waitKey(0)


def crop_rect(img, rect):
    # get the parameter of the small rectangle
    center, size, angle = rect[0], rect[1], rect[2]
    center, size = tuple(map(int, center)), tuple(map(int, size))

    # get row and col num in img
    height, width = img.shape[0], img.shape[1]

    # calculate the rotation matrix
    M = cv2.getRotationMatrix2D(center, angle, 1)
    # rotate the original image
    img_rot = cv2.warpAffine(img, M, (width, height))

    # now rotated rectangle becomes vertical and we crop it
    img_crop = cv2.getRectSubPix(img_rot, size, center)

    return img_crop, img_rot


if __name__ == "__main__":
    main()

In the above code, we first find the rectangle enclosing the text area based on the four points we provide using the cv2.minAreaRect() method. Then in function crop_rect(), we calculate a rotation matrix and rotate the original image around the rectangle center to straighten the rotated rectangle. Finally, the rectangle text area is cropped from the rotated image using cv2.getRectSubPix method.

The orignal, rotated and cropped image are shown below:

We can see clearly that some parts of the text are cut out in the final result. Of course, you can padd the image before hand, and crop the rectangle2 from the padded image, which will prevent the cutting-out effect.

The better way

Is there a better way? Yes!

In the above code, when we want to draw the rectangle area in the image, we use cv2.boxPoints() method to get the four corner points of the real rectangle. We also know the width and height of rectanle from rect. Then we can directly warp the rectangle from the image using cv2.warpPerspective() function. The following scrit shows an example:

import cv2
import numpy as np


def main():
    img = cv2.imread("big_vertical_text.jpg")
    # points for test.jpg
    cnt = np.array([
            [[64, 49]],
            [[122, 11]],
            [[391, 326]],
            [[308, 373]]
        ])
    print("shape of cnt: {}".format(cnt.shape))
    rect = cv2.minAreaRect(cnt)
    print("rect: {}".format(rect))

    # the order of the box points: bottom left, top left, top right,
    # bottom right
    box = cv2.boxPoints(rect)
    box = np.int0(box)

    print("bounding box: {}".format(box))
    cv2.drawContours(img, [box], 0, (0, 0, 255), 2)

    # get width and height of the detected rectangle
    width = int(rect[1][0])
    height = int(rect[1][1])

    src_pts = box.astype("float32")
    # corrdinate of the points in box points after the rectangle has been
    # straightened
    dst_pts = np.array([[0, height-1],
                        [0, 0],
                        [width-1, 0],
                        [width-1, height-1]], dtype="float32")

    # the perspective transformation matrix
    M = cv2.getPerspectiveTransform(src_pts, dst_pts)

    # directly warp the rotated rectangle to get the straightened rectangle
    warped = cv2.warpPerspective(img, M, (width, height))

    # cv2.imwrite("crop_img.jpg", warped)
    cv2.waitKey(0)


if __name__ == "__main__":
    main()

Now the cropped rectangle becomes

If you check carefully, you will notice that there are some black area in the cropped image. That is because the a small part of the detected rectangle is out of the bound of the image. To remedy this, you may pad the image a little bit and do the crop after that. An example is given here (By me :)).

About the angle of the return rectangle

The last element of the returned rect is the detected angle of the rectangle. But it has confused a lot of people, for example, see here and here

This angle is in the range $[-90, 0)$. After much experiment, I have found that the relationship between the rectangle orientation and output angle of minAreaRect(). It can be summarized in the following image

The following description assume that we have a rectangle with unequal height and width length, i.e., it is not square.

If the rectangle lies vertically (width < height), then the detected angle is -90. If the rectangle lies horizontally, then the detected angle is also -90 degree.

If the top part of the rectangle is in first quadrant, then the detected angle decreases as the rectangle rotate from horizontal to vertical position, until the detected angle becomes -90 degrees. In first quadrant, the width of detected rectangle is longer than its height.

If the top part of the detected rectangle is in second quadrant, then the angle decreases as the rectangle rotate from vertical to horizontal position. But there is a difference between second and first quadrant. If the rectangle approaches vertical position but has not been in vertical position, its angle approaches 0. If the rectangle approaches horizontal position but has not been in horizontal position, its angle approaches -90 degrees.

This post here is also good in explaining this.

Conclusion

In this post, I compared the two methods to crop the rotated rectangle from the image and also explained the meaning of angle returned by cv2.minAreaRect() method. Overall, I like the second method since it does not require rotating the image and can deal with this problem more elegantly with less code.

References


  1. Note that we only need to provide roughly the 4 corner points of the rectangle. Then we can use OpenCV to find the exact the rectangle enclosing the rectangle text area.
  2. Note that you need to re-calculate the corner points of the rectangle after padding the original image.