python将图片合成剪映

 

一. 概述

先看一张来自http://www.vintagephoto.tv/mb.shtml 的图片:
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-j1JuaB3X-1583201443765)(http://www.vintagephoto.tv/images/chapel.jpg)]
以前没有彩色照片的时代, 使用不同的filters拍三张RGB各个通道的图.
现在使用数字图像处理技术可以重建出古老的1909年的彩照!
很有趣的一项技术, 虽然现在已经用不到了.

但是, 拍摄3张照片要2~6秒. 拍静物还好, 要是拍人就重建出伪影了:
下图中, 左边是直接重建的, 右边是使用Image Alignment技术重建的:
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-zgPHGqEc-1583201443766)(https://www.learnopencv.com/wp-content/uploads/2015/07/image-alignment-example-opencv-1024x545.jpg)]
来自: https://www.learnopencv.com/image-alignment-ecc-in-opencv-c-python/
是不是很神奇!
看到了100年前的俄罗斯大叔彩照!

那么, 下面我们就研究一下怎么使用OpenCV实现Image Alignment.

二. Motion Models

大叔的三张照片相对位置有偏移(Motion), 我们可以对这些Motion建模.

1. Translation ( MOTION_TRANSLATION ) . 平移, 大家都知道. 只要x和y两个参数.
2. Euclidean ( MOTION_EUCLIDEAN ). 欧几里得变换(rigid transform, 刚体变换). 有平移和旋转 .需要x, y和旋转角度angle三个参数. 注意正方形大小形状都没有变.
3. Affine ( MOTION_AFFINE ) . 仿射变换. 有平移, 旋转, 缩放和切变(shear). 需要6个参数. 平行线仍然是平行线. 但是相交直线的角度会变化. 平移和欧几里得变换可以看成Affine的特例.
   OpenCV文档中公式:
   
   有了变换矩阵, warpAffine函数可以实现图像的Affine变换.
4. Homography ( MOTION_HOMOGRAPHY ) . 单应矩阵变换. 有部分3D变换的效果. 需要8个参数(存为3*3矩阵). 正方形可以变为任意的四边形. 有了变换矩阵, warpPerspective函数可以实现图像的Homography变换.

三．ＥＣＣ算法

文献：Parametric Image Alignment using Enhanced Correlation Coefficient Maximization
http://xanthippi.ceid.upatras.gr/people/evangelidis/george_files/PAMI_2008.pdf ECC的优点:

1. 对图像对比度和亮度变化适应性强.
2. 虽然目标函数是非线性的, 但是有简单的迭代解法.

四．简单的例子

先看一个简单的两张图片对齐的例子:

/**
 * OpenCV Image Alignment  Example
 *
 * Copyright 2015 by Satya Mallick <[e-mail]>
 *
 */

#include "opencv2/opencv.hpp"

using namespace cv;
using namespace std;

int main(void)
{
    // Read the images to be aligned
    Mat im1 = imread("image1.jpg",CV_LOAD_IMAGE_COLOR);
    Mat im2 = imread("image2.jpg",CV_LOAD_IMAGE_COLOR);
    
    // Convert images to gray scale;
	Mat im1_gray, im2_gray;
	cvtColor(im1, im1_gray, CV_BGR2GRAY);
	cvtColor(im2, im2_gray, CV_BGR2GRAY);

    // Define the motion model
    const int warp_mode = MOTION_EUCLIDEAN;

    // Set a 2x3 or 3x3 warp matrix depending on the motion model.
    Mat warp_matrix;
    
    // Initialize the matrix to identity
    if ( warp_mode == MOTION_HOMOGRAPHY )
        warp_matrix = Mat::eye(3, 3, CV_32F);
    else
        warp_matrix = Mat::eye(2, 3, CV_32F);

    // Specify the number of iterations.
    int number_of_iterations = 5000;
    
    // Specify the threshold of the increment
    // in the correlation coefficient between two iterations
    double termination_eps = 1e-10;
    
    // Define termination criteria
    TermCriteria criteria (TermCriteria::COUNT+TermCriteria::EPS, number_of_iterations, termination_eps);

    // Run the ECC algorithm. The results are stored in warp_matrix.
    findTransformECC(
                     im1_gray,
                     im2_gray,
                     warp_matrix,
                     warp_mode,
                     criteria
                 );

    // Storage for warped image.
    Mat im2_aligned;

    if (warp_mode != MOTION_HOMOGRAPHY)
        // Use warpAffine for Translation, Euclidean and Affine
        warpAffine(im2, im2_aligned, warp_matrix, im1.size(), INTER_LINEAR + WARP_INVERSE_MAP);
    else
        // Use warpPerspective for Homography
        warpPerspective (im2, im2_aligned, warp_matrix, im1.size(),INTER_LINEAR + WARP_INVERSE_MAP);

    // Show final result
    imshow("Image 1", im1);
    imshow("Image 2", im2);
    imshow("Image 2 Aligned", im2_aligned);
    waitKey(0);

}

输出结果:

也可以用Python:

#!/usr/bin/python

'''
    OpenCV Image Alignment  Example
    
    Copyright 2015 by Satya Mallick <[e-mail]>
    
'''

import cv2
import numpy as np


if __name__ == '__main__':
    
    # Read the images to be aligned
    im1 =  cv2.imread("images/image1.jpg");
    im2 =  cv2.imread("images/image2.jpg");
    
    # Convert images to grayscale
    im1_gray = cv2.cvtColor(im1,cv2.COLOR_BGR2GRAY)
    im2_gray = cv2.cvtColor(im2,cv2.COLOR_BGR2GRAY)

    # Find size of image1
    sz = im1.shape

    # Define the motion model
    warp_mode = cv2.MOTION_TRANSLATION

    # Define 2x3 or 3x3 matrices and initialize the matrix to identity
    if warp_mode == cv2.MOTION_HOMOGRAPHY :
        warp_matrix = np.eye(3, 3, dtype=np.float32)
    else :
        warp_matrix = np.eye(2, 3, dtype=np.float32)

    # Specify the number of iterations.
    number_of_iterations = 5000;
    
    # Specify the threshold of the increment
    # in the correlation coefficient between two iterations
    termination_eps = 1e-10;
    
    # Define termination criteria
    criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, number_of_iterations,  termination_eps)

    # Run the ECC algorithm. The results are stored in warp_matrix.
    (cc, warp_matrix) = cv2.findTransformECC (im1_gray,im2_gray,warp_matrix, warp_mode, criteria)


    if warp_mode == cv2.MOTION_HOMOGRAPHY :
        # Use warpPerspective for Homography
        im2_aligned = cv2.warpPerspective (im2, warp_matrix, (sz[1],sz[0]), flags=cv2.INTER_LINEAR + cv2.WARP_INVERSE_MAP)
    else :
        # Use warpAffine for Translation, Euclidean and Affine
        im2_aligned = cv2.warpAffine(im2, warp_matrix, (sz[1],sz[0]), flags=cv2.INTER_LINEAR + cv2.WARP_INVERSE_MAP);

    # Show final results
    cv2.imshow("Image 1", im1)
    cv2.imshow("Image 2", im2)
    cv2.imshow("Aligned Image 2", im2_aligned)
    cv2.waitKey(0)

五. 实战

直接用上述的方法做大叔三张图的Alignment, 效果不好.
因为RGB三个通道的灰度值的相关性没有那么大. 例如大叔的衣服, 三张图中三种样子.
但是, 在梯度空间(Gradient Domain)里, RGB三图的梯度相关性是较高的.

…even though the intensities may be different in the three channels,
the edge map generated by object and color boundaries are consistent.

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-G8i4Yphy-1583201443771)(https://www.learnopencv.com/wp-content/uploads/2015/06/emir-394x1024.jpg)]

C++实现大概用了2分半钟. 时间还是比较长的.

/**
 * OpenCV Image Alignment  Example
 *
 * Copyright 2015 by Satya Mallick <[e-mail]>
 *
 */

#include "opencv2/opencv.hpp"

using namespace cv;
using namespace std;

Mat GetGradient(Mat src_gray)
{ 
	Mat grad_x, grad_y;
	Mat abs_grad_x, abs_grad_y;

	int scale = 1; 	
	int delta = 0; 
	int ddepth = CV_32FC1; ;
    
    // Calculate the x and y gradients using Sobel operator

	Sobel( src_gray, grad_x, ddepth, 1, 0, 3, scale, delta, BORDER_DEFAULT );
	convertScaleAbs( grad_x, abs_grad_x );

	Sobel( src_gray, grad_y, ddepth, 0, 1, 3, scale, delta, BORDER_DEFAULT );
	convertScaleAbs( grad_y, abs_grad_y );

    // Combine the two gradients
    Mat grad;
	addWeighted( abs_grad_x, 0.5, abs_grad_y, 0.5, 0, grad );

	return grad; 

} 


int main( int argc, char** argv )
{

	// Read 8-bit color image.
    // This is an image in which the three channels are
    // concatenated vertically.
	Mat im =  imread("F:\\GitWareHouse\\learnopencv\\ImageAlignment\\images\\emir.jpg", IMREAD_GRAYSCALE);
	
    // Find the width and height of the color image
    Size sz = im.size();
    int height = sz.height / 3;
	int width = sz.width; 
 
    // Extract the three channels from the gray scale image
	vector<Mat>channels; 
	channels.push_back(im( Rect(0, 0,         width, height)));
	channels.push_back(im( Rect(0, height,    width, height))); 
	channels.push_back(im( Rect(0, 2*height,  width, height))); 

    // Merge the three channels into one color image
	Mat im_color; 
	merge(channels,im_color);
    
    // Set space for aligned image.
    vector<Mat> aligned_channels;
    aligned_channels.push_back(Mat(height, width, CV_8UC1));
    aligned_channels.push_back(Mat(height, width, CV_8UC1));
    
    // The blue and green channels will be aligned to the red channel.
    // So copy the red channel
    aligned_channels.push_back(channels[2].clone());


    // Define motion model
	const int warp_mode = MOTION_AFFINE;
    
    // Set space for warp matrix.
	Mat warp_matrix;
    
    // Set the warp matrix to identity.
    if ( warp_mode == MOTION_HOMOGRAPHY )
		warp_matrix = Mat::eye(3, 3, CV_32F);
	else
		warp_matrix = Mat::eye(2, 3, CV_32F);

    // Set the stopping criteria for the algorithm.
    int number_of_iterations = 5000;
	double termination_eps = 1e-10;
    
    TermCriteria criteria(TermCriteria::COUNT+TermCriteria::EPS,
                          number_of_iterations, termination_eps);

    
    // Warp the blue and green channels to the red channel
	for ( int i = 0; i < 2; i++)
	{
        
		double cc = findTransformECC (
                                      GetGradient(channels[2]),
                                      GetGradient(channels[i]),
                                      warp_matrix,
                                      warp_mode,
                                      criteria
                                      );

		cout << "warp_matrix : " << warp_matrix << endl;  
		cout << "CC " << cc << endl; 
		if (cc == -1)
		{
			cerr << "The execution was interrupted. The correlation value is going to be minimized." << endl;
			cerr << "Check the warp initialization and/or the size of images." << endl << flush;
		}


		if (warp_mode == MOTION_HOMOGRAPHY)
            // Use Perspective warp when the transformation is a Homography
            warpPerspective (channels[i], aligned_channels[i], warp_matrix, aligned_channels[0].size(), INTER_LINEAR + WARP_INVERSE_MAP);
        else
            // Use Affine warp when the transformation is not a Homography
            warpAffine(channels[i], aligned_channels[i], warp_matrix, aligned_channels[0].size(), INTER_LINEAR + WARP_INVERSE_MAP);
        
	} 
	
    // Merge the three channels
    Mat im_aligned;
    merge(aligned_channels, im_aligned);

    // Show final output
	imshow("Color Image", im_color);
	imshow("Aligned Image", im_aligned);
  	waitKey(0);	

}

Aligned 结果:

如果加入TickMeter 计时功能:

143秒, 时间还是挺长的.

再附上Python代码:

#!/usr/bin/python

'''
    OpenCV Image Alignment  Example
    
    Copyright 2015 by Satya Mallick <[e-mail]>
    
'''

import cv2
import numpy as np

def get_gradient(im) :
    # Calculate the x and y gradients using Sobel operator
    grad_x = cv2.Sobel(im,cv2.CV_32F,1,0,ksize=3)
    grad_y = cv2.Sobel(im,cv2.CV_32F,0,1,ksize=3)
    # Combine the two gradients
    grad = cv2.addWeighted(np.absolute(grad_x), 0.5, np.absolute(grad_y), 0.5, 0)
    return grad


if __name__ == '__main__':
    
    
    # Read 8-bit color image.
    # This is an image in which the three channels are
    # concatenated vertically.
    
    im =  cv2.imread("images/emir.jpg", cv2.IMREAD_GRAYSCALE);

    # Find the width and height of the color image
    sz = im.shape
    print sz
    height = int(sz[0] / 3);
    width = sz[1]

    # Extract the three channels from the gray scale image
    # and merge the three channels into one color image
    im_color = np.zeros((height,width,3), dtype=np.uint8 )
    for i in xrange(0,3) :
        im_color[:,:,i] = im[ i * height:(i+1) * height,:]

    # Allocate space for aligned image
    im_aligned = np.zeros((height,width,3), dtype=np.uint8 )

    # The blue and green channels will be aligned to the red channel.
    # So copy the red channel
    im_aligned[:,:,2] = im_color[:,:,2]

    # Define motion model
    warp_mode = cv2.MOTION_HOMOGRAPHY

    # Set the warp matrix to identity.
    if warp_mode == cv2.MOTION_HOMOGRAPHY :
            warp_matrix = np.eye(3, 3, dtype=np.float32)
    else :
            warp_matrix = np.eye(2, 3, dtype=np.float32)

    # Set the stopping criteria for the algorithm.
    criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 5000,  1e-10)

    # Warp the blue and green channels to the red channel
    for i in xrange(0,2) :
        (cc, warp_matrix) = cv2.findTransformECC (get_gradient(im_color[:,:,2]), get_gradient(im_color[:,:,i]),warp_matrix, warp_mode, criteria)
    
        if warp_mode == cv2.MOTION_HOMOGRAPHY :
            # Use Perspective warp when the transformation is a Homography
            im_aligned[:,:,i] = cv2.warpPerspective (im_color[:,:,i], warp_matrix, (width,height), flags=cv2.INTER_LINEAR + cv2.WARP_INVERSE_MAP)
        else :
            # Use Affine warp when the transformation is not a Homography
            im_aligned[:,:,i] = cv2.warpAffine(im_color[:,:,i], warp_matrix, (width, height), flags=cv2.INTER_LINEAR + cv2.WARP_INVERSE_MAP);
        print warp_matrix

    # Show final output
    cv2.imshow("Color Image", im_color)
    cv2.imshow("Aligned Image", im_aligned)
    cv2.waitKey(0)