Keywords

1 Introduction

With the popularity of digital cameras, online photo albums and social networks, we can easily find thousands of photos of the same scenes, especially for the famous landmarks. And these kinds of photo collections have been applied in high level computer vision problems, including visual photo tourism [1], scene completion [2], landmark recognition [3], content-based image retrieval [4], photo stitching [5], and time-lapse [6]. However, these online images are usually taken with different capture devices [7], different illuminations, and different views, which are the reasons for the different color styles in the same photo collection.

In order to obtain better results of the above mentioned interesting computer vision applications, the color consistency or color correction is an essential pre-processing procedure before implementing the main algorithms. The most simple color correction method is the auto color adjustment technique, which is implemented in most commercial photo editing tools. However, this kind of method makes the color correction for each image independently, not considering the color distribution relations among all images in the same collection. Another kind of method is manually adjusting all images by professional photographers, which requires a significant amount of processing time for a large number of photos.

Recently, Park et al. [8] proposed a method to optimize color consistency in community photo collections depicting the same scenes. Their method is based on the assumption of a color correction model, which is a simple combination of white balance and gamma correction. Park et al.’s method is globally making color mapping in the three color channels respectively, and may cause gamut problems, since the dark pixel values are easily saturated to 0 and the bright pixel values are easily saturated to 255 in this simple global parameter-based model. Another disadvantage of their algorithm is that the final results are highly depending on the color distribution of all images. In other words, some low quality images with bad color distribution will affect the quality of other images in the same collection.

In order to avoid gamut problems [9, 10] and reduce the effects from bad quality images, we present an automatic color consistency algorithm for photo collections based on image quality assessment and range-preserving histogram specification. In our method, input images are divided into two sets, namely high quality image set and low quality image set. And then the images having high quality will be used to produce corresponding color balanced images. At last, the images classified in the low quality image set will be color transferred with the corrected high quality images as the reference. These three stages can obtain color consistency among the whole set of images and not cause the disadvantages existing in Park et al.’s method.

The rest of this paper is organized as follows. The relevant previous work is summarized in Sect. 2. The overview of the proposed algorithm is described in Sect. 3 and the detailed description is given in Sect. 4. The experiments and comparisons are shown in Sect. 5. Finally, we give the conclusion and discuss the future work in Sect. 6.

2 Related Work

In this section, we summarize the previous related work and categorize these methods by the number of input images. Color image enhancement is an ancient image processing technique, which is normally applied to a single image based on the statistic information of the input image or some specialized filters. Color transfer is a technique of transferring the color style of a reference image to a test image. Color balance for image collections is a new and interesting problem, which can enforce thousands of images to share the same color style.

2.1 Color Enhancement for a Single Image

Color image enhancement is an ancient and challenging task in image processing. Histogram equalization [11] is a useful method for image enhancement. Sapiro and Caselles [12] proposed a PDE-based image histogram modification method, which was further extended to color image enhancement by many researchers [13,14,15,16,17]. The automatic color enhancement (ACE) algorithm [18] is a popular method, which has been used in many photo editing software. Recently, Yuan and Sun [19] proposed an automatic exposure correction method, which was based on region-level exposure evaluation and detail-preserving S-curve adjustment. This kind of algorithm operates on each image independently, so they can not obtain color consistency among all images in the same collection.

2.2 Color Transfer Between Two Images

Color transfer between two images has received significant attention in recent years. The goal of this kind of algorithm is transferring the color style of a reference image to a test image, which can enforce the corresponding two images to share the same color style. Reinhard et al. [20] proposed a simple statistics-based algorithm to transfer colors between two images, which was also extended by many researchers. Papadakis et al. [21] proposed a variational model for color image histogram transfer, which used the energy functional minimization to obtain the goal of transferring the image color style and maintaining the image geometry. Hristova et al. [22] presented a style-aware robust color transfer method, which was based on the style feature clustering and the local chromatic adaptation transform. Hwang et al. [23] proposed a geometry-based color transfer algorithm using probabilistic moving least squares. A more detailed review of this kind of algorithm can been seen in [24].

2.3 Color Balance for Image Collections

Automatic color consistency for photo collections firstly obtained attention in HaCohen et al. [25]. Their method used the non-rigid dense correspondence algorithm [26] to obtain matched features, and then constructed a graph with edges linking image pairs sharing the similar contents. Color consistency was achieved by globally optimizing three piece-wise quadratic curves over the whole graph, which can minimize the color difference between all correspondences. However, their method is highly depending on dense and accurate correspondences (high computational complexity), which makes this algorithm unsuitable for a large number of images in photo collections.

A more efficient and robust color consistency algorithm for photo collections was proposed by Park et al. [8]. They firstly recovered sparse point correspondences in the same image collection, and then constructed a simple color correction model which can be considered the combination of white balance and gamma correction. Then a robust low-rank matrix factorization method was used to estimate the unknown parameters of the defined model. However, this correction model does not consider gamut problems and negative effects resulting from low quality images in the same collection.

3 Overview

Given an input photo collection as a sequence of images \(I=\{I_1, I_2, I_3,..., I_n\}\) which describe the same scene with different views, different capture devices or different illustrations, our goal is to make these images share the same color style and to be color consistent in the shared contents.

Our method is inspired by [8], which has given a whole framework to minimizing color difference among all input images. However, we do not adopt the simple global parameter-based model to correct colors, which may result in gamut problems. The contributions of our work are classifying input images based on image quality metrics and finding suitable target histograms for the range-preserving histogram specification.

Figure 1 illustrates the overall framework of our color consistency algorithm, which consists of three main stages.

1. Image quality assessment: use non-reference measures to assess the image visual quality, and then construct two image sets having respectively high quality and low quality.

2. Color balance: use matched points between images having high quality to construct correspondences, and then obtain early corrected images with linear transforms. These early images can produce the target histograms. At last, the range-preserving histogram specification is applied to each image.

3. Color transfer: use matched points between low quality images and corrected high quality images to construct correspondences, and then obtain final corrected images with linear transforms (can produce target histograms) and the range-preserving histogram specification.

Fig. 1.
figure 1

The overall framework of the proposed method

Full size image

4 Color Consistency without Gamut Problems

4.1 Image Quality Assessment by Non-reference Measures

As discussed in Sect. 1, the results of Park et al.’s method [8] depend on the color distribution of all images in the same collection. This kind of algorithm may obtain some bad results when there are many low quality images in the same image collection. So we firstly make quality assessment for input images, which can sort the images by their visual quality. Then we make color balance for the images ranking high quality. At last, this kind of color style obtained from high visual quality images will be propagated to the images which are ranked lower visual quality in the assessment procedure. This kind of processing can reduce bad effects from lower visual quality images.

Much effort [27, 28] has been made in recent years to design non-reference image quality assessment metrics. Panetta et al. [27] proposed a non-reference color image quality measurement combining the contrast, colorfulness and sharpness metrics. In other words, this measurement is a linear weighting of three terms assessing the contrast, color and edge information respectively. We simplify this assessment method and adopt the contrast and the sharpness metrics in our method. The measurement IQA (Image Quality Assessment) is written in Eq. (1).

$$\begin{aligned} IQA = w_1*Contrast + w_2*Sharpness~, \end{aligned}$$
(1)

where \(w_1\) is the weighting of the contrast measure, \(w_2\) is the weighting of the sharpness measure, Contrast is defined in Eq. (2), Sharpness is defined in Eq. (3).

$$\begin{aligned} \begin{aligned}&Contrast = AME(I\_Y) \\&AME(I\_Y) = \frac{1}{k_1*k_2}\sum _{l=1}^{k_1}\sum _{k=1}^{k_2}\Big (log\big (\frac{I\_Y(max,k,l)+I\_Y(min,k,l)}{I\_Y(max,k,l)-I\_Y(min,k,l)}\big )\Big )^{-0.5} ~, \end{aligned} \end{aligned}$$
(2)

where AME is the Michelson-Law measure of enhancement [29], \(I\_Y\) is the values of Y channel of the input image in the YCbCr color space, the image is divided into \(k_1*k_2\) blocks, \(I\_Y(min,k,l)\) is the minimum of the block (k, l) in \(I\_Y\), \(I\_Y(max,k,l)\) is the maximum of the block (k, l) in \(I\_Y\), k and l are the row and column index of blocks in the image.

$$\begin{aligned} \begin{aligned}&Sharpness = \sum _{c=1}^{3}\lambda _c EME(GrayEdge_c) \\&EME(GrayEdge) = \frac{2}{k_1*k_2}\sum _{l=1}^{k1}\sum _{k=1}^{k2}log\big (\frac{GrayEdge(max,k,l)}{GrayEdge(min,k,l)}\big ) ~, \end{aligned} \end{aligned}$$
(3)

where c is the index of the color channel, EME is the measure of enhancement [30], \(GrayEdge_c\) is the edge map of the corresponding color channel, \(\lambda _c\) is the weighting of each color channel, the image is divided into \(k_1*k_2\) blocks. GrayEdge(min, k, l) and GrayEdge(max, k, l) are respectively the minimum and maximum of the block (k, l) in GrayEdge, k and l are the row and column index of blocks in the image.

After this kind of quality assessment, we obtain the corresponding visual quality values of each image, which are written by \(IQA = \{IQA_1, IQA_2, IQA_3, ... , IQA_n\}\). Then these images can be sorted by descending the values of IQA. Some best quality images \(I_{high} = \{I_{high(1)}, I_{high(2)}, ... , I_{high(s)}\}\) will be chosen for make color balance in the next stage. s is the total number of images in the set \(I_{high}\). The rest images are labeled by \(I_{low} = \{I_{low(1)}, I_{low(2)}, ... , I_{low(t)}\}\), where \(t = n-s\).

4.2 Color Balance Without Gamut Problems

In this stage, the goal is to make color balance for the image set \(I_{high} = \{I_{high(1)}, I_{high(2)}, ... , I_{high(s)}\}\). We adopt the global color correction method, which is robust to feature matching errors and has low computational complexity. We do not define the simple correction models which were defined in [8, 25, 26]. For example, the correction model in [8] is the simulation of white balance and gamma correction, which is depending on only two parameters for each image and may result in gamut problems. Our color balance method consists of two parts, linear transforms to obtain target histograms and range-preserving histogram specifications to produce corrected images. The detailed algorithm is described in Algorithm 1.

figure a

For the step 7 in Algorithm 1, we combine the hue-preserving color image enhancement framework [9] and the fast exact histogram specification [15, 16], which can avoid gamut problems. Given a color image IM, we firstly get the corresponding intensity image G using Eq. (4). Then the histogram specification will be applied to G, the result is \(\hat{G}\). At last, we use the hue-preserving enhancement framework to obtain the final corrected color image \(\hat{IM}\), which is computed using the Eq. (5) for the three color channels respectively.

$$\begin{aligned} G = 0.299*I\_R + 0.587*I\_G + 0.114*I\_B ~, \qquad \qquad \qquad \end{aligned}$$
(4)

where \(I\_R, I\_G, I\_B\) are respectively the red color channel, green color channel and blue color channel of the corresponding image IM.

$$\begin{aligned} \begin{aligned}&\hat{im(k)} = \frac{\hat{G(k)}}{G(k)}im(k),\ \ \qquad \qquad \qquad \qquad \qquad if \ \frac{\hat{G(k)}}{G(k)} <=1, \\&\hat{im(k)} = \frac{255-\hat{G}(k)}{255-G(k)}\big (im(k)-G(k)\big )+\hat{G(k)}, \ if \ \frac{\hat{G}(k)}{G(k)}>1, \end{aligned} \end{aligned}$$
(5)

where im is the corresponding color channel of the image IM, \(\hat{im}\) is the corresponding color channel of the final corrected image \(\hat{IM}\), k is the index of pixels in each gray image.

4.3 Color Transfer Without Gamut Problems

In the first stage, we divided the input image collection into two image sets, based on a non-reference image quality assessment method. The first image set has been color balanced in the previous stage. And the other image set having lower quality will be color corrected in this stage. We use the idea of color transfer or color propagation between two images to finish the task at this stage. The detailed algorithm is described in Algorithm 2.

figure b

For the step 6 in Algorithm 2, we adopt the same method described in the previous stage, which can avoid gamut problems.

5 Experiments

5.1 Image Datasets

We had run our algorithm and the comparison algorithm [8] on the landmark images download from Flickr, which were also collected as a public test dataset on the project website (landmark3d.codeplex.com) [3]. This dataset was popularly used as a benchmark for evaluating and comparing different computer vision algorithms. In our experiments, four landmarks had been chosen to test color consistency algorithms. Arc Triomphe, Notre Dame Cathedral, Statue of Liberty, Trevi Fountain are the names of these landmarks. For each landmark, we chose 20 images having different color styles, which were captured with different cameras, different illuminations, different views by different photographers. Some examples will be shown in the following part.

5.2 Results

For the comparison algorithm, we use the original codes shared on the author’s homepage and the parameters are the default values. Our method is implemented on the Matlab platform. Figures 2, 3, 4 and 5 show the results of Park et al.’s method [8] and our method. Since there is no efficient object evaluation metric to assess the performance of color consistency algorithms for photo collections, we compare the results with the subject method based on the color distribution and visual quality of corrected images.

Fig. 2.
figure 2

Results on the photo collection Arc Triomphe, only four images are shown. The upper row shows the original images, the middle row shows the results by Park et al.’s method [8], the lower row shows our results. Red and yellow squares indicate gamut problems in Park et al.’s method. (Color figure online)

Full size image
Fig. 3.
figure 3

Results on the photo collection Notre Dame Cathedral, only four images are shown. The upper row shows the original images, the middle row shows the results by Park et al.’s method [8], the lower row shows our results. Red squares indicate gamut problems in Park et al.’s method. (Color figure online)

Full size image
Fig. 4.
figure 4

Results on the photo collection Trevi Fountain, only four images are shown. The upper row shows the original images, the middle row shows the results by Park et al.’s method [8], the lower row shows our results. Red squares indicate gamut problems in Park et al.’s method. (Color figure online)

Full size image
Fig. 5.
figure 5

Results on the photo collection Statue of Liberty, only five images are shown. The upper row shows the original images, the middle row shows the results by Park et al.’s method [8], the lower row shows our results.

Full size image

The results shown in Fig. 2 demonstrate that our approach can enforce the image collection to share a similar color style and not cause gamut problems. The matched patches from the original images and the corrected images can easily describe the color consistency after our method. The yellow squares indicate that the compared algorithm causes gamut problems in the dark regions. In the same way, the red squares indicate that it causes gamut problems in the bright regions. Our method do not result in this kind of disadvantage, since we adopt a range preserving exact histogram specification in the proposed algorithm.

In Fig. 3, the results of Park et al.’s method and our approach show these two methods both enforce the images to share a similar color style. However, our method outperforms the compared algorithm on the bright sky regions, which is easy to cause gamut problems in Park et al.’s method. The results shown in Fig. 4 are similar to the results of Fig. 3, which indicates that the two approaches both make color consistency well and Park et al.’s method can not correct pixels in bright and dark regions.

In Fig. 5, it is not easy to compare the results from the whole images on the left of this figure. However, we can know that our results are slightly better than the compared algorithm, from the re-sized matched patches on the right of the figure.

6 Conclusion

In this work, we have proposed a color consistency approach for image collections, based on range-preserving histogram specification. This method can enforce the same image collection to share the same color style and not to cause gamut problems. The main contributions of our work are reducing the effects of bad visual quality images to the whole image collection based on non-reference image quality assessment method, and finding suitable target histograms for range-preserving histogram specifications. The experiments demonstrate the proposed method outperforms the compared algorithm in terms of color consistency and avoiding gamut problems.

The proposed method is also based on a global model, since the global histogram specification is adopted. Generally speaking, global model may result in some color artifacts when the input images have some bright and dark regions. So we will focus on the local-based model in the future.