Charles Xu

COMS 4732 Project 1: Images of the Russian Empire: Colorizing the Prokudin-Gorskii photo collection

Introduction

The Prokudin-Gorskii photo collection is a historical archive containing more than 10,000 glass plate images captured between 1908 and 1915 by Russian photographer Sergei Mikhailovich Prokudin-Gorskii. These photographs offer a rare view into the daily life, architecture, and landscapes of the Russian Empire during a time of profound political and social transformation. The Library of Congress digitized the negatives of these glass plates and made them available to the public. In this project, we use image plates selected from the Prokudin-Gorskii collection to experiment with image processing techniques.

Method

Simple Alignment

The naive approach iterates over all possible displacements in both x and y directions within a window of 15 pixels. Fixing one channel as a reference, we move the other channels and evaluate the alignment score at each step. There are two metrics we use: L2 Euclidean distance and normalized cross-correlation (NCC).

L2 Norm (Sum of Squared Differences) is defined as:

\[\text{L2 Norm} = \sum \sum \left( (\text{image1} - \text{image2})^2 \right)\]

Normalized Cross-Correlation (NCC) is defined as:

\[\text{NCC} = \frac{\text{image1}}{||\text{image1}||} \cdot \frac{\text{image2}}{||\text{image2}||}\]

For the L2 norm, we simply compute the sum of squared differences between pixels of the base image and the shifted image. A lower L2 norm indicates better matching.

For NCC, we treat each image as a vector. We take the zero-centered, normalized vectors and compute their dot product. NCC performs consistently better in my experiments.

Multi-Resolution Pyramid Alignment

The pyramid search algorithm recursively scales down the image by half and performs naive alignment on the scaled images to improve efficiency and handle larger displacements.

Implementation

Simple Alignment

In our implementation, we fix the blue image as the base plane and shift the green and red images. For simple alignment, I implemented a function named find_displacement_naive(image_1, image_2, window_size=15, NCC=False) which exhaustively computes the score for all combinations of displacements. The final displacement is chosen by minimizing the L2 distance or maximizing the NCC score. NCC performs consistently better than L2 distance in my experiments.

Multi-Resolution Pyramid Alignment

For pyramid alignment, I implemented the find_displacement_pyramid function on top of find_displacement_naive, which reduced runtime and enhanced performance for high-resolution images. I found that using NCC further improved the results. The challenge was determining the optimal number of pyramid layers. Setting a fixed number of layers is not ideal because different image sizes require different depths. Therefore, I ensured that the smallest image had at least 16 to 32 pixels and calculated the number of layers using a logarithmic function. We precompute the images for all pyramid levels and iteratively refine the displacement from the coarsest level to the finest.

I also experimented with dynamically adjusting the window size as the scaling factor increased, but this resulted in longer runtimes without noticeable performance improvements. To balance runtime and accuracy, I chose a constant window size (e.g., 2 pixels for refinement levels) as the optimal value.

Bell & Whistle

To make the image look more natural, I employed two additional techniques: gradient detection and cropping. The Sobel gradient functions as an edge detection filter. It highlights areas of the image where intensity changes significantly. This feature improves alignment accuracy by focusing on elements that are more structural significant. Also, image borders typically do not contribute to the alignment process, so cropping the outer borders helps eliminate noise that may be inconsistent with the main image content.