Photoconsistency constraint and novel view generation

Given a set of input images such as:

consider the problem of genrating a novel view.

1. Finely sample the back-projected ray from a site in the target super resolved view between a range $Z_{min}$ to $Z_{max}$ . For each $z$ in this range, project the 3D point ${\bf X}(z)$ to each of the low resolution input images and compute

   $${c}(p,z)=\{{\bf g}_k(p'(z)) \;\vert \;\;\; 1\leq k \leq n\}$$ (1)

   
   
   as a set of possible colours corresponding to depth $z$ .
   

2. For each $z$ do a nearest neighbour clustering of ${c}(p,z)$ in $YCbCr$ space, with a suitable distance threshold on colour differences, and remove all but the top few dominant clusters. If there are no dominant clusters then we remove $z$ from the list of candidate depths at $p$ .

3. For all candidate $z$ values at site $p$ we write

   $${\mathcal C}(p,z)=\big \{{\mathcal I}(p,z,i)\big \}_{i=1\dots m_{pz}}$$ (2)

   
   
   where $m_{pz}$ is the number of dominant clusters, and ${\mathcal I}(p,z,i)$ is the set of the image indices that belong to the $i^{th}$ dominant cluster. For each depth $z$ .

4. Define a correlation function

   $$fcorr(z)=\frac{ \sum_{i=1}^{m_{pz}} corr(patch({\mathcal I}(p,z,i)))}{\sum_{i=1}^{m_{pz}} {{\mathcal I}(p,z,i)\vert \choose 2} }$$ (3)

   
   
   where $patch({\mathcal I} (p,z,i))$ is the set of $3 \times 3$ patches around the low resolution pixel ${\bf g}_k(p')$ and $corr(patch({\mathcal I} (p,z,i)))$ is the sum of pair-wise correlations between patches of ${\mathcal I}(p,z,i)$ .

5. Form a set of candidate depths $d_p$ at site $p$ by choosing the local minimas of $fcorr(z)$ .

6. More than one dominant clusters at any $z$ suggests occlusion
   

7. Resolve occlusion using the ordinal visibility constraint:
   If ${\bf X}(z_1)$ and ${\bf X}(z_2)$ are any two points on two back projected rays then ${\bf X}(z_1)$ can occlude ${\bf X}(z_2)$ only if $z_1<z_2$ .
8. Novel view result:

   (a)	(b)
   (c)	(d)

   (a) One of the input images (out of 40). (b) Photo-consistent depth map, note that the depth map is not accurate at occluding pixels and at homogeneous parts. (c) Occlusion-map: the pixels for which multiple clusters are formed, and are marked white. (d) The generated novel view. Note that though the depth map generated is not accurate, it provides correct photo-consistent colour in the novel view.
   
   Generated novel view and the difference image of novel view with the ground truth.