Merge pull request #1941 from surgical-vision:quasi-dense-stereo

Implementation of Quasi Dense Stereo algorithm. (#1941)

* initial commit.

* Remove license header.

* Fix python wrap flags

* Change std::string to cv::String, in function declarations, to resolve compilation issues.

* Add python wrapper extending header

* Fix python wrapper conflicts

* Fix implicit type conversions

* Change C API types and enums to C++.

* Remove redundant included headers and move wanted headers to src/precomp.hpp

* Remove saturate header

* Remove unnecessary python wrapping flags

* Removed defaults parameter header

* Split declaration and implementation of the class using Pimpl.

* Fix to comply with new public API.

* Remove unnecessary modules

* Fix maybe-uninitialized warnings on linux

* Migration to stereo module

* Remove CV_PROP_RW flag.

* Remove CV_EXPORTS flags from class members.

* Fix: Removed misplaced flag

* Remove empty lines.

* Move queue to private headers.

* Fix default arguments of public methods.

* Add authors information and switch to the compact version of license header.

* Reorganize and fix markdown files. Create a table of content and move tutorials in new directories. Modify samples and tutorials to use snippet and include Doxygen commands.

* Change argument name dMatch->denseMatch, to avoid confusion with cv::DMatch build-in type.

* Remove duplicate snippet.

* Fix: change vector resize to reserve.

* Fix: replace extensive license header with the compact version.

modules/README.md

@@ -54,7 +54,7 @@ $ cmake -D OPENCV_EXTRA_MODULES_PATH=<opencv_contrib>/modules -D BUILD_opencv_<r
 
 - **sfm**: Structure from Motion -- This module contains algorithms to perform 3d reconstruction from 2d images. The core of the module is a light version of Libmv.
 
-- **stereo**: Stereo Correspondence -- Stereo matching done with different descriptors: Census / CS-Census / MCT / BRIEF / MV.
+- **stereo**: Stereo Correspondence -- Stereo matching done with different descriptors: Census / CS-Census / MCT / BRIEF / MV and dense stereo correspondence using Quasi Dense Stereo method.
 
 - **structured_light**: Structured Light Use -- How to generate and project gray code patterns and use them to find dense depth in a scene.
 

modules/stereo/CMakeLists.txt

 set(the_description "Stereo Correspondence")
-ocv_define_module(stereo opencv_imgproc opencv_features2d opencv_core opencv_calib3d)
+ocv_define_module(stereo opencv_imgproc opencv_features2d opencv_core opencv_calib3d opencv_tracking opencv_video)

modules/stereo/README.md

@@ -2,3 +2,10 @@ Stereo Correspondence with different descriptors
 ================================================
 
 Stereo matching done with different descriptors: Census / CS-Census / MCT / BRIEF / MV.
+
+Quasi Dense Stereo
+======================
+
+Quasi Dense Stereo is method for performing dense stereo matching.
+The code uses pyramidal Lucas-Kanade with Shi-Tomasi features to get the initial seed correspondences.
+Then these seeds are propagated by using mentioned growing scheme.

modules/stereo/doc/stereo.bib

+@InProceedings{Stoyanov2010,
+author="Stoyanov, Danail
+and Scarzanella, Marco Visentini
+and Pratt, Philip
+and Yang, Guang-Zhong",
+editor="Jiang, Tianzi
+and Navab, Nassir
+and Pluim, Josien P. W.
+and Viergever, Max A.",
+title="Real-Time Stereo Reconstruction in Robotically Assisted Minimally Invasive Surgery",
+booktitle="Medical Image Computing and Computer-Assisted Intervention (MICCAI 2010)",
+year="2010",
+publisher="Springer Berlin Heidelberg",
+address="Berlin, Heidelberg",
+pages="275--282",
+abstract="The recovery of 3D tissue structure and morphology during robotic assisted surgery is an important step towards accurate deployment of surgical guidance and control techniques in minimally invasive therapies. In this article, we present a novel stereo reconstruction algorithm that propagates disparity information around a set of candidate feature matches. This has the advantage of avoiding problems with specular highlights, occlusions from instruments and view dependent illumination bias. Furthermore, the algorithm can be used with any feature matching strategy allowing the propagation of depth in very disparate views. Validation is provided for a phantom model with known geometry and this data is available online in order to establish a structured validation scheme in the field. The practical value of the proposed method is further demonstrated by reconstructions on various in vivo images of robotic assisted procedures, which are also available to the community.",
+isbn="978-3-642-15705-9"
+}
+
+@article{Lhuillier2000,
+abstract = {A new robust dense matching algorithm is introduced. The algorithm$\backslash$nstarts from matching the most textured points, then a match propagation$\backslash$nalgorithm is developed with the best first strategy to dense matching.$\backslash$nNext, the matching map is regularised by using the local geometric$\backslash$nconstraints encoded by planar affine applications and by using the$\backslash$nglobal geometric constraint encoded by the fundamental matrix. Two most$\backslash$ndistinctive features are a match propagation strategy developed by$\backslash$nanalogy to region growing and a successive regularisation by local and$\backslash$nglobal geometric constraints. The algorithm is efficient, robust and can$\backslash$ncope with wide disparity. The algorithm is demonstrated on many real$\backslash$nimage pairs, and applications on image interpolation and a creation of$\backslash$nnovel views are also presented},
+author = {Lhuillier, Maxime and Quan, Long},
+doi = {10.1109/ICPR.2000.905620},
+file = {:home/dimitrisps/Desktop/ucl/papers/quasiDenseMatching.pdf:pdf},
+isbn = {0-7695-0750-6},
+issn = {10514651},
+journal = {Proceedings-International Conference on Pattern Recognition},
+number = {1},
+pages = {968--972},
+title = {{Robust dense matching using local and global geometric constraints}},
+volume = {15},
+year = {2000}
+}

modules/stereo/include/opencv2/stereo.hpp

@@ -49,6 +49,7 @@
 #include "opencv2/core/affine.hpp"
 #include "opencv2/stereo/descriptor.hpp"
 #include "opencv2/stereo/matching.hpp"
+#include <opencv2/stereo/quasi_dense_stereo.hpp>
 
 /**
 @defgroup stereo Stereo Correspondance Algorithms

modules/stereo/include/opencv2/stereo/quasi_dense_stereo.hpp

+// This file is part of OpenCV project.
+// It is subject to the license terms in the LICENSE file found in the top-level directory
+// of this distribution and at http://opencv.org/license.html.
+
+//authors: Danail Stoyanov, Evangelos Mazomenos, Dimitrios Psychogyios
+
+
+//__OPENCV_QUASI_DENSE_STEREO_H__
+#ifndef __OPENCV_QUASI_DENSE_STEREO_H__
+#define __OPENCV_QUASI_DENSE_STEREO_H__
+
+
+
+#include <opencv2/core.hpp>
+
+
+namespace cv
+{
+namespace stereo
+{
+/** \addtogroup stereo
+ *  @{
+ */
+
+
+// A basic match structure
+struct CV_EXPORTS Match
+{
+    cv::Point2i p0;
+    cv::Point2i p1;
+    float	corr;
+
+    bool operator < (const Match & rhs) const//fixme  may be used uninitialized in this function
+    {
+        return this->corr < rhs.corr;
+    }
+};
+struct CV_EXPORTS PropagationParameters
+{
+    int	corrWinSizeX;			// similarity window
+    int	corrWinSizeY;
+
+    int borderX;					// border to ignore
+    int borderY;
+
+    //matching
+    float correlationThreshold;	// correlation threshold
+    float textrureThreshold;		// texture threshold
+
+    int	  neighborhoodSize;		// neighborhood size
+    int	  disparityGradient;	// disparity gradient threshold
+
+    // Parameters for LK flow algorithm
+    int lkTemplateSize;
+    int lkPyrLvl;
+    int lkTermParam1;
+    float lkTermParam2;
+
+    // Parameters for GFT algorithm.
+    float gftQualityThres;
+    int gftMinSeperationDist;
+    int gftMaxNumFeatures;
+
+};
+
+
+/**
+ * @brief Class containing the methods needed for Quasi Dense Stereo computation.
+ *
+ * This module contains the code to perform quasi dense stereo matching.
+ * The method initially starts with a sparse 3D reconstruction based on feature matching across a
+ * stereo image pair and subsequently propagates the structure into neighboring image regions.
+ * To obtain initial seed correspondences, the algorithm locates Shi and Tomashi features in the
+ * left image of the stereo pair and then tracks them using pyramidal Lucas-Kanade in the right image.
+ * To densify the sparse correspondences, the algorithm computes the zero-mean normalized
+ * cross-correlation (ZNCC) in small patches around every seed pair and uses it as a quality metric
+ * for each match. In this code, we introduce a custom structure to store the location and ZNCC value
+ * of correspondences called "Match". Seed Matches are stored in a priority queue sorted according to
+ * their ZNCC value, allowing for the best quality Match to be readily available. The algorithm pops
+ * Matches and uses them to extract new matches around them. This is done by considering a small
+ * neighboring area around each Seed and retrieving correspondences above a certain texture threshold
+ * that are not previously computed. New matches are stored in the seed priority queue and used as seeds.
+ * The propagation process ends when no additional matches can be retrieved.
+ *
+ *
+ * @sa This code represents the work presented in @cite Stoyanov2010.
+ * If this code is useful for your work please cite @cite Stoyanov2010.
+ *
+ * Also the original growing scheme idea is described in @cite Lhuillier2000
+ *
+ */
+
+class CV_EXPORTS QuasiDenseStereo
+{
+public:
+    /**
+     * @brief destructor
+     * Method to free all the memory allocated by matrices and vectors in this class.
+     */
+    virtual ~QuasiDenseStereo() = 0;
+
+
+    /**
+     * @brief Load a file containing the configuration parameters of the class.
+     * @param[in] filepath The location of the .YAML file containing the configuration parameters.
+     * @note default value is an empty string in which case the default parameters will be loaded.
+     * @retval 1: If the path is not empty and the program loaded the parameters successfully.
+     * @retval 0: If the path is empty and the program loaded default parameters.
+     * @retval -1: If the file location is not valid or the program could not open the file and
+     * loaded default parameters from defaults.hpp.
+     * @note The method is automatically called in the constructor and configures the class.
+     * @note Loading different parameters will have an effect on the output. This is useful for tuning
+     * in case of video processing.
+     * @sa loadParameters
+     */
+    virtual int loadParameters(cv::String filepath) = 0;
+
+
+    /**
+     * @brief Save a file containing all the configuration parameters the class is currently set to.
+     * @param[in] filepath The location to store the parameters file.
+     * @note Calling this method with no arguments will result in storing class parameters to a file
+     * names "qds_parameters.yaml" in the root project folder.
+     * @note This method can be used to generate a template file for tuning the class.
+     * @sa loadParameters
+     */
+    virtual int saveParameters(cv::String filepath) = 0;
+
+
+    /**
+     * @brief Get The sparse corresponding points.
+     * @param[out] sMatches A vector containing all sparse correspondences.
+     * @note The method clears the sMatches vector.
+     * @note The returned Match elements inside the sMatches vector, do not use corr member.
+     */
+    virtual void getSparseMatches(std::vector<stereo::Match> &sMatches) = 0;
+
+
+    /**
+     * @brief Get The dense corresponding points.
+     * @param[out] denseMatches A vector containing all dense matches.
+     * @note The method clears the denseMatches vector.
+     * @note The returned Match elements inside the sMatches vector, do not use corr member.
+     */
+    virtual void getDenseMatches(std::vector<stereo::Match> &denseMatches) = 0;
+
+
+
+    /**
+     * @brief Main process of the algorithm. This method computes the sparse seeds and then densifies them.
+     *
+     * Initially input images are converted to gray-scale and then the sparseMatching method
+     * is called to obtain the sparse stereo. Finally quasiDenseMatching is called to densify the corresponding
+     * points.
+     * @param[in] imgLeft The left Channel of a stereo image pair.
+     * @param[in] imgRight The right Channel of a stereo image pair.
+     * @note If input images are in color, the method assumes that are BGR and converts them to grayscale.
+     * @sa sparseMatching
+     * @sa quasiDenseMatching
+     */
+    virtual void process(const cv::Mat &imgLeft ,const cv::Mat &imgRight) = 0;
+
+
+    /**
+     * @brief Specify pixel coordinates in the left image and get its corresponding location in the right image.
+     * @param[in] x The x pixel coordinate in the left image channel.
+     * @param[in] y The y pixel coordinate in the left image channel.
+     * @retval cv::Point(x, y) The location of the corresponding pixel in the right image.
+     * @retval cv::Point(0, 0) (NO_MATCH)  if no match is found in the right image for the specified pixel location in the left image.
+     * @note This method should be always called after process, otherwise the matches will not be correct.
+     */
+    virtual cv::Point2f getMatch(const int x, const int y) = 0;
+
+
+    /**
+     * @brief Compute and return the disparity map based on the correspondences found in the "process" method.
+     * @param[in] disparityLvls The level of detail in output disparity image.
+     * @note Default level is 50
+     * @return cv::Mat containing a the disparity image in grayscale.
+     * @sa computeDisparity
+     * @sa quantizeDisparity
+     */
+    virtual cv::Mat getDisparity(uint8_t disparityLvls=50) = 0;
+
+
+    static cv::Ptr<QuasiDenseStereo> create(cv::Size monoImgSize, cv::String paramFilepath = cv::String());
+
+
+    PropagationParameters Param;
+};
+
+} //namespace cv
+} //namespace stereo
+
+/** @}*/
+
+#endif // __OPENCV_QUASI_DENSE_STEREO_H__

modules/stereo/samples/dense_disparity.cpp

+#include <opencv2/core.hpp>
+#include <opencv2/highgui.hpp>
+#include <fstream>
+#include <opencv2/stereo.hpp>
+
+
+
+
+using namespace cv;
+using namespace std;
+
+
+int main()
+{
+//!     [load]
+    cv::Mat rightImg, leftImg;
+    leftImg = imread("./imgLeft.png", IMREAD_COLOR);
+    rightImg = imread("./imgRight.png", IMREAD_COLOR);
+//!     [load]
+
+
+//!     [create]
+    cv::Size frameSize = leftImg.size();
+    Ptr<stereo::QuasiDenseStereo> stereo = stereo::QuasiDenseStereo::create(frameSize);
+//!     [create]
+
+
+//!     [process]
+    stereo->process(leftImg, rightImg);
+//!     [process]
+
+
+//!     [disp]
+    uint8_t displvl = 80;
+    cv::Mat disp;
+    disp = stereo->getDisparity(displvl);
+    cv::namedWindow("disparity map");
+    cv::imshow("disparity map", disp);
+//!     [disp]
+
+
+    cv::namedWindow("right channel");
+    cv::namedWindow("left channel");
+    cv::imshow("left channel", leftImg);
+    cv::imshow("right channel", rightImg);
+
+
+//!     [export]
+    vector<stereo::Match> matches;
+    stereo->getDenseMatches(matches);
+    std::ofstream dense("./dense.txt", std::ios::out);
+    for (uint i=0; i< matches.size(); i++)
+    {
+        dense << matches[i].p0 << matches[i].p1 << endl;
+    }
+    dense.close();
+//!     [export]
+
+
+
+    cv::waitKey(0);
+
+    return 0;
+}

modules/stereo/samples/export_param_file.cpp

+#include <opencv2/core.hpp>
+#include <opencv2/stereo.hpp>
+
+using namespace cv;
+using namespace std;
+
+int main(int argc, char* argv[])
+{
+//!     [create]
+    Ptr<stereo::QuasiDenseStereo> stereo =  stereo::QuasiDenseStereo::create(cv::Size(5,5));
+//!     [create]
+
+
+//!     [write]
+    std::string parameterFileLocation = "./parameters.yaml";
+    if (argc > 1)
+        parameterFileLocation = argv[1];
+    stereo->saveParameters(parameterFileLocation);
+//!     [write]
+
+    return 0;
+}

modules/stereo/tutorials/qds_export_parameters/qds_export_parameters.markdown

+Exporting a template parameter file {#tutorial_qds_export_parameters}
+==================
+
+Goal
+----
+
+In this tutorial you will learn how to
+
+-   create a simple parameter file template.
+
+@include ./samples/export_param_file.cpp
+
+## Explanation:
+
+The class supports loading configuration parameters from a .yaml file using the method `loadParameters()`.
+This is very useful for fine-tuning the class' parameters on the fly. To extract a template of this
+parameter file you run the following code.
+
+We create an instance of a `QuasiDenseStereo` object. Not specifying the second argument of the constructor,
+makes the object to load default parameters.
+@snippet ./samples/export_param_file.cpp create
+By calling the method `saveParameters()`, we store the template file to the location specified by `parameterFileLocation`
+@snippet ./samples/export_param_file.cpp write

modules/stereo/tutorials/qds_quasi_dense_stereo/qds_quasi_dense_stereo.markdown

+Quasi dense Stereo {#tutorial_qds_quasi_dense_stereo}
+==================
+
+Goal
+----
+
+In this tutorial you will learn how to
+
+-   Configure a QuasiDenseStero object
+-   Compute dense Stereo correspondences.
+
+@include ./samples/dense_disparity.cpp
+
+## Explanation:
+
+The program loads a stereo image pair.
+
+After importing the images.
+@snippet ./samples/dense_disparity.cpp load
+We need to know the frame size of a single image, in order to create an instance of a `QuasiDesnseStereo` object.
+
+@snippet ./samples/dense_disparity.cpp create
+
+Because we didn't specify the second argument in the constructor, the `QuasiDesnseStereo` object will
+load default parameters.
+
+We can then pass the imported stereo images in the process method like this
+@snippet ./samples/dense_disparity.cpp process
+
+The process method contains most of the functionality of the class and does two main things.
+-   Computes a sparse stereo based in "Good Features to Track" and "pyramidal Lucas-Kanade" flow algorithm
+-   Based on those sparse stereo points, densifies the stereo correspondences using Quasi Dense Stereo method.
+
+After the execution of `process()` we can display the disparity Image of the stereo.
+@snippet ./samples/dense_disparity.cpp disp
+
+
+At this point we can also extract all the corresponding points using `getDenseMatches()` method and export them in a file.
+@snippet ./samples/dense_disparity.cpp export

modules/stereo/tutorials/table_of_content_quasi_dense_stereo.markdown

+Quasi Dense Stereo (stereo module) {#tutorial_table_of_content_quasi_dense_stereo}
+==========================================================
+
+Quasi Dense Stereo is method for performing dense stereo matching. `QuasiDenseStereo` implements this process.
+The code uses pyramidal Lucas-Kanade with Shi-Tomasi features to get the initial seed correspondences.
+Then these seeds are propagated by using mentioned growing scheme.
+
+-   @subpage tutorial_qds_quasi_dense_stereo
+
+    Example showing how to get dense correspondences from a stereo image pair.
+
+-   @subpage tutorial_qds_export_parameters
+
+    Example showing how to genereate a parameter file template.