Primal-dual algorithm

This is an implementation of primal-dual algorithm, based on the C++
source code by Vadim Pisarevsky. It was extended to handle the denoising
based on multiple observations. It also contains documentation and
tests.

modules/optim/doc/optim.rst

@@ -9,3 +9,4 @@ optim. Generic numerical optimization
 
     linear_programming
     downhill_simplex_method
+    primal_dual_algorithm

modules/optim/doc/primal_dual_algorithm.rst

+Primal-Dual Algorithm
+=======================
+
+.. highlight:: cpp
+
+optim::denoise_TVL1
+---------------------------------
+
+Primal-dual algorithm is an algorithm for solving special types of variational
+problems (that is, finding a function to minimize some functional)
+. As the image denoising, in particular, may be seen as the variational
+problem, primal-dual algorithm then can be used to perform denoising and this
+is exactly what is implemented.
+
+It should be noted, that this implementation was taken from the July 2013 blog entry [Mordvintsev]_, which also contained 
+(slightly more general) ready-to-use
+source code on Python. Subsequently, that code was rewritten on C++ with the usage of openCV by Vadim Pisarevsky
+at the end of July 2013 and finally it was slightly adapted by later authors.
+
+Although the thorough discussion and justification
+of the algorithm involved may be found in [ChambolleEtAl]_, it might make sense to skim over it here, following [Mordvintsev]_. To
+begin with, we consider the 1-byte gray-level images as the functions from the rectangular domain of pixels
+(it may be seen as set :math:`\left\{(x,y)\in\mathbb{N}\times\mathbb{N}\mid 1\leq x\leq n,\;1\leq y\leq m\right\}`
+for some :math:`m,\;n\in\mathbb{N}`) into :math:`\{0,1,\dots,255\}`. We shall denote the noised images as :math:`f_i` and with this
+view, given some image :math:`x` of the same size, we may measure how bad it is by the formula
+
+.. math::
+        \left\|\left\|\nabla x\right\|\right\| + \lambda\sum_i\left\|\left\|x-f_i\right\|\right\|
+
+:math:`\|\|\cdot\|\|` here denotes :math:`L_2`-norm and as you see, the first addend states that we want our image to be smooth
+(ideally, having zero gradient, thus being constant) and the second states that we want our result to be close to the observations we've got.
+If we treat :math:`x` as a function, this is exactly the functional what we seek to minimize and here the Primal-Dual algorithm comes
+into play.
+
+.. ocv:function:: void optim::denoise_TVL1(const std::vector<Mat>& observations,Mat& result, double lambda, int niters)
+
+    :param observations: This array should contain one or more noised versions of the image that is to be restored.
+
+    :param result: Here the denoised image will be stored. There is no need to do pre-allocation of storage space, as it will be automatically allocated, if necessary.
+
+    :param lambda: Corresponds to :math:`\lambda` in the formulas above. As it is enlarged, the smooth (blurred) images are treated
+    more favorably than detailed (but maybe more noised) ones. Roughly speaking, as it becomes smaller, the result will be
+    more blur but more sever outliers will be removed.
+    
+
+    :param niters: Number of iterations that the algorithm will run. Of course, as more iterations as better, but it is hard to quantitatively refine this statement, so just use the default and increase it if the results are poor.
+
+
+.. [ChambolleEtAl] A. Chambolle, V. Caselles, M. Novaga, D. Cremers and T. Pock, An Introduction to Total Variation for Image Analysis, http://hal.archives-ouvertes.fr/docs/00/43/75/81/PDF/preprint.pdf (pdf)
+
+.. [Mordvintsev] Alexander Mordvintsev, ROF and TV-L1 denoising with Primal-Dual algorithm, http://znah.net/rof-and-tv-l1-denoising-with-primal-dual-algorithm.html (blog entry)

modules/optim/include/opencv2/optim.hpp

@@ -96,6 +96,7 @@ enum
 };
 
 CV_EXPORTS_W int solveLP(const Mat& Func, const Mat& Constr, Mat& z);
+CV_EXPORTS_W void denoise_TVL1(const std::vector<Mat>& observations,Mat& result, double lambda=1.0, int niters=30);
 }}// cv
 
 #endif

modules/optim/src/denoise_tvl1.cpp

+#include "precomp.hpp"
+#define ALEX_DEBUG
+#include "debug.hpp"
+#include <vector>
+#include <algorithm>
+
+#define ABSCLIP(val,threshold) MIN(MAX((val),-(threshold)),(threshold))
+
+namespace cv{namespace optim{
+    
+    class AddFloatToCharScaled{
+        public:
+            AddFloatToCharScaled(float scale):_scale(scale){}
+            inline float operator()(float a,uchar b){
+                return a+_scale*((float)b);
+            }
+        private:
+            float _scale;
+    };
+
+    void solve_TVL1(const Mat& img, Mat& res, double _clambda, int niters)
+    {
+        const float L2 = 8.0f, tau = 0.02f, sigma = 1./(L2*tau), theta = 1.f, img_scale = 1.f/255;
+        float clambda = (float)_clambda, threshold = clambda*tau;
+        const int workdepth = CV_32F;
+
+        int i, x, y, rows=img.rows, cols=img.cols;
+
+        Mat X, P = Mat::zeros(rows, cols, CV_MAKETYPE(workdepth, 2));
+        img.convertTo(X, workdepth, 1./255);
+
+        for( i = 0; i < niters; i++ )
+        {
+            float currsigma = i == 0 ? 1 + sigma : sigma;
+
+            // P_ = P + sigma*nabla(X)
+            // P(x,y) = P_(x,y)/max(||P(x,y)||,1)
+            for( y = 0; y < rows; y++ )
+            {
+                const float* x_curr = X.ptr<float>(y);
+                const float* x_next = X.ptr<float>(std::min(y+1, rows-1));
+                Point2f* p_curr = P.ptr<Point2f>(y);
+                float dx, dy, m;
+                for( x = 0; x < cols-1; x++ )
+                {
+                    dx = (x_curr[x+1] - x_curr[x])*currsigma + p_curr[x].x;
+                    dy = (x_next[x] - x_curr[x])*currsigma + p_curr[x].y;
+                    m = 1.f/std::max(std::sqrt(dx*dx + dy*dy), 1.f);
+                    p_curr[x].x = dx*m;
+                    p_curr[x].y = dy*m;
+                }
+                dy = (x_next[x] - x_curr[x])*currsigma + p_curr[x].y;
+                m = 1.f/std::max(std::abs(dy), 1.f);
+                p_curr[x].x = 0.f;
+                p_curr[x].y = dy*m;
+            }
+
+            // X1 = X + tau*(-nablaT(P))
+            // X2 = X1 + clip(img - X1, -clambda*tau, clambda*tau)
+            // X = X2 + theta*(X2 - X)
+            for( y = 0; y < rows; y++ )
+            {
+                const uchar* img_curr = img.ptr<uchar>(y);
+                float* x_curr = X.ptr<float>(y);
+                const Point2f* p_curr = P.ptr<Point2f>(y);
+                const Point2f* p_prev = P.ptr<Point2f>(std::max(y - 1, 0));
+
+                x = 0;
+                float x_new = x_curr[x] + tau*(p_curr[x].y - p_prev[x].y);
+                x_new += std::min(std::max(img_curr[x]*img_scale - x_new, -threshold), threshold);
+                x_curr[x] = x_new + theta*(x_new - x_curr[x]);
+
+                for( x = 1; x < cols; x++ )
+                {
+                    x_new = x_curr[x] + tau*(p_curr[x].x - p_curr[x-1].x + p_curr[x].y - p_prev[x].y);
+                    x_new += std::min(std::max(img_curr[x]*img_scale - x_new, -threshold), threshold);
+                    x_curr[x] = x_new + theta*(x_new - x_curr[x]);
+                }
+            }
+        }
+
+        res.create(X.rows,X.cols,CV_8U);
+        X.convertTo(res, CV_8U, 255);
+    }
+
+    void denoise_TVL1(const std::vector<Mat>& observations,Mat& result, double lambda, int niters){
+
+        CV_Assert(observations.size()>0 && niters>0 && lambda>0);
+#if 0
+        solve_TVL1(observations[0],result,lambda,niters);
+        return;
+#endif
+
+        const float L2 = 8.0f, tau = 0.02f, sigma = 1./(L2*tau), theta = 1.f, img_scale = 1.f/255;
+        float clambda = (float)lambda, threshold = clambda*tau;
+        float s=0;
+        const int workdepth = CV_32F;
+
+        int i, x, y, rows=observations[0].rows, cols=observations[0].cols,count;
+        for(i=1;i<observations.size();i++){
+            CV_Assert(observations[i].rows==rows && observations[i].cols==cols);
+        }
+
+        Mat X, P = Mat::zeros(rows, cols, CV_MAKETYPE(workdepth, 2));
+        observations[0].convertTo(X, workdepth, 1./255);
+        std::vector< Mat_<float> > Rs(observations.size());
+        for(count=0;count<Rs.size();count++){
+            Rs[count]=Mat::zeros(rows,cols,workdepth);
+        }
+
+        for( i = 0; i < niters; i++ )
+        {
+            float currsigma = i == 0 ? 1 + sigma : sigma;
+
+            // P_ = P + sigma*nabla(X)
+            // P(x,y) = P_(x,y)/max(||P(x,y)||,1)
+            for( y = 0; y < rows; y++ )
+            {
+                const float* x_curr = X.ptr<float>(y);
+                const float* x_next = X.ptr<float>(std::min(y+1, rows-1));
+                Point2f* p_curr = P.ptr<Point2f>(y);
+                float dx, dy, m;
+                for( x = 0; x < cols-1; x++ )
+                {
+                    dx = (x_curr[x+1] - x_curr[x])*currsigma + p_curr[x].x;
+                    dy = (x_next[x] - x_curr[x])*currsigma + p_curr[x].y;
+                    m = 1.f/std::max(std::sqrt(dx*dx + dy*dy), 1.f);
+                    p_curr[x].x = dx*m;
+                    p_curr[x].y = dy*m;
+                }
+                dy = (x_next[x] - x_curr[x])*currsigma + p_curr[x].y;
+                m = 1.f/std::max(std::abs(dy), 1.f);
+                p_curr[x].x = 0.f;
+                p_curr[x].y = dy*m;
+            }
+
+
+            //Rs = clip(Rs + sigma*(X-imgs), -clambda, clambda)
+            for(count=0;count<Rs.size();count++){
+                std::transform<MatIterator_<float>,MatConstIterator_<uchar>,MatIterator_<float>,AddFloatToCharScaled>(
+                        Rs[count].begin(),Rs[count].end(),observations[count].begin<uchar>(),
+                        Rs[count].begin(),AddFloatToCharScaled(-sigma/255.0));
+                Rs[count]+=sigma*X;
+                min(Rs[count],clambda,Rs[count]);
+                max(Rs[count],-clambda,Rs[count]);
+            }
+
+            for( y = 0; y < rows; y++ )
+            {
+                const uchar* img_curr = observations[0].ptr<uchar>(y);
+                float* x_curr = X.ptr<float>(y);
+                const Point2f* p_curr = P.ptr<Point2f>(y);
+                const Point2f* p_prev = P.ptr<Point2f>(std::max(y - 1, 0));
+
+                // X1 = X + tau*(-nablaT(P))
+                x = 0;
+                s=0.0;
+                for(count=0;count<Rs.size();count++){
+                    s=s+Rs[count](y,x);
+                }
+                float x_new = x_curr[x] + tau*(p_curr[x].y - p_prev[x].y)-tau*s;
+                    // X = X2 + theta*(X2 - X)
+                x_curr[x] = x_new + theta*(x_new - x_curr[x]);
+           
+
+                for(x = 1; x < cols; x++ )
+                {
+                    s=0.0;
+                    for(count=0;count<Rs.size();count++){
+                        s+=Rs[count](y,x);
+                    }
+                        // X1 = X + tau*(-nablaT(P))
+                    x_new = x_curr[x] + tau*(p_curr[x].x - p_curr[x-1].x + p_curr[x].y - p_prev[x].y)-tau*s;
+                        // X = X2 + theta*(X2 - X)
+                    x_curr[x] = x_new + theta*(x_new - x_curr[x]);
+                }
+            }
+        }
+
+        result.create(X.rows,X.cols,CV_8U);
+        X.convertTo(result, CV_8U, 255);
+    }
+}}

modules/optim/test/test_denoise_tvl1.cpp

+#include "test_precomp.hpp"
+#include "opencv2/highgui.hpp"
+
+void make_noisy(const cv::Mat& img, cv::Mat& noisy, double sigma, double pepper_salt_ratio,cv::RNG& rng){
+    noisy.create(img.size(), img.type());
+    cv::Mat noise(img.size(), img.type()), mask(img.size(), CV_8U);
+    rng.fill(noise,cv::RNG::NORMAL,128.0,sigma);
+    cv::addWeighted(img, 1, noise, 1, -128, noisy);
+    cv::randn(noise, cv::Scalar::all(0), cv::Scalar::all(2));
+    noise *= 255;
+    cv::randu(mask, 0, round(1./pepper_salt_ratio));
+    cv::Mat half = mask.colRange(0, img.cols/2);
+    half = cv::Scalar::all(1);
+    noise.setTo(128, mask);
+    cv::addWeighted(noisy, 1, noise, 1, -128, noisy);
+}
+void make_spotty(cv::Mat& img,cv::RNG& rng, int r=3,int n=1000){
+    for(int i=0;i<n;i++){
+        int x=rng(img.cols-r),y=rng(img.rows-r);
+        if(rng(2)==0){
+            img(cv::Range(y,y+r),cv::Range(x,x+r))=(uchar)0;
+        }else{
+            img(cv::Range(y,y+r),cv::Range(x,x+r))=(uchar)255;
+        }
+    }
+}
+
+bool validate_pixel(const cv::Mat& image,int x,int y,uchar val){
+    printf("test: image(%d,%d)=%d vs %d - %s\n",x,y,(int)image.at<uchar>(x,y),val,(val==image.at<uchar>(x,y))?"true":"false");
+    return (image.at<uchar>(x,y)==val);
+}
+
+TEST(Optim_denoise_tvl1, regression_basic){
+    cv::RNG rng(42);
+    cv::Mat img = cv::imread("lena.jpg", 0), noisy,res;
+    if(img.rows!=512 || img.cols!=512){
+        printf("\tplease, put lena.jpg from samples/c in the current folder\n");
+        printf("\tnow, the test will fail...\n");
+        ASSERT_TRUE(false);
+    }
+
+    const int obs_num=5;
+    std::vector<cv::Mat> images(obs_num,cv::Mat());
+    for(int i=0;i<images.size();i++){
+        make_noisy(img,images[i], 20, 0.02,rng);
+        //make_spotty(images[i],rng);
+    }
+
+    //cv::imshow("test", images[0]);
+    cv::optim::denoise_TVL1(images, res);
+    printf("hello here!\n");
+    //cv::imshow("denoised", res);
+    //cv::waitKey();
+
+#if 0
+    ASSERT_TRUE(validate_pixel(res,248,334,179));
+    ASSERT_TRUE(validate_pixel(res,489,333,172));
+    ASSERT_TRUE(validate_pixel(res,425,507,104));
+    ASSERT_TRUE(validate_pixel(res,489,486,105));
+    ASSERT_TRUE(validate_pixel(res,223,208,64));
+    ASSERT_TRUE(validate_pixel(res,418,3,78));
+    ASSERT_TRUE(validate_pixel(res,63,76,97));
+    ASSERT_TRUE(validate_pixel(res,29,134,126));
+    ASSERT_TRUE(validate_pixel(res,219,291,174));
+    ASSERT_TRUE(validate_pixel(res,384,124,76));
+#endif
+
+#if 1
+    ASSERT_TRUE(validate_pixel(res,248,334,194));
+    ASSERT_TRUE(validate_pixel(res,489,333,171));
+    ASSERT_TRUE(validate_pixel(res,425,507,103));
+    ASSERT_TRUE(validate_pixel(res,489,486,109));
+    ASSERT_TRUE(validate_pixel(res,223,208,72));
+    ASSERT_TRUE(validate_pixel(res,418,3,58));
+    ASSERT_TRUE(validate_pixel(res,63,76,93));
+    ASSERT_TRUE(validate_pixel(res,29,134,127));
+    ASSERT_TRUE(validate_pixel(res,219,291,180));
+    ASSERT_TRUE(validate_pixel(res,384,124,80));
+#endif
+
+}