###press q to start, when all images are created in each class folder, you should copy all images from ../data/images_ape, ../data/images_ant, ../data/images_cow and ../data/images_plane into ../data/images_all folder as a collection of images for network tranining and feature extraction, when all images are copyed correctlly, proceed on.
###press q to start, when all images are created in images_all folder as a collection of images for network tranining and feature extraction, then proceed on.
###After this demo, the binary files of images and labels will be stored as 'binary_image' and 'binary_label' in current path, you should copy them into the leveldb folder in Caffe triplet training, for example: copy these 2 files in <caffe_source_directory>/data/linemod and rename them as 'binary_image_train', 'binary_image_test' and 'binary_label_train', 'binary_label_train'.
###We could start triplet tranining using Caffe
```
...
...
@@ -70,19 +70,22 @@ $ cd <opencv_contrib>/modules/cnn_3dobj/samples/build
#Demo2:
###Convert data into leveldb format from folder ../data/images_all for feature extraction afterwards. The leveldb files including all data will be stored in ../data/dbfile. If you will use the OpenCV defined feature extraction process, you could also skip Demo2 for data converting, just run Demo3 after Demo1 for feature extraction because Demo3 also includes the db file converting process before feature extraction.
###feature extraction, this demo will convert a set of images in a particular path into leveldb database for feature extraction using Caffe.
###feature extraction, this demo will convert a set of images in a particular path into leveldb database for feature extraction using Caffe and outputting a binary file including all extracted feature.
```
$ ./feature_extract_test
```
###This will extract feature from a set of images in a folder as vector<cv::Mat> for further classification and a binary file with containing all feature vectors of each sample. Pay attention: if it's warning you that 'Check failed: leveldb::DB::Open(options, outputdb, &db).ok()', the reason is that there is alreay leveldb files in ../data/dbfile as previous running of Demo2 or Demo3, just delete all files in ../data/dbfile and run Demo3 again.
###This will extract feature from a set of images in a folder as vector<cv::Mat> for further classification and a binary file with containing all feature vectors of each sample.
###After running this, you will get a binary file storing features in ../data/feature folder, I can provide a Matlab script reading this file if someone need it. If you don't need the binary file, the feature could also be stored in vector<cv::Mat> for directly classification using the softmax layer as shown in Demo4.
==============
#Demo4:
###Classifier
###Classifier, this will extracting the feature of a single image and compare it with features of gallery samples for prediction. Just run:
@@ -60,10 +60,6 @@ the use of this software, even if advised of the possibility of such damage.
#include <glog/logging.h>
#include <google/protobuf/text_format.h>
#include <leveldb/db.h>
//#include <opencv2/calib3d/calib3d.hpp>
#include <opencv2/viz/vizcore.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/highgui/highgui_c.h>
#define CPU_ONLY
#include <caffe/blob.hpp>
#include <caffe/common.hpp>
...
...
@@ -71,6 +67,10 @@ the use of this software, even if advised of the possibility of such damage.
#include <caffe/proto/caffe.pb.h>
#include <caffe/util/io.hpp>
#include <caffe/vision_layers.hpp>
#include "opencv2/viz/vizcore.hpp"
#include "opencv2/highgui.hpp"
#include "opencv2/highgui/highgui_c.h"
#include "opencv2/imgproc.hpp"
usingstd::string;
usingcaffe::Blob;
usingcaffe::Caffe;
...
...
@@ -161,6 +161,38 @@ class CV_EXPORTS_W DataTrans
/** @brief Extract feature into a binary file and vector<cv::Mat> for classification, the model proto and network proto are needed, All images in the file root will be used for feature extraction.
*/
};
classCV_EXPORTS_WClassification
{
private:
caffe::shared_ptr<caffe::Net<float>>net_;
cv::Sizeinput_geometry_;
intnum_channels_;
cv::Matmean_;
std::vector<string>labels_;
voidSetMean(conststring&mean_file);
/** @brief Load the mean file in binaryproto format.
/** @brief Wrap the input layer of the network in separate cv::Mat objects(one per channel). This way we save one memcpy operation and we don't need to rely on cudaMemcpy2D. The last preprocessing operation will write the separate channels directly to the input layer.
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials provided
* with the distribution.
* * Neither the name of Willow Garage, Inc. nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
* FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
* COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
* BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
* CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
* ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
*/
#include <opencv2/cnn_3dobj.hpp>
#include <iomanip>
usingnamespacecv;
usingnamespacestd;
usingnamespacecv::cnn_3dobj;
intmain(intargc,char**argv)
{
constStringkeys="{help | | this demo will convert a set of images in a particular path into leveldb database for feature extraction using Caffe.}"
"{src_dir | ../data/images_all/ | Source direction of the images ready for being converted to leveldb dataset.}"
"{src_dst | ../data/dbfile | Aim direction of the converted to leveldb dataset. }"
"{attach_dir | ../data/dbfile | Path for saving additional files which describe the transmission results. }"
"{channel | 1 | Channel of the images. }"
"{width | 64 | Width of images}"
"{height | 64 | Height of images}"
"{caffemodel | ../data/3d_triplet_iter_10000.caffemodel | caffe model for feature exrtaction.}"
"{network_forDB | ../data/3d_triplet_galleryIMG.prototxt | Network definition file used for extracting feature from levelDB data, causion: the path of levelDB training samples must be wrotten in in .prototxt files in Phase TEST}"
"{save_feature_dataset_names | ../data/feature/feature_iter_10000.bin | Output of the extracted feature in form of binary files together with the vector<cv::Mat> features as the feature.}"
"{extract_feature_blob_names | feat | Layer used for feature extraction in CNN.}"
"{num_mini_batches | 4 | Batches suit for the batches defined in the .proto for the aim of extracting feature from all images.}"
"{device | CPU | Device: CPU or GPU.}"
"{dev_id | 0 | ID of GPU.}"
"{network_forIMG | ../data/3d_triplet_testIMG.prototxt | Network definition file used for extracting feature from a single image and making a classification}"
"{mean_file | ../data/images_mean/triplet_mean.binaryproto | The mean file generated by Caffe from all gallery images, this could be used for mean value substraction from all images.}"
"{label_file | ../data/dbfileimage_filename | A namelist including all gallery images.}"
"{target_img | ../data/images_all/2_13.png | Path of image waiting to be classified.}"
"{num_candidate | 6 | Number of candidates in gallery as the prediction result.}";
cv::CommandLineParserparser(argc,argv,keys);
parser.about("Demo for Sphere View data generation");
std::cout<<std::endl<<"All images in: "<<std::endl<<src_dir<<std::endl<<"have been converted to levelDB data in: "<<std::endl<<src_dst<<std::endl<<"for extracting feature of gallery images in classification step efficiently, this convertion is not needed in feature extraction of test image"<<std::endl;
@@ -66,11 +66,11 @@ int main(int argc, char* argv[])
"{channel | 1 | Channel of the images. }"
"{width | 64 | Width of images}"
"{height | 64 | Height of images}"
"{pretrained_binary_proto | ../data/3d_triplet_iter_10000.caffemodel | caffe model for feature exrtaction.}"
"{feature_extraction_proto | ../data/3d_triplet_train_test.prototxt | network definition in .prototxt the path of the training samples must be wrotten in in .prototxt files in Phase TEST}"
"{save_feature_dataset_names | ../data/feature/feature_iter_10000.bin | the output of the extracted feature in form of binary files together with the vector<cv::Mat> features as the feature.}"
"{caffemodel | ../data/3d_triplet_iter_10000.caffemodel | caffe model for feature exrtaction.}"
"{network_forDB | ../data/3d_triplet_galleryIMG.prototxt | network definition in .prototxt the path of the training samples must be wrotten in in .prototxt files in Phase TEST}"
"{featurename_bin | ../data/feature/feature_iter_10000.bin | the output of the extracted feature in form of binary files together with the vector<cv::Mat> features as the feature.}"
"{extract_feature_blob_names | feat | the layer used for feature extraction in CNN.}"
"{num_mini_batches | 6 | batches suit for the batches defined in the .proto for the aim of extracting feature from all images.}"
"{num_mini_batches | 4 | batches suit for the batches defined in the .proto for the aim of extracting feature from all images.}"
"{device | CPU | device}"
"{dev_id | 0 | dev_id}";
cv::CommandLineParserparser(argc,argv,keys);
...
...
@@ -86,14 +86,16 @@ int main(int argc, char* argv[])
std::cout<<std::endl<<"All images in: "<<std::endl<<src_dir<<std::endl<<"have been converted to levelDB data in: "<<std::endl<<src_dst<<std::endl<<"for extracting feature of gallery images in classification step efficiently, this convertion is not needed in feature extraction of test image"<<std::endl;
std::cout<<std::endl<<"All featrues of images in: "<<std::endl<<src_dir<<std::endl<<"have been extracted as binary file(using levelDB data) in:"<<std::endl<<featurename_bin<<std::endl<<"for analysis in Matlab and other software, this function also outputting a vector<cv::Mat> format gallery feature used for classificatioin.";