Using [GPGPU](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units)
via OpenCL for applications performance enhancements is quite a modern trend now.
Some CV algo-s (e.g. image filtering) run much faster on a GPU than on a CPU.
Recently it has become possible on Android OS.
The most popular CV application scenario for an Android operated device is starting camera in preview mode, applying some CV algo to every frame
and displaying the preview frames modified by that CV algo.
Let's consider how we can use OpenCL in this scenario. In particular let's try two ways: direct calls to OpenCL API and recently introduced OpenCV T-API
(aka [Transparent API](https://docs.google.com/presentation/d/1qoa29N_B-s297-fp0-b3rBirvpzJQp8dCtllLQ4DVCY/present)) - implicit OpenCL accelerations of some OpenCV algo-s.
Application structure
---------------------
Starting Android API level 11 (Android 3.0) [Camera API](http://developer.android.com/reference/android/hardware/Camera.html)
allows use of OpenGL texture as a target for preview frames.
Android API level 21 brings a new [Camera2 API](http://developer.android.com/reference/android/hardware/camera2/package-summary.html)
that provides much more control over the camera settings and usage modes,
it allows several targets for preview frames and OpenGL texture in particular.
Having a preview frame in an OpenGL texture is a good deal for using OpenCL because there is an
[OpenGL-OpenCL Interoperability API (cl_khr_gl_sharing)](https://www.khronos.org/registry/cl/sdk/1.2/docs/man/xhtml/cl_khr_gl_sharing.html),
allowing sharing OpenGL texture data with OpenCL functions without copying (with some restrictions of course).
Let's create a base for our application that just configures Android camera to send preview frames to OpenGL texture and displays these frames
on display without any processing.
A minimal `Activity` class for that purposes looks like following:
public void surfaceCreated(SurfaceHolder holder) {
super.surfaceCreated(holder);
}
@Override
public void surfaceDestroyed(SurfaceHolder holder) {
super.surfaceDestroyed(holder);
}
@Override
public void surfaceChanged(SurfaceHolder holder, int format, int w, int h) {
super.surfaceChanged(holder, format, w, h);
}
@Override
public void onResume() {
super.onResume();
mRenderer.onResume();
}
@Override
public void onPause() {
mRenderer.onPause();
super.onPause();
}
}
@endcode
__Note__: we use two renderer classes: one for legacy [Camera](http://developer.android.com/reference/android/hardware/Camera.html) API
and another for modern [Camera2](http://developer.android.com/reference/android/hardware/camera2/package-summary.html).
A minimal `Renderer` class can be implemented in Java (OpenGL ES 2.0 [available](http://developer.android.com/reference/android/opengl/GLES20.html) in Java),
but since we are going to modify the preview texture with OpenCL let's move OpenGL stuff to JNI.
Here is a simple Java wrapper for our JNI stuff:
@code{.java}
public class NativeGLRenderer {
static
{
System.loadLibrary("opencv_java3"); // comment this when using OpenCV Manager
System.loadLibrary("JNIrender");
}
public static native int initGL();
public static native void closeGL();
public static native void drawFrame();
public static native void changeSize(int width, int height);
}
@endcode
Since `Camera` and `Camera2` APIs differ significantly in camera setup and control, let's create a base class for the two corresponding renderers:
@code{.java}
public abstract class MyGLRendererBase implements GLSurfaceView.Renderer, SurfaceTexture.OnFrameAvailableListener {
protected final String LOGTAG = "MyGLRendererBase";
protected SurfaceTexture mSTex;
protected MyGLSurfaceView mView;
protected boolean mGLInit = false;
protected boolean mTexUpdate = false;
MyGLRendererBase(MyGLSurfaceView view) {
mView = view;
}
protected abstract void openCamera();
protected abstract void closeCamera();
protected abstract void setCameraPreviewSize(int width, int height);
public void onResume() {
Log.i(LOGTAG, "onResume");
}
public void onPause() {
Log.i(LOGTAG, "onPause");
mGLInit = false;
mTexUpdate = false;
closeCamera();
if(mSTex != null) {
mSTex.release();
mSTex = null;
NativeGLRenderer.closeGL();
}
}
@Override
public synchronized void onFrameAvailable(SurfaceTexture surfaceTexture) {
//Log.i(LOGTAG, "onFrameAvailable");
mTexUpdate = true;
mView.requestRender();
}
@Override
public void onDrawFrame(GL10 gl) {
//Log.i(LOGTAG, "onDrawFrame");
if (!mGLInit)
return;
synchronized (this) {
if (mTexUpdate) {
mSTex.updateTexImage();
mTexUpdate = false;
}
}
NativeGLRenderer.drawFrame();
}
@Override
public void onSurfaceChanged(GL10 gl, int surfaceWidth, int surfaceHeight) {
But instead of writing OpenCL code by yourselves you may want to use __OpenCV T-API__ that calls OpenCL implicitly.
All that you need is to pass the created OpenCL context to OpenCV (via `cv::ocl::attachContext()`) and somehow wrap OpenGL texture with `cv::UMat`.
Unfortunately `UMat` keeps OpenCL _buffer_ internally, that can't be wrapped over either OpenGL _texture_ or OpenCL _image_ - so we have to copy image data here:
- @note We have to make one more image data copy when placing back the modified image to the original OpenGL texture via OpenCL image wrapper.
- @note By default the OpenCL support (T-API) is disabled in OpenCV builds for Android OS (so it's absent in official packages as of version 3.0),
but it's possible to rebuild locally OpenCV for Android with OpenCL/T-API enabled: use `-DWITH_OPENCL=YES` option for CMake.
@code{.cmd}
cd opencv-build-android
path/to/cmake.exe -GNinja -DCMAKE_MAKE_PROGRAM="path/to/ninja.exe" -DCMAKE_TOOLCHAIN_FILE=path/to/opencv/platforms/android/android.toolchain.cmake -DANDROID_ABI="armeabi-v7a with NEON" -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON path/to/opencv
path/to/ninja.exe install/strip
@endcode
To use your own modified `libopencv_java3.so` you have to keep inside your APK, not to use OpenCV Manager and load it manually via `System.loadLibrary("opencv_java3")`.
Performance notes
-----------------
To compare the performance we measured FPS of the same preview frames modification (_Laplacian_) done by C/C++ code (call to `cv::Laplacian` with `cv::Mat`),
by direct OpenCL calls (using OpenCL _images_ for input and output), and by OpenCV _T-API_ (call to `cv::Laplacian` with `cv::UMat`) on _Sony Xperia Z3_ with 720p camera resolution:
* __C/C++ version__ shows __3-4 fps__
* __direct OpenCL calls__ shows __25-27 fps__
* __OpenCV T-API__ shows __11-13 fps__ (due to extra copying from `cl_image` to `cl_buffer` and back)
