buildlb.rst 9.34 KB
Newer Older
1
.. buildlb.rst:
2

Leona C's avatar
Leona C committed
3
###############
4
Build and Test
Leona C's avatar
Leona C committed
5
###############
6

7
* :ref:`default_ngflags`
Leona C's avatar
Leona C committed
8
* :ref:`ngraph_plaidml_backend`
9

10 11 12 13 14
There are a few common paths to take when manually building the |project| 
from source code. Today nGraph supports various developers working on all 
parts of the :abbr:`Deep Learning (DL)` stack, and the way you decide to 
build or install components ought to depend on the capabilities of your 
hardware, and how you intend to use it.
L.S. Cook's avatar
L.S. Cook committed
15

16 17 18
A "from scratch" source-code build of the nGraph Library enables the CPU, 
``Interpreter``, and unit tests by default. See :ref:`default_ngflags` 
for more detail.
L.S. Cook's avatar
L.S. Cook committed
19

20 21 22 23 24 25 26 27 28 29 30 31 32 33
A "from scratch" source-code build that defaults to the PlaidML backend 
contains rich algorithm libraries akin to those that were previously available 
only to developers willing to spend extensive time writing, testing, and 
customizing kernels. An ``NGRAPH_PLAIDML`` dist can function like a framework 
that lets developers compose, train, and even deploy :abbr:`DL (Deep Learning)` 
models in their preferred language on neural networks of any size. This is 
a good option if, for example, you are working on a laptop with a high-end 
GPU that you want to use for compute. See :ref:`ngraph_plaidml_backend` 
for instructions on how to build.

In either case, there are some prerequisites that your system will need 
to build from sources.

.. _prerequisites:
34

35 36
Prerequisites
-------------
37 38 39 40 41 42

.. csv-table::
   :header: "Operating System", "Compiler", "Build System", "Status", "Additional Packages"
   :widths: 25, 15, 25, 20, 25
   :escape: ~

43
   CentOS 7.4 64-bit, GCC 4.8, CMake 3.9.0, supported, ``wget zlib-devel ncurses-libs ncurses-devel patch diffutils gcc-c++ make git perl-Data-Dumper`` 
44 45
   Ubuntu 16.04 or 18.04 (LTS) 64-bit, Clang 6, CMake 3.5.1 + GNU Make, supported, ``build-essential cmake clang-format-6.0 clang-tidy-6.0 clang-6.0 git curl zlib1g zlib1g-dev libtinfo-dev unzip autoconf automake libtool``
   Clear Linux\* OS for Intel® Architecture version 28880+, Clang 8.0, CMake 3.14.2, experimental, bundles ``machine-learning-basic c-basic python-basic python-basic-dev dev-utils``
46 47


48 49 50 51
.. _default_ngflags:

Building nGraph from source
===========================
52

53 54 55 56
.. important:: The default :program:`cmake` procedure (no build flags) will  
   install ``ngraph_dist`` to an OS-level location like ``/usr/bin/ngraph_dist``
   or ``/usr/lib/ngraph_dist``. Here we specify how to build locally to the
   location of ``~/ngraph_dist`` with the cmake target ``-DCMAKE_INSTALL_PREFIX=~/ngraph_dist``. 
57 58

All of the nGraph Library documentation presumes that ``ngraph_dist`` gets 
59 60 61
installed locally. The system location can be used just as easily by 
customizing paths on that system. See the :file:`ngraph/CMakeLists.txt` 
file to change or customize the default CMake procedure.
62

63 64
* :ref:`ubuntu`
* :ref:`centos`
65

66

L.S. Cook's avatar
L.S. Cook committed
67
.. _ubuntu:
68

69 70
Ubuntu LTS build steps
----------------------
71

72 73
The process documented here will work on Ubuntu\* 16.04 (LTS) or on Ubuntu 
18.04 (LTS).
74

75 76
#. Ensure you have installed the :ref:`prerequisites` for Ubuntu\*.

77 78 79 80
#. Clone the `NervanaSystems` ``ngraph`` repo:

   .. code-block:: console

81
      $ git clone https://github.com/NervanaSystems/ngraph.git
82 83 84 85 86 87 88 89 90
      $ cd ngraph

#. Create a build directory outside of the ``ngraph/src`` directory 
   tree; somewhere like ``ngraph/build``, for example:

   .. code-block:: console

      $ mkdir build && cd build

L.S. Cook's avatar
L.S. Cook committed
91
#. Generate the GNU Makefiles in the customary manner (from within the 
92 93
   ``build`` directory). This command enables ONNX support in the library  
   and sets the target build location at ``~/ngraph_dist``, where it can be 
94
   found easily.
95 96 97

   .. code-block:: console

98
      $ cmake .. -DNGRAPH_ONNX_IMPORT_ENABLE=ON  -DCMAKE_INSTALL_PREFIX=~/ngraph_dist
99 100 101 102

   **Other optional build flags** -- If running ``gcc-5.4.0`` or ``clang-3.9``, 
   remember that you can also append ``cmake`` with the prebuilt LLVM option 
   to speed-up the build.  Another option if your deployment system has Intel® 
103 104
   Advanced Vector Extensions (Intel® AVX) is to target the accelerations 
   available directly by compiling the build as follows during the cmake 
105 106
   step: ``-DNGRAPH_TARGET_ARCH=skylake-avx512``.  
   
107 108
   .. code-block:: console

109
      $ cmake .. [-DNGRAPH_TARGET_ARCH=skylake-avx512]   
110 111

#. Run ``$ make`` and ``make install`` to install ``libngraph.so`` and the 
112
   header files to ``~/ngraph_dist``:
113 114 115

   .. code-block:: console
      
116
      $ make   # note: make -j <N> may work, but sometimes results in out-of-memory errors if too many compilation processes are used
117
      $ make install
118 119 120 121

#. (Optional, requires `doxygen`_, `Sphinx`_, and `breathe`_). Run ``make html`` 
   inside the ``doc/sphinx`` directory of the cloned source to build a copy of 
   the `website docs`_ locally. The low-level API docs with inheritance and 
L.S. Cook's avatar
L.S. Cook committed
122 123
   collaboration diagrams can be found inside the ``/docs/doxygen/`` directory. 
   See the :doc:`project/doc-contributor-README` for more details about how to 
124
   build documentation for nGraph.
L.S. Cook's avatar
L.S. Cook committed
125

126

L.S. Cook's avatar
L.S. Cook committed
127
.. _centos: 
128

129 130
CentOS 7.4 build steps
----------------------
131 132 133

The process documented here will work on CentOS 7.4.

134 135 136
#. Ensure you have installed the :ref:`prerequisites` for CentOS\*, 
   and update the system with :command:`yum`.

137 138 139 140 141 142 143 144
   .. code-block:: console

      $ sudo yum update

#. Install Cmake 3.4:

   .. code-block:: console
    
145 146 147
      $ wget https://cmake.org/files/v3.4/cmake-3.5.0.tar.gz      
      $ tar -xzvf cmake-3.5.0.tar.gz
      $ cd cmake-3.5.0
148 149
      $ ./bootstrap --system-curl --prefix=~/cmake
      $ make && make install     
150

151
#. Clone the `NervanaSystems` ``ngraph`` repo via HTTPS and use Cmake 3.5.0 to 
152 153
   build nGraph Libraries to ``~/ngraph_dist``. This command enables ONNX 
   support in the library (optional). 
154

155 156 157
   .. code-block:: console

      $ cd /opt/libraries 
158
      $ git clone https://github.com/NervanaSystems/ngraph.git
159
      $ cd ngraph && mkdir build && cd build
160
      $ ~/cmake/bin/cmake .. -DCMAKE_INSTALL_PREFIX=~/ngraph_dist -DNGRAPH_ONNX_IMPORT_ENABLE=ON 
161 162 163
      $ make && sudo make install 


Leona C's avatar
Leona C committed
164
.. _ngraph_plaidml_backend:
165

Leona C's avatar
Leona C committed
166 167
Building nGraph-PlaidML from source
===================================
168

Leona C's avatar
Leona C committed
169 170
The following instructions will create the ``~/ngraph_plaidml_dist`` 
locally:
171

Leona C's avatar
Leona C committed
172
#. Ensure you have installed the :ref:`prerequisites` for your OS.
173

Leona C's avatar
Leona C committed
174
#. Install the prerequisites for the backend. Our hybrid ``NGRAPH_PLAIDML``
175 176 177 178 179 180 181 182 183 184
   backend works best with Python3 versions. We recommend that you use a 
   virtual environment, due to some of the difficulties that users have 
   seen when trying to install outside of a venv.

   .. code-block:: console

      $ sudo apt install python3-pip
      $ pip install plaidml 
      $ plaidml-setup

Leona C's avatar
Leona C committed
185
#. Clone the source code, create and enter your build directory:
186 187 188 189 190 191

   .. code-block:: console

      $ git clone https://github.com/NervanaSystems/ngraph.git
      $ cd ngraph && mkdir build && cd build

Leona C's avatar
Leona C committed
192
#. Prepare the CMake files as follows: 
193 194 195 196 197

   .. code-block:: console

      $ cmake .. -DCMAKE_INSTALL_PREFIX=~/ngraph_plaidml_dist -DNGRAPH_CPU_ENABLE=OFF -DNGRAPH_PLAIDML_ENABLE=ON 

Leona C's avatar
Leona C committed
198
#. Run :command:`make` and ``make install``. Note that if you are building 
199 200 201 202 203 204 205 206 207 208 209 210 211 212
   outside a local or user path, you may need to run ``make install`` as the 
   root user.

   .. code-block:: console

      $ make
      $ make install

   This should create the shared library ``libplaidml_backend.so`` and 
   nbench. Note that if you built in a virtual environment and run 
   ``make check`` from it, the Google Test may report failures. Full 
   tests can be run when PlaidML devices are available at the machine 
   level.

Leona C's avatar
Leona C committed
213 214
For more about working with the PlaidML backend from nGraph, see our 
API documentation :doc:`backends/plaidml-ng-api/index`. 
215 216


217 218
macOS\* development
--------------------
219

L.S. Cook's avatar
L.S. Cook committed
220 221
.. note:: Although we do not currently offer full support for the macOS platform, 
   some configurations and features may work.
222 223 224 225 226 227 228 229 230 231

The repository includes two scripts (``maint/check-code-format.sh`` and 
``maint/apply-code-format.sh``) that are used respectively to check adherence 
to ``libngraph`` code formatting conventions, and to automatically reformat code 
according to those conventions. These scripts require the command 
``clang-format-3.9`` to be in your ``PATH``. Run the following commands 
(you will need to adjust them if you are not using bash):

.. code-block:: bash

232
   $ brew install llvm@3.9 automake
233 234 235 236
   $ mkdir -p $HOME/bin
   $ ln -s /usr/local/opt/llvm@3.9/bin/clang-format $HOME/bin/clang-format-3.9
   $ echo 'export PATH=$HOME/bin:$PATH' >> $HOME/.bash_profile

237 238
Testing the build 
=================
239

240 241 242 243 244 245 246 247 248 249
We use the `googletest framework`_ from Google for unit tests. The ``cmake`` 
command automatically downloaded a copy of the needed ``gtest`` files when 
it configured the build directory.

To perform unit tests on the install:

#. Create and configure the build directory as described in our 
   :doc:`buildlb` guide.

#. Enter the build directory and run ``make check``:
250 251 252 253 254 255 256
   
   .. code-block:: console

      $ cd build/
      $ make check


257
.. _doxygen: http://www.doxygen.nl/index.html
258 259 260 261 262
.. _Sphinx:  http://www.sphinx-doc.org/en/stable/
.. _breathe: https://breathe.readthedocs.io/en/latest/
.. _llvm.org: https://www.llvm.org 
.. _NervanaSystems: https://github.com/NervanaSystems/ngraph/blob/master/README.md
.. _ONNX: http://onnx.ai
263
.. _website docs: https://ngraph.nervanasys.com/docs/latest/
264
.. _googletest framework: https://github.com/google/googletest.git
265
.. _PlaidML: https://github.com/plaidml/plaidml