Commit dcd71a59 authored by L.S. Cook's avatar L.S. Cook Committed by Scott Cyphers

Re-align docs with code example for mnist (#1364)

* Re-align docs with code example for mnist

* Also fix dist_mnist and add context highlight
parent 9a4125ef
......@@ -4,13 +4,9 @@
Derive a trainable model
#########################
Documentation in this section describes one of the ways to derive and run a
trainable model from an inference model.
We can derive a trainable model from any graph that has been constructed with
weight-based updates. For this example named ``mnist_mlp.cpp``, we start with
a hand-designed inference model and convert it to a model that can be trained
with nGraph.
Documentation in this section describes one of the possible ways to turn a
:abbr:`DL (Deep Learning)` model for inference into one that can be used
for training.
Additionally, and to provide a more complete walk-through that *also* trains the
model, our example includes the use of a simple data loader for uncompressed
......@@ -25,19 +21,26 @@ MNIST data.
- :ref:`update`
.. _understanding_ml_ecosystem:
.. _automating_graph_construction:
Automating graph construction
==============================
Understanding the ML ecosystem
===============================
In a :abbr:`Machine Learning (ML)` ecosystem, it makes sense to use automation
and abstraction whereever possible. nGraph was designed to automatically use
the "ops" of tensors provided by a framework when constructing graphs. However,
nGraph's graph-construction API operates at a fundamentally lower level than a
typical framework's API, and writing a model directly in nGraph would be somewhat
akin to programming in assembly language: not impossible, but not the easiest
thing for humans to do.
In a :abbr:`Machine Learning (ML)` ecosystem, it makes sense to take advantage
of automation and abstraction as much as possible. As such, nGraph was designed
to integrate wtih graph construction endpoints (AKA *ops*) handed down to it
from a framework. Our graph-construction API, therefore, needs to operate at a
fundamentally lower level than a typical framework's API. For this reason,
writing a model directly in nGraph would be somewhat akin to programming in
assembly language: not impossible, but not exactly the easiest thing for humans
to do.
To make the task easier for developers who need to customize the "automatic",
construction of graphs, we've provided some demonstration code for how this
could be done. We know, for example, that a trainable model can be derived from
any graph that has been constructed with weight-based updates.
The following example named ``mnist_mlp.cpp`` represents a hand-designed
inference model being converted to a model that can be trained with nGraph.
.. _model_overview:
......@@ -74,20 +77,25 @@ Inference
---------
We begin by building the graph, starting with the input parameter
``X``. We define a fully-connected layer, including a parameter for
weights and bias.
``X``. We also define a fully-connected layer, including parameters for
weights and bias:
.. literalinclude:: ../../../examples/mnist_mlp/mnist_mlp.cpp
:language: cpp
:lines: 124-136
:lines: 127-139
Repeat the process for the next layer,
.. literalinclude:: ../../../examples/mnist_mlp/mnist_mlp.cpp
:language: cpp
:lines: 141-149
We repeat the process for the next layer, which we
normalize with a ``softmax``.
and normalize everything with a ``softmax``.
.. literalinclude:: ../../../examples/mnist_mlp/mnist_mlp.cpp
:language: cpp
:lines: 137-149
:lines: 151-153
.. _loss:
......@@ -95,13 +103,13 @@ normalize with a ``softmax``.
Loss
----
We use cross-entropy to compute the loss. nGraph does not currenty
have a cross-entropy op, so we implement it directly, adding clipping
to prevent underflow.
We use cross-entropy to compute the loss. nGraph does not currenty have a core
op for cross-entropy, so we implement it directly, adding clipping to prevent
underflow.
.. literalinclude:: ../../../examples/mnist_mlp/mnist_mlp.cpp
:language: cpp
:lines: 151-169
:lines: 154-169
.. _backprop:
......@@ -109,16 +117,15 @@ to prevent underflow.
Backprop
--------
We want to reduce the loss by adjusting the weights. We compute the
adjustments using the reverse mode autodiff algorithm, commonly
referred to as "backprop" because of the way it is implemented in
interpreted frameworks. In nGraph, we augment the loss computation
with computations for the weight adjustments. This allows the
calculations for the adjustments to be further optimized.
We want to reduce the loss by adjusting the weights. We compute the adjustments
using the reverse-mode autodiff algorithm, commonly referred to as "backprop"
because of the way it is implemented in interpreted frameworks. In nGraph, we
augment the loss computation with computations for the weight adjustments. This
allows the calculations for the adjustments to be further optimized.
.. literalinclude:: ../../../examples/mnist_mlp/mnist_mlp.cpp
:language: cpp
:lines: 172-175
:lines: 171-175
For any node ``N``, if the update for ``loss`` is ``delta``, the
......@@ -129,7 +136,8 @@ update computation for ``N`` will be given by the node
auto update = loss->backprop_node(N, delta);
The different update nodes will share intermediate computations. So to
get the updated values for the weights we just say:
get the updated values for the weights as computed with the specified
:doc:`backend <../programmable/index>`,
.. literalinclude:: ../../../examples/mnist_mlp/mnist_mlp.cpp
:language: cpp
......@@ -153,5 +161,5 @@ compile clones of the nodes.
.. literalinclude:: ../../../examples/mnist_mlp/mnist_mlp.cpp
:language: cpp
:lines: 220-226
:lines: 221-226
......@@ -19,18 +19,12 @@ should be added after the steps needed to complete the
.. literalinclude:: ../../../examples/mnist_mlp/dist_mnist_mlp.cpp
:language: cpp
:lines: 188-191
:lines: 180-196
:emphasize-lines: 9-12
Also since we are using OpenMPI in this example, we need to initialize and
finalize MPI.
.. literalinclude:: ../../../examples/mnist_mlp/dist_mnist_mlp.cpp
:language: cpp
:lines: 112
.. literalinclude:: ../../../examples/mnist_mlp/dist_mnist_mlp.cpp
:language: cpp
:lines: 295
finalize MPI with ``MPI::Init();`` and ``MPI::Finalize();`` at the beginning
and the end of the code used to deploy to devices; see the `full raw code`_.
Finally, to run the training on two nGraph devices, invoke :command:`mpirun`.
This will run on a single machine and launch two processes.
......@@ -41,7 +35,5 @@ This will run on a single machine and launch two processes.
$ mpirun -np 2 dist_mnist_mlp
.. _OpenMPI: https://www.open-mpi.org/software/ompi/v3.1
.. _full raw code: https://github.com/NervanaSystems/ngraph/blob/master/doc/examples/mnist_mlp/dist_mnist_mlp.cpp
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment