Stubs for Provenance documentation (#3781)

* Initial draft start

* Add provenance doc

* Another effort at resolving doc build warning

* Revert file

* Revise provenance intro

* Ensure Clang dependencies update with new minimum version of clang needed

* PR review feedback, cleanup other change added to collab_ngai

doc/sphinx/source/buildlb.rst

@@ -41,8 +41,8 @@ Prerequisites
    :escape: ~
 
    CentOS 7.4 64-bit, GCC 4.8, CMake 3.9.0, supported, ``wget zlib-devel ncurses-libs ncurses-devel patch diffutils gcc-c++ make git perl-Data-Dumper`` 
-   Ubuntu 16.04 or 18.04 (LTS) 64-bit, Clang 6, CMake 3.5.1 + GNU Make, supported, ``build-essential cmake clang-3.9 clang-format-3.9 git curl zlib1g zlib1g-dev libtinfo-dev unzip autoconf automake libtool``
-   Clear Linux\* OS for Intel® Architecture version 28880, Clang 8.0, CMake 3.14.2, experimental, bundles ``machine-learning-basic c-basic python-basic python-basic-dev dev-utils``
+   Ubuntu 16.04 or 18.04 (LTS) 64-bit, Clang 6, CMake 3.5.1 + GNU Make, supported, ``build-essential cmake clang-format-6.0 clang-tidy-6.0 clang-6.0 git curl zlib1g zlib1g-dev libtinfo-dev unzip autoconf automake libtool``
+   Clear Linux\* OS for Intel® Architecture version 28880+, Clang 8.0, CMake 3.14.2, experimental, bundles ``machine-learning-basic c-basic python-basic python-basic-dev dev-utils``
 
 
 .. _default_ngflags:

doc/sphinx/source/conf.py

@@ -73,11 +73,11 @@ author = 'Intel Corporation'
 # built documents.
 #
 # The short X.Y version.
-version = '0.26'
+version = '0.27'
 
 # The Documentation full version, including alpha/beta/rc tags. Some features
 # available in the latest code will not necessarily be documented first
-release = '0.26.0'
+release = '0.27.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

doc/sphinx/source/features.rst

 .. features.rst
 
+:orphan:
+
 .. _features:
 
 Features

doc/sphinx/source/glossary.rst

@@ -17,6 +17,12 @@ Glossary
       A component of nGraph that acts as a backend for a framework,
       allowing the framework to define and execute computations.
 
+   builder
+
+       A builder is a function that creates a sub-graph and returns 
+       a root node to the bridge. Fused ops are preferred over 
+       builders See also: :doc:`provenance/overview`.
+
    data-flow graph
 
       Data-flow graphs are used to implement deep learning models. In  
@@ -67,16 +73,28 @@ Glossary
       additional constant attributes. Every output of an op
       corresponds to a tensor and has an element type and a shape. The
       element types and shapes of the outputs of an op are determined
-      by the inputs and attributes of the op.
+      by the inputs and attributes of the op.   
 
    parameter
 
       In the context of a function graph, a "parameter" refers to what
       "stands in" for an argument in an ``op`` definition.
 
+   provenance
+   
+      The term provenance refers to the matching of device code to framework 
+      sub-graphs; it is analogous to source code locators in conventional 
+      compilers, which associate regions of object code with source files and 
+      line numbers. 
+   
    quantization
 
-      Quantization refers to the conversion of numerical data into a lower-precision representation. Quantization is often used in deep learning to reduce the time and energy needed to perform computations by reducing the size of data transfers and the number of steps needed to perform a computation. This improvement in speed and energy usage comes at a cost in terms of numerical accuracy, but deep learning models are often able to function well in spite of this reduced accuracy. 
+      Quantization refers to the conversion of numerical data into a lower-precision 
+      representation. Quantization is often used in deep learning to reduce the time and 
+      energy needed to perform computations by reducing the size of data transfers and the 
+      number of steps needed to perform a computation. This improvement in speed and energy 
+      usage comes at a cost in terms of numerical accuracy, but deep learning models are 
+      often able to function well in spite of this reduced accuracy. 
 
    result
 

doc/sphinx/source/index.rst

@@ -37,8 +37,6 @@ nGraph Compiler Stack Documentation
    :maxdepth: 1
 
    introduction.rst
-   features.rst
-   project/release-notes.rst
 
 .. toctree::
    :maxdepth: 2
@@ -58,6 +56,7 @@ nGraph Compiler Stack Documentation
    nGraph Core Ops <ops/index.rst>
    core/constructing-graphs/index.rst
    core/passes/passes.rst
+   provenance/index.rst
 
    
 .. toctree::

doc/sphinx/source/introduction.rst

@@ -34,7 +34,7 @@ Motivations
 The current :abbr:`State-of-the-Art (SoTA)` software solution for deep 
 learning computation is to integrate kernel libraries such as Intel® 
 :abbr:`Math Kernel Library for Deep Neural Networks (Intel® MKL DNN)` 
-and Nvidia\*'s CuDNN into deep learning frameworks. These kernel 
+and Nvidia's CuDNN into deep learning frameworks. These kernel 
 libraries offer a performance boost during runtime on specific hardware 
 targets through highly-optimized kernels and other operator-level 
 optimizations.
@@ -166,7 +166,7 @@ computational graphsfrom deep learning frameworks with nGraph IR.
 In conjunction with nGraph's graph-level optimizations, PlaidML automatically
 applies low-level optimizations to improve deep learning performance.
 Additionally, PlaidML offers extensive support for various hardware targets
-due to its abilility to generate code in LLVM, OpenCL, OpenGL, and Metal.
+due to its ability to generate code in LLVM, OpenCL, OpenGL, and Metal.
 
 Given a backend with existing kernel libraries, nGraph can readily support the
 target hardware because the backend only needs to support a few primitive

doc/sphinx/source/provenance/index.rst

+.. provenance/index.rst
+
+Provenance
+##########
+
+.. toctree::
+   :maxdepth: 1
+   
+   overview.rst
+..   other.rst

doc/sphinx/source/provenance/overview.rst

+.. provenance/overview.rst
+
+Basic concepts
+==============
+
+The term :term:`provenance` refers to the matching of device code to 
+framework subgraphs; it is analogous to source code locators in 
+conventional compilers, which associate regions of object code with 
+source files and line numbers. Provenance is *extensible* in that it 
+may also include the chain of passes that lead from the framework graph 
+to the executing code. 
+
+It can associate device code with specific tags added by a framework bridge which 
+correspond to the framework ops that create the nGraph nodes. This works only for 
+those transformations that take place in nGraph: the information stored 
+in the nodes can include additional details about how the device code was 
+chosen. For example, whenever a graph transformation is performed with one 
+of the nGraph core :doc:`Ops <../ops/index>`, a lower level of abstraction 
+can record information about the transformation that may be useful to 
+anyone wondering why a kernel was "chosen"; a complete description of the 
+steps leading to the device kernels being used, as well as all of the 
+framework nodes that led to the kernel, can be obtained. 
+
+
+Existing use cases
+------------------
+
+Currently, every node nGraph touches can optionally have a set of provenance 
+tags, which are strings set by a framework bridge. When a set of nodes is 
+replaced by a new set of nodes, a combination of heuristics and special casing 
+is used to set the tags on the new nodes based on the tags from the old nodes. 
+
+A :term:`builder` is a function that creates a sub-graph and returns a root 
+node to the bridge. The bridge is not necessarily aware of the subgraph, only 
+of the returned node, where it sets tags. The remaining nodes' tags are set 
+by associating a set of nodes, called a *provenance group*, with the node. Any 
+tags added to the node are also added to the nodes in the provenance group.
+
+An updated implementation of the functionality of builders is the *fused op*, 
+a node that can replace itself with a subgraph. When the node is expanded 
+into a subgraph, a vector of values is returned, corresponding to outputs 
+of the original fused op; the tags of the fused op are added to all nodes 
+in the values in reverse dataflow direction, up to (though not including) the 
+input values of the fused op.
+