Update documentation section on graph visualization, fix ngraph bridge link, etc. (#2818)

* Update for v 0.19

* Add rnotes from github releases page

* Add section on graph inspection tools, add link to HE transformer, and fix link to ngraph bridge for Tensorflow

* Fix broken links

* Remove MANYLINUX from list of default flags

* Revise some for new script loc

* Add stub for ngc_util python script

* Might as well link to open source file

* original file from Vasanth Tovinkere

* updated to handle more attributes.

* Update new script loc and details

* Revise intro

* Clarify large graphs generated from script best read on gephi or cytoscape only

* Edit oddly-worded sentence for clarity

* Delete ngraph_json_to_graphml.py

* Delete ngc_util.py

* Add nGraph logo, update color scheme, disallow preprocessing of frontends' markdown UI

* Finalize doc changes for release v0.19

* PR feedback on nGraph-friendly tools

doc/sphinx/ngraph.doxyfile

@@ -50,7 +50,7 @@ DOXYFILE_ENCODING      = UTF-8
 # pixels and the maximum width should not exceed 200 pixels. Doxygen will copy
 # the logo to the output directory.
 
-PROJECT_LOGO           =
+PROJECT_LOGO           = ../sphinx/ngraph_theme/static/logo.png
 
 # The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute) path
 # into which the generated documentation will be written. If a relative path is
@@ -240,7 +240,7 @@ EXTENSION_MAPPING      =
 # case of backward compatibilities issues.
 # The default value is: YES.
 
-MARKDOWN_SUPPORT       = NO
+MARKDOWN_SUPPORT       = YES
 
 # When the TOC_INCLUDE_HEADINGS tag is set to a non-zero value, all headings up
 # to that level are automatically included in the table of contents, even if
@@ -770,7 +770,7 @@ RECURSIVE              = YES
 # Note that relative paths are relative to the directory from which doxygen is
 # run.
 
-EXCLUDE                =
+EXCLUDE                =  
 
 # The EXCLUDE_SYMLINKS tag can be used to select whether or not files or
 # directories that are symbolic links (a Unix file system feature) are excluded
@@ -786,7 +786,9 @@ EXCLUDE_SYMLINKS       = NO
 # Note that the wildcards are matched against the file with absolute path, so to
 # exclude all test directories for example use the pattern */test/*
 
-EXCLUDE_PATTERNS       = */resource/*  
+# EXCLUDE_PATTERNS       = */resource/*  
+EXCLUDE_PATTERNS       = */frontend/*.md 
+
 
 # The EXCLUDE_SYMBOLS tag can be used to specify one or more symbol names
 # (namespaces, classes, functions, etc.) that should be excluded from the
@@ -1083,7 +1085,7 @@ HTML_EXTRA_FILES       =
 # Minimum value: 0, maximum value: 359, default value: 220.
 # This tag requires that the tag GENERATE_HTML is set to YES.
 
-HTML_COLORSTYLE_HUE    = 193
+HTML_COLORSTYLE_HUE    = 222
 
 # The HTML_COLORSTYLE_SAT tag controls the purity (or saturation) of the colors
 # in the HTML output. For a value of 0 the output will use grayscales only. A
@@ -1091,7 +1093,7 @@ HTML_COLORSTYLE_HUE    = 193
 # Minimum value: 0, maximum value: 255, default value: 100.
 # This tag requires that the tag GENERATE_HTML is set to YES.
 
-HTML_COLORSTYLE_SAT    = 215
+HTML_COLORSTYLE_SAT    = 26
 
 # The HTML_COLORSTYLE_GAMMA tag controls the gamma correction applied to the
 # luminance component of the colors in the HTML output. Values below 100
@@ -1102,7 +1104,7 @@ HTML_COLORSTYLE_SAT    = 215
 # Minimum value: 40, maximum value: 240, default value: 80.
 # This tag requires that the tag GENERATE_HTML is set to YES.
 
-HTML_COLORSTYLE_GAMMA  = 80
+HTML_COLORSTYLE_GAMMA  = 63
 
 # If the HTML_TIMESTAMP tag is set to YES then the footer of each generated HTML
 # page will contain the date and time when the page was generated. Setting this
@@ -1154,7 +1156,7 @@ GENERATE_DOCSET        = YES
 # The default value is: Doxygen generated docs.
 # This tag requires that the tag GENERATE_DOCSET is set to YES.
 
-DOCSET_FEEDNAME        = "Doxygen generated docs"
+DOCSET_FEEDNAME        = "nGraph Doxygen generated docs"
 
 # This tag specifies a string that should uniquely identify the documentation
 # set bundle. This should be a reverse domain-name style string, e.g.
@@ -1162,7 +1164,7 @@ DOCSET_FEEDNAME        = "Doxygen generated docs"
 # The default value is: org.doxygen.Project.
 # This tag requires that the tag GENERATE_DOCSET is set to YES.
 
-DOCSET_BUNDLE_ID       = org.doxygen.Project
+DOCSET_BUNDLE_ID       = ai.ngraph.Docset
 
 # The DOCSET_PUBLISHER_ID tag specifies a string that should uniquely identify
 # the documentation publisher. This should be a reverse domain-name style
@@ -1170,7 +1172,7 @@ DOCSET_BUNDLE_ID       = org.doxygen.Project
 # The default value is: org.doxygen.Publisher.
 # This tag requires that the tag GENERATE_DOCSET is set to YES.
 
-DOCSET_PUBLISHER_ID    = org.doxygen.Publisher
+DOCSET_PUBLISHER_ID    = ai.ngraph.TechPubs
 
 # The DOCSET_PUBLISHER_NAME tag identifies the documentation publisher.
 # The default value is: Publisher.
@@ -1358,14 +1360,14 @@ GENERATE_TREEVIEW      = YES
 # Minimum value: 0, maximum value: 20, default value: 4.
 # This tag requires that the tag GENERATE_HTML is set to YES.
 
-ENUM_VALUES_PER_LINE   = 8
+ENUM_VALUES_PER_LINE   = 20
 
 # If the treeview is enabled (see GENERATE_TREEVIEW) then this tag can be used
 # to set the initial width (in pixels) of the frame in which the tree is shown.
 # Minimum value: 0, maximum value: 1500, default value: 250.
 # This tag requires that the tag GENERATE_HTML is set to YES.
 
-TREEVIEW_WIDTH         = 248
+TREEVIEW_WIDTH         = 383
 
 # If the EXT_LINKS_IN_WINDOW option is set to YES, doxygen will open links to
 # external symbols imported via tag files in a separate window.
@@ -1921,7 +1923,7 @@ DOT_FONTSIZE           = 10
 # the path where dot can find it using this tag.
 # This tag requires that the tag HAVE_DOT is set to YES.
 
-DOT_FONTPATH           = ngraph_theme/static/fonts/
+DOT_FONTPATH           = ../sphinx/ngraph_theme/static/fonts/
 
 # If the CLASS_GRAPH tag is set to YES then doxygen will generate a graph for
 # each documented class showing the direct and indirect inheritance relations.

doc/sphinx/source/buildlb.rst

@@ -44,8 +44,8 @@ compatible with a gcc 4.8-based build.)
 The ``default`` build
 ---------------------
 
-Running ``cmake`` with no build flags defaults to the following settings; adjust 
-as needed: 
+Running ``cmake`` with no build flags defaults to the following settings; see
+the ``CMakeLists.txt`` file for other experimental options' details: 
 
 .. code-block:: console 
 
@@ -68,6 +68,7 @@ as needed:
    -- NGRAPH_PLAIDML_ENABLE:           OFF
    -- NGRAPH_DISTRIBUTED_ENABLE:       OFF
 
+
 .. important:: The default :program:`cmake` procedure (no build flags) will  
    install ``ngraph_dist`` to an OS-level location like ``/usr/bin/ngraph_dist``
    or ``/usr/lib/ngraph_dist``. Here we specify how to build locally to the
@@ -80,30 +81,17 @@ paths on that system. See the :file:`ngraph/CMakeLists.txt` file to change or
 customize the default CMake procedure.
 
 
-Install steps
--------------
+Build steps
+-----------
 
 .. _ubuntu:
 
-Ubuntu 16.04
-~~~~~~~~~~~~
+Ubuntu LTS
+~~~~~~~~~~
 
 The process documented here will work on Ubuntu\* 16.04 (LTS) or on Ubuntu 
 18.04 (LTS).
 
-#. (Optional) Create something like ``/opt/libraries`` and (with sudo), 
-   give ownership of that directory to your user. Creating such a placeholder 
-   can be useful if you'd like to have a local reference for APIs and 
-   documentation, or if you are a developer who wants to experiment with 
-   how to :doc:`core/constructing-graphs/execute` using resources available through the 
-   code base.
-
-   .. code-block:: console
-
-      $ sudo mkdir -p /opt/libraries
-      $ sudo chown -R username:username /opt/libraries
-      $ cd /opt/libraries
-
 #. Clone the `NervanaSystems` ``ngraph`` repo:
 
    .. code-block:: console
@@ -161,18 +149,6 @@ CentOS 7.4
 
 The process documented here will work on CentOS 7.4.
 
-#. (Optional) Create something like ``/opt/libraries`` and (with sudo), 
-   give ownership of that directory to your user. Creating such a placeholder 
-   can be useful if you'd like to have a local reference for APIs and 
-   documentation, or if you are a developer who wants to experiment with 
-   how to :doc:`core/constructing-graphs/execute` using resources available through the 
-   code base.
-
-   .. code-block:: console
-
-      $ sudo mkdir -p /opt/libraries
-      $ sudo chown -R username:username /opt/libraries
-
 #. Update the system with :command:`yum` and issue the following commands: 
    
    .. code-block:: console
@@ -244,36 +220,11 @@ To perform unit tests on the install:
       $ make check
 
 
-Adding framework support
-========================
-
-After building and installing nGraph on your system, there are two likely 
-paths for what you'll want to do next: either compile a framework to run a DL 
-training model, or load an import of an "already-trained" model for inference 
-on an Intel nGraph-enabled backend.
-
-For the former case, this early |version|, :doc:`frameworks/index`, 
-can help you get started with a training a model with a supported framework or 
-companion tool. 
-
-* :doc:`MXNet<frameworks/tensorflow_integ>` framework,  
-* :doc:`TensorFlow<frameworks/mxnet_integ>` framework,
-* :doc:`PaddlePaddle<frameworks/paddle_integ>` framework, or
-* :doc:`ONNX<frameworks/onnx_integ>` and the ONNXIFI tool. 
-
-For the latter case, if you've followed a tutorial from `ONNX`_, and you have an 
-exported, serialized model, you can skip the section on frameworks and go 
-directly to our :doc:`core/constructing-graphs/import` documentation. 
-
-Please keep in mind that both of these are under continuous development, and will 
-be updated frequently in the coming months. Stay tuned!  
-
-
 .. _doxygen: https://www.stack.nl/~dimitri/doxygen/
 .. _Sphinx:  http://www.sphinx-doc.org/en/stable/
 .. _breathe: https://breathe.readthedocs.io/en/latest/
 .. _llvm.org: https://www.llvm.org 
 .. _NervanaSystems: https://github.com/NervanaSystems/ngraph/blob/master/README.md
 .. _ONNX: http://onnx.ai
-.. _website docs: http://ngraph.nervanasys.com/docs/latest/
+.. _website docs: https://ngraph.nervanasys.com/docs/latest/
 .. _googletest framework: https://github.com/google/googletest.git

doc/sphinx/source/conf.py

@@ -73,11 +73,11 @@ author = 'Intel Corporation'
 # built documents.
 #
 # The short X.Y version.
-version = '0.18'
+version = '0.19'
 
 # The Documentation full version, including alpha/beta/rc tags. Some features
 # available in the latest code will not necessarily be documented first
-release = '0.18.0'
+release = '0.19.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

doc/sphinx/source/diagnostics/nbench.rst

-.. nbench:
-
-###################
-Working with nBench
-###################
-
-
-
-
-
-
-
-
-
-
-
-
-
-

doc/sphinx/source/diagnostics/visualize.rst

-.. visualize: 
-
-Visualization
-#############
-

doc/sphinx/source/frameworks/index.rst

@@ -6,7 +6,7 @@ Current framework integrations
 .. toctree::
    :maxdepth: 1
    
-   tensorflow_integ.rst
+   tensorflow_connect.rst
    mxnet_integ.rst
    onnx_integ.rst
    paddle_integ.rst

doc/sphinx/source/frameworks/onnx_integ.rst

-.. onnx_integ.rst:
-
-.. ---------------------------------------------------------------------------
-.. Copyright 2017 Intel Corporation
-.. Licensed under the Apache License, Version 2.0 (the "License");
-.. you may not use this file except in compliance with the License.
-.. You may obtain a copy of the License at
-..
-..      http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing, software
-.. distributed under the License is distributed on an "AS IS" BASIS,
-.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-.. See the License for the specific language governing permissions and
-.. limitations under the License.
-.. ---------------------------------------------------------------------------
+.. frameworks/onnx_integ.rst:
 
 
 ONNX Support

doc/sphinx/source/frameworks/tensorflow_connect.rst

+.. frameworks/tensorflow_connect.rst:
+
+Connect TensorFlow\*
+====================
+
+
+See the `README`_ on the `ngraph_bridge repo`_ for the many ways to connect 
+Tensorflow to nGraph, enabling a `DSO`_ backend that can speed up your TensorFlow 
+training and inference workloads.
+
+
+.. _README: https://github.com/tensorflow/ngraph-bridge/blob/master/README.md
+.. _ngraph_bridge repo: https://github.com/tensorflow/ngraph-bridge  
+.. _DSO: http://csweb.cs.wfu.edu/%7Etorgerse/Kokua/More_SGI/007-2360-010/sgi_html/ch03.html

doc/sphinx/source/frameworks/tensorflow_integ.rst

-.. tensorflow_integ.rst:
-
-TensorFlow\* bridge
-===================
-
-See the `ngraph tensorflow bridge README`_ for how to install the `DSO`_ for the 
-nGraph-TensorFlow bridge.
-
-
-.. _DSO: http://csweb.cs.wfu.edu/%7Etorgerse/Kokua/More_SGI/007-2360-010/sgi_html/ch03.html
-.. _ngraph tensorflow bridge README: https://github.com/NervanaSystems/ngraph-tf/blob/master/README.md

doc/sphinx/source/index.rst

@@ -81,18 +81,18 @@ boost compared to native implementations. For a high-level overview, see the
 
 .. toctree::
    :maxdepth: 1
-   :caption: Diagnostics and Visualization
+   :caption: Inspecting Graphs
+
+   inspection/index.rst
+   inspection/performance-profile.rst
+   inspection/debug.rst 
 
-   diagnostics/nbench.rst
-   diagnostics/performance-profile.rst
-   diagnostics/visualize.rst
-   diagnostics/debug.rst 
 
 .. toctree::
    :maxdepth: 1
    :caption: Tutorials
 
-   nGraph.ai Tutorials <https://www.ngraph.ai/tutorials>   
+   tutorials/index.rst
 
 
 .. toctree::
@@ -104,7 +104,8 @@ boost compared to native implementations. For a high-level overview, see the
    project/contribution-guide.rst
    project/governance.rst
    project/doc-contributor-README.rst
-   project/index.rst 
+   project/index.rst
+   project/extras.rst 
    glossary.rst
 
 Indices and tables

doc/sphinx/source/inspection/index.rst

+.. inspection/index: 
+
+Visualization Tools
+###################
+
+nGraph provides serialization and deserialization facilities along with the 
+ability to create image formats. When visualization is enabled, a ``dot`` file 
+is generated, along with a ``png``. The default can be adjusted by setting the 
+``NGRAPH_VISUALIZE_TREE_OUTPUT_FORMAT`` flag to another format, like PDF. 
+
+Note: Large graphs are usually not legible with formats like PDF.  
+
+Large graphs may require additional work to get into a human-readable format. We 
+have provided a script to convert the `most common default output`_, nGraph JSON,
+to an output that is better able to handle detailed graphs; however, we do not 
+offer user support for this script. After running the script, you should have a 
+``.graphml`` that can be imported and inspected with third-party tools like: 
+
+#. `Gephi`_
+
+#. `Cytoscape`_
+
+.. #. `Netron`_ support tentatively planned to come soon
+
+
+.. _CMakeLists.txt: https://github.com/NervanaSystems/ngraph/blob/master/CMakeLists.txt
+.. _most common default output: https://github.com/NervanaSystems/ngraph/contrib/tools/graphml/ngraph_json_to_graphml.py
+.. _Netron: https://github.com/lutzroeder/netron/blob/master/README.md
+.. _Gephi: https://gephi.org
+.. _Cytoscape: https://cytoscape.org
+

doc/sphinx/source/project/extras.rst

+.. project/extras.rst
+
+#######
+Extras
+#######
+
+
+* `Deep Learning on Encrypted Data with HE Transformer for nGraph`_
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+.. _Deep Learning on Encrypted Data with HE Transformer for nGraph: https://www.intel.ai/he-transformer-for-ngraph-enabling-deep-learning-on-encrypted-data/
\ No newline at end of file

doc/sphinx/source/project/release-notes.rst

@@ -14,6 +14,24 @@ See also: https://github.com/NervanaSystems/ngraph/releases for previous version
 CHANGELOG |release|
 ===================
 
++ Add graph visualization tools to doc
++ Update doxygen to be friendlier to frontends
+
+
+Changelog 
+=========
+
+nGraph v0.18.1
+--------------
+
++ Python formatting issue
++ mkl-dnn work-around
++ Event tracing improvements
++ Gaussian error function
++ Begin tracking framework node names
++ ONNX quantization
++ More fusions
+
 
 nGraph v0.17.0-rc.0
 -------------------
@@ -29,9 +47,6 @@ Recent API Changes
 + Pad op takes CoordinateDiff instead of Shape pad values to allow for negative padding.
 
 
-Changelog 
-=========
-
 nGraph v0.16.0-rc.3
 -------------------
 
@@ -44,5 +59,5 @@ nGraph v0.16.0-rc.3
 + Doc updates
 
 
-.. _Format .zip: https://github.com/NervanaSystems/ngraph/archive/v0.17.0-rc.0.zip
-.. _Format tar.gz: https://github.com/NervanaSystems/ngraph/archive/v0.17.0-rc.0.tar.gz
\ No newline at end of file
+.. _Format .zip: https://github.com/NervanaSystems/ngraph/archive/v0.19.0-rc.0.zip
+.. _Format tar.gz: https://github.com/NervanaSystems/ngraph/archive/v0.19.0-rc.0.tar.gz
\ No newline at end of file

doc/sphinx/source/tutorials/index.rst

-.. tutorials/index:
-
+.. guides/index:
 
 .. This will hold the organization of the tutorials we put on ngraph.ai  
 
    .. it will need to be organized in a way that is navigable for the many kinds of frameworks and backends we support in the "Compiler stack".  It will need to be workable with a sitemap structure. The initial example is for the latest nGraph-TensorFlow bridge.  
 
-
-:orphan:
-
-##########
-Tutorials
-##########
+#######
+Guides
+#######
 
 Coming soon 
 
 .. toctree::
    :maxdepth: 1
-
-   
-