* [ONNX] Add support for ONNX sinh op

* [ONNX] Test the sinh op and improve an error msg about not supported op version

* [ONNX] Fix the failing tests by moving sinh to opset 1

* [ONNX] Apply clang-format to fix failing CI builds

* [ONNX] Add and test ONNX cosh op

* [ONNX] Add and test ONNX cosh op

* [ONNX] Add and test ONNX sign op

* [ONNX] Test the sign operator with more challenging values

* [ONNX] Update of supported ops documentation table

* [ONNX] Add null node.

Optional inputs in ONNX standard are represented by empty string.
We need a placeholder to keep information which inputs are not provided.

* Rename class null_node -> NullNode

* Remove unnecesary validate_and_infer_types method

* Add <memory> header

* Change name != "" -> !name.empty()

* Change constructor

* Little description

* Change node type in NullNode

* Add is_null() method

* Docstring

* Add UT

* Use override

* Style check

* Update null_node.cpp

* [ONNX] Move make_ng_constant to Tensor class

* [ONNX] Add workaround for initializers without corresponding inputs

* Fix malformed test models

* clang-format

* Avoid creating multiple constants

* Code review comments

* [ONNX] Add support for ONNX sinh op

* [ONNX] Test the sinh op and improve an error msg about not supported op version

* [ONNX] Fix the failing tests by moving sinh to opset 1

* [ONNX] Apply clang-format to fix failing CI builds

* [ONNX] Add and test ONNX cosh op

* [ONNX] Add and test ONNX cosh op

* -   check to verify if the data_slices shares the same weights

* add the serialized graph

* - explicitly fuse the data slices, so all the parameter partitioned by slices are in contigous memory location
- fixes all the failing test cases

* Enable support for group attribute.

* UT for ConvTranspose with groups.

* Validate group attribute value.

* Move helper function to unnamed namespace.

* Access values with bounds checking.

* Fix spelling.

* UT for Softplus ONNX operator testing edge cases.

* Rename UT model name.

* Handle overflows.

* Add UT for ininite values and check them correctly.

* Update values in comment

* Add broadcasting to variadic OP in opset 8.

* Apply style format.

* Update onnx_import.cpp

* - Added reorder support for rnn weights_layer/iter

* i) fixed compilation issues ii) working but still observing precision error

* i) fixed failing rnn unit test for DEX ii) refactored workspace in RNN mkldnn emitter

* i) added support for src reorder to TNC from NTC

* reorder support for rnn output fron NTC to TNC

* - added support for rnn weight reorder ldgoi -> ldigo
- code refactor for lstm/rnn kernel in mkldnn emitter

* - refactor rnn mkldnnn kernel, change variable names

* fix RNN codegen kernel

* disbale layer rnn fusion pass, to test CI

* method to validate recurrent rnn inputs

* add correlated macthes for Recurrent RNN PM

* - simplify reorder logic for rnn_weights
- fix graph pattern for fusing rnn cell across time steps

* do weights reorders in rnn timesteps fusion

* refactored LSTM graph pass

* - Bug fix for finding the lstm inputs determenstically
- Refactored LSTM graph pass to single pass
- made changes to LSTM RNN time step fusion graph pass

* - use replace_node instead of replace_output in Lstm_step_wise fusion graph pass

* fix compilation error

* Fix GNMT rnn fusion

* check if the node is in use before replacing in RNN graph passes

*  i) fix style ii) fix topo sort issue in RNN graph pass

* style fix

* fix bug in simplify_concat pass

* replaces Lstm1 -> {GOE1, GOE2} -> {Slice1, Slice2} -> Concat -> Lstm2 with Lstm1 -> Lstm2

* cse for convert layout

* addressed PR comments

* - optimization pass to remove  Lstm1 -> {GOE1, GOE2} -> {Slice1, Slice2} -> Lstm2
- conditional fusing of LSTM cells only for the decoder

* made changes to multi layer RNN fusion callback

* fix asserts in RNN op

* - added support to fuse layers when slc=dlc for RNN cells
- bug fix on the sanity checks for RNN Op

* - support RNN layer fusion till slc = dlc
- bug fixes in multi layer rnn fusion call back

* capture reshape in the RNN weights

* Addressed PR comments

* - added comments in multi layer PM call back
- fuse only if slc == DLC across layers

* restore deleted 3_lstm_cell_forward.json file

* fix typo

* fix failing unit tets

* When processing in place slice, do not change the offset of the slice node if the argument pointer comes from function input.

* Address PR feedback: process in place slice after propagating in place input.

* Set INTERMEDIATE role before propagating in place input.

* Do not add temporaries to the variable name map before propagating in place input in codegen.

* Fix a bug in codegen.

* Fix a bug in codegen slice.

* reenable disabled rnn unit test

* fix compiler error

* - bug fix in the slicing logic for the layer fused rnn cell
- fix failing rnn unit test

* - Addressed PR comments
- removed redundant checks from the rnn graph pass
- simplified rnn call back replace node logic

* - added new multilayer rnn *.json file
- fix test case

* [PRIVATE BRANCH] Style fixes (#2080)

* Style fixes

* change order of lstm gates

* [PRIVATE BRANCH] Jbobba/rnn fusion review (#2113)

* Style fixes for single-layer RNN fusion

* Style fixes to multi-layer RNN

* style fix

* disable GPU test

* Unit tests for conv2d causing errors.

* UT for conv3D_bias

* Fix padding order.

`padding below` in nGraph terminology means padding added at the beginning
of the axis. Whereas `padding above` means padding added at the end of
the axis.

* Rename test to sth more descriptive.

* Apply clang-format.

* Fix handling of `SAME_UPPER/LOWER` auto_pads mode for convolution/pooling ops.

* Fix order of padding_below/above.
Signed-off-by: Adam Rogowiec <adam.rogowiec@intel.com>

* Fix error in calculating output data shape.

* Add static keyword for helper function.

* Fix MatMul for cases where left hand side is 1D vector.

- Add unit-test for this case.

* Add new line at the end of file.

* Log warning when dealing with scalars

* Apply clang-format

* Review: fix spelling, rename test model.

* Add ArgMin operator

* Add ArgMax and a basic test case

* Rename variables

* Apply workaround for problems with Reshape on i64

* Review comments

* Review comments

* onnx: add information about a domain to operators set
Signed-off-by: Artur Wojcik <artur.wojcik@intel.com>

* onnx: updates after review
Signed-off-by: Artur Wojcik <artur.wojcik@intel.com>

* onnx: update comments in the code
Signed-off-by: Artur Wojcik <artur.wojcik@intel.com>

* onnx: fix bug in node's description method
Signed-off-by: Artur Wojcik <artur.wojcik@intel.com>

* onnx: fix CentOS compilation
Signed-off-by: Artur Wojcik <artur.wojcik@intel.com>

* onnx: more after review changes
Signed-off-by: Artur Wojcik <artur.wojcik@intel.com>

* split bn into bn_inference bn_training

* fix warnings

* Add GPU support for the new BN ops (#1569)

* Add GPU support and change batchnorm_globalstats test to use BNInference.

* Changed test back to using BNTraining for global stats and updated cudnn backend to account for it.

* Fix issues in merge with master.

* Formatting.

* CPU fixes

* remove 5-arg training BN for now

* more fixes

* python batchnorm changes

* fix onnx_import

* fix a call BatchNormInference c-tor

* yet another fix to BatchNormInference c-tor

* AND yet another fix to batchnorm_inference c-tor

* ops.py

* address adam's feedback

* Remove unnecessary parameter/argument.

* remove batch_norm_training_relu_with_global_stats

* remove bn_relu (training)

* [ONNX] Assert all op types supported

* Apply clang-format

* Address code review comments

* Fix #include statements

* Update ONNX Squeeze Op implementation to conform with doc. Add unit test.

* Apply code-format.

* Correct attribute value type.

* Change used loop structure.

* Modified version of loops.

- Without erase and with minimal computation time complexity.

* Run CI

* reshape sinking working on mnist_conv

* forgot to add reshape_sinking files

* refactoring of binary case

* Quantize/Dequantize case, fix add case, add assert

* address bob and scott's feedback

* debug

* fix a bug where reshapes are removed too early

* adding nGraph Quantize op

* unit test failing for floating point exception

* unit test working in float

* unit test working in uint8

* improved type checking and polished unit test - passing

* quantized axes working

* inclusive project method

* add round mode

* TODO cleanup

* code format

* adding serializer support - fails build

* add serializer support

* make CPU quantize op work; new tests for int8, clamp)

* fix build failure

* fix GPU build issue

* fix GPU unit test manifest

* use quantized offset

* add is_quantized field to element::Type

* add reduce function to coordinate.hpp

* [ONNX] Non-linear operators

* Review fix pt. 1

* Review fix pt. 2

* Non-linear tests

* style check

* Exception fix

* Test fix

* Add missing header.

* Test for ReduceSum

* Simple tests for reductions

- L1/L2/LogSum/LogSumExp/Max/Mean/Min/Prod/SumSquare.

* Add floating point literal suffix

* Fix typo

* [ONNX] Shape operator

* Review fix pt. 1

* Style check

* Move reshape utils down to reshape namespace.

* Reshape operation.

* Reshape operator binding.

* Error fixes.

* Reshape unit tests.

* Move flatten utility function to reshape namespace.

* Fix unused catched exception object

* Add Constant support for int64

* Review fix.

* clang-format

* Review fix part 2.

* Enable output shape as a second node input (only Constant).

* Unit test for "dynamic" output shape (from Constant node).

* Review fixes.

* Make sure second Reshape op input is Constant node.

* [ONNX] Concat operator

* Style fix

* [ONNX] Sum op

* [ONNX] Generic variadic op template

* Add support for Min op

* clang-format

* Add support for Max op

* Add support for Mean op

* Docs, code cleanup

* Docs, code cleanup

* [ONNX] Softmax operator

* Review fix pt. 1

* Review fix pt. 2

* Add softmax test

* Update onnx_import.cpp

* [ONNX] MatMul operator

* Add NL on EOF

* Review fix pt. 1

* Enable Mul OP

* Reshape, broadcasting utils and Gemm op

* Style check

* Review fix pt. 1

* Review fix pt. 2

* Reuse documentation

* Move batch_norm implementation to a .cpp file
* Move split implementation to a .cpp file

* [ONNX] Refactor exceptions

* [ONNX] Attribute helper functions

* [ONNX] Convolution operation

* [ONNX] Add Relu op