* Add 'Range' op with type prop tests (no DynElimination yet)

* Implement DynElimination for Range

* Add bailouts for GPU and INTELGPU backends

* Add some execution tests

* Add missing include for GPU

* Add /bigobj flag for MSVS on unit-test

* Added mode attributes to DynPad and TopK

* Rename sort_type to sort

* Throw error in pad reference implementation for symmetric mode

* Move split utility functions into core builder.

* Move activation functions to nGraph core.

* RNN cell base class.

* LSTM cell fused operator.

* Update LSTM ONNX operator to use LSTMCell fused op.

* Use Constant::create instead of make_constant.

* Remove ngraph:: prefixes and include standard headers.

* Store member shared_ptrs as object.

* Formatting.

* Run validation at the end of constructor.

* Add more doc to ActivationFunction.

* Run FusedOpDecomposition pass two times in interpreter backend.

* Remove unnecesary class member.

* Add node validation.

* Disambiguate constructors.

* Add type property test.

* Formatting and add comment with equations.

* Update IGPU backend with LSTMCell fused op.

* Fix: clip activation function input.

* Unit tests.

* Workaround for nested fused op: run FusedOpDecomposition twice.

* Fix compilation on CentOS and on GPU.

* PR feedback.

* Fix CentOS bugs.

* Address review comments.

Remove stored inputs as class members. Use node inputs directly in
decomposition.

* Fix errors.

* Review feedback: don't use decompose_op while generating Function in UTs.

* Fix merge artifacts.

* Move RNNCellBase to op/util directory.

* Fix typo for avg_pool setter method.

* Set default values for optional inputs.

* Fix typo in comment.

Fix corner case where op::Constant ctor is called with a zero-element shape and a vector of one string (#3082)

* Adding GroupConvTranspose fused operator.

* Add missing header and remove commented code.

* Remove unused variable.

* Add a few more convieniece constructors.

* Add more type prop UTs.

* Remove unused post validation functions.

* Style apply.

* Fix conversion of vector to CoordinateDiff

* Add GroupConvolutionTranspose to intel gpu backend.

* Add documentation.

* Use default (python-like) divide.

* Add execution tests for dynamic reduction ops; fix validation logic

* Add dynamic_GPU.all to manifest

* Be explicit about the dynamic_GPU prefix in manifest

* Added support for ceil mode in AvgPool

* Added ceil mode to MaxPool

* remove extra semicolon

* Add more constructor variants to support pybind which seems to have issues with multiple optional arguments

* More constructor variants for AvgPool

* More constructor variants for MaxPool

* Style fix

* Avoid constructor delegation

* Revert "Avoid constructor delegation"

This reverts commit 8efd59127bc9a16bae93b3c6b67dbcccfa95648f.

* Handle 0 and -1 in shape inputs for dynamic reshape op

* Flag to control semantics of zero values in dynreshape op

* Initial implementation of implicit broadcasting for eltwise ops. Only Add supported

* Addressed PR feedback

* cleanup

* Rename Bcast to Broadcast

* Autobroadcasting support for rest of elementwise ops

* Serializer support for autobroadcast

* Added missing autob serialization for Minimum

* Added execution unit tests and more op types to implicit broadcast elimination

* Addressed PR feedback

* Fixes windows build issue

* RVO optimization per PR feedback

* Draft of FakeQuantize operation along with UTs.

* Add FakeQuantize to implemented operators on IGPU.

* Get back FakeQuantize op case to switch.

* Fix compilation errors.

* Skip test for INTERPRETER backend and disable type_prop tests.

* Initial implementation covering the most basic case

* Cleanup of fake_quantize_with_clip UT

* Reformat the cpu unit tests manifest and unlock anothe fake quant UT

* Handle the clipping case by subtracting input_low from quantization input

* Clip the input data before quantization to avoid Selects

* UT manifest fix

* Obsolete comment removed

* Code formatting

* Broadcast input data for non-scalar in/out params

* Code formatting

* Enable the type prop tests for FakeQuantize

* Dequant the data without using the Dequantize op (fixes an edge case)

* ShuffleChannels implementation

* Validation of ShuffleChannels params

* Implementation of ShuffleChannels decompose_op()

* Formatting adjustments

* Corrected implementation and validation of op params

* Basic test of ShuffleChannels

* Negative axis value test

* Default params for the ShuffleChannels op

* ShuffleChannels test with floats

* ShuffleChannels validation unit tests

* PR comments

* Compilation error fix

* PR feedback and cleanup

* Code formatting adjustment

* Negative axis value documentation

* Docs update (PR feedback)

* PR feedback: shape and axis validation

* Modify axis semantics on shuffle op

* Revert "PR feedback: shape and axis validation"

This reverts commit 21b708e710b91da2a7e37a69c0da1f31c7743b47.

* [Fused] LeakyRelu op

* Add LeakyRelu to serializer

* Add unit tests

* Fix merge branch 'master' into mkarzyns/fused_leaky_relu

* Change broadcasting rules to NumPy style

* Remove std:: and ngraph:: prefixes

* Rename CPU Runtime LeakyRelu to CPULeakyRelu

* Style apply

* Fix cpu_fusion.fuse_leaky_relu test

* Use eigen's tanh in the fused sigmoid multiply kernel (#2946)

* Merge branch 'master' into mkarzyns/fused_leaky_relu

* Add LeakyRelu to Intel GPU backend op list

* Add LeakyRelu to Intel GPU backend op list

* Virtualize some things that crash when layout descriptor is missing

* More shape specialization

* (very bare) skeleton for dyn elimination

* Miscellaneous

* Lift i32->int64-only restriction on constant folding for Convert

* Add constant folding for ShapeOf, and some tests for new constant folders

* Tests for DynElimination

* Rename specialize_shapes to specialize_function, and add a unit test for value substitution

* Roll back overeager API change in dyn slice bprop (it has to handle right-indexed axes; bummer)

* Add a test for dynamic usage of transpose op

* Fix warning/error about variable shadowing

* Strengthen checks in apply_permutation

* Propagate Constant shapes through Transpose

* Add CHANGE_DYNAMIC_STATE where appropriate

* PR feedback, and fix unit test failure

* Fix PR reference in comment

* PR comments

* Comments for helper funcs

* Remove unique_ptr indirection for the AlignedBuffers

* Fix incorrect indexing of AlignedBuffer vector (whoops\!)

* Remove unnecessary CHANGE_DYAMIC_STATEs

* De-update pass property unit test for const folding

* Replace mystery runes with all_pass_property_off

* Split op skeleton

* Two ways to construct a fused Split to be able to use it in onnx importer

* refactor: move the util::split() helper functions to the core

* Split's decompose_op() implementation using a helper function

* Use fused Split in the onnx_importer

* Code formatting

* PR feedback

* Split helpers moved to ngraph/builder

* Basic UT - split a 1D tensor to 3 equal parts

* UT: Split 2D tensor into variable length parts

* Code formatting

* Catch the proper type of exception in the onnx_importer split()

* Initialize members in the correct order

* Type prop tests for Split

* Code formatting

* PR feedback

* SquaredDifference implementation

* Broadcast input before using it

* Simple test of SquaredDifference

* SquaredDifference validation tests

* Formatting adjustments

* Docs correction

* Exclude the unit test on iGPU

* Keep the includes in a single group

* Update intelgpu_backend.cpp

* Update unit_test.manifest

* UT for the broadcasting path

* Extend lp-norm functions to take bias.

* Move lp-norm utilities to nGraph core op/util.

* Move norm files to builder directory.

* Apply clang-format.

* Sceleton for GRN operation.

* Add GRN implementation.

* Fix reshape utility function.

* Address review comments.

* Add using namespace std.

* Review comments.

* Few fixes in grn implementation.

* Clang format.

* Basic UT.

* Fix expected data.

* Add more UT and skip them on IGPU.

* Review comments: const correctness and remove using namespace std statement.

* Unblock GRN on IGPU.

* Get back GRN op case to switch.

* merge error

* Add fused Squeeze op

* Use fused Squeeze op in ONNX importer

* Update serializer

* Add unit tests

* Add type prop tests

* Change Squeeze signature to accept a dynamic input for axes

* Update src/ngraph/op/fused/squeeze.cpp
Co-Authored-By: Adam Rogowiec <adam.rogowiec@intel.com>

* Code review comments

* Fix failing unit test

* Code review comments

* Code review comments

* style

* Add op to iGPU backend

* Add op to iGPU backend

* Temp save.

* Temp save.

* Temp save.

* Temp save.

* Temp save.

* Temp save.

* Temp save.

* Fix compile errors.

* Fix incorrect index.

* Fix UT typo.

* Interpreter passes UT.

* Fix more bugs.

* Apply style.

* Add shape check for updates tensor.

* Merge typo

* Add ScaleShift operator

* Add ScaleShift to serializer

* Add UT for ScaleShift

* Add type_prop tests for ScaleShift

* Style-fix

* Skip tests on Intel GPU

* Review fix 1

* Style fix

* Extend lp-norm functions to take bias.

* Move lp-norm utilities to nGraph core op/util.

* Move norm files to builder directory.

* Normalize fused operator implementation.

* Fused op boilerplate.

* Fix node validation and normalization across spatial axes.

* Add UT normalize across CHW with scalar scale.

* Fix expanding input tensor to 4D.

* Add more UT for 3D and 2D.

* Add more UT, with scale and across HW.

* Update to new localization of l2_norm function.

* Add type_prop UT and update gpu/igpu manifests.

* Apply clang-format.

* Add positive UT for type_prop.

* Update unit test manifests.

* Address review comments.

* Add using namespace std.

* Remove unnecessary std prefixes.

* Remove blacklisted unittests for GPU.

* Apply clang-format.

* Review comments.

* Fix clang errors.

* Basic mean normalization

* Add NVM to serializer

* Add test for mean normalization

* Add support for across_channel atribute

* Add test for mvn_mean_normalization splited by channels

* Assume that data have n and c channels

* Add support for normalize_variance attribute

* Add test for full mean variance normalization

* Add type prop test

* Skip tests on GPU

* Use ngraph builder functions instead of my own

* Update mvn.cpp

* Change order in initializer list

* Review fix

* Enable non-constructor use of shape inference

* Move GPU BatchNormTrainingWithStats shape inference out of the constructor

* Addressed PR feedback