Pruthvi/bi rnn (#2232)

* - Added reorder support for rnn weights_layer/iter

* i) fixed compilation issues ii) working but still observing precision error

* i) fixed failing rnn unit test for DEX ii) refactored workspace in RNN mkldnn emitter

* i) added support for src reorder to TNC from NTC

* reorder support for rnn output fron NTC to TNC

* - added support for rnn weight reorder ldgoi -> ldigo
- code refactor for lstm/rnn kernel in mkldnn emitter

* - refactor rnn mkldnnn kernel, change variable names

* fix RNN codegen kernel

* disbale layer rnn fusion pass, to test CI

* method to validate recurrent rnn inputs

* add correlated macthes for Recurrent RNN PM

* - simplify reorder logic for rnn_weights
- fix graph pattern for fusing rnn cell across time steps

* do weights reorders in rnn timesteps fusion

* refactored LSTM graph pass

* - Bug fix for finding the lstm inputs determenstically
- Refactored LSTM graph pass to single pass
- made changes to LSTM RNN time step fusion graph pass

* - use replace_node instead of replace_output in Lstm_step_wise fusion graph pass

* fix compilation error

* Fix GNMT rnn fusion

* check if the node is in use before replacing in RNN graph passes

*  i) fix style ii) fix topo sort issue in RNN graph pass

* style fix

* fix bug in simplify_concat pass

* replaces Lstm1 -> {GOE1, GOE2} -> {Slice1, Slice2} -> Concat -> Lstm2 with Lstm1 -> Lstm2

* cse for convert layout

* addressed PR comments

* - optimization pass to remove  Lstm1 -> {GOE1, GOE2} -> {Slice1, Slice2} -> Lstm2
- conditional fusing of LSTM cells only for the decoder

* made changes to multi layer RNN fusion callback

* fix asserts in RNN op

* - added support to fuse layers when slc=dlc for RNN cells
- bug fix on the sanity checks for RNN Op

* - support RNN layer fusion till slc = dlc
- bug fixes in multi layer rnn fusion call back

* capture reshape in the RNN weights

* Addressed PR comments

* - added comments in multi layer PM call back
- fuse only if slc == DLC across layers

* restore deleted 3_lstm_cell_forward.json file

* fix typo

* fix failing unit tets

* When processing in place slice, do not change the offset of the slice node if the argument pointer comes from function input.

* Address PR feedback: process in place slice after propagating in place input.

* Set INTERMEDIATE role before propagating in place input.

* Do not add temporaries to the variable name map before propagating in place input in codegen.

* Fix a bug in codegen.

* Fix a bug in codegen slice.

* reenable disabled rnn unit test

* fix compiler error

* - bug fix in the slicing logic for the layer fused rnn cell
- fix failing rnn unit test

* - Addressed PR comments
- removed redundant checks from the rnn graph pass
- simplified rnn call back replace node logic

* - added new multilayer rnn *.json file
- fix test case

* [PRIVATE BRANCH] Style fixes (#2080)

* Style fixes

* change order of lstm gates

* WIP bi rnn

* [PRIVATE BRANCH] Jbobba/rnn fusion review (#2113)

* Style fixes for single-layer RNN fusion

* Style fixes to multi-layer RNN

* added callback routine for bi-directional rnn

* fix rnn op ctor, rnn mkldnn emitter to accomodate bi directional rnn

* style fix

* added helper function for rnn's to query direction and cell_type

* fix clang error

* - unit test case for bi rnn fusion
- style fix

* - updated bi-rnn graph pass to handle reverse and reverse_seq ops in the predicate
- added bi-rnn inter v/s cpu unit test case
- add support to in mkldnn_utils to create_md with tnc/ntc format

* - added enum type to deduce rnn_type

* Addressed PR comments
    - handle reshapes from {t, n, c} to {n, t, c} in the graph pass

* fix style

* fix clang error

* fix style

* i) move enum specific to rnn to seperate header

src/ngraph/runtime/cpu/cpu_external_function.cpp

@@ -1104,6 +1104,7 @@ void runtime::cpu::CPU_ExternalFunction::register_common_passes(ngraph::pass::Ma
     REGISTER_KNOBBED_PASS(RNNFusion, true, runtime::cpu::pass);
     REGISTER_KNOBBED_PASS(AlgebraicSimplification, true, ngraph::pass);
     REGISTER_KNOBBED_PASS(MultiLayerRNNFusion, true, runtime::cpu::pass);
+    REGISTER_KNOBBED_PASS(BiDirectionalRnn, true, runtime::cpu::pass);
     REGISTER_KNOBBED_PASS(CPURnnMatFusion, true, runtime::cpu::pass);
     REGISTER_KNOBBED_PASS(CPUBatchFusion, true, runtime::cpu::pass);
     REGISTER_KNOBBED_PASS(ReshapeSinking, false, ngraph::pass);

src/ngraph/runtime/cpu/mkldnn_emitter.cpp

@@ -28,6 +28,7 @@
 #include "ngraph/runtime/cpu/cpu_tensor_view_wrapper.hpp"
 #include "ngraph/runtime/cpu/mkldnn_invoke.hpp"
 #include "ngraph/runtime/cpu/mkldnn_utils.hpp"
+#include "ngraph/runtime/cpu/op/rnn.hpp"
 #include "ngraph/type/element_type.hpp"
 
 using namespace ngraph::runtime::cpu;
@@ -1052,7 +1053,9 @@ size_t MKLDNNEmitter::build_rnn_forward(const mkldnn::memory::desc& src_layer_de
                                         const mkldnn::memory::desc& weights_iter_desc,
                                         const mkldnn::memory::desc& bias_desc,
                                         const mkldnn::memory::desc& dst_layer_desc,
-                                        const mkldnn::memory::desc& dst_iter_desc)
+                                        const mkldnn::memory::desc& dst_iter_desc,
+                                        const mkldnn::rnn_direction& rnn_direction,
+                                        const mkldnn::algorithm& rnn_algorithm)
 {
     size_t src_layer_index = build_memory_primitive(src_layer_desc);
     size_t src_iter_index = build_memory_primitive(src_iter_desc);
@@ -1062,10 +1065,10 @@ size_t MKLDNNEmitter::build_rnn_forward(const mkldnn::memory::desc& src_layer_de
     size_t dst_layer_index = build_memory_primitive(dst_layer_desc);
     size_t dst_iter_index = build_memory_primitive(dst_iter_desc);
 
-    mkldnn::rnn_cell::desc rnn_cell(mkldnn::algorithm::vanilla_lstm);
+    mkldnn::rnn_cell::desc rnn_cell(rnn_algorithm);
     mkldnn::rnn_forward::desc rnn_layer_desc(mkldnn::prop_kind::forward_training,
                                              rnn_cell,
-                                             mkldnn::rnn_direction::unidirectional_left2right,
+                                             rnn_direction,
                                              src_layer_desc,
                                              src_iter_desc,
                                              weights_layer_desc,
@@ -1073,6 +1076,7 @@ size_t MKLDNNEmitter::build_rnn_forward(const mkldnn::memory::desc& src_layer_de
                                              bias_desc,
                                              dst_layer_desc,
                                              dst_iter_desc);
+
     auto rnn_layer_prim_desc =
         mkldnn::rnn_forward::primitive_desc(rnn_layer_desc, executor::global_cpu_engine);
     auto workspace_index =
@@ -1080,6 +1084,7 @@ size_t MKLDNNEmitter::build_rnn_forward(const mkldnn::memory::desc& src_layer_de
     auto workspace = std::unique_ptr<MKLDNNWorkspace>(
         new MKLDNNWorkspace(rnn_layer_prim_desc.workspace_primitive_desc().get_size()));
     auto workspace_buf_index = insert_workspace(workspace);
+
     size_t rnn_index = insert_primitive(new mkldnn::rnn_forward(
         rnn_layer_prim_desc,
         mkldnn::primitive::at(*m_mkldnn_primitives[src_layer_index]),

src/ngraph/runtime/cpu/mkldnn_emitter.hpp

@@ -37,6 +37,7 @@
 #include "ngraph/runtime/cpu/op/conv_add.hpp"
 #include "ngraph/runtime/cpu/op/conv_bias.hpp"
 #include "ngraph/runtime/cpu/op/conv_relu.hpp"
+#include "ngraph/runtime/cpu/op/rnn_utils.hpp"
 #include "ngraph/shape.hpp"
 #include "ngraph/strides.hpp"
 #include "ngraph/type/element_type.hpp"
@@ -477,7 +478,28 @@ namespace ngraph
                     auto rnn_cell_n_states =
                         static_cast<unsigned long>(rnn_node->get_num_cell_states());
 
-                    if (out[0].get_shape().size() == 2 && (out[0].get_shape()[1] != feature_size))
+                    auto get_mkldnn_rnn_cell_type = [&]() {
+                        switch (rnn_node->get_rnn_type())
+                        {
+                        case rnn_utils::rnntype::vanilla_rnn: return mkldnn::algorithm::vanilla_rnn;
+                        case rnn_utils::rnntype::vanilla_gru: return mkldnn::algorithm::vanilla_gru;
+                        case rnn_utils::rnntype::vanilla_lstm:
+                            return mkldnn::algorithm::vanilla_lstm;
+                        default: throw ngraph_error("unsupported mkldnn rnn algorithm");
+                        }
+                    };
+
+                    auto get_mkldnn_rnn_direction = [&]() {
+                        switch (direction)
+                        {
+                        case 1: return mkldnn::rnn_direction::unidirectional_left2right;
+                        case 2: return mkldnn::rnn_direction::bidirectional_concat;
+                        default: throw ngraph_error("unsupported mkldnn rnn direction");
+                        }
+                    };
+
+                    if (out[0].get_shape().size() == 2 &&
+                        (out[0].get_shape()[1] != direction * feature_size))
                     {
                         throw ngraph_error(
                             "input slc{ht} feature size is not equal to output dlc{ht} feature "
@@ -508,7 +530,7 @@ namespace ngraph
                     Shape wei_iter_tz{
                         num_fused_layers, direction, feature_size, rnn_cell_n_gates, feature_size};
                     Shape bias_tz{num_fused_layers, direction, rnn_cell_n_gates, feature_size};
-                    Shape dst_layer_tz{src_sequence_length_max, batch, feature_size};
+                    Shape dst_layer_tz{src_sequence_length_max, batch, direction * feature_size};
                     Shape dst_iter_tz{
                         num_fused_layers, direction, rnn_cell_n_states, batch, feature_size};
 
@@ -534,7 +556,9 @@ namespace ngraph
                                              wei_iter_md,
                                              bias_md,
                                              dst_layer_md,
-                                             dst_iter_md);
+                                             dst_iter_md,
+                                             get_mkldnn_rnn_direction(),
+                                             get_mkldnn_rnn_cell_type());
                 }
 
                 size_t build_rnn_forward(const mkldnn::memory::desc& src_layer_desc,
@@ -543,7 +567,9 @@ namespace ngraph
                                          const mkldnn::memory::desc& weights_iter_desc,
                                          const mkldnn::memory::desc& bias_desc,
                                          const mkldnn::memory::desc& dst_layer_desc,
-                                         const mkldnn::memory::desc& dst_iter_desc);
+                                         const mkldnn::memory::desc& dst_iter_desc,
+                                         const mkldnn::rnn_direction& rnn_direction,
+                                         const mkldnn::algorithm& rnn_algorithm);
 
                 size_t build_concat(const std::vector<mkldnn::memory::desc>& inputs_data_desc,
                                     const mkldnn::memory::desc& result_desc,

src/ngraph/runtime/cpu/mkldnn_utils.cpp

@@ -366,6 +366,18 @@ mkldnn::memory::desc runtime::cpu::mkldnn_utils::create_blocked_mkldnn_md(
         }
     }
 
+    if (dims.size() == 3)
+    {
+        if (is_perm_sorted(strides, {0, 1, 2}))
+        {
+            return memory::desc(dim, dtype, memory::format::tnc);
+        }
+        if (is_perm_sorted(strides, {1, 0, 2}))
+        {
+            return memory::desc(dim, dtype, memory::format::ntc);
+        }
+    }
+
     if (dims.size() == 4)
     {
         if (is_perm_sorted(strides, {0, 1, 2, 3}))
@@ -450,6 +462,10 @@ memory::desc runtime::cpu::mkldnn_utils::try_get_named_md(const mkldnn_memory_de
     {
     case 1: CANONICALIZE_MD(mkldnn_x); break;
     case 2: CANONICALIZE_MD(mkldnn_nc); break;
+    case 3:
+        CANONICALIZE_MD(mkldnn_tnc);
+        CANONICALIZE_MD(mkldnn_ntc);
+        break;
     case 4:
         CANONICALIZE_MD(mkldnn_nchw);
         CANONICALIZE_MD(mkldnn_nhwc);

src/ngraph/runtime/cpu/op/lstm.cpp

@@ -28,14 +28,15 @@ shared_ptr<Node> op::Lstm::copy_with_new_args(const NodeVector& new_args) const
         throw ngraph_error("Incorrect number of new arguments");
     }
     return make_shared<Lstm>(
-        new_args.at(0), new_args.at(1), new_args.at(2), new_args.at(3), new_args.at(4));
+        new_args.at(0), new_args.at(1), new_args.at(2), new_args.at(3), new_args.at(4), m_rnntype);
 }
 
 op::Lstm::Lstm(std::shared_ptr<Node> src_layer,
                std::shared_ptr<Node> src_iter,
                std::shared_ptr<Node> weights_layer,
                std::shared_ptr<Node> weights_iter,
-               std::shared_ptr<Node> bias)
+               std::shared_ptr<Node> bias,
+               ngraph::runtime::cpu::rnn_utils::rnntype rnn_type)
     : Op("Lstm", check_single_output_args({src_layer, src_iter, weights_layer, weights_iter, bias}))
     , m_output_tensor_shape(src_layer->get_shape())
     , m_output_cell_shape(src_iter->get_shape())
@@ -47,6 +48,7 @@ op::Lstm::Lstm(std::shared_ptr<Node> src_layer,
     , m_num_cell_states(2)
     , m_direction(1)
     , m_num_fused_layers(1)
+    , m_rnntype(rnn_type)
 {
     constructor_validate_and_infer_types();
 

src/ngraph/runtime/cpu/op/lstm.hpp

@@ -17,6 +17,7 @@
 #pragma once
 
 #include "ngraph/op/op.hpp"
+#include "ngraph/runtime/cpu/op/rnn_utils.hpp"
 #include "ngraph/util.hpp"
 
 namespace ngraph
@@ -43,9 +44,11 @@ namespace ngraph
                  std::shared_ptr<Node> src_iter,
                  std::shared_ptr<Node> weights_layer,
                  std::shared_ptr<Node> weights_iter,
-                 std::shared_ptr<Node> bias);
+                 std::shared_ptr<Node> bias,
+                 ngraph::runtime::cpu::rnn_utils::rnntype rnn_type);
             Shape get_output_tensor_shape() const { return m_output_tensor_shape; }
             Shape get_output_cell_shape() const { return m_output_cell_shape; }
+            ngraph::runtime::cpu::rnn_utils::rnntype get_rnn_type() const { return m_rnntype; }
             size_t get_num_timesteps() const { return m_num_timesteps; }
             size_t get_src_sequence_length() const { return m_src_sequence_length; }
             size_t get_gates_per_cell() const { return m_num_gates_per_cell; }
@@ -70,6 +73,7 @@ namespace ngraph
             size_t m_num_cell_states;
             size_t m_direction;
             size_t m_num_fused_layers;
+            ngraph::runtime::cpu::rnn_utils::rnntype m_rnntype;
         };
     }
 }

src/ngraph/runtime/cpu/op/rnn.cpp

@@ -37,7 +37,8 @@ shared_ptr<Node> op::Rnn::copy_with_new_args(const NodeVector& new_args) const
                             m_src_sequence_length,
                             m_num_cell_states,
                             m_direction,
-                            m_num_fused_layers);
+                            m_num_fused_layers,
+                            m_rnntype);
 }
 
 op::Rnn::Rnn(std::shared_ptr<Node> src_layer,
@@ -50,7 +51,8 @@ op::Rnn::Rnn(std::shared_ptr<Node> src_layer,
              size_t src_sequence_length,
              size_t num_cell_states,
              size_t direction,
-             size_t num_fused_layers)
+             size_t num_fused_layers,
+             ngraph::runtime::cpu::rnn_utils::rnntype rnn_type)
     : Op("Rnn", check_single_output_args({src_layer, src_iter, weights_layer, weights_iter, bias}))
     , m_num_timesteps(num_timesteps)
     , m_num_gates_per_cell(num_gates_per_cell)
@@ -58,6 +60,7 @@ op::Rnn::Rnn(std::shared_ptr<Node> src_layer,
     , m_num_cell_states(num_cell_states)
     , m_direction(direction)
     , m_num_fused_layers(num_fused_layers)
+    , m_rnntype(rnn_type)
 {
     constructor_validate_and_infer_types();
     if (src_layer->get_shape().size() != weights_layer->get_shape().size())
@@ -90,8 +93,10 @@ op::Rnn::Rnn(std::shared_ptr<Node> src_layer,
         throw ngraph_error("src_layer size is not equal t*n*c");
     }
 
-    if ((bias->get_shape()[0] / m_num_fused_layers) != (weights_layer->get_shape()[1]) ||
-        (bias->get_shape()[0] / m_num_fused_layers) != (weights_iter->get_shape()[1]))
+    if ((bias->get_shape()[0] / (m_direction * m_num_fused_layers)) !=
+            (weights_layer->get_shape()[1]) ||
+        (bias->get_shape()[0] / (m_direction * m_num_fused_layers)) !=
+            (weights_iter->get_shape()[1]))
     {
         throw ngraph_error("bias and weights_shape are not compatible");
     }
@@ -108,7 +113,7 @@ op::Rnn::Rnn(std::shared_ptr<Node> src_layer,
     set_output_size(2);
     set_output_type(0,
                     src_layer->get_element_type(),
-                    Shape{(m_direction * m_num_timesteps * m_batch_size), m_src_iter_feature_size});
+                    Shape{(m_num_timesteps * m_batch_size), m_direction * m_src_iter_feature_size});
     set_output_type(1,
                     src_layer->get_element_type(),
                     Shape{(m_num_cell_states * m_direction * m_num_fused_layers * m_batch_size),

src/ngraph/runtime/cpu/op/rnn.hpp

@@ -18,6 +18,7 @@
 
 #include "ngraph/op/op.hpp"
 #include "ngraph/runtime/cpu/cpu_backend_visibility.h"
+#include "ngraph/runtime/cpu/op/rnn_utils.hpp"
 #include "ngraph/util.hpp"
 
 namespace ngraph
@@ -57,9 +58,12 @@ namespace ngraph
                                 size_t src_sequence_length,
                                 size_t num_cell_states,
                                 size_t direction,
-                                size_t num_fused_layers);
+                                size_t num_fused_layers,
+                                ngraph::runtime::cpu::rnn_utils::rnntype rnn_type);
             virtual std::shared_ptr<Node>
                 copy_with_new_args(const NodeVector& new_args) const override;
+
+            ngraph::runtime::cpu::rnn_utils::rnntype get_rnn_type() const { return m_rnntype; }
             size_t get_num_timesteps() const { return m_num_timesteps; }
             size_t get_src_sequence_length() const { return m_src_sequence_length; }
             size_t get_gates_per_cell() const { return m_num_gates_per_cell; }
@@ -83,6 +87,7 @@ namespace ngraph
             size_t m_num_cell_states;
             size_t m_direction;
             size_t m_num_fused_layers;
+            ngraph::runtime::cpu::rnn_utils::rnntype m_rnntype;
         };
     }
 }

src/ngraph/runtime/cpu/op/rnn_utils.hpp

+//*****************************************************************************
+// Copyright 2017-2019 Intel Corporation
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//*****************************************************************************
+
+#pragma once
+
+#include <cstddef>
+#include <cstdint>
+
+namespace ngraph
+{
+    namespace runtime
+    {
+        namespace cpu
+        {
+            namespace rnn_utils
+            {
+                // TODO(pruthvi): Populate this enums based of addition of new MKLDNN RNN variants
+                enum rnntype
+                {
+                    vanilla_rnn,
+                    vanilla_gru,
+                    vanilla_lstm
+                };
+            }
+        }
+    }
+}

src/ngraph/runtime/cpu/pass/cpu_fusion.cpp

@@ -70,6 +70,7 @@
 #include "ngraph/runtime/cpu/op/leaky_relu.hpp"
 #include "ngraph/runtime/cpu/op/lstm.hpp"
 #include "ngraph/runtime/cpu/op/matmul_bias.hpp"
+#include "ngraph/runtime/cpu/op/rnn_utils.hpp"
 #include "ngraph/runtime/cpu/op/sigmoid_mul.hpp"
 #include "ngraph/runtime/cpu/op/update_slice.hpp"
 #include "ngraph/util.hpp"
@@ -1780,8 +1781,14 @@ void ngraph::runtime::cpu::pass::CPUFusion::construct_fuse_lstm_recurrent_state(
     auto weights_layer_label = std::make_shared<pattern::op::Label>(element::f32, Shape{100, 400});
     auto weights_iter_label = std::make_shared<pattern::op::Label>(element::f32, Shape{100, 400});
     auto bias_label = std::make_shared<pattern::op::Label>(element::f32, Shape{400});
-    auto lstm1 = std::make_shared<op::Lstm>(
-        src_layer_label, src_iter_label, weights_layer_label, weights_iter_label, bias_label);
+    ngraph::runtime::cpu::rnn_utils::rnntype rnn_type =
+        ngraph::runtime::cpu::rnn_utils::rnntype::vanilla_lstm;
+    auto lstm1 = std::make_shared<op::Lstm>(src_layer_label,
+                                            src_iter_label,
+                                            weights_layer_label,
+                                            weights_iter_label,
+                                            bias_label,
+                                            rnn_type);
 
     auto lstm1_goe0 = std::make_shared<op::GetOutputElement>(lstm1, 0);
     auto lstm1_goe1 = std::make_shared<op::GetOutputElement>(lstm1, 1);

src/ngraph/runtime/cpu/pass/cpu_rnn_fusion.cpp

@@ -39,14 +39,18 @@
 #include "ngraph/op/relu.hpp"
 #include "ngraph/op/reshape.hpp"
 #include "ngraph/op/result.hpp"
+#include "ngraph/op/reverse.hpp"
+#include "ngraph/op/reverse_sequence.hpp"
 #include "ngraph/op/slice.hpp"
 #include "ngraph/op/sum.hpp"
 #include "ngraph/op/tanh.hpp"
 #include "ngraph/pattern/matcher.hpp"
 #include "ngraph/pattern/op/label.hpp"
 #include "ngraph/pattern/op/skip.hpp"
+#include "ngraph/runtime/cpu/mkldnn_utils.hpp"
 #include "ngraph/runtime/cpu/op/lstm.hpp"
 #include "ngraph/runtime/cpu/op/rnn.hpp"
+#include "ngraph/runtime/cpu/op/rnn_utils.hpp"
 #include "ngraph/runtime/cpu/op/sigmoid.hpp"
 
 #define STR(X) #X
@@ -266,6 +270,8 @@ void ngraph::runtime::cpu::pass::LSTMFusion::construct_lstm_fprop()
             auto slc = weights_layer->get_shape()[0];
             auto dic = weights_iter->get_shape()[1] / (lstm_n_gates * direction * layers);
             auto sic = weights_iter->get_shape()[0];
+            ngraph::runtime::cpu::rnn_utils::rnntype rnn_type =
+                ngraph::runtime::cpu::rnn_utils::rnntype::vanilla_lstm;
 
             if (dlc != dic)
             {
@@ -284,8 +290,8 @@ void ngraph::runtime::cpu::pass::LSTMFusion::construct_lstm_fprop()
 
             auto bias = std::make_shared<op::Add>(pattern_map[bias_i2h], pattern_map[bias_h2h]);
 
-            auto lstm_node =
-                std::make_shared<op::Lstm>(src_layer, src_iter, weights_layer, weights_iter, bias);
+            auto lstm_node = std::make_shared<op::Lstm>(
+                src_layer, src_iter, weights_layer, weights_iter, bias, rnn_type);
 
             auto lstm_ht_output = std::make_shared<op::GetOutputElement>(lstm_node, 0);
             auto lstm_ht_ct_output = std::make_shared<op::GetOutputElement>(lstm_node, 1);
@@ -347,12 +353,15 @@ void ngraph::runtime::cpu::pass::RNNFusion::construct_rnn_lstm_fprop()
     auto lstm_bias = std::make_shared<op::Add>(lstm_bias_layer_shared, lstm_bias_iter_shared);
     auto lstm_bias_label =
         std::make_shared<pattern::op::Label>(lstm_bias, nullptr, NodeVector{lstm_bias});
+    ngraph::runtime::cpu::rnn_utils::rnntype ref_rnn_type =
+        ngraph::runtime::cpu::rnn_utils::rnntype::vanilla_lstm;
 
     auto lstm = std::make_shared<op::Lstm>(lstm_src_layer,
                                            lstm_src_iter_label,
                                            lstm_weights_layer_label,
                                            lstm_weights_iter_label,
-                                           lstm_bias_label);
+                                           lstm_bias_label,
+                                           ref_rnn_type);
     auto lstm_goe = std::make_shared<op::GetOutputElement>(lstm, 1);
     // We cannot attach labels to multi-output nodes, so we attach a label to the goe instead
     auto lstm_goe_label =
@@ -403,6 +412,8 @@ void ngraph::runtime::cpu::pass::RNNFusion::construct_rnn_lstm_fprop()
         const size_t num_cell_states = 2;
         const size_t direction = 1;
         const size_t num_fused_rnn_layers = 1;
+        ngraph::runtime::cpu::rnn_utils::rnntype rnn_type =
+            ngraph::runtime::cpu::rnn_utils::rnntype::vanilla_lstm;
 
         NGRAPH_DEBUG << "src_layer: " << join(rnn_src_layer->get_shape());
         NGRAPH_DEBUG << "src_iter: " << join(rnn_src_iter->get_shape());
@@ -459,7 +470,8 @@ void ngraph::runtime::cpu::pass::RNNFusion::construct_rnn_lstm_fprop()
                                              sequence_len,
                                              num_cell_states,
                                              direction,
-                                             num_fused_rnn_layers);
+                                             num_fused_rnn_layers,
+                                             rnn_type);
 
         std::vector<std::shared_ptr<op::Slice>> ht_slice_per_timestep(sequence_len, nullptr);
         auto rnn_ht_goe = std::make_shared<op::GetOutputElement>(rnn, 0);
@@ -564,13 +576,14 @@ void ngraph::runtime::cpu::pass::MultiLayerRNNFusion::construct_multi_layer_rnn_
     auto rnn_weights_layer = std::make_shared<pattern::op::Label>(element::f32, Shape{100, 400});
     auto rnn_weights_iter = std::make_shared<pattern::op::Label>(element::f32, Shape{100, 400});
     auto rnn_bias = std::make_shared<pattern::op::Label>(element::f32, Shape{400});
-
     const size_t ref_number_of_timesteps = 3;
     const size_t ref_number_of_gates_per_cell = 4;
     const size_t ref_src_seq_length = 3;
     const size_t ref_num_rnn_cell_states = 2;
     const size_t ref_rnn_direction = 1;
     const size_t ref_num_of_rnn_fused_layer = 1;
+    ngraph::runtime::cpu::rnn_utils::rnntype ref_rnn_type =
+        ngraph::runtime::cpu::rnn_utils::rnntype::vanilla_lstm;
 
     auto ref_rnn_node = std::make_shared<op::Rnn>(rnn_src_layer,
                                                   rnn_src_iter,
@@ -582,7 +595,8 @@ void ngraph::runtime::cpu::pass::MultiLayerRNNFusion::construct_multi_layer_rnn_
                                                   ref_src_seq_length,
                                                   ref_num_rnn_cell_states,
                                                   ref_rnn_direction,
-                                                  ref_num_of_rnn_fused_layer);
+                                                  ref_num_of_rnn_fused_layer,
+                                                  ref_rnn_type);
 
     auto rnn_goe0 = std::make_shared<op::GetOutputElement>(ref_rnn_node, 0);
 
@@ -630,6 +644,7 @@ void ngraph::runtime::cpu::pass::MultiLayerRNNFusion::construct_multi_layer_rnn_
         size_t num_rnn_cell_states = rnn_nodes[0]->get_num_cell_states();
         size_t rnn_direction = rnn_nodes[0]->get_direction();
         size_t num_fused_rnn_layers = rnn_nodes.size();
+        ngraph::runtime::cpu::rnn_utils::rnntype rnn_type = rnn_nodes[0]->get_rnn_type();
 
         for (auto rnn_node : rnn_nodes)
         {
@@ -690,7 +705,8 @@ void ngraph::runtime::cpu::pass::MultiLayerRNNFusion::construct_multi_layer_rnn_
                                              sequence_len,
                                              num_rnn_cell_states,
                                              rnn_direction,
-                                             num_fused_rnn_layers);
+                                             num_fused_rnn_layers,
+                                             rnn_type);
 
         auto mrnn_ht = std::make_shared<op::GetOutputElement>(rnn, 0);
         auto mrnn_ht_ct = std::make_shared<op::GetOutputElement>(rnn, 1);
@@ -751,3 +767,138 @@ void ngraph::runtime::cpu::pass::MultiLayerRNNFusion::construct_multi_layer_rnn_
         rnn_goe0_label, rnn_src_layer, empty_correlated_matches, callback);
     this->add_matcher(m);
 }
+
+void ngraph::runtime::cpu::pass::BiDirectionalRnn::construct_bidirectional_rnn()
+{
+    auto rnn_left_to_right = std::make_shared<pattern::op::Label>(
+        element::f32, Shape{1, 256}, pattern::has_class<op::Rnn>());
+    auto rnn_right_to_left = std::make_shared<pattern::op::Label>(
+        element::f32, Shape{1, 256}, pattern::has_class<op::Rnn>());
+
+    auto reshape_pred = [](std::shared_ptr<Node> n) {
+        return (std::dynamic_pointer_cast<op::Reshape>(n) != nullptr);
+    };
+    auto rnn_left_to_right_goe0 = std::make_shared<op::GetOutputElement>(rnn_left_to_right, 0);
+    auto rnn_right_to_left_goe0 = std::make_shared<op::GetOutputElement>(rnn_right_to_left, 0);
+
+    auto rnn_rtol_goe0_reshape_ntc =
+        std::make_shared<pattern::op::Skip>(rnn_right_to_left_goe0, reshape_pred);
+    auto rnn_rtol_goe0_reshape_tnc =
+        std::make_shared<pattern::op::Skip>(rnn_rtol_goe0_reshape_ntc, reshape_pred);
+    auto rnn_ltor_goe0_reshape_ntc =
+        std::make_shared<pattern::op::Skip>(rnn_left_to_right_goe0, reshape_pred);
+    auto rnn_ltor_goe0_reshape_tnc =
+        std::make_shared<pattern::op::Skip>(rnn_ltor_goe0_reshape_ntc, reshape_pred);
+
+    auto reverse_seq_predicate = [](std::shared_ptr<Node> node) {
+        return pattern::has_class<op::ReverseSequence>()(node) ||
+               pattern::has_class<op::Reverse>()(node);
+    };
+    auto skip_reverse_seq =
+        std::make_shared<pattern::op::Skip>(rnn_rtol_goe0_reshape_tnc, reverse_seq_predicate);
+    auto concat =
+        std::make_shared<op::Concat>(NodeVector{rnn_ltor_goe0_reshape_tnc, skip_reverse_seq}, 0);
+
+    // Define a call back that needs to called once the DFG matches the pattern
+    ngraph::pattern::graph_rewrite_callback callback = [rnn_left_to_right,
+                                                        rnn_right_to_left](pattern::Matcher& m) {
+
+        auto pattern_map = m.get_pattern_map();
+        auto rnn_ltor_node = std::dynamic_pointer_cast<op::Rnn>(pattern_map[rnn_left_to_right]);
+        auto rnn_rtol_node = std::dynamic_pointer_cast<op::Rnn>(pattern_map[rnn_right_to_left]);
+
+        if (rnn_ltor_node->get_src_sequence_length() != rnn_rtol_node->get_src_sequence_length())
+        {
+            NGRAPH_DEBUG << " Not fusing, timestep of rnn's in both direction should match";
+            return false;
+        }
+
+        if (rnn_ltor_node->get_src_layer_feature_size() !=
+            rnn_rtol_node->get_src_layer_feature_size())
+        {
+            NGRAPH_DEBUG << " Not fusing, feature_size of rnn's in both direction should match";
+            return false;
+        }
+
+        if (rnn_ltor_node->get_src_iter_feature_size() !=
+            rnn_rtol_node->get_src_iter_feature_size())
+        {
+            NGRAPH_DEBUG << " Not fusing, feature_size of rnn's in both direction should match";
+            return false;
+        }
+
+        if (rnn_ltor_node->get_batch_size() != rnn_rtol_node->get_batch_size())
+        {
+            NGRAPH_DEBUG << " Not fusing, feature_size of rnn's in both direction should match";
+            return false;
+        }
+
+        if (rnn_ltor_node->get_src_sequence_length() != rnn_rtol_node->get_src_sequence_length())
+        {
+            NGRAPH_DEBUG << " Not fusing, sequence length  of rnn's in both direction should match";
+            return false;
+        }
+
+        size_t num_time_steps = rnn_ltor_node->get_num_timesteps();
+        size_t lstm_n_gates = rnn_ltor_node->get_gates_per_cell();
+        size_t sequence_len = rnn_ltor_node->get_src_sequence_length();
+        size_t num_rnn_cell_states = rnn_ltor_node->get_num_cell_states();
+        size_t rnn_direction = 2;
+        size_t num_fused_rnn_layers = 1;
+        ngraph::runtime::cpu::rnn_utils::rnntype rnn_type =
+            ngraph::runtime::cpu::rnn_utils::rnntype::vanilla_lstm;
+
+        auto construct_birnn_inputs = [&](int index) {
+
+            auto nodes =
+                NodeVector{rnn_ltor_node->get_argument(index), rnn_rtol_node->get_argument(index)};
+            return std::make_shared<op::Concat>(nodes, 0);
+        };
+
+        auto src_layer = rnn_ltor_node->get_arguments()[0];
+        auto src_iter = construct_birnn_inputs(1);
+        auto weights_layer = construct_birnn_inputs(2);
+        auto weights_iter = construct_birnn_inputs(3);
+        auto bias = construct_birnn_inputs(4);
+
+        auto rnn = std::make_shared<op::Rnn>(src_layer,
+                                             src_iter,
+                                             weights_layer,
+                                             weights_iter,
+                                             bias,
+                                             num_time_steps,
+                                             lstm_n_gates,
+                                             sequence_len,
+                                             num_rnn_cell_states,
+                                             rnn_direction,
+                                             num_fused_rnn_layers,
+                                             rnn_type);
+
+        auto layer_rnn_ht = std::make_shared<op::GetOutputElement>(rnn, 0);
+        size_t batch_size = layer_rnn_ht->get_shape()[0] / num_time_steps;
+        size_t feature_size = layer_rnn_ht->get_shape()[1];
+
+        // if the shape doesnt match, we will logically reshape it to expaned_dims{tnc} from squeezed_dims{t*n, c}
+        std::shared_ptr<Node> layer_rnn_ht_reshape = layer_rnn_ht;
+        if (m.get_match_root()->get_shape() != layer_rnn_ht->get_shape())
+        {
+            layer_rnn_ht_reshape = std::make_shared<op::Reshape>(
+                layer_rnn_ht, AxisVector{0, 1}, Shape{num_time_steps, batch_size, feature_size});
+        }
+
+        // we will check if the node being replaced is in Shape{n, t, c}, if so we will transpose
+        if (m.get_match_root()->get_shape() == Shape{batch_size, num_time_steps, feature_size})
+        {
+            layer_rnn_ht_reshape =
+                std::make_shared<op::Reshape>(layer_rnn_ht_reshape,
+                                              AxisVector{1, 0, 2},
+                                              Shape{batch_size, num_time_steps, feature_size});
+        }
+
+        ngraph::replace_node(m.get_match_root(), layer_rnn_ht_reshape);
+        return true;
+    };
+
+    auto m = std::make_shared<ngraph::pattern::Matcher>(concat, callback);
+    this->add_matcher(m);
+}

src/ngraph/runtime/cpu/pass/cpu_rnn_fusion.hpp

@@ -30,6 +30,7 @@ namespace ngraph
             {
                 class LSTMFusion;
                 class RNNFusion;
+                class BiDirectionalRnn;
                 class MultiLayerRNNFusion;
             }
         }
@@ -77,3 +78,16 @@ public:
 private:
     void construct_multi_layer_rnn_fusion_fprop();
 };
+
+class ngraph::runtime::cpu::pass::BiDirectionalRnn : public ngraph::pass::GraphRewrite
+{
+public:
+    BiDirectionalRnn()
+        : GraphRewrite()
+    {
+        construct_bidirectional_rnn();
+    }
+
+private:
+    void construct_bidirectional_rnn();
+};

test/cpu_fusion.cpp

@@ -38,6 +38,7 @@
 #include "ngraph/op/parameter.hpp"
 #include "ngraph/op/quantize.hpp"
 #include "ngraph/op/relu.hpp"
+#include "ngraph/op/reverse_sequence.hpp"
 #include "ngraph/op/sigmoid.hpp"
 #include "ngraph/op/sum.hpp"
 #include "ngraph/op/tanh.hpp"
@@ -66,6 +67,7 @@
 #include "ngraph/runtime/cpu/op/lstm.hpp"
 #include "ngraph/runtime/cpu/op/matmul_bias.hpp"
 #include "ngraph/runtime/cpu/op/rnn.hpp"
+#include "ngraph/runtime/cpu/op/rnn_utils.hpp"
 #include "ngraph/runtime/cpu/op/sigmoid_mul.hpp"
 #include "ngraph/runtime/cpu/op/update_slice.hpp"
 #include "ngraph/runtime/cpu/pass/cpu_fusion.hpp"
@@ -2180,6 +2182,9 @@ TEST(cpu_fusion, rnn_fprop_1_lstm_cell)
     const int num_rnn_cell_states = 2;
     const int rnn_direction = 1;
     const int num_of_rnn_fused_layer = 1;
+    ngraph::runtime::cpu::rnn_utils::rnntype rnn_type =
+        ngraph::runtime::cpu::rnn_utils::rnntype::vanilla_lstm;
+
     auto rnn_node = make_shared<op::Rnn>(src_layer,
                                          src_iter,
                                          weights_layer,
@@ -2190,7 +2195,9 @@ TEST(cpu_fusion, rnn_fprop_1_lstm_cell)
                                          src_seq_length,
                                          num_rnn_cell_states,
                                          rnn_direction,
-                                         num_of_rnn_fused_layer);
+                                         num_of_rnn_fused_layer,
+                                         rnn_type);
+
     auto rnn_ht_output = make_shared<op::GetOutputElement>(rnn_node, 0);
     auto rnn_ct_output = make_shared<op::GetOutputElement>(rnn_node, 1);
 
@@ -3590,7 +3597,6 @@ TEST(cpu_quant_fusion, qconvb_relu)
         rng.initialize(tensor_val);
         args.push_back(tensor_val);
     }
-
     set_environment("NGRAPH_PASS_ENABLES", "CPUQuantFusion:0", 1);
     auto cpu1_results = execute(cpu_f1, args, "CPU");
     set_environment("NGRAPH_PASS_ENABLES", "CPUQuantFusion:1", 1);
@@ -3808,3 +3814,46 @@ TEST(cpu_quant_fusion, qconvba)
     auto cpu2_results = execute(cpu_f2, args, "CPU");
     EXPECT_TRUE(test::all_close(cpu1_results.at(0), cpu2_results.at(0)));
 }
+
+TEST(cpu_fusion, fuse_bi_directional_rnn)
+{
+    pass::Manager pass_manager;
+    pass_manager.register_pass<runtime::cpu::pass::LSTMFusion>();
+    pass_manager.register_pass<runtime::cpu::pass::RNNFusion>();
+    pass_manager.register_pass<ngraph::pass::AlgebraicSimplification>();
+    pass_manager.register_pass<runtime::cpu::pass::MultiLayerRNNFusion>();
+    pass_manager.register_pass<runtime::cpu::pass::BiDirectionalRnn>();
+    const string json_path = file_util::path_join(SERIALIZED_ZOO, "mxnet/lstm_bi_directional.json");
+    const string json_string = file_util::read_file_to_string(json_path);
+    stringstream ss(json_string);
+    shared_ptr<Function> func = ngraph::deserialize(ss);
+    pass_manager.run_passes(func);
+    // Bidirectional graph pass will folds the reverse seq
+    auto rev_seq_ops = get_ops_of_type<op::Reverse>(func);
+    auto rnn_ops = get_ops_of_type<op::Rnn>(func);
+    EXPECT_EQ(rev_seq_ops.size(), 0);
+    // fuse two bi-directional rnn layers in to one MKLDNN Op
+    EXPECT_EQ(rnn_ops.size(), 1);
+}
+
+TEST(cpu_fusion, bi_rnn_interpreter_vs_cpu)
+{
+    const std::string file_name("mxnet/lstm_bi_directional.json");
+    auto cpu_f = make_function_from_file(file_name);
+    auto int_f = make_function_from_file(file_name);
+    test::Uniform<float> rng(0.0f, 1.0f);
+    vector<vector<float>> args;
+
+    for (shared_ptr<op::Parameter> param : int_f->get_parameters())
+    {
+        vector<float> tensor_val(shape_size(param->get_shape()));
+        rng.initialize(tensor_val);
+        args.push_back(tensor_val);
+    }
+    auto int_results = execute(int_f, args, "INTERPRETER");
+    auto cpu_results = execute(cpu_f, args, "CPU");
+    for (size_t i = 0; i < int_results.size(); i++)
+    {
+        EXPECT_TRUE(test::all_close(cpu_results.at(i), int_results.at(i), 1.0e-4f, 1.0e-4f));
+    }
+}

test/models/mxnet/lstm_bi_directional.json

+[{"name":"Function_12","ops":[{"cacheable":true,"control_deps":[],"element_type":"float","inputs":[],"name":"Parameter_1321","op":"Parameter","outputs":["Parameter_1321_0"],"shape":[40]},{"cacheable":true,"control_deps":[],"element_type":"float","inputs":[],"name":"Parameter_1320","op":"Parameter","outputs":["Parameter_1320_0"],"shape":[40,10]},{"cacheable":true,"control_deps":[],"element_type":"float","inputs":[],"name":"Parameter_1313","op":"Parameter","outputs":["Parameter_1313_0"],"shape":[40]},{"cacheable":true,"control_deps":[],"element_type":"float","inputs":[],"name":"Parameter_1312","op":"Parameter","outputs":["Parameter_1312_0"],"shape":[40,10]},{"cacheable":true,"control_deps":[],"element_type":"float","inputs":[],"name":"Parameter_1234","op":"Parameter","outputs":["Parameter_1234_0"],"shape":[40]},{"cacheable":true,"control_deps":[],"element_type":"float","inputs":[],"name":"Parameter_1233","op":"Parameter","outputs":["Parameter_1233_0"],"shape":[40,10]},{"cacheable":true,"control_deps":[],"element_type":"float","inputs":[],"name":"Parameter_1226","op":"Parameter","outputs":["Parameter_1226_0"],"shape":[40]},{"cacheable":true,"control_deps":[],"element_type":"float","inputs":[],"name":"Parameter_1225","op":"Parameter","outputs":["Parameter_1225_0"],"shape":[40,10]},{"cacheable":false,"control_deps":[],"element_type":"float","inputs":[],"name":"Parameter_1222","op":"Parameter","outputs":["Parameter_1222_0"],"shape":[1,2,10]},{"control_deps":[],"element_type":"float","inputs":[],"name":"Constant_1318","op":"Constant","outputs":["Constant_1318_0"],"shape":[],"value":["0"]},{"control_deps":[],"element_type":"float","inputs":[],"name":"Constant_1345","op":"Constant","outputs":["Constant_1345_0"],"shape":[],"value":["1"]},{"control_deps":[],"element_type":"float","inputs":[],"name":"Constant_1341","op":"Constant","outputs":["Constant_1341_0"],"shape":[],"value":["0"]},{"control_deps":[],"element_type":"float","inputs":[],"name":"Constant_1335","op":"Constant","outputs":["Constant_1335_0"],"shape":[],"value":["1"]},{"control_deps":[],"element_type":"float","inputs":[],"name":"Constant_1328","op":"Constant","outputs":["Constant_1328_0"],"shape":[],"value":["1"]},{"control_deps":[],"element_type":"float","inputs":[],"name":"Constant_1383","op":"Constant","outputs":["Constant_1383_0"],"shape":[],"value":["1"]},{"control_deps":[],"element_type":"float","inputs":[],"name":"Constant_1375","op":"Constant","outputs":["Constant_1375_0"],"shape":[],"value":["1"]},{"control_deps":[],"element_type":"float","inputs":[],"name":"Constant_1368","op":"Constant","outputs":["Constant_1368_0"],"shape":[],"value":["1"]},{"control_deps":[],"element_type":"float","inputs":[],"name":"Constant_1231","op":"Constant","outputs":["Constant_1231_0"],"shape":[],"value":["0"]},{"control_deps":[],"element_type":"float","inputs":[],"name":"Constant_1258","op":"Constant","outputs":["Constant_1258_0"],"shape":[],"value":["1"]},{"control_deps":[],"element_type":"float","inputs":[],"name":"Constant_1254","op":"Constant","outputs":["Constant_1254_0"],"shape":[],"value":["0"]},{"control_deps":[],"element_type":"float","inputs":[],"name":"Constant_1248","op":"Constant","outputs":["Constant_1248_0"],"shape":[],"value":["1"]},{"control_deps":[],"element_type":"float","inputs":[],"name":"Constant_1241","op":"Constant","outputs":["Constant_1241_0"],"shape":[],"value":["1"]},{"control_deps":[],"element_type":"float","inputs":[],"name":"Constant_1298","op":"Constant","outputs":["Constant_1298_0"],"shape":[],"value":["1"]},{"control_deps":[],"element_type":"float","inputs":[],"name":"Constant_1290","op":"Constant","outputs":["Constant_1290_0"],"shape":[],"value":["1"]},{"control_deps":[],"element_type":"float","inputs":[],"name":"Constant_1283","op":"Constant","outputs":["Constant_1283_0"],"shape":[],"value":["1"]},{"control_deps":[],"input_order":[0],"inputs":["Parameter_1321"],"name":"Reshape_1324","op":"Reshape","output_shape":[1,40],"outputs":["Reshape_1324_0"]},{"control_deps":[],"input_order":[0],"inputs":["Parameter_1321"],"name":"Reshape_1364","op":"Reshape","output_shape":[1,40],"outputs":["Reshape_1364_0"]},{"control_deps":[],"input_order":[1,0],"inputs":["Parameter_1320"],"name":"Reshape_1322","op":"Reshape","output_shape":[10,40],"outputs":["Reshape_1322_0"]},{"control_deps":[],"input_order":[1,0],"inputs":["Parameter_1320"],"name":"Reshape_1362","op":"Reshape","output_shape":[10,40],"outputs":["Reshape_1362_0"]},{"control_deps":[],"input_order":[0],"inputs":["Parameter_1313"],"name":"Reshape_1316","op":"Reshape","output_shape":[1,40],"outputs":["Reshape_1316_0"]},{"control_deps":[],"input_order":[0],"inputs":["Parameter_1313"],"name":"Reshape_1360","op":"Reshape","output_shape":[1,40],"outputs":["Reshape_1360_0"]},{"control_deps":[],"input_order":[1,0],"inputs":["Parameter_1312"],"name":"Reshape_1314","op":"Reshape","output_shape":[10,40],"outputs":["Reshape_1314_0"]},{"control_deps":[],"input_order":[1,0],"inputs":["Parameter_1312"],"name":"Reshape_1358","op":"Reshape","output_shape":[10,40],"outputs":["Reshape_1358_0"]},{"control_deps":[],"input_order":[0],"inputs":["Parameter_1234"],"name":"Reshape_1237","op":"Reshape","output_shape":[1,40],"outputs":["Reshape_1237_0"]},{"control_deps":[],"input_order":[0],"inputs":["Parameter_1234"],"name":"Reshape_1279","op":"Reshape","output_shape":[1,40],"outputs":["Reshape_1279_0"]},{"control_deps":[],"input_order":[1,0],"inputs":["Parameter_1233"],"name":"Reshape_1235","op":"Reshape","output_shape":[10,40],"outputs":["Reshape_1235_0"]},{"control_deps":[],"input_order":[1,0],"inputs":["Parameter_1233"],"name":"Reshape_1277","op":"Reshape","output_shape":[10,40],"outputs":["Reshape_1277_0"]},{"control_deps":[],"input_order":[0],"inputs":["Parameter_1226"],"name":"Reshape_1229","op":"Reshape","output_shape":[1,40],"outputs":["Reshape_1229_0"]},{"control_deps":[],"input_order":[0],"inputs":["Parameter_1226"],"name":"Reshape_1275","op":"Reshape","output_shape":[1,40],"outputs":["Reshape_1275_0"]},{"control_deps":[],"input_order":[1,0],"inputs":["Parameter_1225"],"name":"Reshape_1227","op":"Reshape","output_shape":[10,40],"outputs":["Reshape_1227_0"]},{"control_deps":[],"input_order":[1,0],"inputs":["Parameter_1225"],"name":"Reshape_1273","op":"Reshape","output_shape":[10,40],"outputs":["Reshape_1273_0"]},{"control_deps":[],"inputs":["Parameter_1222"],"lower_bounds":[0,0,0],"name":"Slice_1223","op":"Slice","outputs":["Slice_1223_0"],"strides":[1,1,1],"upper_bounds":[1,1,10]},{"control_deps":[],"inputs":["Parameter_1222"],"lower_bounds":[0,1,0],"name":"Slice_1271","op":"Slice","outputs":["Slice_1271_0"],"strides":[1,1,1],"upper_bounds":[1,2,10]},{"axes":[0,1],"control_deps":[],"inputs":["Constant_1318"],"name":"Broadcast_1319","op":"Broadcast","outputs":["Broadcast_1319_0"],"shape":[1,10]},{"axes":[0,1],"control_deps":[],"inputs":["Constant_1345"],"name":"Broadcast_1346","op":"Broadcast","outputs":["Broadcast_1346_0"],"shape":[1,10]},{"axes":[0,1],"control_deps":[],"inputs":["Constant_1341"],"name":"Broadcast_1342","op":"Broadcast","outputs":["Broadcast_1342_0"],"shape":[1,10]},{"axes":[0,1],"control_deps":[],"inputs":["Constant_1335"],"name":"Broadcast_1336","op":"Broadcast","outputs":["Broadcast_1336_0"],"shape":[1,10]},{"axes":[0,1],"control_deps":[],"inputs":["Constant_1328"],"name":"Broadcast_1329","op":"Broadcast","outputs":["Broadcast_1329_0"],"shape":[1,10]},{"axes":[0,1],"control_deps":[],"inputs":["Constant_1383"],"name":"Broadcast_1384","op":"Broadcast","outputs":["Broadcast_1384_0"],"shape":[1,10]},{"axes":[0,1],"control_deps":[],"inputs":["Constant_1375"],"name":"Broadcast_1376","op":"Broadcast","outputs":["Broadcast_1376_0"],"shape":[1,10]},{"axes":[0,1],"control_deps":[],"inputs":["Constant_1368"],"name":"Broadcast_1369","op":"Broadcast","outputs":["Broadcast_1369_0"],"shape":[1,10]},{"axes":[0,1],"control_deps":[],"inputs":["Constant_1231"],"name":"Broadcast_1232","op":"Broadcast","outputs":["Broadcast_1232_0"],"shape":[1,10]},{"axes":[0,1],"control_deps":[],"inputs":["Constant_1258"],"name":"Broadcast_1259","op":"Broadcast","outputs":["Broadcast_1259_0"],"shape":[1,10]},{"axes":[0,1],"control_deps":[],"inputs":["Constant_1254"],"name":"Broadcast_1255","op":"Broadcast","outputs":["Broadcast_1255_0"],"shape":[1,10]},{"axes":[0,1],"control_deps":[],"inputs":["Constant_1248"],"name":"Broadcast_1249","op":"Broadcast","outputs":["Broadcast_1249_0"],"shape":[1,10]},{"axes":[0,1],"control_deps":[],"inputs":["Constant_1241"],"name":"Broadcast_1242","op":"Broadcast","outputs":["Broadcast_1242_0"],"shape":[1,10]},{"axes":[0,1],"control_deps":[],"inputs":["Constant_1298"],"name":"Broadcast_1299","op":"Broadcast","outputs":["Broadcast_1299_0"],"shape":[1,10]},{"axes":[0,1],"control_deps":[],"inputs":["Constant_1290"],"name":"Broadcast_1291","op":"Broadcast","outputs":["Broadcast_1291_0"],"shape":[1,10]},{"axes":[0,1],"control_deps":[],"inputs":["Constant_1283"],"name":"Broadcast_1284","op":"Broadcast","outputs":["Broadcast_1284_0"],"shape":[1,10]},{"control_deps":[],"input_order":[0,1,2],"inputs":["Slice_1223"],"name":"Reshape_1224","op":"Reshape","output_shape":[1,10],"outputs":["Reshape_1224_0"]},{"control_deps":[],"input_order":[0,1,2],"inputs":["Slice_1271"],"name":"Reshape_1272","op":"Reshape","output_shape":[1,10],"outputs":["Reshape_1272_0"]},{"control_deps":[],"inputs":["Broadcast_1319","Reshape_1322"],"name":"Dot_1323","op":"Dot","outputs":["Dot_1323_0"],"reduction_axes_count":1},{"control_deps":[],"inputs":["Broadcast_1232","Reshape_1235"],"name":"Dot_1236","op":"Dot","outputs":["Dot_1236_0"],"reduction_axes_count":1},{"control_deps":[],"inputs":["Reshape_1224","Reshape_1227"],"name":"Dot_1228","op":"Dot","outputs":["Dot_1228_0"],"reduction_axes_count":1},{"control_deps":[],"inputs":["Reshape_1224","Reshape_1358"],"name":"Dot_1359","op":"Dot","outputs":["Dot_1359_0"],"reduction_axes_count":1},{"control_deps":[],"inputs":["Reshape_1272","Reshape_1273"],"name":"Dot_1274","op":"Dot","outputs":["Dot_1274_0"],"reduction_axes_count":1},{"control_deps":[],"inputs":["Reshape_1272","Reshape_1314"],"name":"Dot_1315","op":"Dot","outputs":["Dot_1315_0"],"reduction_axes_count":1},{"control_deps":[],"inputs":["Dot_1323","Reshape_1324"],"name":"Add_1325","op":"Add","outputs":["Add_1325_0"]},{"control_deps":[],"inputs":["Dot_1236","Reshape_1237"],"name":"Add_1238","op":"Add","outputs":["Add_1238_0"]},{"control_deps":[],"inputs":["Dot_1228","Reshape_1229"],"name":"Add_1230","op":"Add","outputs":["Add_1230_0"]},{"control_deps":[],"inputs":["Dot_1359","Reshape_1360"],"name":"Add_1361","op":"Add","outputs":["Add_1361_0"]},{"control_deps":[],"inputs":["Dot_1274","Reshape_1275"],"name":"Add_1276","op":"Add","outputs":["Add_1276_0"]},{"control_deps":[],"inputs":["Dot_1315","Reshape_1316"],"name":"Add_1317","op":"Add","outputs":["Add_1317_0"]},{"control_deps":[],"inputs":["Add_1230","Add_1238"],"name":"Add_1239","op":"Add","outputs":["Add_1239_0"]},{"control_deps":[],"inputs":["Add_1317","Add_1325"],"name":"Add_1326","op":"Add","outputs":["Add_1326_0"]},{"control_deps":[],"inputs":["Add_1239"],"lower_bounds":[0,30],"name":"Slice_1240","op":"Slice","outputs":["Slice_1240_0"],"strides":[1,1],"upper_bounds":[1,40]},{"control_deps":[],"inputs":["Add_1239"],"lower_bounds":[0,10],"name":"Slice_1247","op":"Slice","outputs":["Slice_1247_0"],"strides":[1,1],"upper_bounds":[1,20]},{"control_deps":[],"inputs":["Add_1239"],"lower_bounds":[0,0],"name":"Slice_1257","op":"Slice","outputs":["Slice_1257_0"],"strides":[1,1],"upper_bounds":[1,10]},{"control_deps":[],"inputs":["Add_1239"],"lower_bounds":[0,20],"name":"Slice_1264","op":"Slice","outputs":["Slice_1264_0"],"strides":[1,1],"upper_bounds":[1,30]},{"control_deps":[],"inputs":["Add_1326"],"lower_bounds":[0,30],"name":"Slice_1327","op":"Slice","outputs":["Slice_1327_0"],"strides":[1,1],"upper_bounds":[1,40]},{"control_deps":[],"inputs":["Add_1326"],"lower_bounds":[0,10],"name":"Slice_1334","op":"Slice","outputs":["Slice_1334_0"],"strides":[1,1],"upper_bounds":[1,20]},{"control_deps":[],"inputs":["Add_1326"],"lower_bounds":[0,0],"name":"Slice_1344","op":"Slice","outputs":["Slice_1344_0"],"strides":[1,1],"upper_bounds":[1,10]},{"control_deps":[],"inputs":["Add_1326"],"lower_bounds":[0,20],"name":"Slice_1351","op":"Slice","outputs":["Slice_1351_0"],"strides":[1,1],"upper_bounds":[1,30]},{"control_deps":[],"inputs":["Slice_1240"],"name":"Negative_1243","op":"Negative","outputs":["Negative_1243_0"]},{"control_deps":[],"inputs":["Slice_1247"],"name":"Negative_1250","op":"Negative","outputs":["Negative_1250_0"]},{"control_deps":[],"inputs":["Slice_1257"],"name":"Negative_1260","op":"Negative","outputs":["Negative_1260_0"]},{"control_deps":[],"inputs":["Slice_1264"],"name":"Tanh_1265","op":"Tanh","outputs":["Tanh_1265_0"]},{"control_deps":[],"inputs":["Slice_1327"],"name":"Negative_1330","op":"Negative","outputs":["Negative_1330_0"]},{"control_deps":[],"inputs":["Slice_1334"],"name":"Negative_1337","op":"Negative","outputs":["Negative_1337_0"]},{"control_deps":[],"inputs":["Slice_1344"],"name":"Negative_1347","op":"Negative","outputs":["Negative_1347_0"]},{"control_deps":[],"inputs":["Slice_1351"],"name":"Tanh_1352","op":"Tanh","outputs":["Tanh_1352_0"]},{"control_deps":[],"inputs":["Negative_1243"],"name":"Exp_1244","op":"Exp","outputs":["Exp_1244_0"]},{"control_deps":[],"inputs":["Negative_1250"],"name":"Exp_1251","op":"Exp","outputs":["Exp_1251_0"]},{"control_deps":[],"inputs":["Negative_1260"],"name":"Exp_1261","op":"Exp","outputs":["Exp_1261_0"]},{"control_deps":[],"inputs":["Negative_1330"],"name":"Exp_1331","op":"Exp","outputs":["Exp_1331_0"]},{"control_deps":[],"inputs":["Negative_1337"],"name":"Exp_1338","op":"Exp","outputs":["Exp_1338_0"]},{"control_deps":[],"inputs":["Negative_1347"],"name":"Exp_1348","op":"Exp","outputs":["Exp_1348_0"]},{"control_deps":[],"inputs":["Broadcast_1242","Exp_1244"],"name":"Add_1245","op":"Add","outputs":["Add_1245_0"]},{"control_deps":[],"inputs":["Broadcast_1249","Exp_1251"],"name":"Add_1252","op":"Add","outputs":["Add_1252_0"]},{"control_deps":[],"inputs":["Broadcast_1259","Exp_1261"],"name":"Add_1262","op":"Add","outputs":["Add_1262_0"]},{"control_deps":[],"inputs":["Broadcast_1329","Exp_1331"],"name":"Add_1332","op":"Add","outputs":["Add_1332_0"]},{"control_deps":[],"inputs":["Broadcast_1336","Exp_1338"],"name":"Add_1339","op":"Add","outputs":["Add_1339_0"]},{"control_deps":[],"inputs":["Broadcast_1346","Exp_1348"],"name":"Add_1349","op":"Add","outputs":["Add_1349_0"]},{"control_deps":[],"inputs":["Broadcast_1242","Add_1245"],"name":"Divide_1246","op":"Divide","outputs":["Divide_1246_0"]},{"control_deps":[],"inputs":["Broadcast_1249","Add_1252"],"name":"Divide_1253","op":"Divide","outputs":["Divide_1253_0"]},{"control_deps":[],"inputs":["Broadcast_1259","Add_1262"],"name":"Divide_1263","op":"Divide","outputs":["Divide_1263_0"]},{"control_deps":[],"inputs":["Broadcast_1329","Add_1332"],"name":"Divide_1333","op":"Divide","outputs":["Divide_1333_0"]},{"control_deps":[],"inputs":["Broadcast_1336","Add_1339"],"name":"Divide_1340","op":"Divide","outputs":["Divide_1340_0"]},{"control_deps":[],"inputs":["Broadcast_1346","Add_1349"],"name":"Divide_1350","op":"Divide","outputs":["Divide_1350_0"]},{"control_deps":[],"inputs":["Divide_1253","Broadcast_1255"],"name":"Multiply_1256","op":"Multiply","outputs":["Multiply_1256_0"]},{"control_deps":[],"inputs":["Divide_1263","Tanh_1265"],"name":"Multiply_1266","op":"Multiply","outputs":["Multiply_1266_0"]},{"control_deps":[],"inputs":["Divide_1340","Broadcast_1342"],"name":"Multiply_1343","op":"Multiply","outputs":["Multiply_1343_0"]},{"control_deps":[],"inputs":["Divide_1350","Tanh_1352"],"name":"Multiply_1353","op":"Multiply","outputs":["Multiply_1353_0"]},{"control_deps":[],"inputs":["Multiply_1256","Multiply_1266"],"name":"Add_1267","op":"Add","outputs":["Add_1267_0"]},{"control_deps":[],"inputs":["Multiply_1343","Multiply_1353"],"name":"Add_1354","op":"Add","outputs":["Add_1354_0"]},{"control_deps":[],"inputs":["Add_1267"],"name":"Tanh_1268","op":"Tanh","outputs":["Tanh_1268_0"]},{"control_deps":[],"inputs":["Add_1354"],"name":"Tanh_1355","op":"Tanh","outputs":["Tanh_1355_0"]},{"control_deps":[],"inputs":["Divide_1246","Tanh_1268"],"name":"Multiply_1269","op":"Multiply","outputs":["Multiply_1269_0"]},{"control_deps":[],"inputs":["Divide_1333","Tanh_1355"],"name":"Multiply_1356","op":"Multiply","outputs":["Multiply_1356_0"]},{"control_deps":[],"input_order":[0,1],"inputs":["Multiply_1269"],"name":"Reshape_1270","op":"Reshape","output_shape":[1,1,10],"outputs":["Reshape_1270_0"]},{"control_deps":[],"inputs":["Multiply_1269","Reshape_1277"],"name":"Dot_1278","op":"Dot","outputs":["Dot_1278_0"],"reduction_axes_count":1},{"control_deps":[],"input_order":[0,1],"inputs":["Multiply_1356"],"name":"Reshape_1357","op":"Reshape","output_shape":[1,1,10],"outputs":["Reshape_1357_0"]},{"control_deps":[],"inputs":["Multiply_1356","Reshape_1362"],"name":"Dot_1363","op":"Dot","outputs":["Dot_1363_0"],"reduction_axes_count":1},{"control_deps":[],"inputs":["Dot_1278","Reshape_1279"],"name":"Add_1280","op":"Add","outputs":["Add_1280_0"]},{"control_deps":[],"inputs":["Dot_1363","Reshape_1364"],"name":"Add_1365","op":"Add","outputs":["Add_1365_0"]},{"control_deps":[],"inputs":["Add_1276","Add_1280"],"name":"Add_1281","op":"Add","outputs":["Add_1281_0"]},{"control_deps":[],"inputs":["Add_1361","Add_1365"],"name":"Add_1366","op":"Add","outputs":["Add_1366_0"]},{"control_deps":[],"inputs":["Add_1281"],"lower_bounds":[0,30],"name":"Slice_1282","op":"Slice","outputs":["Slice_1282_0"],"strides":[1,1],"upper_bounds":[1,40]},{"control_deps":[],"inputs":["Add_1281"],"lower_bounds":[0,10],"name":"Slice_1289","op":"Slice","outputs":["Slice_1289_0"],"strides":[1,1],"upper_bounds":[1,20]},{"control_deps":[],"inputs":["Add_1281"],"lower_bounds":[0,0],"name":"Slice_1297","op":"Slice","outputs":["Slice_1297_0"],"strides":[1,1],"upper_bounds":[1,10]},{"control_deps":[],"inputs":["Add_1281"],"lower_bounds":[0,20],"name":"Slice_1304","op":"Slice","outputs":["Slice_1304_0"],"strides":[1,1],"upper_bounds":[1,30]},{"control_deps":[],"inputs":["Add_1366"],"lower_bounds":[0,30],"name":"Slice_1367","op":"Slice","outputs":["Slice_1367_0"],"strides":[1,1],"upper_bounds":[1,40]},{"control_deps":[],"inputs":["Add_1366"],"lower_bounds":[0,10],"name":"Slice_1374","op":"Slice","outputs":["Slice_1374_0"],"strides":[1,1],"upper_bounds":[1,20]},{"control_deps":[],"inputs":["Add_1366"],"lower_bounds":[0,0],"name":"Slice_1382","op":"Slice","outputs":["Slice_1382_0"],"strides":[1,1],"upper_bounds":[1,10]},{"control_deps":[],"inputs":["Add_1366"],"lower_bounds":[0,20],"name":"Slice_1389","op":"Slice","outputs":["Slice_1389_0"],"strides":[1,1],"upper_bounds":[1,30]},{"control_deps":[],"inputs":["Slice_1282"],"name":"Negative_1285","op":"Negative","outputs":["Negative_1285_0"]},{"control_deps":[],"inputs":["Slice_1289"],"name":"Negative_1292","op":"Negative","outputs":["Negative_1292_0"]},{"control_deps":[],"inputs":["Slice_1297"],"name":"Negative_1300","op":"Negative","outputs":["Negative_1300_0"]},{"control_deps":[],"inputs":["Slice_1304"],"name":"Tanh_1305","op":"Tanh","outputs":["Tanh_1305_0"]},{"control_deps":[],"inputs":["Slice_1367"],"name":"Negative_1370","op":"Negative","outputs":["Negative_1370_0"]},{"control_deps":[],"inputs":["Slice_1374"],"name":"Negative_1377","op":"Negative","outputs":["Negative_1377_0"]},{"control_deps":[],"inputs":["Slice_1382"],"name":"Negative_1385","op":"Negative","outputs":["Negative_1385_0"]},{"control_deps":[],"inputs":["Slice_1389"],"name":"Tanh_1390","op":"Tanh","outputs":["Tanh_1390_0"]},{"control_deps":[],"inputs":["Negative_1285"],"name":"Exp_1286","op":"Exp","outputs":["Exp_1286_0"]},{"control_deps":[],"inputs":["Negative_1292"],"name":"Exp_1293","op":"Exp","outputs":["Exp_1293_0"]},{"control_deps":[],"inputs":["Negative_1300"],"name":"Exp_1301","op":"Exp","outputs":["Exp_1301_0"]},{"control_deps":[],"inputs":["Negative_1370"],"name":"Exp_1371","op":"Exp","outputs":["Exp_1371_0"]},{"control_deps":[],"inputs":["Negative_1377"],"name":"Exp_1378","op":"Exp","outputs":["Exp_1378_0"]},{"control_deps":[],"inputs":["Negative_1385"],"name":"Exp_1386","op":"Exp","outputs":["Exp_1386_0"]},{"control_deps":[],"inputs":["Broadcast_1284","Exp_1286"],"name":"Add_1287","op":"Add","outputs":["Add_1287_0"]},{"control_deps":[],"inputs":["Broadcast_1291","Exp_1293"],"name":"Add_1294","op":"Add","outputs":["Add_1294_0"]},{"control_deps":[],"inputs":["Broadcast_1299","Exp_1301"],"name":"Add_1302","op":"Add","outputs":["Add_1302_0"]},{"control_deps":[],"inputs":["Broadcast_1369","Exp_1371"],"name":"Add_1372","op":"Add","outputs":["Add_1372_0"]},{"control_deps":[],"inputs":["Broadcast_1376","Exp_1378"],"name":"Add_1379","op":"Add","outputs":["Add_1379_0"]},{"control_deps":[],"inputs":["Broadcast_1384","Exp_1386"],"name":"Add_1387","op":"Add","outputs":["Add_1387_0"]},{"control_deps":[],"inputs":["Broadcast_1284","Add_1287"],"name":"Divide_1288","op":"Divide","outputs":["Divide_1288_0"]},{"control_deps":[],"inputs":["Broadcast_1291","Add_1294"],"name":"Divide_1295","op":"Divide","outputs":["Divide_1295_0"]},{"control_deps":[],"inputs":["Broadcast_1299","Add_1302"],"name":"Divide_1303","op":"Divide","outputs":["Divide_1303_0"]},{"control_deps":[],"inputs":["Broadcast_1369","Add_1372"],"name":"Divide_1373","op":"Divide","outputs":["Divide_1373_0"]},{"control_deps":[],"inputs":["Broadcast_1376","Add_1379"],"name":"Divide_1380","op":"Divide","outputs":["Divide_1380_0"]},{"control_deps":[],"inputs":["Broadcast_1384","Add_1387"],"name":"Divide_1388","op":"Divide","outputs":["Divide_1388_0"]},{"control_deps":[],"inputs":["Divide_1295","Add_1267"],"name":"Multiply_1296","op":"Multiply","outputs":["Multiply_1296_0"]},{"control_deps":[],"inputs":["Divide_1303","Tanh_1305"],"name":"Multiply_1306","op":"Multiply","outputs":["Multiply_1306_0"]},{"control_deps":[],"inputs":["Divide_1380","Add_1354"],"name":"Multiply_1381","op":"Multiply","outputs":["Multiply_1381_0"]},{"control_deps":[],"inputs":["Divide_1388","Tanh_1390"],"name":"Multiply_1391","op":"Multiply","outputs":["Multiply_1391_0"]},{"control_deps":[],"inputs":["Multiply_1296","Multiply_1306"],"name":"Add_1307","op":"Add","outputs":["Add_1307_0"]},{"control_deps":[],"inputs":["Multiply_1381","Multiply_1391"],"name":"Add_1392","op":"Add","outputs":["Add_1392_0"]},{"control_deps":[],"inputs":["Add_1307"],"name":"Tanh_1308","op":"Tanh","outputs":["Tanh_1308_0"]},{"control_deps":[],"inputs":["Add_1392"],"name":"Tanh_1393","op":"Tanh","outputs":["Tanh_1393_0"]},{"control_deps":[],"inputs":["Divide_1288","Tanh_1308"],"name":"Multiply_1309","op":"Multiply","outputs":["Multiply_1309_0"]},{"control_deps":[],"inputs":["Divide_1373","Tanh_1393"],"name":"Multiply_1394","op":"Multiply","outputs":["Multiply_1394_0"]},{"control_deps":[],"input_order":[0,1],"inputs":["Multiply_1309"],"name":"Reshape_1310","op":"Reshape","output_shape":[1,1,10],"outputs":["Reshape_1310_0"]},{"control_deps":[],"input_order":[0,1],"inputs":["Multiply_1394"],"name":"Reshape_1395","op":"Reshape","output_shape":[1,1,10],"outputs":["Reshape_1395_0"]},{"axis":1,"control_deps":[],"inputs":["Reshape_1270","Reshape_1310"],"name":"Concat_1311","op":"Concat","outputs":["Concat_1311_0"]},{"axis":1,"control_deps":[],"inputs":["Reshape_1357","Reshape_1395"],"name":"Concat_1396","op":"Concat","outputs":["Concat_1396_0"]},{"control_deps":[],"inputs":["Concat_1396"],"name":"Reverse_1397","op":"Reverse","outputs":["Reverse_1397_0"],"reversed_axes":[1]},{"axis":2,"control_deps":[],"inputs":["Concat_1311","Reverse_1397"],"name":"Concat_1398","op":"Concat","outputs":["Concat_1398_0"]},{"control_deps":[],"inputs":["Concat_1398"],"name":"Result_1399","op":"Result","outputs":["Result_1399_0"]}],"parameters":["Parameter_1222","Parameter_1225","Parameter_1226","Parameter_1233","Parameter_1234","Parameter_1312","Parameter_1313","Parameter_1320","Parameter_1321"],"result":["Result_1399"]}]