• Chris Sullivan's avatar
    Cache and use fprop stats in cudnn batchnorm bprop (#1841) · fbc3a940
    Chris Sullivan authored
    * Temp bn update commit.
    
    * Add CUDNNBatchNorm which adds two additional outputs to batchnorm, the batch mean and batch inv variance.
    The batch mean is the same as the output mean if the cummulative average factor is 1.0. Add BatchNormCache pass which replaces all BatchNorm ops that are inputs to BatchNormBackprop
    with CUDNNBatchNorm which outputs the saved batch statistics directly to the backprop step.
    
    * Updated bn cache pass, removed extra tests, added test checking that provided stats are used in bprop instead of batch stats.
    This test was disabled for interpreter as the reference kernel needs to be updated to use provided statistics.
    
    * Formatting.
    
    * Update to new batch norm API.
    
    * CUDNNBatchNorm -> BatchNormTrainingWithStats
    
    * new line
    
    * Preprocess input variance into BN denominator for cudnn (#1885)
    
    * Add explicit cuda kernel to calculate what cuDNN describes as the inverse
    variance. In reality, the backward cudnn kernel for BN requires 1.0f / sqrt(variance + eps),
    which is the batchnorm denominator for each channel (a numerically stable inverse stddev).
    
    This introduces op annotations for batch norm backprop and updates the cudnn_emitter to support the insertion of this cuda kernel when required.
    
    * Disable second test on INTERPRETER.
    fbc3a940
Name
Last commit
Last update
..
files Loading commit data...
models Loading commit data...
ref_generators Loading commit data...
util Loading commit data...
CMakeLists.txt Loading commit data...
algebraic_simplification.cpp Loading commit data...
all_close_f.cpp Loading commit data...
assertion.cpp Loading commit data...
autodiff.in.cpp Loading commit data...
backend_api.cpp Loading commit data...
backend_binary_elementwise.in.cpp Loading commit data...
backend_broadcast.in.cpp Loading commit data...
backend_comparison.in.cpp Loading commit data...
backend_debug_api.cpp Loading commit data...
backend_dot.in.cpp Loading commit data...
backend_one_hot.in.cpp Loading commit data...
backend_performance.cpp Loading commit data...
backend_pool.in.cpp Loading commit data...
backend_reduce.in.cpp Loading commit data...
backend_reshape.in.cpp Loading commit data...
backend_sum.in.cpp Loading commit data...
backend_test.in.cpp Loading commit data...
backend_unary_elementwise.in.cpp Loading commit data...
build_graph.cpp Loading commit data...
builder.cpp Loading commit data...
builder_autobroadcast.cpp Loading commit data...
constant_folding.cpp Loading commit data...
control_dependencies.cpp Loading commit data...
convolution_test.in.cpp Loading commit data...
coordinate.cpp Loading commit data...
copy.cpp Loading commit data...
core_fusion.cpp Loading commit data...
cpio.cpp Loading commit data...
cpu_fusion.cpp Loading commit data...
cpu_reshape_sinking.cpp Loading commit data...
cpu_test.cpp Loading commit data...
cse.cpp Loading commit data...
cudnn.cpp Loading commit data...
distributed.in.cpp Loading commit data...
element_type.cpp Loading commit data...
file_util.cpp Loading commit data...
gpu_fusion.cpp Loading commit data...
gpu_test.cpp Loading commit data...
graph_partition.cpp Loading commit data...
halide.cpp Loading commit data...
hybrid_backend.cpp Loading commit data...
includes.cpp Loading commit data...
inliner.cpp Loading commit data...
input_output_assign.cpp Loading commit data...
main.cpp Loading commit data...
mkldnn.cpp Loading commit data...
nop_elimination.cpp Loading commit data...
onnx_import.cpp Loading commit data...
onnxifi.cpp Loading commit data...
op.cpp Loading commit data...
partial_shape.cpp Loading commit data...
pass_liveness.cpp Loading commit data...
pass_manager.cpp Loading commit data...
pass_memory_layout.cpp Loading commit data...
pattern.cpp Loading commit data...
quantize_cpu.cpp Loading commit data...
reshape_elimination.cpp Loading commit data...
serialize.cpp Loading commit data...
shape.cpp Loading commit data...
tensor.cpp Loading commit data...
type_prop.cpp Loading commit data...
update_reference.sh Loading commit data...
util.cpp Loading commit data...
uuid.cpp Loading commit data...
zero_dim_tensor_elimination.cpp Loading commit data...