Merge branch 'contributing' of https://github.com/haon4/protobuf into contributing

CONTRIBUTING.md

@@ -83,3 +83,13 @@ the final release.
   Ultimately, the green signal will be provided by our testing infrastructure.
   The reviewer will help you if there are test failures that seem not related
   to the change you are making.
+* Apply the "release notes: yes" label if the pull request's description should
+  be included in the next release (e.g., any new feature / bug fix).
+  Apply the "release notes: no" label if the pull request's description should
+  not be included in the next release (e.g., refactoring changes that does not
+  change behavior, integration from Google internal, updating tests, etc.).
+* Apply the appropriate language label (e.g., C++, Java, Python, etc.) to the
+  pull request. This will make it easier to identify which languages the pull
+  request affects, allowing us to better identify appropriate reviewer, create
+  a better release note, and make it easier to identify issues in the future.
+  

Makefile.am

@@ -49,6 +49,7 @@ pkgconfigdir = $(libdir)/pkgconfig
 pkgconfig_DATA = protobuf.pc protobuf-lite.pc
 
 csharp_EXTRA_DIST=                                                           \
+  global.json                                                                \
   csharp/.gitignore                                                          \
   csharp/CHANGES.txt                                                         \
   csharp/Google.Protobuf.Tools.targets                                       \
@@ -56,9 +57,10 @@ csharp_EXTRA_DIST=                                                           \
   csharp/README.md                                                           \
   csharp/build_packages.bat                                                  \
   csharp/build_tools.sh                                                      \
+  csharp/buildall.bat                                                        \
   csharp/buildall.sh                                                         \
   csharp/generate_protos.sh                                                  \
-  csharp/global.json                                                         \
+  csharp/install_dotnet_sdk.ps1                                              \
   csharp/keys/Google.Protobuf.public.snk                                     \
   csharp/keys/Google.Protobuf.snk                                            \
   csharp/keys/README.md                                                      \
@@ -75,6 +77,12 @@ csharp_EXTRA_DIST=                                                           \
   csharp/src/AddressBook/ListPeople.cs                                       \
   csharp/src/AddressBook/Program.cs                                          \
   csharp/src/AddressBook/SampleUsage.cs                                      \
+  csharp/src/Google.Protobuf.Benchmarks/SerializationConfig.cs               \
+  csharp/src/Google.Protobuf.Benchmarks/SerializationBenchmark.cs            \
+  csharp/src/Google.Protobuf.Benchmarks/Program.cs                           \
+  csharp/src/Google.Protobuf.Benchmarks/Google.Protobuf.Benchmarks.csproj    \
+  csharp/src/Google.Protobuf.Benchmarks/Benchmarks.cs                        \
+  csharp/src/Google.Protobuf.Benchmarks/BenchmarkMessage1Proto3.cs           \
   csharp/src/Google.Protobuf.Conformance/Conformance.cs                      \
   csharp/src/Google.Protobuf.Conformance/Google.Protobuf.Conformance.csproj  \
   csharp/src/Google.Protobuf.Conformance/Program.cs                          \
@@ -737,6 +745,7 @@ php_EXTRA_DIST=                                                       \
   php/tests/autoload.php                                              \
   php/tests/bootstrap_phpunit.php                                     \
   php/tests/compatibility_test.sh                                     \
+  php/tests/compile_extension.sh                                      \
   php/tests/descriptors_test.php                                      \
   php/tests/encode_decode_test.php                                    \
   php/tests/gdb_test.sh                                               \

appveyor.bat

@@ -38,7 +38,7 @@ dotnet restore
 dotnet build -c %configuration% || goto error
 
 echo Testing C#
-dotnet test -c %configuration% -f netcoreapp1.0 Google.Protobuf.Test\Google.Protobuf.Test.csproj || goto error
+dotnet test -c %configuration% -f netcoreapp2.1 Google.Protobuf.Test\Google.Protobuf.Test.csproj || goto error
 dotnet test -c %configuration% -f net451 Google.Protobuf.Test\Google.Protobuf.Test.csproj || goto error
 
 goto :EOF

benchmarks/README.md

@@ -65,6 +65,12 @@ include PHP protobuf's src and build the c extension if required.
 ### Node.js
 Node.js benchmark need [node](https://nodejs.org/en/)(higher than V6) and [npm](https://www.npmjs.com/) package manager installed. This benchmark is using the [benchmark](https://www.npmjs.com/package/benchmark) framework to test, which needn't to manually install. And another prerequisite is [protobuf js](https://github.com/protocolbuffers/protobuf/tree/master/js), which needn't to manually install either
 
+### C#
+The C# benchmark code is built as part of the main Google.Protobuf
+solution. It requires the .NET Core SDK, and depends on
+[BenchmarkDotNet](https://github.com/dotnet/BenchmarkDotNet), which
+will be downloaded automatically.
+
 ### Big data
 
 There's some optional big testing data which is not included in the directory
@@ -74,7 +80,7 @@ initially, you need to run the following command to download the testing data:
 $ ./download_data.sh
 ```
 
-After doing this the big data file will automaticly generated in the
+After doing this the big data file will automatically generated in the
 benchmark directory.
 
 ## Run instructions
@@ -209,6 +215,15 @@ $ make js-benchmark
 $ ./js-benchmark $(specific generated dataset file name)
 ```
 
+### C#
+From `csharp/src/Google.Protobuf.Benchmarks`, run:
+
+```
+$ dotnet run -c Release
+```
+
+We intend to add support for this within the makefile in due course.
+
 ## Benchmark datasets
 
 Each data set is in the format of benchmarks.proto:

conformance/Makefile.am

@@ -316,7 +316,7 @@ conformance-java-lite: javac_middleman_lite
 conformance-csharp: $(other_language_protoc_outputs)
 	@echo "Writing shortcut script conformance-csharp..."
 	@echo '#! /bin/sh' > conformance-csharp
-	@echo 'dotnet ../csharp/src/Google.Protobuf.Conformance/bin/Release/netcoreapp1.0/Google.Protobuf.Conformance.dll "$$@"' >> conformance-csharp
+	@echo 'dotnet ../csharp/src/Google.Protobuf.Conformance/bin/Release/netcoreapp2.1/Google.Protobuf.Conformance.dll "$$@"' >> conformance-csharp
 	@chmod +x conformance-csharp
 
 conformance-php:

csharp/.gitignore

@@ -29,3 +29,6 @@ mono/*.exe
 mono/*.dll
 lib/protoc.exe
 *.ncrunch*
+
+# Benchmark output
+BenchmarkDotNet.Artifacts/

csharp/build_packages.bat

 @rem Builds Google.Protobuf NuGet packages
 
 dotnet restore src/Google.Protobuf.sln
-dotnet pack -c Release src/Google.Protobuf.sln /p:SourceLinkCreate=true || goto :error
+dotnet pack -c Release src/Google.Protobuf.sln || goto :error
 
 goto :EOF
 

csharp/buildall.bat

+@rem Builds Google.Protobuf and runs the tests
+
+dotnet build src/Google.Protobuf.sln || goto :error
+
+echo Running tests.
+
+dotnet test src/Google.Protobuf.Test/Google.Protobuf.Test.csproj || goto :error
+
+goto :EOF
+
+:error
+echo Failed!
+exit /b %errorlevel%

csharp/buildall.sh

@@ -10,8 +10,8 @@ dotnet restore $SRC/Google.Protobuf.sln
 dotnet build -c $CONFIG $SRC/Google.Protobuf.sln
 
 echo Running tests.
-# Only test netcoreapp1.0, which uses the .NET Core runtime.
+# Only test netcoreapp2.1, which uses the .NET Core runtime.
 # If we want to test the .NET 4.5 version separately, we could
 # run Mono explicitly. However, we don't have any differences between
-# the .NET 4.5 and netstandard1.0 assemblies.
-dotnet test -c $CONFIG -f netcoreapp1.0 $SRC/Google.Protobuf.Test/Google.Protobuf.Test.csproj
+# the .NET 4.5 and netstandard2.1 assemblies.
+dotnet test -c $CONFIG -f netcoreapp2.1 $SRC/Google.Protobuf.Test/Google.Protobuf.Test.csproj

csharp/compatibility_tests/v3.0.0/src/Google.Protobuf.Test/Google.Protobuf.Test.csproj

@@ -2,7 +2,7 @@
 
   <PropertyGroup>
     <OutputType>Exe</OutputType>
-    <TargetFrameworks>net451;netcoreapp1.0</TargetFrameworks>
+    <TargetFrameworks>net451;netcoreapp2.1</TargetFrameworks>
     <AssemblyOriginatorKeyFile>../../keys/Google.Protobuf.snk</AssemblyOriginatorKeyFile>
     <SignAssembly>true</SignAssembly>
     <PublicSign Condition=" '$(OS)' != 'Windows_NT' ">true</PublicSign>
@@ -24,7 +24,7 @@
     - Visual Studio.
     -->
   <PropertyGroup Condition="'$(OS)' != 'Windows_NT'">
-    <TargetFrameworks>netcoreapp1.0</TargetFrameworks>
+    <TargetFrameworks>netcoreapp2.1</TargetFrameworks>
   </PropertyGroup>
 
 </Project>

csharp/compatibility_tests/v3.0.0/test.sh

@@ -2,17 +2,17 @@
 
 function run_test() {
   # Generate test proto files.
-  ./protoc_1 -Iprotos/src -I../../../src/ --csharp_out=src/Google.Protobuf.Test \
+  $1 -Iprotos/src -I../../../src/ --csharp_out=src/Google.Protobuf.Test \
     --csharp_opt=base_namespace=Google.Protobuf \
     protos/src/google/protobuf/unittest_import_proto3.proto \
     protos/src/google/protobuf/unittest_import_public_proto3.proto \
     protos/src/google/protobuf/unittest_well_known_types.proto
 
-  ./protoc_1 -Iprotos/csharp --csharp_out=src/Google.Protobuf.Test \
+  $1 -Iprotos/csharp --csharp_out=src/Google.Protobuf.Test \
     --csharp_opt=base_namespace=UnitTest.Issues \
     protos/csharp/protos/unittest_issues.proto
 
-  ./protoc_2 -Iprotos/src --csharp_out=src/Google.Protobuf.Test \
+  $2 -Iprotos/src --csharp_out=src/Google.Protobuf.Test \
     --csharp_opt=base_namespace=Google.Protobuf \
     protos/src/google/protobuf/unittest_proto3.proto \
     protos/src/google/protobuf/map_unittest_proto3.proto
@@ -22,7 +22,7 @@ function run_test() {
   dotnet restore src/Google.Protobuf.Test/Google.Protobuf.Test.csproj
   dotnet build -c Release src/Google.Protobuf/Google.Protobuf.csproj
   dotnet build -c Release src/Google.Protobuf.Test/Google.Protobuf.Test.csproj
-  dotnet run -c Release -f netcoreapp1.0 -p src/Google.Protobuf.Test/Google.Protobuf.Test.csproj
+  dotnet run -c Release -f netcoreapp2.1 -p src/Google.Protobuf.Test/Google.Protobuf.Test.csproj
 }
 
 set -ex
@@ -79,26 +79,18 @@ cp ../../keys . -r
 # Test A.1:
 #   proto set 1: use old version
 #   proto set 2 which may import protos in set 1: use old version
-cp old_protoc protoc_1
-cp old_protoc protoc_2
-run_test
+run_test "./old_protoc" "./old_protoc"
 
 # Test A.2:
 #   proto set 1: use new version
 #   proto set 2 which may import protos in set 1: use old version
-cp ../../../src/protoc protoc_1
-cp old_protoc protoc_2
-run_test
+run_test "../../../src/protoc" "./old_protoc"
 
 # Test A.3:
 #   proto set 1: use old version
 #   proto set 2 which may import protos in set 1: use new version
-cp old_protoc protoc_1
-cp ../../../src/protoc protoc_2
-run_test
+run_test "./old_protoc" "../../../src/protoc"
 
-rm protoc_1
-rm protoc_2
 rm old_protoc
 rm keys -r
 rm src/Google.Protobuf -r

csharp/generate_protos.sh

@@ -61,3 +61,9 @@ $PROTOC -Iexamples -Isrc --csharp_out=csharp/src/AddressBook \
 
 $PROTOC -Iconformance -Isrc --csharp_out=csharp/src/Google.Protobuf.Conformance \
     conformance/conformance.proto
+
+# Benchmark protos
+$PROTOC -Ibenchmarks \
+  benchmarks/datasets/google_message1/proto3/*.proto \
+  benchmarks/benchmarks.proto \
+  --csharp_out=csharp/src/Google.Protobuf.Benchmarks

csharp/install_dotnet_sdk.ps1

+#!/usr/bin/env powershell
+# Install dotnet SDK based on the SDK version from global.json
+
+Set-StrictMode -Version 2
+$ErrorActionPreference = 'Stop'
+
+# avoid "Unknown error on a send" in Invoke-WebRequest
+[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
+
+$InstallScriptUrl = 'https://dot.net/v1/dotnet-install.ps1'
+$InstallScriptPath = Join-Path  "$env:TEMP" 'dotnet-install.ps1'
+$GlobalJsonPath = Join-Path $PSScriptRoot '..' | Join-Path -ChildPath 'global.json'
+
+# Resolve SDK version from global.json file
+$GlobalJson = Get-Content -Raw $GlobalJsonPath | ConvertFrom-Json
+$SDKVersion = $GlobalJson.sdk.version
+
+# Download install script
+Write-Host "Downloading install script: $InstallScriptUrl => $InstallScriptPath"
+Invoke-WebRequest -Uri $InstallScriptUrl -OutFile $InstallScriptPath
+&$InstallScriptPath -Version $SDKVersion

csharp/src/AddressBook/AddressBook.csproj

 <Project Sdk="Microsoft.NET.Sdk">
 
   <PropertyGroup>
-    <TargetFramework>netcoreapp1.0</TargetFramework>
+    <TargetFramework>netcoreapp2.1</TargetFramework>
     <OutputType>Exe</OutputType>
     <StartupObject>Google.Protobuf.Examples.AddressBook.Program</StartupObject>
     <IsPackable>False</IsPackable>

csharp/src/Google.Protobuf.Benchmarks/Benchmarks.cs

+// <auto-generated>
+//     Generated by the protocol buffer compiler.  DO NOT EDIT!
+//     source: benchmarks.proto
+// </auto-generated>
+#pragma warning disable 1591, 0612, 3021
+#region Designer generated code
+
+using pb = global::Google.Protobuf;
+using pbc = global::Google.Protobuf.Collections;
+using pbr = global::Google.Protobuf.Reflection;
+using scg = global::System.Collections.Generic;
+namespace Benchmarks {
+
+  /// <summary>Holder for reflection information generated from benchmarks.proto</summary>
+  public static partial class BenchmarksReflection {
+
+    #region Descriptor
+    /// <summary>File descriptor for benchmarks.proto</summary>
+    public static pbr::FileDescriptor Descriptor {
+      get { return descriptor; }
+    }
+    private static pbr::FileDescriptor descriptor;
+
+    static BenchmarksReflection() {
+      byte[] descriptorData = global::System.Convert.FromBase64String(
+          string.Concat(
+            "ChBiZW5jaG1hcmtzLnByb3RvEgpiZW5jaG1hcmtzIkcKEEJlbmNobWFya0Rh",
+            "dGFzZXQSDAoEbmFtZRgBIAEoCRIUCgxtZXNzYWdlX25hbWUYAiABKAkSDwoH",
+            "cGF5bG9hZBgDIAMoDEIgCh5jb20uZ29vZ2xlLnByb3RvYnVmLmJlbmNobWFy",
+            "a3NiBnByb3RvMw=="));
+      descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData,
+          new pbr::FileDescriptor[] { },
+          new pbr::GeneratedClrTypeInfo(null, new pbr::GeneratedClrTypeInfo[] {
+            new pbr::GeneratedClrTypeInfo(typeof(global::Benchmarks.BenchmarkDataset), global::Benchmarks.BenchmarkDataset.Parser, new[]{ "Name", "MessageName", "Payload" }, null, null, null)
+          }));
+    }
+    #endregion
+
+  }
+  #region Messages
+  public sealed partial class BenchmarkDataset : pb::IMessage<BenchmarkDataset> {
+    private static readonly pb::MessageParser<BenchmarkDataset> _parser = new pb::MessageParser<BenchmarkDataset>(() => new BenchmarkDataset());
+    private pb::UnknownFieldSet _unknownFields;
+    [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
+    public static pb::MessageParser<BenchmarkDataset> Parser { get { return _parser; } }
+
+    [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
+    public static pbr::MessageDescriptor Descriptor {
+      get { return global::Benchmarks.BenchmarksReflection.Descriptor.MessageTypes[0]; }
+    }
+
+    [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
+    pbr::MessageDescriptor pb::IMessage.Descriptor {
+      get { return Descriptor; }
+    }
+
+    [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
+    public BenchmarkDataset() {
+      OnConstruction();
+    }
+
+    partial void OnConstruction();
+
+    [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
+    public BenchmarkDataset(BenchmarkDataset other) : this() {
+      name_ = other.name_;
+      messageName_ = other.messageName_;
+      payload_ = other.payload_.Clone();
+      _unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields);
+    }
+
+    [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
+    public BenchmarkDataset Clone() {
+      return new BenchmarkDataset(this);
+    }
+
+    /// <summary>Field number for the "name" field.</summary>
+    public const int NameFieldNumber = 1;
+    private string name_ = "";
+    /// <summary>
+    /// Name of the benchmark dataset.  This should be unique across all datasets.
+    /// Should only contain word characters: [a-zA-Z0-9_]
+    /// </summary>
+    [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
+    public string Name {
+      get { return name_; }
+      set {
+        name_ = pb::ProtoPreconditions.CheckNotNull(value, "value");
+      }
+    }
+
+    /// <summary>Field number for the "message_name" field.</summary>
+    public const int MessageNameFieldNumber = 2;
+    private string messageName_ = "";
+    /// <summary>
+    /// Fully-qualified name of the protobuf message for this dataset.
+    /// It will be one of the messages defined benchmark_messages_proto2.proto
+    /// or benchmark_messages_proto3.proto.
+    ///
+    /// Implementations that do not support reflection can implement this with
+    /// an explicit "if/else" chain that lists every known message defined
+    /// in those files.
+    /// </summary>
+    [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
+    public string MessageName {
+      get { return messageName_; }
+      set {
+        messageName_ = pb::ProtoPreconditions.CheckNotNull(value, "value");
+      }
+    }
+
+    /// <summary>Field number for the "payload" field.</summary>
+    public const int PayloadFieldNumber = 3;
+    private static readonly pb::FieldCodec<pb::ByteString> _repeated_payload_codec
+        = pb::FieldCodec.ForBytes(26);
+    private readonly pbc::RepeatedField<pb::ByteString> payload_ = new pbc::RepeatedField<pb::ByteString>();
+    /// <summary>
+    /// The payload(s) for this dataset.  They should be parsed or serialized
+    /// in sequence, in a loop, ie.
+    ///
+    ///  while (!benchmarkDone) {  // Benchmark runner decides when to exit.
+    ///    for (i = 0; i &lt; benchmark.payload.length; i++) {
+    ///      parse(benchmark.payload[i])
+    ///    }
+    ///  }
+    ///
+    /// This is intended to let datasets include a variety of data to provide
+    /// potentially more realistic results than just parsing the same message
+    /// over and over.  A single message parsed repeatedly could yield unusually
+    /// good branch prediction performance.
+    /// </summary>
+    [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
+    public pbc::RepeatedField<pb::ByteString> Payload {
+      get { return payload_; }
+    }
+
+    [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
+    public override bool Equals(object other) {
+      return Equals(other as BenchmarkDataset);
+    }
+
+    [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
+    public bool Equals(BenchmarkDataset other) {
+      if (ReferenceEquals(other, null)) {
+        return false;
+      }
+      if (ReferenceEquals(other, this)) {
+        return true;
+      }
+      if (Name != other.Name) return false;
+      if (MessageName != other.MessageName) return false;
+      if(!payload_.Equals(other.payload_)) return false;
+      return Equals(_unknownFields, other._unknownFields);
+    }
+
+    [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
+    public override int GetHashCode() {
+      int hash = 1;
+      if (Name.Length != 0) hash ^= Name.GetHashCode();
+      if (MessageName.Length != 0) hash ^= MessageName.GetHashCode();
+      hash ^= payload_.GetHashCode();
+      if (_unknownFields != null) {
+        hash ^= _unknownFields.GetHashCode();
+      }
+      return hash;
+    }
+
+    [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
+    public override string ToString() {
+      return pb::JsonFormatter.ToDiagnosticString(this);
+    }
+
+    [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
+    public void WriteTo(pb::CodedOutputStream output) {
+      if (Name.Length != 0) {
+        output.WriteRawTag(10);
+        output.WriteString(Name);
+      }
+      if (MessageName.Length != 0) {
+        output.WriteRawTag(18);
+        output.WriteString(MessageName);
+      }
+      payload_.WriteTo(output, _repeated_payload_codec);
+      if (_unknownFields != null) {
+        _unknownFields.WriteTo(output);
+      }
+    }
+
+    [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
+    public int CalculateSize() {
+      int size = 0;
+      if (Name.Length != 0) {
+        size += 1 + pb::CodedOutputStream.ComputeStringSize(Name);
+      }
+      if (MessageName.Length != 0) {
+        size += 1 + pb::CodedOutputStream.ComputeStringSize(MessageName);
+      }
+      size += payload_.CalculateSize(_repeated_payload_codec);
+      if (_unknownFields != null) {
+        size += _unknownFields.CalculateSize();
+      }
+      return size;
+    }
+
+    [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
+    public void MergeFrom(BenchmarkDataset other) {
+      if (other == null) {
+        return;
+      }
+      if (other.Name.Length != 0) {
+        Name = other.Name;
+      }
+      if (other.MessageName.Length != 0) {
+        MessageName = other.MessageName;
+      }
+      payload_.Add(other.payload_);
+      _unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields);
+    }
+
+    [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
+    public void MergeFrom(pb::CodedInputStream input) {
+      uint tag;
+      while ((tag = input.ReadTag()) != 0) {
+        switch(tag) {
+          default:
+            _unknownFields = pb::UnknownFieldSet.MergeFieldFrom(_unknownFields, input);
+            break;
+          case 10: {
+            Name = input.ReadString();
+            break;
+          }
+          case 18: {
+            MessageName = input.ReadString();
+            break;
+          }
+          case 26: {
+            payload_.AddEntriesFrom(input, _repeated_payload_codec);
+            break;
+          }
+        }
+      }
+    }
+
+  }
+
+  #endregion
+
+}
+
+#endregion Designer generated code

csharp/src/Google.Protobuf.Benchmarks/Google.Protobuf.Benchmarks.csproj

+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <OutputType>Exe</OutputType>
+    <TargetFramework>netcoreapp2.1</TargetFramework>
+    <IsPackable>False</IsPackable>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <PackageReference Include="BenchmarkDotNet" Version="0.11.4" />
+    <ProjectReference Include="..\Google.Protobuf\Google.Protobuf.csproj" />
+  </ItemGroup>
+
+  <ItemGroup>
+    <EmbeddedResource Include="..\..\..\benchmarks\datasets\google_message1\proto3\dataset.google_message1_proto3.pb" />
+  </ItemGroup>
+
+</Project>

csharp/src/Google.Protobuf.Benchmarks/Program.cs

+#region Copyright notice and license
+// Protocol Buffers - Google's data interchange format
+// Copyright 2019 Google Inc.  All rights reserved.
+// https://github.com/protocolbuffers/protobuf
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions are
+// met:
+//
+//     * Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+//     * Redistributions in binary form must reproduce the above
+// copyright notice, this list of conditions and the following disclaimer
+// in the documentation and/or other materials provided with the
+// distribution.
+//     * Neither the name of Google Inc. nor the names of its
+// contributors may be used to endorse or promote products derived from
+// this software without specific prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#endregion
+
+using BenchmarkDotNet.Running;
+
+namespace Google.Protobuf.Benchmarks
+{
+    /// <summary>
+    /// Entry point, that currently runs the sole benchmark we have.
+    /// Eventually we might want to be able to specify a particular dataset
+    /// from the command line.
+    /// </summary>
+    class Program
+    {
+        static void Main() => BenchmarkRunner.Run<SerializationBenchmark>();
+    }
+}

csharp/src/Google.Protobuf.Benchmarks/SerializationBenchmark.cs

+#region Copyright notice and license
+// Protocol Buffers - Google's data interchange format
+// Copyright 2019 Google Inc.  All rights reserved.
+// https://github.com/protocolbuffers/protobuf
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions are
+// met:
+//
+//     * Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+//     * Redistributions in binary form must reproduce the above
+// copyright notice, this list of conditions and the following disclaimer
+// in the documentation and/or other materials provided with the
+// distribution.
+//     * Neither the name of Google Inc. nor the names of its
+// contributors may be used to endorse or promote products derived from
+// this software without specific prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#endregion
+
+using BenchmarkDotNet.Attributes;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+
+namespace Google.Protobuf.Benchmarks
+{
+    /// <summary>
+    /// Benchmark for serializing (to a MemoryStream) and deserializing (from a ByteString).
+    /// Over time we may wish to test the various different approaches to serialization and deserialization separately.
+    /// </summary>
+    [MemoryDiagnoser]
+    public class SerializationBenchmark
+    {
+        /// <summary>
+        /// All the configurations to be tested. Add more datasets to the array as they're available.
+        /// (When C# supports proto2, this will increase significantly.)
+        /// </summary>
+        public static SerializationConfig[] Configurations => new[]
+        {
+            new SerializationConfig("dataset.google_message1_proto3.pb")
+        };
+
+        [ParamsSource(nameof(Configurations))]
+        public SerializationConfig Configuration { get; set; }
+
+        private MessageParser parser;
+        /// <summary>
+        /// Each data set can contain multiple messages in a single file.
+        /// Each "write" operation should write each message in turn, and each "parse"
+        /// operation should parse each message in turn.
+        /// </summary>
+        private List<SubTest> subTests;
+
+        [GlobalSetup]
+        public void GlobalSetup()
+        {
+            parser = Configuration.Parser;
+            subTests = Configuration.Payloads.Select(p => new SubTest(p, parser.ParseFrom(p))).ToList();
+        }
+
+        [Benchmark]
+        public void WriteToStream() => subTests.ForEach(item => item.WriteToStream());
+
+        [Benchmark]
+        public void ToByteArray() => subTests.ForEach(item => item.ToByteArray());
+
+        [Benchmark]
+        public void ParseFromByteString() => subTests.ForEach(item => item.ParseFromByteString(parser));
+
+        [Benchmark]
+        public void ParseFromStream() => subTests.ForEach(item => item.ParseFromStream(parser));
+
+        private class SubTest
+        {
+            private readonly Stream destinationStream;
+            private readonly Stream sourceStream;
+            private readonly ByteString data;
+            private readonly IMessage message;
+
+            public SubTest(ByteString data, IMessage message)
+            {
+                destinationStream = new MemoryStream(data.Length);
+                sourceStream = new MemoryStream(data.ToByteArray());
+                this.data = data;
+                this.message = message;
+            }
+
+            public void Reset() => destinationStream.Position = 0;
+
+            public void WriteToStream()
+            {
+                destinationStream.Position = 0;
+                message.WriteTo(destinationStream);
+            }
+
+            public void ToByteArray() => message.ToByteArray();
+
+            public void ParseFromByteString(MessageParser parser) => parser.ParseFrom(data);
+
+            public void ParseFromStream(MessageParser parser)
+            {
+                sourceStream.Position = 0;
+                parser.ParseFrom(sourceStream);
+            }
+        }
+    }
+}

csharp/src/Google.Protobuf.Benchmarks/SerializationConfig.cs

+#region Copyright notice and license
+// Protocol Buffers - Google's data interchange format
+// Copyright 2019 Google Inc.  All rights reserved.
+// https://github.com/protocolbuffers/protobuf
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions are
+// met:
+//
+//     * Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+//     * Redistributions in binary form must reproduce the above
+// copyright notice, this list of conditions and the following disclaimer
+// in the documentation and/or other materials provided with the
+// distribution.
+//     * Neither the name of Google Inc. nor the names of its
+// contributors may be used to endorse or promote products derived from
+// this software without specific prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#endregion
+
+using Benchmarks;
+using Google.Protobuf.Reflection;
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Reflection;
+
+namespace Google.Protobuf.Benchmarks
+{
+    /// <summary>
+    /// The configuration for a single serialization test, loaded from a dataset.
+    /// </summary>
+    public class SerializationConfig
+    {
+        private static readonly Dictionary<string, MessageParser> parsersByMessageName = 
+            typeof(SerializationBenchmark).Assembly.GetTypes()
+                .Where(t => typeof(IMessage).IsAssignableFrom(t))
+                .ToDictionary(
+                    t => ((MessageDescriptor) t.GetProperty("Descriptor", BindingFlags.Static | BindingFlags.Public).GetValue(null)).FullName,
+                    t => ((MessageParser) t.GetProperty("Parser", BindingFlags.Static | BindingFlags.Public).GetValue(null)));
+
+        public MessageParser Parser { get; }
+        public IEnumerable<ByteString> Payloads { get; }
+        public string Name { get; }
+
+        public SerializationConfig(string resource)
+        {
+            var data = LoadData(resource);
+            var dataset = BenchmarkDataset.Parser.ParseFrom(data);
+
+            if (!parsersByMessageName.TryGetValue(dataset.MessageName, out var parser))
+            {
+                throw new ArgumentException($"No parser for message {dataset.MessageName} in this assembly");
+            }
+            Parser = parser;
+            Payloads = dataset.Payload;
+            Name = dataset.Name;
+        }
+
+        private static byte[] LoadData(string resource)
+        {
+            using (var stream = typeof(SerializationBenchmark).Assembly.GetManifestResourceStream($"Google.Protobuf.Benchmarks.{resource}"))
+            {
+                if (stream == null)
+                {
+                    throw new ArgumentException($"Unable to load embedded resource {resource}");
+                }
+                var copy = new MemoryStream();
+                stream.CopyTo(copy);
+                return copy.ToArray();
+            }
+        }
+
+        public override string ToString() => Name;
+    }
+}

csharp/src/Google.Protobuf.Conformance/Google.Protobuf.Conformance.csproj

 <Project Sdk="Microsoft.NET.Sdk">
 
   <PropertyGroup>
-    <TargetFramework>netcoreapp1.0</TargetFramework>
+    <TargetFramework>netcoreapp2.1</TargetFramework>
     <OutputType>Exe</OutputType>
     <IsPackable>False</IsPackable>
   </PropertyGroup>

csharp/src/Google.Protobuf.JsonDump/Google.Protobuf.JsonDump.csproj

 <Project Sdk="Microsoft.NET.Sdk">
 
   <PropertyGroup>
-    <TargetFramework>netcoreapp1.0</TargetFramework>
+    <TargetFramework>netcoreapp2.1</TargetFramework>
     <OutputType>Exe</OutputType>
     <IsPackable>False</IsPackable>
   </PropertyGroup>

csharp/src/Google.Protobuf.Test/Collections/RepeatedFieldTest.cs

@@ -750,7 +750,8 @@ namespace Google.Protobuf.Collections
             var list2 = new RepeatedField<double> { SampleNaNs.Regular, SampleNaNs.PayloadFlipped };
             var list3 = new RepeatedField<double> { SampleNaNs.Regular, SampleNaNs.SignallingFlipped };
 
-            EqualityTester.AssertInequality(list1, list2);
+            // All SampleNaNs have the same hashcode under certain targets (e.g. netcoreapp2.1)
+            EqualityTester.AssertInequality(list1, list2, checkHashcode: false);
             EqualityTester.AssertEquality(list1, list3);
             Assert.True(list1.Contains(SampleNaNs.SignallingFlipped));
             Assert.False(list2.Contains(SampleNaNs.SignallingFlipped));

csharp/src/Google.Protobuf.Test/EqualityTester.cs

@@ -49,13 +49,14 @@ namespace Google.Protobuf
             Assert.AreEqual(first.GetHashCode(), second.GetHashCode());
         }
 
-        public static void AssertInequality<T>(T first, T second) where T : IEquatable<T>
+        public static void AssertInequality<T>(T first, T second, bool checkHashcode = true) where T : IEquatable<T>
         {
             Assert.IsFalse(first.Equals(second));
             Assert.IsFalse(first.Equals((object) second));
             // While this isn't a requirement, the chances of this test failing due to
             // coincidence rather than a bug are very small.
-            if (first != null && second != null)
+            // For such rare cases, an argument can be used to disable the check.
+            if (checkHashcode && first != null && second != null)
             {
                 Assert.AreNotEqual(first.GetHashCode(), second.GetHashCode());
             }

csharp/src/Google.Protobuf.Test/Google.Protobuf.Test.csproj

 <Project Sdk="Microsoft.NET.Sdk">
 
   <PropertyGroup>
-    <TargetFrameworks>net451;netcoreapp1.0</TargetFrameworks>
+    <TargetFrameworks>net451;netcoreapp2.1</TargetFrameworks>
     <AssemblyOriginatorKeyFile>../../keys/Google.Protobuf.snk</AssemblyOriginatorKeyFile>
     <SignAssembly>true</SignAssembly>
     <PublicSign Condition=" '$(OS)' != 'Windows_NT' ">true</PublicSign>
@@ -24,7 +24,7 @@
     - Visual Studio.
     -->
   <PropertyGroup Condition="'$(OS)' != 'Windows_NT'">
-    <TargetFrameworks>netcoreapp1.0</TargetFrameworks>
+    <TargetFrameworks>netcoreapp2.1</TargetFrameworks>
   </PropertyGroup>
 
   <ItemGroup>

csharp/src/Google.Protobuf.sln

@@ -2,15 +2,17 @@ Microsoft Visual Studio Solution File, Format Version 12.00
 # Visual Studio 15
 VisualStudioVersion = 15.0.26114.2
 MinimumVisualStudioVersion = 10.0.40219.1
-Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "AddressBook", "AddressBook\AddressBook.csproj", "{AFB63919-1E05-43B4-802A-8FB8C9B2F463}"
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "AddressBook", "AddressBook\AddressBook.csproj", "{AFB63919-1E05-43B4-802A-8FB8C9B2F463}"
 EndProject
-Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Google.Protobuf", "Google.Protobuf\Google.Protobuf.csproj", "{9B576380-726D-4142-8238-60A43AB0E35A}"
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Google.Protobuf", "Google.Protobuf\Google.Protobuf.csproj", "{9B576380-726D-4142-8238-60A43AB0E35A}"
 EndProject
-Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Google.Protobuf.Test", "Google.Protobuf.Test\Google.Protobuf.Test.csproj", "{580EB013-D3C7-4578-B845-015F4A3B0591}"
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Google.Protobuf.Test", "Google.Protobuf.Test\Google.Protobuf.Test.csproj", "{580EB013-D3C7-4578-B845-015F4A3B0591}"
 EndProject
-Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Google.Protobuf.Conformance", "Google.Protobuf.Conformance\Google.Protobuf.Conformance.csproj", "{DDDC055B-E185-4181-BAB0-072F0F984569}"
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Google.Protobuf.Conformance", "Google.Protobuf.Conformance\Google.Protobuf.Conformance.csproj", "{DDDC055B-E185-4181-BAB0-072F0F984569}"
 EndProject
-Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Google.Protobuf.JsonDump", "Google.Protobuf.JsonDump\Google.Protobuf.JsonDump.csproj", "{9695E08F-9829-497D-B95C-B38F28D48690}"
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Google.Protobuf.JsonDump", "Google.Protobuf.JsonDump\Google.Protobuf.JsonDump.csproj", "{9695E08F-9829-497D-B95C-B38F28D48690}"
+EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Google.Protobuf.Benchmarks", "Google.Protobuf.Benchmarks\Google.Protobuf.Benchmarks.csproj", "{D25E4804-4DEA-45AB-9F8C-BA4DBD8E5A07}"
 EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
@@ -38,8 +40,15 @@ Global
 		{9695E08F-9829-497D-B95C-B38F28D48690}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{9695E08F-9829-497D-B95C-B38F28D48690}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{9695E08F-9829-497D-B95C-B38F28D48690}.Release|Any CPU.Build.0 = Release|Any CPU
+		{D25E4804-4DEA-45AB-9F8C-BA4DBD8E5A07}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{D25E4804-4DEA-45AB-9F8C-BA4DBD8E5A07}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{D25E4804-4DEA-45AB-9F8C-BA4DBD8E5A07}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{D25E4804-4DEA-45AB-9F8C-BA4DBD8E5A07}.Release|Any CPU.Build.0 = Release|Any CPU
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE
 	EndGlobalSection
+	GlobalSection(ExtensibilityGlobals) = postSolution
+		SolutionGuid = {7B06C87B-83E1-4F5F-A0DD-6E9AFAC03DAC}
+	EndGlobalSection
 EndGlobal

csharp/src/Google.Protobuf/ByteString.cs

@@ -67,15 +67,6 @@ namespace Google.Protobuf
             {
                 return new ByteString(bytes);
             }
-
-            /// <summary>
-            /// Provides direct, unrestricted access to the bytes contained in this instance.
-            /// You must not modify or resize the byte array returned by this method.
-            /// </summary>
-            internal static byte[] GetBuffer(ByteString bytes)
-            {
-                return bytes.bytes;
-            }
         }
 
         /// <summary>
@@ -119,6 +110,14 @@ namespace Google.Protobuf
             get { return Length == 0; }
         }
 
+#if NETSTANDARD2_0
+        /// <summary>
+        /// Provides read-only access to the data of this <see cref="ByteString"/>.
+        /// No data is copied so this is the most efficient way of accessing.
+        /// </summary>
+        public ReadOnlySpan<byte> Span => new ReadOnlySpan<byte>(bytes);
+#endif
+
         /// <summary>
         /// Converts this <see cref="ByteString"/> into a byte array.
         /// </summary>
@@ -161,7 +160,7 @@ namespace Google.Protobuf
             int capacity = stream.CanSeek ? checked((int) (stream.Length - stream.Position)) : 0;
             var memoryStream = new MemoryStream(capacity);
             stream.CopyTo(memoryStream);
-#if NETSTANDARD1_0
+#if NETSTANDARD1_0 || NETSTANDARD2_0
             byte[] bytes = memoryStream.ToArray();
 #else
             // Avoid an extra copy if we can.
@@ -187,7 +186,7 @@ namespace Google.Protobuf
             // We have to specify the buffer size here, as there's no overload accepting the cancellation token
             // alone. But it's documented to use 81920 by default if not specified.
             await stream.CopyToAsync(memoryStream, 81920, cancellationToken);
-#if NETSTANDARD1_0
+#if NETSTANDARD1_0 || NETSTANDARD2_0
             byte[] bytes = memoryStream.ToArray();
 #else
             // Avoid an extra copy if we can.
@@ -219,6 +218,18 @@ namespace Google.Protobuf
             return new ByteString(portion);
         }
 
+#if NETSTANDARD2_0
+        /// <summary>
+        /// Constructs a <see cref="ByteString" /> from a read only span. The contents
+        /// are copied, so further modifications to the span will not
+        /// be reflected in the returned <see cref="ByteString" />.
+        /// </summary>
+        public static ByteString CopyFrom(ReadOnlySpan<byte> bytes)
+        {
+            return new ByteString(bytes.ToArray());
+        }
+#endif
+
         /// <summary>
         /// Creates a new <see cref="ByteString" /> by encoding the specified text with
         /// the given encoding.

csharp/src/Google.Protobuf/Google.Protobuf.csproj

@@ -7,7 +7,7 @@
     <VersionPrefix>3.7.0</VersionPrefix>
     <LangVersion>6</LangVersion>
     <Authors>Google Inc.</Authors>
-    <TargetFrameworks>netstandard1.0;net45</TargetFrameworks>
+    <TargetFrameworks>netstandard1.0;netstandard2.0;net45</TargetFrameworks>
     <GenerateDocumentationFile>true</GenerateDocumentationFile>
     <AssemblyOriginatorKeyFile>../../keys/Google.Protobuf.snk</AssemblyOriginatorKeyFile>
     <SignAssembly>true</SignAssembly>
@@ -18,6 +18,8 @@
     <PackageLicenseUrl>https://github.com/protocolbuffers/protobuf/blob/master/LICENSE</PackageLicenseUrl>
     <RepositoryType>git</RepositoryType>
     <RepositoryUrl>https://github.com/protocolbuffers/protobuf.git</RepositoryUrl>
+    <!-- Include PDB in the built .nupkg -->
+    <AllowedOutputExtensionsInPackageBuildOutputFolder>$(AllowedOutputExtensionsInPackageBuildOutputFolder);.pdb</AllowedOutputExtensionsInPackageBuildOutputFolder>
   </PropertyGroup>
 
   <!--
@@ -26,11 +28,15 @@
     - Visual Studio.
     -->
   <PropertyGroup Condition="'$(OS)' != 'Windows_NT'">
-    <TargetFrameworks>netstandard1.0</TargetFrameworks>
+    <TargetFrameworks>netstandard1.0;netstandard2.0</TargetFrameworks>
   </PropertyGroup>
 
+  <ItemGroup Condition=" '$(TargetFramework)' == 'netstandard2.0' ">
+    <PackageReference Include="System.Memory" Version="4.5.2" />
+  </ItemGroup>
+
   <ItemGroup>
-    <PackageReference Include="SourceLink.Create.CommandLine" PrivateAssets="All" Version="2.7.6"/>
+    <PackageReference Include="Microsoft.SourceLink.GitHub" Version="1.0.0-beta2-18618-05" PrivateAssets="All" />
   </ItemGroup>
 
 </Project>

csharp/global.json → global.json

 {
   "sdk": {
-    "version": "2.1.3"
+    "version": "2.1.504"
   }
 }

java/core/src/test/java/com/google/protobuf/ExtensionRegistryFactoryTest.java

@@ -30,6 +30,8 @@
 
 package com.google.protobuf;
 
+import protobuf_unittest.NestedExtension;
+import protobuf_unittest.NestedExtensionLite;
 import protobuf_unittest.NonNestedExtension;
 import protobuf_unittest.NonNestedExtensionLite;
 import java.lang.reflect.Method;
@@ -69,6 +71,8 @@ public class ExtensionRegistryFactoryTest extends TestCase {
     void testAdd();
 
     void testAdd_immutable();
+
+    void testExtensionRenamesKeywords();
   }
 
   /** Test implementations for the non-Lite usage of ExtensionRegistryFactory. */
@@ -156,6 +160,23 @@ public class ExtensionRegistryFactoryTest extends TestCase {
       } catch (IllegalArgumentException expected) {
       }
     }
+
+    @Override
+    public void testExtensionRenamesKeywords() {
+      assertTrue(NonNestedExtension.if_ instanceof GeneratedMessage.GeneratedExtension);
+      assertTrue(NestedExtension.MyNestedExtension.default_ instanceof GeneratedMessage.GeneratedExtension);
+
+      NonNestedExtension.MessageToBeExtended msg =
+              NonNestedExtension.MessageToBeExtended.newBuilder()
+                      .setExtension(NonNestedExtension.if_, "!fi")
+                      .build();
+      assertEquals("!fi", msg.getExtension(NonNestedExtension.if_));
+
+      msg = NonNestedExtension.MessageToBeExtended.newBuilder()
+              .setExtension(NestedExtension.MyNestedExtension.default_, 8)
+              .build();
+      assertEquals(8, msg.getExtension(NestedExtension.MyNestedExtension.default_).intValue());
+    }
   }
 
   /** Test implementations for the Lite usage of ExtensionRegistryFactory. */
@@ -202,6 +223,23 @@ public class ExtensionRegistryFactoryTest extends TestCase {
       } catch (UnsupportedOperationException expected) {
       }
     }
+
+    @Override
+    public void testExtensionRenamesKeywords() {
+      assertTrue(NonNestedExtensionLite.package_ instanceof GeneratedMessageLite.GeneratedExtension);
+      assertTrue(NestedExtensionLite.MyNestedExtensionLite.private_ instanceof GeneratedMessageLite.GeneratedExtension);
+
+      NonNestedExtensionLite.MessageLiteToBeExtended msg =
+              NonNestedExtensionLite.MessageLiteToBeExtended.newBuilder()
+              .setExtension(NonNestedExtensionLite.package_, true)
+              .build();
+      assertTrue(msg.getExtension(NonNestedExtensionLite.package_));
+
+      msg = NonNestedExtensionLite.MessageLiteToBeExtended.newBuilder()
+              .setExtension(NestedExtensionLite.MyNestedExtensionLite.private_, 2.4)
+              .build();
+      assertEquals(2.4, msg.getExtension(NestedExtensionLite.MyNestedExtensionLite.private_), 0.001);
+    }
   }
 
   /** Defines a suite of tests which the JUnit3 runner retrieves by reflection. */

java/core/src/test/proto/com/google/protobuf/nested_extension.proto

@@ -43,5 +43,6 @@ package protobuf_unittest;
 message MyNestedExtension {
   extend MessageToBeExtended {
     optional MessageToBeExtended recursiveExtension = 2;
+    optional int32 default = 2002;
   }
 }

java/core/src/test/proto/com/google/protobuf/nested_extension_lite.proto

@@ -45,5 +45,6 @@ import "com/google/protobuf/non_nested_extension_lite.proto";
 message MyNestedExtensionLite {
   extend MessageLiteToBeExtended {
     optional MessageLiteToBeExtended recursiveExtensionLite = 3;
+    optional double private = 2004;
   }
 }

java/core/src/test/proto/com/google/protobuf/non_nested_extension.proto

@@ -45,4 +45,5 @@ message MyNonNestedExtension {}
 
 extend MessageToBeExtended {
   optional MyNonNestedExtension nonNestedExtension = 1;
+  optional string if = 2000;
 }

java/core/src/test/proto/com/google/protobuf/non_nested_extension_lite.proto

@@ -46,4 +46,5 @@ message MyNonNestedExtensionLite {}
 
 extend MessageLiteToBeExtended {
   optional MyNonNestedExtensionLite nonNestedExtensionLite = 1;
+  optional bool package = 2006;
 }

kokoro/linux/csharp/build.sh

@@ -10,7 +10,8 @@
 # Change to repo root
 cd $(dirname $0)/../../..
 
-export DOCKERFILE_DIR=kokoro/linux/64-bit
+export DOCKERHUB_ORGANIZATION=protobuftesting
+export DOCKERFILE_DIR=kokoro/linux/dockerfile/test/csharp
 export DOCKER_RUN_SCRIPT=kokoro/linux/pull_request_in_docker.sh
 export OUTPUT_DIR=testoutput
 export TEST_SET="csharp"

kokoro/linux/dockerfile/test/csharp/Dockerfile

+FROM debian:stretch
+
+# Install dependencies.  We start with the basic ones require to build protoc
+# and the C++ build
+RUN apt-get update && apt-get install -y \
+  autoconf \
+  autotools-dev \
+  build-essential \
+  bzip2 \
+  ccache \
+  curl \
+  gcc \
+  git \
+  libc6 \
+  libc6-dbg \
+  libc6-dev \
+  libgtest-dev \
+  libtool \
+  make \
+  parallel \
+  time \
+  wget \
+  && apt-get clean
+
+# dotnet SDK prerequisites
+RUN apt-get update && apt-get install -y libunwind8 libicu57 && apt-get clean
+
+# Install dotnet SDK via install script
+RUN wget -q https://dot.net/v1/dotnet-install.sh && \
+    chmod u+x dotnet-install.sh && \
+    ./dotnet-install.sh --version 2.1.504 && \
+    ln -s /root/.dotnet/dotnet /usr/local/bin
+
+RUN wget -q www.nuget.org/NuGet.exe -O /usr/local/bin/nuget.exe
+
+ENV DOTNET_SKIP_FIRST_TIME_EXPERIENCE true

kokoro/release/csharp/windows/build_nuget.bat

@@ -2,4 +2,13 @@
 cd /d %~dp0\..\..\..\..
 
 cd csharp
+
+@rem Install dotnet SDK
+powershell -File install_dotnet_sdk.ps1
+set PATH=%LOCALAPPDATA%\Microsoft\dotnet;%PATH%
+
+@rem Disable some unwanted dotnet options
+set DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true
+set DOTNET_CLI_TELEMETRY_OPTOUT=true
+
 call build_packages.bat

kokoro/windows/csharp/build.bat

+@rem enter repo root
+cd /d %~dp0\..\..\..
+
+cd csharp
+
+@rem Install dotnet SDK
+powershell -File install_dotnet_sdk.ps1
+set PATH=%LOCALAPPDATA%\Microsoft\dotnet;%PATH%
+
+@rem Disable some unwanted dotnet options
+set DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true
+set DOTNET_CLI_TELEMETRY_OPTOUT=true
+
+call buildall.bat

kokoro/windows/csharp/continuous.cfg

+# Config file for running tests in Kokoro
+
+# Location of the build script in repository
+build_file: "protobuf/kokoro/windows/csharp/build.bat"
+timeout_mins: 1440

kokoro/windows/csharp/presubmit.cfg

+# Config file for running tests in Kokoro
+
+# Location of the build script in repository
+build_file: "protobuf/kokoro/windows/csharp/build.bat"
+timeout_mins: 1440

objectivec/DevTools/full_mac_build.sh

@@ -163,8 +163,7 @@ if [[ "${DO_AUTOGEN}" == "yes" ]] ; then
   header "Running autogen & configure"
   ./autogen.sh
   ./configure \
-    CPPFLAGS="-mmacosx-version-min=10.9 -Wunused-const-variable -Wunused-function" \
-    CXXFLAGS="-Wnon-virtual-dtor -Woverloaded-virtual"
+    CPPFLAGS="-mmacosx-version-min=10.9 -Wunused-const-variable -Wunused-function"
 fi
 
 if [[ "${DO_CLEAN}" == "yes" ]] ; then

php/ext/google/protobuf/def.c

@@ -853,9 +853,10 @@ static zend_class_entry *register_class(const upb_filedef *file,
 
   fill_namespace(package, php_namespace, &namesink);
   fill_classname(fullname, package, prefix, &namesink, use_nested_submsg);
+  stringsink_string(&namesink, NULL, "\0", 1, NULL);
 
   PHP_PROTO_CE_DECLARE pce;
-  if (php_proto_zend_lookup_class(namesink.ptr, namesink.len, &pce) ==
+  if (php_proto_zend_lookup_class(namesink.ptr, namesink.len - 1, &pce) ==
       FAILURE) {
     zend_error(
         E_ERROR,

php/tests/compile_extension.sh

+#!/bin/bash
+
+EXTENSION_PATH=$1
+
+pushd $EXTENSION_PATH
+make clean || true
+set -e
+# Add following in configure for debug: --enable-debug CFLAGS='-g -O0'
+phpize && ./configure CFLAGS='-g -O0' && make
+popd

php/tests/php_implementation_test.php

@@ -14,8 +14,19 @@ use Google\Protobuf\Internal\GPBType;
 use Google\Protobuf\Internal\GPBWire;
 use Google\Protobuf\Internal\CodedOutputStream;
 
+/**
+ * Please note, this test is only intended to be run without the protobuf C
+ * extension.
+ */
 class ImplementationTest extends TestBase
 {
+    public function setUp()
+    {
+        if (extension_loaded('protobuf')) {
+            $this->markTestSkipped();
+        }
+    }
+
     public function testReadInt32()
     {
         $value = null;

php/tests/test.sh

@@ -7,12 +7,7 @@ export C_INCLUDE_PATH=/usr/local/php-$VERSION/include/php/main:/usr/local/php-$V
 export CPLUS_INCLUDE_PATH=/usr/local/php-$VERSION/include/php/main:/usr/local/php-$VERSION/include/php:$CPLUS_INCLUDE_PATH
 
 # Compile c extension
-pushd ../ext/google/protobuf/
-make clean || true
-set -e
-# Add following in configure for debug: --enable-debug CFLAGS='-g -O0'
-phpize && ./configure CFLAGS='-g -O0' && make
-popd
+/bin/bash ./compile_extension.sh ../ext/google/protobuf
 
 tests=( array_test.php encode_decode_test.php generated_class_test.php map_field_test.php well_known_test.php descriptors_test.php wrapper_type_setters_test.php)
 

src/google/protobuf/compiler/java/java_extension.cc

@@ -68,7 +68,7 @@ void ExtensionGenerator::InitTemplateVars(
     std::map<std::string, std::string>* vars_pointer) {
   std::map<std::string, std::string>& vars = *vars_pointer;
   vars["scope"] = scope;
-  vars["name"] = UnderscoresToCamelCase(descriptor);
+  vars["name"] = UnderscoresToCamelCaseCheckReserved(descriptor);
   vars["containing_type"] =
       name_resolver->GetClassName(descriptor->containing_type(), immutable);
   vars["number"] = StrCat(descriptor->number());
@@ -153,7 +153,7 @@ int ImmutableExtensionGenerator::GenerateNonNestedInitializationCode(
     // Only applies to non-nested extensions.
     printer->Print(
         "$name$.internalInit(descriptor.getExtensions().get($index$));\n",
-        "name", UnderscoresToCamelCase(descriptor_), "index",
+        "name", UnderscoresToCamelCaseCheckReserved(descriptor_), "index",
         StrCat(descriptor_->index()));
     bytecode_estimate += 21;
   }
@@ -165,7 +165,7 @@ int ImmutableExtensionGenerator::GenerateRegistrationCode(
   printer->Print(
     "registry.add($scope$.$name$);\n",
     "scope", scope_,
-    "name", UnderscoresToCamelCase(descriptor_));
+    "name", UnderscoresToCamelCaseCheckReserved(descriptor_));
   return 7;
 }
 

src/google/protobuf/compiler/java/java_extension_lite.cc

@@ -109,7 +109,7 @@ int ImmutableExtensionLiteGenerator::GenerateRegistrationCode(
   printer->Print(
     "registry.add($scope$.$name$);\n",
     "scope", scope_,
-    "name", UnderscoresToCamelCase(descriptor_));
+    "name", UnderscoresToCamelCaseCheckReserved(descriptor_));
   return 7;
 }
 

src/google/protobuf/compiler/java/java_helpers.cc

@@ -76,6 +76,16 @@ const char* kForbiddenWordList[] = {
   "class",
 };
 
+const std::unordered_set<string> kReservedNames = {
+  "abstract", "assert", "boolean", "break", "byte", "case", "catch", "char",
+  "class", "const", "continue", "default", "do", "double", "else", "enum",
+  "extends", "final", "finally", "float", "for", "goto", "if", "implements",
+  "import", "instanceof", "int", "interface", "long", "native", "new", "package",
+  "private", "protected", "public", "return", "short", "static", "strictfp", "super",
+  "switch", "synchronized", "this", "throw", "throws", "transient", "try", "void", 
+  "volatile", "while",
+};
+
 const int kDefaultLookUpStartFieldNumber = 40;
 
 bool IsForbidden(const std::string& field_name) {
@@ -196,6 +206,14 @@ std::string UnderscoresToCamelCase(const MethodDescriptor* method) {
   return UnderscoresToCamelCase(method->name(), false);
 }
 
+std::string UnderscoresToCamelCaseCheckReserved(const FieldDescriptor* field) {
+  std::string name = UnderscoresToCamelCase(field);
+  if (kReservedNames.find(name) != kReservedNames.end()) {
+    return name + "_";
+  }
+  return name;
+}
+
 std::string UniqueFileScopeIdentifier(const Descriptor* descriptor) {
   return "static_" + StringReplace(descriptor->full_name(), ".", "_", true);
 }

src/google/protobuf/compiler/java/java_helpers.h

@@ -83,6 +83,9 @@ std::string UnderscoresToCapitalizedCamelCase(const FieldDescriptor* field);
 // of lower-casing the first letter of the name.)
 std::string UnderscoresToCamelCase(const MethodDescriptor* method);
 
+// Same as UnderscoresToCamelCase, but checks for reserved keywords
+std::string UnderscoresToCamelCaseCheckReserved(const FieldDescriptor* field);
+
 // Similar to UnderscoresToCamelCase, but guarentees that the result is a
 // complete Java identifier by adding a _ if needed.
 std::string CamelCaseFieldName(const FieldDescriptor* field);

src/google/protobuf/stubs/strutil.h

@@ -43,7 +43,7 @@
 namespace google {
 namespace protobuf {
 
-#ifdef _MSC_VER
+#if defined(_MSC_VER) && _MSC_VER < 1800
 #define strtoll  _strtoi64
 #define strtoull _strtoui64
 #elif defined(__DECCXX) && defined(__osf__)

tests.sh

@@ -117,6 +117,12 @@ build_csharp() {
   internal_build_cpp
   NUGET=/usr/local/bin/nuget.exe
 
+  # Disable some unwanted dotnet options
+  export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true
+  export DOTNET_CLI_TELEMETRY_OPTOUT=true
+
+  # TODO(jtattermusch): is this still needed with "first time experience"
+  # disabled?
   # Perform "dotnet new" once to get the setup preprocessing out of the
   # way. That spews a lot of output (including backspaces) into logs
   # otherwise, and can cause problems. It doesn't matter if this step
@@ -467,6 +473,16 @@ build_php5.5_c() {
   # popd
 }
 
+build_php5.5_mixed() {
+  use_php 5.5
+  pushd php
+  rm -rf vendor
+  composer update
+  /bin/bash ./tests/compile_extension.sh ./ext/google/protobuf
+  php -dextension=./ext/google/protobuf/modules/protobuf.so ./vendor/bin/phpunit
+  popd
+}
+
 build_php5.5_zts_c() {
   use_php_zts 5.5
   cd php/tests && /bin/bash ./test.sh 5.5-zts && cd ../..
@@ -497,6 +513,16 @@ build_php5.6_c() {
   # popd
 }
 
+build_php5.6_mixed() {
+  use_php 5.6
+  pushd php
+  rm -rf vendor
+  composer update
+  /bin/bash ./tests/compile_extension.sh ./ext/google/protobuf
+  php -dextension=./ext/google/protobuf/modules/protobuf.so ./vendor/bin/phpunit
+  popd
+}
+
 build_php5.6_zts_c() {
   use_php_zts 5.6
   cd php/tests && /bin/bash ./test.sh 5.6-zts && cd ../..
@@ -552,6 +578,16 @@ build_php7.0_c() {
   # popd
 }
 
+build_php7.0_mixed() {
+  use_php 7.0
+  pushd php
+  rm -rf vendor
+  composer update
+  /bin/bash ./tests/compile_extension.sh ./ext/google/protobuf
+  php -dextension=./ext/google/protobuf/modules/protobuf.so ./vendor/bin/phpunit
+  popd
+}
+
 build_php7.0_zts_c() {
   use_php_zts 7.0
   cd php/tests && /bin/bash ./test.sh 7.0-zts && cd ../..
@@ -615,6 +651,16 @@ build_php7.1_c() {
   fi
 }
 
+build_php7.1_mixed() {
+  use_php 7.1
+  pushd php
+  rm -rf vendor
+  composer update
+  /bin/bash ./tests/compile_extension.sh ./ext/google/protobuf
+  php -dextension=./ext/google/protobuf/modules/protobuf.so ./vendor/bin/phpunit
+  popd
+}
+
 build_php7.1_zts_c() {
   use_php_zts 7.1
   cd php/tests && /bin/bash ./test.sh 7.1-zts && cd ../..
@@ -632,6 +678,10 @@ build_php_all_32() {
   build_php5.6_c
   build_php7.0_c
   build_php7.1_c $1
+  build_php5.5_mixed
+  build_php5.6_mixed
+  build_php7.0_mixed
+  build_php7.1_mixed
   build_php5.5_zts_c
   build_php5.6_zts_c
   build_php7.0_zts_c