Merge pull request #4913 from xfxyjwf/protoc

Update protoc build scripts.

protoc-artifacts/README.md

@@ -7,13 +7,31 @@ build and publish a ``protoc`` executable (a.k.a. artifact) to Maven
 repositories. The artifact can be used by build automation tools so that users
 would not need to compile and install ``protoc`` for their systems.
 
+If you would like us to publish protoc artifact for a new platform, please send
+us a pull request to add support for the new platform. You would need to change
+the following files:
+
+* [build-protoc.sh](build-protoc.sh): script to cross-build the protoc for your
+  platform.
+* [pom.xml](pom.xml): script to upload artifacts to maven.
+* [build-zip.sh](build-zip.sh): script to package published maven artifacts in
+  our release page.
+
+## Maven Location
+The published protoc artifacts are available on Maven here:
+
+    http://central.maven.org/maven2/com/google/protobuf/protoc/
+
 ## Versioning
 The version of the ``protoc`` artifact must be the same as the version of the
 Protobuf project.
 
 ## Artifact name
 The name of a published ``protoc`` artifact is in the following format:
-``protoc-<version>-<os>-<arch>.exe``, e.g., ``protoc-3.0.0-alpha-3-windows-x86_64.exe``.
+``protoc-<version>-<os>-<arch>.exe``, e.g., ``protoc-3.6.1-linux-x86_64.exe``.
+
+Note that artifacts for linux/macos also have the `.exe` suffix but they are
+not windows binaries.
 
 ## System requirement
 Install [Apache Maven](http://maven.apache.org/) if you don't have it.
@@ -29,95 +47,116 @@ generate the configure script.
 
 Under the protobuf project directory:
 
+
 ```
-$ ./autogen.sh && ./configure && make
+$ ./autogen.sh
 ```
 
-## To install artifacts locally
-The following command will install the ``protoc`` artifact to your local Maven repository.
+### Build the artifact for each platform
+
+Run the build-protoc.sh script under this protoc-artifacts directory to build the protoc
+artifact for each platform.  For example:
+
 ```
-$ mvn install
+$ cd protoc-artifacts
+$ ./build-protoc.sh linux x86_64 protoc
 ```
 
-## Cross-compilation
-The Maven script will try to detect the OS and the architecture from Java
-system properties. It's possible to build a protoc binary for an architecture
-that is different from what Java has detected, as long as you have the proper
-compilers installed.
+The above command will produce a `target/linux/x86_64/protoc` binary under the
+protoc-artifacts directory.
 
-You can override the Maven properties ``os.detected.name`` and
-``os.detected.arch`` to force the script to generate binaries for a specific OS
-and/or architecture. Valid values are defined as the return values of
-``normalizeOs()`` and ``normalizeArch()`` of ``Detector`` from
-[os-maven-plugin](https://github.com/trustin/os-maven-plugin/blob/master/src/main/java/kr/motd/maven/os/Detector.java).
-Frequently used values are:
-- ``os.detected.name``: ``linux``, ``osx``, ``windows``.
-- ``os.detected.arch``: ``x86_32``, ``x86_64``
+For a list of supported platforms, see the comments in the build-protoc.sh
+script. We only use this script to build artifacts on Ubuntu and MacOS (both
+with x86_64, and do cross-compilation for other platforms.
+
+### Tips for building for Linux
+We build on Centos 6.9 to provide a good compatibility for not very new
+systems. We have provided a ``Dockerfile`` under this directory to build the
+environment. It has been tested with Docker 1.6.1.
+
+To build a image:
 
-For example, MinGW32 only ships with 32-bit compilers, but you can still build
-32-bit protoc under 64-bit Windows, with the following command:
 ```
-$ mvn install -Dos.detected.arch=x86_32
+$ docker build -t protoc-artifacts .
 ```
 
+To run the image:
+
+```
+$ docker run -it --rm=true protoc-artifacts bash
+```
+
+To checkout protobuf (run within the container):
+
+```
+$ # Replace v3.5.1 with the version you want
+$ wget -O - https://github.com/google/protobuf/archive/v3.5.1.tar.gz | tar xvzp
+```
+
+### Windows build
+We no longer use scripts in this directory to build windows artifacts. Instead,
+we use Visual Studio 2015 to build our windows release artifacts. See our
+[kokoro windows build scripts here](../kokoro/release/protoc/windows/build.bat).
+
+To upload windows artifacts, copy the built binaries into this directory and
+put it into the target/windows/(x86_64|x86_32) directory the same way as the
+artifacts for other platforms. That will allow the maven script to find and
+upload the artifacts to maven.
+
 ## To push artifacts to Maven Central
 Before you can upload artifacts to Maven Central repository, make sure you have
 read [this page](http://central.sonatype.org/pages/apache-maven.html) on how to
 configure GPG and Sonatype account.
 
-You need to perform the deployment for every platform that you want to
-support. DO NOT close the staging repository until you have done the
-deployment for all platforms. Currently the following platforms are supported:
-- Linux (x86_32, x86_64 and cross compiled aarch_64)
-- Windows (x86_32 and x86_64) with
-  - Cygwin64 with MinGW compilers (x86_64)
-  - MSYS with MinGW32 (x86_32)
-  - Cross compile in Linux with MinGW-w64 (x86_32, x86_64)
-- MacOSX (x86_32 and x86_64)
-
-As for MSYS2/MinGW64 for Windows: protoc will build, but it insists on
-adding a dependency of `libwinpthread-1.dll`, which isn't shipped with
-Windows.
+Before you do the deployment, make sure you have built the protoc artifacts for
+every supported platform and put them under the target directory. Example
+target directory layout:
+
+    + pom.xml
+    + target
+      + linux
+        + x86_64
+          protoc.exe
+        + x86_32
+          protoc.exe
+      + macos
+        + x86_64
+          protoc.exe
+        + x86_32
+          protoc.exe
+      + windows
+        + x86_64
+          protoc.exe
+        + x86_32
+          protoc.exe
+
+You will need to build the artifacts on multiple machines and gather them
+together into one place.
 
 Use the following command to deploy artifacts for the host platform to a
 staging repository.
+
 ```
 $ mvn clean deploy -P release
 ```
+
 It creates a new staging repository. Go to
 https://oss.sonatype.org/#stagingRepositories and find the repository, usually
-in the name like ``comgoogle-123``.
-
-You will want to run this command on a different platform. Remember, in
-subsequent deployments you will need to provide the repository name that you
-have found in the first deployment so that all artifacts go to the same
-repository:
-```
-$ mvn clean deploy -P release -Dstaging.repository=comgoogle-123
-```
-
-A 32-bit artifact can be deployed from a 64-bit host with
-``-Dos.detected.arch=x86_32``
-
-An arm64 artifact can be deployed from x86 host with
-``-Dos.detected.arch=aarch_64``
-
-A windows artifact can be deployed from a linux machine with
-``-Dos.detected.name=windows``
-
-When you have done deployment for all platforms, go to
-https://oss.sonatype.org/#stagingRepositories, verify that the staging
-repository has all the binaries, close and release this repository.
+in the name like ``comgoogle-123``. Verify that the staging repository has all
+the binaries, close and release this repository.
 
 ## Upload zip packages to github release page.
 After uploading protoc artifacts to Maven Central repository, run the
 build-zip.sh script to bulid zip packages for these protoc binaries
 and upload these zip packages to the download section of the github
 release. For example:
+
 ```
-$ ./build-zip.sh 3.0.0-beta-4
+$ ./build-zip.sh 3.6.0
 ```
+
 The above command will create 5 zip files:
+
 ```
 dist/protoc-3.0.0-beta-4-win32.zip
 dist/protoc-3.0.0-beta-4-osx-x86_32.zip
@@ -125,67 +164,14 @@ dist/protoc-3.0.0-beta-4-osx-x86_64.zip
 dist/protoc-3.0.0-beta-4-linux-x86_32.zip
 dist/protoc-3.0.0-beta-4-linux-x86_64.zip
 ```
+
 Before running the script, make sure the artifacts are accessible from:
 http://repo1.maven.org/maven2/com/google/protobuf/protoc/
 
-### Tips for deploying on Linux
-We build on Centos 6.6 to provide a good compatibility for not very new
-systems. We have provided a ``Dockerfile`` under this directory to build the
-environment. It has been tested with Docker 1.6.1.
-
-To build a image:
-```
-$ docker build -t protoc-artifacts .
-```
-
-To run the image:
-```
-$ docker run -it --rm=true protoc-artifacts bash
-```
-
-To checkout protobuf (run within the container):
-```
-$ # Replace v3.5.1 with the version you want
-$ wget -O - https://github.com/google/protobuf/archive/v3.5.1.tar.gz | tar xvzp
-```
-
-### Tips for deploying on Windows
-Under Windows the following error may occur: ``gpg: cannot open tty `no tty':
-No such file or directory``. This can be fixed by configuring gpg through an
-active profile in ``.m2\settings.xml`` where also the Sonatype password is
-stored:
-```xml
-<settings>
-  <servers>
-    <server>
-      <id>sonatype-nexus-staging</id>
-      <username>[username]</username>
-      <password>[password]</password>
-    </server>
-  </servers>
-  <profiles>
-    <profile>
-      <id>gpg</id>
-      <properties>
-        <gpg.executable>gpg</gpg.executable>
-        <gpg.passphrase>[password]</gpg.passphrase>
-      </properties>
-    </profile>
-  </profiles>
-  <activeProfiles>
-    <activeProfile>gpg</activeProfile>
-  </activeProfiles>
-</settings>
-```
-
-### Tested build environments
+## Tested build environments
 We have successfully built artifacts on the following environments:
 - Linux x86_32 and x86_64:
-  - Centos 6.6 (within Docker 1.6.1)
-  - Ubuntu 14.04.2 64-bit
-- Linux aarch_64: Cross compiled with `g++-aarch64-linux-gnu` on Ubuntu 14.04.2 64-bit
-- Windows x86_32: MSYS with ``mingw32-gcc-g++ 4.8.1-4`` on Windows 7 64-bit
-- Windows x86_32: Cross compile with ``i686-w64-mingw32-g++ 4.8.2`` on Ubuntu 14.04.2 64-bit
-- Windows x86_64: Cygwin64 with ``mingw64-x86_64-gcc-g++ 4.8.3-1`` on Windows 7 64-bit
-- Windows x86_64: Cross compile with ``x86_64-w64-mingw32-g++ 4.8.2`` on Ubuntu 14.04.2 64-bit
+  - Centos 6.9 (within Docker 1.6.1)
+  - Ubuntu 14.04.5 64-bit
+- Linux aarch_64: Cross compiled with `g++-aarch64-linux-gnu` on Ubuntu 14.04.5 64-bit
 - Mac OS X x86_32 and x86_64: Mac OS X 10.9.5

protoc-artifacts/build-protoc.sh

 #!/bin/bash
 
-# Builds protoc executable into target/protoc.exe; optionally build protoc
-# plugins into target/protoc-gen-*.exe
-# To be run from Maven.
-# Usage: build-protoc.sh <OS> <ARCH> <TARGET>
-# <OS> and <ARCH> are ${os.detected.name} and ${os.detected.arch} from os-maven-plugin
-# <TARGET> can be "protoc" or "protoc-gen-javalite"
+# Builds protoc executable into target/<OS>/<ARCH>/protoc.exe; optionally builds
+# protoc plugins into target/<OS>/<ARCH>/protoc-gen-*.exe
 #
-# The script now supports cross-compiling windows and linux-arm64 in linux-x86
-# environment. Required packages:
-# - Windows: i686-w64-mingw32-gcc (32bit) and x86_64-w64-mingw32-gcc (64bit)
-# - Arm64: g++-aarch64-linux-gnu
+# Usage: ./build-protoc.sh <OS> <ARCH> <TARGET>
+#
+# <TARGET> can be "protoc" or "protoc-gen-javalite". Supported <OS> <ARCH>
+# combinations:
+#   HOST   <OS>    <ARCH>   <COMMENT>
+#   cygwin windows x86_32   Requires: i686-w64-mingw32-gcc
+#   cygwin windows x86_64   Requires: x86_64-w64-mingw32-gcc
+#   linux  linux   aarch_64 Requires: g++-aarch64-linux-gnu
+#   linux  linux   x86_32
+#   linux  linux   x86_64
+#   linux  windows x86_32   Requires: i686-w64-mingw32-gcc
+#   linux  windows x86_64   Requires: x86_64-w64-mingw32-gcc
+#   macos  osx     x86_32
+#   macos  osx     x86_64
+#   mingw  windows x86_32
+#   mingw  windows x86_64
+#
+# Before running this script, make sure you have generated the configure script
+# in the parent directory (i.e., run ./autogen.sh there).
 
 OS=$1
 ARCH=$2
 MAKE_TARGET=$3
 
 if [[ $# < 3 ]]; then
-  echo "No arguments provided. This script is intended to be run from Maven."
+  echo "Not enough arguments provided."
   exit 1
 fi
 
@@ -156,13 +167,8 @@ checkDependencies ()
 
 echo "Building protoc, OS=$OS ARCH=$ARCH TARGET=$TARGET"
 
-# Nested double quotes are unintuitive, but it works.
-cd "$(dirname "$0")"
-
-WORKING_DIR=$(pwd)
 CONFIGURE_ARGS="--disable-shared"
 
-TARGET_FILE=target/$MAKE_TARGET.exe
 if [[ "$OS" == windows ]]; then
   MAKE_TARGET="${MAKE_TARGET}.exe"
 fi
@@ -242,10 +248,18 @@ fi
 
 export CXXFLAGS LDFLAGS
 
-cd "$WORKING_DIR"/.. && ./configure $CONFIGURE_ARGS &&
-  cd src && make clean && make $MAKE_TARGET &&
-  cd "$WORKING_DIR" && mkdir -p target &&
-  cp ../src/$MAKE_TARGET $TARGET_FILE ||
+# Nested double quotes are unintuitive, but it works.
+cd "$(dirname "$0")"
+
+WORKING_DIR="$(pwd)"
+BUILD_DIR="build/$OS/$ARCH"
+TARGET_FILE="target/$OS/$ARCH/$MAKE_TARGET.exe"
+
+mkdir -p "$BUILD_DIR" && cd "$BUILD_DIR" &&
+  ../../../../configure $CONFIGURE_ARGS &&
+  cd src && make $MAKE_TARGET -j8 &&
+  cd "$WORKING_DIR" && mkdir -p $(dirname $TARGET_FILE) &&
+  cp $BUILD_DIR/src/$MAKE_TARGET $TARGET_FILE ||
   exit 1
 
 if [[ "$OS" == osx ]]; then

protoc-artifacts/pom.xml

@@ -33,36 +33,7 @@
     </connection>
   </scm>
   <build>
-    <extensions>
-      <extension>
-        <groupId>kr.motd.maven</groupId>
-        <artifactId>os-maven-plugin</artifactId>
-        <version>1.5.0.Final</version>
-      </extension>
-    </extensions>
     <plugins>
-      <plugin>
-        <groupId>org.codehaus.mojo</groupId>
-        <artifactId>exec-maven-plugin</artifactId>
-        <version>1.1.1</version>
-        <executions>
-          <execution>
-            <phase>compile</phase>
-            <goals>
-              <goal>exec</goal>
-            </goals>
-          </execution>
-        </executions>
-        <configuration>
-          <executable>bash</executable>
-          <arguments>
-            <argument>build-protoc.sh</argument>
-            <argument>${os.detected.name}</argument>
-            <argument>${os.detected.arch}</argument>
-            <argument>protoc</argument>
-          </arguments>
-        </configuration>
-      </plugin>
       <plugin>
         <groupId>org.codehaus.mojo</groupId>
         <artifactId>build-helper-maven-plugin</artifactId>
@@ -77,8 +48,38 @@
             <configuration>
               <artifacts>
                 <artifact>
-                  <file>${basedir}/target/protoc.exe</file>
-                  <classifier>${os.detected.name}-${os.detected.arch}</classifier>
+                  <file>${basedir}/target/linux/x86_64/protoc.exe</file>
+                  <classifier>linux-x86_64</classifier>
+                  <type>exe</type>
+                </artifact>
+                <artifact>
+                  <file>${basedir}/target/linux/x86_32/protoc.exe</file>
+                  <classifier>linux-x86_32</classifier>
+                  <type>exe</type>
+                </artifact>
+                <artifact>
+                  <file>${basedir}/target/windows/x86_64/protoc.exe</file>
+                  <classifier>windows-x86_64</classifier>
+                  <type>exe</type>
+                </artifact>
+                <artifact>
+                  <file>${basedir}/target/windows/x86_32/protoc.exe</file>
+                  <classifier>windows-x86_32</classifier>
+                  <type>exe</type>
+                </artifact>
+                <artifact>
+                  <file>${basedir}/target/osx/x86_64/protoc.exe</file>
+                  <classifier>osx-x86_64</classifier>
+                  <type>exe</type>
+                </artifact>
+                <artifact>
+                  <file>${basedir}/target/osx/x86_32/protoc.exe</file>
+                  <classifier>osx-x86_32</classifier>
+                  <type>exe</type>
+                </artifact>
+                <artifact>
+                  <file>${basedir}/target/linux/aarch_64/protoc.exe</file>
+                  <classifier>linux-aarch_64</classifier>
                   <type>exe</type>
                 </artifact>
               </artifacts>