Merge pull request #1139 from ewen-naos-nz/zos

z/OS: Updated to build libzmq git master

builds/zos/README.md

@@ -12,6 +12,8 @@ Tested build combinations:
 
 * ZeroMQ 4.0.4, using IBM XL C/C++ compiler, as XPLINK in LP64 mode
 
+* ZeroMQ 4.1-git, using IBM XL C/C++ compiler, as XPLINK in ILP32 mode
+
 Other combinations are likely to work, possibly with minor changes,
 but have not been tested.  Both static library and DLL modes have been
 tested.
@@ -201,9 +203,18 @@ pass. There are two tests that are expected to fail:
     programs -- most programs use threads or fork without exec, but
     not both.
 
-These two "expected to fail" tests are listed as XFAIL_TESTS, and
+0.  `test_diffserv`: tests ability to set IP_TOS ([IP Type of
+    Service](http://en.wikipedia.org/wiki/Type_of_service), or
+    [DiffServ](http://en.wikipedia.org/wiki/Differentiated_Services_Code_Point))
+    values on sockets.  While z/OS UNIX System Services has the
+    preprocessor defines required, it appears not to support the
+    required functionality (call fails with "EDC8109I Protocol not
+    available.")
+
+These three "expected to fail" tests are listed as XFAIL_TESTS, and
 `runtests` will still consider the test run successful when they fail
-as expected.
+as expected.  (`builds/zos/runtests` will automatically skip these
+"expected to fail" tests if running "all" tests.)
 
 In addition `test_security_curve` does not do any meaningful testing,
 as a result of the CURVE support not being compiled in; it requires
@@ -212,6 +223,13 @@ ported to z/OS UNIX System Services yet.
 
 Multicast (via `libpgm`) is also not ported or compiled in.
 
+[TIPC](http://hintjens.com/blog:70), a cluster IPC protocol,
+is only supported on Linux, so it is not compiled into the z/OS
+UNIX System Services port -- and the tests are automatically skipped
+if running "all" tests.  (However they are not listed in XFAIL_TESTS
+because without the TIPC support there is no point in even running
+them, and it would be non-trivial to track them by hand.)
+
 
 ## ZeroMQ on z/OS UNIX System Services: Library portability notes
 
@@ -278,6 +296,18 @@ which is done in the `cxxall` script.  (The "3" value exposes later
 pthreads functionality like `pthread_atfork`, although ZeroMQ does not
 currently use all these features.)
 
+If compiling on a *recent* version of z/OS UNIX System Services it
+may be worth compiling with:
+
+    _UNIX03_THREADS=1
+
+which enables a later version of the threading support, potentially
+including `pthread_getschedparam` and pthread_setschedparam`; at
+present in the z/OS UNIX System Services port these functions are
+hidden and never called.  (See [IBM z/OS pthread.h
+documentation](http://pic.dhe.ibm.com/infocenter/zos/v1r11/index.jsp?topic=/com.ibm.zos.r11.bpxbd00/pthrdh.htm)
+for details on the differences.)
+
 
 ## `platform.hpp` on z/OS UNIX System Services
 
@@ -286,6 +316,11 @@ pre-built `platform.hpp` file.  (By default `src/platform.hpp` is
 dynamically generated as a result of running the `./configure` script.)
 The master version of this is in `builds/zos/platform.hpp`.
 
+Beware that this file contains the version number for libzmq (usually
+included during the configure phase).  If taking the `platform.hpp` from
+an older version to use on a newer libzmq be sure to update the version
+information near the top of the file.
+
 The pre-built file is used because z/OS does not have the GNU auto tools
 (`automake`, `autoconf`, `libtool`, etc) installed, and particularly the
 libtool replacement does not work properly with the IBM XL C/C++
@@ -317,18 +352,20 @@ syntax):
 
 *   set `CPPFLAGS` to for the feature macros required, eg:
 
-        setenv CPPFLAGS "-D_XOPEN_SOURCE_EXTENDED=1 -D_OPEN_THREADS=3 -D_OPEN_SYS_SOCK_IPV6"
+        setenv CPPFLAGS "-D_XOPEN_SOURCE_EXTENDED=1 -D_OPEN_THREADS=3 -D_OPEN_SYS_SOCK_IPV6 -DZMQ_HAVE_ZOS"
 
 *   set `CXXFLAGS` to enable XPLINK:
 
         setenv CXXFLAGS "-Wc,xplink -Wl,xplink -+"
 
 *   run configure script with `--disable-eventfd` (`sys/eventfd.h` does
-    not exist, but the test for its existance has a false positive on 
-    z/OS UNIX System Services, apparently due to the way the `c++` 
-    compiler wrapper passes errors back from the IBM XL C/C++ compiler), viz:
+    not exist, but the test for its existance has a false positive on
+    z/OS UNIX System Services, apparently due to the way the `c++`
+    compiler wrapper passes errors back from the IBM XL C/C++ compiler),
+    and with `--with-poller=poll` because `poll` is the most advanced
+    of the file descriptor status tests available on z/OS.  That is:
 
-        ./configure --disable-eventfd
+        ./configure --disable-eventfd --with-poller=poll
 
 All going well several Makefiles, and `src/platform.hpp` should be
 produced.  Two additional changes are required to `src/platform.hpp`
@@ -369,3 +406,58 @@ but note:
 
 However running `./configure` to regenerate `src/platform.hpp` may 
 be useful for later versions of ZeroMQ which add more feature tests.
+
+
+## Transferring from GitHub to z/OS UNIX System Services
+
+The process of transferring files from GitHub to z/OS UNIX System
+Services is somewhat convoluted because:
+
+*   There is not a port of git for z/OS UNIX System Services; and
+
+*   z/OS uses the EBCDIC (IBM-1047) character set rather than the
+    ASCII/ISO-8859-1 character set used by the ZeroMQ source code
+    on GitHub
+
+A workable transfer process is:
+
+*   On an ASCII/ISO-8859-1/UTF-8 system with `git` (eg, a Linux system):
+
+        git clone https://github.com/zeromq/libzmq.git
+        git archive --prefix=libzmq-git/ -o /var/tmp/libzmq-git.tar master
+
+*   On a ASCII/ISO-8859-1/UTF-8 system with `tar`, and `pax`, and
+    optionally the GNU auto tools (eg, the same Linux system):
+
+        mkdir /var/tmp/zos
+        cd /var/tmp/zos
+        tar -xpf /var/tmp/libzmq-git.tar
+        cd libzmq-git
+        ./autogen.sh             # Optional: to be able to run ./configure
+        cd ..
+        pax -wf /var/tmp/libzmq-git.pax libzmq-git
+        compress libzmq-git.pax  # If available, reduce transfer size
+
+*   Transfer the resulting file (`libzmq-git.pax` or `libzmq-git.pax.Z`)
+    to the z/OS UNIX System Services system.  If using FTP be sure to
+    transfer the file in `bin` (binary/Image) mode to avoid corruption.
+
+*   On the z/OS UNIX System Services system, unpack the `pax` file and
+    convert all the files to EBCDIC with:
+
+        pax -o from=iso8859-1 -pp -rvf  libzmq-git-2014-07-23.pax
+
+    or if the file was compressed:
+
+        pax -o from=iso8859-1 -pp -rvzf libzmq-git-2014-07-23.pax.Z
+
+The result should be a `libzmq-git` directory with the source in
+EBCDIC format, on the z/OS UNIX System Services system ready to start
+building.
+
+See also the [`pax` man
+page](http://pic.dhe.ibm.com/infocenter/zos/v1r13/index.jsp?topic=%2Fcom.ibm.zos.r13.bpxa500%2Fr4paxsh.htm),
+some [`pax` conversion
+examples](http://pic.dhe.ibm.com/infocenter/zos/v1r13/index.jsp?topic=%2Fcom.ibm.zos.r13.bpxa400%2Fbpxza4c0291.htm),
+and [IBM's advice on ASCII to EBCDIC conversion
+options](http://www-03.ibm.com/systems/z/os/zos/features/unix/bpxa1p03.html)

builds/zos/cxxall

@@ -4,7 +4,7 @@
 # additional compile arguments.
 #
 # Written by Ewen McNeill <ewen@imatix.com>, 2014-07-19
-# Updated by Ewen McNeill <ewen@imatix.com>, 2014-07-22
+# Updated by Ewen McNeill <ewen@imatix.com>, 2014-07-24
 #---------------------------------------------------------------------------
 
 VERBOSE="${VERBOSE:-}"    # Set to non-empty for already done status
@@ -22,7 +22,7 @@ ZCXX="${BIN_DIR}/zc++"
 
 # Determine compile flags
 CPPFLAGS="-D_XOPEN_SOURCE_EXTENDED=1 -D_OPEN_THREADS=3 -D_OPEN_SYS_SOCK_IPV6"
-CXXFLAGS="-DHAVE_CONFIG_H -D_REENTRANT -D_THREAD_SAFE -DZMQ_FORCE_POLL"
+CXXFLAGS="-DZMQ_HAVE_ZOS -DHAVE_CONFIG_H -D_REENTRANT -D_THREAD_SAFE -DZMQ_USE_POLL"
 case $(pwd) in
   *src)   CXXFLAGS="${CXXFLAGS} -I."
           ;;

builds/zos/makeclean

@@ -2,6 +2,7 @@
 # Remove built object files and test executables
 #
 # Written by Ewen McNeill <ewen@imatix.com>, 2014-07-22
+# Updated by Ewen McNeill <ewen@imatix.com>, 2014-07-24
 #---------------------------------------------------------------------------
 
 set -e    # Stop on errors
@@ -22,7 +23,7 @@ TESTS="${TOP}/tests"
 
 # Remove object/library files
 echo "Removing libzmq built files"
-(cd "${SRC}" && rm -f *.o *.a *.dbg *.x libzmq* *.so)
+(cd "${SRC}" && rm -f *.o *.a *.dbg *.x *.so libzmq)
 
 # Remove test object files
 echo "Removing libzmq tests"

builds/zos/platform.hpp

@@ -10,6 +10,14 @@
 /* Define to 1 if you have the `clock_gettime' function. */
 /* #undef HAVE_CLOCK_GETTIME */
 
+/* Define to 1 if you have the declaration of `LOCAL_PEERCRED', and to 0 if
+   you don't. */
+#define HAVE_DECL_LOCAL_PEERCRED 0
+
+/* Define to 1 if you have the declaration of `SO_PEERCRED', and to 0 if you
+   don't. */
+#define HAVE_DECL_SO_PEERCRED 0
+
 /* Define to 1 if you have the <dlfcn.h> header file. */
 #define HAVE_DLFCN_H 1
 
@@ -37,6 +45,9 @@
 /* Define to 1 if you have the <inttypes.h> header file. */
 #define HAVE_INTTYPES_H 1
 
+/* Define to 1 if you have the `gssapi_krb5' library (-lgssapi_krb5). */
+/* #undef HAVE_LIBGSSAPI_KRB5 */
+
 /* Define to 1 if you have the `iphlpapi' library (-liphlpapi). */
 /* #undef HAVE_LIBIPHLPAPI */
 
@@ -134,9 +145,6 @@
    */
 #define LT_OBJDIR ".libs/"
 
-/* Define to 1 if your C compiler doesn't accept -c and -o together. */
-#define NO_MINUS_C_MINUS_O 1
-
 /* Name of package */
 #define PACKAGE "zeromq"
 
@@ -147,7 +155,7 @@
 #define PACKAGE_NAME "zeromq"
 
 /* Define to the full name and version of this package. */
-#define PACKAGE_STRING "zeromq 4.0.4"
+#define PACKAGE_STRING "zeromq 4.1.0"
 
 /* Define to the one symbol short name of this package. */
 #define PACKAGE_TARNAME "zeromq"
@@ -156,7 +164,7 @@
 #define PACKAGE_URL ""
 
 /* Define to the version of this package. */
-#define PACKAGE_VERSION "4.0.4"
+#define PACKAGE_VERSION "4.1.0"
 
 /* Define as the return type of signal handlers (`int' or `void'). */
 #define RETSIGTYPE void
@@ -168,7 +176,10 @@
 #define TIME_WITH_SYS_TIME 1
 
 /* Version number of package */
-#define VERSION "4.0.4"
+#define VERSION "4.1.0"
+
+/* Enable militant API assertions */
+/* #undef ZMQ_ACT_MILITANT */
 
 /* Force to use mutexes */
 /* #undef ZMQ_FORCE_MUTEXES */
@@ -197,12 +208,18 @@
 /* Have Linux OS */
 /* #undef ZMQ_HAVE_LINUX */
 
+/* Have LOCAL_PEERCRED socket option */
+/* #undef ZMQ_HAVE_LOCAL_PEERCRED */
+
 /* Have MinGW32 */
 /* #undef ZMQ_HAVE_MINGW32 */
 
 /* Have NetBSD OS */
 /* #undef ZMQ_HAVE_NETBSD */
 
+/* Have NORM protocol extension */
+/* #undef ZMQ_HAVE_NORM */
+
 /* Have OpenBSD OS */
 /* #undef ZMQ_HAVE_OPENBSD */
 
@@ -222,7 +239,10 @@
 /* #undef ZMQ_HAVE_SOLARIS */
 
 /* Whether SO_KEEPALIVE is supported. */
-/* #undef ZMQ_HAVE_SO_KEEPALIVE */
+#define ZMQ_HAVE_SO_KEEPALIVE 1
+
+/* Have SO_PEERCRED socket option */
+/* #undef ZMQ_HAVE_SO_PEERCRED */
 
 /* Whether TCP_KEEPALIVE is supported. */
 /* #undef ZMQ_HAVE_TCP_KEEPALIVE */
@@ -236,6 +256,9 @@
 /* Whether TCP_KEEPINTVL is supported. */
 /* #undef ZMQ_HAVE_TCP_KEEPINTVL */
 
+/* Have TIPC support */
+/* #undef ZMQ_HAVE_TIPC */
+
 /* Have uio.h header. */
 #define ZMQ_HAVE_UIO 1
 

builds/zos/runtests

@@ -7,81 +7,11 @@
 # defines the order in which tests are run.
 #
 # Written by Ewen McNeill <ewen@imatix.com>, 2014-07-19
-# Updated by Ewen McNeill <ewen@imatix.com>, 2014-07-22
+# Updated by Ewen McNeill <ewen@imatix.com>, 2014-07-24
 #---------------------------------------------------------------------------
 
 set -e    # Stop if a test fails
 
-# Test order taken from tests/Makefile.am
-noinst_PROGRAMS="test_system
-                 test_pair_inproc
-                 test_pair_tcp
-                 test_reqrep_inproc
-                 test_reqrep_tcp
-                 test_hwm
-                 test_reqrep_device
-                 test_sub_forward
-                 test_invalid_rep
-                 test_msg_flags
-                 test_connect_resolve
-                 test_immediate
-                 test_last_endpoint
-                 test_term_endpoint
-                 test_monitor
-                 test_router_mandatory
-                 test_router_raw_empty
-                 test_probe_router
-                 test_stream
-                 test_disconnect_inproc
-                 test_ctx_options
-                 test_ctx_destroy
-                 test_security_null
-                 test_security_plain
-                 test_security_curve
-                 test_iov
-                 test_spec_req
-                 test_spec_rep
-                 test_spec_dealer
-                 test_spec_router
-                 test_spec_pushpull
-                 test_req_correlate
-                 test_req_relaxed
-                 test_conflate
-                 test_inproc_connect
-                 test_issue_566
-                 test_abstract_ipc
-                 test_many_sockets
-                 test_shutdown_stress
-                 test_pair_ipc
-                 test_reqrep_ipc
-                 test_timeo
-                 test_fork"
-
-if [ -n "${1}" ]; then
-  TESTS="$*"     # Run tests on command line
-else
-  TESTS="${noinst_PROGRAMS}"
-fi
-
-# Explanation of tests expected to fail:
-# test_abstract_ipc: Relies on Linux-specific IPC functionality
-# test_fork:         Relies on using pthreads _after_ fork, _before_ exec
-#
-# (Note: there _must_ be spaces either side of the tests names to match)
-XFAIL_TESTS="
-             test_abstract_ipc
-             test_fork
-            "
-
-verbose() {
-   echo "Starting: $@" >&2
-   "$@"
-}
-
-# Uncomment TESTS_ENVIRONMENT=verbose to see "Starting: TEST" messages
-#TESTS_ENVIRONMENT=verbose
-TESTS_ENVIRONMENT=
-
 #---------------------------------------------------------------------------
 # Change to tests directory if necessary
 
@@ -113,6 +43,48 @@ else
   exit 1
 fi
 
+#---------------------------------------------------------------------------
+# Explanation of tests expected to fail:
+# test_abstract_ipc: Relies on Linux-specific IPC functionality
+# test_fork:         Relies on using pthreads _after_ fork, _before_ exec
+# test_diffserv:     Needs IP_PROTO, IP_TOS setsockopt(); the headers
+#                    are present on z/OS UNIX System Services, but
+#                    fails with:
+#                    EDC8109I Protocol not available. (./ip.cpp:164)
+#    NOTE: not listed as a valid IP setsockopt() option at:
+#       http://pic.dhe.ibm.com/infocenter/zos/v2r1/index.jsp?topic=%2Fcom.ibm.zos.v2r1.bpxbd00%2Fsetopt.htm
+#
+XFAIL_TESTS="test_abstract_ipc|test_fork|test_diffserv"
+
+# BUILD_TIPC is not defined, so we skip all these tests
+SKIP_TESTS="test_.*_tipc"
+
+# Extract list of all test programs from tests/Makefile.am
+#
+# Excluding tests we know will fail because of missing functionality
+#
+noinst_PROGRAMS=$(grep "test_" Makefile.am | egrep -v "_SOURCES|XFAIL" |
+                  sed 's/noinst_PROGRAMS.* test/test/; s/^ *//; s/ *\\ *$//;' |
+                  grep -v "${SKIP_TESTS}" | egrep -v "${XFAIL_TESTS}")
+#echo "Found tetsts: ${noinst_PROGRAMS}"
+
+# Run tests on command line, or all tests we found
+if [ -n "${1}" ]; then
+  TESTS="$*"     # Run tests on command line
+else
+  TESTS="${noinst_PROGRAMS}"
+fi
+
+verbose() {
+   echo "Starting: $@" >&2
+   "$@"
+}
+
+# Uncomment TESTS_ENVIRONMENT=verbose to see "Starting: TEST" messages
+#TESTS_ENVIRONMENT=verbose
+TESTS_ENVIRONMENT=
+
+#---------------------------------------------------------------------------
 # Explicitly add SRCDIR into library serach path, to make sure we
 # use our just-built version
 LIBPATH="${SRCDIR}:/lib:/usr/lib"

src/metadata.hpp

@@ -31,7 +31,7 @@ namespace zmq
     {
         public:
 
-            typedef std::map <const std::string, const std::string> dict_t;
+            typedef std::map <std::string, const std::string> dict_t;
 
             metadata_t (const dict_t &dict);
             virtual ~metadata_t ();

src/thread.cpp

@@ -104,6 +104,7 @@ void zmq::thread_t::stop ()
 
 void zmq::thread_t::setSchedulingParameters(int priority_, int schedulingPolicy_)
 {
+#if !defined ZMQ_HAVE_ZOS
     int policy = 0;
     struct sched_param param;
 
@@ -122,6 +123,7 @@ void zmq::thread_t::setSchedulingParameters(int priority_, int schedulingPolicy_
 
     rc = pthread_setschedparam(descriptor, policy, &param);
     posix_assert (rc);
+#endif
 }
 
 #endif

tests/test_connect_delay_tipc.cpp

@@ -27,6 +27,8 @@
 #undef NDEBUG
 #include <assert.h>
 
+#include "testutil.hpp"
+
 int main (void)
 {
     int val;

tests/test_sub_forward_tipc.cpp

@@ -26,6 +26,8 @@
 #undef NDEBUG
 #include <assert.h>
 
+#include "testutil.hpp"
+
 int main (void)
 {
     fprintf (stderr, "test_sub_forward running...\n");

tests/test_term_endpoint_tipc.cpp

@@ -27,6 +27,8 @@
 #undef NDEBUG
 #include <assert.h>
 
+#include "testutil.hpp"
+
 int main (void)
 {
     int rc;