Add ZMQ_DELAY_ATTACH_ON_CONNECT sockopt

Update bind and connect documentation for clarity

Rename the pipeset to terminating_pipes, as suggested by Martin H. Adds
asserts to test the pipe is contained in the terminating set where
appropriate.

The previous note in connect introduced but did not elaborate on the
requirement to bind before connect in inproc. As that discussion is in
detail in the inproc docs link to there.

Based on discussion with Steve O on the list, make the difference
between bind and connect usage more clear, and add a note reflecting the
fact that inproc must have bind before connect to reinforce the
information in zmq_inproc.

This patch, salvaged from a trainwreck accidental merge earlier, adds a
new sockopt, ZMQ_DELAY_ATTACH_ON_CONNECT which prevents a end point
being available to push messages to until it has fully connected, making
connect work more like bind. This also applies to reconnecting sockets,
which may cause message loss of in-queue messages, so it is sensible to
use this in conjunction with a low HWM and potentially an alternative
acknowledgement path.

Notes on most of the individual commits can be found the repository log.

This patch, salvaged from a trainwreck accidental merge earlier, adds a
new sockopt, ZMQ_DELAY_ATTACH_ON_CONNECT which prevents a end point
being available to push messages to until it has fully connected, making
connect work more like bind. This also applies to reconnecting sockets,
which may cause message loss of in-queue messages, so it is sensible to
use this in conjunction with a low HWM and potentially an alternative
acknowledgement path.

Notes on most of the individual commits can be found the repository log.

Revert "After speaking with Ben Gray and the discussion on the mailing list, this is an attempt to create a sockopt to allow connecting pipes to not immediately be available for traffic. The problem is in a PUSH to many PULL situation, where there is a connect to a PULL which is not there. This connect will immediately create a pipe (unlike bind), and traffic will be load balanced to that pipe. This means if there is a persistently unavailable end point then the traffic will queue until HWM is hit, and older messages will be lost."

This reverts commit fe3fb419.

Revert "Remove extra brackets as suggested by Martin H, and fix up a comment which was missing a word"

This reverts commit b79aaaf4.

This reverts commit 297af954.

Revert "Remove the extra outpipe handling as the session is quite capable of delaying the creation of the pipe until the connection has happened. Simply don't build the pipe, and let it do that automatically."

This reverts commit 06485d92.

Revert "Fix a number of whitespace issues in various parts of the code, add validation to most calls on the test and take a first stab at implementing the reconnection pipe blocking."

This reverts commit 6f6466f0.

Revert "Fix condition so that PGM and EPGM sockets always create pipes immediately, even if delay_attach_on_connect is set. This allows passing through the icanhasall flag, and is realistic given the fact those protocols should be able to connect immediately"

This reverts commit b5ace39e.

This reverts commit 0b3478d6.

Revert "Rebinding on reconnection to allow for the pipe blocking will set the event sink again, which will cause an assert to be triggered. I have modified that to allow for setting the same sink to be OK. I believe this should be ok - on termination, if a pipe hasn't reconnected, it would just attempt to be erased from the pipes list again, which is reasonable"

This reverts commit 1566091b.

Revert "On the advice of Martin Hurton, removed the new command type and just terminated the pipe in a reconnect situation, and notified the socket of the same. This handles the blocking properly, but at the cost of potentially losing in flight messages. However, this is a reasonable trade off given how much simpler it makes the patch."

This reverts commit c13f1d52.

Revert "Remove unnecessary extra test in pipe assertion - now we are terminating pipes we don't need to allow the case in which a pipe is rebound to the same sink"

This reverts commit 29f8d9ca.

This reverts commit 55cbdfcf.

Revert "Removing unnecessary outpipe values that had been used for reconnecting existing pipes - no longer needed when using a pipe term for the delay_attach situation."

This reverts commit ace7c99b.

This reverts commit 09956dee.

Revert "When detaching a pipe, as well as checking the delay on connect sockopt is set, also ensure that the protocol is not pgm or epgm as we are not implementing the functionality for multicase types"

This reverts commit 5008f385.

Revert "The previous code to block the socket from receiving on that pipe during a disconnect was vulnerable to a race condition. This code calls with the terminate functions of both ends of the pipe - I believe this should be safer. This required storing a pointer to the socket end of the pipe"

This reverts commit 336f7272.

Revert "Reverted to a simpler shutdown. This seems to disconnect and reconnect the pipe properly, but there is a problem in overall shutdown when the pipe has blocked and reconnected - the session seems to get terminated() called on it only in shutdown for the original pipe, by which point it has been replaced. I am not sure at the moment why this only happens then, but this does mean this patch is broken at the moment"

This reverts commit b84b0079.

Revert "Use the hiccup mechanism to notify the socket end of the pair of the change in state, and have it shutdown that end, and shutdown the local end normally. This seems to resolve the shutdown and race condition issues."

This reverts commit 67497a26.

Revert "Code formatting, and clean up so that xhiccuped is only handled in case the pipe isn't terminating"

This reverts commit 5da289cd.

This reverts commit 3053f7e3.

Revert "As Martin pointed out, there is a race condition in the old code where a pipe could start shutting down after disconnection, but the new one could connect first. This connection would not get a pipe created for it, so the messages could never flow. The simplest way round this would be a flag, but it is possibly for a very bouncy but fast connection to go up and down twice I imagine, so instead I have added a counter. This starts at zero, and will null out the pipe if terminate is called while it is zero. On a disconnect situation the counter is incremented, and the pipe is the not nulled if the value is non zero. In the terminated function it is decremented for each pipe that is shut down, and the assertion that the terminated pipe == the current pipe is skipped while it is non-zero. This should deal with the race condition and not allow any extra terminated() calls without hitting the assertion."

This reverts commit a5f7300d.

Revert "Filter read and write activated calls from the pipe to the session, and delay shutdown in terminated until the final pipe is shutdown."

This reverts commit a90c1db7.

This reverts commit d8f3487e.

This reverts commit 029d3dfa.

This reverts commit dde69fb9.

This reverts commit 33459029, reversing
changes made to 889b0e6f.

This reverts commit 7b105865, reversing
changes made to 33459029.

This reverts commit 4aa5ba3d.
Unintentional merge

This commit removes the countdown flag and adds a set to store the pipes
that are currently being disconnected.

Fix race conditions in {tcp,ipc}_connecter

Once the object has been terminated, it is unsafe for this object
to refer to its parent.

The bug was responsible for occasional
test_shutdown_stress failures.

Add a new section to the connect delay test that disconnects and
reconnects a socket to ensure that some messages are appropriately
blocked.