Unpredictable abortive shutdown on client side

2015-04-30T17:11:33

I'm working with a fairly basic server/client setup, where both are located on the same network. They communicate via Winsock2 blocking sockets over TCP/IP, and are doing so perfectly fine.

However, for the scenario described below, the client sometimes sees an abortive connection termination (RST). It goes right roughly 99 out of 100 times, but that last time annoyingly fails some tests and therefore, my whole build. It is completely unpredictable when and where it happens, and so reproducing the problem has so far eluded me.

If I understand the the relevant MSDN page correctly, the nominal connection termination sequence for blocking sockets can be summarized as:

Client               |  Server
-----------------------------
shutdown(SD_SEND)    |  
                     |  send() response data
i=recv() until i==0  |  shutdown(SD_SEND) 
closesocket()        |  closesocket() 

In my setup it is necessary to

  1. do a relatively expensive operation (let's call it expensive_operation()) depending on whether a portion of the received data (let's say, 512 bytes) contains a trigger value. The server is single-threaded, so expensive_operation() effectively stops recv()ing the data stream until expensive_operation() is complete
  2. initiate a server shutdown sequence if the client sends a particular sentinel value, let's call it 0xDEADBEEF.

My client is implemented such that the sentinel value is always sent last, so after sending it, no other data is sent:

  1. send( "data data data 0xDEADBEEF" ) to server
  2. shutdown(SD_SEND) <------- FAILURE OCCURS HERE
  3. recv() until 0 bytes received
  4. closesocket()

Whenever the server receives 0xDEADBEEF, it confirms the shutdown request and continues termination:

  1. recv() 512 bytes of data or until 0 bytes are returned
  2. Check for trigger. If a trigger is found, perform expensive_operation() and go back to step 1, otherwise continue
  3. Check for sentinel value. If sentinel is not found, go back to step 1.
  4. If the sentinel is found:
    1. send( confirmation ) to client
    2. shutdown(SD_SEND)
    3. closesocket()
    4. all the normal server shutdown stuff

I can understand that if the client intends to send more data after the sentinel, this will result in abortive connection termination -- because the server actively terminates the connection. This is completely expected and by design, and I can indeed reliably reproduce this behavior.

However, in the nominal case, the sentinel is always last in the sequence, which indeed always happens as evidenced by the relevant log entries, and indeed graceful connection termination happens as expected most of the time. But not always...

As I said, it happens randomly and sporadically, so I can't produce a code snippet that reliably reproduces the problem. The only thing that's consistent is that the failure always occurs when calling shutdown() in the client...

I suspect it's more of a design flaw, or some synchronization issue I'm not handling yet, rather than a problem with the code (although I'd be happy to provide the relevant code snippets).

So is there anything obvious I'm overlooking here?

Copyright License:
Author:「Rody Oldenhuis」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/29963619/unpredictable-abortive-shutdown-on-client-side

About “Unpredictable abortive shutdown on client side” questions

I'm working with a fairly basic server/client setup, where both are located on the same network. They communicate via Winsock2 blocking sockets over TCP/IP, and are doing so perfectly fine. Howev...
I am getting SSL handshake failures randomly when number of concurrent users increases. Though the number is not that big, with only 100 users or so SSL handshake keeps failing randomly. I have set...
when a tcp client establishes a tcp connection with my TCP server, after it sends several packets, I want to make a hard/abortive close on this TCP connection, how to do it in linux C? the hard/ab...
I was writing lots of code in Eclipse when suddenly unpredictable shutdown has occurred. Now when I open Eclipse I have only the window in workspace and very long one line (crossed red line). I've...
When working with the Lync SDK, starting up a client side-by-side, and then shutting it down -it leaves an orphaned host process that never quits. I have to manually kill the process by either code...
How do i remotely shutdown client/s system from server end ? Also, how do i run any command on client/s command prompt ? How do i send the line of codes to the client end from the server side ? I'v...
The client does some ssl::stream&lt;tcp_socket&gt;::async_read_some()/ssl::stream&lt;tcp_socket&gt;::async_write() calls and at some point needs to exit, i.e. it needs to shutdown the connection.
I am working on a .net rich client server application, the client call the server thru web service, what could happen if the client is suddently shutdown(connection closed) when it is calling the s...
Using non-blocking (20 ms cycle) TCP connection in linux, I got a problem: when I close socket from the server side [close(sd) or shutdown(sd,2);close(sd)], the client poll() receives no POLLHUP e...
I'm building a WCF service,after a client connects to this service,when the service disconnect, the client doesn't notice that, and it doesn't fire any action, i would like to close client form whe...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.