HttpResponse.disconnect() broken for ApacheHttpTransport

[rawler]

When using Apache as the underlying transport, issuing response.getContent().close() blocks waiting for the entire response to finish. If the source is the content from a GCS blob, this can potentially be many GB, which defeats the intention of disconnect().

Calling HttpResponse.disconnect() before .getContent().close() falls on the problem that disconnect() starts by calling getContent().close() before calling LowLevelHttpHandler.disconnect(). The wrong order, compared to the intended way for ApacheHttpClient.

The net effect is that the application-code get's blocked until all the blob-content have been read and discarded.

[chanseokoh]

This is breaking us. When we call httpResponse.disconnect(), this makes the connection closed first. Not knowing that, ContentLengthInputStream tries to consume and discard the remaining bytes in close(), and it throws an exception as read() returns -1 (end of stream) while they believe there are remaining bytes to consume.

 Caused by: org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 2; received: 0)
 	at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178)
 	at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:198)
 	at org.apache.http.impl.io.ContentLengthInputStream.close(ContentLengthInputStream.java:101)
 	at org.apache.http.impl.execchain.ResponseEntityProxy.streamClosed(ResponseEntityProxy.java:142)
 	at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)
 	at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)
 	at com.google.api.client.http.HttpResponse.ignore(HttpResponse.java:427)
 	at com.google.api.client.http.HttpResponse.disconnect(HttpResponse.java:441)

[chanseokoh]

A user can consistently reproduce the above error just with

new ApacheHttpTransport().createRequestFactory()
    .reqFactory.buildGetRequest(new GenericUrl(url))
    .execute()
    .disconnect();

And if you look at the comments and thumbs-ups in the linked issue, there are a lot of users hitting this error.

I think until this error can be properly handled, we should revert the change in #1315.

[chanseokoh]

Oh, actually, I see #1409 already open.