GMN failure on large object transfer

[mbjones]

The following was reported by Ross for a failure trying to upload a large 5GB file to GMN using create() from the Java client. Here's Ross' report:

we encountered three "Read Timeout" errors (see past history) in the past two days - one of the archives was small and we were able to submit it again right away, and it worked the second time. The other two both contain a file around 5GB in size, so it took substantially longer (hours) for the member node to error out. Could this be related to production issues this week or otherwise?

SEVERE: unable to create resource map
org.dataone.service.exceptions.ServiceFailure: class org.dataone.client.exception.ClientSideException: /Read timed out
	at org.dataone.client.rest.HttpMultipartRestClient.doPostRequest(HttpMultipartRestClient.java:470)
	at org.dataone.client.v2.impl.MultipartMNode.create(MultipartMNode.java:495)
	at util.ArchiveSubmitter.createFile(ArchiveSubmitter.java:573)
	at util.ArchiveSubmitter.createFolderMap(ArchiveSubmitter.java:218)
	at util.ArchiveSubmitter.createBaseFolderMap(ArchiveSubmitter.java:149)
	at util.ArchiveSubmitter.submit(ArchiveSubmitter.java:99)
	at models.archive.ArchivePackage.submit(ArchivePackage.java:241)
	at controllers.Archive.submitArchive(Archive.java:242)
	at play.mvc.ActionInvoker.invokeWithContinuation(ActionInvoker.java:536)
	at play.mvc.ActionInvoker.invoke(ActionInvoker.java:473)
	at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:467)
	at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:436)
	at play.mvc.ActionInvoker.invoke(ActionInvoker.java:159)
	at Invocation.HTTP Request(Play!)
Caused by: org.dataone.client.exception.ClientSideException: /Read timed out
	... 14 more
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.client.cache.CachingExec.callBackend(CachingExec.java:578)
	at org.apache.http.impl.client.cache.CachingExec.execute(CachingExec.java:259)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.dataone.client.rest.RestClient.doRequest(RestClient.java:330)
	at org.dataone.client.rest.RestClient.doRequestMMBody(RestClient.java:306)
	at org.dataone.client.rest.RestClient.doPostRequest(RestClient.java:226)
	at org.dataone.client.rest.HttpMultipartRestClient.doPostRequest(HttpMultipartRestClient.java:464)
	... 13 more

timestamps in logs indicate the archive submission took hours to fail (at 4:18am) after being submitted (createFolderMap @ 7:47pm) to the member node

timestamps in logs indicate the archive submission took hours to fail (at 4:18am) after being submitted (createFolderMap @ 7:47pm) to the member node
May 14, 2020 7:47:59 PM play.Logger info
INFO: submitArchive: 6456544
May 14, 2020 7:47:59 PM play.Logger info
INFO: createFolderMap
May 15, 2020 4:18:44 AM play.Logger niceThrowable
SEVERE: unable to create resource map
org.dataone.service.exceptions.ServiceFailure: class org.dataone.client.exception.ClientSideException: /Read timed out
	at org.dataone.client.rest.HttpMultipartRestClient.doPostRequest(HttpMultipartRestClient.java:470)
	at org.dataone.client.v2.impl.MultipartMNode.create(MultipartMNode.java:495)
	at util.ArchiveSubmitter.createFile(ArchiveSubmitter.java:573)
	at util.ArchiveSubmitter.createFolderMap(ArchiveSubmitter.java:218)
	at util.ArchiveSubmitter.createBaseFolderMap(ArchiveSubmitter.java:149)

this occurs when MN.create is called using the Java API to GMN