Question: Why does an NFS mount using the sync option cause nginx and/or Chrome (and only Chrome; not Safari, Firefox, Edge or curl) to chunk or restart the file upload?
Some background:
I have nginx in front of a server cluster. We added file uploading and we're using nginx to handle the upload and then pass the uploaded file body via its X-FILE header.
Until recently we were only processing uploads on the nginx server, due to the issues with other application servers in the cluster being unable to access the client_body_temp_path (/tmp in this case).
We run on AWS, so I provisioned an EFS (aka NFS) share and mounted this at /mnt/efs and then changed nginx client_body_temp_path to point at /mnt/efs/tmp.
I mounted the NFS share on all the application servers and they can now access the file uploads on /mnt/efs/tmp, but the uploaded files are often zero-length when they're initially read by the other clients, due to the NFS asynchronous writes.
I changed the NFS mount just on the nginx server so it's mounted with sync:
Initial mount:
example.eu-west-1.amazonaws.com:/ /mnt/efs nfs4 vers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,_netdev,nofail
New mount:
example.eu-west-1.amazonaws.com:/ /mnt/efs nfs4 vers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,_netdev,nofail,sync
I remounted the share and tried some uploads and happy days, the other NFS clients see the non-empty files in /mnt/efs/tmp.
But with some more testing it turns out that uploaded files over about 10M get restarted or chunked by Chrome.
What happens:
nginx starts to receive the upload (in the video below, I'm sending a 32.7MB file), but at 6.4MB, a new upload starts and then eventually nginx decides that the client's stopped sending, so issues a 499 which causes Chrome to abort the upload and it all fails. You can see all this happening here, as I watch the files coming in to /mnt/efs/tmp:
http://jay.gooby.org/media/video/chrome%20failing.mp4
The upload initially comes in as 0000000006 but then when it gets to 6.4M, a second upload 0000000007 starts (but 0000000006 continues to 12M), and then when 0000000007 hits 6.4, a third, 0000000008 starts (0000000007 continues to 12M). Once 0000000008 hits 12M ngingx issues the 499 and Chrome aborts the upload.
Other useragents don't have this problem. I'm able to upload the same file with Safari, Firefox and curl. Chrome on Mac and Windows both have the same issue.
Here's curl sending the file with no problems: it arrives as 0000000010 single 32M file:
http://jay.gooby.org/media/video/curl%20working.mp4
After tearing my hair out for a bit, I removed the sync option from the NFS mount on the nginx server and lo and behold, files uploaded with Chrome arrive in one piece as a single file. So what gives?
nginx is behind an Elastic IP, but not an ELB, and the upload is over http2.
nginx is compiled with --with-threads option
Update 1: That 6.4mb value seemed significant, as that's the point that each new upload chunk began/restarted. 6.4mb is 51200000 bits which seems like a quite specific value...
The initial bytes of the three files are identical, so it's as if Chrome decided to start resending at the 6.4mb point:
0 Answers