nginx stuck and setting CLOSE_WAIT when using worker_processes > 1

Everything works nicely if I have worker_processes=1, or I run one request at a time. If I open a browser on a fresh nginx install and holds F5 down a couple of seconds, the sockets gets lingering as CLOSE_WAIT, and nginx wont do anything (other than stuffing more sockets in CLOSE_WAIT..)

The more workers I use, the sooner it gets stuck. Just one of the workers has listed any sockets though..

Any idea what might be causing this?