Your configuration should be correct.
The problem, strangely enough, seems to be mod_deflate.
After having reproduced your configuration successfully here (not getting a 404), I also got the 5 second delay. However, I noticed that when the UA omits gzip from its Accept-Headers, the page is displayed/received instantaneously. You can test this for yourself with wget.
Interestingly, further debugging with strace shows that apache sends the contents of your FallbackResource to the socket of your client without noticeable difference in delay for both cases. This is also evident on the wire, where a reply paket is sent from the server to the client after the HTTP Request without any notable delay1.
It seems that when using mod_deflate in this case however, the UA doesn't know when the data sent by the server ends, and thus does not render anything before the TCP connection times out2 and is forcibly closed by the server. This is compliant to HTTP/1.0, where a closed connection signifies end-of-content.
For HTTP/1.1, the server has other means available to signal the end-of-content–but none of which seem to happen here.
Wether the bug lurks in mod_dir or mod_deflate however, is beyond my available time right now. I got it to work flawlessy by disabling gzip compression; as a workaround until the issue is fixed for good, you can selectively disable gzip.
1) This tells us that the problem does not stem from unflushed buffers on the server.
2) By default, the timeout is 5 seconds with apache - that's where your 5 seconds are from.
