Anything in the logs from the storefront server? Might be worth a packet capture and analysis to see which side of the conversation is initiating a time out?
I can understand this must be frustrating and I can only provide what works for me in my organization. It’s likely something that differs between our infrastructure configuration.
just thinking out loud here, doesn’t happen in your main central data centers but it does in the mills (branch offices I’m guessing). Are those “branch” location connected “internally” through a GRE tunnel? Any WAN acceleration between them? That can all add to packet overhead, we’ve had a few remote location going across a GRE tunnel where we’ve dropped the MTU size in WMS from the default of 1500 to 1340 and that corrected some connectivity issues for those locations.
Again, not the same situation and I don’t know your infrastructure but we found the issue through deep packet inspection and found oversized packets being fragmented and one side of the conversation would then stop. Lowering the MTU from default helped correct it. Also allowing UDP 1494 and 2598 has helped internally as well.