TCP IPERF test for maximum bandwidth: use -l and -P not -w ?!

There is always the question: how to test circuit for maximum bandwidth - with UDP or TCP traffic? They are very different, and IPERF implements them differently. As it was mentioned above for UDP, -l parameter (buffer size) can not be larger 64k - the maximum size of UDP packet. Well, for TCP it works differently. Running IPERF TCP test with default parameters between two computers which Gigabit cards are connected back to back, gave only 35Mbits/sec... Increasing -l parameter improved throughput significantly:

click for full-size

On the left we have remote desktop of Windows 7, running IPERF in server mode, on the right Windows XP with IPERF client. It looks like for this circuit the value 550Mbits/sec is close to the maximum. Maybe it's because TCP window is large: 2.5M? Trying without it:

click for full-size

Using TCP window either default 8Kbytes or 2.5Mbytes doesn't make a difference at all - with large IPERF buffer size. Again - these results are valid only for IPERF implementation on Windows. There is a well-known fact, that to get better perforance on decent circuits it's worth to increase TCP window. Maybe it's not the case with IPERF on Gigabit speed, where -l parameter makes real difference instead.
There is another well-known fact, that to get real circuit performance on TCP, someone should use multiple TCP sessions. Even when testing Internet privider circuit with FTP file upload, it's worth to try running multiple FTP sessions at once. IPERF provides similar functionality with -P parameter:

click for full-size

Apparently in this test, two TCP streams didn't push higher bandwidth, probably because circuit is already working to its maximum. At lower speed, multiple TCP streams really help to feel what the circuit is capable for.

Thus, with high-bandwidth TCP tests with IPERF, not only TCP window size but also buffer size and number of streams should be used.