CDC-EEM vs. CDC-ECM Protocols - Improvements in TCP Throughput

The CDC-EEM protocol is intended as an alternative to the CDC-ECM protocol for applications that do not require a Control Plane. The CDC-EEM protocol simplifies some of the requirements of the USB Configuration (no control plane) and allows for multiple network frames to be sent as part of a single USB Transfer. This study will provide some indication of the benefits of using the features of CDC-EEM to allow for multiple network frames to be coalesced into a single USB Transfer. 

NOTE:  the newer CDC-NCM protocol is another alternative to CDC-ECM that retains the Control Plane. The CDC-NCM encapsulation is different in structure from EEM but has similar capabilities to CDC-EEM for sending multiple datagrams per transfer. CDC-NCM throughput numbers will be very similar to CDC-EEM for most devices.

Overview

The USB implementations impose problems when trying to get maximum throughput for other protocols such as TCP. TCP requires low Round Trip Times (RTT) and high bandwidth to get good throughput. As originally designed TCP is very good at measuring effective throughput and only offering enough data to the network that will fit without frames being dropped. Modern networking stacks (e.g. current Linux or Windows or Mac OSX) implement various TCP extensions that allow this to happen even on very fast networks (i.e. > 100Mbit/s).

While the USB has a reasonably high speed (> 400Mbit/s) the effective speed is lower due to various constraints in the design. These have the effect of increasing latency which impacts RTT which tends to reduce effective throughput for TCP. The primary constraint is in how the network frames are encapsulated and sent across the USB. ECM has a simple strategy of one network frame for each transfer. EEM allows multiple network frames per transfer.

We show that using EEM’s ability for sending multiple frames in a single USB Transfer can increase throughput by 50%. Our best case uni-directional test measured about 11 Mbytes/s for ECM and up to 26.5 MBytes/s for EEM. Our best case bi-directional test measured about 8 Mbytes/s x 2 for ECM and 14.5 Mbytes/s x 2 for EEM.

Caveats

This is a test that should be close to the best case scenario.

• uses USB Data Cable to allow two USB Hosts to be used, these are not CPU or Memory bandwidth limited
• use of Data Cable does add a small amount of latency for data transfered across the bus (two transfers as well as data errors introduced by the Data Cable.)
• typical USB Peripherals may have limited CPU or Memory bandwidth which may impact overall throughput

Methodology
See draft-constantine-ippm-tcp-throughput-tm-03 for an overview of testing TCP throughput.

Our overall goal was to test end to end TCP throughput. Specifically the maximum rate that data can be sent from an application on one system to an application on another system using TCP where the two systems comprise the USB Host and USB Peripheral.

Our secondary goal was to observe the actual bandwidth consumed at the USB Bus level and try and understand how well the TCP layer is being serviced by the protocol implementation and the USB transport layer with respect to the goal if getting the maximal amount of data sent in the least amount of time.

Throughput Tests


Each system is setup with the following:

• iperf 
• Lecroy USB 2.0 Analyzer 

Two scripts are used to automate iperf.
Server:

while sleep 1; do

iperf -s -f M -i 2

done

Client:

while sleep 1; do

iperf -c 169.254.1.1 -t 60 -f M -i 2 # start a single transfer

iperf -c 169.254.1.1 -t 60 -f M -i 2 -d # start a bi-directional transfer

done

This alternates between a sixty second, single transfer test and a sixty second, two transfer, bi-directional test.

Three sets of tests were run:

1. ECM 
2. EEM - use EEM but with a maximum of one datagram per transfer 
3. EEM - use EEM with a maximum of six datagrams per transfer

Description	Test	Datagrams per Transfer	iperf TCP Throughput MBytes/s	USB Bandwidth MBytes/s	USB Bandwidth Percent (%)
ECM	Single	1	11	12	28
ECM	Bi-Directional	1	8x2	16	32
EEM	Single	1	10.5	12	28
EEM	Bi-Directional	1	9x2	18	35
EEM	Single	6	26.5	28	55
EEM	Bi-Directional	6	14.5x2	28	55

Table 1. Windows - Data Cable - Windows Tests

Figure 1.1: shows the USB Bandwidth used during the EEM 6 tests.

Figure 1.2: shows the throughput for the ECM tests

Two different test setup were used:

1. Two Windows 7 systems connteced with a USB Data Cable 
2. Windows 7 USB Host connted to an Overo Peripheral

The two Windows setup is useful as it allows to systems with fast CPU / Memory to set a benchmark. Normally two USB Hosts cannot be connected directly together. A USB Data Cable allows for that. It implements two BULK data points for each USB Host and simply forwards data packets received on the OUT endpoint on one side to the IN endpoint on the other. There is some additional latency because each USB packet has to traverse two buses. Thus the observed rates for this setup may be slightly lower than the theoretical limit. 

The Windows systems used for this test are:

• Windows 7/64
• IBM Lenovo ThinkCentre 
• Intel ® Pentium ® 4 CPU 3.00 GHZ 1.50 GHZ 
• Intel ® 82801G (ICH7 Family) USB2 Enhanced Host Controller 

Conclusions

The use of CDC-EEM allowed for approximately a 240% increase in throughput for a single uni-directional TCP stream and about 180% increase in throughput for two transfers in a bi-directional transfer as measured at the TCP level.

USB Bandwidth usage goes up by between 170% and 200% depending on the which test was used.