Economics of Computer Network Overheads

A 1 GB file downloaded from the Internet will have around ~ 53 MB of additional (invisible) data

This data is not part of the actual file, nor we can see it anywhere once the download is completed.

Simply put, this additional data is called communication overhead, an essential component which contains all control information required for the file to travel the vast computer networks.

So what's the concern? Especially given the fast Internet connections and unlimited data plans we have?

Well, what if the download file size is 10 GB? Now the new overhead unavoidably increases to not less than ~534 MB.

Let's push it further, for a 100 GB download, the overhead data goes as high as ~5.3 GB and it gets large for large files/data downloads.

This additional overhead data is not a concern for small network infrastructures, but reducing even 1% of the overhead becomes crucial for large scale enterprise networks, especially within Data Center and DC2DC scenarios.

 So what exactly is this 'overhead data' and are there any ways to reduce them?

Grab your coffee and buckle up for a lengthy article.

Before we see the math, it's important to discus basics.

Internet Protocol (v4)

Communications in Computer Networks and the Internet happens using Internet Protocol. IP, along with its payload (user data) is called an IP Packet, and this study deals with only IPv4.

IP Packets are self-contained units that can travel networks of any scale independently to reach their destinations. A typical IP Packet (without payload) is illustrated below.

Minimum IP Header size is 20 Bytes (without IP Options), this includes the first 5 rows above (32 bit each) and these 5 are must to have on any IP Packet, the obvious ones most of us know are the Source and Destination IP Addresses (Explanation of each field).

To put it in perspective, when we open a website from a browser, the user request from the browser becomes the Payload, sort of like the contents enclosed within a traditional mail.

Where that payload has to go and other control information becomes part of the Header, like the From and To address we write on the mail envelop.

Transmission Control Protocol

Internet Protocol is a connection less, best effort, unreliable communication protocol, meaning, it doesn't have ANY mechanism to tell us if the data is delivered or not.

To fix the problem of reliability, another protocol is used along with IP, and its name is, you guessed it, TCP.

Like in IP, TCP too has its own header section, illustrated below.

Minimum TCP Header size is 20 Bytes (without options), like in IP, first 5 rows in TCP Header as illustrated are a must for any TCP based communications, options may be included but we are ignoring them here.

TCP takes care of end-to-end reliability for a given communication, in the TCP/IP - 5 Layer Model (called as TCP/IP Model from now on) or the deprecated OSI Model it sits on top of Network Layer (IP), meaning, when we open a website from a browser, the request before going to IP, comes down to TCP.

Browsing websites happen using HTTP/HTTPS protocols, these protocols are designed to use TCP for reliability at the Transport Layer.

TCP encapsulate the data from HTTP/HTTPS with its header information and goes down as payload to IP at Network Layer.

IP encapsulates this payload (TCP Payload + TCP Header) and builds the IP Packet with its own header information and goes down to Data Link Layer as a payload.

Ethernet

Ethernet is a Media Access Control protocol, together with Logic Link Control 802.2, it is part of Data Link Layer in the TCP/IP Model. There are several versions of Ethernet, of which, Ethernet II is popular and is the industry standard.

Ethernet encapsulates the Network Layer IP Packet with its own Header and a CRC trailer, illustrated below.

Preamble + SFD (Start Frame Delimiter) are not part of the actual Ethernet Header, they are added to signal the start of an Ethernet Frame, end of the Frame is signaled by the underlying encoding format, usually the extra two bits in the 10 bit code of 8B/10B encoding scheme used in Ethernet.

Encapsulation

Whatever we saw till now is the process of encapsulation, at each layer of the TCP/IP Model, end user's request gets encapsulated with the respective protocol headers as it goes down.

Data, before entering a Layer, is called Service Data Unit (SDU), once encapsulated with the protocol headers, it is called as Protocol Data Unit (PDU).

PDUs at each layer have names depending on the protocol selected by the upper layer.

The block sizes are not fixed, they are relative to the user data and protocol headers, the fixed size is just to illustrate.

As we can see, there's additional data that's getting added to the actual user data before transmitting as 0's and 1's at the Physical Layer.

At the receiving end, the data goes through the process of decapsulation (opposite of encapsulation), where at each layer, it's respective header info is stripped and sent to proper layer above for processing, before reaching the end user application.

Maximum Transmission Unit

This parameter dictates the maximum PDU size that can be transmitted within a single Network Layer transaction.

In simple terms, MTU tells the maximum size on an IP Packet to encapsulate within an Data Link Layer Frame.

MTU logic (RFC 791):

• If: The TL (Total Length) field in the IP Header is less than or equal to MTU size, then the IP Packet is sent to Data Link Layer protocol for encapsulation.
• Else If: Discard the IP Packet if TL is greater than MTU and DF (Don't Fragment) bit is set to 1 in the IP Header.
• Else: Fragment the IP Packet into multiple blocks where each block is not greater than MTU.

The imposition of MTU has to do with improving efficiency, back then, bandwidths were low and sending large IP Packets over low bandwidth links meant huge delays, congestion and resulting errors. Also, memory buffers in Network Interface Cards were very expensive!

To address these concerns, size limit is imposed on the size of IP Packets, this varies for several Data Link Layer protocols, for Ethernet, MTU is fixed at 1500 Bytes.

Ethernet MTU can vary in present day implementations, like Jumbo Frames where it is set at 9000 Bytes. Though Jumbo Frames are not part of IEEE 802.3 standards, it is used as a vendor feature on supported hardware largely due to the availability of higher bandwidths and less expensive memory. Maybe ... In future Jumbos becomes a standard.

OK, done with the basics.

Overhead Math

Assumptions:

1. Options in both TCP and IP Headers are ignored, Only the fields other than Options are considered, so, 20 Bytes is the minimum size for TCP or IP Header.
2. Ethernet is considered at the Data Link Layer where the MTU is set to 1500 Bytes, meaning, 1460 is the maximum user data that an Ethernet Frame can carry, 1460 Bytes because, we need to add TCP and IP overheads which would sum up to 1500 Bytes (1460 Bytes user data + 20 Bytes IP header + 20 Bytes TCP header).
3. Even though UDP is integral part of network communications, only TCP is assumed here. UDP overheads are negligible after all.
4. When you see 1 GB, it means 1,000,000,000 Bytes, not 1,073,741,824 Bytes. SI Prefix (decimal) is used for file sizes, unlike in Microsoft Windows file system and Linux's "df" command where Binary Prefix is used.
5. SI Prefix is used for measuring network throughput too, this is in-fact true in real world. 1 Gbit/s is 1,000,000,000 bits per second. For example, on perfect ideal conditions(no latency), an Ethernet medium(1500 MTU) is expected to take ~8.5 seconds to download 1 GB file over a 1 Gbit/s link. Not 1 sec.
6. Packets losses, handshakes, duplicates, congestion, latency and retires are ignored, ideal conditions are assumed for calculations.
7. Interframe/Interpacket Gap is considered in the calculations. This is the interval/pause time between two Ethernet Packets on the Physical Medium, it's a brief recovery time between two Ethernet Packets which allows the receiver to prepare(clock recovery) for the next Ethernet Packet.

The standard minimum Interframe/Interpacket gap for transmission is 96 bit times, meaning, the time it takes to transmit 96 bits(12 Bytes) of data on the medium.

Calculations:

For 1 GB Data:

• With 1460 Bytes as maximum data payload, 1 GB is divided into 684,932 blocks (Without TCP and IP headers, refer point 2 in assumptions).
• At Transport Layer, 20 Bytes overhead is added by TCP for each data block.
• Down at Network Layer, 20 Bytes overhead is added by IP for each TCP Segment.
• Once at Data Link Layer, Ethernet would add its own overhead of 26 Bytes to each Network Layer PDU (nothing but IP Packet).
• When on wire, an additional 12 Bytes are consumed for each Ethernet Packet for Interframe / Interpacket Gap.

• This sums to a total overhead of 78 Bytes for each Ethernet Packet carrying a maximum user payload (block) of 1460 Bytes.

What this tells us?

• That there's an overhead of ~5.3% for 1 GB Data.
• ~95% efficiency due to overheads.
• And 0.95 Gbit/s of practical throughput (in ideal conditions) on 1 Gbit/s interface.

Formulas

That's the additional ~53 MB we saw at the beginning of the article. 684,932 blocks (maximum 1460 Bytes each) multiplied by 78 Bytes of overhead.

For 1 TB Data:

• Is divided into 684,931,507 blocks (Without TCP and IP headers, point 2 in assumptions).
• Adding 78 Bytes of overhead to each block gives a total overhead of ~53.4 GB!

However, the overhead and the efficiency percentages remain the same, as-long-as given MTU size is fully utilized. What happens when the actual user data is less than the MTU?

For 1 KB Data:

• Is divided? No, usable data size in 1500 MTU is 1460 Bytes, and this is greater than 1 KB so this can be stuffed in to a single packet.
• Adding 78 Bytes to a single packet of 1 KB payload would give an overhead of 7.8%, which is slightly more than the ~5.3% we saw in our previous cases, higher overhead percentage is due to the under utilization of MTU.

What if the application just want to send 1 Byte over the network? Yes, some applications do.

For 1 Byte Data:

• Can be fit into a single packet? Yes of-course.
• There is no way 78 Bytes of overhead can be avoided, and here, it is way more than the actual data being transmitted leading to an overhead of 7,800%!

Understanding these overheads is essential in designing network applications. There are some applications like the Telnet which uses one Packet for every character typed (read here). Network overheads would be significant for small data payloads that are less than the prescribed MTU.

How to Reduce Overheads?

Obvious solution is to increase the MTU value. This is already done with Jumbo Frames where the Ethernet MTU is set at 9000 Bytes.

How much difference does it make? Let's see.

For 1 GB Data:

• Maximum user payload per Ethernet Packet would be 8960 Bytes with 9000 Bytes as MTU (without TCP and IP Headers).
• With that, 1 GB is divided into 111,608 blocks.
• Adding 78 Bytes overhead (TCP+IP+Ethernet+IPG/IFG) would give ~8.7 MB of total overhead, that's just 0.87% from the ~5.3% we saw with 1500 Bytes MTU.
• And that would give us an efficiency of ~99.1%. Significant improvement!

Good enough reason to tell your boss to start implementing Jumbo Frames in your infrastructure and get a good salary hike, right? Wrong!

Reasons:

Higher MTU values are proved to provide better overall efficiency and reduced CPU cycles for network processing. However, they can degrade the performance of low bandwidth networks. Large Ethernet Packets traveling on low bandwidth networks would hold the interfaces from transmitting/receiving other Ethernet Packets until finished.

Example: A 9038 Byte Ethernet Packet traveling on a 56 kbit/s link would hold the interface for 1.29 seconds! No other communication can happen until the transmission is finished. Not a problem anymore because, these days, we get minimum 100 Mbit/s Ethernet ports even on a cheap home Wi-Fi router.

If network throughput is not a problem anymore then why Jumbo Frame didn't replace the default 1500 Byte MTU for Ethernet?

Because IEEE says so, so does the whole Internet.

Part of Ethernet's success has to do with its backward compatibility, meaning, we can interconnect Gigabit Ethernet(1/10/100 Gbit/s), Fast Ethernet(100 Mbit/s) and Ethernet(10 Mbit/s) switch ports and they work just fine, considering the standard 1500 MTU is used.

When an L2 switch with Ethernet port receives a Jumbo Frame, and if its interface is configured at standard 1500 MTU, it'll promptly drops it.

To fix this, we'll need to configure its interfaces to accept Jumbo Frames, unfortunately, Fast Ethernet and anything below that MAY NOT support Jumbo Frames, however, this has changed in recent years with vendors enabling support for Jumbos.

Configuring just a switch port is not going to increase the efficiency, we'll have to ensure entire data path is configured to accept Jumbo Frames, if not, intermediate nodes would drop them. Also, ISP networks, at-least the customer bound ones work on Ethernet's standard 1500 Bytes MTU, enabling Jumbo Frames towards ISP bound traffic may cause problems.

Honestly speaking, 1500 MTU is the reason why Internet works.

Most networks, at-least the ones I've seen uses Jumbo Frames for specific scenarios, especially Data Center NAS,SAN environments, storage vendors recommend using Jumbo Frames for higher efficiency and these environments should ensure MTU consistency is maintained across the network.

Planning is everything, that goes without saying right?

By now if you haven't wondered about Jumbo Frame's 9000 Bytes size then you should. Why is it fixed at 9000 Bytes? Why not pump it up to the IPv4's theoretical limit of 64 KB?

The Cyclic Redundancy Check algorithm used for calculating the Ethernet Frame checksum has a limit of up to 12,000 Bytes, anything above that would be error prone.

I guess, 9000 Bytes was chosen for maximum payload and also to accommodate overheads without breaching the limit. This may not be true with every vendor, some vendors have different values for Jumbo Frames, this is expected to happen until IEEE makes it a standard.

Summary

The case study showed the network overheads and efficiency percentages for Wired Ethernet medium under ideal conditions. We saw, increasing MTU value can benefit the overall network performance with Jumbo Frames as an example.

Since the calculations were done with assumptions in place, we didn't consider several other overheads, like the ones added by the user Application, VLAN Tags on Ethernet Frames which are most common in networks.

If we would've considered a real-world case, the final overhead would've been more than our theoretical ~5.3%. In case of IPSec, the overall efficiency would go less than 90% (read here).

Jumbo Frames can be used to decrease the overheads and increase the overall efficiency of the network, but without proper planning, it may lead to a network wide outage.

Thank you for reading, feel free to highlight inconsistencies/mistakes.