Internet protocols slim down for speed

Until now, technical development for 5G has concentrated on New Radio: getting more bits through the ether, more reliably, with lower latency, and using less power. With 3GPP Release 15 now in its final stages, attention is turning to other aspects, and ETSI ISG NGP (Industry Specification Group, Next Generation Protocols) is studying technology that will use those bits more efficiently while reducing latency and power consumption in the rest of the system.

Current packet routing in mobile networks uses the internet protocol (IP) suite, which dates from the 1970s and 80s when digital electronics technology was very different from today. Devices did not make the frequent changes to their point of attachment that are characteristic of cellular networks, leading to an addressing scheme that does not natively support mobility. Access to the network was sufficiently restricted to make security within the network no great issue, unlike today when denial-of-service and other attacks are frequent. Some of the features of IP were arguably wrong even back then. It was felt that, to reduce memory requirements, routers should not keep ‘state’ and communication should be connectionless. This made it like the postal service where every packet carries an address label with all the information needed to deliver it to its destination, rather than a connection-oriented structure, like the telephone system, where there are separate phases of setting up the connection, exchanging data, and disconnecting. While this avoids the need to keep per-flow state in the network, it requires packet headers to contain globally-significant addresses and switching-points to hold information about the routes to all possible addresses.

At about the same time, I was designing a connection-oriented system which kept per-flow state using less space than required for the kind of routing tables needed for connectionless systems. Address fields in packet headers, being only locally significant, were smaller and thus quicker for an eight-bit micro to process, and the messages that set connections up were able to use a variety of different ways to identify the remote device or process instance, including the name of any service it was offering.

PACKET ROUTING IN MOBILE NETWORKS USES THE INTERNET PROTOCOL SUITE, WHICH DATES FROM THE 1970S AND 80S

Note that connection-oriented packet networks are not the same technology as the circuit switching used in older telecommunications networks; they do not have to be fixed-bandwidth but can offer the same kind of bandwidth-on-demand service as other packet networks.

Another issue is that address fields in IP networks conflate different functions. The IP address acts both as a locator and as an identifier of the point of attachment. Port numbers have to be used to identify the syntax and semantics of a message as well as the process instance that will receive it.

When IP was new, logic had limited capability and most processing was done by code running on a CPU. Now even a small FPGA (Field Programmable Gate Array) contains thousands of gates, and functions such as packet forwarding can be done entirely by logic. Note that the term 'code' is used instead of 'software' because FPGA logic can also be thought of as software.

Processing by logic is very different to processing by code. There is more parallelism, with different components forming a production line in which each function performs a simple operation on the data before passing it on to the next. Processing of incoming packets is typically clocked at the rate at which data arrives from the interface, with dual-ported memories being used so that a process that writes packets to a buffer does not compete for memory bandwidth with the process that reads them.

In code, by contrast, a CPU can only execute one process at a time, with each process requiring multiple instructions, and all data passing over a single memory interface. This requires a much faster system clock in order to keep up with the data, which, in turn, means the power consumption will be higher. A significant amount of power is additionally consumed by the process of decoding the instructions that form the program. Multi-processor systems help to mitigate this but hardware packet processing can be many times faster.

Switches can now be divided into a control plane implemented in code, which makes decisions about how packets will be routed, and a data (or forwarding) plane implemented in logic, which copies each incoming packet to the appropriate output. The two processes are linked by a forwarding table written by the control plane. The data plane uses information from the header of each incoming packet to locate the table entry that shows what is to be done with the packet.

With a connection-oriented network there is a clear separation between the planes. Set-up and disconnection are control plane functions implemented using messages that can carry a wide variety of information for the network and for the remote entity. Multiple addressing schemes can be supported as well as information on traffic characteristics, data format, security, and charging. Forwarding table entries are created by the set-up process and erased on disconnection, and packet headers only need to contain a locally-significant reference to the table entry. All other information is in the table entry or, if not needed by the data plane, in the control plane’s memory.

In the connectionless case, the distinction is much less clear; routing decisions (in the control plane) can only use information that is in the packet header (in the data plane).

Although IP was originally intended to be purely connectionless, the way it is used looks increasingly connection-oriented; most packets being part of either a TCP session or an RTP stream. Some of the consequences of this can be quite absurd. An application using the Sockets API creates a socket and makes a 'connect' call to tell the operating system where it wants to send the data. To send a packet, it supplies the data and a 'handle' identifying the socket, with the operating system appending headers to it that include MAC addresses, IP addresses, and port numbers. The data plane in each switch must then search its forwarding table for a match with this information to, in effect, identify the socket. It would be so much simpler to use the connection-oriented paradigm, setting up the route when the call connect is made.

This article is from the CW Journal archive.

Connectionless networks provide a 'best-effort' service. In switches, packets are held in queues until their output is free and packets will be lost if the space allocated to a queue fills up. This kind of service is appropriate to communication between computing processes, which in NGP is called IT (Information Technology) traffic. An application can transmit whenever it has data to send, and protocols such as TCP are designed to cope with the packet loss and variable delays which are a feature of a best-effort service, as well as the duplication and misordering which can occur in connectionless networks.

Much of the current traffic on networks consists of continuous media, such as audio and video (AV) traffic. For applications involving a conversation, or any kind of control or feedback loop, latency is important. If latency reaches above a certain level the service is unusable, and there can even be noticeable effects at much lower levels.

An important feature of AV traffic is that packets are sent at regular intervals. For instance, in the case of an uncompressed stereo audio signal there might be a 48-byte packet every 125 microseconds. Resources can be reserved along the path packets will follow to ensure queues do not get too large, thus limiting the latency and eliminating packet loss. A switch can have separate queues for AV traffic to give it priority over the IT traffic, in the same way that bus lanes give buses priority over other traffic.

Some further functionality is needed to ensure the system works properly: 'traffic shaping' to stop the packets becoming bunched together, and 'policing' to ensure a flow does not exceed the amount of data for which a reservation has been made. This is called an asynchronous AV service because the various media flows are not aligned with each other, so packets for several different flows may arrive simultaneously.

To eliminate queues, a synchronous AV service is required, in which each flow is allocated specific timeslots in which to transmit. This provides lower latency than can be achieved with an asynchronous service and also makes policing and traffic shaping unnecessary.

The system proposed in ISG NGP provides two separate services within a single connection-oriented framework: a best-effort service for IT traffic, and a synchronous service for AV flows. Each service is simple to implement, and between them they provide all the necessary facilities to meet the requirements listed previously. Wired links are formatted to carry the AV service according to a schedule, with all bytes not occupied by AV packets being used to carry the IT service. This has been prototyped using 1Gb/s twisted-pair Ethernet physical layer.

Addresses only appear in the control plane set-up messages where they can take a wide variety of forms, and flows can easily be rerouted when a device changes its point of attachment. The set-up process is able to perform any security checks required which is done before any packets can be forwarded. Control plane messages include a globally-unique flow identifier; detailed procedures will be developed after the basic architecture is finalised.

IT packet payloads can be from one to 2,000 bytes. The header format on each link is specific to the physical layer technology; this allows new technologies to be introduced without requiring the kind of system-wide upheaval that was needed for the change from IPv4 to IPv6. Typically the header simply contains the payload length and a label which is an index into the data plane forwarding table. Thus a header is of minimal size and the forwarding logic has minimal complexity. The prototype implementation uses 13 bits plus a three-bit CRC for the label but a larger field could be used if forwarding tables have more than 8,192 entries, or different data protection could be used (or none, if bit errors on the link always result in an invalid symbol). Flows can be aggregated by adding another label, in the same way as an MPLS (MultiProtocol Label Switching) push operation.

AV packets are very different. They were originally planned to have a similar size range to IT packets, to carry anything from a single audio sample to a sizeable piece of video data, with time slots of similarly variable size allocated for their transmission. This has been found to be impractical and slots are now proposed to be 64 bytes, with AV packets having a single-byte header (including a flag which can be used to mark the end of a message) and zero to 63 bytes payload. This is superficially similar to an Asynchronous Transfer Mode (ATM) cell but, unlike ATM, if a packet does not fill the slot the remaining bytes are available to the IT service. Note that an MPEG-2 Transport Stream packet fits into three AV packets with one byte to spare.

The way that AV packets are routed is also different. In the prototype implementation, links are formatted into subframes containing 30 or 31 slots, which at 1Gb/s is about 15 microseconds worth of data. In a switch there is a buffer memory which can hold one subframe from each port and which is separate from the memory that holds the buffers for IT packets. The packet in each incoming slot is written to this buffer, where it stays until overwritten by the packet in the equivalent location in the following subframe. This means there is no routing table at the input, and no label is needed in the packet header. Indeed, it is not necessary to read any part of an AV packet in order to forward it and this may allow AV flows to be switched in the optical domain.

There is a very simple mechanism that keeps a constant phase relationship between subframes on any two links; it does not require clocks to be synchronised, and the subframes are not related to any other references, such as media clocks. For each output there is a routing table holding a schedule which shows from where in the buffer to take the packet to transmit in each slot.

In the prototype implementation this repeats every millisecond. To route a flow, the control plane chooses an outgoing slot which occurs during the 15 microseconds in which the required packet is in the buffer, and sets it to read from the appropriate location. Note that a flow can be multicast simply by routing it to more than one output.

The AV service may also be a better way to implement large file transfers. Much of the functionality of TCP is to match the transmission rate to the network and the recipient, and to repeat dropped packets. With the AV service, the data rate can be negotiated up-front and packets will not be dropped. Instead of frequent acknowledgements, all that is needed will be a report at the end of the transfer that can be a control plane message which, in the case of success, is combined with the one that terminates the connection. A similar mechanism could be used to multicast a software update to a large number of users.

In 2018 the ISG NGP needs to firm up on recommendations for 3GPP release 17. The specification of the proposed technology needs to be refined and it needs to be tested in a 5G system. The ETSI group's homepage is at http://www.etsi.org/technologies-clusters/technologies/next-generation-protocols. As well as the formal group in ETSI, there is an informal group in UK5G that is open to anyone who is interested. To ask to join it, please send an e-mail to: j@ninetiles.com.

Of course, neither ETSI nor 3GPP is responsible for internet standards, and the new system will (at least initially) be restricted to the core and access networks, tunnelling internet traffic to the Packet Gateway in the same way that GPRS Tunnelling Protocol (GTP) does now. However, it is expected that much of the traffic that currently goes to the internet will instead go to Mobile Edge Computing (MEC) servers in the operator's network. The new technology can use the same APIs as current systems with extensions to support the new facilities, especially if the new API being developed by the IETF TAPS working-group is adopted.