Sufficient set of fields required for IPFIX exporter implementation
There are multiple traffic telemetry protocols available from vendors and each of them has their own cons and pros. In this article we will focus on the most advanced one called IPFIX. It's a standardized vendor neutral protocol which offers ways to export almost all possible characteristics and meta information about network traffic.
RFC uses term IPFIX exporter for device or software which processes or forwards traffic and then sends summarised information about it to collector which receives it and provides human friendly overview of traffic. Alternatively IPFIX exporter may be called IPFIX agent or IPFIX probe. These terms have exactly same meaning.
At FastNetMon LTD we work with traffic telemetry protocols on a daily basis and our DDoS detection product supports all widely used traffic telemetry protocols and was battle tested with implementations of these protocols from more then dozen vendors.
Usually, we work with protocols vendors have to offer as we need to deploy our product rapidly in response to DDoS attacks against customer infrastructure and we're clearly not in position to make alterations in network design and topology until DDoS threat is remediated.
Effective DDoS mitigation significantly depends on quality of network telemetry protocols available for particular network and that's a reason why we allocate significant amount of time to investigate and evaluate implementations from different vendors with main focus to address issues and improve existing implementations.
Recently we got opportunity to work on exciting project of early stage telemetry implementation for one vendor. It may be very unique implementation as normally IPFIX collector developers have very weak influence on network vendors. Sadly such lack of industry connections negatively affects quality of network visibility in general.
The IPFIX protocol has more than 500 documented entities and it’s very challenging from the IPFIX exporter developer side to decide which fields are the most important and must be in the IPFIX packet.
When you see list of available fields you got feeling that it's like trying to build something from Lego without having any idea what you're trying to assemble
I would would like to share our opinion about the fields which must be in the IPFIX packet for regular IPv4 and IPv6 traffic. We will not cover cases of MPLS and VXLAN traffic in this overview. I'll use entity names which will match with the field “name” in this document from IANA.
To explicitly distinguish IPv4 and IPv6 traffic on collector and to use space in IPFIX template the most efficient way we recommend using two different templates for exporting information about IPv4 and IPv6 traffic.
One of the templates will carry source IPv4 (sourceIPv4Address, #8) and destination (destinationIPv4Address, #12) IPv4 addresses and another one will carry source IPv6 (sourceIPv6Address, #27) and destination (destinationIPv6Address, #28) IPv6 addresses.
To provide reliable way to distinguish them we recommend adding field ipVersion, #60 which will help the collector to get the version of protocol.
| IPv4 template | Field length | IPv6 template | Field length |
|---|---|---|---|
| ipVersion, #60 | 1 byte | ipVersion, #60 | 1 byte |
| sourceIPv4Address, #8 | 4 byte | sourceIPv6Address, #27 | 16 bytes |
| destinationIPv4Address, #12 | 4 byte | destinationIPv6Address, #28 | 16 bytes |
If you cannot use two templates for some reasons you may put both IPv4 and IPv6 addresses to same template and use field ipVersion, #60 to simplify protocol version selection for collector. Please ensure that addresses for not used protocol are set to zero for IPv4 and all zeros for IPv6.
| IPv4 + IPv6 template | Field length |
|---|---|
| ipVersion, #60 | 1 byte |
| sourceIPv4Address, #8 | 4 byte |
| destinationIPv4Address, #12 | 4 byte |
| sourceIPv6Address, #27 | 16 bytes |
| destinationIPv6Address, #28 | 16 bytes |
We do not cover cases when any prefix level aggregation is taking place and assume that all IPv4 traffic is exported for /32 and all IPv6 traffic is exported for /128.
Such list of fields does not provide enough information for the collector to understand what's going on in network. To make it more useful we need to add the number of received bytes and packets. To export number of bytes or octets we can use field octetDeltaCount, #1 and packetDeltaCount, #2. Both counters are 8 byte long.
As next step we need to add visibility about protocol which was observed by network device and we can use 1 byte long protocolIdentifier, #4 for both IPv4 (protocol) and IPv6 (next header) packets.
For TCP and UDP protocols we have source and destination packet numbers and this information may be very useful too. We can export it using sourceTransportPort, #8 and destinationTransportPort, #11 2 byte fields.
One may argue that we need to add ICMP protocol codes too but that’s how things are getting more complicated and pretty different in case of IPv4 and IPv6 and I prefer to keep it out of scope for this article.
To export TCP flags we can use tcpControlBits, #6 which may be 1 or 2 byte long. What is the best way? I think the less error prone way is to just use 2 byte format which completely matches the same section in TCP header and includes data offset, reserved and all regular TCP flags.
Let's look on our current templates
| IPv4 template | Field length | IPv6 template | Field length |
|---|---|---|---|
| ipVersion, #60 | 1 byte | ipVersion, #60 | 1 byte |
| sourceIPv4Address, #8 | 4 byte | sourceIPv6Address, #27 | 16 bytes |
| destinationIPv4Address, #12 | 4 byte | destinationIPv6Address, #28 | 16 bytes |
| octetDeltaCount, #1 | 8 bytes | octetDeltaCount, #1 | 8 bytes |
| packetDeltaCount, #2 | 8 bytes | packetDeltaCount, #2 | 8 bytes |
| protocolIdentifier, #4 | 1 byte | protocolIdentifier, #4 | 1 byte |
| sourceTransportPort, #8 | 2 byte | sourceTransportPort, #8 | 2 byte |
| destinationTransportPort, #11 | 2 byte | destinationTransportPort, #11 | 2 byte |
| tcpControlBits, #6 | 2 byte | tcpControlBits, #6 | 2 byte |
These templates provide enough information to offer decent traffic visibility on the collector side and if you’re limited on time then you may stop here. If you’re looking to provide the best experience for customers please continue reading.
On almost all network devices we have input / ingress and output / egress interface numbers and we can export this information using ingressInterface, #10 and egressInterface, #14. Both fields are 4 byte long.
On routers you may have access to source and destination ASNs and these fields can be exported using bgpSourceAsNumber, #16 and bgpDestinationAsNumber, #17. Both fields are 4 byte to accommodate 32 bit ASN numbers. Network engineers love peering reports and you will make them happy by providing this information.
Another very important field is flow direction which may be ingress or egress. It’s exceptionally useful field from collector perspective and if you have this information on device, you can export it using 1 byte field flowDirection, #61.
The IPFIX flow tracking process is very complicated on its own and it’s crucial to have some visibility that it works fine. To accomplish it we need to provide flow duration to the collector. Such information will help it to identify issues when flow aggregating does not work as expected or flow timeouts are not configured correctly. Some collectors may rely on this information in their bandwidth calculation and approximation algorithms.
IPFIX has options to export flow duration and alternative option to export time when flow was observed first and when it was observed last. In real world scenarios we haven’t seen any use of duration field by vendors and majority of vendors in industry use flow start and flow end time: flowStartMilliseconds, #152 and flowEndMilliseconds, #153 encoded as 8 byte integers in milliseconds since 1970
These fields clearly need more clarification. Both start and end flow times are named in a slightly incorrect way and we need to talk about flow export timeout.
Let’s imagine you have a long TCP session which was established a while ago and stays active for many hours and we transfer a significant amount of traffic over it. We clearly want to know about such traffic transfers using IPFIX. What are our options?
We can wait until this session will be terminated and then send a huge hour-long flow which has flowStartMilliseconds set to time when TCP session was established and flowEndMilliseconds is set to time when it was normally terminated. Can we afford to wait for hours before observing what’s going on? We clearly cannot.
For that purpose IPFIX implementations on routers have logic which can cut such long flows into shorter pieces and export them individually. Granularity of such chopping is controlled via an option usually called active flow timeout and it's normally measured in seconds.
It means that even if the TCP session is still active we will for X second then then export the amount of data transferred for the last X seconds using individual flow. In this case industry (excluding Juniper) agrees that flowStartMilliseconds will be set to start of this chunk of flow and flowEndMilliseconds will be set to time when this chunk of flow finished. Unfortunately RFC does not focus on this specific topic and I hope we will address it in future in another article.
Finally, we have following list of fields:
| IPv4 template | Field length | IPv6 template | Field length |
|---|---|---|---|
| ipVersion, #60 | 1 byte | ipVersion, #60 | 1 byte |
| sourceIPv4Address, #8 | 4 byte | sourceIPv6Address, #27 | 16 bytes |
| destinationIPv4Address, #12 | 4 byte | destinationIPv6Address, #28 | 16 bytes |
| octetDeltaCount, #1 | 8 bytes | octetDeltaCount, #1 | 8 bytes |
| packetDeltaCount, #2 | 8 bytes | packetDeltaCount, #2 | 8 bytes |
| protocolIdentifier, #4 | 1 byte | protocolIdentifier, #4 | 1 byte |
| sourceTransportPort, #8 | 2 byte | sourceTransportPort, #8 | 2 byte |
| destinationTransportPort, #11 | 2 byte | destinationTransportPort, #11 | 2 byte |
| tcpControlBits, #6 | 2 byte | tcpControlBits, #6 | 2 byte |
| ingressInterface, #10 | 4 bytes | ingressInterface, #10 | 4 bytes |
| egressInterface, #14 | 4 bytes | egressInterface, #14 | 4 bytes |
| bgpSourceAsNumber, #16 | 4 bytes | bgpSourceAsNumber, #16 | 4 bytes |
| bgpDestinationAsNumber, #17 | 4 bytes | bgpDestinationAsNumber, #17 | 4 bytes |
| flowDirection, #61 | 1 byte | flowDirection, #61 | 1 byte |
| flowStartMilliseconds, #152 | 8 bytes | flowStartMilliseconds, #152 | 8 bytes |
| flowEndMilliseconds, #153 | 8 bytes | flowEndMilliseconds, #153 | 8 bytes |
With all these fields in place RFC compliant collectors will be able to provide lots of insights about different aspects of your network operations. By following RFC for field lengths you increase chances that your implementation will work fine with collectors from different vendors.
I'll be happy to hear your feedback and you can reach me via email or via LinkedIN.