By Pavel Odintsov — Sep 4, 2024

sFlow on Huawei: story of scarcity and redundancy

Hello! I’m Pavel and I’m CTO and co-founder of FastNetMon LTD, London, 🇬🇧. We’re cyber security software vendor and we develop DDoS 🎯 detection and mitigation platform for Telecoms.

This blog post is result of research and cooperation on this matter with Blažej Krajňák.

Netflow and IPFIX for DDoS mitigation

To efficiently detect DDoS attacks we need to know what traffic passed over customer network and we need to know it as soon as possible. Sadly most popular (but clearly not best for DDoS detection) traffic telemetry protocols such as Netflow and IPFIX are not great from perspective of delivering traffic information fast as they have flow timeout delay which ranges between 15 and 60 seconds. Such delay is a mandatory part of protocol design and it's required to implement flow aggregation logic which is foundation of those protocols.

To efficiently detect DDoS we need to have access to basically all fields available in IPv4, IPv6, UDP, TCP, ICMP packet headers as any of them can be used by attacker to craft attack. Unfortunately, some fields which are absolutely crucial to detect DDoS efficiently are missing in many implementations of Netflow and IPFIX agents.The most important ones are TTL and IP fragmentation flags (more fragments, don't fragment, fragment offset).

sFlow is a silver bullet for DDoS detection

According to our experience at FastNetMon sFlow is one of the best protocols for DDoS detection and for many years it keeps one of the top places in my presentations (my slides from SANOG 37) about DDoS detection.

sFlow protocol benefits for DDoS detection

What is the catch?

We all known that it sounds too good to be true and you're absolutely correct to be sceptical. Basically all known implementations of sFlow protocol have some kind of issues.

For quite long time sFlow was considered as traffic telemetry protocol for switches but many router vendors (Juniper, Huawei) added support for it in their router product lines.

Sadly Juniper's sFlow protocol implementation for MX, PTX and QFX platforms has such incredibly long list of limitations that it barely possible to use it for medium and large scale networks.

On the contrary Huawei's sFlow implementation is considered by us as production grade and we have significant number of customers using it in production environment without any issues.

Huawei and sFlow: redundancy

One of the peculiarity of Huawei's sFlow implementation is redundancy: it actually exports more then needed. Let's look on sFlow packet from Huawei NE40E-F1A-14H24Q in Wireshark:

As you may notice we have three sections which carry raw packets headers:

Raw packet header which carries first 76 bytes of whole Ethernet frame
Ethernet frame data section which carries 32 bytes of Ethernet header only
IPv4 packet header which carries 40 bytes from only IPv4 header

Let's look on it with greater precision in Wireshark:

As you may notice same information is exported multiple times. Raw packet header has all the information needed for attack detection and presence of Ethernet header and IPv4 header is clearly redundant.

I asked co-author of sFlow Neil McKee from InMon Corp about his opinion about such behaviour and he pointed me to sFlow standard which tells us following:

💡

A flow_sample must contain packet header information. The prefered format for reporting packet header information is the sampled_header. However, if the packet header is not available to the sampling process then one or more of sampled_ethernet, sampled_ipv4, sampled_ipv6 may be used.

According to this information presence of Ethernet and IPv4 headers is redundant when raw packet header is here and it leads to additional overhead for sFlow agent and sFlow collector as each UDP packet can carry only 4-5 packet samples instead of 8-10 when only raw packet header presents.

Huawei and sFlow: scarcity of place when vlan in place

On contrary to previous approach when Huawei exports more then needed we observed multiple cases when it actually exports way less then needed.

Let's look on following example:

In this example (probably because of presence of VLAN tag) Huawei router exported only 52 bytes of packet in raw header section. As you may notice we do not have all information about TCP packet as it did not fit into 52 bytes and we got only first 2 bytes of TCP header.

Let's look on TCP packet structure:

Our regular logic failed to parse it correctly as we expected that whole TCP header should be present but we had only 2 first bytes of TCP packet instead of full 20 byte long TCP header. Fortunately, we got source and destinations ports but very important for DDoS detection information about TCP flags is missing.

We had to implement workaround to read only source and destinations ports in TCP packet header which are located right after start of TCP header. Very likely other packet parser implementations will discard such packets completely. It's important for sFlow agent implementer to keep in mind that minimum packet size for telemetry changes when we have VLAN or MPLS tags in place. Some devices allow to change length of packet header captured in sFlow but it's pretty rare and it may not be possible due to hardware constraints.

Lack of TCP flags in network telemetry is a very serious issue as TCP SYN flood attacks are one of the most dangerous and popular ones and it's crucial to have TCP flags in telemetry.

Huawei and sFlow: 32 bytes is enough for everyone

After seeing missing parts of TCP header we saw yet another fun case when Huawei router did not export even complete IPv4 header in sFlow sample. In happened for traffic which was dropped / discarded partially by BGP Flow Spec rate limiter:

You may notice that even source and destination IPv4 addresses are missing.

In this case we received only 32 bytes of packet and instead of getting whole IPv4 header which has length of 20 bytes we got only 14 bytes from it. I would like to remind structure of IPv4 packet and it looks this way:

In this case last field we were able to parse was checksum and both source IPv4 and destination IPv4 fields are missing. In compare with just missing TCP flags this issue is very serious as it completely breaks any capability to detects attacks and it's considered as critical issue by us.

Sadly this issue was officially confirmed by Huawei TAC:

💡

This is a normally behavior that the discard packets will fill only L2 header + 12bytes.

Curiously enough but we still can retrieve source and destination IP addresses from redundant IPv4 header carried next after raw header:

Huawei and sFlow: missing discarded flag

sFlow protocol provides an option to explicitly mark that specific sample was dropped by router by using special format of output interface (format 1) but Huawei clearly does not use it for discarded traffic:

As this traffic was discarded we should expect output interface format set to 1 and Huawei did not behave this way

Huawei and sFlow: missing ASN section for discarded traffic

Sadly we found yet another issue with discarded traffic. Unfortunately ASN information section called extended gateway data is missing for discarded traffic but presents for normal traffic:

Such issue leads to need for sFlow collector software to lookup ASN information for dropped traffic using data provided by third party providers which may not reflect information on customer network.

Conclusions

sFlow is extremely powerful protocol and Huawei's effort to support it on their router product line is admirable but some improvements still needed to polish support and address rough edges.

I hope more vendors will consider this as success story and will add sFlow support to their product lines.

Authors