Rate limiting

Why limit?

There are many reasons why limiting the rate at which Netfilter rules will match a packet can be a good idea. One such reason would be to limit the size of the log which can be generated in response to dropped packets. Another would be to limit the rate at which connections to the ssh server will be accepted to make brute-force password attacks more time consuming.

The rate limiting discussed here should not be confused with bandwidth limiting or QoS. It is not designed to be used to implement this type of solution and, in fact, will probably result in a horrible loss of connectivity if you try to implement such a solution using the technologies described below.

 

What to limit by?

The Netfilter system includes a variety of matchers which we can use to implement rate limiting solutions. A non-exhaustive list is given below along with a brief description of each module. We shall be covering all of these in turn in the following sections.

limit

This matcher can be used to limit matching of a rule to a rate specified. Any rule which includes this matcher will only match while the limit has not been exceeded.

connlimit

This matcher can be used to limit matching of a rule based on the number of existing active connections from a given host or address block. Any rule which includes this matcher will only match while the number of connections is above, or below if negated, the number specified.

hashlimit

This matcher can be used to limit matching of a rule to a rate specified on a per address, or per address-port tuple, basis. Any rule which includes this matcher will only match while the limit has not been exceeded for the specified source or destination address.

recent

This matcher can be used to create, update, and perform actions based on the contents of dynamic lists of addresses. It can be used to create extremely complex rules and is ideal for creating dynamic behaviours such as automated retaliation and port "knocking" activated rules.

As you can see there is a matcher for pretty much every imaginable scenario.

Simple rate limiting

Let's start with a simple example which uses the limit matcher to limit the maximum quantity of entries which can be sent to the logging subsystem in a given period.

This may sound unnecessary but each log file entry is approximately 200 bytes. Assuming a rate of 1 log entry per second a log of roughly 16 megabytes can be generated every single day! This would grow to consume nearly 6 gigabytes of storage after a year. For this reason any log generating lines should always be limited.

 

Now that we know that a rate of one log entry per second is probably enough as a long term average we are half way to writing our rule. We are only half way as the limit matcher takes two parameters. The second is the maximum burst rate at which packets will be matched. As packets can come in fairly fast during a suspicious event we should probably set the burst limit fairly high. I have arbitrarily decided on a value of twenty for this example.

We can add simple rate limiting to our example LOGDROP rule by modifying it as shown below. Lines to be removed are shown crossed out.

iptables -N LOGDROP
iptables -A LOGDROP -j LOG --log-prefix 'FIREWALL - DROP:' --log-level info
iptables -A LOGDROP -m limit --limit 1/second --limit-burst 20 \
   -j LOG --log-prefix 'FIREWALL - DROP:' --log-level info
iptables -A LOGDROP -j DROP

While simple rate limiting is adequate for controlling the size of a log file it is not really suitable for much else. We could use it to limit the number of connection attempts to a particular service in any given period, for example, but as it pays no regard to who is attempting to connect this would just be a recipe for an easy denial of service attack. In the next section we shall examine some more functional rate limiters which can be used to perform such tasks.

Connection limiting

Sometimes it can be useful to be able to control the number of simultaneous connections which may be opened to a particular resource from a given host or network. A good example of this would be the ssh protocol as each connection requires a fairly significant quantity of system resources to maintain.

As remote shell access is a fairly commonly offered service, and one which it is desirable to exert some level of control over, let's modify our firewall configuration script to only allow a maximum of two simultaneous ssh connections from any address. This can be done by inserting a rule into our script as shown below.

The connlimit module has been dropped from some recent kernels. If that is the case with the kernel you are using then you shall have to either make do without or add it to the kernel manually using patch-o-matic.

 

iptables -A TCP-IN-REQ   -p tcp --dport ssh -m state --state NEW \
   -m connlimit --connlimit-above 2 -j REJECT --reject-with icmp-admin-prohibited
iptables -A TCP-IN-REQ   -p tcp --dport ssh -m state --state NEW         -j ACCEPT
iptables -A TCP-IN-REQ   -p tcp --dport ssh -m state --state ESTABLISHED -j ACCEPT
iptables -A TCP-OUT-RESP -p tcp --sport ssh -m state --state ESTABLISHED -j ACCEPT

You can see from the example rule above that we have decided to reject any new connection attempts when there are already two or more connections established. To inform the client of the rejection we are sending an ICMP admin-prohibited message which is probably not the use for which it was intended but it is as good as any other.

This matcher limits connections on a per address basis. If more than one system is hidden behind a single address, such as a network connected to the Internet using NAT or a transparent proxy intercepting web traffic, they will be treated as a single host by this matcher.

 

Per address / port rate limiting

As the connlimit matcher has been removed from recent kernels, and some people don't like the idea of patching their kernel by hand, in this section we shall be taking a quick look at another matcher which can be used to mitigate attempts at brute force password cracking and denial of service attacks. This matcher is called hashlimit and can be used to limit the rate of matching packets by source or destination address or address-port tuple.

The hashlimit matcher, like the simple limit matcher we met earlier, accepts two parameters which control the rate of packets to be matched. These are --hashlimit and --hashlimit-burst respectively. Unlike the limit matcher the hashlimit matcher is not automatically associated with the rule to which it is bound but requires a name to be specified with the --hashlimit-name parameter. This allows the same limit to be applied to more than one rule providing far greater flexibility than the simple limit matcher.

In addition to the parameters mentioned above the hashlimit matcher also requires the match mode to be specified using the --hashlimit-mode parameter. This is specified as a comma-separated list of the following options:

dstip

The matcher will record entries in the hash table based on the destination address of the packets.

dstport

The matcher will record entries in the hash table based on the destination port of the packets. If this option is specified alone then the matcher will perform in an identical way to the plain limit matcher when applied to a single port.

srcip

The matcher will record entries in the hash table based on the source address of the packets.

srcport

The matcher will record entries in the hash table based on the source port of the packets.

The example code given below uses the hashlimit matcher to limit the number of connection attempts to particular service to one per minute based on the source address from which the connection attempts originate. In this example we are limiting access to the ssh server as we have seen an increasing number of brute force attacks in recent months. The same rule can be applied to other services however you should make sure that you specify a different hashlimit-name unless you actually want the same limit to be spread across multiple rules.

iptables -A TCP-IN-REQ   -p tcp --dport ssh -m state --state NEW \
  -m hashlimit --hashlimit-name SSH --hashlimit 1/minute --hashlimit-burst 1 \
  --hashlimit-mode srcip --hashlimit-htable-expire 300000 -j ACCEPT
iptables -A TCP-IN-REQ   -p tcp --dport ssh -m state --state NEW         -j ACCEPT
iptables -A TCP-IN-REQ   -p tcp --dport ssh -m state --state ESTABLISHED -j ACCEPT
iptables -A TCP-OUT-RESP -p tcp --sport ssh -m state --state ESTABLISHED -j ACCEPT

The above rule is by no means perfect. In fact, it can open you to a denial of service attack if the attacker spoofs their source address when sending the syn packets. We shall be examining possible solutions to this weakness in a later section.

 

Complex limiting

In addition to the limit, hashlimit and connlimit matchers discussed so far the Netfilter system comes with another module which allows for the controlled handling of packets based on rate, as well as anything else which can be programmed using the matchers which are already available. This module provides a matcher called recent.

The recent matcher achieves this level of flexibility by operating in a subtly different way to the matchers which we have covered so far. Instead of matching based on a rigid predefined-plan the recent matcher allows the match logic to be coded using user-defined rules and chains.

This level of flexibility makes the recent matcher extremely powerful as it can be used to quite literally program the Netfilter system to respond in any way you desire. In the advanced section of this guide we shall explore the use of this matcher to detect port scans and take some very limited retaliatory action. For now, however, we shall just give a brief description of the available options.

name

This parameter is used to specify the name of the list which all other options will operate on. If no name is specified then the 'DEFAULT' list will be used.

set

If this option is present in a match specifier then the source address of the current packet will be added to the list. If the source address is already present then the timestamp and hitcount values will be updated accordingly. This option always returns true so that further options will be evaluated.

rcheck

This option can be used to query the list for the source address of the current packet. If the source address is present in the list then this option will return true. The source address will not be added to the list by this option.

update

Like the rcheck option above this option is used to query the list for the presence of the source address of the current packet. Unlike the rcheck option this option will add the source address to the list after the test is performed. The associated timestamp and hitcount values will also be updated.

remove

This option causes the source address to be removed from the list if it is present. If the source address was present in the list then this option returns true, it will return false if the address was not found.

seconds

This option can be used to narrow the match of any of the other options to only include the address if it has a timestamp with a value less than that specified. As this option is used to narrow the match of another test it must be used in conjunction with either the rcheck or the update option.

hitcount

Like the seconds option above the hitcount option is used to narrow the match of another option to only include the address if the associated hitcount value is greater than, or equal to, that specified. As with the update option above it must be used in conjunction with either the rcheck or the update option.

rttl

This option allows you to further restrict the match of an address with a packet to include the TTL value. This option is often used to try to detect when an attacker is forging their source address in an attempt to cause a denial of service to legitimate users. As with the other match restriction commands it must be used in conjunction with either the rcheck or the update option.

As you can see the related matcher is fairly complex. We shall not attempt to explore the functionality which can be implemented using this matcher in this section as it would be impossible to do it justice. We shall be returning to it later when we cover more advanced topics such as port scan detection and port "knocking".