The Stock Exchange under fire
NZX has had a difficult week. The subject of a series of ongoing, sophisticated denial-of-service style attacks, it has seen trading halted day after day. The fallout will be far reaching, and the government has called in the GCSB to assist with the efforts to respond to the attacks.
The particular type of attack, called a distributed denial-of-service (DDoS) attack, that has been levelled against the NZX, has the potential to cause huge disruption to businesses. So exactly what is a DDoS attack, and is there anything companies can do to protect themselves from a similar attack?
What is DDoS?
First up, some background on the internet
The internet is made up of web sites and services that sit waiting for new requests to come in, from anywhere and anyone. In general there is no limit on who or what can make requests of a web service, and every request has a small cost associated with it. That cost comes in the form of internet bandwidth (think “megabits” of data) that is used by that request, as well as the time the service has to spend dealing with the request and potentially sending a response.
Web services are configured to deal with a certain, generally quite predictable, level of requests so that everyone making a request gets a timely response. However if a greedy requester makes a very large number of requests at once, the service can become overloaded and other legitimate users will not receive responses to their requests.
Bandwidth is key
Most web services still have only up to only about 1 Gbps (“gigabits per second” or “a billion zeros or ones transmitted per second”) of bandwidth available to them, and the actual web server that serves up the responses might be able to process even less information than this.
Most businesses in New Zealand have perhaps as much as 0.9 Gbps of bandwidth available for browsing the internet at work, so although 1 Gbps is the most common capacity of a single web service, it isn’t actually an enormous amount of capacity for one web service to serve the whole world with.
To address this limitation, it is possible to spread out a single web service across multiple networks and web servers, so they can all share the load or requests, and so the others can pick up the load if one of them is overwhelmed or offline for any reason. This is generally known as load-balancing and we’ll talk more about this below.
It is also possible to put your web service behind a larger capacity connection, of say 10 Gbps, or 40 Gbps.
Unfortunately, no matter how much capacity you have for your service, and how much load-balancing or other techniques you use, there will always be a limit to the amount of capacity your web-facing services will have. Even more unfortunately, there is essentially no limit on the quantity or size of requests that can be sent to your web service from the whole of the internet. It is easy to see, therefore, that at a certain level of malicious requests, a web service will become overwhelmed and legitimate users will “denied service” from that web service. When this is done intentionally, it is called a Denial of Service (DoS) attack.
Plain old DoS
A simple DoS attack seeks to prevent (ie deny) a normal level of response from a web service, like a web site or a cloud service.
We should point out that there are other ways to DoS a web site or web service. These include hacking the domain name or domain naming services (DNS) associated with that web service, obtaining a valid certificate to impersonate the target service, hacking the web service itself, or DoSsing the service provider infrastructure on which the service is hosted. However, although these have a similar effect, they are generally not referred to as DoS attacks.
DDoS – Distributed Denial of Service
So what does the “distributed” part mean? A DDoS is a type of DoS attack that comes not from a single or limited set of attackers, but from a large number of attacking systems that are distributed around the internet.
There are two key points to this distribution of the attack sources: first, the attack is generally not coming from a single IP address, country or even region of the world; and, secondly, the attack is coming from a potentially large number of systems that have previously been compromised or co-opted by the attacker.
These two features allow the attacker to essentially be anonymous (they are not directly involved in sending the malicious requests) and also to significantly increase the size of the attack, well beyond what would be possible from a single or small number of attacker systems.
What’s more, it is actually possible for attackers to further “amplify” attacks coming from each compromised or co-opted system. Imagine an attacker making a single request to a co-opted anonymous server on the internet, and causing that server to respond with tens of thousands of responses all directed at a web service. These many co-opted anonymous systems, combined with amplification attacks, make it possible to create huge storms of traffic that gets directed at an innocent web service.
No matter what capacity that innocent web service has, it will be possible to overwhelm the service via a DDoS attack, particularly if there is amplification involved. The largest recorded DDoS attack to date involved 2.3 Tbps (that’s 2,300 Gbps) being levelled at a single service. There is no combination of counter-measures that can maintain normal web service against this level of request traffic.
The news isn’t great
The bottom-line, unfortunately, is that DDoS attacks are the hardest form of attack the defend against on the internet as it is currently designed. Notwithstanding the measures we discuss below for mitigating DDoS risks, there is always a limit to which those measures can defend against a DDoS attack. If someone decides they really want to DDoS your web services, they will be able to.
Sorry that isn’t great news, but on the positive side we generally do not see sustained periods of key web services being unavailable due to DDoS attacks, so the counter-measures generally do seem to deter or defend against all but the most determined attackers.
So what can be done to avoid or limit DDoS?
As mentioned above, the main thing you are trying to do, in avoiding the worst impact of DDoS attacks, is to have a large capacity for responding to requests, and to be able to rapidly block requests coming from malicious requesters on the internet.
Load-balancing of a web service is a good place to start. Load balancing simply involves having multiple instances of a single service, with the “load” of incoming requests being “balanced” across those multiple instances. This is useful for general resilience and uptime of the relevant web service, and it can increase service capacity manifold times (double, triple, quadruple, etc).
Although important, and a good place to start, these are, however, quite modest increases in capacity compared to the potential threat from modern DDoS amplification attacks.
Another important step is to place some protection between the web service and the internet. This protection is normally in the form of a firewall or even more specialist security device that can detect DoS patterns and respond by blocking traffic from the sources of the attack. Modern business-grade firewalls generally have such protections available.
It is worth noting, however, that firewalls typically only have about the same level of bandwidth available as the web service they are protecting (eg 1 Gbps, or perhaps 10Gbps). It should be clear therefore that firewalls ultimately can’t protect from significant DDoS attacks that result in tens or hundreds of gigabits of attack traffic. They can become overwhelmed just as quickly as the service they are trying to protect.
Firewalls are important for many other protections, and for attack visibility and event management, but they are not a particularly effective protection against determined DDoS attacks.
Using an IaaS
The next level of defence is to place your web services in a leading Infrastructure as a Service (IaaS). When you use a large, modern IaaS as the location from which your web services are serviced, you are piggy-backing on significant, enterprise-grade internet connectivity, in the sense that you are getting a piece (eg 1 Gbps) or a much larger chunk of internet capacity.
Although you should not think of this as a silver bullet for DDoS attacks, the leading IaaS providers (like Microsoft Azure, Amazon AWS and Google Compute Cloud) all have some degree of somewhat hidden protection for modest DDoS attacks targeting your public IP addresses.
In short, they take steps to try to ensure that DDoS attacks levelled at one customer’s services can’t significantly impact other customers’ services. Whilst this might seem like cold comfort for you as a customer of the IaaS, it is certainly better to be sitting behind Microsoft’s Azure internet connectivity than behind a smaller internet service provider (traditionally called “ISPs”).
Spreading content delivery
The next important level of defence is to spread your web service content delivery out widely across the internet, using what’s called a Content Delivery Network (CDN).
How do CDNs work?
CDNs are services that specialise in getting your services close to the people who are using them (because closeness makes a difference to speed), and increasingly they also offer security and availability protections in front of those services. Essentially they take a copy of your web service’s data and place it in multiple locations on the internet. When a request for your service arrives at that location, the response is sent directly from that location rather than having to come back to your web service (known as the “origin server” in this context).
In the extreme case, your web service serves no responses at all to the internet – the only time it actually receives requests is when some of its content changes, and the CDN needs to come and get the latest content. In these “full caching” (aka “full blocking”) scenarios, requesters on the internet never actually know about your underlying web service – they are only ever talking to the CDN.
In fact it is possible to prevent any requests coming from the “raw internet” to the web service, and only allow requests made by the CDN itself to the web service (to pick up the latest content).
How CDNs can help
In such a full-blocking scenario, it becomes the CDN’s job to avoid and resist DDoS attacks.
CDN providers often sit on top of IaaS providers as discussed above, and so you generally benefit both from the IaaS protections as the CDN-specific protections, to whatever service level you have purchased. Of course there are differing service levels and costs associated with the different levels and capacities of protection.
Why CDNs don’t always help
One key problem with the CDN approach is that very few web-facing services are made up of completely static data. A simple marketing website is generally quite static, in that the content doesn’t change particularly often. Such websites can often be placed into full-blocking mode.
However, there are many websites and services that simply can’t be in full blocking mode, because their content is regularly changing, or because they are a service that fundamentally requires traffic to come back to the main web server in order for the service to work at all.
It is worth noting, also, that above some level of attack traffic, your CDN will stop protecting you – either because it is above your service-level, or they have to take measures to stop your attack affecting all of their other customers. That generally involves taking your services off the internet, so there is nothing for the attacker to target any longer. That, of course, is just as good a result for the attacker as actually sending too much traffic to your web service.
So, with today’s internet, there are things that can be done to limit the impact of a DDoS attack on your web services. There are some important best practices that you really ought to comply with, to limit modest attacks.
More comprehensive and effective protections involve risk-cost trade-offs, with costs potentially relating to the project time associated with the architecture of your services as much as to technical protection services.
At the limit, however, there are DDoS attacks that are beyond the industry’s and the internet’s current ability to effectively respond to.
This doesn’t argue for doing nothing, but it is an important and unwelcome fact to keep in mind.