Skip to content
Rabin Poudyal
Page 9-1
9-1 Incidents

9.12Dyn DNS attack and the Mirai botnet

A botnet of hijacked cameras, video recorders and routers flooded Dyn's DNS (Domain Name System) service, and users could not look up the addresses of some of the internet's biggest sites.

Dyn · 21 October 2016 · Attacks at scale

  1. 1Read
    Blast radius
    Dyn's Managed DNS customers, and the people trying to reach their sites. In the first wave mainly users whose lookups went to Dyn's points of presence on the US East Coast; the second wave was more global. Dyn names no customers; the Mirai paper, citing press reports, lists Amazon, GitHub, Netflix, PayPal, Reddit and Twitter.
    Duration
    Two damaging waves by Dyn's own times: about 11:10 to 13:20 Coordinated Universal Time (UTC), and 15:50 to 17:00 UTC, with residual impact from other sources until about 20:30 UTC. Dyn's status page marked the incident resolved at 22:17 UTC.
  2. 2Follow

    What happened, in order

    1. 1 Aug 2016

      Mirai first appears. It spreads by trying factory-default user names and passwords over Telnet, and infects nearly 65,000 internet-connected devices in its first 20 hours.

    2. Sep 2016

      Mirai attacks the hosting company OVH and Brian Krebs's security blog; the Krebs attack measures 623 gigabits per second. On 30 September its source code is published on a hacking forum.

    3. 14 Oct 2016

      The US Computer Emergency Readiness Team (US-CERT) warns of the threat from Mirai, noting that it uses a short list of 62 common default user names and passwords.

    4. 21 Oct, 11:10 UTC

      Dyn sees elevated traffic against its Managed DNS platform in Asia Pacific, South America, Eastern Europe and US West. The attack then turns on its US East points of presence with floods of TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) packets to port 53, the DNS port, from a large number of addresses.

    5. 21 Oct, 13:20 UTC

      Traffic shaping, rebalancing through anycast policies, internal filtering and scrubbing services are fully deployed, and service is back to normal.

    6. 21 Oct, 15:50 UTC

      A second, more globally spread attack begins, using the same protocols. Dyn substantially recovers by 17:00 UTC, with residual impact until about 20:30.

    7. 21 Oct, 22:17 UTC

      Dyn marks the incident resolved. A third attempted attack is mitigated without customer impact, by Dyn's account.

    8. 26 Oct 2016

      Dyn publishes its analysis: Mirai was the primary source of attack traffic, from up to 100,000 malicious endpoints, revising its first estimate of tens of millions of addresses.

  3. 3Find

    Root cause

    Note

    DNS turns a name such as github.com into an address. A company that uses a managed DNS provider has that provider's servers give the authoritative answer for its names, and every lookup that is not already cached ends up there. Dyn's servers were flooded with TCP and UDP traffic to port 53. Where they could not answer, users' resolvers could not find the address of a customer's site, so the site was unreachable even though nothing on it had failed.

    The traffic came mostly from Mirai, a botnet of internet-connected devices. Mirai scanned for devices with Telnet open and tried ten user name and password pairs chosen from a built-in list of 62 factory defaults. The paper by Antonakakis and colleagues, which tracked it from its first appearance, puts it at 200,000 to 300,000 infections in September 2016 and a peak of 600,000 in late November. Most were security cameras, video recorders and home routers. Infections did not survive a reboot, but a rebooted device with the same password was simply infected again.

    The attack fed on itself. As Dyn's servers failed to answer, resolvers around the world retried, which Dyn measured at 10 to 20 times normal traffic. That legitimate retry storm made the attack look far larger than it was: Dyn first reported tens of millions of addresses, and later revised that to up to 100,000 malicious endpoints.

    Many devices, one provider Mirai infected cameras, video recorders and routers by trying a list of 62 default user names and passwords over Telnet, and its source code was public from 30 September 2016. Up to 100,000 infected endpoints flooded Dyn's Managed DNS with traffic to port 53. Lookups for customer sites stopped resolving, and resolvers' retries pushed traffic to 10 to 20 times normal. The first wave ran from 11:10 to 13:20 UTC, mainly on the US East Coast; the second, more global, from 15:50 to 17:00 UTC on 21 October 2016. The devices The botnet The DNS provider The sites Infected devices cameras, DVRs, routers Mirai botnet up to 100,000 endpoints Dyn Managed DNS floods to port 53 Customer sites names stop resolving Default passwords 62 pairs, over Telnet Source code public, 30 Sep 2016 Resolver retries 10 to 20x normal volume wave 1: 11:10 to 13:20 UTC, mainly US East; wave 2: 15:50 to 17:00 UTC, more global; 21 October 2016
    Devices taken over through default passwords flooded one DNS provider, resolvers' retries multiplied the load, and sites that depended on it could not be found.
  4. 4Trace

    Why it got through

    The weakness was in devices no one at Dyn or its customers owned. Each camera or recorder shipped with a default password and an open Telnet port, and a device that worked well enough to watch a doorway gave its owner no reason to change it. The cost of an insecure default fell almost entirely on other people, and the US-CERT warning, a week before, could only advise owners to change passwords that many of them did not know existed.

    For the sites that went dark, one provider stood between users and every name they owned. Dyn's own statements say users routed to its East Coast servers could not reach some customers' sites while users on the West Coast could, so the outage followed the provider's geography, not the sites'. Dyn does not say how many of its customers had a second DNS provider, but a site whose names only Dyn answered had nowhere else for a resolver to go. The paper adds a twist: the attack commands also targeted PlayStation's name servers, which Dyn hosted, and the authors judge that the attacker was likely aiming at gaming infrastructure and disrupted everyone else on the way.

  5. 5Check

    The lesson

    Your availability is capped by your least redundant dependency, and DNS sits in front of everything. And an insecure default on a cheap device is not a private choice: at scale it becomes everyone's problem.

  6. 6Change

    What changed afterwards

    Dyn said it was extending and scaling the protective measures it had put in place during the attack, sharing what it learnt with other infrastructure providers, and working with a criminal investigation. It declined to speculate about motive or identity.

    The wider argument was about devices. US-CERT's advice was to reboot an infected device offline and change its default password before reconnecting, keep it patched and turn off Universal Plug and Play. The Mirai paper went further: randomised default passwords are only a first step, and devices should move from default-open ports to default-closed, with automatic updates, owner notifications and a plan for end of life.

  7. 7Verify

    Where the sources differ

    • Dyn's first statement reported "10s of millions of discrete IP addresses". Its analysis four days later revised that to "up to 100,000 malicious endpoints", blaming the retry storm. Dyn gave the paper's authors 107,500 addresses, of which 71 percent matched Mirai scanning they had seen, so other hosts may have been involved.
    • The retry storm was 10 to 20 times normal traffic. The 40 to 50 times figure often quoted is Dyn's early observation of TCP packet bursts, not retries. Reports of 1.2 terabits per second were, in Dyn's words, a claim it was unable to verify.
    • Dyn's statements name no customers, and say there was no system-wide outage: users on the West Coast could reach the sites. The list of affected sites here comes from the Mirai paper, which cites press reports. No source used here says Europe was cut off.
    • Mirai is often described as cameras and video recorders. The paper says routers were also a large share and that media reports overstated the cameras and recorders. The Krebs attack is dated 20 September at over 620 gigabits per second by US-CERT, and 21 September at 623 by the paper.
    • None of the sources used here advises using several DNS providers. The point about single-provider DNS in the lesson is this page's inference from how DNS resolution works, not a finding in Dyn's analysis.
  8. 8Source

    Sources

  9. 9Go