Inside the discovery of the massive SolarWinds cyberattacks: Security firm FireEye's top expert explains how his team found a 'needle in a haystack of needles'

  • Charles Carmakal, the chief technology officer of Mandiant, FireEye's incident response division, was in the middle of the sprawling SolarWinds supply-chain cyberattacks.
  • FireEye was hacked – a humbling experience for a cybersecurity firm – but Carmakal's team pivoted to use the experience for good.
  • With a team of 100 and daily calls until midnight, FireEye was able to provide actionable intelligence to other companies as well as government agencies. 
  • 2020 was filled with major cybersecurity events and Carmakal believes that the SolarWinds attacks drive home the takeaway that industry must work together.
  • Visit Business Insider's homepage for more stories.

Multiple US government departments as well as an estimated 18,000 companies suffered breaches in the sweeping SolarWinds supply-chain attacks, discovered in early December, that many experts see as historic and industry-changing. 

But of all of them, the cybersecurity company FireEye played a unique role: It is both an industry leader in responding to cyberattacks as well as, ironically, the first victim to discover this one in its own systems. And as the chief technology officer of FireEye's incident response division, Mandiant, it was Charles Carmakal's job to figure out what happened.

"It's a very humbling experience being part of a security company dealing with a hack of its own systems," says Carmakal. Yet in what experts call an impressive response, FireEye's humbling experience led to information that helped shut down the paralyzing cyberattack that shook the world since early December.  

Carmakal, a 39-year-old Florida native who became obsessed with cybersecurity in middle school, has led the $4.6 billion Silicon Valley firm through a remarkable pivot. In the scope of two weeks, FireEye went from the cybersecurity company that got hacked to the incident-response experts sharing detailed forensics that no one else has. After an initial dip when FireEye announced the hack December 8, the company's stock is up 41% over the last two weeks, clocking in at $21.87 a share as of Thursday.

Other cybersecurity players have praised Carmakal's team since the incident.

"We commend FireEye for their disclosure and collaboration, so that we can all be better prepared," Microsoft spokesman Jeff Jones said.

FireEye remains "the de facto solution for incident response," says Bryson Bort, the CEO of Scythe and a special advisor in 2020 to the US Cybersecurity and Infrastructure Security Agency (CISA). "That's not going to change." 

The SolarWinds attacks are the finale of a year that thrust cybersecurity into the spotlight, as the industry pulled together to protect remote workers, hospitals battling COVID-19, and the election. Carmakal believes the SolarWinds attacks drive home a key lesson learned in 2020: cooperation in the industry is key. "It's getting people to think about security differently than before," Carmakal says. "It proves the value of intelligence-sharing." 

But first, Carmakal had to gather the intelligence in the first place. 

'Looking for a needle in a haystack made of needles'

In early December, Carmakal's team first noticed what he calls "suspicious activity on our network." 

"We couldn't figure out why," he says. The team developed several hypotheses, including a possible connection to the ubiquitous IT-management firm SolarWinds.

"I asked our investigators to really dig into the SolarWinds systems," Carmakal says. 

Like some 300,000 other companies, FireEye uses SolarWinds to monitor traffic, performance, and outages on its computer network. Because SolarWinds' software was a third-party product with code approved by its maker, a subtle hack would be extremely difficult to find. Carmakal's team would have to scour every part of the software that touched FireEye's network. 

The team looked through 50,000 lines of computer code, he says, searching for any slight anomaly. "They were looking for a needle in a haystack made of needles." 

They found the right needle. 

100 people and daily bridge calls until midnight 

Carmakal was in a video conference with other C-suite executives when he got a message from a malware analyst on his team saying that he'd found, deep inside SolarWinds' software, a "backdoor" – a hacker's secret entrance into a company's systems planted in computer code. We need to talk about this immediately, the analyst messaged Carmakal. 

"I told the people on the call, 'I need to drop — and take some of you with me,'" he recalls. "And it was a call I was leading."

FireEye had been hacked – and perhaps many others had, too. "When they actually found a backdoor code in SolarWinds, it meant the team had truly found something," Carmakal said. "The level of energy on the team surged." 

More than 100 people in the company of 3,200 began working on the hack and the company set up a virtual "conference room" for its experts to gather. "A bunch of us were on bridge calls every day until midnight," Carmakal says. "We couldn't just tell the cybersecurity community something bad had happened. We had to give them actionable intelligence."

What they found was intimidating. A global attack dating back months, to as early as March. 

'They'd read our blogs. They'd read everybody's blogs.'

Carmakal discovered a level of sophistication he had never seen before in an attack. The hackers – clearly a large and very meticulous team – had carried out an ingenious attack "supply-chain" attack, meaning that it started with a vendor and spread to its customers. 

First, the hackers broke into SolarWinds' system. Then, they injected the backdoor code into the software and somehow forced it to be included in the patching system SolarWinds used to update its software. The hackers faked approval for that addition to SolarWinds' software and, finally, learned how to exploit SolarWinds' connection to thousands of clients.    

"The amount of infrastructure they'd built to pull this off, and the level of discipline was uncanny," Carmakal says. "This is a group that has been in the game for a very long time."

He believes the group patiently gathered intelligence on the way many different companies approach cybersecurity by reading the technical writings their teams share to help the industry. "They'd read our blogs," he says. "They'd read everybody's blogs." 

The hackers accessed FireEye's own hacking tools, which could be used in attacks on nation states or other companies. Bat Carmakal says he doesn't believe that will happen. 

"We have seen no unauthorized use of our tools yet, and I don't think we will," he says. 

FireEye put out so many alerts about the tools that enterprises will be able to defend against them, he says, easily identifying anyone using them as a criminal: "We burned our own tools." 

Some experts, including US Secretary of State Mike Pompeo, have identified the hackers as working for the Russian government. But Carmakal isn't ready to go there. "A lot of people listen to what we say about attribution, and we don't know that conclusively," he says. "So we aren't saying that."

Carmakal's team gave the impressive hackers the unceremonious name UNC2452 – the name referencing that the attackers are unclassified (not officially determined to be a nation-state yet), with the first intrusion on March 24, with a 52nd update to the SolarWinds code.

SolarWinds pointed customers to this web page with updates on the attacks, and provided the following statement from company president Kevin Thompson: "Security and trust in our software are the foundation of our commitment to our customers. We strive to implement and maintain appropriate administrative, physical, and technical safeguards, security processes, procedures, and standards designed to protect our customers." 

In many enterprises, the hackers still have access to internal systems

As a 16-year-old company with many government clients, FireEye was able to help agencies respond to the national security concerns of the SolarWinds attacks quickly. And as a legacy company in the cybersecurity field, it was also able to rally the community with guidance and tools. 

Carmakal and FireEye reached out to government agencies, "helping some to respond, offering some coaching and mentoring, too." His team led customers through detection of the SolarWinds intrusion, and published guidance for the industry. 

"Some other companies have told us that if the attacks hadn't hit FireEye, we may not have known about this for a long time," Carmakal says. There was never hesitation to share the intelligence FireEye gathered from its own hack, Carmakal says: "There is no question that when you share information about threats, you all benefit."

The good news is, the supply-chain hack has stopped spreading. FireEye and Microsoft collaborated on a "kill switch," a program that finds the intrusion on company's networks and de-activates the malicious software. 

But for thousands of companies, stopping the spread of the SolarWinds attacks is just beginning. Hackers remain in their midst. "For organizations already compromised, the threat actor likely still has access," Carmakal says. In the past week, more than 50 companies have reached out to FireEye for help, he says. 

The SolarWinds attacks underscored one of 2020's hard lessons: Devastating attacks require the cybersecurity industry to pull together, he says, sharing tools and offering free services. When ransomware hackers began targeting hospitals, endangering the lives of people in the middle of a pandemic, any competitive concerns disappeared. 

"It was something we could all get behind," he says. "The hackers had crossed a line."

That lesson will survive, he believes, in part because of what happened with UNC2452, a dramatic year's final act: "This is one of the events people in our industry will be thinking about for a very long time."

Source: Read Full Article