Re: Security breach at multiple Federal agencies via SolarWinds

[Rick Moen <rick@linuxmafia.com>]

Quoting goossbears (acohen36@gmail.com):

> Further thoughts and insights on this from Michael P, Rick M, Thomas L, and 
> anyone else here?

I'll pass on my late-night posting to CABAL's mailing list.  As an
additional comment, cybersecurity firm FireEye, cited below as one of
the victims of the software-chain infiltration, i.e., one of
SolarWinds's customers who bought and ran the trojaned Orion Platform 
network-management software, was also a _key good guy_.  FireEye 
figured out that their retail copies of Orion were up to no good (had 
briefly breached FireEye corporate security from inside the firm's own
networks) and alerted Department of Homeland Security (and alerted
SolarWinds).
https://www.bloomberg.com/news/articles/2020-12-15/fireeye-stumbled-across-solarwinds-breach-while-probing-own-hack

My point is that Texas-based proprietary software company SolarWinds, Inc. 
had been utterly clueless about having had their entire software
production chain taken over for months, and had to be informed of their 
stunning incompetence and its catastrophic effects by a customer.

The phrase 'You had _one_ job!' comes to mind.
https://www.youtube.com/watch?v=zHCzlCoDBCI

One obvious lesson for Linux users is that it's a reminder that blithely
running some chump corporation's proprietary software exposes you to
risks that you would avoid if you said 'I'll pass' -- and that
code-signing can be just another way to go wrong with confidence...
as three million users of Google Chrome and Microsoft Edge are finding
out:
https://arstechnica.com/information-technology/2020/12/up-to-3-million-devices-infected-by-malware-laced-chrome-and-edge-add-ons/
"How could the Vimeo Video Downloader extension have been unsafe?  It was 
signed by the [Google|Microsoft] online store!"

I suspect I'll write about the latter story on CABAL's mailing list.



Date: Tue, 15 Dec 2020 22:52:30 -0800
From: Rick Moen <rick@linuxmafia.com>
To: conspire@linuxmafia.com
Subject: Security breach @ multiple Federal agencies via SolarWinds Orion software
Organization: If you lived here, you'd be $HOME already.

In this posting, I'll be trying in real time to figure out the
substantive reality behind a current news story.  Example:
https://gizmodo.com/feds-still-trying-to-determine-how-screwed-they-are-aft-1845888076

Headline is:
Feds Still Trying to Determine How Screwed They Are After Massive SolarWinds Hack
by Tom McKay

RM: There are recurring problems with IT press coverage of security
items, especially security breaches.  1. Where, as is frequently the
case, somebody messed up, the details go underreported because the
people who know don't want to talk about it.  2. IT reporters usually 
don't understand security very well, and tend to uncritically crib from
press releases.

  A cyberattack that began by targeting an IT firm used by numerous
  federal government agencies, Fortune 500 companies, and other high-value
  targets is shaping up to be a historic event.

  The U.S. government is still reeling after the detection of a massive
  foreign intrusion into federal computer systems at agencies including—at
  a minimum—the Department of Homeland Security, the Treasury, and the
  Commerce Department; [...]

  Those responsible built a backdoor into Orion, an IT management
  software produced by SolarWinds, possibly by breaking into Microsoft
  email accounts and other systems, according to the Wall Street Journal
  [link].  They then used it to contaminate software updates provided by
  the company with malware in March and June 2020. 

To unpack that:  Private US company SolarWinds, Inc. publishes
proprietary MS-Windows software for businesses to help manage their
networks, systems, and information technology infrastructure.  For
obvious reasons, any such software is itself security-sensitive and runs
with elevated privilege.  In Spring 2020, the Russian Federation Foreign
Intelligence Service ('SVR'), specifically its APT29 aka Cozy Bear team,
 managed to break into the crown jewels at SolarWinds, Inc., gaining
control of a software signing key for the production software chain,
which was then used to gain 'tokens' for other highly privileged roles
at SolarWinds, and among other thing insert remote-backdoor software
into a binary software library (SolarWinds.Orion.Core.BusinessLayer.dll) 
used in future releases of SolarWinds's Orion Platform software product.
So, the 'malicious' code in question then went out signed by
SolarWinds's release-code key, and so went out automatically to
customers as supposedly authentic code.

This root-level compromise of a piece of widely used commercial
off-the-shelf (COTS) software snagged _lots_ of victims.  Those who've
admitted getting suckered include:

o  NATO
o  US Treasury Dept.
o  US Commerce Dept. National Telecommunications & Information Administration
o  US Dept. of Homeland Security
o  EU Parliament
o  UK Health Service
o  UK Home Office
o  cybersecurity firm FireEye (!)
o  pharmaceutical and biopharmaceutical company AstraZeneca (probably)


How did SolarWinds, Inc. get H4X0Red?  Maybe, by being really
mind-bogglingly stupid?

https://www.msn.com/en-us/news/politics/notorious-hacker-fxmsp-sold-access-to-solarwinds-machines-report/ar-BB1bXZwj

  [...]
  Vinoth Kumar, a security researcher, told the outlet that he warned
  SolarWinds that their update server could have been accessed by "any
  attacker" with ease last year because the password was set to
  "solarwinds123." Kumar first notified the company of the issue on
  November 19, 2019 and the company responded three days later, according
  to emails he supplied to Newsweek.

  Kumar believes the vulnerability may have been present as far back as
  June 2018.
  [...]

Or maybe not?

  The recent breach, allegedly by Russian hackers, is also unlikely to
  be directly related to the password vulnerability since it took place
  months after the issue was remedied.

Doesn't seem reassuring, anyway.

SolarWinds, Inc. asks customers to un-fsck themselves as follows:

  SolarWinds asks customers currently using Orion Platform v2020.2 with
  no hotfix installed or 2020.2 HF 1 to upgrade to Orion Platform version
  2020.2.1 HF 2 as soon as possible to ensure the security of your
  environment.

https://www.solarwinds.com/securityadvisory/faq

If I were a customer, I'd want the answer to the question 'What
happened, guys, and why should I feel reassured that it cannot ever
happen again?'  Is that addressed in their security advisory FAQ, you
ask?

  Why didn’t SolarWinds catch this vulnerability before it happened?

  This attack was very complex and sophisticated. The vulnerability was
  crafted to evade detection and only run when detection was unlikely.

Um, guys?

  How do you know the new build is secure?

  We have limited access rights to our build environment to only those
  necessary and added additional controls to limit access further. As an
  added precaution, we are using a new code signing certificate for our
  new builds.

Um, _guys_?  Why did this fail the first time?

  With these processes in place how was your code compromised?

  We are not aware that the SolarWinds code base was compromised.[...]

You're kidding.

  [...]Our initial investigations point to an issue in the supply chain
  resulting in a compromise of our product that inserted a vulnerability
  within its Orion monitoring products which, if present and activated,
  could potentially allow an attacker to compromise the server on which
  the Orion products run.

'An issue in the supply chain'?

Here's the thing:  If you do code-signing competently, you can no longer 
pass the buck to 'the supply chain', because any (hypothetical)
tampering downstream from your crown-jewels signing machine would result
in the modified software no longer validating as signed by the signing
key of record.

So, the logical inference is that the above is poppycock, that
SolarWinds's code-signing infrastructure, the crown jewels, was indeed
compromised.  And, by implication, SolarWinds, Inc. is either in denial
about this fact and is delusional, or is clumsily lying.  The latter 
interpretation would be a little more reassuring than the former, IMO.


Let's see what CSO Online says:
https://www.csoonline.com/article/3601508/solarwinds-supply-chain-attack-explained-why-organizations-were-not-prepared.html

  SolarWinds stated that its customers included 425 of the US Fortune
  500, the top ten US telecommunications companies, the top five US
  accounting firms, all branches of the US Military, the Pentagon, the
  State Department, as well as hundreds of universities and colleges
  worldwide.

  The SolarWinds software supply chain attack also allowed hackers to
  access the network of US cybersecurity firm FireEye [...]

/me reads many more paragraphs.

Nope, no useful insights from CSO Online.

Brian Krebs (https://krebsonsecurity.com/) has started to cover the
story, but in fairness it's quite new.  (I expect he will have useful
things to say, and recommend his site.)

There's a subReddit to follow the story:
https://www.reddit.com/r/Solarwinds/



I'm going to have to close out this posting without any pretence of 
having reached grand conclusions:  Possibly, more will come out.
However, if I had to guess, based on available evidence, the root cause
will turn out to involve SolarWinds, Inc. security incompetence -- 
made worse by the shortage of transparency that is typical with
proprietary software companies.

You might wonder:  Could something similar happen with, for example, the 
Debian Project?  The simple answer is 'yes', but there is competent 
management of key-signing both at the ftp-master build machines and 
among the individual maintainers of Debian packages.  Basically, 
the all-volunteer Debian Project routinely does _way_ better than this 
major-name software company did.

-- 
You received this message because you are subscribed to the Google Groups "BerkeleyLUG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to berkeleylug+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/berkeleylug/20201217191540.GI28791%40linuxmafia.com.