Re: Security breach at multiple Federal agencies via SolarWinds

["Michael Paoli" <Michael.Paoli@cal.berkeley.edu>]

> From: goossbears <acohen36@gmail.com>
> Subject: Security breach at multiple Federal agencies via SolarWinds
> Date: Thu, 17 Dec 2020 10:32:22 -0800 (PST)

> Further thoughts and insights on this from Michael P, Rick M, Thomas L, and
> anyone else here?

Okay, I'm not going to add a whole lot to this.  Notably as it's already
been discussed fairly well elsewhere/elselist.

E.g. the conspire list.
http://linuxmafia.com/mailman/listinfo/conspire
http://linuxmafia.com/pipermail/conspire/2020-December/date.html
I think Rick Moen - and others, have already quite well covered it.
Though I might come from slightly different perspective,
and maybe if I were sufficiently motivated, I might find a few
minor bits to quibble slightly over, but I think Rick already pretty
much has it spot on well covered.

Reddit also has tons of stuff on SolarWinds too.  "stuff" - great
expert highly qualified stuff, lots of the unwashed masses, ... it's
The Internet - there's quite the mix.  I've mostly only barely skimmed
some of it, and certainly not tried to follow all of it, on Reddit
or elsewhere.

Lots of other places/sources too ... of, "of course", varying quality.

So, I'll add some bits and my commentary ... not necessarily in any
particular order - but maybe I'll try

First of all, yes, it's a big deal.  When widely deployed
hardware/software/firmware/whatever is greatly used as critical part of
lots of IT infrastructure and quite widely deployed - rightly and/or
wrongly - and it's majorly compromised, it's a big deal.
This would tend to apply to stuff like - widely used/deployed
critical flaws in - operating systems, network equipment,
(")security(") software/hardware, management/monitoring/control software
and products, control systems, etc.
So, yep, suffice it to say this is quite bad.

How bad?  Well, much of that depends not only how widely deployed,
and where, but how much - or little - one put trust into such.
Highly trust and put lots of faith and control into not-so-trustworthy
dubious stuff, and it can bite one - and very hard.  Be much more
skeptical, don't trust anything too much, well use defense in depth
https://en.wikipedia.org/wiki/Defense_in_depth_(computing)
and then, well, not great, but at least not nearly so bad and the
damage is much more limited and controlled/isolated.
But hey, you know "all" (or at least so many of) those, e.g.
Federal agencies ... that typically get a "D" or "F" on their
report cards on security, and most of 'em don't get a "C" or
better ... well, what do you think happens with agencies like that
when they widely deploy something that ought be secure, but it's really
not that secure and reliable, and something really bad happens with it.
Yeah, ... that.  Very bad.

Additionally, something gravely lacking here in so many places,
adequate monitoring and detection.  This should'a been caught a whole
lot sooner and stopped and shut down ... but ... it wasn't.
Not that that's trivial to do, but that so many failed to do it,
yeah, that's also a big deal.  That means a whole lot of nastiness
was going on a quite a long while before anybody noticed something
was up.  So, that makes bad much worse.

One of the posts I saw mentioned, Debian, and "what if", as if what if
something like that had happened to Debian?  Well, fair number of
years back, it kind'a did.  A Debian Developer (DD)'s key was
compromised.  And some bad folks started doing some nasty stuff - even
using a Zero-day
https://en.wikipedia.org/wiki/Zero-day_(computing)
exploit.
Well, unlike many others, Debian's got their sh*t together.  They
detected this in highly short order (day or two or less?), shut that
sh*t down, clamped down tightly on everything, checked everything, fixed
all the issues and damage, and slowly and carefully reinstated all or
most all services that were in place before.  So, since Debian caught it
so dang (relatively) fast, the damage was quite limited.  There was
still a fair bit of clean-up, but much of that was mostly precautionary
over stuff that might've been exposed - what was actually altered was
relatively minimal and detected and shut down in quite short order.
Let's see ... I'm fuzzy on the details - it was years ago, so ... for
folks that may want to read more ...
https://www.debian.org/News/2003/20031202
... yeah, ... compromise to detection ... 29 hours.
And it was actually a sniffed password that was the initial vector into
the compromise.
SolarWinds ... what many months or more?  Closer to 29 weeks than
29 hours.  Yeah, seriously not good.

And yeah, everyone ought pay ample attention to security.  But also,
many that are "jucier" targets ought pay a helluva lot more attention to
security, as they're much more attractive targets for attackers,
and the compromise risks are also much higher.

Yeah, ... once upon a time, I was working at a major financial
institution.  SATAN
https://en.wikipedia.org/wiki/Security_Administrator_Tool_for_Analyzing_Networks
was released.  And, within hours or less, said financial institution
saw being probed by SATAN.  Well, said financial institution
pays attention - this did not go without notice.  And follow-up.
It was tracked back.  Somebody working somewhere else at some
other employer, without anyone's authorization and approval, and
way outside what they were supposed to be doing there, was using SATAN
to poke at and prod/scan said financial institution.
Well, said financial institution got in touch with the employer
from which the attacks were originating, it got tracked back to the
person doing it, and they were summarily instafired.  (There may have
been further consequences/actions/outcomes, but I didn't specificly
hear).

Anyway, folks need pay attention, and especially jucier targets need pay
stringent attention and take appropriate measures - most notably to not
only try to prevent various attacks/compromises, and the like, but also
highly important - to detect such when they occur.  Because there are
always threats, and not all will be fully prevented all the time ahead
of time.  Stuff happens.  E.g. insider job - how well does your security
software and hardware prevent that?  Yeah, that's a tough one.  Okay,
*when* it happens, can you at least detect that it happened?  And
probably also with enough info to know who done it (or from whence it
came, and all possible available relevant details - like full traffic
captures of the stuff that happened?).

So ... a whole hella lot of SolarWinds users/customers, were not only
compromised by SolarWinds, but also, they generally failed to detect
that they'd been compromised and SolarWinds likewise failed to detect
the compromise,  and unfortunately too for many of them,
lack of or insufficient defense in depth, the compromises went
relatively deep and insidious.  So, yeah, a relatively big deal.

And, another thing.  SolarWinds.  People, companies, institutions,
whatever ... human(s) and/or run/operated by humans ... stuff happens,
thing break, folks make mistakes, screw up, whatever.  Okay, not great,
but deal with it - fix/repair it as feasible, take the appropriate
actions as feasible to ensure it doesn't happen again, fess up to it,
tell the truth, promise to improve and actually do so, and move on.
But ... not SolarWinds.  If there was any question that they were
sh*t before, it's no longer a question.  What did they do?
They placed the blame for their own incompetency and screw ups
upon the innocent.  Who/what did they blame and do they blame?
Did they blame themselves?  No.  Did they blame the attackers?  No.
They blame Open Source.  Well, f*ck SolarWinds, they've shown
themselves to be not only incompetent, but scum.
So, reminds me of recent news story.  Major fuel pipeline leak.
And ... the news story blamed ... tree roots.  Wrong!  The tree roots
were doing what tree roots do, what they've evolved over millions
of years to do, and highly predictable.  It's not like the
trees have a highly advanced well educated conscious society of
highly ethical trees, and, well, this one bad seed knew better
but intentionally misbehaved and screwed around with a fuel pipeline.
So, yeah, we're gonna send that bad tree to prison.  Nope, that's
not how it works.  The supposedly advanced civilization of humans
stupidly put a fuel pipeline where tree roots could get to and damage
and breach it, and not only did they do that, but they didn't
quickly detect and stop the problem, or even detect it before they
had a breach or major breach.  So, the fault lies with those that
made poor decisions.  Not with some tree roots.
Likewise SolarWinds.  They got caught with their pants down.
I don't know if they failed to use suspenders or belts, or what
their problem is, but pants down ... and they're blaming gravity
for their gross exposure.  Well, it ain't the fault of gravity.
It's not like the gravity of the situation couldn't be predicted.
And, yeah, doesn't help to have solarwinds123 as password on
external Internet accessible interfaces, either.  SolarWinds
reeks of incompetence ... and clearly now also scum and sleaze,
if that weren't already clear before.  And f*ck 'em for blaming
Open Source.  I think they ought be forced to do any and all their
work henceforward with absolutely no use of open source in any
way whatsoever directly or indirectly.  We can probably start
that by ripping out most or all of their network stacks,
most or all of their DNS access, probably most if not all of
the tools and software they use to build and test their
products, and sell and market them, etc., and keep going,
and rip out all the other Open Source out from under 'em,
and then lets see what they can do from there - let 'em
flounder and drown in nothing but their own rhetoric,
and limited to only using and interacting with
software that isn't Open Source and wasn't built,
developed, or delivered, using Open Source.
Oh yeah, and screw Equifax too - they likewise blamed Open Source
for their incompetence.
Well, was bit earlier, but in any case, SolarWinds blames
Open Source.
https://thwack.solarwinds.com/t5/Geek-Speak-Blogs/The-Pros-and-Cons-of-Open-source-Tools/ba-p/478665
"
Security becomes a major issue. Anyone can be hacked. However, the risk
is far less when it comes to proprietary software. Due to the nature of
open-source software allowing anyone to update the code, the risk of
downloading malicious code is much higher. One source referred to using
open-source software as "eating from a dirty fork." When you reach
in the drawer for a clean fork, you could be pulling out a dirty
utensil. That analogy is right on the money.
"
https://www.veracode.com/blog/security-news/are-we-eating-dirty-fork

"anyone to update the code".  Oh really, tell me exactly how *anyone*
can change the code in the Debian Linux kernel I'm running.
Start 'splainin' to me, ... go ahead now.
And tell me again how your proprietary closed source code is so
much better because we can't look at it and we should trust you.
And likewise how your security practices aren't open for most or all
to inspect and that's so much better.  How's that workin' out for you?
And your supply chain - don't have that open for inspection so we
can see if anything goes wrong.  How's that going for you?

https://en.wikipedia.org/wiki/SolarWinds#2020_supply_chain_attack
"On December 13, 2020, The Washington Post reported"
"The company stated in an SEC filing that fewer than 18,000 of its
33,000 Orion customers were affected".
Oh, fewer that more than half of.  Lovely.  Odd double-speak way of
saying "most", or "the majority of".
"indications of compromise dating back to the spring of 2020"
$ awk '/^2020.*Vernal Equinox/ {print $1}' ~/calendar
2020-03-19
$ date -I -d '2020-03-19 + 29 weeks'
2020-10-08
$ awk '/^2020.*Summer Solstice / {print $1}' ~/calendar
2020-06-20
$ date -I -d '2020-06-20 + 29 weeks - 1 day'
2021-01-08
Yup, still looks to me a lot closer to 29 weeks than 29 hours.
Don't worry, that closed source stuff is only about 24x7=168 times or
so worse that Open Source.
"November 2019, a security researcher notified SolarWinds that their FTP
server had a weak password of 'solarwinds123', warning that 'any hacker
could upload malicious [files]' that would then be distributed to
SolarWinds customers."
Yep, quality operation there.
"SolarWinds" ... "employee passwords had been posted on GitHub in 2019."
Uh huh, ... top notch.
"SolarWinds said they would revoke the compromised certificates by
December 21, 2020".
Yep, ... right on top of it - major security breach, sure, we'll get
to fixin' the major bleeding in a bit over a week or so.
https://en.wikipedia.org/wiki/2020_United_States_federal_government_data_breach
"Trump" ... "on December 19, 2020, 'everything is well under control'"
Uh huh ... just like COVID-19.  And it's gonna magically go away too,
right?
"had gone undetected for months" - yep, that's how you make bad lots
worse.
"allowed the attackers to" ... "perform federated authentication across
victim resources."
You put your trust in *what* software, and *what* level of trust?
And from what quality/integrity of software source?  Oh, closed source
"trust me" model?  What could go wrong?
"Within days of its discovery, at least 200 organizations around the
world had been found to be affected by the attack"
Don't worry, it's only most of Orion's 33,000 customers.
Nobody important like, ... oh, numerous Federal agencies, Microsoft,
FireEye, UK Home Office, the UK National Health Service,
the North Atlantic Treaty Organization (NATO), the European Parliament
... oops.

So, yeah, it's a big deal.  It matters where one places one's trust, and
how much trust one places there.  Good thing at least we've got a
president that we can ... oh sh*t.

--
You received this message because you are subscribed to the Google Groups "BerkeleyLUG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to berkeleylug+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/berkeleylug/20201220000941.186362b9oefo06ww%40webmail.rawbw.com.