[PLUG] Responsible system/network administration 101 [was: Internet Janitors (WAS: VPS Hosting)]

[Rich Kulawiec via plug <plug@lists.phillylinux.org>]

[ I've finally found time to respond to this.  Sorry.  Been a little busy. ]

On Sun, Jul 12, 2020 at 02:32:00PM -0400, brent timothy saner via plug wrote:
> As someone who's worked in hosting in various forms in the past, I can
> tell you platitudes and demands like this are nice, but they don't work
> in a practical form because Real-Life(TM) is a little more grey than
> black-and-white.

As someone who's been running 'net-connected systems and networks for
a very long time, I can tell you that this is EXACTLY how things work
at quality operations.

It is the responsibility of *everyone*, whether they're running a small
hosting farm or a global network of datacenters, a /8 or a /32 with
a Raspberry Pi on it, to ensure that their operation isn't a chronic
and/or systemic source of abuse/attacks.

*How* they do that is not my problem.  My problem is doing it for the
things that I run, and I've done that -- starting from the design stage,
going through implementation and on to operation.  That's why you don't
see attacks/abuse from the things that I run.  That's also why there
are certain things that I've chosen *not* to do, because I didn't feel
that I could secure them adequately and I knew that if I didn't, the
fallout from that would impact others.

I expect no more and no less from everyone else.  To be blunt,
if you (rhetorical you) can't run it right: don't run it at all.

In some cases solving these kinds of problems is easy, for example:

> 1.) You're a VPS provider. You seee a massive outbound surge of email
> traffic (DST 25/TCP, 465/TCP, etc.). Is this instance a spam service? Or
> is it hosting a massively popular newsletter?

This is where your network of spamtraps -- you know, the one that you
set up very carefully *before* you allowed people to send massive outbound
surges of email from your operation -- can come in handy.  This is also
where feedback loops with major email providers can come in handy.  This
is also where judicious throttling of anything displaying exponential
behavior can come in handy, because it's probably abuse.

Also: did you vet your customer?  Did you look at their DNS history to see
where they were hosted previously?  If they've been through multiple VPS
services before yours, then maybe they just weren't happy or those VPS
services sucked or maybe they outgrew them.  OR: maybe they're spammers
who have been repeatedly kicked off other providers.

While you were vetting them, did you look at what their domains are?

Here's an example.  Last week the FBI sent out a notice about quite a
few domains that were impersonating the US census bureau, e.g.:

        uscensus-gov(.)us
        uscensus(.)co
        uscensus(.)net
        uscensus(.)us
        uscensusbureau-gov(.)us
        uscensusbureau(.)co
        uscensusbureau(.)net
        uscensusbureau(.)org

and *many* more.  Gosh, d'ya think if a customer shows up and asks
you to provision domains with those names *in a census year* *in an
election year* that they might be up to something and maybe, just maybe,
you should consider hitting "hold" on that process while you take
the time to compose a letter telling them to pound sand?

(Incidentally, I have the FBI announcement as a PDF if anybody wants a copy
and can't find it.  It's "FLASH_Spoofing_US_Census_Bureau_Websites-1.pdf.)

(Also incidentally, I took the time to see where all of those domains were
hosted.  The usual suspects turned up: OVH, Namecheap, Amazon, Digital Ocean,
Softlayer, etc.)

Bottom line:

Techniques for detecting spammers at every point from when they walk in
the door to when they fire up the abuse are very well-known and people like
me have written about them ad nauseum for years and years.  Which ones
work the best depends on the environment, and none of them are perfect.
But if you deploy multiple ones, then you greatly increase your chances
of detecting an incident before anybody else does, and if you do it really
well, you have a decent shot at preventing an incident from happening.

There are no valid excuses for failure to do these things.  None.


The same is largely true of abuse/attacks via other methods, such as in
the examples that you enumerated and I've elided.  There are no perfect
diagnostic methods for any of them; however, due diligence starting
before someone's even a customer and continuing throughout their tenure
usually suffices.

Now, there *are* some abusers/attackers who are clever enough to try
to fly under the radar and some of them are quite good at it.  That's
where reliance on and quick response to abuse reports is critical.

But note two things: first, abuse reports should never be the first line
of defense - they should be the last.  The design/operations goal should
be to make it unecessary for people to file them, because what you're
essentially doing by making them necessary is forcibly conscripting other
people to do your job for you. Those reports are documentation of your
failures -- which is why it's a good reason, port-mortem, to study them
and figure out how not to fail that way again.  And second, one good thing
about an abuser/attacker who is deliberately bandlimiting in order to stay
under the radar is that they're doing is that they're doing less of it.
"Less" isn't as good as "none" but it's certainly better than "more".
It also discourages them from hanging around: abusers/attackers talk
to each other, y'know, and if you run an operation where they have a
hard time signing up and a hard time doing their thing and a hard time
sticking around...they'll probably go somewhere else.

There are plenty of "somewhere elses", sadly.  And that is what
firewall rulesets are for.

> The vast majority of the abuses seen on the Internet
> are due to misconfigured services that attackers compromise and take
> advantage of, not first-level abuse.

No.  As someone has been studying abuse for decades, I can tell that this
*was* true once upon a time, but it's not true now.  The vast majority
of abuse today is due to very poorly run operations that will sign up
anyone, let them do anything, ignore abuse reports (if they even accept
them), and/or shrug their shoulders when they're asked to discharge
their basic professional responsibilities.  To be sure, some of that
abuse is amplified by misconfigured services -- so sure, we could (and
should) take the folks running those to task and get them to fix
their stuff.  But it's the origination points that are the root
cause of the problem(s).

You're paying for this.  I'm paying for this.  We're all paying for this.
And we're collectively paying a lot, as in "billions".  (The global market
just for DoS mitigation goods/services is in the billions...and that's
just one sector.)

Those are costs that those operations are *not* paying -- by not having
good processes and good people in place   Those costs are being shifted
onto you and me and everyone else.  We pay for their negligence
with our resources, including our time and attention.  Every CPU cycle,
every dollar, every minute of our time that's been invested in fending
off abuse/attacks is a cost that we paid because someone else didn't.
(However, they quite happily pocketed the revenue from the abuser/attacker.)

Those are the easy-to-see, easy-to-measure costs: there are others,
like customer irritation due to spam we didn't block or slow response
of web servers due to the side effects of DoS mitigation.  Just because
those are harder to quantify doesn't mean they don't exist.  We're paying
all of those costs too.

If it helps, think of the analogy of an industrial plant located along
a river.  That plant is supposed to treat its discharge water *before*
it pumps it back into the river...but if it skips the capital investment
in the appropriate equipment and it doesn't hire people to run that gear,
then its profits go up -- because it avoids the costs of remediation.

It's everyone downstream who pays for that, in easy-to-measure ways
like the expense of treating the water, and in hard-to-measure ways like
decreased lifespans due to pollution.  Again, the costs don't go away:
they're just shifted.

> What you're suggesting, in other words, is the digital equivalent of
> "thoughtcrime" - applying ill intent/malignance onto a situation with no
> inside knowledge before any actual violation was committed.

Oh, just stop.  Nobody is asking, nobody has ever asked, nobody will ever
ask for system admins to be pre-cognitive or the Internet police or any
other such thing.  They just need to do their damn jobs, and that starts
with "don't run an operational hazard to the entire rest of the Internet".

How hard is it -- really -- to look at a list of domain names like
the ones above and quickly come to the conclusion that they are up to
no good and that there is absolutely no way any operation should ever
provide them with any services of any kind?

Anyone who can't manage that should GTFO.  It doesn't necessarily make
them a bad person, but it does mean that they don't have what it takes
to be a system or network admin in 2020.


As an aside: I don't like this.  At all.  This isn't how it was, and
this isn't how it's supposed to be.  But this is how it is, despite
the best efforts of a lot of people (including me) who tried to keep
things from getting to this point.  So now we're here, and it sucks,
but it would suck a lot less if everyone who runs anything plugged
into the Internet would put the same level of attention into stopping
outbound attacks/abuse as they do into stopping inbound attacks/abuse.

That's responsible system/network administration 101.  It always has been.

---rsk
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug