Re: [PLUG] spamassassin help: create a rule to score by sender TLD

[Rich Kulawiec <rsk@gsp.org>]

On Wed, Oct 19, 2016 at 01:30:37PM +0200, ac wrote:
> funny. but it proves my point exactly.
> 
> no I am sending from a well known and maintained ipv4 space, from a
> reputable .com mail server... that is blacklisted exactly nowhere -
> anytime in the past ten+ years
> 
> and you are not receiving my email...
> 
> imagine you relied on your email as a business tool (to buy food)
> 
> you would be screwed.

No, it doesn't prove your point.  Let me try to correct some of your
misconceptions about how email and the Internet work (or don't).

With very few exceptions, nobody is obligated to furnish services to
anybody.  (The exceptions are things like (1) government entities required
to furnish services to citizens and (2) people who have contractual
agreements to do so.)  We *mostly* do so because the Internet works
best via cooperation.

And for many years, that cooperation worked beautifully.  We all took
care of our own operations *and* we took care to make sure that what
we were doing didn't have a negative effect on others.  (And when it
did?  We fixed it and apologized and made sure it didn't happen again.)

But that was a long time ago.  It's all different now.  It's quite routine
for large operations to be systemic, chronic sources of abuse and attacks,
sometimes on a massive basis.  Sometimes this is because the people running
those are malicious.  Sometimes it's because they're incompetent.  But
whatever the cause, it still has to be dealt with.  With that in mind,
let me quote one of the best things ever said on NANOG:

	If you give people the means to hurt you, and they do it, and
	you take no action except to continue giving them the means to
	hurt you, and they take no action except to keep hurting you,
	then one of the ways you can describe the situation is "it isn't
	scaling well".  --- Paul Vixie

For a number of years, it was possible to adequately defend mail systems
by enumerating the addresses, domains, network allocations, etc. from
which SMTP abuse emanated and blocking those.  That's because the number
of benign sources greatly outnumbered the malicious ones, and thus it
was reasonable to try this approach.

But it's not reasonable any more, because that underlying assumption
(goodness >> badness) is no longer true.  Not even close.

Thanks to a combination of factors, including botnets, senseless
proliferation of TLDs, hijacked networks, negligently-run cloud
operations, and other things, it's less reasonable every day.  Sad,
unfortunate, annoying, inconvenient...but true.  And not only have
the attack/abuses become far more numerous, they've become much
more virulent and much more capable of inflicting damage.  Anyone
who reads full-disclosure or breachexchange or any of the numerous
mailing lists/web sites cataloging the ongoing litany of massive
security failures knows this because they get new lessons in it all
day, every day.  And email-borne attacks are a big part of that.

So one of the things that anyone who runs an email operation of any
size/diversity and *actually pays attention to their logs* will notice
is that the amount of badness now greatly exceeds the amount of goodness.
For example, at this particular domain on these particular servers,
inbound traffic is about 95% to 98% badness depending on the metrics used.

	[ This also true, by the way, of other services.  It's no
	longer reasonable to leave, let's say, SSH service open to
	the world and then try to reactively block hosts that attack it.
	That is now a deprecated approach in all but a very few use
	cases.  A far better approach is to block *everything* and
	then only permit connections originating from the places that
	real live actual legitimate connections will originate.  This
	is facilitated by using a decent firewall -- like "pf" -- and
	by using network allocations -- like those at ipdeny.com.

	Or as I recently put it to someone, "Why are your logs overflowing
	with SSH attempts originating in China when, in the entire history
	of your operation, nobody has ever actually logged in from there?"

	Yes, this is more work.  But provided you use the right tools,
	provided you use revision control, and provided you pay attention
	to your own logs, it's really not that hard.  I know, I've done
	it many times many different ways for many years.

	Or you can spend a small fortune in time and money and energy
	doing pointless SIEM on things that you would never even have
	seen had you done something sensible like "drop all packets
	from Chinese IP space with destination port 22 because nobody
	is ever going to log in from there".

	Default permit as a security strategy is dead.  Evolve or perish. ]

Thus "might someone with a domain in .me want to send me email one day?"
is no longer the correct question.  The correct question is "do I care
to arrange things so that they can be successful?" and my answer is "no,
probably not".  It's not about you.  It's about what I want.  I can
deny you SMTP service because you have a .me domain or the sending IP
address is an even number or because it's Friday and I felt like it.
You don't get a vote because it's not your service.

Of course I don't recommend being arbitrary and capricious: that would
be foolish.  I made the decision to block .me based on a rather large
amount of data, including detailed logfile analysis spanning a number
of years on a number of diverse mail servers *and* the data I maintain as
part of an anti-spam research project that covers well over 100M domains.

Yes, that might one day have some possible negative consequence for me
or for other people with addresses in this domain.  I'm okay with that,
and it's *my* decision to be okay/not-okay with that because I run it.
Again: you don't get a vote.  You are free to run *your* services
*your* way and block .ru or everyone with a "j" in their address or
block nobody or whatever you want.  And I don't get a vote about that.

Shifting gears slightly:

One of things that lots of people need to get past is that refusing
an email message is a horribly bad insurmountably bad heinously bad
thing.  It's not, provide (a) you do it properly (b) you do it
consistently (c) you do it noisily.  "Properly" means with a suitable
SMTP 5xx response and hopefully with an error message that says
something useful.  (Although some horrible mail systems, like Exchange,
are known to mangle those error messages, thereby destroying their
usefulness.)  "Consistently" means don't accept a message and then
deny another one an hour apart.  "Noisily" means don't just accept
the message and then silently discard it.  These things are important
because *if* it's a mistake, these maximize the probability of
someone noticing it.

And if it's a mistake, so what?  Senders receive a response that makes
it abundantly clear why their message was refused and how they can
report it (if they think that rejection was in error).  They might
choose to do that, they might not: their call.  I think if they're
sufficiently interested in sending email they probably will.
And if they're not?  Almost never worth worrying about.

	[ Side note: in addition to providing a very clear, very
	easy way for senders to report suspected FPs, I also do
	periodic log analysis to look for FPs.  I recommend that
	for everyone: yeah, it's after-the-fact, but of course
	there are no logs to analyze before-the-fact.  This isn't
	a panacea by any means, but I think it's due diligence
	and it's often quite informative. ]

There is far too much pointless hand-wringing over false positives
(real and/or perceived).  That energy is far better invested in
log analysis (as I've repeatedly emphasized in this message and
elsewhere) because unless you have a detailed understanding of
what your mail system is doing (or isn't) you're simply not in
a position to decide how to change its behavior.

Moreover, the issues arising from false positives are dwarfed by all
the other issues.  Read the nanog or mailop or outages or other
mailing lists for a few years, and keep track of the number of
incidents involving mail disappearing into blackholes or mass
mail account compromises or mail being refused (rather than deferred)
because of DNS hiccups or unannounced deployment of not-ready-for-prime-time
measures or or or...

	[ While I was composing this message, a timely thread
	on nanog appeared discussing yet another serious DDoS attack.
	That single attack will probably disrupt far more email than all
	of our combined false positives for the year.  Or recall
	this little gem:

	Yahoo breaks every mailing list in the world including the IETF's
	http://www.ietf.org/mail-archive/web/ietf/current/msg87153.html

	There were probably more FPs generated by that bonehead move than
	any of our mail servers will produce during their lifetime. ]

This is not to say that FPs are never a problem.  Yeah, sometimes
they are.  But compared to all this other stuff, they're really
rather insignificant.  We have much larger fish to fry.  So my advice
to everyone is -- as I said above -- make sure you do refusals
properly and then *don't worry about it*.  Soon enough, any mistakes
you've made will become readily apparent and you can fix them.

Hopefully you'll also learn from them. ;)

And if you have users (other than yourself): abandon default permit. [1]
It's no longer viable.   Carefully answer the question "where do I
really need/want mail from?" and then implement it.  Don't hesitate
to block 37 countries or 281 TLDs if that's what *your* answer indicates
will work for *you*.  Because otherwise you're in the position of my
client (see above) watching the logs scroll by too fast to read,
wasting resources on a problem that can be solved in 60 seconds with
a single firewall rule and a list of allocations in CIDR format.

Or worse, you will be in the positions of other clients who were much
too worried about false positive and far too blase' about the risks
of permitting email from all kinds of dubious sources...until the
day they were burned by it, and it cost them a thousand times more
than any imaginable FP ever could.  So if my recommendations seem
draconian: yeah, they are.  I think it's a better alternative than
volunteering to be the next object lesson.

I don't like that it's come to this.  I argued vociferously against
allowing this to come to pass, but I didn't win that argument.  Oh well.
So now that it's reality, there's no point in denying it or pretending
things are otherwise.  Competent email system defense is now based on
an aggressive default-deny strategy grounded in the operational requirements
of receivers -- not senders.

---rsk

[1] I have to give a good deal of credit to Marcus Ranum, for writing
the single best thing on security I've ever read (and I've read *a lot*):

	The Six Dumbest Ideas in Computer Security
	http://www.ranum.com/security/computer_security/editorials/dumb/

Dumb idea #1: default permit.

___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug