Dustin Getz on 16 Nov 2011 10:15:31 -0800


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: is lisp the answer? /linkbait


but can we generalize software systems construction such that, in future universities, we can teach "software is constructed by composing three fundamental operatations: map, reduce, filter"? is it correct to say, that in the future, software constructed with patterns other than this will be disdained as against common knowledge, in the same way that everyone knows that assembly isn't the tool you use for a web app? if we could establish the fundamental absoluteness of sequence-oeriented abstractions... would systems integration work become, as you say, "factory work"?

systems integration is hard because the underlying systems aren't designed to be composable, or at least in big systems it never turns out that way. i wonder if systems never turn out composable because they are implemented using patterns that aren't composable (by people who don't undertand referential transparency, or even know what it is). and if all systems were implemented, from the smallest, most tactical level, all the way up, using composable patterns -- referentially transparent patterns -- the natural outcome is composable software components. which would mean "finding the right glue" isn't hard anymore.

On Wed, Nov 16, 2011 at 12:04 PM, Aaron Feng <aaron.feng@gmail.com> wrote:


"The plumber is often the most experienced programmer on a team. ...
software plumbing connects things together. It deals with things other
people don’t want to see or think about. And it’s crucial." and he
implies that systems integration is one of the hardest parts of
software (systems?) engineering.

It is an interesting description, and I feel it's very true.
System integration (aka enterprise development), is definitely
non-trivial, and time consuming.  With my experience, using
the correct tool (might not be the most exciting thing technically)
can reduce that complexity down to factory work.
 

A HN commenter `nadam` wrote: "Most of the enterprise developers' job
is connecting components together. I hate it. I hate modifying
existing systems and hate connecting them together. It is usually
hard, but the kind of hard which is not really fulfilling for me. It
is about fighting with accidental complexity created by other people."

I feel it's not fulfilling to most people because there's no quick wins in
these tasks.
 

`fleitz` (5800 karma) responded (summarizing): "It's not that
difficult when you work with the right tools. Most of these problems
boil down to a couple operations: map, reduce, filter ... [to and from
webservices] ... [only need four types to do this] ... Using the usual
sequence operators pretty much anything is possible with those inputs,
outputs, and transforms (pipes) and you avoid a lot of overhead making
data pipelines." [2] worth a click imo.

I responded, questioning his assertion that "most problems can be
solved with sequence operators". I would love if this was true, but i
see no evidence of "big" software systems being successfully
constructed in this fashion. "People build BIG BALLS OF MUD because
they work. In many domains, they are the only things that have been
shown to work. Indeed, they work where loftier approaches have yet to
demonstrate that they can compete." (quoting an old essay, this
actually might not be true anymore! facebook/twitter/google/amazon)
[3] I elaborate a bit in the HN thread.

Integration is often (always?) messy.  I feel the same way about software in general.
There's no such thing as "pretty" non-trivial programs.  It might start out that way, but
the quality usually degrades over time.
 

Anyway, I can't help but wonder if the lisp paper we discuss tonight,
and the underlying lambda calculus, forms a proof that yes, all
computable problems may be decomposed into sequence operations, and i
wonder if that proof can be extended to convincingly conclude that
sequence-oriented decomposition of systems-integration problems is a
better* approach than big balls of mud. I recognize that many
intuitively feel this is true, but I have yet to find and understand
an argument that concludes this beyond all doubt.

I don't necessarily think when it comes to system integration, there's one tool
that is the right answer.  It's highly dependent on the problems you are solving.
In most cases, it's just using the "right" glue to bond everything together.

If someone can generalize all integration problems, please let me know :)

Thanks for sharing,

Aaron