This has been an interesting discussion to follow, and I look forward to seeing
what comes of it in the long term. However, I feel that one thing doesn't seem
to have been raised in any detail, and this is the question of who writes the
applications code for these packages.
Generally speaking, certainly with the big monolithic packages (I don't mean
monolithic in a technical sense of a single program as opposed to a number of
images executing in some coordinated and cooperative way, I mean monolithic
in the sense of being a homogeneous whole, like IRAF) there are the people
who create the infrastructure for the system, the people who write the
individual applications using that infrastructure, and then there are the
people who use the results.
There's often an overlap, of course. There are
end-users who love tinkering with computers and writing their own neat stuff,
and they'll get into the system and create new applications. There are other
end-users - even in this very computer-literate world of astronomy - who
want nothing whatsoever to do with what's inside a package, who's only concern
is that it reduce their data quickly and easily. Many of these are quite happy
to learn complex incantations of commands, but don't want to have to understand
what's going on inside - and they don't have any patience with technical
explanations that such and such can't be done because it would violate some
internal layering constraints of the software. I have a lot of sympathy with
these users.
What sells a system to most end users is applications. It isn't
necessarily ease-of-use, it isn't the flexibility of the file format, it isn't
the elegance of the scripting language, it most certainly isn't the fact that
it's built using object-oriented theories of software design. Most importantly,
it isn't the potential of the system that will sell it, it's the actuality.
Now, whatever system gets designed, someone has to write the applications. And
if you have a system, you have a set of rules for writing applications. You
have an API. Am I wrong in thinking that no matter what sort of a system you
end up with, no matter what sort of rules you define for interoperability and
all the rest, you have to end up with an API? When someone comes to you and
says, "How do I write an application to do such-and-such?" what are you going
to tell them?
So you have to sell a system to applications programmers (or you have to
employ them yourself, which is an expensive way around the problem). That
means the API has to be able to do what they want it to do, and it has to
be easy to use it to develop new applications. Making it easy to use is
mostly a case of providing good documentation, particularly friendly
stuff that explains how to write a simple application and then goes on to
show how the bells and whistles are added. It's also a case of providing
enough good template applications.
Making it possible to do what they want it to do can be very complex, and
I think it's been getting harder recently. It's my observation that most
'ordinary' astronomers who have an idea for an algorithm and are sufficiently
programming-literate to code it, want to write something
tightly integrated. They want to write something that will pick out the stars
in their image using their new algorithm, but they want to be able to interact
with the algorithm to help it along, so they want to be able to control the
display, they want to be able to draw a red line around the stars they found
and so on. And they want a prompt-response phase as the program lets them tweak
the way it works. This used to be easy (you were writing for a specific machine
with a specific image display and there were always ways of writing directly to
the display if the API didn't have exactly what you wanted). Now we have much
'better' systems (we think they're better because they're more portable, they
define strict layers between the application and the 'virtual' display - which
can now be almost anything running on almost any type of machine) which
won't let them do this because you can't even be sure the monitor can display
a red line.
I don't have any good solutions to this, but I think it's a problem. Almost
by definition - it's a selection effect, like so much in astronomy - the
people reading this message (if any) the people submitting these ideas
about scripting languages and OOP systems, the people who'll go to ADASS and
attend the BoF discussion, are all people who are not like their typical
prospective user, and are (the main point of this message) not even like their
typical prospective writer of applications. And yet the success of the system
will depend on its acceptance by the typical end user, and this will be
to a large extent determined by its acceptance by the people who are to
write the huge range of applications for it.
On a couple of occasions I've used the word 'sell' here. It's a useful
way of looking at things. If this were commerce, if we were designing a new
operating system for a PC, we would be trying first to sell it to the
applications vendors. One you've persuaded everyone to write applications
for your platform, you're made - which is why Apple are worrying that people
are writing for OLE instead of OpenDoc, no matter what the technical merits of
the two might be. Of course, you sell things to applications vendors by
persuading them that all the end-users will use this platform, so the thing's
circular.
Of course, you do need sound technical decisions as the foundation of
what you're trying to build. You need a good, comprehensive, flexible file
system (hierarchical, extensible, self-defining, preferably) because you
need to be able to handle the programmer who wants not only the error
arrays and the quality information you've already thought about, but also
need the list of fibre locations from the spectrograph (which you'd not
thought about until right now). You need a display sub-system that you can
interact with in detail - that will let you draw a red line around the
pixels you want and will let you move the cursor explicitly to a point in
the image, and all the other things you can't always do easily. You need
a way to build a complex application out of simpler applications using a
scripting language that an ordinary user can understand (how many of your
astronomers can use awk or even perl with fluency, by the way?) You need
to make it possible to write large integrated applications that let you
move the cursor to a point in the graphics window and answer a prompt at
the same time because that's how a large subset of potential applications
writers want to write applications. You need to be able to write an application
in a language that most target application writers already know or can be
persuaded it's in their interests to learn.
But you need to be able to do this without producing an API whose documentation
comes in too many volumes to fit on my bookshelf (or in my mind, just as
importantly) and whose learning curve is so almost vertical. I'm sorry I'm
having to say 'you' - I'd like to be there myself.
Good luck, I'll look forward to seeing the outcomes.