ANDF - some information, part 2

I am the DRA person who put out the long posting giving some information
on ANDF/TDF. This was posted to many different Usenet groups. It has
stimulated some interesting correspondence. We are answering technical
details directly, but I thought I would from time to time draw a number of
issues together and put them into a general mailing. This is the first
such:


There have been comments about how ANDF would affect the market place.
The issue that ANDF potentially destroys segmentation and will drive down
prices is quoted as one reason why system vendors are not rushing to
support ANDF. My view is that what we are seeing in the U**X market is the
maturing of this market into a high volume / low (per unit) value,
commodity market place. This move is inevitable but it will create winners
and losers. System vendors have to decide the balance between keeping
their profit margins up for as long as possible vs being seen as missing
the mass-market boat. Different system vendors have taken different
viewpoints, and within each system vendor one finds groupings that span
the spectrum from protectionist to expansionist. I think the simplistic
view that system vendors are against ANDF is just that - simplistic.


What is true is that ANDF potentially accelerates the commoditisation of
the market and hence arouses strong emotions. DRA has always believed that
we would have to dig in for a long campaign. Our strategy is three-fold:


- get portability products out of the door so that ISVs can use them to
improve the portability of their products, even if they continue to ship
in binary. ISVs will then have first hand experience whether their testing
on individual platforms has ceased to throw up problems. If fullscale
testing is found to be unnecessary then the ANDF scenario becomes a risk
that ISVs might be willing to take, possibly with a smallscale
qualification test run at installation time.


- sell the compiler technology that we have developed to create an income
stream that underwrites the ANDF programme at DRA. Having established such
an income stream we can present ANDF as a technology that is not going
away - undermining those opponents who hope it will just roll over and
die.


- work with the user-oriented standards organisations to put pressure on
the suppliers. The use that TenDRA can play as an API checker/enforcer
makes it very attractive to such organisations.


Does ANDF make piracy easier? First of all we expect ISVs to start using
our tools whilst still shipping in binaries. Once ANDF is used for
distribution we would expect the applications to use licensing managers to
control piracy.


Does ANDF really support many languages or are there language specific
installers?


Our objective was to achieve no performance degradation for C, C++,
Fortran and Ada. Each producer defines its own token definition library,
but we would expect all installers to support these four languages (other
language producers may generate ANDF tokens for language idioms and supply
default token expansion definitions, but particular installers could
supply token expansions that are more efficient for that target). For
these other languages we are having to balance performance against
increased complexity in installers / token definition libraries. We should
however be able to do very much better job than those portable compilers
that output C. Just how good is the subject of substantial research effort
(with the generous support of the European Commission's Esprit programme).
We have done exploratory work on dynamic languages and are working with
Micro Focus on Cobol. If I had to speculate I would say that we will be
able to support portable compilers for the languages mentioned on the
Usenet which produce sufficiently good code that most users would find the
performance acceptable. ANDF is designed to be extendible, so it is
possible to adapt to improve performance for important new languages, but
such extensions will have to be well proven by research work before being
adopted. To that extent we may be seen as breaking the stranglehold of C
and C++.


Can ANDF be used for persistent and distributed OO systems? We have made
great efforts to make ANDF as flexible as possible. We are definitely much
better than using C. Until ANDF is used in anger for these interesting
applications we cannot be sure the extent (if any) of ANDF's limitations.
The open process by which ANDF has, and continues to be, reviewed by the
community should help ensure that as many possibilities as possible are
considered.


An interesting question was raised concerning ANDF - would it have been
better to modify C so that it could be used as an ANDF rather than
starting from scratch?


The first point to raise is that the great attraction of using C is the
universal availability of good compilers. This means that any changes to C
would have to accepted and implemented by the whole industry - not a
trivial issue to put it mildly!


A second point is that C was not primarily designed for this job and it is
a bit like using a screwdriver to pound nails. It is however an
illuminating exercise to contrast how a modified C would compare with
ANDF:


First, consider code efficiency. A number of other emails have mentioned a
list of language features that need to be added to C for efficient
compilation of those languages that need them (there are a number of
obvious candidates for inclusion that have been mentioned in other
mailings). In addition to these major items we have uncovered many
detailed areas where generation of C and code generation through an
existing C compiler generates grossly inefficient code for a particular
class of input programs. Either these areas must be catered for by
extensions to C, or alternatively C compilers should be advised to
recognise particular patterns of C that are generated by producers for
other languages. As an example tuning DRA's C compiler (producer plus
installer) for the F2C output of a SPEC 89 program (matrix300 - without
the pre-processing trick) improved runtime performance by a factor of 2.
Thus C would have to be extensively extended and/or compilers tuned for
generation by other languages. This is possible but would be a big effort
- unsurprisingly an effort comparable to the effort that has gone into
designing and implementing ANDF.


Secondly, consider the header file problem I described in the first "ANDF
- some information" mailing. Namely that existing headers for APIs are in
fact implementations for a particular machine, and make unspecified API
details, such as order of struct members, visible. C would have to be
extended in a similar manner to the DRA producer to allow architecture
neutral headers to be handled. This raises an interesting issue - that
DRA's C producer implements a non-standard extension to ANSI C. There are
two reasons we can get away with this:


1. Because of ANDF our producer can generate code wherever there is an
installer, and the large investment in installers means that soon all
major architectures can be reached.


2. We are ensuring that ISVs can create a single source tree for their
products which can easily be specialised to be compiled by a non-ANDF C
compiler.


(note : our extensions are in the Public Domain and we would be happy to
offer them for standardisation.)


A third point is that most existing compilers are not designed with
machine-generated C in mind. This manifests itself in many ways:


- machine-generated C can cause C compilers to crash.


- Getting debugging information into a foreign language front-end is hard.


- automated installation if the machine-generated C is distributed as an
ANDF is difficult.


Fourthly there is the practical problem of having to do a full C
compilation, including lexical analysis and parsing the C, which can make
compile times for the language generating C excessive. Machine generated C
can be very voluminous.


Fifthly, C as a human readable stream of characters is a bit of a pain to
generate, whereas the tree-structured ANDF is designed to be easy to
machine generate. In addition ANDF is easy to machine process (e.g. as
ANDF to ANDF portable optimisations). Limiting the generated C to be
within ANSI C's minimum translation limits would be very difficult.


In summary, extending C to be an ANDF is technically possible - in fact
the technical design of ANDF is not dissimilar from an extended C.
However, the effort involved would be comparable to that which had to be
spent developing ANDF. Because of the constraint imposed by starting from
C the result would be slightly less efficient and much less convenient to
use. In addition ANDF is purpose-built and so is designed for its role -
this gives it a number of advantages (for example it is designed to aid
program construction from ANDF fragments). The main benefit of using C
would be to reuse much of the existing implementations of C compilers -
however the design level of ANDF, which is comparable to C, means that
reusing existing compiler technology is relatively easy.


We suspect that if we had implemented ANDF as an extension to C we would
probably have implemented it using an intermediate language very like the
current ANDF.


I will conclude this (overlong) mailing by answering directly some
questions I spotted in the last couple of weeks:


"something like Gimpel's C-shroud would fill the need"
(Robert Dewar) 2 Dec 93 15:49:48 GMT


C-Shroud does not address either the portability or API conformance part
of the distribution/portability/API triangle. Nor does it offer any
multi-language capabilities beyond C (see discussion above).


"technical success but a commercial flop"
(Donald Lewine) Thu, 2 Dec 93 13:48:38 GMT


Thanks for the technical success bit, we are working on the commercial
flop aspect. In fact I think commercial flop is wrong - that implies it
was launched and failed, when in fact it was never launched (it was almost
a question of strangled at birth) . We are working with others to do a
proper launch of useful products that will initially address the
portability/API parts of the distribution/portability/API triangle.


"sell software ... never even *tested* my software on these
yet-to-be-released platforms"


and


"the cost of *creating* software far outweighs the cost of reproducing
or distributing it"
(Ronald F. Guilmette) Wed, 8 Dec 1993 10:45:38 GMT


On the first point I would reiterate that putting out products that
address the portability/API part of the triangle which can used for
conventional binary distribution, will allow ISVs to assess whether ports
of products do consistently work first time. (Obviously) we use our own
stuff internally and there is nothing like having things work consistently
first time for increasing confidence.


On the second point, ISVs have not agreed with this view in conversation
with us. Porting and testing are sometimes larger costs for ISVs than
software creation. Distribution is smaller but perceptible. It all depends
on the size of the ISV what really hurts. For example, for some small
ISVs the costs of buying and running the necessary hardware platforms is a
significant consideration. In general I would discourage people from
simplistic statements about ISVs costs - the picture is far from simple.


"ANDF may be a reasonable delivery mechanism, but I don't see that it
addresses the really hard problems in writing software that runs on Unix
systemS" (ie API usage)
(James Larus) Fri, 10 Dec 1993 14:08:16 GMT


"serious vendors who statically link all math libraries into their
application, because they won't take the chance of linking with a
dynamic-shared-object that might possibly be changed on them"
(John R. Mashey) Thu, 9 Dec 1993 02:32:33 GMT


I have lumped these together because I think the second point is a
detailed version of the second. The problem in the Open Systems world is
that there are (supposedly) homogeneous interfaces (APIs, libraries) from
heterogeneous suppliers. In a homogeneous (proprietary) world such as
Multics, dynamic libraries were a no-brainer. Getting UNIX interfaces as
solid as in a proprietary world is (in my view) essential to the long term
success of UNIX. ANDF tools address the issues of static API checking (of
both applications and implementations). Combined with a technology for
better semantic test suites for implementation branding (such as ADL -
Assertion Definition Language, and TDD - Test-data Description language,
from the PrimaVera group at Sun Microsystems Labs Inc) it is possible that
the API issue could be solved.


That all folks!


Nic Peeling


TenDRA Project Manager


DRA Malvern


email: peeling@hermes.mod.uk (internet)
              peeling@dra.hmg.gb (alternate internet address)
--

Return to the comp.compilers page.
Search the comp.compilers archives again.