trollout-D17
===05.10.24
Subject: ï INDIRECT DOCUMENTS AT LAST! Now for a Humanist Computer Agenda.
From: Ted Nelson, grebnetugƒxanadu.net
Cited addresses:
transliterature.org/transquoter
xanadu.com.au/transquoter
transliterature.org
transcopyright.org
trans© 2005 T. Nelson, stable at hyperland.com/trollout.txt
and xanadu.com/trollout.txt
Permission is given to redistribute this but only in its entirety.
Dear World:
The tekkies have hijacked literatureñ with the best intentions, of course!-) ñ but now the humanists have to get it back.
Nearly every form of electronic documentñ Word, Acrobat, HTML, XMLñ represents some business or ideological agenda. Many believe Word and Acrobat are out to entrap users; HTML and XML enact a very limited kind of hypertext with great internal complexity. All imitate paper and (internally) hierarchy.
For years, hierarchy simulation and paper simulation have been imposed throughout the computer world and the world of electronic documents. Falsely portrayed as necessitated by "technology," these are really just the world-view of those who build software. I believe that for representing human documents and thought, which are parallel and interpenetratingñ some like to say "intertwingled"ñ hierarchy and paper simulation are all wrong.
This note is to announce a very special and very different piece of open-source software you can download and use now, for electronic documents radically different from anything out thereñ and a bigger plan.
I propose a different document agenda: I believe we need new electronic documents which are transparent, public, principled, and freed from the traditions of hierarchy and paper. In that case they can be far more powerful, with deep and rich new interconnections and propertiesñ able to quote dynamically from other documents and buckle sideways to other documents, such as comments or successive versions; able to present third-party links; and much more.
Most urgently: if we have different document structures we can build a new copyright realm, where everything can be freely and legally quoted and remixed in any amount without negotiation.
It's time for an alternative to today's document systems, and we the loyal opposition have a proposal.
>>>Humanists please jump to transliterature.org, since what follows will be somewhat technical.
But first, some background. This will take a while.
BEFORE THE WEB, A GREATER DREAM
Long before there was a World Wide Web, there was a project with greater intent. This was Project Xanadu*, a bunch of clever, cynical idealists who believed in a dream of world-wide hypertextñ somewhat like the web, but deeper and more powerful and more integrated, rooted in literary ideas, and mindful from the beginning of the copyright problems that would come. The project started unofficially in 1960 when I began to think about world-wide screen publishing, but grew to involve about a hundred participants and supporters over the last half-century.
(Note that I flip between "we" and "I" because this piece culminates work and ideas shared by a number of others over the decades; but I am presently acting alone, so whenever appropriate I am including those others by pronoun.)
Even from the beginning, we planned on unrestricted publishing of hypertext by millions of people; but web-like documents were only the beginning, only one possible form.
The Xanadu project asked at the beginningñ not, "How do we imitate paper?", but "What if we could write in midair, without enclosing rectangles? What new ways can thoughts be connected and presented?" Many ideas and screen maneuvers came to mind, but they always sharpened down to this question: "How can electronic documents on the screen IMPROVE on paper?" And our key answer was: "Keep each quotation connected to its original context."
This idea (now called "transclusion") is the center of our work and the center of my own beliefs. I believe it will give great powers to authors, readersñ and publishers. And transclusion is what we are now delivering.
But till now the world has gone in a very different direction. At Xerox PARC they candy-coated tradition and called it the future. They candy-coated hierarchical directories into 'folders' and candy-coated lump files into font-lavish simulations of paper. (This was all intended, mind you, to support secretaries.) To view it all they created an aviary of ever-flapping windows with no visible connections. (For our radically different connected-windowing proposal of 1972, see xanadu.com.au/ted/TN/PARALUNE/paraviz.html
and download the much-later prototype by Ian Heath, xanadu.com/cosmicbook. See also "The Heart of Connection," in bibliography at end.)
This PARC pantheon of effects and structures (miscalled "the GUI") has taken over everywhere, to the detriment of the world's authors, editors and readers. It has become the standard computer paradigmñ the same on Windows, Macintosh and Linux: the simulation of hierarchy and the simulation of paper as frozen by Xerox PARC, with each document a lump file. Nearly everything on computers today elaborates these traditions. (The most extreme example of gratuitous paper simulation is Adobe Acrobat, a canopic jar to keep documents from escaping.)
These traditions are miscalled "computer basics" as if they were cosmically necessary. I believe today's computer world is based on tekkie misunderstandings of human thought and human life.
The Xanadu project, on the other hand, is based on the structure of connected ideas, which we represented by open parallel data. In the early eighties we found a generalized format and delivery system for all documents, allowing unbreakable deep interconnection (links and transclusions) in many layers and vast quantity. Links may be of many types, which anyone may put on any documents from outside, since they are parallel and external. All links may be followed in any direction (not just two directions, since they can be n-ary.) The markup and links outside the content are what we mean by open parallelism.
Perhaps most important, this method can keep all quoted materials connected to their original sources (our original idea of transclusion). Among other things, this implies a vividly simple copyright system where anything may be quoted freely, because it is easy to arrange the payment of royalty to publishers for those portions brought from different documents. These methods can provide windows, doorways, tunnels into all the world's documentsñ at least those documents opted into this form of rights managementñ making it easy for people to sample and anthologize broadly from the great Niagara of copyrighted materials, and in principle making all documents freely re-usable without copyright violation. ("Freely" in Stallman's sense of "free speech, not free beer.")
This is why veterans of the Xanadu project see today's Balkanized document formats, including the Babel of World Wide Web file types, as way overcomplicated, far too restrictive, and fundamentally broken.
But the Xanadu project went wrong-footed. Along the way we had political/implementational screwups and we lost our place in line. Thirty-two years after we started, another hypertext system caught onñ far more traditional, packaging together the standard traditions of lump files, hierarchy simulation and paper simulation. It bound the links unreachably inside the lump files, making the links one-way.
In recent years the Xanadu project has been derided, disgraced and largely forgotten. That will change. If Xerox PARC was the leading university of software teams, big and conventional and smug, Project Xanadu was the Black Mountain College, small and feisty and defiantly original. Also like Black Mountain, also disbanded, its influence has been much wider than people know.
WHAT WENT WRONG
Project Xanadu progressed slowly but well through decades of no funding, my colleagues creating the great xu88 design in 1980-1. In 1983, because the others demanded freedom to find backers, I signed a deal (the infamous "Silver Agreement") to let the others make the technical decisions provided I could oversee the publishing system (my central concern being our open copyright model). Five years later, backing hit like a hurricane. But sloshed by money, swollen by newcomers and wholly out of my hands, the project spiralled out of control with all the classic mistakes at once: too many cooks, bridges too far, horses in midstream, and Second System Syndrome. And the new people took the software in another direction, digressing from open parallelism.
It all crashed; four years and millions in funding were wasted and the new software was unfinishable. By the time the smoke cleared I was left standing with only the trademark in hand, to pick up the pieces by going back to the previous version. The other participants, less committed, went their separate ways, except for Roger Gregory and briefly a few others.
BERNERS-LEE, AND THE IRRUPTION OF THE WEB
On the day in 1992 that Autodesk funding collapsed, a young man came to see me in my office. He showed me a simpleminded hypertext system he had cooked up. I was polite, didn't say anything negative about it, and took him to lunch. Since then I have watched aghast as this and shallow system, doing only small parts of what we were trying to do (and in a completely wrong way), has taken over the world.
Now, I have great liking and respect for Tim Berners-Lee, who is a good and decent and honorable and very nice guy, of whom I have not the slightest personal criticism whatever. I believe that his ideals are probably the same as mine at some level of abstraction.
All that said, I don't think Tim and I agree on anything in the universe. He bases his ideas on computer tradition: hierarchy, and legacy mechanisms of files and directories. I base my ideas on the nature of ideas and literature and what I believe human beings need for keeping track of ideas and presenting them, for which I believe the imposition of hierarchy, files and exposed directories are highly destructive. It goes on from there.
What Tim could not show me in 1992ñ someone else's work, the other half that made the web take offñ had not yet come out of the cornfields of Illinois. What we now call the web browser was created by gallivanting college students (Marc Andreessen and Eric Bina), brilliant programmers throwing together a random salad of convenient effects as fast as they could. Now their decisions are hallowed, as "web formats" to be cherished and standardized (except, of course, browsers never seem to match). These quick choices made at the U. of Illinois defined the basics of what could be on a "web page" and especially how it would interact.
The World Wide Webñ Tim's early design as boxed up and enhanced by the lads in Illinoisñ has validated all our early predictions for the benefits and wonderfulness of anarchic world-wide hypertext publishing, where anyone can publish internationally, without prior restraint, at very low cost. ("Most people don't want to publish," said arch-publisher William Jovanovich to me in 1966. I said everyone did. "Oh, you mean VANITY publishing," he said. Since he was my boss, I had to stifle the urge to explain that ALL publishing is vanity publishing.)
Why don't I like the web? I hate its flapping and screeching and emphasis on appearance; its paper-simulation rectangles of Valuable Real Estate, artifically created by the NCSA browser, now hired out to advertisers; its hierarchies exposed and imposed; its untyped one-way links only from inside the document. (The one-way links hidden under text were a regrettable simplification of hypertext which I assented to in '68 on the HES project. But that's another story.) Only trivial links are possible; there is nothing to support careful annotation and study; and, of course, there is no transclusion.
For the last decade I have studied high and low, trying to figure out how to fix the web to do what I believe in, looking at server kludges, code embedments, Javascript and Java tweaks, frames, database approaches, blah blah blah, because there was no obvious way to go in the great salad of the web's intricate narrow options. (I believe millions of others experience this daily, but with not enough knowledge to be indignant.)
Finally I realized: nothing further that I believe in can be done on the World Wide Web, period. So why compromise with the World Wide Web at all?
That was the breakthrough. Let's just simplify the Xanadu structure to work in the present environment. BACK TO THE FUTURE! Perhaps another dumbdown of Xanadu can still get traction todayñ de-generalizing it, dropping capabilities in order to piggyback on existing protocols and servers. After all, dumbing down Xanadu sure worked well for Tim Berners-Lee!
STARTING OVER
I have now adapted and simplified Xanadu (reference version xu88.1) to the existing ambient structures of files and protocolsñ
ï as a usable mini-system for transquotation, downloadable now (and yes, it's open source)
ï as the general design of a new document infrastructure, Transliterature*, to support profuse linking, transclusion, and game-like 3D documents.
But Transliterature will have to have new viewersñ fortunately a lot less complicated than web browsers.
Transliterature (see transliterature.org) is intended as an alternative system of electronic documents, zoomable, animatable, with vast numbers of connections where desiredñ including of course the connection of re-used content to its original contexts. Currently Transliterature is a sketchy set of open-source specs for a wholly different kind of document and electronic literature, and the specs are still evolving. Nonetheless, there's enough there that bright kids could easily get up a prototype; my only request is that such prototypes (if any) be brought forward against the specs as they evolve.
I regret not being able to put up a more polished presentation, but time is very scarce and getting scarcer. I would have done the illustrations with a draw program, but hand sketches are much quicker. (I note that Laurence Tribe, Harvard Law professor and Supreme Court advocate, uses his own sketches unabashedly, and so must I.)
THE TRANSQUOTER*, A PRESENT DELIVERABLE
But Transliterature may take a long time. For a quick foretaste, help yourself to the Xanadu Transquoterñ
ï explained at transliterature.org/transquoter
ï downloadable at xanadu.com.au/transquoter
ï programmed by Andrew Pam.
The Transquoter allows you to create a document which quotes dynamically from all over the net (textfiles and web pages) and keeps each quoted portion connected to its original context. Just as we always said, transclusive documents. (Only last summer, a web founder told me this was impossible.) I believe the Transquoter is the first deployed program for dynamic quotation with maintained transclusive connection.
THE TRANSLITERARY PLAN
The Transquoter is of course just the opening shot, a come-on for Transliterature. The Transquoter, and someday Transliterature, will facilitate sending and publishing of new documents drawing freely on pre-existing content, all remaining connected their original contexts.
The transliterary design is client-side for numerous reasons, including both legal issues and ease of deployment. It should be usable by anyone from anywhere, and requires no special servers (though a boost can be provided by the popular EPrints server from the University of Southampton, one of our project hosts.) Its open parallel data structure is extremely simple (streams of content portions to which streams of relations are applied).
Like it or not, discerning readers will have to acknowledge that this new Transliterature design represents a plausible and simple infrastructure for a completely different world of hypermediañ resolutely nonhierarchical, free-form and no longer constrained to the web browser (though it can project to the web browser by limiting its capabilities).
Students write to me for help with their homework, saying they have to write essays about the original 1960 vision of world-wide hypertext, and how does it play out today? Well, students everywhere, the World Wide Web was, let's say, the first 15% of that vision. Transliterature/xu88 should provide the tools for the next 50%, including especially our copyright initiative.
I don't want to kick over the chessboard, just enlarge it. A lot. Do I give a flap about "web standards"? Let's put it this way: I think I feel about web standards the way Tim felt about my standards in 1992. Ask him.
Many will be quick to call the Transliterature design "Vaporware," even though the Transquoter exists. But Transliterature is an agenda, not a promise, and I offer no dates of availability. (I believe something isn't "vaporware" till you've promised itñ a mistake I don't intend to make again.)
The real issue is: are we right? If the Xanadu model is workable, as embodied in Transliterature, perhaps there are big changes in store. I hope people of ability will study this design, which is much more accessible than the Xanadu version on which it is based.
VERY DIFFERENT IDEAS
Transliterature should make possible any shape of document in 3D gamespace or even more dimensions, but we won't go there right now. (What call the client? Perhaps "Flowser*," FLying brOWSER ?-)
But more important than appearance, it should make possible
ï deep profuse overlapping links by anyone, by anyone and on anything, user-selectable
ï everything quotable and connectable and annotatable, both into and from transliterary documents
ï import-viewing of documents from a variety of formats, and maintaining stable connections to them
ï indirect delivery (with its extraordinary advantages, such as unbreaking links to absolute addresses)
ï being able to see all content and connections raw
ï every portion connected to its original source (1-level transclusion. Note that "transclusion" is now defined as "the same content knowably in more than one place")
ï user-selectable views, effects and markup, leaving behind fixed paper simulation (as propagandized by the expression "WYSIWYG"), offering instead the more libertarian WYSIWYL (What You See Is What You Like)
ï 3D animated text, 3D zooms and sworfs (swooping morphs), transparent and fade-in overlays (WYSIWYNC, What You See Is What You Never Could (before))
At least and at last, the Transliterature design offers a simple, workable, lightweight infrastructure to make all these things possible.
Transliterature may seem complicated to those fixated on the web's original simplicity, but in fact Transliterature is far simpler than the baroque web of today.
THE COPYRIGHT ISSUE
Our copyright solution has always been the Holy Grail of the Xanadu Project, and the idea is still good as gold. But first we had to have indirect documents. More is needed, but now it can start.
The world's copyright wrangling is now totally polarizedñ Valenti and Bertelsmann versus a million kidsñ but polarization, with the right glasses, sometimes shows what no one else can see. Our polarized glasses (try them on at transcopyright.org) show a worldñ a possible new community of documents on the Netñ where everyone can re-use and re-mix freely and legally, even with paid content. It requires indirect documents that bring in content by reference (transclusion again)ñ but that is now possible with the Xanadu Transquoter. Which is just the beginning. Please share our vision.
The transcopyright proposal is a win-win solution to support everyone: readers, authors, and later commercial publishersñ in a proposed new system of commerce based on microsale.
A lot of the open source people say, "How dare you be in favor of payment? Everything should be free!" Answer: we're talking about the real world, where content is owned under law and already sold on line; and we are now asking publishers to sell it in minute amounts.
Think of it this way: How can digital rights management be the most open and the most beneficial for all? Transcopyright publishing is our answerñ a daylight and legally valid method within the iron reality of a world where content is sold.
THE WIRED ATTACK
Before I leave the keyboard where I now sit, I must face once again the issue of the Wired attack. The article, "The Curse of Xanadu," was published ten years ago now, but it comes up in nearly every discussion of Project Xanadu and my work. The article is a sewer of lies, concealments and fabrications, steaming with malice, signed by an author whom we may refer to as Gory Jackal.
The purpose of the article was to dishonor and destroy our work, to annihilate our reputations and our ideas, to hide the depth and integrity of the Xanadu project and present us as clueless bozos; to make sure we had no access to respect or funding, even in the dot-com feeding frenzy that was underway; and above all to deny us any credit for the thinking behind the World Wide Web. So far its dastardly purposes have been quite successful.
During the course of the article, Jackal successively implies:
1) that I am a terrifyingly reckless driver;
2) that I am a drug addict;
3) that I am mentally defective;
4) that my every utterance in the course of my life has been incoherent and offensive;
5) that my work was driven by ignorance;
6) that my Xanadu colleagues and I were slap-happy, deluded twits attempting the impossible with toothpicks and string;
7) that my colleague Roger Gregory is an ignorant "repairman" (on account of a job he once had);
8) that we were all clinically insane.
However obliquely averred, these are all damned lies.
Jackal ransacks my life (even my childhood) for suggestive scraps to be presented with loathing and mockery. He has a morbid interest in the contents of my pockets (to which he devotes paragraphs) but not even a perfunctory interest in my ideas, misstating them left and right. So busy is he with his duties as judge, jury, executioner and psychoanalyst that he has little time to get things right, misdating the Silver Agreement by five years, misdating the start of nondisclosure agreements by eighteen years, feigning astonishment at our ups and downs as if not knowing this is how labor-of-love projects go.
EXAMPLE OF HIS CONCEALMENTS: Nowhere does Jackal mention the deep media background I brought to the computer field at the age of 22: that in the late forties I had watched a new medium being born sitting behind my father in television control rooms; that I had won prizes in poetry and playwriting; that I had acted on television and the professional stage; that I had published a book and a long-playing record; and that I had written what was apparently the first rock musical (it ran for two nights as scheduled at Swarthmore College in November 1957). Jackal deigns to mention my 26-minute student film but only to claim falsely that it was unfinished.
My early experience in these many projects across the media board made me extremely confident as a designer and media innovator, and led me to recognize at once the potential of the computer screen and hypertext publishing even long before I saw a computer screen. It was this background that gave me an auteurist, lone film-maker's perspective on how software should be developedñ as a branch of cinema and under the visionary supervision of a director who controls all aspects.
You would think that media background was relevant to the understanding of my work, wouldn't you? Instead Jackal just calls me a "strange researcher."
EXAMPLE OF HIS LIES: Jackal repeatedly describes me as "ignorant," and what are the particulars please? His repeated assertions of my ignorance gurgle down to two claims: that I didn't know enough about advanced software to be discouraged (HAH!), and that I didn't know why others doubted my ideas. These are both abject lies; I knew perfectly well why others didn't think world-wide hypertext could be done, essentially because they didn't want to understand what we were actually attempting.
There are more like that.
Jackal deceitfully and viciously presents my Xanadu colleague Roger Gregory as an ignorant dreamer, repeatedly referring to him as a "repairman" (somehow intended to suggest cluelessness), where in fact Roger is a brilliant generalist and polymathñ and yes, he IS a rocket scientist (see U.S. Patent no. 6,212,876).
Jackal's article deserves careful analysis for the cleverness and subtlety of its deceptions, and I intend it will become known to posterity as a classic of deceit next to the Protocols of Zion. But that is for another day. The hell with Gory Jackal; I think he was just a paid assassin, a liar for hire, and that the real perpetrators were publisher-editors Rossetto, Metcalfe and Kelly, who had all pretended to be friends of mine, and without all of whose enthusiastic support this attack could not have been so lovingly and lavishly mounted. Each of them knew, I am sure, both the article's deceit and the vicious consequences of that deceit for my colleagues and especially me, the only participant who could not leave Xanadu behindñ an all-out full-frontal assault on my life, my character, my intellect, and everything I stand for, hope for and believe in.
There have been various conspiratorial theories about why they did this, but I that's not necessary. It was simply a witch-burning.
As the principal celebrants and sycophants of the World Wide Web, Wired's Gang of Three had a vested interest in silencing dissent. By burning us in public they reinforced the prejudices of all their readers and assured them that the prevailing computer paradigm was not challenged and that they would have to learn and understand nothing new.
To hell too with Rossetto, Metcalfe and Kelly. But my concern is that out of snottiness, malice and Schadenfreude, by inflicting these setbacks and disgraces to our work, these unspeakable individuals may have destroyed one of the few great possibilities the human race ever had: an electronic publishing network where contents could be freely combined and remixed under legal copyright, with each portion being purchased from the original, and everything deeply linkable and annotatableñ while being changed. All this works in the Xanadu and Transliterature designs.
I have no choice but to fight on.
Transliterature is announced as an open source project, but there's no schedule and no money and no people. The Oxford Internet Institute has provided a wonderful breathing-space in which to pull together this work, but mundane pressures will shortly slow me down. This may not be finished in my lifetime. But the important thing is to start.
Thanks partly to Wired, the fascist formats have largely taken over. But perhaps this can be D-Day.
Ted Nelson
===THANKS
My nearly two years at the Oxford Internet Institute have given me time and breathing space for this work, and Wadham College (Oxford) has provided the Fellowship, in both senses, to help me go on. Special thanks to William Dutton and Dame Stephanie Shirley of the Oxford Internet Institute, to Wadham College (Oxford) and Kenneth Woods for supporting my fellowship there, to Wendy Hall and the Department of Electronics and Computer Science at the University of Southampton for support and inspiration during my residence in Southampton; to Chris Gutteridge and Stevan Harnad at the University of Southampton and the EPrints project for adding portion and context service to the EPrints server, to Helen Ashman and the Department of Computer Science at the University of Nottingham for my first office that had a view of swans. Extreme special thanks to my brilliant, amiable and most ingenious collaborator, Andrew Pam of Xanadu Australia. Thanks to my many other recent collaborators and well-wishers in the United Kingdom, Japan, Finland and France. Thanks to about a hundred alumni and supporters of the Xanadu Project, and especially its principal designers Roger Gregory, Mark Miller and Stuart Greene. Thanks to Doug Engelbart for his great and enduring inspiration. And thanks most of all to my collaborator, fellow traveller and sweetpartner of long standing and patience, Marlene Mallicoat.
===BIBLIOGRAPHY
Defining book on Project Xanadu: Theodor Holm Nelson, 'Literary Machines'. 1981 and later editions. (Editions since 1985 describe the tumbler addressing structure of xu88 in some detail; omitted was the secret that all internal mechanisms are based on tumbler arithmetic, e.g. rearrangement by permutation matrices of tumbler spans.)
Peer-reviewed ACM article on the history of the Xanadu project, http://portal.acm.org/citation.cfm?id=346033
If you don't have an ACM account, http://xanadu.com.au/ted/XUsurvey/xuDation.html
Peer-reviewed ACM article on transpathic media, TN's "The heart of connection: hypermedia unified by transclusion" at http://portal.acm.org/citation.cfm?id=208344.208353. See also Krottmaier, "Issues of Transclusions," http://coronet.iicm.edu/denis/pubs/elearn2002b.pdf.
Peer-reviewed British Computer Society article on TN's ZigZag* nonhierarchical database and software engine, http://jodi.tamu.edu/Articles/v05/i01/Nelson/
=== LEGACY XANADU SOFTWARE, NOW OPEN SOURCE
Code for referential Xanadu xu88 (now also called "Udanax Green") is maintained by Roger Gregory at Udanax.com, and now said to be working. It is reported that Jeff Rush has translated it into Python. I look forward to merging Transliterature with fully-functional xu88 Xanadu, but who knows when.
See discussion of Xanadu principles (especially enfilades) at xanadu.com/tech/. A good deal of Xanadu documentation will be found at sunless-sea.org, maintained by Jeff Rush. However, in that documentation you may find some confusion between Udanax Green (xu88) and the very different Udanax Gold (xu92), both being conflated as "Xanadu" without distinction.
("Udanax Gold" or xu92, the completely different design done in the Autodesk period, is not compatible with Transliterature or Xanadu xu88. It is considered by some to be of great theoretical interest, but it is very very far from working.)
==="ON THE SHOULDERS OF GIANTS" DISCLAIMER
Transliterature is an adaptation and simplification of referential Xanadu (xu88), which was designed by Roger Gregory, Mark Miller and Stuart Greene. That design was their ingenious fulfillment of the wish list and specs we worked out in the Xanadu design summer of 1979.
It is high time they get full credit for the depth and brilliance of their full design, which is much deeper than Transliterature.
=== * ABOUT TRADEMARKS
Trademark law offers excellent benefits to the little guy as well as big corporations. The following I claim as trademarks for open source software to avoid semantic creep: Transliterature, Transquoter, Flowser, LUSTR, Transcopyright. The following are registered software trademarks for commercial purposes: Xanadu, ZigZag.
ó30ïñ
span: