Editorial File for Top Level of ECI/MCI
=======================================

Most of this document is directed at issues relating to the use of
Text Encoding Initiative (TEI) compliant markup using ISO-8879
(Standard Generalized Markup Language (SGML)).  Those not interested
in these issues should concentrate on sections I, IV, V and VI.

I. Structure of the ECI/MCI

Conceptually, and as a TEI-compliant SGML document, The ECI/MCI is
structured as a hierarchical tree of corpora, which in turn may
contain further sub-corpora or documents. A document may in turn
consist of several computer files.

	ECI-CORPUS
	  ECI-TYPE
	    CORPUS
              [SUB-CORPUS]
		DOCUMENT
		    FILE

This structure is mapped more or less straightforwardly onto the
directory structure rooted in the data directory of this CD.  The
details are described in the file doc/dirstrct.txt.  An introduction
to other directories on this CD can be found in the README file.

II.  ECI/MCI Corpora

A brief summary of the contents of each corpus can be found in the
file doc/corpdesc.txt.

The corpus as a whole, and each of its component corpora, are
accessible as SGML document instances with Doctype eci.corpus, using
the DTD lib/tei/eci.dtd, which is to a large extent parasitic on a
preliminary version of the TEI P2 release (as represented by the files
publically available as of 5 January 1994 (hereafter referred to as
P2[5.i.94])), using the prose and corpus specialisations.  Individual
documents below the sub-corpus level are of Doctype eci.  eci and
eci.corpus are aliases for TEI.2 and teiCorpus.2 respectively.  As
such the corpus as a whole, each component corpus or sub-corpus and
each document has a TEI header giving information about its form and
content.

III.  Element Structure

Although the ECI DTD is based on the TEI P2[5.i.94] DTDs quite closely, and
directly incorporates them with few changes (see section VII below),
only a small amount of the richness provided by the TEI is actually
made use of.

Most of the corpora have only a modest amount of markup in the body of
their documents, typically using <divn> for basic structuring of
components (chapters, issues, articles, etc.) and <p> for basic
textual units.  'rend' attributes are used to record aspects of the
original markup thought to be of possible value.  Corpora of type eci1
(see doc/dirstrct.txt) have little or no non-SGML-encoded structure,
e.g. using blank lines or indentation.  In contrast corpora of type
eci2 have their structure only partially captured in SGML markup, and
may include a range of more or less idiosyncratic indications of
structuring and/or rendition.

IV.  The Text/Markup Invariant

All the corpora of types eci1 and eci2 observe what we call the
Text/Markup Invariant: Every line in a data file (.eci file) is either
all text or all markup, and a line is a markup line if and only if it
begins with a left angle bracket (<).  This makes processing only the
text from data files, without any markup, straightforward.  It has
the negative consequence that line breaks from the original material
are not preserved, particularly where relatively low-level markup
occurs in the midst of text, e.g. for quoting or highlighting.

V.  Character Sets

The majority of the data in ECI/MCI is encoded using the ISO-8859-1
(ISO Latin 1) character set.  Some use is also made of ISO Latin 2
(for Czech), ISO Latin 5 (Cyrillic, for Bulgarian and Russian) and 7
(Greek).  All of these character sets have 256 characters, i.e. they
use 8 bits per character.  They are also all virtually identical to
ASCII for the first 128 character codes.  Some support is provided,
mostly for UN*X environments, for displaying and printing the full
character inventory of these character sets -- see src/unix/isoscrpt
and lib/fonts for more information.

VI.  The Use of Entities for Unusual or Unknown Characters

A limited use is made of the SGML general entity mechanism whereever
necessary.  This occurs in two general kinds of circumstances: where
the basic syntax of SGML and/or the Text Markup invariant disallows
certain characters, and where a document uses characters not present
in the ISO-8859 character set being used, or the character in ISO-8859
appropriate to render a character from an original
non-ISO-8859-encoded corpus is not clear.

Three entities are used in the first case: amp, lab, and sq, standing
for ampersand (&), left angle bracket (<) and single quote (')
respectively.  The first is used in plain text in place of any
ampersand which would be taken by SGML to introduce an entity, i.e.
where it is followed by an ASCII alphabetic, period (.) or dash (-),
e.g. "to&fro" in the original will appear as "to&amp;fro", but note
that e.g. "Little & Large" will appear as is.  Left angle brackets are
similarly replaced with invocations of lab, e.g. "<nnn>" as
"&lab;nnn>" when no more systematic semantically-based markup has been
used, e.g. "<quote rend='<<>>'>Okay</quote>, she said.".  In a very
small number of cases where to leave a left angle bracket in text,
even though it did not otherwise require replacement, would violate the
Text/Markup invariant, the lab entity has also been used, e.g.

    . . .
    </hi>
    <164> was the next relevant part . . .

  becomes

    . . .
    </hi>
    &lab;164> was the next relevant part . . .

Finally in a very few cases where the value of an attribute contained
both single and double quotes, the single quotes were of necessity
replaced with the sq entity, e.g.

    <omit desc='"That&sq;s a lie", a;kjdf said' reason=garbled resp=eci>

Note that the -e switch to bin/unix/textonly will replace these entities
when they occur in text with their definitions.

The second kind of entity use arises because many of the corpora
included here came to us in character sets other than ISO-8859-n.  In
some cases characters occured in the source character set whose glyph
had no character in the appropriate ISO-8859 set, e.g. the m-dash, for
which we have used the entity mdash.  In other cases we simply could
not be sure what glyph was intended, but leaving the character code as
is would result in confusion, e.g. the character whose glyph in
ISO-8859-1 is the division sign (÷) occurs twice in the ger03
corpus, but is presented using the entity divide as whatever was
intended it is certainly NOT the division glyph.  Finally in a few
cases two different codes were used with the same apparent glyph
intended, in which case only one was translated as the character for
that glyph, and the other(s) presented using an entity.

All these cases are discussed in the individual .edt files for the
corpora where they occur, and the entities are defined in the
corresponding .ent files.

Note that using the -ecorpid switch to bin/unix/textonly will replace these
entities when they occur in text with their definitions.

VII.  Changes to the TEI P2 Distribution

We have made only a small number of changes to the P2 DTDs as they
were publically available on 5 January 1994.  In almost all cases
these changes have been effected using the mechanisms provided by the
P2 DTDs themselves for this purpose.

The following sections discuss changes made across the board, in files
found in lib/tei.  Local changes required for individual components
are to be found in their .ent file, documented in the associated .edt
file.

VII.1  Changes to P2[5.i.94] NOT Using the Approved Mechanisms

The most pervasive change of this sort is to convert all external
entity references to PUBLIC, so that file-system dependence is not
built-in.  This of course means that any use of this material requires
an entity manager which can be directed to find files in the right
places: bin/unix/eci and src/perl/ecipath.pl (q.v.) implements this
for sgmls.

The other major unapproved change was to the definition of the
parameter entity %specialPara, which is used for the content of
elements such as <quote>, <note> and <corr>.  In trying to allow for
either a sequence of paragraphs or plain text but not both, the
original P2[5.i.94] definition fell foul of the dread "mixed content"
problem, with the result that material in the standard ECI form of one
tag per line was invalid, e.g.
  <quote>
  <p>
  . . .
  </p>
  </quote>
The change we have made avoids this problem by allowing arbitrary
mixtures of paragraphs and plain text as the content of such elements,
which is an infelicity of a different, but we feel a lesser, kind.

Three minor changes also had to be done by direct editing, namely adding
a 'type' attribute to the <p> and <byline> elements, and returning the
default for the 'type' attribute in %a.divn to #IMPLIED (from #CURRENT).

VII.2  Changes to P2[5.i.94] Using the Approved Mechanisms

P2[5.i.94] was a transitional release, which in our judgement was not
quite self-consistent.  The entities %n.seg and %n.gap were therefore
redefined (in eci.dtd) to "s" and "omit" to preserve consistency with
prior usage.  These changes are identified as such in the eci.dtd
file.

Other minor changes increased the range of allowable locations for the
<omit>, <dateline>, <q> and <time> elements.

Finally two new elements were introduced:  <tail>, parallel to <head>
in allowed content but at the end of divs instead of the beginning,
and <div8>, as in a few cases we actually needed a div below <div7>,
and the definition of <eci> was slightly changed from that provided
for <TEI.2>, to allow for entire corpora to be included as SUBDOCs in
a relatively well-structured way.

VII.3  Further customisation

The eci dtd begins with the inclusion of an external public entity
"eciover", which by default resolves to an empty file,
lib/tei/eciover.ent.  But if you need to override any aspect of the
eci dtd, you need only arrange the search path of your application to
find an alternative resolution for that inclusion, wherein you may
include such overriding declarations as may be required.  Note that
for the use of sgmls provided by bin/unix/eci and oneeci, the
environment variable ECI_PATCH can be used to assist in this (see
bin/unix/eci for further information).

VIII.  SGML Processing of Part of the Corpus Only.

The top-level mci.sgm file can be viewed on the one hand as simply a
locus of documentation and bibliographic information.  But it also contains a
rich structure of marked sections controlled by parameter entities,
which can be used to allow processing of various subsets of the
complete ECI/MCI contents.

All these entities are all given default "IGNORE" values in mci.ent,
so by default processing mci.sgm will actually result in an error
after displaying the header, as there is no data content in this case.
But this can be overriden by setting one or more of these entities to
"INCLUDE", using e.g. command-line -i arguments to sgmls or by
interpolating a non-empty file to resolve the search for the public
external entity "eciover" (see above) to provide pre-empting
definitions.

The following types of control parameter entities are provided:

1) Individual corpora entities:  For every component, there is a
parameter with the same name as the component, e.g. %bul01, %mul05,
%eng22.  Setting these to "INCLUDE" will produce the material from
the named component.

2) Language entities: For every language represented in more than one
component, a parameter with the three-letter language code, e.g. %eng,
%tur.  Setting these to "INCLUDE" will produce material from all
components whose primary language is as specified.

3) Corpus type entitities: For the four corpus types 1-4 (see section
III above and doc/dirstrct.txt for more information), there are
appropriate parameters %eci1, %eci2, %eci3 and %eci4.  Setting these to
include will produce all components of the given type.

4) Finally, more for completeness than any expectatation of utility,
the entity %all is provided, which if set to "INCLUDE" will give you
the Whole Thing.

For example, providing an override file with the contents given below
would direct an application to all the dutch data as well as the
english data from the mul05 multilingual corpus:

<!ENTITY % dut "INCLUDE">
<!ENTITY % eng15 "INCLUDE">

IX.  Taking the Covers Off: The Entity and Inclusion Structure of ECI Corpora

This section is for SGML afficianados only, and applies to those files
which are marked up using the eci dtd.

In order to render the amount of non-data SGML material in data files
to a minimum, .eci files do not contain TEI header material
themselves, but rather are all intended to be slotted in as the
content of the <body> element of a standard TEI document instance.
The basic TEI framework of header and text is (indirectly)
instantiated in a .sgm file which (indirectly) references the .eci
file(s).  In order to avoid unnecessary repetition of material in the
.sgm files, heavy use is made of entity references.
Component-specific header information is actually located in .ent
files, whence different amounts are retrieved depending on the amount
of material being processed.  The framework of entities and marked
sections which accomplishes this is spread across lib/tei/eci.dtd and
lib/tei/ecisubc.ent.

The basic inclusion structure which results from all this looks as
follows, at the file level:

For whole corpora:

xxxnn.sgm [via doctype] eci.dtd [directly] eciover.ent
				           xxxnn.ent
					   ecisubc.ent
				           tei2.dtd [directly] rest of P2
	  [via entities] xxxnn*.eci

The resultant document instance looks schematically like this:

<eci.corpus id=xxxnn>
 corpus header with corpus-dependent material from entities in xxxnn.ent
<eci id=xxxnna>
 sub-corpus header with subcorpus-dependent material from entities in xxxnn.ent
 sub-corpus data from xxxnna*.eci
</eci>
<eci id=xxxnn[b-?]> . . . </eci> repeated for as many sub-corpora as there are
</eci.corpus>

For individual sub-corpora:

xxxnnl.sgm [via doctype] eci.dtd [directly] eciover.ent
				            xxxnn.ent
					    ecisubc.ent
				            tei2.dtd [directly] rest of P2
	  [via entities] xxxnnl*.eci

The resultant document instance looks schematically like this:

<eci id=xxxnnl>
 sub-corpus header with subcorpus-dependent material from entities in xxxnn.ent
 sub-corpus data from xxxnnl*.eci
</eci>

Both corpus and sub-corpus .sgm files set a handful of crucial
parameter entities which guide the process of inclusion and
instantiation outlined above:

  %corp.code	the three letter plus two number identifier of the corpus.
		Given as xxxnn in the examples above.  Used to construct
		the .ent and .eci external references.

  %eci.corpus/
  %eci.single	One of these is INCLUDE in every .sgm file.  Co-varies with
		DOCTYPE eci.corpus versus DOCTYPE eci.

  %comp.code	In the case of DOCTYPE eci/%eci.single=INCLUDE, this
		gives the single letter sub-corpus identifier.  Given as
		l in the examples above.  Used to construct the .eci external
		references, and to select the approriate entities for the
		sub-corpus header.

  %eci.grouped  If INCLUDE, then at least some sub-corpora are composed of
		multiple data files, for which entity declarations can be
		found in the .ent file.  In this case all data files
		will have names of the form xxxnnlnn.eci.  The alternative
		is %eci.simple,	in which case all the data file names
		are of the form xxxnnl.eci and can be constructed directly.
		This case does not actually occur, see section X below.

The .ent file defines one further structural entity, namely
%corp.n.components, to be INCLUDE for that value of n (<= 10) which
accurately reflects the number of sub-corpora of which the corpus is
made up.  It also defines entities from which the corpus header is
constructed, and for each sub-corpus defines entities to provide the
title, copyright and bibliographic information for the sub-corpus
header, each indexed by the sub-corpus component letter.  Many, but
not all, .ent files also define entities which provide tag usage
statistics for the corpus as a whole and its sub-corpora.

Note that each .eci file contains a subset of the subcorpus-dependent
information relevant to itself in SGML comment lines at the beginning
of the file, including corpus and component ids, title and copyright
information.  Since SGML comments begin with "<", this does not
compromise the Text/Markup Invariant.

The decision to identify sub-corpora is necessarily subjective in some
cases.  It has sometimes been made for principled reasons, where there
is a clear need to identify major qualitive differences in source of
material, which in turn demands a header.  Equally often in practice,
it has been done in order to avoid creating files of unwieldy size.
This latter consideration is also the basis for making a particular
sub-corpus of type eci.grouped:  a rough rule of thumb has been to
avoid creating individual files of more than 3 Megabytes in size.  The
entity structure which supports all this is of moderate complexity,
but the line has been drawn at allowing some sub-corpora of a given
corpus to be eci.grouped and others eci.simple:  if any are grouped,
they all are.  Similarly, in the very simplest cases where a corpus
contains only a single data file, this is none-the-less denoted
xxxnna.eci, and the corpus treated as if it has a single sub-corpus.

On balance it is our judgement that the internal complexity of the
mechanisms used is justified by the simplicity of the external view it
presents to the user:  we believe that it is important not to
over-burden those whose interest in SGML and/or the TEI is minimal
with to many angle-brackets in the first instance, but also to provide
all the necessary information at the end of the day for those who DO
care.

X.  Why Not One .Sgm File Per .Eci File?

For ungrouped corpora only one .sgm file is provided, for the corpus
as a whole, i.e. xxxnn.sgm.  For grouped corpora, in addition one .sgm
file is provided for each sub-corpus, i.e. xxxnnl.sgm.  Where SGML
processing of individual data files is desired but no explicit .sgm
file is present, i.e. either for xxxnnl.eci files in the case of
ungrouped corpora, or xxxnnlnn.eci files in the case of grouped
corpora, an implicit .sgm file can be created on the basis of the
filename alone.  For UN*X systems, an implementation of this can be
found in bin/unix/oneeci, q.v.  Non-UN*X systems can take this file as
a specification of how to achieve the same result.  Again, the
decision to provide explicit .sgm files in some cases and not in
others is somewhat arbitrary---implicit nnnxxl.sgm files could be
constructed on the fly for grouped as well as ungrouped corpora, and
we could even manage the corpus-level nnnxx.sgm file as well if
necessary.  We have tried to strike a balance between utility and
superfluity, hopefully the right one.