When my colleagues and I sat down to design the conferencing system that would become known as CoSy, we did not immediately start worrying about file layouts, record structures, and the like. We started thinking about how the user would view and use the system, and then worked backward from that. Of course, the design process is never quite that simple -- we had to go through several iterations, because something that we'd thought might be a great idea at the user level turned out to be impossibly complicated to implement, but could be done easily with a small change to the user interface.
Still, this was just a prototype. However, in a system of this sort, the files created with the prototype tend to get inherited by successive generations of the software. Think how annoyed users would get if the old conferences became unreadable whenever a change to the software was made! (Indeed, one of the very first conferences on the prototype, for discussion amongst the developers and test users, was still going strong a year later when the software was well into its second release, and on its third hardware home!) Thus, the organization of the information (the storage architecture) in the system can have profound implications to later development of the software, and it is much more difficult to say "Well, with version 4 we'll go to a totally new file organization" - because then conversion programs would need writing, and some of the imaginable conversions may not even make sense (because of the lack of data never stored in the original files, say).
In this article I'm going to talk about some of the tradeoffs that are involved in deciding how to store conferencing system information. Since I am responsible for most of the detailed design and the coding of the U of Guelph's CoSy system, many examples will come from that. I will make some comparisons with other systems that I know a bit about, COM for example, but bear in mind that I'm not an expert on these and may make some erroneous assumptions. In any case, it is not the specifics of any particular conferencing system that are important, but the implications and limitations of certain designs - these designs may be considered hypothetical.
Now I have to decide whether to approach this subject from the top down - starting at the user interface layer or at least general file layout layer and working down to character coding -- or from the bottom up, starting with deciding how to store characters in messages. Good programming practices is to start at the top down. In any case, one of the first decisions made was to use Unix to develop the system, and Unix doesn't care what a file contains -- it is just a stream of bytes.
The top level organization is obvious: conferences. All conferencing systems have these (surprised?) and they can be viewed as structures in their own right. On all systems that I am familiar with, a conference contains a collection of messages (which may be further grouped or structured, but we'll get to that later), together with information about who is reading or contributing to that collection (or has permission to). There is usually other data associated with a conference too, such as who is allowed to manipulate the conference in some special way (such as add or remove participants, messages, or groups of messages -- this person is usually referred to as the conference moderator), and perhaps descriptive information such as the date the conference was created, a short narrative about the purpose of the conference, and so on. Information about which messages have been read by particular participants (stored to simplify the presentation of new messages to the user) can also be viewed as being conference information, or it can be viewed as being user information and belonging with other data about the user.
We've now come to the first decision point. Does "last message read" information belong in a user-related file or in a conference-related file? Keeping it in the conference file (here the term "file" is used loosely, a conference "object" may be made up of several files. At this point in the discussion these details are unimportant, and we leave them for a lower level in our top-down approach) simplifies such housekeeping operations as adding a user to a conference (or removing one), removing messages, and so on. On the other hand, an element of redundancy is lost by doing this. If such information is stored in user files, then the system becomes easier to reconstruct should something catastrophic happen to a conference file (I am thinking more of software error than hardware problem here, since full backups are done regularly).
In CoSy I chose the latter approach, with some redundancy. Each conference has connected with it a list of the members of that conference, and each user has a list of the conferences to which he or she belongs. Should problems occur, one list can be rebuilt from the others in the system. The "messages read" information is kept with the users' lists - since this information changes often when a user is signed on and reading or writing messages, this list can be read into memory and continually updated without the disk I/O that would be needed if this information were kept with conference files. COM and Telecenter also keep this information in the user files, although the details are significantly different.
Strictly speaking, conferences aren't really the top level. In any type of business environment (and the administration of a university is a business, if a strange one) there have to be classes of conference. For example, there should be public or open conferences (or forums, if you prefer) which anyone can browse through or contribute to. Indeed, these are usually the most popular and active on the system. However, there are some discussions that should have restricted access, (closed), although you may want to post their existence, so that someone who is interested enough can ask to be invited in. Finally, there are those that are closed and which the participants have no intention of letting anyone else into - these might be private conversations between two or three people, for example. These unlisted conferences are called "confidential" in CoSy parlance.
We knew at the outset that we would want these three classes of conference, but they can all be treated as the same sort of thing (indeed, a CoSy conference can be switched from one class to another). Thus, there is no real "class layer" on top of the conference layer - rather, the information logically belongs with each conference as to which type it is.
Another topmost layer, however, is that of groups of related conferences. This is an especially useful concept in an application where there are many open conferences, electronic publishing for example. The new user is likely to be a bit bewildered at the choice, so some method of grouping these is desirable. Thus there might be a "languages" group, and "operating systems" group, and so on, each of these comprising a number of separate conferences. Had we thought of this idea at the beginning, we would have been tempted to incorporate it into the file structure, using UNIX's directory hierarchies. However, this concept did not arise until later, as the number of active conferences grew, and the group structure is more virtual than real. A group is really just a named list of conferences, nothing more. In fact I think this is preferable than the group/conference hierarchy that we might have implemented, since it permits the users to transfer easily from one conference to another without worrying about whether they are crossing a group boundary or not.
On the other hand, there are some advantages to having some sort of group boundary in terms of giving the users easier control over how they read their messages, although the differences are better appreciated if you've actually used CoSy. However, I'll try to explain. In CoSy, each user has his or her own list of conferences to which they belong. CoSy also provides a simple mechanism for reading messages - just hitting the RETURN key will cause an automatic search for the next unread message. The search starts within the topic you are already connected to, if any, and continues on to the other topics in the same conference if the user has read all the messages in the topic. If all the messages in the conference have been read, the search starts at the top of the user's list of conferences and progresses down through it, stopping at the first unread message.
With few conferences, or a relatively quiet system, this is perfectly acceptable. An "order" command is provided so that users can rearrange conferences on their lists, to place higher priority ones at the top. However, in a situation where you are a member of many conferences, and there are many other people on-line at the same time, who are adding messages, you can sometimes find yourself "ping-ponging" from one conference to another as messages get added elsewhere while you are reading the messages in one conference. It would be nice to be able to group conferences in such a way that the ones you use for work, say, are distinct from those you participate in out of interest, so that you are not distracted by messages in lower priority conferences. This is not really as significant a problem as I have perhaps made out, as the "order" command can deal with this, although it does illustrate that any feature can have both positive and negative effects.
Telecenter has no concept of groups of conferences in that sense, but does use the UNIX file permissions to define open and closed classes of conference. COM (and most other conferencing systems) also allows open, closed and unlisted conferences. Neither of these systems support the concept of named groups of conferences, however.
Not all conferencing systems have a named level below conferences - they just start right in with messages, although there is usually some structure to the way in which messages are presented. It seemed logical to us, though, that within a given conference there should be the option of dividing discussion into several areas. One analogy is of agenda items at a face-to-face meeting, although with a computer conferencing system it is possible to carry out discussion on all the agenda items simultaneously! (At least, with the appearance of simultaneity -- such concepts get a bit muddled in the asynchronous world of computer conferencing).
This is easy to do in UNIX, we just implemented each conference as a directory, and the topics as files within that directory. (I am simplifying slightly). Going this route had the bonus of making it easy to add attributes to a conference - another file could be added within the conference directory. The "UNIX-ness" of all this is transparent to the users, who are never exposed to the raw system.
COM does not provide for this hierarchical conference/topic structure. However, the method COM uses to store messages and other information does provide a number of other features. COM does most of its own storage management, relying little on the underlying operating system. This is discussed more fully below.
Although Telecenter uses the UNIX file system -- indeed, relies on it heavily, conferences are not subdivided into topics. Possibly the implementors decided that the extra layer of complexity that would be added to the user commands was not worth the slight advantage. Indeed, on the Guelph CoSy system, many conferences have only one topic, but some have as many as seven, eight or even more (there is no inherent limit).
PARTICIPATE (PARTI on THE SOURCE) uses a different structuring concept altogether, and allows each note in a conference potentially to become the root of another conference, building into a many-branched tree structure. The comment structure of messages within a CoSy topic is similar to this, although the branches remain within the topic and a member of the conference has access to all messages in all branches. (A new conference can be formed from them by copying, if desired).
Now we get down to the real meat of a conferencing system -- the messages. It is also here that the differences in implementation really start to show up. In Telecenter, for example, each message is a separate UNIX file. These are grouped into conferences following the normal UNIX directory system, and a message can be shared by two or more conferences by using UNIX filesystem "links", although there is no user friendly way to do this. Given this implementation, several things become easy and tempting. For example, a user's new messages are links stored in the user's directory, connecting to the conference messages, and they are unlinked as read. This makes for a system that is easy to construct, using existing UNIX commands and writing the Telecenter commands as UNIX command files or execs (called "shell scripts" in UNIX parlance). It reportedly took Mike Pearson about one man-week's programming effort to get the first version of Telecenter running. (The CoSy prototype took about six or seven man-weeks of programming, plus of course design, documentation and testing time). The disadvantages to this approach lie in the storage overhead of all those separate files, as well as the program overhead from using command scripts. While an excellent prototyping method, the system is slow (and non portable except to other Unix systems).
As mentioned briefly above, COM does all its own file management, and indeed is almost an operating system unto itself. COM reserves several large chunks of file space, and then manages its own "internal" file system using this space. Several of these files are used to store such information as names of all objects and their addresses (in effect, a directory); information about the numbers of objects and details about users, and so on. Of most interest in this discussion though is the TEXT file and its associated OBJ and DIR files. Actual message text and some information about author and creation date is stored in the TXT file. More detailed information about the text (rather than the text itself) is stored in the OBJ and DIR files; this includes pointers linking comments together with the texts that they comment on. (CoSy handles such comment pointers differently, as described below). All the relevant information about record lengths, position in file, and so in is maintained by COM. Additionally, COM buffers file blocks in and out of core. If a user accesses a particular message, COM first checks to see if that block is already in memory. The use of UNIX provides this buffering automatically for CoSy and other UNIX-based systems.
Thus, COM handles files and messages almost as an operating system unto itself. This allows a very flexible file structure -- as in Telecenter, all COM messages are separate entities, linked by pointers to wherever they belong. Thus, a COM message can be in several conferences and several people's mail baskets, and need not be replicated. This saves storage space, and obviates the complications of "copy" and "file" commands to manipulate messages. A message can be copied to another conference, or as mail to another user, just by manipulating links. On the other hand, the development effort was large, since in effect a major part of an operating system (storage allocation, file buffering) was written. (An early version of COM was about 30,000 lines of DEC-10 assembler, and the design stage estimate for PortaCOM was 90,000 lines of Pascal. The CoSy prototype was about 3000 lines of C, excluding the editor; it is now about 12000 lines, including the editor but excluding it is now quite a bit larger). COM also imposes some structure on the content of messages that is not present in CoSy or Telecenter, but may be in other systems. For example, most non-printable ASCII characters are stripped off when text is input, and lines are restricted in length. (Lines are formatted to the users desired line length on output, up to a maximum of 76 characters). This is perfectly acceptable, even desirable, in an conferencing system consisting solely of text messages. However it precludes the later incorporation of graphic messages (messages containing graphic codes or information) without some serious reprogramming.
CoSy compromises by doing its own storage management within a single topic, but falling back on the operating system's facilities for other functions. Thus, a topic is a single file containing the text of messages as well as associated information, such as the author of the message, the date and time written, and pointer information linking comments to the original message on which they comment. This pointer information is stored in a separate file in COM. In CoSy, a topic file can be viewed as made up of variable length records, consisting of a fixed "header" portion containing data about the message (including comment pointers) and a variable length "text" portion containing the actual message.
There are two ways of storing information of variable length. One is to include a "length" field as part of the record, the other is to have a unique terminator sequence signifying the end of the record. Since we needed a header field anyway, and to speed up scans through the topic file (by doing a direct disk seek to the next header, rather than reading sequentially through the intervening text), I chose the former method, encoding the text length as part of the header. This storage method turned out to have significant advantages for later development.
Since there is no special "magic" character indicating the end of a message, then there is no limitation on what data is stored in a text field of a message. Since UNIX imposes no structure on user files either, it can be an arbitrary stream of bytes. This potentially useful feature was preserved by not doing any filtering of what the user entered as a message. Non printable ASCII characters were left alone on input, and instead converted to displayable form on output. By providing an option to turn off this output filtering, (and carriage return insertion for wrapping long text lines), we automatically have the capability of storing and sending graphic messages (or any other byte stream, for that matter). The first tests of this were done early on, using Tektronix codes, and were quite successful. The currently supported code is NAPLPS (North American Presentation Level Protocol Syntax) which is an ANSI and international standard of representing graphs and images as special text and picture drawing instructions. (See the four-part article by Fleming and Frezza, "NAPLPS: A New Standard for Text and Graphics" in the February through May 1983 issues of BYTE for more about NAPLPS). This would have been impossible, or at least very difficult, had we limited ourselves by imposing any sort of structure as messages were input, a point in favour of not imposing limits on yourself until you have to.
Assuming that the basic requirements of a conferencing system are present, it is the user interface that can make or break the success and usability of a system. This has more to do with careful choice of command vocabulary, the wording of prompts and help files, design of the command parser, and intelligent choice of defaults than it does with the underlying storage architecture. Of course, I like to think we made mostly correct decisions in this are with CoSy. But implementation decisions can have ramifications on how user friendly a system is perceived to be. For example, the implementation of Telecenter largely as a collection of UNIX C-shell command files places limits on the user interface. For example, the syntax of commands is quite rigid, and unlike CoSy, abbreviations cannot be used. (This was somewhat alleviated in Telecenter by rewriting parts of it in C).
The particular file system design can have more subtle effects than that. As we've already seen with the example of exchanging graphics via CoSy, the underlying architecture (in this case, the storage of message records in the topic file) can have a significant influence on how easy or difficult it is to implement some feature useful to the user. There can be other ramifications, too, in terms of how users cope with the volume of information available on a busy conferencing system.
This has been mentioned already in parallel with the different levels of organization in a system, but let's look at this a little closer here.
For example, suppose you are a participant in two similar but distinct conferences on a given system, and you have something to say that really fits both conferences. You may also want to explicitly mail the same message to someone who is not participating in either conference, but is a user of the system. In COM, this can be done at a single stroke, because each message or "text" is only stored once, and all references to it (whether from a conference or an individual's mailbox) are via special pointers. Telecenter also stores each message as a single physical entity, although using the UNIX filesystem to maintain links or pointers to the appropriate places. However, in the early version of Telecenter with which I am familiar, there is no single command to direct a message to several different conferences at once - the appropriate links would need to be added later. CoSy does not provide such a linking mechanism at the message level. To place a message in more than one conference, it has to be physically copied from one to the other, increasing slightly the disk space used. There are of course commands to facilitate this copying. However, it is possible for a whole topic to be shared between two or more conferences, with the messages entered by members of either conference available to all.
In the systems described (CoSy, COM, and Telecenter) the messages entered into a conference (or conference topic) may be either new thoughts or specific references or answers to other messages. In all of these also there is some indication when a given message is indeed a reference to or comment on another message. This contextual information is very useful, indeed necessary if an answering comment consists solely of "I agree" or "no". Telecenter doesn't go much beyond providing the number of the original message, but both CoSy and COM let you know, when you are reading a message, if there are any comments or responses to it. This is useful when reading an item that asks a question - it is helpful to know if anyone else has responded to it yet, in which case you may not need to bother answering. COM and CoSy also provide ways for reading all the responses to a message before reading the next chronologically entered message.
Other systems have similar mechanisms for handling messages and responses. In Participate, for example, not only may any given message have comments to it, forming a tree structure of comments, comments to comments, and so on, but any such message may become the root of a whole new named conference. CoSy also supports trees of comments to comments, but to start a new conference (or topic) at an arbitrary point, the relevant branches must be explicitly copied. In my opinion this keeps the system as a whole easier to manage, and is perhaps less bewildering for users, but dedicated Parti users might disagree. PicoSpan, developed by Marcus Watts, is another UNIX-based conferencing system. However, the structure of its messages and comments is perhaps more like a comb than a tree. In a PicoSpan conference, there can be sequential messages, each of which may have a number of responses to it. The user has the option of reading the responses before going on to the next message, as in COM or CoSy. However, in PicoSpan the responses are handled differently than the base level sequential messages, and there is no additional level of branching. Hence a comb-structure, with the base level messages forming the spine of the comb, and the responses forming the teeth. In some respects this can be viewed as a hypothetical CoSy conference in which any participant could add a new topic, and members were only allowed to comment to the first message in a topic. [[**NOTE to Donna or tech editor: I may have blown the above about Picospan. This is how I remember it working, having used it a few times many months ago. I can't put my hands on any current documentation, and maybe you can in fact comment to responses. I don't think so, but you should check it out - you are likely to have faster access to this info than me - Picospans line is always busy, and my account has probably long since been deleted...A.M.**]]
This top-down comparison of some conferencing systems has illustrated some of the different tradeoffs and compromises involved in designing real systems that are intended for a broad audience. I've also shown how different decisions about how the data is stored - the storage architecture - can have later implications on what can or cannot be easily done in terms of adding new features or subsystems to existing programs. To really do the latter point justice would probably require a book length article, but I'm sure many of you are familiar with the experience of coming back to make a change in a program and thinking "darn, if I'd done it this way instead, it would have been easy". (Or, if you're lucky or exceptionally farsighted "gee I'm glad I did it that way, this new feature just slides right in").
Any system design involves compromises. The decisions made generally depend more on the intended users and usage than any absolute scale of "good" or "efficient" design. But it is always a good idea to keep the maximum number of options open for as long as possible.
Telecenter was developed by Mike Pearson while at the International Institute for Applied Systems Analysis in Laxenburg, Austria. This effort is described in "Creating An Adaptive Computerized Conferencing System on UNIX", Michael M.L. Pearson and James E. Kulp, IIASA, Laxenburg, Austria 1981.
COM was developed by Jacob Palme, et al., at the Swedish National Defence Research Institute, Stockholm. Info here is extracted from "The COM Teleconferencing System Functional Specification", FOA Report C 10164-M6(H9), File COMSPE.VIP revised 22 Oct, 1980. (By Jacob Palme, Stefan Arnborg, Lars Enderin, Carl Meyer and Torgny Tholerus) as well as from other documents on COM from the Stockholm University Computing Centre - QZ, and from private communications.