Pedagogical Framework Documentation: A Research Proposal

Ian Chai
chai@uiuc.edu
Department of Computer Science
University of Illinois at Urbana-Champaign
1304 West Springfield Avenue, Urbana, IL 61801

Presented at:
Empirical Studies of Programmers '97 Student Workshop
and OOPSLA '97 poster session

Abstract
Although object-oriented frameworks are a powerful technique for reuse, they are hard to learn because they tend to have a lot of internal structure. The classic tutorial-based documentation does not work very well as people are bad at following step-by-step instructions. This is a project to figure out principles for writing pedagogical framework documentation by using human experimental trials of different documentation styles.

Introduction

In recent years, object-oriented frameworks have emerged as an important way to achieve better code reuse. However, a major obstacle stands in the way of their wide-spread use: mature frameworks tend to be large and contain complicated interrelations among the objects therein. Programmers new to a framework are often overwhelmed by the steep learning curve, and require much hand-holding by an expert.

This is exacerbated by the fact that most framework documentation is written in-house by programmers who would rather be programming than writing documentation. They also usually have no theory of what works well in documentation.

Various people have proposed formal descriptions of frameworks, for example [Helm 90]. However, formal descriptions are not a good way for newcomers to learn a system instead, they mainly help people who are already familiar with a system to figure out various details. This is comparable with the difference between a How to program in Pascal textbook and the denotational semantics for Pascal.

Johannes Samtinger proposed an object-oriented approach to documenting object-oriented systems [Samtinger 94]. He proposed that the documentation should be also object-oriented, and support inheritance. However, this is mainly useful for systems documentation and not for application programmer documentation. Mary Elizabeth Raven and Richard Thomson took a look at how his system might be extended to user documentation, which is organized around how to rather than what is [Raven 94]. They note certain drawbacks to the system Samtinger proposed, and proposed their own broader definition, emphasizing modularity and reusability.

Traditionally, computer documentation comes in the form of step-by-step instructions. The research of John M. Carroll shows that this does not work well in general because people are bad at following instructions. What people are good at is self-motivated instruction, where the interests of the learner can direct the learning. He found that providing minimal instructions that promote exploration and understanding is much more effective than step-by-step instructions [Carroll 90].

Patterns have emerged as a good way to document and pass on technical knowledge. Patterns and pattern languages are ways to describe best practices and good designs, and to capture experience in such a way that it is possible for others to reuse this experience. C. Alexander first used patterns for architecture [Alexander 77, Alexander 79]. They were adapted to computer science, resulting in books like Design Patterns: Elements of Reusable Object Oriented Software [Gamma 95].

Ralph Johnson has demonstrated the feasibility of documenting the HotDraw framework using patterns [Johnson 92]. He and Kent Beck argued that one could use design patterns to derive architectures, once again using the HotDraw framework as an example [Beck 94].

Both patterns and minimalist documentation have a modular nature, while traditional step-by-step tutorial-style documentation does not.

I am comparing the combination of patterns and minimalist documentation with traditional step-by-step tutorial-style documentation. Which style allows individuals to learn how to use object-oriented reusable frameworks faster? I am searching for guidelines for writing framework documentation. Note that what I am aiming for is a tutorial for newcomers and not a reference for experts.

Patterns, Minimalist Documentation, and the Web

Patterns

Patterns1 are a documentation technique have been used to document good object-oriented design practices (eg. [Gamma 95]), software packages and frameworks (eg. [Johnson 92]) as well as how to do many other things. Some of the diverse things people have documented using patterns include designing web pages [Orenstein 95], how to teach in a classroom [Anthony 95], how to run an organization [Coplien 97], and how to write patterns themselves[Meszaros 96].

They are a solution to a problem in a context. Although they may come in different formats or forms, patterns almost always have a statement of the problem, the context in which to solve the problem, and a solution given that context.

One of the objectives of a pattern is to get readers to understand some of the rationale for the solution, so that they can decide when to apply the pattern. In contrast, a traditional, step-by-step tutorial usually gives readers the steps to take without explaining why they should take them.

Patterns often refer to other patterns when a problem or its solution is too big to discuss in one sitting. Hence, they lend themselves well to hypertext presentations such as those found on the Web.

Gerard Meszaros and Jim Doble [Meszaros 96] say that patterns should have these elements:

Minimalist Documentation

Minimalist documentation2 is based on the idea that people do not want irrelevant information when trying to learn a particular task hence the term minimalist. Also based on the idea that people are not good at following step-by-step instructions, they attempt to give the reader short pages or index cards of information that they can read in whatever order suits them. Carroll's experiments [Carroll 90] indicate that people learn better when they decide what they want to know next, instead of the book they happen to be reading.

[Rosson 90] gives an example of a case study of applying minimalist instruction to teaching Smalltalk. [Gong 90] gives an example of applying minimalist instruction to writing manuals incrementally.

Because each minimalist page or card contains little information, like patterns, they, too, often refer to other pages or cards. Hence, like patterns, they lend themselves well to hypertext presentations.

Carroll [Carroll 90] gives these guidelines for minimalist documentation:

The Web

The idea or presenting books or book-like material online is not a new one. Back in 1979 Nicholas Negroponte discussed the concept in his article Books without Pages. [Negroponte 79]. With the World-Wide Web, this idea has exploded [Dicks 96][Kalmbach 96][Mehlenbacher 96]. However, the explosion of the web did not necessarily mean that we got better online writing. I think this comment [Sheppard 97] by Dan Sheppard illustrates one of the problems:
Paper is calm.

It looked for a while that paper could be augmented, calmly, with hypertext, which allowed cross-referencing, something paper wasn't very good at. But look at a typical corporate web-page now, it appears to be in a state of constant alarm, like a Vietnam Veteran running knife in hand, screaming, through the University Library.

Calm is good.

As we design web pages for documentation, we should keep this in mind. Java animations, bells and whistles, etc. can quickly become excessive. Documentation should be calm.

The Web provides a convenient and widely available hyperlinked environment to present documents. The nature of both minimalist documentation and patterns lend themselves to a hyperlinked environment: since each pattern or minimal document has to be small, they often refer to related documents that elaborate on other aspects of the problem.

In fact, when reading a pattern language on paper, many are struck by how natural it would be if they could click on the reference to another pattern and go directly there. This is why many pattern languages will include page numbers as a poor man's hyperlink to references.

Patterns, Minimalist Documentation, and the Web

What does it look like when you combine patterns and minimalist documentation? To understand this, let me first discuss how they are similar and different.

Both patterns and minimalist documentation

The main tension between patterns and minimalist documentation is the question of giving context and forces to a problem. With the other differences, one or the other style is indifferent to the issue, and thus we can accommodate the insistent one.

As I noted earlier, patterns have a hyperlinked feel, and thus the web is a natural environment for them.

Preliminary Research

Because in general computer scientists do not perform human trials, I took a class from Educational Psychology on evaluation. As the class project, I did a comparison of two different kinds of documentation for VisualWorks Smalltalk's graphical user interface framework. I ran these trials on students from Ralph Johnson's Smalltalk class, after they had learned non-graphical smalltalk programming, but before they had learned the graphical user interface framework.

The first set of documentation, the representative of the traditional printed manual, was two chapters from The VisualWorks Tutorial from ParcPlace Digitalk [ParcPlace 95]. They are Chapter 4, Creating a Graphical User Interface and Chapter 5, Developing the Domain Models.

For the second set of documentation, the representative of minimalist patterns, I wrote some web pages called Visualessons, [Chai 96] using the guidelines from [Meszaros 96] and [Carroll 90] stated above.

For a full report on this experiment, see [Chai 97].

Results of the Preliminary Research: from the VisualWorks Tutorial

Results of the Preliminary Research: from Visualessons

Flaws in the Preliminary Research

Because this was my first time doing human experimental trials, I made a number of mistakes.

Future Research

The future research will be along similar lines with the preliminary research, but attempt to avoid its flaws. I am using Habanero as the base framework for these trials.

The Habanero Project

Habanero is a project of the National Center for Supercomputing Applications (NCSA). It is a framework for collaborative computing written in Java.

Collaborative computing is an attempt to reproduce the benefits of multiple computer users working together in one site when they are actually not physically present. A collaborative computing environment allows users at different sites to log in to the same session, and then see what others are doing. Then instead of having to explain to the other person what you are doing, you can show them directly on their own computer monitor.

The Habanero environment allows users to run hablets collaborative programs. All users logged in to the same session can see what the other users are doing. The Habanero framework allows programmers to turn Java applets into hablets.

In order to convert applets into hablets, you need to include the Habanero code and change your code to call it. You would already have written event-handling code in your applet. Since in Habanero things that happen locally need to be reflected to other users, you need to capture the events and pass them along. You will also need to migrate some objects to other locations.

I am writing documentation to teach Java programmers how to convert their applets into hablets.

Research Questions

There are many interesting research questions to test, and each introduces another parameter dimension to the experiment. In order to properly test all these questions, I would need more trials than I am likely to be able to get, and have to write too many sample documents. Thus, I need to select a limited number of these parameters to test.

Here are the possible questions. Most of them were highlighted by the preliminary research.

  1. Which is better: traditional step-by-step tutorial instructions or smaller pages of information, with random-access hyperlinks?

  2. The tension between Patterns insisting on giving background materials and reasoning context and forces, vs. Minimalist ideas about assuming a context that the reader is already familiar with, teaching them just what they need to get going, and let them explore.

  3. Should examples be interspersed between the documentation, or set apart and just referred to by the documentation?

  4. Should there be one connected example, or many small examples?

  5. Does linking into design patterns help, or just confuse the reader with extraneous material?

  6. Is the comfort of paper better, or the convenience of hyperlinked web pages?

  7. Is iterative development of documents better?
Question 7 is obviously true, so requires no testing.

Mark Guzdial's work with STABLE [Guzdial 97] indicate that the answer to question 6 may be that hyperlinked web pages are better. His work is not proof that it is better, but simply point in that direction.

Questions 1 through 5 are relevant questions, but it is not feasible in a study of this size to test every single one of them, because it would require too many trials for every question we test, we could double the number of trials we have to do.

Planned Experimental Procedure

The experimental subjects should be reasonably experienced programmers, and have at least a working knowledge of Java.

I intend to show the subject how to use the Habanero user interface, and let them try out some of the sample hablets. Then I will give them the code to a small applet, and one of the various sets of documentation, selected at random. I will tell them that their task is to turn that applet into a hablet, and the documentation will show them how.

I will then observe them as they try to solve the problem, prompting them to voice their thoughts so that I could transcribe them, along with my observations of what they are doing.

When they have completed the exercise, or if they are hopelessly stuck and unable to complete the exercise, I will stop the trial. Then, I will be free to answer any questions they may have about things they did not understand, since that will be outside the trial and will not affect it.

After the trials, I will collate the comments into representative samples that illustrate what worked well and what did not work well. I intend to also compare the time it took for them to complete the exercise, to obtain some qualitative results.

The sample size will be around 75. I plan to run trials in 2 stages.

The First Stage

I will test questions 1 and 2 by running trials with three sets of documentation: My hypothesis is that either Patterns-style documentation with context and forces, or Minimalist-style documentation without context and forces, will be the most effective.

The Second Stage

The first stage's results will tell us which one of the three methods works best. I plan then to combine that method with one of these variations to test question 3: My hypothesis is that examples interspersed between the documentation will be the most effective.

Questions 4, 5 and 6 will have to wait for a future study. Each of the above variations will have approximately 15 subjects.

Example documents

I have already written the patterns-style documentation with context and forces and made them available at http://st-www.cs.uiuc.edu/~chai/writing/pepperypatterns/. Appendix 1 shows an example of such a document. This also illustrates examples interspersed between the documentation.

Appendix 2 shows an example of a minimalist-style documentation without the context and forces. This also illustrates examples interspersed between the documentation.

Appendix 3 shows an example of patterns-style documentation with context and forces with examples set apart and just referred to by the documentation.

References

[Alexander 77]
C. Alexander, S. Ishikawa, and M. Silverstein, A Pattern Language, Oxford University Press, 1977
[Alexander 79]
C. Alexander, A Timeless Way of Building, Oxford University Press, 1979
[Anthony 95]
Dana Anthony, Patterns for Classroom Education, [PloP 95]. Also available on the web at http://st-www.cs.uiuc.edu/~chai/writing/classroom-ed.html
[Beck 94]
Kent Beck and Ralph Johnson, Patterns Generate Architectures, Proceedings of the 8th European Conference, ECOOP '94, Lecture Notes in Computer Science, Volume 821. Postscript version available at ftp://st.cs.uiuc.edu/pub/papers/patterns/patterns-generate-archs.ps
[Brown 92]
Ann L. Brown, Design Experiments: Theoretical and Methodological Challenges in Creating Complex Interventions in Classroom Settings, Journal of the Learning Sciences, 2(2) 141-178, Lawrence Erlbaum Associates, 1992.
[Carroll 90]
John M. Carroll, The Nurnberg Funnel: Designing Minimalist Instruction for Practical Computer Skill, MIT Press, 1990
[Chai 96]
Ian Chai, Visualessons, URL http://st-www.cs.uiuc.edu/~chai/writing/visualessons
[Chai 97]
Ian Chai, Visualessons: A comparison between traditional paper manual tutorials and web-based minimalist documentation written in pattern style URL http://st-www.cs.uiuc.edu/~chai/research/esp/
[Coplien 97]
Jim Coplien, Organizational Patterns URL http://www.bell-labs.com/cgi-user/OrgPatterns/OrgPatterns?OrganizationalPatterns
[Dicks 96]
R. Stanley Dicks, Pages, Books, the Web, and Virtual Reality: A Response to Negroponte's Books Without Pages, * (Asterisk) The Journal of Computer Documentation, Volume 20, Number 3, August 1996, ACM SIGDOC.
[Gamma 95]
Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides, Design Patterns: Elements of Reusable Object-Oriented Software, Addison Wesley, 1994
[Gong 90]
Richard Gong and Jay Elkerton, Designing Minimal Documentation Using a GOMS Model: A Usability Evaluation of an Engineering Approach, Conference on Human Factors in Computing Systems (CHI) 90 Proceedings, Addison-Wesley Press, 1990.
[Guzdial 97]
Mark Guzdial, Tools to Support Project-Based Learning, 5 May 1997, Beckman Institute Complex Systems Colloquium, and personal communication regarding the STABLE project. Dr. Guzdial's URL is http://www.cc.gatech.edu/gvu/people/Faculty/Mark.Guzdial.html
[Helm 90]
Richard Helm, Ian Holland, Dipayan Gangopadhyay, Contracts: Specifying Behavioral Compositions in Object-Oriented Systems, OOPSLA '90 Proceedings, ACM Press, 1990.
[Johnson 92]
Ralph Johnson, Documenting Frameworks Using Patterns, OOPSLA '92 Proceedings, ACM Press, 1992. Postscript version available at ftp://st.cs.uiuc.edu/pub/papers/HotDraw/documenting-frameworks.ps
[Kalmbach 96]
James Kalmbach, Books without Pages/Pages without Books: Revisiting the Geography of Interface, * (Asterisk) The Journal of Computer Documentation, Volume 20, Number 3, August 1996, ACM SIGDOC.
[Mehlenbacher 96]
Brad Mehlenbacher, Spaces Without Places: Politicizing Nicholas Nigroponte's Technologizing of Books Without Pages, * (Asterisk) The Journal of Computer Documentation, Volume 20, Number 3, August 1996, ACM SIGDOC.
[Meszaros 96]
Gerard Meszaros and Jim Doble, MetaPatterns: A Pattern Language for Pattern Writing, PLoP '96, Washington University Technical Report #wucs-97-07 URL http://www.cs.wustl.edu/~schmidt/PLoP-96/meszaros.ps
[Negroponte 79]
Nicholas Negroponte, Books without pages, ICC '79 Conference Record, IEEE.
[Orenstein 95]
Robert Orenstein, A HTML 2.0 Pattern Language, [PloP 95]. A web-based version is available at http://www.anamorph.com/docs/patterns/default.html
[ParcPlace 95]
ParcPlace-Digitalk, VisualWorks Tutorial Rev 2.1, ParcPlace Digitalk, 1995
[PLoP 94]
Jim Coplien & Doug Schmidt, editors, Pattern Languages of Program Design, (Patterns from Pattern Languages of Programming '94) Addison-Wesley Press, 1995.
[PLoP 95]
John Vlissides, Jim Coplien, Norm Kerth, editors, Pattern Languages of Program Design 2, (Patterns from Pattern Languages of Programming '95) Addison-Wesley Press, 1995.
[Raven 94]
Mary Elizabeth Raven and Richard Thomson, Can Principles of Object-Oriented System Documentation Be Applied to User Documentation?, * (Asterisk, ACM SIGDOC Journal of Computer Documentation) Volume 18, ACM Press, 1994.
[Rosson 90]
Mary Beth Rosson, John M. Carroll, and Rachel K.E. Bellamy, Smalltalk Scaffolding: A Case Study of Minimalist Instruction, Conference on Human Factors in Computing Systems (CHI) '90 Proceedings, Addison-Wesley Press, 1990.
[Samtinger 94]
Johannes Sametinger, Object-Oriented Documentation * (Asterisk, ACM SIGDOC Journal of Computer Documentation) Volume 18, ACM Press, 1994.
[Sheppard 97]
Dan Sheppard, Paper is Calm, original USENET posting to oxbridge.tat, copied to the Patterns Discussion Group, patterns@cs.uiuc.edu. Also available at URL http://st-www.cs.uiuc.edu/~chai/writing/paper-calm.html.

Footnotes

  1. See [Chai 96, Gamma 94, Alexander 77, Alexander 79] for examples of patterns.
  2. See [Carroll 90] for examples of minimalist documentation.

Back to Ian's website.
If you spot any mistakes, please let me know.