Would you like to make this site your homepage? It's fast and easy...
Yes, Please make this my home page!
Pedagogical Framework Documentation: A Research Proposal
Ian Chai
chai@uiuc.edu
Department of Computer Science
University of Illinois at Urbana-Champaign
1304 West Springfield Avenue, Urbana, IL 61801
Presented at:
Empirical Studies of
Programmers '97
Student
Workshop
and OOPSLA '97 poster session
|
Abstract
Although object-oriented frameworks are a powerful
technique for reuse, they are hard to learn because they
tend to have a lot of internal structure. The classic
tutorial-based documentation does not work very well
as people are bad at following step-by-step
instructions. This is a project to figure out principles
for writing pedagogical framework documentation by
using human experimental trials of different
documentation styles.
|
|
Introduction
In recent years, object-oriented frameworks have
emerged as an important way to achieve better code
reuse. However, a major obstacle stands in the way of
their wide-spread use: mature frameworks tend to be
large and contain complicated interrelations among
the objects therein. Programmers new to a framework
are often overwhelmed by the steep learning curve, and
require much hand-holding by an expert.
This is exacerbated by the fact that most framework
documentation is written in-house by programmers
who would rather be programming than writing
documentation. They also usually have no theory of
what works well in documentation.
Various people have proposed formal descriptions of
frameworks, for example [Helm 90]. However, formal
descriptions are not a good way for newcomers to learn
a system instead, they mainly help people who are
already familiar with a system to figure out various
details. This is comparable with the difference between
a How to program in Pascal textbook and the
denotational semantics for Pascal.
Johannes Samtinger proposed an object-oriented
approach to documenting object-oriented systems
[Samtinger 94]. He proposed that the documentation
should be also object-oriented, and support
inheritance. However, this is mainly useful for
systems documentation and not for application
programmer documentation. Mary Elizabeth Raven
and Richard Thomson took a look at how his system
might be extended to user documentation, which is
organized around how to rather than what is [Raven
94]. They note certain drawbacks to the system
Samtinger proposed, and proposed their own broader
definition, emphasizing modularity and reusability.
Traditionally, computer documentation comes in the
form of step-by-step instructions. The research of
John M. Carroll shows that this does not work well in
general because people are bad at following
instructions. What people are good at is
self-motivated instruction, where the interests of the
learner can direct the learning. He found that providing
minimal instructions that promote exploration and
understanding is much more effective than
step-by-step instructions [Carroll 90].
Patterns have emerged as a good way to document and
pass on technical knowledge. Patterns and pattern
languages are ways to describe best practices and good
designs, and to capture experience in such a way that it
is possible for others to reuse this experience. C.
Alexander first used patterns for architecture
[Alexander 77, Alexander 79]. They were adapted to
computer science, resulting in books like Design
Patterns: Elements of Reusable Object Oriented
Software [Gamma 95].
Ralph Johnson has demonstrated the feasibility of
documenting the HotDraw framework using patterns
[Johnson 92]. He and Kent Beck argued that one could
use design patterns to derive architectures, once again
using the HotDraw framework as an example [Beck
94].
Both patterns and minimalist documentation have a
modular nature, while traditional step-by-step
tutorial-style documentation does not.
I am comparing the combination of patterns and
minimalist documentation with traditional
step-by-step tutorial-style documentation. Which
style allows individuals to learn how to use
object-oriented reusable frameworks faster? I am
searching for guidelines for writing framework
documentation. Note that what I am aiming for is a
tutorial for newcomers and not a reference for experts.
Patterns, Minimalist Documentation, and the Web
Patterns
Patterns1
are a documentation technique have been
used to document good object-oriented design
practices (eg. [Gamma 95]), software packages and
frameworks (eg. [Johnson 92]) as well as how to do
many other things. Some of the diverse things people
have documented using patterns include designing web
pages [Orenstein 95], how to teach in a classroom
[Anthony 95], how to run an organization [Coplien
97], and how to write patterns themselves[Meszaros
96].
They are a solution to a problem in a context.
Although they may come in different formats or forms,
patterns almost always have a statement of the
problem, the context in which to solve the problem,
and a solution given that context.
One of the objectives of a pattern is to get readers to
understand some of the rationale for the solution, so
that they can decide when to apply the pattern. In
contrast, a traditional, step-by-step tutorial usually
gives readers the steps to take without explaining why
they should take them.
Patterns often refer to other patterns when a problem
or its solution is too big to discuss in one sitting.
Hence, they lend themselves well to hypertext
presentations such as those found on the Web.
Gerard Meszaros and Jim Doble [Meszaros 96] say that
patterns should have these elements:
- Pattern Name: so that people can refer to
the pattern
- Problem: the description of the problem it
solves.
- Solution: there may be different solutions to
the same problem depending on the context.
- Context: the circumstances of the problem
imposes constraints on the solution.
- Forces: often contradictory considerations
that must be taken into account when choosing a
solution to a problem.
Minimalist Documentation
Minimalist documentation2
is based on the idea that
people do not want irrelevant information when trying
to learn a particular task hence the term minimalist.
Also based on the idea that people are not good at
following step-by-step instructions, they attempt to
give the reader short pages or index cards of
information that they can read in whatever order suits
them. Carroll's experiments [Carroll 90] indicate that
people learn better when they decide what they want to
know next, instead of the book they happen to be
reading.
[Rosson 90] gives an example of a case study of
applying minimalist instruction to teaching
Smalltalk. [Gong 90] gives an example of applying
minimalist instruction to writing manuals
incrementally.
Because each minimalist page or card contains little
information, like patterns, they, too, often refer to
other pages or cards. Hence, like patterns, they lend
themselves well to hypertext presentations.
Carroll [Carroll 90] gives these guidelines for
minimalist documentation:
- Training on Real Tasks: people are more
motivated to do an exercise when it relates directly to
something useful they want to do.
- Getting Started Fast: if there is too much to
read before readers get to typing something on the
computer, they will lose interest and miss things.
- Reasoning and Improvising: instead of rote
exercises, people learn better with more personally
meaningful challenges.
- Reading in Any Order: topics are brief and
allow readers to chose whatever order seems best to
them.
- Coordinating System and Training: instead of
giving all the detailed steps, let the learner interact
with the system.
- Supporting Error Recognition and Recovery:
instead of giving step-by-step instructions that
assume readers will repeat them flawlessly, expect
them to fail and give them resources to understand how
to recover.
- Exploiting Prior Knowledge: instead of using
insider jargon , use analogies to readers' prior
experience to help them understand.
- Using the Situation: take advantage of the
expectations learners bring to the situation.
The Web
The idea or presenting books or book-like material
online is not a new one. Back in 1979 Nicholas
Negroponte discussed the concept in his article Books
without Pages. [Negroponte 79]. With the World-Wide
Web, this idea has exploded [Dicks 96][Kalmbach
96][Mehlenbacher 96]. However, the explosion of the
web did not necessarily mean that we got better online
writing. I think this comment [Sheppard 97] by Dan
Sheppard illustrates one of the problems:
Paper is calm.
It looked for a while that paper could be augmented,
calmly, with hypertext, which allowed
cross-referencing, something paper wasn't very good
at. But look at a typical corporate web-page now, it
appears to be in a state of constant alarm, like a
Vietnam Veteran running knife in hand, screaming,
through the University Library.
Calm is good.
As we design web pages for documentation, we should
keep this in mind. Java animations, bells and
whistles, etc. can quickly become excessive.
Documentation should be calm.
The Web provides a convenient and widely available
hyperlinked environment to present documents. The
nature of both minimalist documentation and patterns
lend themselves to a hyperlinked environment: since
each pattern or minimal document has to be small,
they often refer to related documents that elaborate on
other aspects of the problem.
In fact, when reading a pattern language on paper,
many are struck by how natural it would be if they
could click on the reference to another pattern and go
directly there. This is why many pattern languages
will include page numbers as a poor man's hyperlink to
references.
Patterns, Minimalist Documentation, and the Web
What does it look like when you combine patterns and
minimalist documentation? To understand this, let me
first discuss how they are similar and different.
Both patterns and minimalist documentation
The main tension between patterns and minimalist
documentation is the question of giving context and
forces to a problem. With the other differences, one or
the other style is indifferent to the issue, and thus we
can accommodate the insistent one.
As I noted earlier, patterns have a hyperlinked feel,
and thus the web is a natural environment for them.
Preliminary Research
Because in general computer scientists do not perform
human trials, I took a class from Educational
Psychology on evaluation. As the class project, I did a
comparison of two different kinds of documentation
for VisualWorks Smalltalk's graphical user interface
framework. I ran these trials on students from Ralph
Johnson's Smalltalk class, after they had learned
non-graphical smalltalk programming, but before
they had learned the graphical user interface
framework.
The first set of documentation, the representative of
the traditional printed manual, was two chapters from
The VisualWorks Tutorial from ParcPlace Digitalk
[ParcPlace 95]. They are Chapter 4, Creating a
Graphical User Interface and Chapter 5, Developing
the Domain Models.
For the second set of documentation, the
representative of minimalist patterns, I wrote some
web pages called Visualessons, [Chai 96] using the
guidelines from [Meszaros 96] and [Carroll 90] stated
above.
For a full report on this experiment, see [Chai 97].
Results of the Preliminary Research: from the
VisualWorks Tutorial
- The biggest problem with the traditional
tutorial way of teaching was that people did not like to
read a lot of material. People are notoriously bad at
following step-by-step instructions, although
Computer Scientists are better than most at this, due to
their training. This is consistent with the findings of
Carroll [Carroll 90]
- Another frequent problem was that the
subjects constantly struggled with the tension of
trying to learn enough to do the task at hand while
trying to follow the tutorial. Thus subjects are often
tempted to skip material as they try to discover just
the information they need.
- The subjects also tried to skip sections that
describe things they thought they already understood.
Due to the dependency of the tutorial upon past steps,
some subjects ended up skipping steps they needed to
have performed to follow the tutorial.
- The tutorial enforced a certain order on the
tasks, even when no particular order was necessary.
This sometimes causes confusion and
misunderstanding.
- Subjects often felt overwhelmed with the
amount they needed to read to get to the nuggets of
information they wanted.
- The book sometimes told the subject to do
something which did not have a visible effect, leaving
the subject confused as to whether or not something
happened. Thus, when giving such instructions, we
must warn the reader that there will be no visible
effect, even though the step is done correctly.
Results of the Preliminary Research: from
Visualessons
- The hyperlinked pages with the index of topics
allowed the subjects to proceed in a mostly non-linear
way, picking only the pages which seem relevant to
the task at hand. However, there were still certain
dependencies which sometimes confused the subjects.
One such dependency was the page that oriented the
subjects as to how to use the web pages. At first, this
page was called the Introduction. I discovered that
nobody would read it with that name. After I renamed it
Read Me First the change was remarkable. Computer
Scientists are used to skipping introductions, while
they are used to reading Read Me First documents.
- Most of the subjects liked using examples,
perhaps because it gives a concrete feel to the
material. But some of them tried to learn solely from
the examples and failed to read the patterns, which hurt
their performance. Patterns are necessary to learn why
things are done.
Perhaps a good solution would be to have a
running example section at the bottom of each
pattern, with buttons to take the reader to the back and
forth across the example.
There was one subject, however, who
downloaded the example code, filed it into their
VisualWorks image (ie. loaded it into the interpreter environment), and never looked at the web pages
again! He tried to figure it all out from the example
code. As a result, he never did discover the GUI-builder
at all. It is unclear how one would deal with such a
person a person who will not read your
documentation, no matter how well-written.
One other subject did a somewhat less extreme
version of this: she kept the web pages going, but
seldom referred to them. Instead, she would try to
puzzle out things herself in VisualWorks. This
subject, however, liked the few VisualWorks windows
with a small context-sensitive help section at the
bottom. Perhaps this kind of person would read the
patterns if they were integrated with VisualWorks in a
context-sensitive way, as opposed to being separate
web pages.
- Many subjects complained that it took a long
time before they could see a tangible running program.
Perhaps Visualessons should start with a simple hello,
world example. We need, however, to balance this
against the drawbacks mentioned earlier against
step-by-step tutorials by keeping the example short
and quick.
- Subjects sometimes get confused when the
patterns tell them to press the x button or select the y
menu because they have difficulty locating the window
with the referenced button or menu. A [picture] button
next to such commands which pops up a screen shot of
the window may help subjects orient themselves
better.
- Those subjects who noticed the index found
it helpful for deciding what patterns to look at.
However, many subjects failed to remember the index
even though they saw it on the first page. Some way of
making the subjects aware of the index perhaps a
bigger, more noticeable, button may help mitigate
this.
- A few subjects complained that they needed
to flip to the relevant pages before being able to figure
out if the widget was what they needed or not, instead
of being able to make that determination at a glance.
When there is a list of patterns which are just nouns or
short noun-phrases (as in the list of widgets), we need
to provide a short description. This is especially
important when the noun phrase is misleading, as in
input field, which may also be used for output.
- It is impossible to write a perfect set of
documentation in one attempt. The principle of
iterative development which has been well-proven in
software engineering is applicable here as well. To
have good documentation, try it out on users, and then
go back and fix the rough spots they discover.
Flaws in the Preliminary Research
Because this was my first time doing human
experimental trials, I made a number of mistakes.
- While iterative refinement of documentation
seems to be a good way of writing framework
documentation, by iteratively improving
Visualessons during the trials, I compromised the
experimental data. While it is possible to consider
this a valid design experiment [Brown 92], I think that
in future research, I should do the iterative refinement
before I run the experimental trials.
- The sample size was too small to draw
conclusive qualitative results.
- I did a different number of trials for The
VisualWorks Tutorial and Visualessons.
- The examples The VisualWorks Tutorial and
Visualessons presented were very different. This may
have affected the subjects' ability to do the sample
program independently of the merits of the writing
styles.
- In some cases, the subjects themselves picked
which form of documentation to use, instead of being
randomly picked.
Future Research
The future research will be along similar lines with the
preliminary research, but attempt to avoid its flaws. I
am using Habanero as the base framework for these
trials.
The Habanero Project
Habanero is a project of the National Center for
Supercomputing Applications (NCSA). It is a
framework for collaborative computing written in
Java.
Collaborative computing is an attempt to reproduce
the benefits of multiple computer users working
together in one site when they are actually not
physically present. A collaborative computing
environment allows users at different sites to log in to
the same session, and then see what others are doing.
Then instead of having to explain to the other person
what you are doing, you can show them directly on
their own computer monitor.
The Habanero environment allows users to run hablets
collaborative programs. All users logged in to the
same session can see what the other users are doing.
The Habanero framework allows programmers to turn
Java applets into hablets.
In order to convert applets into hablets, you need to
include the Habanero code and change your code to call
it. You would already have written event-handling code
in your applet. Since in Habanero things that happen
locally need to be reflected to other users, you need to
capture the events and pass them along. You will also
need to migrate some objects to other locations.
I am writing documentation to teach Java programmers
how to convert their applets into hablets.
Research Questions
There are many interesting research questions to test,
and each introduces another parameter dimension to
the experiment. In order to properly test all these
questions, I would need more trials than I am likely to
be able to get, and have to write too many sample
documents. Thus, I need to select a limited number of
these parameters to test.
Here are the possible questions. Most of them were
highlighted by the preliminary research.
- Which is better: traditional step-by-step
tutorial instructions or smaller pages of information,
with random-access hyperlinks?
- The tension between Patterns insisting on
giving background materials and reasoning context
and forces, vs. Minimalist ideas about assuming a
context that the reader is already familiar with,
teaching them just what they need to get going, and let
them explore.
- Should examples be interspersed between the
documentation, or set apart and just referred to by the
documentation?
- Should there be one connected example, or
many small examples?
- Does linking into design patterns help, or
just confuse the reader with extraneous material?
- Is the comfort of paper better, or the
convenience of hyperlinked web pages?
- Is iterative development of documents
better?
Question 7 is obviously true, so requires no testing.
Mark Guzdial's work with STABLE [Guzdial 97]
indicate that the answer to question 6 may be that
hyperlinked web pages are better. His work is not
proof that it is better, but simply point in that
direction.
Questions 1 through 5 are relevant questions, but it is
not feasible in a study of this size to test every single
one of them, because it would require too many trials
for every question we test, we could double the number
of trials we have to do.
Planned Experimental Procedure
The experimental subjects should be reasonably
experienced programmers, and have at least a working
knowledge of Java.
I intend to show the subject how to use the Habanero
user interface, and let them try out some of the sample
hablets. Then I will give them the code to a small
applet, and one of the various sets of documentation,
selected at random. I will tell them that their task is to
turn that applet into a hablet, and the documentation
will show them how.
I will then observe them as they try to solve the
problem, prompting them to voice their thoughts so
that I could transcribe them, along with my
observations of what they are doing.
When they have completed the exercise, or if they are
hopelessly stuck and unable to complete the exercise,
I will stop the trial. Then, I will be free to answer any
questions they may have about things they did not
understand, since that will be outside the trial and will
not affect it.
After the trials, I will collate the comments into
representative samples that illustrate what worked well
and what did not work well. I intend to also compare
the time it took for them to complete the exercise, to
obtain some qualitative results.
The sample size will be around 75. I plan to run trials
in 2 stages.
The First Stage
I will test questions 1 and 2 by running trials with
three sets of documentation:
- Traditional step-by-step tutorial instructions
- Patterns-style documentation with context and
forces
- Minimalist-style documentation without
context and forces
My hypothesis is that either Patterns-style
documentation with context and forces, or
Minimalist-style documentation without context and
forces, will be the most effective.
The Second Stage
The first stage's results will tell us which one of the
three methods works best. I plan then to combine that
method with one of these variations to test question 3:
- Examples interspersed between the
documentation
- Examples set apart and just referred to by the
documentation
My hypothesis is that examples interspersed between
the documentation will be the most effective.
Questions 4, 5 and 6 will have to wait for a future
study. Each of the above variations will have
approximately 15 subjects.
Example documents
I have already written the patterns-style
documentation with context and forces and made them
available at
http://st-www.cs.uiuc.edu/~chai/writing/pepperypatterns/.
Appendix 1 shows an example
of such a document. This also illustrates examples
interspersed between the documentation.
Appendix 2 shows an example of a
minimalist-style documentation without the context
and forces. This also illustrates examples interspersed
between the documentation.
Appendix 3 shows an example of patterns-style
documentation with context and forces with examples
set apart and just referred to by the documentation.
-
[Alexander 77]
-
C. Alexander, S. Ishikawa, and M. Silverstein, A
Pattern Language, Oxford University Press, 1977
- [Alexander 79]
-
C. Alexander, A Timeless Way of Building, Oxford
University Press, 1979
-
[Anthony 95]
-
Dana Anthony, Patterns for Classroom Education,
[PloP 95]. Also available on the web at
http://st-www.cs.uiuc.edu/~chai/writing/classroom-ed.html
-
[Beck 94]
-
Kent Beck and Ralph Johnson, Patterns Generate
Architectures, Proceedings of the 8th European
Conference, ECOOP '94, Lecture Notes in Computer
Science, Volume 821. Postscript version available at
ftp://st.cs.uiuc.edu/pub/papers/patterns/patterns-generate-archs.ps
-
[Brown 92]
-
Ann L. Brown, Design Experiments: Theoretical and
Methodological Challenges in Creating Complex
Interventions in Classroom Settings, Journal of the
Learning Sciences, 2(2) 141-178, Lawrence Erlbaum
Associates, 1992.
-
[Carroll 90]
-
John M. Carroll, The Nurnberg Funnel: Designing
Minimalist Instruction for Practical Computer Skill,
MIT Press, 1990
-
[Chai 96]
-
Ian Chai, Visualessons, URL http://st-www.cs.uiuc.edu/~chai/writing/visualessons
-
[Chai 97]
-
Ian Chai, Visualessons: A comparison between
traditional paper manual tutorials and web-based
minimalist documentation written in pattern style
URL http://st-www.cs.uiuc.edu/~chai/research/esp/
-
[Coplien 97]
-
Jim Coplien, Organizational Patterns
URL http://www.bell-labs.com/cgi-user/OrgPatterns/OrgPatterns?OrganizationalPatterns
-
[Dicks 96]
-
R. Stanley Dicks, Pages, Books, the Web, and Virtual
Reality: A Response to Negroponte's Books Without
Pages, * (Asterisk) The Journal of Computer
Documentation, Volume 20, Number 3, August 1996,
ACM SIGDOC.
-
[Gamma 95]
-
Erich Gamma, Richard Helm, Ralph Johnson, and
John Vlissides, Design Patterns: Elements of Reusable
Object-Oriented Software, Addison Wesley, 1994
-
[Gong 90]
-
Richard Gong and Jay Elkerton, Designing Minimal
Documentation Using a GOMS Model: A Usability
Evaluation of an Engineering Approach, Conference
on Human Factors in Computing Systems (CHI) 90
Proceedings, Addison-Wesley Press, 1990.
-
[Guzdial 97]
-
Mark Guzdial, Tools to Support Project-Based
Learning, 5 May 1997, Beckman Institute Complex
Systems Colloquium, and personal communication
regarding the STABLE project. Dr. Guzdial's URL is
http://www.cc.gatech.edu/gvu/people/Faculty/Mark.Guzdial.html
-
[Helm 90]
-
Richard Helm, Ian Holland, Dipayan Gangopadhyay,
Contracts: Specifying Behavioral Compositions in
Object-Oriented Systems, OOPSLA '90 Proceedings,
ACM Press, 1990.
-
[Johnson 92]
-
Ralph Johnson, Documenting Frameworks Using
Patterns, OOPSLA '92 Proceedings, ACM Press, 1992.
Postscript version available at
ftp://st.cs.uiuc.edu/pub/papers/HotDraw/documenting-frameworks.ps
-
[Kalmbach 96]
-
James Kalmbach, Books without Pages/Pages without
Books: Revisiting the Geography of Interface, *
(Asterisk) The Journal of Computer Documentation,
Volume 20, Number 3, August 1996, ACM SIGDOC.
-
[Mehlenbacher 96]
-
Brad Mehlenbacher, Spaces Without Places:
Politicizing Nicholas Nigroponte's Technologizing of
Books Without Pages, * (Asterisk) The Journal of
Computer Documentation, Volume 20, Number 3,
August 1996, ACM SIGDOC.
-
[Meszaros 96]
-
Gerard Meszaros and Jim Doble, MetaPatterns: A
Pattern Language for Pattern Writing, PLoP '96,
Washington University Technical Report
#wucs-97-07 URL http://www.cs.wustl.edu/~schmidt/PLoP-96/meszaros.ps
-
[Negroponte 79]
-
Nicholas Negroponte, Books without pages, ICC '79
Conference Record, IEEE.
-
[Orenstein 95]
-
Robert Orenstein, A HTML 2.0 Pattern Language,
[PloP 95]. A web-based version is available at
http://www.anamorph.com/docs/patterns/default.html
-
[ParcPlace 95]
-
ParcPlace-Digitalk, VisualWorks Tutorial Rev 2.1,
ParcPlace Digitalk, 1995
-
[PLoP 94]
-
Jim Coplien & Doug Schmidt, editors, Pattern
Languages of Program Design, (Patterns from Pattern
Languages of Programming '94) Addison-Wesley
Press, 1995.
-
[PLoP 95]
-
John Vlissides, Jim Coplien, Norm Kerth, editors,
Pattern Languages of Program Design 2, (Patterns
from Pattern Languages of Programming '95)
Addison-Wesley Press, 1995.
-
[Raven 94]
-
Mary Elizabeth Raven and Richard Thomson, Can
Principles of Object-Oriented System Documentation
Be Applied to User Documentation?, * (Asterisk, ACM
SIGDOC Journal of Computer Documentation) Volume
18, ACM Press, 1994.
-
[Rosson 90]
-
Mary Beth Rosson, John M. Carroll, and Rachel K.E.
Bellamy, Smalltalk Scaffolding: A Case Study of
Minimalist Instruction, Conference on Human Factors
in Computing Systems (CHI) '90 Proceedings,
Addison-Wesley Press, 1990.
-
[Samtinger 94]
-
Johannes Sametinger, Object-Oriented Documentation
* (Asterisk, ACM SIGDOC Journal of Computer
Documentation) Volume 18, ACM Press, 1994.
-
[Sheppard 97]
-
Dan Sheppard, Paper is Calm, original USENET
posting to oxbridge.tat, copied to the Patterns
Discussion Group, patterns@cs.uiuc.edu.
Also available at URL http://st-www.cs.uiuc.edu/~chai/writing/paper-calm.html.
Footnotes
- See [Chai 96, Gamma 94, Alexander 77,
Alexander 79] for examples of patterns.
- See [Carroll 90] for examples of minimalist
documentation.
Back to Ian's website.
If you spot any mistakes, please let me know.