Observer Handbook

A Collective Tool Demonstration

Observer Handbook

A CASCON 1999 Workshop

Dr. Margaret-Anne Storey
Assistant Professor
Department of Computer Science
University of Victoria
PO Box 3055 STN CSC
Victoria, BC Canada V8W 3P6
Tel: +1 (250) 721 8796
Fax: +1 (250) 721 7292
mstorey@csr.uvic.ca

Susan Elliott Sim
Department of Computer Science
University of Toronto
10 Kings College Rd.
Toronto, Ontario Canada M5S 3G4
Tel. +1 (416) 978-4158
Fax. +1 (416) 978-4765
simsuz@cs.utoronto.ca

Thank you for agreeing to be an observer for the Collective Tool Demonstration. We have invited developers of program comprehension tools from IBM and university research to participate in a formal demonstration by applying their tools to a common subject system. You will be observing the XXX team as they use their tool to analyse the subject system. You should consider yourself as an "apprentice" and that during the day your goal is to develop a mastery of XXX. In particular, you should consider how the tool could be used in your work as a software developer. You will report on your experiences on Day 2 of the workshop.

We will be presenting the tool teams a scenario in which they play developers of the subject system, xfig. The subject system is written in C and consists of approx. 30,000 LOC. Your team will be using XXX to analyze xfig and complete two sets of tasks. The first set of tasks is required, and consist of tasks such as creating documentation to provide an overview of a software system. The second set is optional (although we would like them to do at least one of these tasks) and consists of maintenance tasks such as fixing a bug or adding a new feature.

We have attached a copy of the handbook which we have given to the tool developers, we encourage you to read the tasks at the beginning of the experiment so that you are familiar with them.

The remainder of this document is organized as follows:

General Instructions: these outline the basic procedures you should follow throughout the day.
Preparing your presentation: reviews the main points you should include in your 10 minute presentation for Wednesday. Read this now as it will help you focus your observations.
Questionnaire: this section describes a questionnaire which you need to complete on behalf of the tool team.

General Instructions

You should consider yourself as an "apprentice" and that your goal during the day is to develop a mastery of the XXX. You will need to pay attention to both the nuts-and-bolts aspects of running the tools and the concepts behind the analysis that they team performs. In other words, how they use XXX and why they use XXX the way they do. You will therefore need to ask the team members what they are doing and how they are using the tool. The team members may work silently, do not be afraid to ask questions, for example:

What are you doing now?
Why did you do that?
Could you elaborate on that?
Are things not working as expected?

Your interruptions may break their concentration, so you need to walk a fine line between finding out what they are doing and not interrupting them unduly.

Please record important observations as you go along, as it is hard to remember important details later on.

We will all pause for lunch 12-1PM.

Read the points to be considered for preparing your presentation on Wednesday (see below). These points will help you focus your observations.

We would like you to fill out a short questionnaire on behalf of your team. The questionnaire can be found on the last page of this handbook. When you have completed the questionnaire, please detach the page and give it to Peggy or Susan.

Preparing your presentation

We would like you to prepare a short 10-minute presentation for Wednesday. Consider some of the following questions as you prepare your presentation:

Would you use XXX as a developer?
Do you feel that XXX has a place in your organization? For which tasks?
Do you consider XXX difficult to use?
After observing for the day, do feel you are capable of using the most of XXX's functionality?
If you had not observed XXX being used, how long do you think it would have taken you to become an expert in the tool?
To what extent did the developers use the XXX to perform the maintenance tasks?
Your team may have used other tools to perform the assigned tasks, would you find any of these tools useful in your work context?

We will have a laptop available with Powerpoint if you wish to use this to prepare your talk. We also have some transparencies and pens if you would prefer to use those.

Questionnaire

Whenever it is convenient, please complete the following questionnaire for the team participants (you may summarize their responses on this page).
Note: Ask for more than yes or no answers, dig a little bit to find out how much they have worked on or browsed such programs before.

Tool Team Name:
Observer Name:

Have you ever used (ran) the xfig drawing package before? To what extent did you use it?
Have you used (run) other drawing packages? Which ones, and to what extent?
Have you written, modified or browsed the source code for xfig before today?
Have you written, modified or browsed the source code for another drawing package before?