xfig 3.2.1 Structured Demonstration Scoring Sheet
IntroductionThe purpose of this structured demonstration is to evaluate the amount of initial effort that an end user needs to invest to use a reverse engineering or program comprehension tool. This effort is expended in installing the tool and applying it to a subject system. The return on this initial investment is considered through the required and optional tasks in the demonstration. This scoring sheet is a guide to recording observations of this inital investment and return, both quantitatively and qualitatively. This questionnaire can be printed out and filled in, or it can be completed by filling in the blanks electronically.1.0 Tool Description1.1 Name of Tool:1.2 Vendor/Research Group: 1.3 Short Description of Tool:
1.4 Intended Purpose of Tool:
1.5 User Interface (Check Any That Apply)
2.0 Installation2.1 Description of Distribution Package2.1.1 Method of Distribution (Check One) __ Off-the-shelf in store
2.1.2 Materials Included in Distribution (Check All That Apply)
Other:
2.2 Size of Distribution Package (Fill in the Appropriate Blanks)
2.3 Time Required to Install 2.3.1 How much elapsed time did it take to install the program?
You should count the time from when you tried to obtain a copy of the program
until you had an executing program. If you had to wait for an order
to come in, or a repair to the distribution, you should include that time.
2.3.2 How much working time did it take to install the program?
You should count only the time that you spent in front of the computer
installing the software or obtaining more information to install the software.
2.3.4 How many attempts were required to install the program?
3.0 Loading the Subject System3.1 Time Required to LoadIf the tool you are using does not parse the subject system or build a repository, skip ahead to Section 4.0. 3.1.1 How much elapsed time did it take to load the subject system?
You should count the time from when had a working copy of the program installed
to when the subject system was completely parsed and the tool could be
used on it.
3.1.2 How much working time did it take to load the subject system?
You should count only the time that you spent in front of the computer
loading the subject system, writing scripts to assist in the task, or obtaining
more information to do so.
3.1.3 How many attempts were made before successfully loading the subject system? 3.2 Changes Made to Parser
3.2.1 What parts of the tool had to be changed to load the subject system
successfully?
3.2.2 How many lines of code were changed to load the subject system? _____ added
3.2.3 Did you need to modify the subject system to load it into the tool? __ yes
If so, please describe the changes:
3.3 Data Repository After parsing the source code, tools often store the data in a (persistent) internal repository. 3.3.1 Amount of Memory Used By Tool Alone
3.3.2 Amount of Memory Used By Tool With Subject System Loaded
3.3.3 Amount of Disk Space Used By Persistent Repository
3.3.4 Format of Persistent Repository (Check All That Apply)
File Format Used:
4.0 Applicability of Tool to Tasks4.1 Task 1.1 Evaluating Structure4.1.1 How much working time was required to complete this task?
4.1.3 How useful was the tool for this task?
4.1.4 Would you use the tool again to evaluate program structure?
4.1.5 Please list ALL the tools you used to complete this task.
You should include general purpose tools as well, such as editors.
4.2 Task 2.1 Feature Change 4.2.1 How much working time was required to complete this task?
4.2.2 Approximately, what proportion of this time was spent using the
reverse engineering/program comprehension tool?
4.2.3 How useful was the tool for this task?
4.2.4 Would you use the tool again to change an existing feature?
4.2.5 Please list ALL the tools you used to complete this task.
You should include general purpose tools as well, such as editors.
4.3 Task 2.2 Feature Add 4.3.1 How much working time was required to complete this task?
4.3.2 Approximately, what proportion of this time was spent using the
reverse engineering/program comprehension tool?
4.3.3 How useful was the tool for this task?
4.3.4 Would you use the tool again to add a feature?
4.3.5 Please list ALL the tools you used to complete this task.
You should include general purpose tools as well, such as editors.
4.4 Task 2.3 Defect Repair 4.4.1 How much working time was required to complete this task?
4.4.2 Approximately, what proportion of this time was spent using the
reverse engineering/program comprehension tool?
4.4.3 How useful was the tool for this task?
4.4.4 Would you use the tool again to change an existing feature?
4.4.5 Please list ALL the tools you used to complete this task.
You should include general purpose tools as well, such as editors.
5.0 General Comments5.1. Generally, how useful did you find the tool?5.2 Would you use this tool in your daily work as a professional developer?
Please comment.
5.3 Please give any comments you have on the structured demonstration
(materials, design, etc.).
6.0 Team Description 6.1 Complete the following table for each team member.
|