ICS 280: Research Methodology for Software
Winter Quarter 2004

Assignment 2: Categorizing Papers

Due Thursday, February 26, 2004

In this assignment, you will work in groups to categorize papers according to a taxonomy of research strategies and submit a joint report on how the papers were categorized, the process used to arrive at consensus, and the quality of the taxonomy.

Objectives.

Form a Group. This assignment is to be done in groups of two or more. Each group member will need to read every paper to be categorized, and the group must agree on the final categorization. The papers selected should use a range of research strategies.

Group Size   Minimum Number of
Papers Categorized
2   10
3   8
4   5

Select a Taxonomy. Choose one of these taxonomies to use in this assignment:

Alternatively, you may choose a taxonomy published elsewhere or create one of your own based on course materials. If you create one, it should have good coverage of the range of possible research strategies and have good operational definitions.

Categorize the Papers. Each group member should read each of the papers and independently assess them. An assessment consists of

The group should then meet and pool their assessments. The group needs to come to consensus on the assessments for all papers, either by changing the categorization or modifying the taxonomy. In particular, the following procedure is suggested.

  1. Each group member presents her or his assessment of all the papers with a short explanation of the choices made.
  2. If there is consensus, stop.
  3. Examine the problematic papers for sources of disagreement.
  4. Examine the taxonomy for sources of disagreement. Make modifications to the taxonomy, if necessary.
  5. Each group member independently prepares a new categorization.
  6. Return to Step 1, repeat until there is consensus.

Your report should include the initial categorizations, a discussion of how consenus was reached, how many rounds were required, and the final categorization. If consensus is not possible, explain why.

Critique the Taxonomy. Is the taxonomy that you used a good one? If not, why? Here are some criteria for you to consider: coverage, completeness, clarity of definitions for the categories, and possible improvements.