ECSCW 2003 Workshop
To be held at ECSCW 2003 in Helsinki, Sept. 14, 2003
Gloria Mark and Bonnie A. Nardi
Introduction
We propose a full day workshop to discuss the problems and opportunities of measuring the impact of collaborative technologies. Methodologies such as reference tasks (Sparck-Jones, 1998; Whittaker, Terveen and Nardi, 2000) are well worked out for performance-based measurement of precisely defined tasks performed under experimental conditions. However, measuring impacts in the field, and in particular, over longer time horizons, is problematic. Given Moore’s law, as well as the immense creativity of a large well-educated global workforce, we are designing and deploying new collaborative technologies at ever increasing rates. Many technologies, such as robotics and affective computing, have potentially radical impacts.
Albert Einstein observed, "Not everything that counts can be counted, and not everything that can be counted counts." How do we measure the impacts of collaborative technologies in order to assess them? The objective of this workshop is to examine the practice of measurement in the context of evaluating collaborative systems. How do we know if a system in CSCW is "good"? What should the CSCW community mean by a "good" system?
In Software Engineering, formal metrics are used to determine the efficacy of a system. In CSCW, such metrics do not exist. Yet often there is an expectation, for example, in industry, that we provide metrics. In the field of Information Systems, system effects are measured through variables such as productivity, satisfaction, and so forth. Interpretation of these indirect measures is problematic because intervening variables may affect the group's behavior. For example, if a group feels that they are being pioneers by testing a system they may perform better. Other areas, such as Organization Science, measure system adoption. But the reasons for adoption are varied and may not reflect on the quality of the system. In short, current measures have serious validity problems.
In this workshop we raise the question of measurement and interpretation for CSCW systems. We will focus on issues including what can/should be measured and how, as well as broader issues of context, ethics, and politics of measures for CSCW systems.
Issues for discussion at the workshop include:
The goal of the workshop is to achieve a better understanding of how metrics can be used in the field of CSCW. The objectives that we will target for the end of the day are: 1) to generate a set of metrics that are feasible to use in CSCW, 2) to identify problems and constraints that are associated with using metrics in CSCW, and 3) to generate a set of research topics whereby metrics can be further investigated.
Workshop participants
The workshop will have a maximum of 15 participants. Participants will be selected on the basis of a 3-5 page position paper that addresses one of the workshop themes. Participants who have special expertise in methods of evaluation or assessment for collaborative systems will be sought, as well as those who are interested in taking the practice of measurement itself as a problem of study. The workshop will solicit participation from researchers in areas such as computer science, psychology, anthropology, and social studies of science. Industry and academia will be represented.
Workshop format
The workshop will be structured to emphasize group discussion. From past experiences in attending and organizing workshops, we have found that breaking into smaller groups stimulates very productive discussion. The workshop will consist of a series of small group discussions alternating with whole group discussions throughout the day. Based on the workshop submissions, we will select three different themes to discuss. For each theme, workshop participants will break up into three groups for discussion. Each subgroup will then present the results of their discussion back to the larger group. This will be followed by a discussion with the entire group. We will repeat this process two more times during the day, varying the participants in each subgroup. The last session of the day will consist of an entire group discussion to summarize the important themes that emerged. We will plan for how we can carry forward the results of the workshop through a website, publications or research agendas.
Submission
Please submit a 3-5 page position paper relating to one of the workshop themes. Send the paper to:
and
Deadline for submission: June 20, 2003
Notification of acceptance: by July 7, 2003
Note that early ECSCW registration is July 20, 2003, so this will give you enough time to register early for the conference.
Workshop Organizers
Gloria Mark is a faculty member in the Interactive and Collaborative Technologies group at the University of California, Irvine. Prior to this, she had been a research scientist at the GMD in Bonn, Germany. Her work in CSCW involves studying the effects of technology use on group behaviors. Technologies she has studied include collaborative hypermedia, shared electronic workspaces, desktop conferencing, and HDTV video conferencing. Her main research interest is in studying the effects of virtual collocation.
Bonnie A. Nardi is an anthropologist at Agilent Laboratories in Palo Alto, California. She is the author of A Small Matter of Programming: Perspectives on End User Computing (MIT Press, 1993), the editor of Context and Consciousness: Activity Theory and Human Computer Interaction (MIT Press, 1996) and the co-author of Information Ecologies: Using Technology with Heart (MIT Press, 1999). She has conducted field studies of collaborative technology use in offices, hospitals, schools, libraries, laboratories, and network operations centers.
References
Sparck-Jones, K. (1998). Summary performance comparisons: TREC2, TREC3,
TREC4, TREC5, TREC 6. In Voorhees, E. M., & Harman, D. K. (Eds.),
Proceedings of the Sixth Text Retrieval Conference (TREC-7), 1998.
Whittaker, S., Terveen, L., and Nardi, B. (2000). Let's stop pushing the envelope and start addressing it: A reference task agenda for HCI. Human Computer Interaction, 15, 75-106.