Reviews for the SWEETPEA Paper

[The paper on our work with Barney (here) was accepted for the CHI'99 conference, although it pretty much split the reviewers. Here is what they had to say.]

Dear Colleague,

Thank you very much for submitting the following paper to CHI 99: SWEETPEA: Software Tools for Programmable Embodied Agents

We are happy to inform you that this paper has been accepted for inclusion in the CHI Technical Program. As an accepted paper, your work will be published in the CHI Conference Proceedings. You will shortly be receiving instructions for preparing and formating your paper for publication.

Attached to this email you will find a set of comments from the paper's reviewers, together with the meta-review prepared by a member of the Program Committee. These reviews contain detailed comments on your paper that you may find useful when revising it for inclusion in the conference proceedings. Note that the meta-review is primarily a summary of the other reviews, sometimes including a recommendation to the Program Committee on how to interpret the reviewers' scores. We hope that you will respond to reviewers' suggestions for improvement. Feel free to contact us if you have any questions about preparing your final contribution to the proceedings, or about your presentation at the conference.

It is expected that authors with accepted papers will attend the CHI Conference and present their papers. Advance Programs will be mailed in the beginning of January, informing you when your paper is scheduled for presentation. We urge you to register for the conference early to secure the pre-registration discount.

Thank you for your interest in CHI 99. We are looking forward to meeting you in Pittsburgh.

Kate Ehrlich and William Newman
CHI 99 Papers Co-Chairs
chi99-papers@acm.org


Reviews for Paper #:99092
Paper Title: SWEETPEA: Software Tools for Programmable Embodied Agents
NOTE: Some punctuation characters (i.e., ' represented as ? ) may have reproduced incorrectly in the reviews. We apologize for any confusion this causes.


Document ID: 99092-0 META-REVIEW

1. Briefly summarize the paper in your own words, making sure to identify the paper's contribution to the field of HCI:

The authors discuss how devices like the Actimates "Barney" can be adapted as control and information-providing devices. This required reverse-engineering the operation of the device, learning how to control it for new purposes, and -- most importantly -- thinking carefully about what it means for such a device to play an interactive role with people, and how this affects how we think about computing.

2. Select the phrase that best describes the paper's contribution:

Design for interactive system

3. Review of the paper:

This is and will be a very controversial paper. Reviewers split very cleanly on this paper: Half saw it as a very significant investigation into how external, portable, character-like devices can play a role (literally!) in our interactive/computational life; the other half saw the paper as an almost-trivial writeup of some people fiddling around with a toy.

I feel very strongly that the paper should be accepted. It is certainly a non-traditional paper that addresses a non-traditional topic with non-traditional technology. But I believe the authors have done it carefully and well. They have not simply fiddled around with a toy, but have thought carefully about notions of role and character, and how even a simple device like that studied here can offer insights into computing. This is not to say that I expect we will soon all have Barneys sitting next to our machines. But I do expect that future systems will be able to learn a great deal from the work here, which is where the direct benefit will come from. And it certainly won't be a boring presentation....

4. Comment on the paper's written presentation:

The paper is very well written; the occasional bits of humor don't detract from its content (or, at least, it didn't for me). I do agree with the points of some reviewers that too much space is devoted to the reverse-engineering aspects of the project, and that the "Barbie" discussion could be safely omitted. Figures are fine; international readers should have no problems.

5. Provide any other comments that may be useful to the authors.


Document ID: 99092-1

1. Briefly summarize the paper in your own words, making sure to identify the paper's contribution to the field of HCI:

The paper describes an experiment in using a commercial toy (a Barney doll) as an interaction device.

2. Select the phrase that best describes the paper's contribution:

Experience: including design briefing / case history

3. Review of the paper:

This has to be one of the funniest papers I have ever read and I certainly do not believe CHI should reject a paper merely because it treated a subject humorously. However, I felt there were some more serious problems with the paper.

In particular, I found the work not particularly significant in that it reports on one summer's worth of experimentation with one toy. There is a laudable attempt to produce general principles and guidelines from this experience but I would rather see a broader base of experience before reaching for such guidelines.

Finally, I find the author(s) use of the toy in a work/office context to be inappropriate. Their idea of using "PEAs" (Programmable Embodied Agents) follows in a tradition of using plush toys that goes back at least to Druin, but these uses have been educational and entertainment-oriented. One of the important principles of graspable user interfaces is that the form of the interface tool accords with the task for which it is to be used. Mapping the waving of Barney's arms to the status of a copier is arbitrary at best.

4. Comment on the paper's written presentation:

The presentation is quite good and the humor helps a good deal.

5. Provide any other comments that may be useful to the authors.

6. Rating of Paper's acceptability (1 is low, 5 is high):

2

7. Rate your expertise in the topic area of this paper (1 is low, 5 is high):

4


Document ID: 99092-2

1. Briefly summarize the paper in your own words, making sure to identify the paper's contribution to the field of HCI:

This paper is part reverse engineering of the ActiMates Barney (a big hit at CHI'98 and promising to have a continued presence at CHI'99), part exploration of bleeding edge HCI issues like character-based interfaces and embodied interaction, and part sheer fun. The authors have deconstructed Barney's API in order to use its capabilities to express other applications, particularly as proxies for or awareness of system activity or other users. The contribution of the paper is the exploration of the use of embodied interaction through a character, and (at least for the authors) the demonstration that "off the shelf" technology can be adapted to support new applications.

2. Select the phrase that best describes the paper's contribution:

Tool for interactive system design/development

3. Review of the paper:

This work addresses one of the most interesting new areas of HCI, namely character-based interfaces and embodied interaction. The authors provide an excellent, if short, review of related work, including ubiquitous computing and Ishii's tangible bits work at MIT. The "lessons learned" from the literature review (last paragraph in the Related Work section) is the most valuable conceptual contribution of the paper, and it would be great to have a fuller discussion of it, particularly in the context of the framework presented later in the paper. To that end, the discussion of Barbie might be dropped, since it is not needed to understand the rest of the paper (or at least condensed into a paragraph, since Barbie is not studied or discussed further). The discussion of how embodied interactions and character-based interfaces are different from "computer" interfaces, and what kinds of messages they are particularly well-suited for is one of the main benefits others can gain from this work.

The authors stress (over-stress, I believe) the value of the demonstration that these kinds of applications can be built from "off the shelf" components (i.e., by hacking, or is it back-hacking? Barney), but it is not clear how satisfactory their Barney behaviors are, particularly without the luxury of arbitrary speech. Without a fairly complete ability to program Barney (wrt the behaviors he has), it seems difficult to argue that it is possible to consider it a development platform, and it tends to make the applications the authors create seem a bit like "stupid Barney tricks." But they are not just that -- at least, to the extent that they constitute a meaningful testbed for exploring ideas about embodied interfaces & characters, and the authors discuss their applications in these terms. So although I think the paper is acceptable the way it currently stands, I think it would be much stronger and make a more significant contribution to the CHI community if these aspects were better highlighted in the discussion, and better i ntegrated with the literature mentioned in the introduction as related work. For example, all of the sections at the end elaborating the framework should be discussed together and summarized at the end -- I read through them all, but did not come away with any unified idea about what had been discovered about character-based interfaces and/or embodied interaction.

The work is original and valid.

4. Comment on the paper's written presentation:

The writing is clear and appropriate for an international audience. The figures are gorgeous; too bad they can't be reproduced in color (it would not be worth taking them out of context to get them in color). I don't get the "SWEETPEA" in the title, though -- ok, she is a character in Popeye, but that won't be understood by many non-Americans. PEA is programmable embodied agents -- is SWEET supposed to be "software tools"?? The acronym/term is never used in the paper. I would suggest that the title just be "software tools for programmable embodied agents" (or feel free to preface it by "Tangible Stuffed Bits" -- see below ;-)

5. Provide any other comments that may be useful to the authors.

I thoroughly enjoyed reading this paper, and am sure it will be well-received by the CHI'99 audience. By the end, I was envisioning all the beanie babies, teletubbies, and other stuffed characters my colleagues and I keep atop our monitors and on our desks coming to life in a veritable "tangible stuffed bits" universe. But rather than recoil in horror, it reveals something of what the future of billions of (physically) detached, networked objects might hold. I think this is the kind of paper that will entertain as it broadens thought; perhaps not the most erudite of HCI work, but definitely making a contribution.

6. Rating of Paper's acceptability (1 is low, 5 is high):

4

7. Rate your expertise in the topic area of this paper (1 is low, 5 is high):

3


Document ID: 99092-3

1. Briefly summarize the paper in your own words, making sure to identify the paper's contribution to the field of HCI:

This paper describes the use of various "multimedia" children's toys as user interface components.

2. Select the phrase that best describes the paper's contribution:

Experience: including design briefing / case history

3. Review of the paper:

This is a sort of interesting paper but it is really in the form of a product announcement. There appears to be no science here and very little engineering. I got the feeling that this was the description of some out-of-hours activity stimulated by the desire to make a toy work in a way not originally intended by the manufacturers.

A real problem I have with the work is that most of it appears to be reverse engineering how to make the product behave as desired instead of as intended.

At the end of the day the phrase is very much "So what?"

4. Comment on the paper's written presentation:

Essentially OK except that the author sometimes uses first person singular and sometime first person plural. This really irritates since the implication is that for part of the work there was a team but I do not think this is the case.

5. Provide any other comments that may be useful to the authors.

6. Rating of Paper's acceptability (1 is low, 5 is high):

2

7. Rate your expertise in the topic area of this paper (1 is low, 5 is high):

3


Document ID: 99092-4

1. Briefly summarize the paper in your own words, making sure to identify the paper's contribution to the field of HCI:

The paper explores the use of new, networked toys (e.g., Talk-With-Me Barbie, Actimates Barney) as forms of computational interaction.

2. Select the phrase that best describes the paper's contribution:

Interaction technique

3. Review of the paper:

I have to say I couldn't tell if this was serious or not. The idea of using that annoying little dinosaur in office contexts had me questioning the author's sanity for a bit. But I decided that it probably was for real.

Contribution. Pretty clear to me that there's something here. Using existing toy devices to provide additional i/o channels is an interesting tweak on the some of the tangible/ambient media work pioneered by Hiroshi Ishii and others. Unlike Ishii, the authors propose exploiting features of well-known characters using off-the-shelf toys. Although we are seeing more techniques that exploit physical objects as i/o devices, i think there's something important about the use of well known characters. Plus the use of these technologies in office settings vs. playrooms.

Having said that, I'll try to provide some suggestions to the authors. My main complaint is the overemphasis on reverse engineering Barney. A huge chunk of the paper talks about how the designers had to interpret Barney's MIDI stream. For one, I suspect they could have asked Erik Strommen at Microsoft for the protocol. Two, I could care less about its protocol. Every toy will have its own protocol until some set of "toy networking" standards are developed, so it just doesn't matter. What *does* matter are the types of interaction that you can squeeze out of the purple one. Other reviewers may disagree with me on this one. But I feel that whole "Exploring Barney's Technology" section could be eliminated. Or replaced by the sentence, "We had to spend time understanding the communications protocol before we could develop an API."

Same goes for the talk about LPC and Barney-speech. I don't care about these issues, I only care about what you can do with the current implementation.

The other complaint is the (in my opinion) huge assumption that people like to communicate with autonomous characters in computing environments. The authors give the example of the Microsoft Paper Clip agent as an interactive proxy.

While they don't say whether the paper clip man is a successful example or not, I know I've never met *anyone* who actually likes that feature. I mean *nobody*.

Ben Schneiderman has posed many arguments against the use of anthropomorphic characters, yet his pleas are almost always ignored by those working on character-based agents. Whether or not people really want talking dinosaurs, it is at least useful to acknowledge that there is some dispute over this. Instead of boldly stating, "these characters embody a stronger sense of identity and agency than can be conveyed by a typical graphical interface. As a result, then, they can serve better as channels for carrying particular sorts of information ... (p. 2, related work section)" It just seems like a jump to imagine that a winking Barney gives me any more context than a dialog box stating, "i messed up your print job." I'm being grumpy about this point, but I think it's fair to state the arguments against anthropomorphic agents.

The paper does a nice job of spelling out the various applications that have been developed with Barney. The real issue for me is the lack of any evaluations - formal or informal - of these applications. I'd be happy with knowing how people in the authors' workspace felt about Barney at the print queue or flagging incoming mail. Do people really see this type of technology as useful? Whether anthropomorphic or not, it would be useful to get a sense for reactions to computing that lives outside the monitor and keyboard. This paper does what much ubiquitous computing research falls prey to: nice technology, where's the impact? So while I think the work is interesting, it does nothing to advance my understanding of how such devices could be deployed in my workspace.

4. Comment on the paper's written presentation:

generally well written. argument comes across loud and clear. i loved the figures... as long as international readers understand "super-dee-duper"(tm), i think the paper satisfies that point as well.

5. Provide any other comments that may be useful to the authors.

I griped a lot, but I do think the paper adds a new dimension to the ubiquitous computing paradigm. Again, one day I'm sure someone will take the time to actually think about what these things do for people, but until then...

So I weigh towards acceptance of the paper. But if informal feedback from users could be added, I'd feel even stronger about the work.

6. Rating of Paper's acceptability (1 is low, 5 is high):

4

7. Rate your expertise in the topic area of this paper (1 is low, 5 is high):

5


Document ID: 99092-5

1. Briefly summarize the paper in your own words, making sure to identify the paper's contribution to the field of HCI:

Identifies interactive devices entitled Programmable Embodied Agents and describes a quick study of features of an prototype PEA Interactive Barney, resulting in an "Interaction Framework".

2. Select the phrase that best describes the paper's contribution:

Interaction technique

3. Review of the paper:

Concise introduction clearly stating area of discussion and aims of the paper. The Related Work section is not very comprehensive: the last paragraph is not justified in the context of related previous work in this area. In addition, it has a limited set of references relying on past CHI papers as a mainstay and the author does not take into account non-US work or work in the filed of Intelligent Agents (other than by Laurel) as I would have expected. The Technical Basis section is overlong with too much detail on interaction protocols: this part of the paper is more suitable for a technical report rather than a CHI submission. It would have been much more appropriate to focus on details of the software infrastructure: the figure given (Fig. 2) is fairly meaningless without explanation and claims that the applications are "lightweight, portable and seamless" are not justified. More information on the widget set would have been useful and made this paper more interesting and innovative: there is enough informa tion to whet the appetite but not enough for a comprehensive understanding of what the author's aims were. Again, Figure 4 is not explained at all well and the framework is not introduced or described clearly. The Channel/Person category especially hard to understand: simply referring the reader to other references is not sufficient. In all, the work, as the author states, was a "brief exploration" of a field which merits more detailed and careful investigation and a more carefully reasoned and thoughtful presentation.

Contribution - This is a contribution to a field which may well prove to be a very promising one in terms of interaction techniques and new designs. However this is obviously initial exploratory work, presented and written (I would guess) very quickly and needs a much more careful and well-designed study with all assumptions clearly stated and rationale suitably justified. This paper does not do that but may be of interest as the basis of future studies. Validity - Very little to comment on in terms of rationale: work presented seems to be mostly in terms of "these are my experiences".

Originality - Original but with a limited base.

4. Comment on the paper's written presentation:

Written presentation - Sometimes well and clearly written but lacking in consistency. The text does become dense in places (Related Work's last paragraphs for example) and with a number of throw-way lines or a "chatty" style in others. In addition, the references in the paper are not in CHI standard format.

5. Provide any other comments that may be useful to the authors.

6. Rating of Paper's acceptability (1 is low, 5 is high):

2

7. Rate your expertise in the topic area of this paper (1 is low, 5 is high):

3