Informatics 45 Spring 2010, Project #3: The Great Beyond

Project Plan due: Saturday, May 15, 11:59pm
Program due: Monday, May 24, 11:59pm
Lessons Learned due: Tuesday, May 25, 11:59pm

You are required to work on this project individually

Introduction

The first two projects helped you to take two large steps that extended the boundaries of what you could accomplish by programming. The first project asked you to build a program with a graphical user interface; the second asked you to build a program that could establish a socket connection and send an image from one program to another. The overarching lesson for most of you, I hope, is that it didn't take a lot of new Java skills to solve either of these problems. There were a few things that were new (the GUI was easier if you used anonymous inner classes; sockets required some light multithreading, especially when mixed with GUIs), but what you knew how to do already went a long way; the difficulty was mostly in discovering the details of the APIs that you needed to solve the problem, which can be overcome with a combination of prototyping, experimentation, adapting the code examples from lecture (or others you may have found online), and searching the Internet for information when you're stuck.

The second project, in particular, dramatically pushed the boundaries of what you could accomplish outward because, for many of you, it was your first time writing a program that could communicate with a separate program. It introduced communication protocols and showed that it doesn't take much code to get two programs talking to each other via a network, while underscoring the importance of both programs agreeing on the specifics of how the conversation should be structured. Once you have this tool in your toolbox, a whole new world opens up: suddenly, the idea that you should be able to write a program that combines, say, Google search queries, the Internet Movie Database, and your favorite social network to find people who like movies similar to the ones you like doesn't seem far-fetched.

The challenge in getting programs to share information is twofold. Firstly, there's purely a software engineering problem: a protocol has to be designed that both programs can use to have their conversation. Secondly, there's a social problem: if the same person (or group of people) isn't writing both programs, it's necessary for them to agree on the protocol ahead of time, then to implement it. This is a potentially catastrophic problem — how could you ever get Google to agree to use your protocol just to communicate with you? — that is largely solved by the presence of standards, such as those defined by the World Wide Web Consortium. Standards help in two ways: by providing a detailed communication protocol whose details have already been hammered out, and also by allowing programs to interoperate with many other programs without having to support a separate protocol for each of them. In fact, standard protocols often have standard implementations, so that you won't have to code up the details yourself; for example, Java has built-in support for a number of standard protocols, including HTTP (Hypertext Transfer Protocol, the protocol that your browser uses to download web pages) among others.

HTTP is of particular importance, since it is the protocol that is used for virtually all web communications. This is not limited only to the conversation that your browser has with a web server in order to download a web page, though that conversation certainly uses HTTP. HTTP is the protocol that underlies a growing variety of program-to-program communications using web protocols, broadly called web services, where web sites or other software systems communicate directly with one another, or where programs other than web browsers fetch data from and affect change on web sites or other software systems. Fortunately, since HTTP support is built directly into Java, we can write programs that use web-based APIs without having to handle low-level details of the protocol, though there are some details you'll need to be familiar with if you want to use the provided implementation effectively.

This project gives you the opportunity to explore a small slice of the vast sea of possibilities presented by web APIs and web services. You'll get the chance to choose a problem related to the web that you're interested in solving by writing a Java program — though I'll provide one for you to solve if you're having trouble choosing — and then to investigate the details and implement your solution. The planning phase will be of special importance in this project, since part of how you'll settle on the details of your problem will be based on experimentation and prototyping, so that you can understand what you can reasonably expect yourself to accomplish in the relatively short time provided. We hope that, when you're done, you'll have a program that you can actually use on a day-to-day basis, rather than one that you'll submit to us and never look at again.

Reminder: Do not partner up!

For this project, your work is expected to completed individually, so do not partner up and do not use pair programming.

The problem

This project provides you with a choice of problems to work on. You're asked to choose between a problem that I've posed or a problem of your own. The choice is entirely yours; while we encourage you to explore your own ideas if you have them, there's neither a penalty for deciding to solve the provided problem instead nor extra credit for choosing your own. Do consider accepting the freedom that's being offered here, but realize that the freedom is its own reward.

What is Twitter?

Twitter is a web site that combines what is called microblogging (sharing short messages on the Internet, rather than longer articles) with what is called moblogging (the ability to post those messages "on the go," by using, for example, a mobile phone). Twitter has emerged as an exciting Internet phenomenon; its popularity has not only generated hype, but has even elicited academic research such as the study summed up by this paper. Twitter is claimed by many to be useful for a variety of purposes. Check out this blog for a randomly-chosen person's ideas on what Twitter is good for.

Twitter provides you with a relatively small number of features, such as these:

Post short messages of no more than 140 characters called status updates or, more colloquially, tweets
Register yourself as a follower of another user's messages, so that you'll be notified whenever they tweet
See who your followers are
Keep track of which tweets are your favorites
Post and receive tweets using the text messaging feature of your mobile phone

Arising from this small feature set is a potentially powerful tool for social networking, online conversation, and even guerrilla journalism. Like many such tools, it's not the web interface that's important; it's the data and how it interconnects that makes it powerful. For that reason, Twitter provides an API for fetching and manipulating this data outside of the confines of a web browser. It's this API that you will explore if you choose to solve the provided problem.

The provided problem: Manipulating your own Twitter account via the Twitter API using Java

As we saw in lecture, Twitter provides an API for interacting with Twitter even if you're not using your web browser. That API is documented in a fair amount of detail at the following link:

Twitter API Documentation

Before you get started, you'll want to create a new Twitter account to interact with: go to twitter.com and click the button that encourages you to "Get Started — Join!" (If you already have a Twitter account, you might not want to use it for your experimentation at first, so it's a good idea to create a fresh account that you can play with.) Especially if you're new to Twitter, be sure to spend a little bit of time experimenting with it, so you can be clear on how it works. Post updates, sign yourself up as a follower of a few other people, check out who's following you, and so on. Understanding how Twitter works is important context for understanding how to work through this problem.

If you choose to work on the provided problem, your program should meet the following requirements.

It should monitor messages posted by friends, looking for messages containing keywords specified by the user. So, for example, you might watch for messages with the words "concert" and "tonight" in them, for example.
The program should display only messages that have these keywords. The program should support any number of keywords and should allow users to specify whether messages should require all or any of the keywords.
It's fine for the program to require the user to specify a new list of keywords each time the program starts; you can also save this information into a file and reload it each time, if you prefer.
The program should also provide the ability for the user to post a public response to any of the messages that it displays.
- Note that, by convention, messages that are posted publicly but are intended for a particular user begin with an "@" followed by the recipient's name. So, for example, to post a reply to the i45s2010 account we saw in class, you might post this message: "@i45s2010: I agree". The "@" convention should be handled by the program automatically.
The program should not check for messages so quickly that it overwhelms the limited number of requests per hour allowed by the Twitter API. Remember to leave yourself room under that limit to post the replies.
- You may make the rate at which your program checks for new messages configurable, or you can hard-code a rate that you think is reasonable. If it's configurable, it would be wise to hard-code a limit that won't allow the user to blow through Twitter's API limits.
Provides a graphical user interface that makes access to the information and functionality simple and straightforward. As usual, the design of that user interface is up to you.
Stick to XML output from each Twitter web "method," rather than the other alternatives (JSON, RSS, Atom) that Twitter provides. Java has built-in support for XML, as we saw in lecture; the others will require you to find or write code that supports them.

Be respectful of the rules governing the use of the Twitter API, including the limited number of web "method" calls per hour; note that this limit may affect your ability to work on and test your program at the pace you'd like, so be sure to plan for the effect this limit will have on your work. (One way to handle this, for example, is to be sure that you make requests more slowly or, at least, less frequently while testing your program.) When the API requires authentication, use only usernames and passwords that belong to you.

You will certainly run up against situations where you receive responses that you were not expecting, hit limitations that you were not aware of, or notice behavior that does not seem to make sense. While we will do our best to help you with these, we are not seasoned Twitter API experts, so we may not always have an immediate answer to your question. (This is a very typical issue in real-world software development: no one is an expert on everything, and any one person is expert at only a tiny fraction of what's out there in the world. The ability to overcome this is paramount if you want to be able to build and maintain real software.) Note that Internet searches are likely to help, and that there is a discussion group for users of the Twitter API.

Coming up with your own problem instead

Rather than working on the problem provided above, you are free to come up with your own problem instead, involving Twitter or any other web site/service (or combination of web sites/services) you'd like. In general, anything goes, though there are a few ground rules that you'll be asked to follow if you choose your own problem, just so that everyone can be on a somewhat level playing field.

You will need to use at least one web site/web service API that is accessible from Java. Ideally, you'll stick with APIs like the ones we experimented with in lecture, but you're welcome to use others that work differently than those we discussed if you're so inclined. You're also not strictly limited to protocols that sit atop HTTP; you can do something with, say, instant messaging, though be aware that you may be making your job significantly harder if you're using something with a proprietary protocol rather than a standardized one.
You are required to call at least three different web "methods" as part of your solution. The substantially interesting part of your program should involve either the creation, consumption, or combination of information sent to or received from these methods.
The goal of your program would ideally be to automate something that is not normally easy to do using the regular browser-based interface provided by your chosen web site.
You are required to avoid violating any license agreement or other rules put in place by your chosen web site(s)/service(s). For example, Twitter requires that you make no more than a certain number of requests (web "method" calls) per hour, so you wouldn't want to write a Twitter "spamming" tool. (For ethical reasons, you might well want to avoid this kind of tool anyway.)
If your chosen API requires authentication, you are required to use only usernames, passwords, etc., that belong to you. (You are free to — and may very well want to — create new accounts just for this course, assuming that it is allowed by the site's license agreement.)
- You will need to give us an account to use for testing, so be sure that this is allowed by the site's license agreement (e.g., do not give us an account you've paid for). Note that this may limit your choices about what problems you can work on.
You are permitted to use pre-existing code that you find online to make the job of communication easier, so long as that code is licensed, at least, for personal (non-commercial) use, and so long as you make it very clear what code came from other sources than yourself and what those sources were. This means, for example, that you could use an open source library to simplify connecting to your chosen web API and receiving its result; we expect all other code to be written by you. Of course, you're not permitted to take code from other students in the course or students who have taken this course previously, nor are you permitted to have other people write code specifically for you. If you're not sure whether you're allowed to use code that you find, be sure to ask us.

If you're working on your own problem, you'll need to get it approved by me early in the process. By Thursday, May 13 at 8:00pm, you'll need to send me an email that describes:

What problem you plan to solve — be detailed enough that I can assess whether the problem is solvable in the time allowed; don't say, for example, "I plan to automate web searching using Google and Yahoo!"
What web site(s)/service(s) are involved
What APIs and/or open source libraries you plan to use — please include links to API documentation, since it's a good bet that you'll be choosing something I've never used before
Roughly what you expect your user interface will look like when you're done, or at least what functionality you expect to provide

The importance of embracing freedom

It may make some of you uncomfortable that you're being encouraged to find your own problem to solve in this project. This may not be something you've been asked to do often in school — though you'll find that it happens increasingly the farther you get, especially if you go to graduate school. You may feel uncomfortable because you're not sure what you should do, becuase you're not sure whether your idea is more than you can handle in the time provided, or because you don't feel it's clear how you'll be graded relative to your peers. In short, doing your own thing will feel like a risk, and you may be risk-averse when it comes to courses in which you'll be receiving a grade.

The thing to remember here is that I've left this project open-ended because I've wanted you to be able to find problems that excite you, as opposed to wanting to grade you on the quality of the problem you chose. No one will be graded down specifically because they chose to work on the provided problem rather than finding their own. In fact, no one will be graded on their choice of problem at all, so long as we approved your problem during the planning phase. You will instead be graded on the quality of what you've delivered: whether it solves the problem posed, whether you provided us enough information to figure out how to use it, and so on.

If you have a problem that you're interested in solving, let go of your fears and embrace the freedom to do it. We'll help you decide whether your problem is solvable in the given time, we'll try to help you whittle it down to something manageable if it's too big, and the problem of grading will work itself out. (And, though it's easy for me to say, I'll remind you that grades are not the purpose of your education; don't let them keep you from getting what you came for!)

The planning phase

Particularly if you decide to work on something other than the provided Twitter problem — but even if you don't — you are going to want to spend some time planning your program and experimenting with some prototype code before you implement your final program. As before, prototyping is going to be very important, since you'll be working with technologies that are almost certain to be new to you.

The project plan

As always, you'll be required to write a project plan that summarizes the planning you did, addressing at least the following questions. (You're free, again, to include any additional information you'd like; spending a little time putting your thoughts down in writing is a good way to get your thoughts organized.)

Describe in a fair amount of detail what problem you plan to solve. (If you plan to work on the provided problem, you can just say "I plan to work on the provided problem" and move on.)
How will your user interface be designed? What will the user interface do while the program communicates with the web site/service? How will you show, in the user interface, that you couldn't establish a connection or couldn't parse the result successfully? (This can happen.)
List the tasks you expect to have to complete before you've finished your implementation. We're not expecting you to break this down at the level of individual methods or classs; we'd just like to see what the big-picture tasks are. Which tasks are dependent on which others?
What features did you prototype, if any? How did you prototype them?
Guided by your prototyping and past experience, what aspects of the project pose the biggest risk to your success? Where do you anticipate having the most difficulty?
Guided by your prototyping and past experience, what aspects of the project pose the least risk to your success? What parts do you expect to be able to handle with relative ease?

The project plan is due on Saturday, May 15 at 11:59pm. See the section titled "Deliverables" below for more information about how to submit the various parts of your project.

Writing the program

The program should be written entirely in Java. The GUI should be built using the Swing library. In addition to Swing, you are free to use any part of the Java library that you would find helpful. Additionally, you are permitted to use other components (e.g., open source components) to help you connect to your web site/service or parse its results, though not for any other part of your solution.

In addition to delivering your program, you'll also need to submit a brief document that explains how to use the program. You won't need to say much here — a page or so will suffice for most of your programs, I'd guess — but we are going to need to be told enough information that we can understand and use your program. In that document, you will need to supply us with a username and password, or other authentication information necessary, for us to use the program with your chosen web site/service, if authentication is necessary. (Note that we have a Twitter account that we can use for testing, but any other site will require you to give us an account we can use freely.) Assume that we will do anything we'd like with the account while testing your program, so do not give us an account you care about. (It's usually going to be best for you to create a new account for your work in this course, then to give it to us as part of your submission. Please put enough information into the account — messages, photos, links, or whatever — for it to be an interesting test of your program.) Be sure that it is within the license agreement of your chosen web site/service to give out this authentication information (e.g., do not give us an account you've paid for). Note that this may limit your choices about what problems you can work on.

Lastly, if you used any open source libraries provided as JAR files, you'll need to include those JAR files in your submission.

The program is due on Monday, May 24 at 11:59pm. See the section titled "Deliverables" below for more information about how to submit the various parts of your project.

It may seem unreasonable, at first blush, that you only have six days to write the program, but note that I'm expecting that you will have written some code already during the planning phase, so this deadline is not as tight as it initially seems. Note, also, that this project may not require as much code as you might imagine, since Java's library includes built-in support for web protocols like HTTP.

Assessing the lessons learned

Once you've completed your implementation and submitted it, take a little time to reflect on your experience by writing a lessons learned document. Your lessons learned document should reflect on, at least, the following questions.

To what extent did you follow your project plan while implementing the program? What features did you implement differently than you specified in your plan? Why did you decide to implement them differently?
What are a couple of your biggest successes on this project — features that you felt turned out well, challenges that you were able to overcome? What forces do you think helped you to be so successful in these cases?
What are a couple of aspects of this project that could have gone better? Why do you think they were problematic and what will you do differently next time to ensure that you'll handle them better?
What did you think of the web site/service API that you chose? Did it make it easier to solve your problem, or did you find yourself fighting it?
In what ways did your chosen web site/service API behave as you expected to? In what ways did it surprise you? If you could have changed one thing about the API, what would it have been?

The lessons learned document is due on Tuesday, May 25 at 11:59pm. See the section titled "Deliverables" below for more information about how to submit the various parts of your project.

Deliverables

You are required to deliver the three parts of the project to Checkmate, an ICS-built online assignment submission system. Follow this link for a discussion of how to submit files via Checkmate. Be aware that I'll be holding you to all of the rules specified in that document, including the one that says that you're responsible for submitting the version of the project that you want graded. We won't regrade a project simply because you submitted the wrong version accidentally.

There are three parts to this project, each with its own due date:

The project plan, which is a Microsoft Word (.doc or .docx), Rich Text Format (.rtf), or PDF document. This is due on Saturday, May 15 at 11:59pm.
The program, which is three things: a collection of .java files comprising your program, a document explaining how to use it, and any JAR files containing open source libraries that you used (if any). Please submit a zip file containing all of your these things, but no other files (e.g., compiled .class files, other parts of your Eclipse workspace). Be sure to zip the .java files while still in the folder structure imposed by Eclipse and Java when you use packages. The program is due on Monday, May 24 at 11:59pm.
The lessons learned document, which is another Microsoft Word (.doc or .docx), Rich Text Format (.rtf), or PDF document. This is due on Tuesday, May 25 at 11:59pm.

After you've submitted your work, take a moment to be sure that you submitted all of the files you intended to; if you missed one, we won't be able to compile and run your program, which can result in a substantial penalty, since we won't be able to evaluate your program's correctness.