Topic
 
|
1/22/2009 Lecture Notes |
|
|
Kick-off: |
|
|
Prospect.1 New Orleans Biennial 2008 |
|
|
Learning Objective: |
|
|
"Be able to describe |
|
the steps of a crawling algorithm |
|
|
the desired characteristics of a web crawler |
|
|
why a good crawler is a complex piece of software |
|
|
how a website operator can control web crawler behavior |
|
|
how DNS resolution impacts crawling |
|
|
Survey Results (probably later) |
|
|
Cards |
|
|
Review Cards |
|
|
What does fetch mean? |
|
|
Do we have to be polite? |
|
|
Concerned about my grade |
|
|
Work in groups. Work efficiently. |
|
|
Where does the seed set come from? |
|
|
How do we demo for 4 if it's going to take so long? |
|
|
How do you get permission to crawl databases directly? |
|
|
Spider traps are still confusing |
|
|
Video break |
|
|
None today |