I went through that code. Elaborate a bit more on how you want your web crawler to be.
Things to consider will be.
1. Java networking concept
2. HTTP Protocol and HTML
3. Text Parsing (Remember how he extracted <a href ..
Commons HTTP/HTTP Component, Regex, Simple IO are welcome as well.
Program you used is not ROBUST enough. Better you start from scratch.