Open a URL and read it line by line (Works in Eclipse but not from Command Line)
Hi, I am trying to use this code which works fine in Eclipse but when I try to execute the program from the command line it just hangs forever and can never accesses the URL.
I am trying to run this program from my home computer on a localhost and I have no idea why this isn't working. I read online and some articles suggested using a proxy but I am not behind a proxy or anything. And again this code works fine when I run it in Eclipse so I guess Eclipse is auto-configuring settings to get this code to work.
URL link = new URL("http://www.yahoo.com");
System.out.println("got to this line 1");
BufferedReader in = new BufferedReader(new InputStreamReader(link.openStream()));
System.out.println("got to this line 2");
//InputStream in = link.openStream();
String inputLine = "";
int count = 0;
while ((inputLine = in.readLine()) != null)
{
System.out.println("line: " + inputLine);
site = site + "\n" + inputLine;
}
in.close();
System.out.println("got to this line 3");
It starts printing the website but never gets to "got to this line 3". It always stops printing the website at the same spot and gives me a connection reset error.
This was also posted here. If either thread finds a solution I will post it:
http://www.javaprogrammingforums.com...mand-line.html
Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)
I ran this peace of code from my computer in command line and it worked fine, i can't figure out what is wrong!
Could you post the complete code please, maybe there is another kind of problem?
Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)
Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)
Well it consists of multiple classes and everything so I can't put all of the code. But here is the function I use below. I'm 99% sure it has to do with the JVM and proxy settings or something. Because the program starts and just hangs forever when it tries to open the URL. The code works perfectly in Eclipse though.
public static String websiteToString(String websiteLink) throws IOException
{
String site = "";
try
{
/* Read a URL and parse it's source.
* Add each line of the source to a string */
//System.setProperty("java.net.useSystemProxies","tr ue");
//System.getProperties().put("proxySet", "true");
//System.setProperty("http.proxyHost", "webcache.mydomain.com");
//System.setProperty("http.proxyPort", "80");
URL link = new URL("http://www.yahoo.com");
BufferedReader in = new BufferedReader(new InputStreamReader(link.openStream()));
//InputStream in = link.openStream();
String inputLine = "";
int count = 0;
while ((inputLine = in.readLine()) != null)
{
site = site + "\n" + inputLine;
}
in.close();
}
catch (IOException e)
{
System.out.println("ERROR: Website " + websiteLink + " can't be opened and/or found.");
}
site = site.replaceAll("<html([^<]*)>", "<html>"); // replace all <html xml:lang="en" lang="en" xmlns:xml="xml"> with <html>
// this is important because xpath can't parse an expression with <html> tags with junk in it
return site;
}
Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)
Quote:
Originally Posted by
doWhile 
Yes I'm sorry I didn't know your two websites were "related" or had the viewers? I will post a solution if I find on in the other thread to not waste anyone's time.
Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)
I edited the code to look like this in my program and re-ran it from the command line.
URL link = new URL("http://www.yahoo.com");
System.out.println("got to this line 1");
BufferedReader in = new BufferedReader(new InputStreamReader(link.openStream()));
System.out.println("got to this line 2");
//InputStream in = link.openStream();
String inputLine = "";
int count = 0;
while ((inputLine = in.readLine()) != null)
{
site = site + "\n" + inputLine;
}
in.close();
System.out.println("got to this line 3");
It gets to "got to this line 2" but not "got to this line 3".
Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)
One question:
in the method websiteToString() the URL object takes the wesiteLink as a parameter not the yahoo.com right?
Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)
I edited the code to look like this in my program and re-ran it from the command line.
URL link = new URL("http://www.yahoo.com");
System.out.println("got to this line 1");
BufferedReader in = new BufferedReader(new InputStreamReader(link.openStream()));
System.out.println("got to this line 2");
//InputStream in = link.openStream();
String inputLine = "";
int count = 0;
while ((inputLine = in.readLine()) != null)
{
System.out.println("line: " + inputLine);
site = site + "\n" + inputLine;
}
in.close();
System.out.println("got to this line 3");
It starts printing the website but never gets to "got to this line 3".
Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)
Quote:
Originally Posted by
DeNiS_M One question:
in the method websiteToString() the URL object takes the wesiteLink as a parameter not the yahoo.com right?
Yes that can be ignored for now. The real program does not run this method on http://www.yahoo.com. I am just using http://www.yahoo.com for testing purposes. I normally use the link I pass in from the parameters but since I know there is nothing wrong with yahoo.com I am just going to use that for now until I get this working.
Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)
Quote:
Originally Posted by
rosco544 Yes I'm sorry I didn't know your two websites were "related" or had the viewers? I will post a solution if I find on in the other thread to not waste anyone's time.
It has nothing to do with the sites being "related" (they're not). We're just tired of working to help folks only to find that they've solved their issue on another site hours before. We're all volunteers and value our time, and we prefer to help others who respect this. The bottom line is if you cross-post anywhere, you should provide links to all cross-posts in all threads.
Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)
Quote:
Originally Posted by
Fubarable It has nothing to do with the sites being "related" (they're not). We're just tired of working to help folks only to find that they've solved their issue on another site hours before. We're all volunteers and value our time, and we prefer to help others who respect this. The bottom line is if you cross-post anywhere, you should provide links to all cross-posts in all threads.
Ok, as I said before though if I find a solution anywhere I would post and let everybody know in both threads. I'm not trying to waste anyone's time that is why I am trying to post as much information as possible.
Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)
Quote:
Originally Posted by
rosco544 Ok, as I said before though if I find a solution anywhere I would post and let everybody know in both threads. I'm not trying to waste anyone's time that is why I am trying to post as much information as possible.
That's fine, but still provide links for all cross-posts as we should be able to follow all discussions.
Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)
I can't figure out what is going wrong but i will make a suggestion:
Code:
String site = "";
BufferedReader in= null;
try
{
URL link = new URL(args[0]);
in = new BufferedReader(new InputStreamReader(link.openStream()));
//InputStream in = link.openStream();
String inputLine = "";
int count = 0;
while ((inputLine = in.readLine()) != null)
{
site = site + "\n"+ inputLine;
}
in.close();
site = site.replaceAll("<html([^<]*)>", "<html>");
System.out.println(site);
System.out.println("*********Here***************");
}
catch(MalformedURLException me)
{
System.out.println("Inside malformed exception: "+me.getMessage());
}
catch(IOException ioe)
{
System.out.println("**********************Inside ioexception: "+ioe.getMessage());
}
finally
{
if(in == null) in.close();
}
Could you run this piece of code and reply to me please?
Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)
Quote:
Originally Posted by
DeNiS_M I can't figure out what is going wrong but i will make a suggestion:
Code:
String site = "";
BufferedReader in= null;
try
{
URL link = new URL(args[0]);
in = new BufferedReader(new InputStreamReader(link.openStream()));
//InputStream in = link.openStream();
String inputLine = "";
int count = 0;
while ((inputLine = in.readLine()) != null)
{
site = site + "\n"+ inputLine;
}
in.close();
site = site.replaceAll("<html([^<]*)>", "<html>");
System.out.println(site);
System.out.println("*********Here***************");
}
catch(MalformedURLException me)
{
System.out.println("Inside malformed exception: "+me.getMessage());
}
catch(IOException ioe)
{
System.out.println("**********************Inside ioexception: "+ioe.getMessage());
}
finally
{
if(in == null) in.close();
}
Could you run this piece of code and reply to me please?
It just hangs and doesn't get to any of your debug statements.
When I got back to my original code and add the debug messages I am getting a connection reset message.
Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)
Have you checked the firewall settings.Maybe there is something wrong there!
Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)
Quote:
Originally Posted by
DeNiS_M Have you checked the firewall settings.Maybe there is something wrong there!
I"m just on Windows Firewall with AVG Free Edition. What settings would I even look for that would be causing this error.
I also don't think it's the firewall because it begins to read the website but it just hangs. I inserted a print statement inside the while loop and it starts to print the website but just stops randomly before finishing.
Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)
Finally solved it with this code found here. Not sure why this works but it works: Java HttpURLConnection (how to open and read a url with Java) | devdaily.com
public String websiteToString(String desiredUrl) throws Exception
{
URL url = null;
BufferedReader reader = null;
StringBuilder stringBuilder;
try
{
// create the HttpURLConnection
url = new URL(desiredUrl);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
// just want to do an HTTP GET here
connection.setRequestMethod("GET");
// uncomment this if you want to write output to this url
//connection.setDoOutput(true);
// give it 15 seconds to respond
connection.setReadTimeout(15*1000);
connection.connect();
// read the output from the server
reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
stringBuilder = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null)
{
stringBuilder.append(line + "\n");
}
String site = stringBuilder.toString();
site = site.replaceAll("<html([^<]*)>", "<html>"); // replace all <html xml:lang="en" lang="en" xmlns:xml="xml"> with <html>
// this is important because xpath can't parse an expression with <html> tags with junk in it
return site;
}
catch (Exception e)
{
e.printStackTrace();
throw e;
}
finally
{
// close the reader; this can throw an exception too, so
// wrap it in another try/catch block.
if (reader != null)
{
try
{
reader.close();
}
catch (IOException ioe)
{
ioe.printStackTrace();
}
}
}
}
