Results 1 to 17 of 17
Like Tree1Likes
  • 1 Post By DeNiS_M

Thread: Open a URL and read it line by line (Works in Eclipse but not from Command Line)

  1. #1
    rosco544 is offline Member
    Join Date
    Sep 2011
    Posts
    10
    Rep Power
    0

    Default Open a URL and read it line by line (Works in Eclipse but not from Command Line)

    Hi, I am trying to use this code which works fine in Eclipse but when I try to execute the program from the command line it just hangs forever and can never accesses the URL.

    I am trying to run this program from my home computer on a localhost and I have no idea why this isn't working. I read online and some articles suggested using a proxy but I am not behind a proxy or anything. And again this code works fine when I run it in Eclipse so I guess Eclipse is auto-configuring settings to get this code to work.


    URL link = new URL("http://www.yahoo.com");
    System.out.println("got to this line 1");
    BufferedReader in = new BufferedReader(new InputStreamReader(link.openStream()));
    System.out.println("got to this line 2");
    //InputStream in = link.openStream();
    String inputLine = "";
    int count = 0;
    while ((inputLine = in.readLine()) != null)
    {
    System.out.println("line: " + inputLine);
    site = site + "\n" + inputLine;
    }
    in.close();
    System.out.println("got to this line 3");

    It starts printing the website but never gets to "got to this line 3". It always stops printing the website at the same spot and gives me a connection reset error.

    This was also posted here. If either thread finds a solution I will post it:
    http://www.javaprogrammingforums.com...mand-line.html
    Last edited by rosco544; 09-17-2011 at 12:43 AM.

  2. #2
    DeNiS_M is offline Member
    Join Date
    Sep 2011
    Location
    Athens Greece
    Posts
    29
    Rep Power
    0

    Default Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)

    I ran this peace of code from my computer in command line and it worked fine, i can't figure out what is wrong!
    Could you post the complete code please, maybe there is another kind of problem?
    Fubarable likes this.

  3. #3
    doWhile is offline Moderator
    Join Date
    Jul 2010
    Location
    California
    Posts
    1,641
    Rep Power
    7

    Default Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)


  4. #4
    rosco544 is offline Member
    Join Date
    Sep 2011
    Posts
    10
    Rep Power
    0

    Default Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)

    Well it consists of multiple classes and everything so I can't put all of the code. But here is the function I use below. I'm 99% sure it has to do with the JVM and proxy settings or something. Because the program starts and just hangs forever when it tries to open the URL. The code works perfectly in Eclipse though.

    public static String websiteToString(String websiteLink) throws IOException
    {
    String site = "";

    try
    {

    /* Read a URL and parse it's source.
    * Add each line of the source to a string */
    //System.setProperty("java.net.useSystemProxies","tr ue");
    //System.getProperties().put("proxySet", "true");
    //System.setProperty("http.proxyHost", "webcache.mydomain.com");
    //System.setProperty("http.proxyPort", "80");
    URL link = new URL("http://www.yahoo.com");

    BufferedReader in = new BufferedReader(new InputStreamReader(link.openStream()));
    //InputStream in = link.openStream();
    String inputLine = "";
    int count = 0;
    while ((inputLine = in.readLine()) != null)
    {
    site = site + "\n" + inputLine;
    }
    in.close();

    }
    catch (IOException e)
    {
    System.out.println("ERROR: Website " + websiteLink + " can't be opened and/or found.");
    }

    site = site.replaceAll("<html([^<]*)>", "<html>"); // replace all <html xml:lang="en" lang="en" xmlns:xml="xml"> with <html>
    // this is important because xpath can't parse an expression with <html> tags with junk in it
    return site;
    }

  5. #5
    rosco544 is offline Member
    Join Date
    Sep 2011
    Posts
    10
    Rep Power
    0

    Default Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)

    Yes I'm sorry I didn't know your two websites were "related" or had the viewers? I will post a solution if I find on in the other thread to not waste anyone's time.
    Last edited by rosco544; 09-16-2011 at 11:41 PM.

  6. #6
    rosco544 is offline Member
    Join Date
    Sep 2011
    Posts
    10
    Rep Power
    0

    Default Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)

    I edited the code to look like this in my program and re-ran it from the command line.

    URL link = new URL("http://www.yahoo.com");
    System.out.println("got to this line 1");
    BufferedReader in = new BufferedReader(new InputStreamReader(link.openStream()));
    System.out.println("got to this line 2");
    //InputStream in = link.openStream();
    String inputLine = "";
    int count = 0;
    while ((inputLine = in.readLine()) != null)
    {
    site = site + "\n" + inputLine;
    }
    in.close();
    System.out.println("got to this line 3");

    It gets to "got to this line 2" but not "got to this line 3".

  7. #7
    DeNiS_M is offline Member
    Join Date
    Sep 2011
    Location
    Athens Greece
    Posts
    29
    Rep Power
    0

    Default Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)

    One question:
    in the method websiteToString() the URL object takes the wesiteLink as a parameter not the yahoo.com right?

  8. #8
    rosco544 is offline Member
    Join Date
    Sep 2011
    Posts
    10
    Rep Power
    0

    Default Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)

    I edited the code to look like this in my program and re-ran it from the command line.

    URL link = new URL("http://www.yahoo.com");
    System.out.println("got to this line 1");
    BufferedReader in = new BufferedReader(new InputStreamReader(link.openStream()));
    System.out.println("got to this line 2");
    //InputStream in = link.openStream();
    String inputLine = "";
    int count = 0;
    while ((inputLine = in.readLine()) != null)
    {
    System.out.println("line: " + inputLine);
    site = site + "\n" + inputLine;
    }
    in.close();
    System.out.println("got to this line 3");

    It starts printing the website but never gets to "got to this line 3".

  9. #9
    rosco544 is offline Member
    Join Date
    Sep 2011
    Posts
    10
    Rep Power
    0

    Default Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)

    Quote Originally Posted by DeNiS_M View Post
    One question:
    in the method websiteToString() the URL object takes the wesiteLink as a parameter not the yahoo.com right?

    Yes that can be ignored for now. The real program does not run this method on http://www.yahoo.com. I am just using http://www.yahoo.com for testing purposes. I normally use the link I pass in from the parameters but since I know there is nothing wrong with yahoo.com I am just going to use that for now until I get this working.

  10. #10
    Fubarable's Avatar
    Fubarable is offline Moderator
    Join Date
    Jun 2008
    Posts
    19,316
    Blog Entries
    1
    Rep Power
    26

    Default Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)

    Quote Originally Posted by rosco544 View Post
    Yes I'm sorry I didn't know your two websites were "related" or had the viewers? I will post a solution if I find on in the other thread to not waste anyone's time.
    It has nothing to do with the sites being "related" (they're not). We're just tired of working to help folks only to find that they've solved their issue on another site hours before. We're all volunteers and value our time, and we prefer to help others who respect this. The bottom line is if you cross-post anywhere, you should provide links to all cross-posts in all threads.

  11. #11
    rosco544 is offline Member
    Join Date
    Sep 2011
    Posts
    10
    Rep Power
    0

    Default Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)

    Quote Originally Posted by Fubarable View Post
    It has nothing to do with the sites being "related" (they're not). We're just tired of working to help folks only to find that they've solved their issue on another site hours before. We're all volunteers and value our time, and we prefer to help others who respect this. The bottom line is if you cross-post anywhere, you should provide links to all cross-posts in all threads.
    Ok, as I said before though if I find a solution anywhere I would post and let everybody know in both threads. I'm not trying to waste anyone's time that is why I am trying to post as much information as possible.

  12. #12
    Fubarable's Avatar
    Fubarable is offline Moderator
    Join Date
    Jun 2008
    Posts
    19,316
    Blog Entries
    1
    Rep Power
    26

    Default Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)

    Quote Originally Posted by rosco544 View Post
    Ok, as I said before though if I find a solution anywhere I would post and let everybody know in both threads. I'm not trying to waste anyone's time that is why I am trying to post as much information as possible.
    That's fine, but still provide links for all cross-posts as we should be able to follow all discussions.

  13. #13
    DeNiS_M is offline Member
    Join Date
    Sep 2011
    Location
    Athens Greece
    Posts
    29
    Rep Power
    0

    Default Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)

    I can't figure out what is going wrong but i will make a suggestion:

    Java Code:
    String site = "";
            BufferedReader in= null;
            try
            {
                URL link = new URL(args[0]);
    
                in = new BufferedReader(new InputStreamReader(link.openStream()));
                //InputStream in = link.openStream();
                String inputLine = "";
                int count = 0;
                while ((inputLine = in.readLine()) != null)
                {
                    site = site + "\n"+ inputLine;
                }
                in.close();
                site = site.replaceAll("<html([^<]*)>", "<html>");
                System.out.println(site);
                System.out.println("*********Here***************");
            }
            catch(MalformedURLException me)
            {
                System.out.println("Inside malformed exception: "+me.getMessage());
            }
            catch(IOException ioe)
            {
                System.out.println("**********************Inside ioexception: "+ioe.getMessage());
            }
            finally
            {
                if(in == null) in.close();
            }
    Could you run this piece of code and reply to me please?
    Last edited by DeNiS_M; 09-17-2011 at 12:08 AM.

  14. #14
    rosco544 is offline Member
    Join Date
    Sep 2011
    Posts
    10
    Rep Power
    0

    Default Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)

    Quote Originally Posted by DeNiS_M View Post
    I can't figure out what is going wrong but i will make a suggestion:

    Java Code:
    String site = "";
            BufferedReader in= null;
            try
            {
                URL link = new URL(args[0]);
    
                in = new BufferedReader(new InputStreamReader(link.openStream()));
                //InputStream in = link.openStream();
                String inputLine = "";
                int count = 0;
                while ((inputLine = in.readLine()) != null)
                {
                    site = site + "\n"+ inputLine;
                }
                in.close();
                site = site.replaceAll("<html([^<]*)>", "<html>");
                System.out.println(site);
                System.out.println("*********Here***************");
            }
            catch(MalformedURLException me)
            {
                System.out.println("Inside malformed exception: "+me.getMessage());
            }
            catch(IOException ioe)
            {
                System.out.println("**********************Inside ioexception: "+ioe.getMessage());
            }
            finally
            {
                if(in == null) in.close();
            }
    Could you run this piece of code and reply to me please?
    It just hangs and doesn't get to any of your debug statements.

    When I got back to my original code and add the debug messages I am getting a connection reset message.
    Last edited by rosco544; 09-17-2011 at 12:31 AM.

  15. #15
    DeNiS_M is offline Member
    Join Date
    Sep 2011
    Location
    Athens Greece
    Posts
    29
    Rep Power
    0

    Default Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)

    Have you checked the firewall settings.Maybe there is something wrong there!

  16. #16
    rosco544 is offline Member
    Join Date
    Sep 2011
    Posts
    10
    Rep Power
    0

    Default Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)

    Quote Originally Posted by DeNiS_M View Post
    Have you checked the firewall settings.Maybe there is something wrong there!
    I"m just on Windows Firewall with AVG Free Edition. What settings would I even look for that would be causing this error.

    I also don't think it's the firewall because it begins to read the website but it just hangs. I inserted a print statement inside the while loop and it starts to print the website but just stops randomly before finishing.
    Last edited by rosco544; 09-17-2011 at 12:38 AM.

  17. #17
    rosco544 is offline Member
    Join Date
    Sep 2011
    Posts
    10
    Rep Power
    0

    Default Re: Open a URL and read it line by line (Works in Eclipse but not from Command Line)

    Finally solved it with this code found here. Not sure why this works but it works: Java HttpURLConnection (how to open and read a url with Java) | devdaily.com

    public String websiteToString(String desiredUrl) throws Exception
    {
    URL url = null;
    BufferedReader reader = null;
    StringBuilder stringBuilder;

    try
    {
    // create the HttpURLConnection
    url = new URL(desiredUrl);
    HttpURLConnection connection = (HttpURLConnection) url.openConnection();

    // just want to do an HTTP GET here
    connection.setRequestMethod("GET");

    // uncomment this if you want to write output to this url
    //connection.setDoOutput(true);

    // give it 15 seconds to respond
    connection.setReadTimeout(15*1000);
    connection.connect();

    // read the output from the server
    reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
    stringBuilder = new StringBuilder();

    String line = null;
    while ((line = reader.readLine()) != null)
    {
    stringBuilder.append(line + "\n");
    }

    String site = stringBuilder.toString();
    site = site.replaceAll("<html([^<]*)>", "<html>"); // replace all <html xml:lang="en" lang="en" xmlns:xml="xml"> with <html>
    // this is important because xpath can't parse an expression with <html> tags with junk in it

    return site;
    }
    catch (Exception e)
    {
    e.printStackTrace();
    throw e;
    }
    finally
    {
    // close the reader; this can throw an exception too, so
    // wrap it in another try/catch block.
    if (reader != null)
    {
    try
    {
    reader.close();
    }
    catch (IOException ioe)
    {
    ioe.printStackTrace();
    }
    }
    }
    }

Similar Threads

  1. Line Reader changeing to read 1 line
    By Javanooby in forum New To Java
    Replies: 15
    Last Post: 05-10-2011, 04:34 AM
  2. Replies: 1
    Last Post: 02-13-2011, 03:09 PM
  3. Replies: 2
    Last Post: 07-02-2010, 02:20 PM
  4. Replies: 3
    Last Post: 04-15-2010, 02:41 AM
  5. Replies: 4
    Last Post: 09-10-2009, 10:18 AM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •