Page 1 of 2 12 LastLast
Results 1 to 20 of 23
  1. #1
    Lil_Aziz1's Avatar
    Lil_Aziz1 is offline Senior Member
    Join Date
    Dec 2009
    Location
    United States
    Posts
    343
    Rep Power
    5

    Default URLConnection Efficiency

    I've written a program based off a game called Runescape. This program looks up a given item's price. Here it is:

    Java Code:
    import java.io.BufferedReader;
    import java.io.BufferedWriter;
    import java.io.InputStreamReader;
    import java.io.OutputStreamWriter;
    import java.net.URL;
    import java.net.URLConnection;
    import java.net.URLEncoder;
    import java.util.regex.Matcher;
    import java.util.regex.Pattern;
    
    
    public class RSGrandExchange {
    	
    	public static void main(String[] args) throws Exception {
    		long t_i = System.currentTimeMillis();
    		String item = "blue partyhat";
    		System.out.printf("Price of %s is %s. It took %dms to find it.%n", 
    				item, price(item), System.currentTimeMillis() - t_i);
    	}
    	
    	public static String price (String item) throws Exception {
    		
    		item = URLEncoder.encode(item, "UTF-8");
    		
    		URL RSSite = new URL("http://services.runescape.com/m=itemdb_rs/results.ws");
    		URLConnection RSConnection = RSSite.openConnection();
    		
    		//Ability to write using POST method.
    		RSConnection.setDoOutput(true);
    		BufferedWriter RSOut = new BufferedWriter(new OutputStreamWriter(
    				RSConnection.getOutputStream()));
    		RSOut.write("query=" + item + "&price=all&members=");
    		RSOut.close();
    		
    		BufferedReader RSIn = new BufferedReader(new InputStreamReader(
    				RSConnection.getInputStream()));
    		String text;
    		
    		Pattern regex = Pattern.compile("^<td><.+?alt=\"(.+?)\">");
    		Matcher matcher;
    		/*
    		 * Regex will match line 4.
    			...
    			1. </thead>
    			2. <tbody>
    			3. <tr class="row_b">
    			4. <td><img src="http://services.runescape.com/m=itemdb_rs/3092_obj_
    				sprite.gif?id=1042" alt="Blue partyhat"></td>
    			5. <td><a href="http://services.runescape.com/m=itemdb_rs/Blue_
    				partyhat/viewitem.ws?obj=1042"> Blue partyhat</a></td>
    			6. <td>560.0m</td>
    			7. <td><span class="stay">0</span></td>
    			...
    			
    		 */
    		
    		while ((text = RSIn.readLine()) != null) {
    			if (text.contains("did not return any results"))
    				return "Not found.";
    			
    			matcher = regex.matcher(text);
    			if (matcher.find()) {
    				RSIn.readLine();
    				//The price of the item is on line 6.
    				return RSIn.readLine().replaceAll("<[^>]*>", "");
    			}
    
    		}
    		RSIn.close();
    		return "Not found.";
    	}
    }
    The problem is, the program is very inefficient. It takes an avg of 1.2 second to get the result. Here is the output:
    Java Code:
    Price of blue partyhat is 560.0m. It took 1214ms to find it.
    I realize it's not java's fault, but I was wondering if I could make it more efficient. Maybe use the GET method (I have yet to look up how to do that on Java)? Or perhaps my code personifies regex abuse. I don't know. Any help is appreciated.

    Thanks in advance!
    Last edited by Lil_Aziz1; 08-18-2010 at 03:58 AM.
    "Experience is what you get when you don't get what you want" (Dan Stanford)
    "Rise and rise again until lambs become lions" (Robin Hood)

  2. #2
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    17,883
    Rep Power
    25

    Default

    It'd be a lot faster if you didn't use the internet. Use a localhost server.

    Have you tried timing all the sections of your code to see where the long execution times were?

  3. #3
    Lil_Aziz1's Avatar
    Lil_Aziz1 is offline Senior Member
    Join Date
    Dec 2009
    Location
    United States
    Posts
    343
    Rep Power
    5

    Default

    I know nothing about localhost servers. :( Maybe you could direct me to a useful article/link?

    I'm assuming the longest execution is the while loop.

    Java Code:
    import java.io.BufferedReader;
    import java.io.BufferedWriter;
    import java.io.InputStreamReader;
    import java.io.OutputStreamWriter;
    import java.net.URL;
    import java.net.URLConnection;
    import java.net.URLEncoder;
    import java.util.regex.Matcher;
    import java.util.regex.Pattern;
    
    
    public class Runescape {
    	
    	public static String[] price(String item) throws Exception {
    		
    		item = URLEncoder.encode(item, "UTF-8");
    		
    		long t_1 = System.currentTimeMillis(), t_2, t_3, t_4;
    		
    		URL RSSite = new URL("http://services.runescape.com/m=itemdb_rs/results.ws");
    		URLConnection RSConnection = RSSite.openConnection();
    		
    		//Ability to write using POST method.
    		RSConnection.setDoOutput(true);
    		BufferedWriter RSOut = new BufferedWriter(new OutputStreamWriter(
    				RSConnection.getOutputStream()));
    		RSOut.write("query=" + item + "&price=all&members=");
    		RSOut.close();
    		
    		t_2 = System.currentTimeMillis();
    		System.out.printf("The opening headers took %dms%n", t_2-t_1);
    		
    		BufferedReader RSIn = new BufferedReader(new InputStreamReader(
    				RSConnection.getInputStream()));
    		String text;
    		
    		t_3 = System.currentTimeMillis();
    		System.out.printf("Preparation for input took %dms%n", t_3-t_2);
    		
    		Pattern regex = Pattern.compile("^<td><.+?alt=\"(.+?)\">");
    		Matcher matcher;
    		/*
    		 * Regex will match line 4.
    			...
    			1. </thead>
    			2. <tbody>
    			3. <tr class="row_b">
    			4. <td><img src="http://services.runescape.com/m=itemdb_rs/3092_obj_
    				sprite.gif?id=1042" alt="Blue partyhat"></td>
    			5. <td><a href="http://services.runescape.com/m=itemdb_rs/Blue_
    				partyhat/viewitem.ws?obj=1042"> Blue partyhat</a></td>
    			6. <td>560.0m</td>
    			7. <td><span class="stay">0</span></td>
    			...
    			
    		 */
    		t_4 = System.currentTimeMillis();
    		System.out.printf("Initializing regex took %dms%n", t_4-t_3);
    		
    		while ((text = RSIn.readLine()) != null) {
    			if (text.contains("did not return any results"))
    				return null;
    			
    			matcher = regex.matcher(text);
    			if (matcher.find()) {
    				String[] result = new String[2];
    				result[0] = RSIn.readLine().replaceAll("<[^>]*>\\s?", "");
    				result[1] = RSIn.readLine().replaceAll("<[^>]*>", "");
    				System.out.printf("The while loop took %dms%n", System.currentTimeMillis()-t_4);
    				return result;
    			}
    		}
    		RSIn.close();
    		return null;
    	}
    	public static String combat(String rsn) {
    		
    		return null;
    	}
    }
    Output:
    Java Code:
    The opening headers took 120ms
    Preparation for input took 505ms
    Initializing regex took 1ms
    The while loop took 557ms
    Price of Red h'ween mask is 132.0m - It took 1186ms to find it.
    My hypothesis was correct, but I definitely didn't anticipate the following code to take half a second!
    Java Code:
    		BufferedReader RSIn = new BufferedReader(new InputStreamReader(
    				RSConnection.getInputStream()));
    		String text;
    "Experience is what you get when you don't get what you want" (Dan Stanford)
    "Rise and rise again until lambs become lions" (Robin Hood)

  4. #4
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    17,883
    Rep Power
    25

    Default

    That would probably be the server/internet using that time.

  5. #5
    Sno's Avatar
    Sno
    Sno is offline Senior Member
    Join Date
    Apr 2010
    Posts
    195
    Rep Power
    5

    Default

    I would agree with Norm,

    You can make things completely efficient only to find out your network is slow...

    Interesting that your doing such a complex thing without understanding localhost.

    All local host means is "This Computer" so, when you start up a server, (like tomcat, or jboss, glassfish or any other type of server) you can access your localhost by either 127.0.0.1 or http://localhost:8080/

    but must have a server running.

    Ideally local host is a place for you to do developments and view thing before their release.
    :rolleyes: ~ Sno ~ :rolleyes:
    '-~ B.S. Computer Science ~-'

  6. #6
    Lil_Aziz1's Avatar
    Lil_Aziz1 is offline Senior Member
    Join Date
    Dec 2009
    Location
    United States
    Posts
    343
    Rep Power
    5

    Default

    Unfortunately, I know nothing about MySQL, localhost, CSS (is that even relevant?), and PHP. It's not like I refuse to learn or I'm lazy, it's just that I don't know where to start and want to learn C++/more Java first, which might be a terrible idea. I don't know.

    When I type 127.0.0.1 or http://localhost:8080/ into my address bar, I get nothing, meaning I don't have a server running.
    Also, how will I be able to connect to a website, specifically RuneScape - MMORPG - The No.1 Free Online Multiplayer Game, without being connected to the internet? Or is this question too advance for my infinitesimal knowledge of everything but Java? Thanks a ton btw.
    "Experience is what you get when you don't get what you want" (Dan Stanford)
    "Rise and rise again until lambs become lions" (Robin Hood)

  7. #7
    Sno's Avatar
    Sno
    Sno is offline Senior Member
    Join Date
    Apr 2010
    Posts
    195
    Rep Power
    5

    Default

    For fun, Download tomcat 6.
    Apache Tomcat - Apache Tomcat 6 Downloads

    under Core download the zip.

    Extract the contents to C:/

    go to C:/tomcat 6/bin/ and click Startup.bat

    than try your local host.
    :rolleyes: ~ Sno ~ :rolleyes:
    '-~ B.S. Computer Science ~-'

  8. #8
    Lil_Aziz1's Avatar
    Lil_Aziz1 is offline Senior Member
    Join Date
    Dec 2009
    Location
    United States
    Posts
    343
    Rep Power
    5

    Default

    Ooo http://localhost:8080/ works. So even if I'm disconnected, I can connect to this page? Neat stuff. Now how can I get a website such as RuneScape - MMORPG - The No.1 Free Online Multiplayer Game to work on localhost?
    "Experience is what you get when you don't get what you want" (Dan Stanford)
    "Rise and rise again until lambs become lions" (Robin Hood)

  9. #9
    Sno's Avatar
    Sno
    Sno is offline Senior Member
    Join Date
    Apr 2010
    Posts
    195
    Rep Power
    5

    Default

    Yup, if your disconnected you can still develop on that page.

    You can't place runescape on the localhost but you can make a dummy data list of the items.

    So in this case, I would use a database or a static area to keep the data.

    but try this

    download and install Install PHP 5 Apache MySQL on Windows : WampServer

    now you can access http://localhost:8080/phpmyadmin

    and you will have MySql installed, and phpmyadmin for a database access.

    which is a database
    default login is root, and it has no password.

    Here you can create a database, tables and add data.
    :rolleyes: ~ Sno ~ :rolleyes:
    '-~ B.S. Computer Science ~-'

  10. #10
    Lil_Aziz1's Avatar
    Lil_Aziz1 is offline Senior Member
    Join Date
    Dec 2009
    Location
    United States
    Posts
    343
    Rep Power
    5

    Default

    The prices change everyday though.
    "Experience is what you get when you don't get what you want" (Dan Stanford)
    "Rise and rise again until lambs become lions" (Robin Hood)

  11. #11
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    17,883
    Rep Power
    25

    Default

    The localhost server is used for testing. It could provide a base, minimum execution time for your code.

  12. #12
    Lil_Aziz1's Avatar
    Lil_Aziz1 is offline Senior Member
    Join Date
    Dec 2009
    Location
    United States
    Posts
    343
    Rep Power
    5

    Default

    Quote Originally Posted by Sno View Post
    Yup, if your disconnected you can still develop on that page.

    You can't place runescape on the localhost but you can make a dummy data list of the items.

    So in this case, I would use a database or a static area to keep the data.

    but try this

    download and install Install PHP 5 Apache MySQL on Windows : WampServer

    now you can access http://localhost:8080/phpmyadmin

    and you will have MySql installed, and phpmyadmin for a database access.

    which is a database
    default login is root, and it has no password.

    Here you can create a database, tables and add data.
    This is all so vague for me. Do you know any source(s) that could teach me all this stuff (for free lol). I wouldn't mind Google as one of your sources but what should I search?

    Btw, I can't say thank you enough times for all your help.
    "Experience is what you get when you don't get what you want" (Dan Stanford)
    "Rise and rise again until lambs become lions" (Robin Hood)

  13. #13
    Lil_Aziz1's Avatar
    Lil_Aziz1 is offline Senior Member
    Join Date
    Dec 2009
    Location
    United States
    Posts
    343
    Rep Power
    5

    Default

    So the question is, how can I tell the URLConnection that the file name is "/search" since the action name is "/search" ? Even if this isn't the problem, I would still like to know how to do so. Here is what I have so far:

    Java Code:
    import java.io.BufferedReader;
    import java.io.BufferedWriter;
    import java.io.InputStreamReader;
    import java.io.OutputStreamWriter;
    import java.net.URL;
    import java.net.URLConnection;
    
    
    public class Google {
    	
    	public static void main (String[] args) throws Exception {
    		URL Google = new URL("http://www.google.com/");
    		//or
    		//URL Google = new URL("http://www.google.com/search");
    
    		URLConnection GConnection = Google.openConnection();
    
    		GConnection.setDoOutput(true);
    		BufferedWriter GoogleOut = new BufferedWriter(new OutputStreamWriter(
    				GConnection.getOutputStream()));
    		GoogleOut.write("hl=en&source=hp&ie=ISO-8859-1&q=Java+Tutorials&btnG=Google+Search");
    		GoogleOut.close();
    		
    		BufferedReader GoogleIn = new BufferedReader(new InputStreamReader(
    				GConnection.getInputStream())); //line 22
    		String text;
    		
    		while ((text = GoogleIn.readLine()) != null)
    			System.out.println(text);
    	}
    
    }
    This is what the link is suppose to look like:
    http://www.google.com/search?hl=en&s...=Google+Search


    Output:
    Java Code:
    Exception in thread "main" java.io.IOException: Server returned HTTP response code: 405 for URL: http://www.google.com
    	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
    	at Google.main(Google.java:22)
    Last edited by Lil_Aziz1; 08-19-2010 at 06:04 AM.
    "Experience is what you get when you don't get what you want" (Dan Stanford)
    "Rise and rise again until lambs become lions" (Robin Hood)

  14. #14
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    17,883
    Rep Power
    25

    Default

    Do you have any debug tools to see what a browser sends to the site?
    Then another to see what your code is sending to the site
    To see what the difference is.

    I use a proxy server that displays the traffic going thru it.

    Why are you using a POST? Your posted URL looks like GET
    To use a GET, put the query string on the URL.

  15. #15
    Lil_Aziz1's Avatar
    Lil_Aziz1 is offline Senior Member
    Join Date
    Dec 2009
    Location
    United States
    Posts
    343
    Rep Power
    5

    Default

    OOh. I use to use Fiddler for that. Do you know how I could send traffic from my Java program to Fiddler?

    I would love to use GET if I knew how to tell the URLConnection how.
    "Experience is what you get when you don't get what you want" (Dan Stanford)
    "Rise and rise again until lambs become lions" (Robin Hood)

  16. #16
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    17,883
    Rep Power
    25

    Default

    GConnection.setDoOutput(true);
    I think this could be what sets it to POST. Try commenting it out and seeing what happens.

  17. #17
    Lil_Aziz1's Avatar
    Lil_Aziz1 is offline Senior Member
    Join Date
    Dec 2009
    Location
    United States
    Posts
    343
    Rep Power
    5

    Default

    I'm not sure if that's it. I was looking over the HttpURLConnection class and there is a method called setRequestMethod(String method)

    Java Code:
    setRequestMethod
    
    public void setRequestMethod(String method)
                          throws ProtocolException
    
        Set the method for the URL request, one of:
    
            * GET
            * POST
            * HEAD
            * OPTIONS
            * PUT
            * DELETE
            * TRACE 
    
        are legal, subject to protocol restrictions. The default method is GET.
    
        Parameters:
            method - the HTTP method 
        Throws:
            ProtocolException - if the method cannot be reset or if the requested method isn't valid for HTTP.
        See Also:
            getRequestMethod()
    "Experience is what you get when you don't get what you want" (Dan Stanford)
    "Rise and rise again until lambs become lions" (Robin Hood)

  18. #18
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    17,883
    Rep Power
    25

    Default

    With the GET method all of the args go on the URL.
    With POST, you need to write the args on a separate line that follows a blank line following the request header.

  19. #19
    Lil_Aziz1's Avatar
    Lil_Aziz1 is offline Senior Member
    Join Date
    Dec 2009
    Location
    United States
    Posts
    343
    Rep Power
    5

    Default

    I did this and got a HTTP Error 403: HTTP Error 403 Forbidden Explained
    Java Code:
    public class Google {
    	
    	public static void main (String[] args) throws Exception {
    		URL Google = new URL("http://www.google.com/search?hl=en&source=hp&ie=" +
    				"ISO-8859-1&q=Java+Tutorials&btnG=Google+Search");
    		//or
    		//URL Google = new URL("http://www.google.com/search");
    		URLConnection GConnection = Google.openConnection();
    
    		BufferedReader GoogleIn = new BufferedReader(new InputStreamReader(
    				GConnection.getInputStream()));
    		String text;
    		
    		while ((text = GoogleIn.readLine()) != null)
    			System.out.println(text);
    	}
    }
    EDIT: Just saw your thread. So yea there is no point in even writing anything.
    "Experience is what you get when you don't get what you want" (Dan Stanford)
    "Rise and rise again until lambs become lions" (Robin Hood)

  20. #20
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    17,883
    Rep Power
    25

    Default

    A previous post on some forum found that some servers don't like just anybody getting their stuff. So you need to pretend to be a good guy browser by setting the User-Agent: field in the request header to mimic a browser:

    GConnection.setRequestProperty( "User-Agent", "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; H010818)" );

Page 1 of 2 12 LastLast

Similar Threads

  1. An Efficiency Question
    By Revenna in forum Java 2D
    Replies: 0
    Last Post: 06-25-2010, 08:22 AM
  2. URLConnection not timing out as expected
    By StephenS in forum Networking
    Replies: 2
    Last Post: 04-02-2010, 04:10 AM
  3. How to set request parameters to an URLConnection
    By somesh A in forum Networking
    Replies: 0
    Last Post: 04-10-2009, 09:08 AM
  4. method efficiency
    By TheWave in forum Advanced Java
    Replies: 0
    Last Post: 02-13-2008, 05:11 AM
  5. Opening URLConnection
    By Java Tip in forum Java Tip
    Replies: 0
    Last Post: 11-24-2007, 08:37 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •