Results 1 to 17 of 17
  1. #1
    asheesh is offline Member
    Join Date
    Apr 2008
    Posts
    1
    Rep Power
    0

    Default reading URL using java through proxy server

    hi all:)
    i made a java application through user can access webpage through proxy server, but this works for only some pages like (http://www.iita-conference.org,yahoo.com,iiitk.ac.in etc) it not works for like rediff.com, ieeeexplore,orkut etc. i also post the code plz tell me where is the problem? why it not works for all URLs?
    if possible plz send code too.

    package ieee;

    import java.io.InputStream;
    import java.net.Authenticator;
    import java.net.HttpURLConnection;
    import java.net.PasswordAuthentication;
    import java.net.URL;
    import java.util.Properties;
    import java.util.logging.Level;
    import java.util.logging.Logger;

    /**
    *
    * @author sknavin
    */
    public class Main {

    /**
    * @param args the command line arguments
    */
    private static String loadPage(URL parURL) throws Exception {

    Properties sysProps = System.getProperties();
    sysProps.put("proxySet", "true");
    sysProps.put("proxyHost", "172.31.1.3");
    sysProps.put("proxyPort", "8080");

    Authenticator authenticator = new Authenticator() {

    public PasswordAuthentication getPasswordAuthentication() {
    return (new PasswordAuthentication("userID","password".toCharA rray()));
    }
    };
    Authenticator.setDefault(authenticator);

    long lTime = System.currentTimeMillis();
    StringBuffer sb = new StringBuffer("");
    String http = parURL.toString();
    // Create a connection.
    HttpURLConnection urlConnection =
    (HttpURLConnection) parURL.openConnection();
    urlConnection.setRequestMethod("GET");
    urlConnection.setDoOutput(true);
    urlConnection.setDoInput(true);
    urlConnection.setRequestProperty("USER-AGENT", "Mozilla/2.02Gold (WinNT; I)");
    //
    urlConnection.setRequestProperty("Content-type", "application/x-www-form-urlenCcoded");
    urlConnection.setAllowUserInteraction(true);

    urlConnection.connect();

    InputStream in =
    ((HttpURLConnection) urlConnection).getInputStream();
    int length = urlConnection.getContentLength();
    for (int n = 0; n < length; n++) {
    sb.append((char) in.read());
    }

    return sb.toString();
    }

    public static void main(String[] args) {
    try {
    // type your intended url here
    URL url = new URL("http://www.iita-conference.org/");
    String s = loadPage(url);
    System.out.println("S=" + s);


    } catch (Exception ex) {
    Logger.getLogger(Main.class.getName()).log(Level.S EVERE, null, ex);
    }


    }
    }

  2. #2
    DonCash's Avatar
    DonCash is offline Moderator
    Join Date
    Aug 2007
    Location
    London, UK
    Posts
    240
    Rep Power
    9

    Default

    Hello asheesh,

    Welcome to the Java Forums.

    It seems to me this is because the sites are blocking connectings from programs that are not web browsers. What you need to do is to trick these sites into thinking your program is a web browser.

    Try this:

    Java Code:
    import java.net.*;
    import java.io.*;
    
    class httpconnect {
    
    public static void main(String[] args) throws Exception {
    
    URL url = new URL("http://www.anywebsite.com/");
    
    URLConnection spoof = url.openConnection();
    
    spoof.setRequestProperty(
    "User-Agent", "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; H010818)" );
    
    BufferedReader in = new BufferedReader(new InputStreamReader(spoof.getInputStream()));
    
    String strLine = "";
    
    while ((strLine = in.readLine()) != null) {
    
            System.out.println(strLine);
    		
        }
    in.close();
      }
    }
    Note this bit of code:

    Java Code:
    spoof.setRequestProperty( 
    "User-Agent", "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; H010818)" );
    This is what tricks the sites into thinking you are using a web browser.

    I'm sure this will fix your problem!
    Did this post help you? Please me! :cool:

  3. #3
    Sarinam is offline Senior Member
    Join Date
    Jun 2008
    Posts
    121
    Rep Power
    0

    Default

    Quote Originally Posted by DonCash View Post
    Hello asheesh,

    Welcome to the Java Forums.

    It seems to me this is because the sites are blocking connectings from programs that are not web browsers. What you need to do is to trick these sites into thinking your program is a web browser.

    Try this:

    Java Code:
    import java.net.*;
    import java.io.*;
    
    class httpconnect {
    
    public static void main(String[] args) throws Exception {
    
    URL url = new URL("");
    
    URLConnection spoof = url.openConnection();
    
    spoof.setRequestProperty(
    "User-Agent", "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; H010818)" );
    
    BufferedReader in = new BufferedReader(new InputStreamReader(spoof.getInputStream()));
    
    String strLine = "";
    
    while ((strLine = in.readLine()) != null) {
    
            System.out.println(strLine);
    		
        }
    in.close();
      }
    }
    Note this bit of code:

    Java Code:
    spoof.setRequestProperty( 
    "User-Agent", "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; H010818)" );
    This is what tricks the sites into thinking you are using a web browser.

    I'm sure this will fix your problem!
    i have also face same problem.Some url open in code and some one is not.I cann't understand.Plz help me to solve the problem.If you need code then i will give you..

    Thxs in Advance..

  4. #4
    Nicholas Jordan's Avatar
    Nicholas Jordan is offline Senior Member
    Join Date
    Jun 2008
    Location
    Southwest
    Posts
    1,018
    Rep Power
    8

    Thumbs up face same problem

    Sarinam: Stick with sites that will let you do a little bit. Trying to spoof can get you on hotlists too easily because of people who intentionally break protocols.

    Most sites do not have the time or patience to separate beginners and so on and if they did do not wish to get involved in all the details. If you really want to do something along the lines of communicating with a server you should be sufficiently skilled to either hire a server or write one yourself.

    The sample code has an http server already in the docs that ship with the JDK - and as well it implements some advanced cryptographic authentication that often is sold for hundreds of dollars.

    If you want help writing a server, we can do that and will do so eagerly to keep you moving forward. There will be a forum here for that purpose.

  5. #5
    Sarinam is offline Senior Member
    Join Date
    Jun 2008
    Posts
    121
    Rep Power
    0

    Default

    Quote Originally Posted by Nicholas Jordan View Post
    If you want help writing a server, we can do that and will do so eagerly to keep you moving forward. There will be a forum here for that purpose.

    Yes i want ot do...I have code which is working good for some url.If you need then i will give you code..

  6. #6
    Sarinam is offline Senior Member
    Join Date
    Jun 2008
    Posts
    121
    Rep Power
    0

    Default

    Quote Originally Posted by Nicholas Jordan View Post
    Sarinam: Stick with sites that will let you do a little bit. Trying to spoof can get you on hotlists too easily because of people who intentionally break protocols.

    Most sites do not have the time or patience to separate beginners and so on and if they did do not wish to get involved in all the details. If you really want to do something along the lines of communicating with a server you should be sufficiently skilled to either hire a server or write one yourself.

    The sample code has an http server already in the docs that ship with the JDK - and as well it implements some advanced cryptographic authentication that often is sold for hundreds of dollars.

    If you want help writing a server, we can do that and will do so eagerly to keep you moving forward. There will be a forum here for that purpose.

    So give me idia to do that or some code

  7. #7
    Nicholas Jordan's Avatar
    Nicholas Jordan is offline Senior Member
    Join Date
    Jun 2008
    Location
    Southwest
    Posts
    1,018
    Rep Power
    8

    Default download sample server

    Java Code:
    class SimpleSample
    {
          Server SERVER;
          Client   CLIENT; 
          SimpleSample()
          {
               SERVER = new Server();
               CLIENT = new Server();
               SERVER.start();
               CLIENT.start();
           }
           public static void main( String[]args)
           {
              SimpleSample simpleton = new SimpleSample():
           }
    }
    class Server extends Thread
    {
       public void run(){}
    }
    
    class Client extends Thread
    {
       public void run(){}
    }
    That might even compile or someting. There are no methods, no activity ~ we have a great challenge in determining what and how to do it. It is telling of the power of the Java Libraries ( classes provided with the development kit ) that we can do https: with the sample server. I have not figured out how to do that, right now I am writing a Servlet, it will run on a sample code server designed to host servlets.

    It runs locally, thus achieving a development platform for testing.
    Introduction to Programming Using Java.
    Cybercartography: A new theoretical construct proposed by D.R. Fraser Taylor

  8. #8
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,308
    Rep Power
    25

    Default

    power of the Java Libraries ( classes provided with the development kit ) that we can do https:
    I wrote a proxy server long ago, but was unable to get it to work with https. I haven't looked at the new classes and methods available since 1.4?.
    Could you lead me to what I need to make my server work with https?

    Thanks,
    Norm

  9. #9
    Mir
    Mir is offline Senior Member
    Join Date
    Mar 2008
    Posts
    447
    Rep Power
    7

    Default

    Quote Originally Posted by Norm View Post
    I wrote a proxy server long ago, but was unable to get it to work with https. I haven't looked at the new classes and methods available since 1.4?.
    Could you lead me to what I need to make my server work with https?

    Thanks,
    Norm
    Hi Norm

    I have a same problem..My proxy server work on only http.I havn't found any type of example or classes ..

    If you fiend the solution then plz give me also..

    Thxs Mir

  10. #10
    Nicholas Jordan's Avatar
    Nicholas Jordan is offline Senior Member
    Join Date
    Jun 2008
    Location
    Southwest
    Posts
    1,018
    Rep Power
    8

    Default Proxy server is in sample code.

    Quote Originally Posted by Norm View Post
    (... snip ...)
    Could you lead me to what I need to make my server work with https?

    Thanks,
    Norm
    This is over my head so I am doing the best available skills:
    Java Code:
    public abstract class Server {
    
        ServerSocketChannel ssc;
        SSLContext sslContext = null;
    
        static private int PORT = 8000;
        static private int BACKLOG = 1024;
        static private boolean SECURE = false;
    
        Server(int port, int backlog,
    	    boolean secure) throws Exception {
    
    	if (secure) {
    	    createSSLContext();
    	}
    
    	ssc = ServerSocketChannel.open();
    	ssc.socket().setReuseAddress(true);
    	ssc.socket().bind(new InetSocketAddress(port), backlog);
        }
        //.....
    C:\.......\.....\Server.java ( sample sources in jdk install )

    Probably it is JKD - 5.x comments state @version 1.2, 04/07/26 ( some date, huh? )

    I will google it: Secure Internet Programming with Java 2, Standard Edition (J2SE) 1.4

    {snippet from cited url}

    ClientHello: The client sends the server information such as SSL protocol version, session id, and cipher suites information such cryptographic algorithms and key sizes supported.
    .......

    sounds like what one could use to proxy across https - a very appealing idea for any site with a need to maintain business standards.

    {edit: probably would need two servers running in same process, may not be achievable }
    Introduction to Programming Using Java.
    Cybercartography: A new theoretical construct proposed by D.R. Fraser Taylor

  11. #11
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,308
    Rep Power
    25

    Default

    Thanks for the info. Will have to look at all that.
    PS. I need a text for Java 1.5 and 1.6, can you recommend any?
    My current library is old: 1.2
    Thanks,
    Norm

  12. #12
    Niveditha's Avatar
    Niveditha is offline Senior Member
    Join Date
    May 2008
    Posts
    307
    Rep Power
    7

    Default

    Hi,
    Try one more thing, i am not sure if it works..
    In the internet options->connections->LAN Settings.
    Check the "use a proxy Server" and Advanced button gets enabled. click on advanced button and enter "localhost" or the IP addr in "Do not use proxy Server for addressing begining with".

    We do the same for our Client-Server Application.
    To finish sooner, take your own time....
    Nivedithaaaa

  13. #13
    Niveditha's Avatar
    Niveditha is offline Senior Member
    Join Date
    May 2008
    Posts
    307
    Rep Power
    7

    Default

    Hello zlanhgn,
    Please reply in English so that everyone can understand, not evryone here will be aware of your language.
    To finish sooner, take your own time....
    Nivedithaaaa

  14. #14
    CaptainMorgan's Avatar
    CaptainMorgan is offline Moderator
    Join Date
    Dec 2007
    Location
    NewEngland, US
    Posts
    835
    Rep Power
    8

    Default

    User has been banned. Unless the topic was Chinese aerospace board meetings. Please report posts that are inappropriate - it is a tremendous help to us. :)
    Last edited by CaptainMorgan; 08-20-2008 at 10:34 AM.
    Vote for the new slogan to our beloved Java Forums! (closes on September 4, 2008)
    Want to voice your opinion on your IDE/Editor of choice? Vote now!
    Got a little Capt'n in you? (drink responsibly)

  15. #15
    fishtoprecords's Avatar
    fishtoprecords is offline Senior Member
    Join Date
    Jun 2008
    Posts
    571
    Rep Power
    7

    Default

    In general, doing a proxy for HTTPS is hard. And sometimes, its impossible, as SSL is aimed at preventing "man in the middle attacks" and a proxy is exactly a man in the middle.

    I debug my stuff completely using just HTTP and after it full works, turn on SSL. If I've really got the code working, its just works.

  16. #16
    Nicholas Jordan's Avatar
    Nicholas Jordan is offline Senior Member
    Join Date
    Jun 2008
    Location
    Southwest
    Posts
    1,018
    Rep Power
    8

    Post text recomendation for 1.5+

    Quote Originally Posted by Norm View Post
    Thanks for the info. Will have to look at all that.
    PS. I need a text for Java 1.5 and 1.6, can you recommend any?
    My current library is old: 1.2
    Thanks,
    Norm
    Use Cay S. Horstmann. Possibly anything by Manning if you have the time and budget. There is a four volume set by O'Reilly that is rather exhaustive in it's coverage but may not cover anything past 1.2 The general approach I would advise for someone of your skills is to get the two volume set by Sedgewick, such work exceeds the power of directly using the libraries.
    Introduction to Programming Using Java.
    Cybercartography: A new theoretical construct proposed by D.R. Fraser Taylor

  17. #17
    voscom is offline Member
    Join Date
    Apr 2010
    Posts
    2
    Rep Power
    0

    Default

    i have also face same problem.Some url open in code and some one is not.I cann't understand.Plz help me to solve the problem.If you need code then i will give you..

    Thank you very much.

Similar Threads

  1. Does OS intervene when reading Java text files
    By Tina G in forum Advanced Java
    Replies: 1
    Last Post: 04-07-2008, 02:29 PM
  2. reading dir in java applets
    By willemjav in forum Java Applets
    Replies: 3
    Last Post: 02-07-2008, 12:36 AM
  3. reading textfile from java problem
    By saytri in forum New To Java
    Replies: 1
    Last Post: 01-17-2008, 02:13 AM
  4. Question abt.reading xml file using java
    By gvi in forum Advanced Java
    Replies: 6
    Last Post: 11-08-2007, 05:48 PM
  5. Help with IRC server in java
    By mathias in forum Networking
    Replies: 1
    Last Post: 08-07-2007, 06:51 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •