reading URL using java through proxy server

[asheesh]

hi all
i made a java application through user can access webpage through proxy server, but this works for only some pages like (http://www.iita-conference.org,yahoo.com,iiitk.ac.in etc) it not works for like rediff.com, ieeeexplore,orkut etc. i also post the code plz tell me where is the problem? why it not works for all URLs?
if possible plz send code too.

package ieee;

import java.io.InputStream;
import java.net.Authenticator;
import java.net.HttpURLConnection;
import java.net.PasswordAuthentication;
import java.net.URL;
import java.util.Properties;
import java.util.logging.Level;
import java.util.logging.Logger;

/**
*
* @author sknavin
*/
public class Main {

/**
* @param args the command line arguments
*/
private static String loadPage(URL parURL) throws Exception {

Properties sysProps = System.getProperties();
sysProps.put("proxySet", "true");
sysProps.put("proxyHost", "172.31.1.3");
sysProps.put("proxyPort", "8080");

Authenticator authenticator = new Authenticator() {

public PasswordAuthentication getPasswordAuthentication() {
return (new PasswordAuthentication("userID","password".toCharA rray()));
}
};
Authenticator.setDefault(authenticator);

long lTime = System.currentTimeMillis();
StringBuffer sb = new StringBuffer("");
String http = parURL.toString();
// Create a connection.
HttpURLConnection urlConnection =
(HttpURLConnection) parURL.openConnection();
urlConnection.setRequestMethod("GET");
urlConnection.setDoOutput(true);
urlConnection.setDoInput(true);
urlConnection.setRequestProperty("USER-AGENT", "Mozilla/2.02Gold (WinNT; I)");
//
urlConnection.setRequestProperty("Content-type", "application/x-www-form-urlenCcoded");
urlConnection.setAllowUserInteraction(true);

urlConnection.connect();

InputStream in =
((HttpURLConnection) urlConnection).getInputStream();
int length = urlConnection.getContentLength();
for (int n = 0; n < length; n++) {
sb.append((char) in.read());
}

return sb.toString();
}

public static void main(String[] args) {
try {
// type your intended url here
URL url = new URL("http://www.iita-conference.org/");
String s = loadPage(url);
System.out.println("S=" + s);


} catch (Exception ex) {
Logger.getLogger(Main.class.getName()).log(Level.S EVERE, null, ex);
}


}
}

[DonCash]

Hello asheesh,

Welcome to the Java Forums.

It seems to me this is because the sites are blocking connectings from programs that are not web browsers. What you need to do is to trick these sites into thinking your program is a web browser.

Try this:

import java.net.*;
import java.io.*;

class httpconnect {

public static void main(String[] args) throws Exception {

URL url = new URL("http://www.anywebsite.com/");

URLConnection spoof = url.openConnection();

spoof.setRequestProperty(
"User-Agent", "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; H010818)" );

BufferedReader in = new BufferedReader(new InputStreamReader(spoof.getInputStream()));

String strLine = "";

while ((strLine = in.readLine()) != null) {

        System.out.println(strLine);
		
    }
in.close();
  }
}

Note this bit of code:

spoof.setRequestProperty( 
"User-Agent", "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; H010818)" );

This is what tricks the sites into thinking you are using a web browser.

I'm sure this will fix your problem!

[Sarinam]

Hello asheesh,

Welcome to the Java Forums.

It seems to me this is because the sites are blocking connectings from programs that are not web browsers. What you need to do is to trick these sites into thinking your program is a web browser.

Try this:

Code:

import java.net.*; import java.io.*; class httpconnect { public static void main(String[] args) throws Exception { URL url = new URL(""); URLConnection spoof = url.openConnection(); spoof.setRequestProperty( "User-Agent", "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; H010818)" ); BufferedReader in = new BufferedReader(new InputStreamReader(spoof.getInputStream())); String strLine = ""; while ((strLine = in.readLine()) != null) { System.out.println(strLine); } in.close(); } }

Note this bit of code:

Code:

spoof.setRequestProperty( "User-Agent", "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; H010818)" );

This is what tricks the sites into thinking you are using a web browser.

I'm sure this will fix your problem!

i have also face same problem.Some url open in code and some one is not.I cann't understand.Plz help me to solve the problem.If you need code then i will give you..

Thxs in Advance..

[Nicholas Jordan]

Sarinam: Stick with sites that will let you do a little bit. Trying to spoof can get you on hotlists too easily because of people who intentionally break protocols.

Most sites do not have the time or patience to separate beginners and so on and if they did do not wish to get involved in all the details. If you really want to do something along the lines of communicating with a server you should be sufficiently skilled to either hire a server or write one yourself.

The sample code has an http server already in the docs that ship with the JDK - and as well it implements some advanced cryptographic authentication that often is sold for hundreds of dollars.

If you want help writing a server, we can do that and will do so eagerly to keep you moving forward. There will be a forum here for that purpose.

[Sarinam]

If you want help writing a server, we can do that and will do so eagerly to keep you moving forward. There will be a forum here for that purpose.

Yes i want ot do...I have code which is working good for some url.If you need then i will give you code..

[Sarinam]

Sarinam: Stick with sites that will let you do a little bit. Trying to spoof can get you on hotlists too easily because of people who intentionally break protocols.

Most sites do not have the time or patience to separate beginners and so on and if they did do not wish to get involved in all the details. If you really want to do something along the lines of communicating with a server you should be sufficiently skilled to either hire a server or write one yourself.

The sample code has an http server already in the docs that ship with the JDK - and as well it implements some advanced cryptographic authentication that often is sold for hundreds of dollars.

If you want help writing a server, we can do that and will do so eagerly to keep you moving forward. There will be a forum here for that purpose.

So give me idia to do that or some code

[Nicholas Jordan]

class SimpleSample
{
      Server SERVER;
      Client   CLIENT; 
      SimpleSample()
      {
           SERVER = new Server();
           CLIENT = new Server();
           SERVER.start();
           CLIENT.start();
       }
       public static void main( String[]args)
       {
          SimpleSample simpleton = new SimpleSample():
       }
}
class Server extends Thread
{
   public void run(){}
}

class Client extends Thread
{
   public void run(){}
}

That might even compile or someting. There are no methods, no activity ~ we have a great challenge in determining what and how to do it. It is telling of the power of the Java Libraries ( classes provided with the development kit ) that we can do https: with the sample server. I have not figured out how to do that, right now I am writing a Servlet, it will run on a sample code server designed to host servlets.

It runs locally, thus achieving a development platform for testing.

[Norm]

power of the Java Libraries ( classes provided with the development kit ) that we can do https:

I wrote a proxy server long ago, but was unable to get it to work with https. I haven't looked at the new classes and methods available since 1.4?.
Could you lead me to what I need to make my server work with https?

Thanks,
Norm

[Mir]

I wrote a proxy server long ago, but was unable to get it to work with https. I haven't looked at the new classes and methods available since 1.4?.
Could you lead me to what I need to make my server work with https?

Thanks,
Norm

Hi Norm

I have a same problem..My proxy server work on only http.I havn't found any type of example or classes ..

If you fiend the solution then plz give me also..

Thxs Mir

[Nicholas Jordan]

(... snip ...)
Could you lead me to what I need to make my server work with https?

Thanks,
Norm

This is over my head so I am doing the best available skills:

public abstract class Server {

    ServerSocketChannel ssc;
    SSLContext sslContext = null;

    static private int PORT = 8000;
    static private int BACKLOG = 1024;
    static private boolean SECURE = false;

    Server(int port, int backlog,
	    boolean secure) throws Exception {

	if (secure) {
	    createSSLContext();
	}

	ssc = ServerSocketChannel.open();
	ssc.socket().setReuseAddress(true);
	ssc.socket().bind(new InetSocketAddress(port), backlog);
    }
    //.....

C:\.......\.....\Server.java ( sample sources in jdk install )

Probably it is JKD - 5.x comments state @version 1.2, 04/07/26 ( some date, huh? )

I will google it: Secure Internet Programming with Java 2, Standard Edition (J2SE) 1.4

{snippet from cited url}

ClientHello: The client sends the server information such as SSL protocol version, session id, and cipher suites information such cryptographic algorithms and key sizes supported.
.......

sounds like what one could use to proxy across https - a very appealing idea for any site with a need to maintain business standards.

{edit: probably would need two servers running in same process, may not be achievable }

[Norm]

Thanks for the info. Will have to look at all that.
PS. I need a text for Java 1.5 and 1.6, can you recommend any?
My current library is old: 1.2
Thanks,
Norm

[Niveditha]

Hi,
Try one more thing, i am not sure if it works..
In the internet options->connections->LAN Settings.
Check the "use a proxy Server" and Advanced button gets enabled. click on advanced button and enter "localhost" or the IP addr in "Do not use proxy Server for addressing begining with".

We do the same for our Client-Server Application.

[Niveditha]

Hello zlanhgn,
Please reply in English so that everyone can understand, not evryone here will be aware of your language.

[CaptainMorgan]

User has been banned. Unless the topic was Chinese aerospace board meetings. Please report posts that are inappropriate - it is a tremendous help to us.

[fishtoprecords]

In general, doing a proxy for HTTPS is hard. And sometimes, its impossible, as SSL is aimed at preventing "man in the middle attacks" and a proxy is exactly a man in the middle.

I debug my stuff completely using just HTTP and after it full works, turn on SSL. If I've really got the code working, its just works.

[Nicholas Jordan]

Thanks for the info. Will have to look at all that.
PS. I need a text for Java 1.5 and 1.6, can you recommend any?
My current library is old: 1.2
Thanks,
Norm

Use Cay S. Horstmann. Possibly anything by Manning if you have the time and budget. There is a four volume set by O'Reilly that is rather exhaustive in it's coverage but may not cover anything past 1.2 The general approach I would advise for someone of your skills is to get the two volume set by Sedgewick, such work exceeds the power of directly using the libraries.