Results 1 to 2 of 2
  1. #1
    raj.mscking@gmail.com's Avatar
    raj.mscking@gmail.com is offline Senior Member
    Join Date
    Jan 2012
    Location
    TamilNadu
    Posts
    330
    Blog Entries
    1
    Rep Power
    7

    Default Issue on Web scraping and REST API calls using Jsoup

    Dear All!

    I am trying to scraping a web-site using Jsoup.

    I have Login successfully and while move to the next page (Result page) the data are loading by Restful web-service as JSON. need to pass the cookies and secret keeys too.

    Is it possible to read web-service using Jsoup?


    Code to login

    (this is working fine)

    Java Code:
    	public static Connection.Response login(Connection.Response response) throws IOException
    	{
            response = Jsoup.connect("https://www.xxxyyy.com/login")
                    .cookies(response.cookies())
                    .data("loginmsrfParam", "hdwuhdwugu258==")
                    .data("user_name", "*************")
                    .data("user_pasword", "********")
                    .method(Connection.Method.POST)
                    .followRedirects(true)
                    .execute(); 
    		return response;
    		
    	}

    Code to call result page (Web-Service)

    (problematic area)

    Java Code:
    	public static Connection.Response get_Search_Result(Connection.Response response) throws IOException
    	{
    		
       	 String url_Search_ = "https://www.xxxyyy.com/search/result?count=10&guides=List()&keywords=%22android%20developer%22%20AND%20%22Us%22&origin=OTHER&q=guided&start=0";
            response = Jsoup
                    .connect(url_Search_)
                    .header("Accept", "application/nnd.xxxyyy.normalized+json")
                    .ignoreContentType(true)
                    .header("Accept-Encoding", "gzip, deflate, br")
                    .header("Accept-Language", "en-US,en;q=0.5")
                    .header("Connection", "keep-alive")
                    .cookies(response.cookies())
                    .header("Msrf-Token", mmsrrgToken)
                    .header("Host", "www.xxxyyy.com")
                    .header("Referer", "https://www.xxxyyy.com/")
                    .header("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0")
                    .header("X-LI-Lang", "en_US")
                    .header("X-LI-Track", "{'clientVersion':'1.0.*','osName':'web','timezoneOffset':5.5,'deviceFormFactor':'DESKTOP'}")
                    .header("X-Requested-With", "XMLHttpRequest")
                    .header("X-RestLi-Protocol-Version", "2.0.0")
                    .header("mile-instance", mile_ist.text())
                    .method(Connection.Method.GET)
                    .followRedirects(true)
                    .execute();
    		return response;
    		
    	}


    Error Message while reading restful Web-Service

    Java Code:
    org.jsoup.HttpStatusException: HTTP error fetching URL. Status=403, URL=https://www.xxxyyy.com/search/result?count=10&guides=List()&keywords=%22android%20developer%22%20AND%20%22Us%22&origin=OTHER&q=guided&start=0
    	at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:679)
    	at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:628)
    	at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:260)
    	at tp.tri.autolgn.SimpleScraping.SimpleScraping.get_Search_Result(SimpleScraping.java:156)
    	at tp.tri.autolgn.SimpleScraping.SimpleScraping.main(SimpleScraping.java:21)

    I need your guidance on this, Guide me please..,

    Thanks..,
    Last edited by raj.mscking@gmail.com; 06-22-2017 at 05:54 PM.
    - Raj

  2. #2
    raj.mscking@gmail.com's Avatar
    raj.mscking@gmail.com is offline Senior Member
    Join Date
    Jan 2012
    Location
    TamilNadu
    Posts
    330
    Blog Entries
    1
    Rep Power
    7

    Default Re: Issue on Web scraping and REST API calls using Jsoup

    HI!

    Anyone can guide me please, i am suffering for long time..,
    - Raj

Similar Threads

  1. scraping using Jsoup
    By gvs048 in forum New To Java
    Replies: 12
    Last Post: 06-07-2013, 09:00 AM
  2. Html scraping Site Loads Wrong Jsoup Java
    By kevinn205 in forum Advanced Java
    Replies: 1
    Last Post: 08-27-2012, 09:19 PM
  3. Scraping Data From a Website
    By atomant in forum New To Java
    Replies: 2
    Last Post: 05-16-2012, 08:25 PM
  4. Web Scraping, web mining
    By mitukhan in forum Java Software
    Replies: 0
    Last Post: 09-19-2011, 12:44 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •