Results 1 to 13 of 13
  1. #1
    WhiteBay is offline Member
    Join Date
    Apr 2017
    Posts
    6
    Rep Power
    0

    Default Please Help - Java Web Scraper

    Im trying to edit a web scraper that uses the following:

    Please Help - Java Web Scraper-lib.jpg

    The problem is the website address that was used for scrapping has changed. I need to update the website used to scrape data.
    Any help is appreciated. I can pay if someone can walk me through what to do.

  2. #2
    SurfMan's Avatar
    SurfMan is offline Godlike
    Join Date
    Nov 2012
    Location
    The Netherlands
    Posts
    1,989
    Rep Power
    8

    Default Re: Please Help - Java Web Scraper

    Welcome to the risky world of scraping. One change on the website can mess it all up. Are you saying you have an application that now fails due to a changed website and you need that application adapted? Do you have the source?

    The files you are showing are just the libraries used, of which JSoup is the library for the actual scraping.
    "It's not fixed until you stop calling the problem weird and you understand what was wrong." - gimbal2 2013

  3. #3
    WhiteBay is offline Member
    Join Date
    Apr 2017
    Posts
    6
    Rep Power
    0

    Default Re: Please Help - Java Web Scraper

    The website went down, and the data is now at a different website. Its very similar to the one that went down, it is an easy change, but i'm lost, as i'm brand new to programming.

  4. #4
    SurfMan's Avatar
    SurfMan is offline Godlike
    Join Date
    Nov 2012
    Location
    The Netherlands
    Posts
    1,989
    Rep Power
    8

    Default Re: Please Help - Java Web Scraper

    So do you have the source?
    "It's not fixed until you stop calling the problem weird and you understand what was wrong." - gimbal2 2013

  5. #5
    WhiteBay is offline Member
    Join Date
    Apr 2017
    Posts
    6
    Rep Power
    0

    Default Re: Please Help - Java Web Scraper

    I just have the program that scrapes from the website that went down. Not sure if i have the source. I can send you what I have over skype or teamviewer

  6. #6
    SurfMan's Avatar
    SurfMan is offline Godlike
    Join Date
    Nov 2012
    Location
    The Netherlands
    Posts
    1,989
    Rep Power
    8

    Default Re: Please Help - Java Web Scraper

    Upload it to some file share thingie (WeTransfer or similar) and post the link. If you have the source, and its license allows you to modify and redistribute it, we can probably make modifications.
    "It's not fixed until you stop calling the problem weird and you understand what was wrong." - gimbal2 2013

  7. #7
    WhiteBay is offline Member
    Join Date
    Apr 2017
    Posts
    6
    Rep Power
    0

    Default Re: Please Help - Java Web Scraper

    Sorry, doesn't look like it

  8. #8
    SurfMan's Avatar
    SurfMan is offline Godlike
    Join Date
    Nov 2012
    Location
    The Netherlands
    Posts
    1,989
    Rep Power
    8

    Default Re: Please Help - Java Web Scraper

    In that case, if you're interested in programming, you now have a challenge :) JSoup isn't that hard to understand if you know a bit of Java and if you also know a bit of CSS-selectors, it's not hard to do at all.

    Having said that, website scraping is not allowed in the majority of websites according to their Terms and Conditions. You might want to check that on the target website itself. Consider alternative options, for example, if it's a supplier, you can ask if they have a pricelist available for download.
    "It's not fixed until you stop calling the problem weird and you understand what was wrong." - gimbal2 2013

  9. #9
    SurfMan's Avatar
    SurfMan is offline Godlike
    Join Date
    Nov 2012
    Location
    The Netherlands
    Posts
    1,989
    Rep Power
    8

    Default Re: Please Help - Java Web Scraper

    I've looked, unfortunately the file you sent contains no sources. You can delete the file from WeTransfer now.
    "It's not fixed until you stop calling the problem weird and you understand what was wrong." - gimbal2 2013

  10. #10
    SurfMan's Avatar
    SurfMan is offline Godlike
    Join Date
    Nov 2012
    Location
    The Netherlands
    Posts
    1,989
    Rep Power
    8

    Default Re: Please Help - Java Web Scraper

    Here is a Github project that might be of help: https://github.com/fiacre/FMCSA-scraper
    "It's not fixed until you stop calling the problem weird and you understand what was wrong." - gimbal2 2013

  11. #11
    WhiteBay is offline Member
    Join Date
    Apr 2017
    Posts
    6
    Rep Power
    0

    Default Re: Please Help - Java Web Scraper

    I found this searching how to edit java without source code... Is something like this do able?

    "The java decompiler you found (http://java.decompiler.free.fr/) will take the compiled jar file you have and create java source files. You will then have to edit the source files to make the changes you want and recompile."

  12. #12
    SurfMan's Avatar
    SurfMan is offline Godlike
    Join Date
    Nov 2012
    Location
    The Netherlands
    Posts
    1,989
    Rep Power
    8

    Default Re: Please Help - Java Web Scraper

    I know about decompiling. However, decompiling equals reverse engineering, which is illegal according most software licenses. That's why I did not (and won't) suggest to do that. Unless you have the license that shows that it's ok, or have written permission from the author, you should assume it's illegal.

    I know it sounds way over the top for such a relative little tool, but it am not going to be held responsible for a massive copyright claim. So I am not joining the reverse engineering party.
    "It's not fixed until you stop calling the problem weird and you understand what was wrong." - gimbal2 2013

  13. #13
    WhiteBay is offline Member
    Join Date
    Apr 2017
    Posts
    6
    Rep Power
    0

    Default Re: Please Help - Java Web Scraper

    I understand. Good to know its possible though. Thank you for the info.

Similar Threads

  1. screen scraper for java applet website
    By sheriek in forum Java Applets
    Replies: 3
    Last Post: 09-25-2008, 09:48 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •