Page 1 of 3 123 LastLast
Results 1 to 20 of 41
Like Tree4Likes

Thread: Download All Files from a Directory

  1. #1
    cjburkey01 is offline Member
    Join Date
    Dec 2013
    Posts
    46
    Blog Entries
    2
    Rep Power
    0

    Question Download All Files from a Directory

    Hello, I was wondering, "How do I download all the files in a directory on the internet(hence[download])" And was wondering how I would do this. So if anyone knows how, please tell me

  2. #2
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,306
    Rep Power
    25

    Default Re: Download All Files from a Directory

    What connection do you have with the location of the files? Is there a server?
    If you don't understand my response, don't ignore it, ask a question.

  3. #3
    gimbal2 is online now Just a guy
    Join Date
    Jun 2013
    Location
    Netherlands
    Posts
    3,773
    Rep Power
    5

    Default Re: Download All Files from a Directory

    He mentions the internet, so I assume he wants to make yet another mass-downloader.
    "Syntactic sugar causes cancer of the semicolon." -- Alan Perlis

  4. #4
    SurfMan's Avatar
    SurfMan is offline Godlike
    Join Date
    Nov 2012
    Location
    The Netherlands
    Posts
    926
    Rep Power
    2

    Default Re: Download All Files from a Directory

    Programming is all about cracking your "problem" into smaller problems, and then again in even more smaller problems until you can solve each one seperately and then connect it all together. Now, how would you go and make this a smaller problem? No, the answer is not "post it on a forum and wait until people hand me Teh Code™". The answer is to look at all the words in your question, and try and see what's behind it. "Download" involves some actions, "all the files" imply something. So here we go, it's your lucky day:

    Original problem: "Download all the files in a directory off teh interwebs".
    Java Code:
    Sub-problems: 
    - Get the page where the URL points to
      Sub problems:
         * Make an HTTP connection
         * Get the output of the HTTP request
    - Parse the html, looking for file references
      Sub problems:
         * Create a parser that scans the HTML and looks for <a href="...">...</a> occurences (or whatever HTML your files are contained in)
         * Store file references in a list
    - Download the files
      Sub problems:
         * Iterate over the list
         * Create an HTTP connection for the file
         * Get the output of the HTTP request
         * Store the output as a file on some filesytem
    This list could be even longer, but I'll leave that as an exercise for the reader.

    Programming is all about breaking stuff down into smaller, manageble chunks, and solve each chunk. If you can't do that (or don't want to), then programming is not your thing. If you really want to learn how to code, analyzing your problem is a priceless skill.


    Edit: put the list in code tags to preserve formatting
    Last edited by SurfMan; 12-10-2013 at 10:16 AM.
    gimbal2 and Tolls like this.
    "It's not fixed until you stop calling the problem weird and you understand what was wrong." - gimbal2™ © 2013

  5. #5
    gimbal2 is online now Just a guy
    Join Date
    Jun 2013
    Location
    Netherlands
    Posts
    3,773
    Rep Power
    5

    Default Re: Download All Files from a Directory

    That is an excellent answer!
    "Syntactic sugar causes cancer of the semicolon." -- Alan Perlis

  6. #6
    cjburkey01 is offline Member
    Join Date
    Dec 2013
    Posts
    46
    Blog Entries
    2
    Rep Power
    0

    Default Re: Download All Files from a Directory

    Quote Originally Posted by SurfMan View Post
    Programming is all about cracking your "problem" into smaller problems, and then again in even more smaller problems until you can solve each one seperately and then connect it all together. Now, how would you go and make this a smaller problem? No, the answer is not "post it on a forum and wait until people hand me Teh Code™". The answer is to look at all the words in your question, and try and see what's behind it. "Download" involves some actions, "all the files" imply something. So here we go, it's your lucky day:

    Original problem: "Download all the files in a directory off teh interwebs".
    Java Code:
    Sub-problems: 
    - Get the page where the URL points to
      Sub problems:
         * Make an HTTP connection
         * Get the output of the HTTP request
    - Parse the html, looking for file references
      Sub problems:
         * Create a parser that scans the HTML and looks for <a href="http://www.java-forums.org/new-java/...">...</a> occurences (or whatever HTML your files are contained in)
         * Store file references in a list
    - Download the files
      Sub problems:
         * Iterate over the list
         * Create an HTTP connection for the file
         * Get the output of the HTTP request
         * Store the output as a file on some filesytem
    This list could be even longer, but I'll leave that as an exercise for the reader.

    Programming is all about breaking stuff down into smaller, manageble chunks, and solve each chunk. If you can't do that (or don't want to), then programming is not your thing. If you really want to learn how to code, analyzing your problem is a priceless skill.


    Edit: put the list in code tags to preserve formatting
    Well, that did explain it better, but to be more specific, there are about 20(ish) files in this directory on my website. I am making a Minecraft Modpack installer. I have everything but the download all the files code, I have a single file downloader that works, but I don't want to list out all the files. I would rather have it scan "http://cjburkey.com/minecraft-downloads/techpack/mods/" and run the
    Java Code:
    download(mc, iteratedfile);
    thing, btw, mc is my variable that holds the minecraft directory of the user. And iteratedfile is the file that is downloaded kind-a. Hope that cleared up any questions

  7. #7
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,306
    Rep Power
    25

    Default Re: Download All Files from a Directory

    Many parts of this project need to be defined.
    What code runs on both sides of this transfer? What protocol is used? What does the client send to the server? What will the server return to the client?
    If you don't understand my response, don't ignore it, ask a question.

  8. #8
    cjburkey01 is offline Member
    Join Date
    Dec 2013
    Posts
    46
    Blog Entries
    2
    Rep Power
    0

    Arrow Re: Download All Files from a Directory

    Quote Originally Posted by Norm View Post
    Many parts of this project need to be defined.
    What code runs on both sides of this transfer? What protocol is used? What does the client send to the server? What will the server return to the client?
    OK, Well, I wasn't planning on code running on the website, the client is running a Java application attemping to download all the files. The client sends a request to the server to download the files. The server will return the file to the client(or not return anything[I'm still a newb :P])

  9. #9
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,306
    Rep Power
    25

    Default Re: Download All Files from a Directory

    I wasn't planning on code running on the website
    There has to be code running on the server for it to be a server. Do you mean that you won't add any code to the server?

    There are different protocols for communicating with a server, which one are you planning on using? Not all servers support all protocols.
    If you don't understand my response, don't ignore it, ask a question.

  10. #10
    cjburkey01 is offline Member
    Join Date
    Dec 2013
    Posts
    46
    Blog Entries
    2
    Rep Power
    0

    Arrow Re: Download All Files from a Directory

    Quote Originally Posted by Norm View Post
    There has to be code running on the server for it to be a server. Do you mean that you won't add any code to the server?

    There are different protocols for communicating with a server, which one are you planning on using? Not all servers support all protocols.
    Sorry if this is confusing you, I can't explain well. I don't plan on adding code to the server. And from tests before, downloading files from this directory one by one will work as it can download files. I suppose it can run protocols.

  11. #11
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,306
    Rep Power
    25

    Default Re: Download All Files from a Directory

    Better do some research to see what protocols are and how to use them. That's what the P at the end of HTTP and FTP means.
    If you don't understand my response, don't ignore it, ask a question.

  12. #12
    SurfMan's Avatar
    SurfMan is offline Godlike
    Join Date
    Nov 2012
    Location
    The Netherlands
    Posts
    926
    Rep Power
    2

    Default Re: Download All Files from a Directory

    The solution is surprisingly simple. Store a text file on the server that contains all the filenames you want to download. Your minecraft mod will first download that list, then reads it, then downloads each file that is in there. You can even make sure that each line is a full URL that you can feed to a HttpURLConnection.
    Tolls and gimbal2 like this.
    "It's not fixed until you stop calling the problem weird and you understand what was wrong." - gimbal2™ © 2013

  13. #13
    cjburkey01 is offline Member
    Join Date
    Dec 2013
    Posts
    46
    Blog Entries
    2
    Rep Power
    0

    Exclamation Re: Download All Files from a Directory

    Quote Originally Posted by SurfMan View Post
    The solution is surprisingly simple. Store a text file on the server that contains all the filenames you want to download. Your minecraft mod will first download that list, then reads it, then downloads each file that is in there. You can even make sure that each line is a full URL that you can feed to a HttpURLConnection.
    I know! But how? I don't know how to do this. I am still newb. And btw, I was putting it in it's own program.

  14. #14
    SurfMan's Avatar
    SurfMan is offline Godlike
    Join Date
    Nov 2012
    Location
    The Netherlands
    Posts
    926
    Rep Power
    2

    Default Re: Download All Files from a Directory

    "It's not fixed until you stop calling the problem weird and you understand what was wrong." - gimbal2™ © 2013

  15. #15
    cjburkey01 is offline Member
    Join Date
    Dec 2013
    Posts
    46
    Blog Entries
    2
    Rep Power
    0

    Arrow Re: Download All Files from a Directory

    Like I said before(I even looked at this exact thread), I already know how to download a file. But I was wondering how I download all the files in an online directory.

  16. #16
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,306
    Rep Power
    25

    Default Re: Download All Files from a Directory

    I was wondering how I download all the files in an online directory.
    Go back to the beginning of this thread and read it again. That is what we've all been talking about here.
    If you don't understand my response, don't ignore it, ask a question.

  17. #17
    SurfMan's Avatar
    SurfMan is offline Godlike
    Join Date
    Nov 2012
    Location
    The Netherlands
    Posts
    926
    Rep Power
    2

    Default Re: Download All Files from a Directory

    If you were to follow my example, there are number of mod-files + 1 to download. Since you already know how to download a file, your problem is solved.
    "It's not fixed until you stop calling the problem weird and you understand what was wrong." - gimbal2™ © 2013

  18. #18
    cjburkey01 is offline Member
    Join Date
    Dec 2013
    Posts
    46
    Blog Entries
    2
    Rep Power
    0

    Default Re: Download All Files from a Directory

    Quote Originally Posted by SurfMan View Post
    If you were to follow my example, there are number of mod-files + 1 to download. Since you already know how to download a file, your problem is solved.
    So I should list the files in the java like:
    Java Code:
    download("yada1", "yadaplace");
    download("yada2", "yadaplace");
    download("yada3", "yadaplace");
    download("yada4", "yadaplace");
    download("yada5", "yadaplace");
    download("yada6", "yadaplace");
    etc

  19. #19
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,306
    Rep Power
    25

    Default Re: Download All Files from a Directory

    Using SurfMan's technique, the code would download ONE file that contained the paths to the other files. It would read the file that was downloaded to get the paths to the other files.
    If you don't understand my response, don't ignore it, ask a question.

  20. #20
    cjburkey01 is offline Member
    Join Date
    Dec 2013
    Posts
    46
    Blog Entries
    2
    Rep Power
    0

    Question Re: Download All Files from a Directory

    Quote Originally Posted by Norm View Post
    Using SurfMan's technique, the code would download ONE file that contained the paths to the other files. It would read the file that was downloaded to get the paths to the other files.
    So could I have an example of how it would read the file and download each file?

Page 1 of 3 123 LastLast

Similar Threads

  1. Download large files from web
    By Asolpdrag in forum New To Java
    Replies: 3
    Last Post: 02-04-2012, 05:47 PM
  2. Download Files From A Website
    By anthropamorphic in forum Networking
    Replies: 3
    Last Post: 10-28-2011, 03:45 AM
  3. download files from internet
    By sublixt in forum Advanced Java
    Replies: 2
    Last Post: 07-31-2011, 12:30 AM
  4. How to. Files and Directory
    By ocean in forum New To Java
    Replies: 4
    Last Post: 12-06-2009, 06:23 PM
  5. Replies: 2
    Last Post: 03-28-2009, 06:50 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •