Results 1 to 13 of 13
  1. #1
    Dark's Avatar
    Dark is offline Senior Member
    Join Date
    Apr 2011
    Location
    Camp Lejuene, North Carolina
    Posts
    643
    Rep Power
    4

    Default Working in theory, a inquisitive file output question.

    So a while back I was looking at a brute force dictionary program some little script kiddie had written in visual basic. There were severe limitations to this program, like that it couldn't write a list of combinations that were over 7 characters long because of the way it wrote to a file.

    Now I want to try out to see if using Java something similar can be accomplished without a drawback. Not for the idea of making a brute force dictionary program, but because I was wondering if writing large amounts of data to a file would have the same draw backs in Java.

    The visual basic program had to store all the combinations inside the RAM, which is an obviously terrible idea. If I used a BufferedWriter(FileWriter) I would be able to append to the end of the file, making it possible to store less in the memory.

    Would I be correct in seeking this combination for this task? I don't know if a BufferedWriter keeps the strings that it will write to the file in its memory before writing it or if it will append the file one line at a time.

    Another question is that if I append a file a line at a time, is there any known drawbacks to this? Performance issues maybe? I haven't been able to quench my thirst for knowledge of this subject on Google so far, but that's half due to the fact that this government web doesn't much like the idea of pursuing knowledge and blocks the websites.

    EDIT: Spelled the title wrong
    Last edited by Dark; 07-19-2011 at 09:13 PM.
    • Use [code][/code] tags when posting code. That way people don't want to stab their eyes out when trying to help you.
    • +Rep people for helpful posts.

  2. #2
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,412
    Rep Power
    25

    Default

    BufferedReader keeps the strings that it will write to the file in its memory before writing it
    Until JosAH comes and explains it, here is my opinion.
    The BufferedReader does a bit of reading ahead to keep the user supplied with data in a smooth fashion and to smooth out the disk accesses.
    Whoops just noticed: Reader and write. I assume they should match.
    For writing, the same principle, do the I/O to smooth out the disk accesses.

  3. #3
    JosAH's Avatar
    JosAH is online now Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,525
    Blog Entries
    7
    Rep Power
    20

    Default

    Maybe I just don't understand your question, but a BufferedWriter uses a fixed size buffer; only when its completely filled up it gets written to the underlying Writer (or when the BufferedWriter is closed or flushed). Normally a 4K character sized buffer is used (8KB) which nicely matches the disk buffer size. Trying to use a bigger buffer just costs you more RAM and doesn't do much (performance wise speaking).

    kind regards,

    Jos

    p.s. try to avoid opening a file, append one line to it and close it again afterwards; all in a loop. That is slow.
    cenosillicaphobia: the fear for an empty beer glass

  4. #4
    Dark's Avatar
    Dark is offline Senior Member
    Join Date
    Apr 2011
    Location
    Camp Lejuene, North Carolina
    Posts
    643
    Rep Power
    4

    Default

    Oops, typo. I fixed it. Yes I meant them both to be writers.

    So until JosAH comes by and graces us with his knowledge of the subject, you're saying that if I were to pursue this program I could indeed make it more efficient than the one made in Visual Basic. I was looking at the BufferedWriter flush method, and from my understanding of it, that would cause it to write everything it has in memory at the moment to the file correct? If that's true then even if the BufferedWriter did store everything into memory I could simply implement a counter and flush it accordingly no?
    • Use [code][/code] tags when posting code. That way people don't want to stab their eyes out when trying to help you.
    • +Rep people for helpful posts.

  5. #5
    Dark's Avatar
    Dark is offline Senior Member
    Join Date
    Apr 2011
    Location
    Camp Lejuene, North Carolina
    Posts
    643
    Rep Power
    4

    Default

    Ah my response was too slow.

    @JosAH, I was planning on leaving the file open just pushing the information to the opened file each time I go to write to it. I haven't even begun coding yet, I was just curious on if it was possible and how. I figured it was possible, but seeing as I don't have a computer nearby that I can code in Java (and run it) I was going to seek out the theory behind it first.

    So what you're saying is that I could simply just use a buffered writer and it won't cause memory problems because it will write to the file every time it hits its memory limit?
    • Use [code][/code] tags when posting code. That way people don't want to stab their eyes out when trying to help you.
    • +Rep people for helpful posts.

  6. #6
    JosAH's Avatar
    JosAH is online now Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,525
    Blog Entries
    7
    Rep Power
    20

    Default

    Quote Originally Posted by Dark View Post
    Ah my response was too slow.

    @JosAH, I was planning on leaving the file open just pushing the information to the opened file each time I go to write to it. I haven't even begun coding yet, I was just curious on if it was possible and how. I figured it was possible, but seeing as I don't have a computer nearby that I can code in Java (and run it) I was going to seek out the theory behind it first.

    So what you're saying is that I could simply just use a buffered writer and it won't cause memory problems because it will write to the file every time it hits its memory limit?
    I don't know; what made me curious is this phrase in your OP:

    The visual basic program had to store all the combinations inside the RAM, which is an obviously terrible idea.
    What combinations? Can you tell us some more about the actual problem before we attempt to optimze something that doesn't need to be optimized?

    kind regards,

    Jos
    cenosillicaphobia: the fear for an empty beer glass

  7. #7
    Dark's Avatar
    Dark is offline Senior Member
    Join Date
    Apr 2011
    Location
    Camp Lejuene, North Carolina
    Posts
    643
    Rep Power
    4

    Default

    Ok, the idea behind the massive volume of information that will be printed to the file is every combination of the characters in a string like "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVW XYZ1234567890!@#$%^^^&&*()" up to specified length starting from 1 character, up to the number specified for max length.

    Ie
    When the loop got to 3 characters long it would be printing
    aaa
    aab
    aac to the file.

    When it finished all possible combinations of three letters it would then print
    aaaa
    aaab
    aaac until it finished with that one too.

    I understand this is an extremely large volume of information, billions of combinations, maybe even more. I haven't attempted the program, I figured that to make the files a bearable size, every couple thousand lines would then close the old file and create a new one. That way, whenever a program accesses the newly created file, it wouldn't be loading gigs of combinations into the ram for use.
    • Use [code][/code] tags when posting code. That way people don't want to stab their eyes out when trying to help you.
    • +Rep people for helpful posts.

  8. #8
    JosAH's Avatar
    JosAH is online now Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,525
    Blog Entries
    7
    Rep Power
    20

    Default

    Are the characters in that String all unique?

    kind regards,

    Jos
    cenosillicaphobia: the fear for an empty beer glass

  9. #9
    Dark's Avatar
    Dark is offline Senior Member
    Join Date
    Apr 2011
    Location
    Camp Lejuene, North Carolina
    Posts
    643
    Rep Power
    4

    Default

    Yes, there are no repeating letters or numbers in the entered string of acceptable combination values. Seeing as it already will use that letter multiple times, there is no need to have multiple a's in the string. I plan on filtering out the submitted values to make it so there are no repeating values. I also planned on splitting it into an array.
    • Use [code][/code] tags when posting code. That way people don't want to stab their eyes out when trying to help you.
    • +Rep people for helpful posts.

  10. #10
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,412
    Rep Power
    25

    Default

    It sounds like a base N exercise, where N is the number of characters in the String.
    If it were base 10 you would start at 0 and counting normally progress thru 9999999999....

  11. #11
    JosAH's Avatar
    JosAH is online now Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,525
    Blog Entries
    7
    Rep Power
    20

    Default

    Quote Originally Posted by Norm View Post
    It sounds like a base N exercise, where N is the number of characters in the String.
    If it were base 10 you would start at 0 and counting normally progress thru 9999999999....
    Yep, you only have to know how many digits are supposed to be in the result, e.g. 0,00, 000 etc. If you know that you don't even have to generate all these combinations, just two numbers will do: number of digits and the rank.

    kind regards,

    Jos
    cenosillicaphobia: the fear for an empty beer glass

  12. #12
    bnson is offline Artemis
    Join Date
    Jul 2011
    Posts
    26
    Rep Power
    0

    Default

    Hello,

    My english not good, specialized technical words......

    I tried to understand your purpose, and if I understand correctly you want to write a file from a large data and memory are not allowed to do so?

  13. #13
    Dark's Avatar
    Dark is offline Senior Member
    Join Date
    Apr 2011
    Location
    Camp Lejuene, North Carolina
    Posts
    643
    Rep Power
    4

    Default

    Hmm, that sounds like an interesting approach that I will have to check out.

    However my question was more of if its possible to write all these combination's to external files so that other programs could use it. I know you could write to these files, but I was wondering if I would run into memory problems if I tried to write this many combination's to a file. From the looks of it, the answer is no. However it wouldn't be the best idea to do it this way.
    • Use [code][/code] tags when posting code. That way people don't want to stab their eyes out when trying to help you.
    • +Rep people for helpful posts.

Similar Threads

  1. Specimen exam quesiton
    By elliotHenry in forum New To Java
    Replies: 14
    Last Post: 06-14-2011, 09:18 PM
  2. OO Class Theory Question
    By Dark in forum New To Java
    Replies: 22
    Last Post: 04-20-2011, 02:16 PM
  3. Replies: 0
    Last Post: 09-30-2010, 08:16 PM
  4. Minimax game theory with mancala
    By jigglywiggly in forum New To Java
    Replies: 2
    Last Post: 01-01-2010, 12:04 PM
  5. [SOLVED] Need help in theory of Java in netbeans
    By kirly in forum New To Java
    Replies: 3
    Last Post: 11-11-2008, 05:47 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •