Results 1 to 6 of 6
  1. #1
    datathinker is offline Member
    Join Date
    Jul 2013
    Location
    Beijing, China
    Posts
    2
    Rep Power
    0

    Default Data processing without Database

    Struggling in coding for some years, I have experienced many different tough problems in my work. However, I never complain about these difficult issues, instead, I really appreciate them because they made me a better programmer.

    Similarly some days earlier, I came up with an issue about Data process (here refers to Data query, Select result set from data source after processing). Whenever DATA is involved in our java application, what comes to our mind immediately is Database, because it’s easy, powerful, but costly. So here I would like to discuss how does java process Data without database?

    In my application, my data-source is an Excel sheet downloaded from a finance website; I have to figure out the Shares rising for N consecutive days in a certain period. For the computation on structured data, programmers usually embed the SQL statements in the Java code, and access the database server via JDBC.
    This is what we did conventionally when Database is employed.

    Without Database, I tried many ways to work this out. Initially I tried to process Excel Sheet Data with java coding, but I failed because it became very complex for java to complete simply data computation. One issue is how to store these data imported from Excel Sheet. No more options except List. How about aggregating, filter, sorting and grouping, we can not imagine how complicated this is. How to figure out the Shares rising for N consecutive days in a certain period from an Excel Sheet downloaded from finance web? I consulted many people and they shared their similar experience with me. One of the solutions they shared with me is called Moderator edit: removed which I never heard of before. At first look, it is similar with Excel cell style, it is a little bit new for me, but it is not difficulty to use. In my case, the code for this issue is given below:

    A1: =file(“E:\\targetData.xlsx”).import@t() //import data
    A2=A1.select(Date>=paramStart&&Date<=paramEnd) //filter by data
    A3=A2.group(Code) //group by Code
    A4=A3.(~.sort(Date)) //sort by Date in each group
    A5=A4.(~.dup@t()) //change to Tseq(structured data)
    A6=A5.(~.derive((Close-Close[-1]):INC)) //add a colum, increased price
    A7=A6.(~derive(if(INC>0,CID=CID[-1]+1,0):CID)) // add a column, continuous increased days
    A8=A7.select(~.max(CID)>=ParamNDays) //stock max CID is bigger than N days
    A9=A8.(~.Code) //finally, stock code

    My difficult issue is solved, but it is with the help of a data process middleware Moderator edit: removed, and I failed to work out this purely with java or other tools. Here I would like to see what would you do if you are having similar case? Please share with me and put you idea or code in your reply, thanks.
    Last edited by DarrylBurke; 08-12-2013 at 10:52 AM. Reason: Removed link

  2. #2
    DarrylBurke's Avatar
    DarrylBurke is offline Member
    Join Date
    Sep 2008
    Location
    Madgaon, Goa, India
    Posts
    11,242
    Rep Power
    19

    Default Re: Data processing without Database

    I have no personal experience with it, but I believe the most common way of working with an Excel workbook from Java is via Apache POI. You might like to research that.

    Since you say you want to avoid the middleware you named, the two links to it that you included in your post are irrelevant. I've removed them.

    If you plan to post any more code, please go through the Guide For New Members and BB Code List - Java Programming Forum - Learn Java Programming

    db
    If you're forever cleaning cobwebs, it's time to get rid of the spiders.

  3. #3
    gimbal2 is offline Just a guy
    Join Date
    Jun 2013
    Location
    Netherlands
    Posts
    4,030
    Rep Power
    6

    Default Re: Data processing without Database

    Or JExcel. I believe that JExcel is a little more resource friendly than POI, but I don't know what its capabilities are.
    "Syntactic sugar causes cancer of the semicolon." -- Alan Perlis

  4. #4
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default Re: Data processing without Database

    If your excel does not have "mixed" columns (i.e. the first few rows in a column contains strings and the rest of the rows in that column numbers, etc) than you can also use JDBC and the JDBC-ODBC bridge, as long as the ODBC Driver is available.

    P.S. The point about "mixed" columns is an ODBC/Excel problem (at least it was before Excel 2007, haven't played with that aspect after that), not a Java one. The data contained in the first few rows will determine the columns type, and then when data of a different type appears in that column a "null" is most often returned, rather than the value contained in the column.

    P.S. @gimbal2, does JExcel work with Excel 2007+, I didn't think it did.

  5. #5
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    12,014
    Rep Power
    20

    Default Re: Data processing without Database

    "How about aggregating, filter, sorting and grouping"

    But...that *is* a database.
    Why are you avoiding one?
    Even a simple embedded one would give you access to much of that and not cost a penny.

    Yes, you would need to parse the Excel sheet into a database structure, but you're going to have to parse it anyway.
    Please do not ask for code as refusal often offends.

    ** This space for rent **

  6. #6
    DarrylBurke's Avatar
    DarrylBurke is offline Member
    Join Date
    Sep 2008
    Location
    Madgaon, Goa, India
    Posts
    11,242
    Rep Power
    19

    Default Re: Data processing without Database

    datathinker:

    As you edited your post to add back the name of a third party software you had earlier provided two links for and which I had removed, I conclude that you're more interested in spamming the name of that software than you are in getting any useful advice.

    I've removed the name again and I'm closing this thread. Any more spammy activity and you will be banned. There will be no further warning.

    db

    THREAD CLOSED
    If you're forever cleaning cobwebs, it's time to get rid of the spiders.

Similar Threads

  1. Can someone please help me persist data to a database
    By alexandra12 in forum New To Java
    Replies: 2
    Last Post: 08-24-2012, 03:24 PM
  2. Replies: 0
    Last Post: 01-05-2011, 10:53 PM
  3. how to insert data in database
    By Vinay Bhatia in forum JDBC
    Replies: 6
    Last Post: 11-03-2010, 04:06 PM
  4. Not able to data into database
    By nash1990 in forum JavaServer Pages (JSP) and JSTL
    Replies: 1
    Last Post: 08-11-2010, 05:01 PM
  5. Data processing API <algorithm>
    By roughjj in forum Advanced Java
    Replies: 4
    Last Post: 09-08-2009, 09:34 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •