Welcome! Log In Create A New Profile

Advanced

java accessing csv file

Posted by Anonymous User 
Announcements Last Post
Announcement SoC Curricula 09/30/2017 01:08PM
Announcement Demarcation or scoping of examinations and assessment 02/13/2017 07:59AM
Announcement School of Computing Short Learning Programmes 11/24/2014 08:37AM
Announcement Unisa contact information 07/28/2011 01:28PM
Anonymous User
java accessing csv file
December 12, 2008 12:25PM
Hi

I have some java code that reads a csv file, does some stuff with it and writes to another csv file. the original csv files has 250300 lines of info, but the second file only has 250150 (as an example). why am I losing ~150 lines of info? How can I make sure that all lines from 1st file are put into 2nd file?

Thanks
celene
avatar Re: java accessing csv file
December 12, 2008 12:26PM
post code
Anonymous User
Re: java accessing csv file
December 12, 2008 12:31PM
I wrote a similar thing once and found that a has.next() check will strip a line. I used an external lib. Want it?
Re: java accessing csv file
December 12, 2008 12:31PM
How are you reading/writing? Perhaps a sort/duplicate issue?
Anonymous User
Re: java accessing csv file
December 12, 2008 12:35PM
File file = new File("something.csv"winking smiley;
FileWriter writer = new FileWriter("newFile.csv"winking smiley;
BufferedReader bufRdr = new BufferedReader(new FileReader(file));
String line = null;

//read each line of text file
while((line = bufRdr.readLine()) != null)
{
//does things and writes to newFile.csv
}



even when I just read one and and wrote straight to the newfile , it also skipped a few hundred lines
Re: java accessing csv file
December 12, 2008 12:36PM
yeah, it's the //does things and writes line that we need, though. smiling smiley
avatar Re: java accessing csv file
December 12, 2008 12:37PM
Maybe there are a couple hundred lines that are blank and is not being copied over to the new file?
Anonymous User
Re: java accessing csv file
December 12, 2008 12:37PM
maybe the file encoding is incompatible and java is reading the file wrong?
Anonymous User
Re: java accessing csv file
December 12, 2008 12:39PM
the most simple version that does not work properly:


while((line = bufRdr.readLine()) != null)
{

StringTokenizer st = new StringTokenizer(line,","winking smiley;
while (st.hasMoreTokens())
{
writer.write(st.nextToken());
writer.write(","winking smiley;

}
writer.write("\n"winking smiley;
}
Anonymous User
Re: java accessing csv file
December 12, 2008 12:40PM
it writes the blank lines to the new file. it does read the old file and it does write to the new file, just not everything
Anonymous User
Re: java accessing csv file
December 12, 2008 12:41PM
tokenizer class will make you cry
Re: java accessing csv file
December 12, 2008 12:42PM
Have you been able to manually check which lines are not being written over? And can you give us two or three of those?
Anonymous User
Re: java accessing csv file
December 12, 2008 12:45PM
unfortunately I can't give lines, its confidential info

I have checked and the last ~150 lines are always missing in the last file. I see that even the one piece of data is chopped in half and only half appears on the last line of new file.
Re: java accessing csv file
December 12, 2008 12:49PM
So, the 150 lines are getting lost at the end, apart from half of the very last line?

Did you carefully check those missing lines for anything weird about them? E.g. no or different style commas or line breaks or funny characters?
Anonymous User
Re: java accessing csv file
December 12, 2008 12:56PM
something like that, yes.

each line shows an email address, the message title, a batch number, a message number and a delivered status. There is nothing "special" about those last lot of lines that makes them different from all the rest

their batch numbers are the same a s a number of lines that do get copied over, so is the message title and the email addresses do occur elsewhere in the info I am copying.
Anonymous User
Re: java accessing csv file
December 12, 2008 01:10PM
/**********************************************************************************************************************    
 /   NOTES: I discarded the Jakarta lib for another: Ostermann's CSV lib. 

 * http://ostermiller.org/utils/doc/com/Ostermiller/util/CSVParser.html
 *It uses a 2 dimensional array.  I had to hard code the width references to the arrays. I figure that the data would stay in a static format.

 **********************************************************************************************************************/  

public void load(){
        try{
            FileReader fin = new FileReader("c:\\cstuff\\Stock.csv"winking smiley;

            CSVParser c = new CSVParser(fin);
            c.getLine(); //strip away the header text from the csv data file
            String body[][] = c.getAllValues();
            for (int i = 0; i < body.length; i++){

                Stock s = new Stock();
                s.setGrade(body[0]);
                s.setWidth(Integer.valueOf(body[1]));
                s.setDepth(Integer.valueOf(body[2]));
                s.setLength(Integer.valueOf(body[3]));

                list.add(s);
            }
           fin.close();
        }catch(IOException e){
            System.out.println("Load I/O exception"winking smiley;
            e.printStackTrace();
        }

    }//getting the Stock data from the files

avatar Re: java accessing csv file
December 12, 2008 01:18PM
Are you correctly closing the file and flushing the buffer?

--
"Knowledge has much better uses than self-pity and superiority"
Re: java accessing csv file
December 12, 2008 01:27PM
Are other duplicate lines properly copied over?

To test, please put in a dummy line (e.g.copy and paste the very first line of the file) just before the first line that is not copied over properly, and run again. Does that line get copied over correctly? If so, your problem lies with something in the file. If not, your problem lies with the code, I suspect.
avatar Re: java accessing csv file
December 12, 2008 01:30PM
Celene, isn't there maybe a size restriction on the length of the lines that you are copying? Have that problem before in a similar scenario with copy/paste.
Anonymous User
Re: java accessing csv file
December 13, 2008 05:51PM
malcolm Wrote:
-------------------------------------------------------
> Are you correctly closing the file and flushing
> the buffer?

good point smiling smiley
Re: java accessing csv file
December 17, 2008 07:23PM
malcolm Wrote:
-------------------------------------------------------
> Are you correctly closing the file and flushing
> the buffer?

Yip - be sure to flush the buffer before killing the writer object.
Anonymous User
Re: java accessing csv file
December 17, 2008 08:43PM
um, it's taken care of by Java. Also, if you long beards had RTF api, you would see that there is no flush included. I didn't call c.close(), so sue me tongue sticking out smiley This code is ancient anyways.

On a lighter note: If programming languages were religions
avatar Re: java accessing csv file
December 17, 2008 09:51PM
Haven't used java for a long time so won't argue with you on "Java automatically flushing" Rick but ...
Quote

I have checked and the last ~150 lines are always missing in the last file. I see that even the one piece of data is chopped in half and only half appears on the last line of new file.
... sounds suspiciously like the symptoms of a buffer that is not being flushed correctly still to me. Especially the line being chopped in half part.

I suggest calling .flush() on the file writer explicitly at the end of the program, and putting either a debug point or some kind of log message after the flush call to ensure that the call is actually been reached. If the problem still persists after this then at least this possibility can be ruled out.

--
"Knowledge has much better uses than self-pity and superiority"
avatar Re: java accessing csv file
December 17, 2008 10:51PM
@rick: thanx for that light hearted link - I'm definately between "the devil and the deep blue sea" with my VB & C++ exposure! Not to mention Java & Delphi.
Anonymous User
Re: java accessing csv file
December 17, 2008 11:28PM
ok, I RTF api too: Java doesn't automagically flush, there is a flush() under java.io. The ostermiller framework does use java.io.* so there can be a flush() called, but not directly through that api. Hopefully, the stranger who wrote that reusable code knew what he was doing winking smiley

Celene was using StringTokenizer. That class is known on the tubes as a steaming pile.

Quote
Java cookbook 2nd ed
CSV is deceptive. It looks simple at first glance, but the values may be quoted or unquoted. If quoted, they may further contain escaped quotes. This far exceeds the capabilities of the StringTokenizer class
Anonymous User
Re: java accessing csv file
December 18, 2008 07:41AM
thanks everyone, I did a writer.flush() and it works smiling smiley

I am very bad when it comes to closing and flushing things (except toilets)
avatar Re: java accessing csv file
December 18, 2008 07:46AM
celene Wrote:
-------------------------------------------------------
> thanks everyone, I did a writer.flush() and it
> works smiling smiley
>
> I am very bad when it comes to closing and
> flushing things (except toilets)


LOL grinning smiley
Anonymous User
Re: java accessing csv file
December 18, 2008 08:04AM
Thanks for putting that one to rest, Celene smiling smiley
avatar Re: java accessing csv file
December 18, 2008 08:42AM
As a further bit of trivia in C++ writing std::endl to a stream will cause a flush to occur in addition to a newline character.
While writing "\n" will only cause a newline character.
I see Celene was using "\n" so using Javas version of std::endl(Does Java have a version of this?) would probably also get rid of the problem, although admittedly some performance loss may result, but hey its Java anyway so performance is obviously already not important tongue sticking out smiley.

--
"Knowledge has much better uses than self-pity and superiority"
Anonymous User
Re: java accessing csv file
December 18, 2008 09:40AM
OutputStream.flush() from java.io.OutputStream won't cause a '\n' to be written. But if one needs that functionality, you could override it? I doubt that the performance hit will be measurable.
Sorry, only registered users may post in this forum.

Click here to login