Fast copying of data to a file

Question

hello i wish to know if there is a way to copy data to a file faster then the following approach used below ? What i am doing is to write matrix/array data to a .cvs file. there are on an average 150000 elements in the array/matrix with each element having three double data type members.The code is here: var path = Path.Combine(myData KinectData- + time + .txt); // start writing the data to file for (int i = 0; i { System.IO.File.AppendAllText(@path Convert.ToString(p1[i].X) +   + Convert.ToString(p1[i].Y) +   + Convert.ToString(p1[i].Z) + Environment.NewLine); System.IO.File.AppendAllText(@path Convert.ToString(p2[i].X) +   + Convert.ToString(p2[i].Y) +   + Convert.ToString(p2[i].Z) + Environment.NewLine); } The time taken for this process is close to 3 min which i wish to minimize.any solution will be of great value to me. Thank you

Vulpes · Answer

I'd see if this is any faster: var path = Path.Combine(myData KinectData- + time + .txt); // start writing the data to file int batchSize = 1000; int numBatches = maxLen/batchSize; int remainder = maxLen % batchSize; if (remainder > 0) ++numBatches; var sb = new System.Text.StringBuilder(132 * batchSize); using(var sw = new System.IO.StreamWriter(path true)) { for(int b = 0; b { if( (b == numBatches - 1) && remainder > 0) batchSize = remainder; for (int j = 0; j { int i = b * 1000 + j; sb.Append(p1[i].X); sb.Append( ); sb.Append(p1[i].Y); sb.Append( ); sb.Append(p1[i].Z); sb.AppendLine(); sb.Append(p2[i].X); sb.Append( ); sb.Append(p2[i].Y); sb.Append( ); sb.Append(p2[i].Z); sb.AppendLine(); } sw.Write(sb.ToString()); if (b } }

Vulpes · Answer

There were two points which struck me about your original code: 1. You were doing a lot of string concatenations. 2. You were appending to the file after every iteration and - worse still - opening and closing it each time. The first problem can be solved by using a StringBuilder and the second by only appending to the file after (say) every 1000 iterations and by opening and closing the file just the once. Also if we make the capacity of the StringBuilder big enough then it won't need to keep increasing the size of its internal buffer which slows things down. So I figured a maximum of 20 characters for each of the 6 doubles 8 characters for the spaces/commas and 4 characters for the new lines (
) making 132 in total multiply by 1000 to give 132000. Also if you just clear the StringBuilder after each batch (by setting its Length to zero) one instance of it will suffice :)

Sanju Singh · Answer

wonderful! thanks for the quick answers Vulpes.

Sanju Singh · Answer

hello Vulpes. I am sorry for the last comment here. the method you have suggested here works perfectly fine and the results are amazing. it took close to 1 sec to do the same task ! Thanks for the wonderful technique. I would like to know why i was taking so long for doing the same task? Is it the no of times write is done which made the difference ? thanks

Sanju Singh · Answer

hello Vulpes. i tried this method.A file does get created but the data it contain is either NaN OR O only like this NaNNaNNaN 000 NaNNaNNaN 000 NaNNaNNaN 000 N aNNaNNaN 000 N aNNaNNaN 000 NaNNaNNaN 000 and so on.