Out of Memory exception while using ReadLine() to read large text stream repeatedly

Question

Problem: I have a very large text file which size is around 156M. my application needs to read through this text file again and again. There are roughly 1.5 millions of lines in this file and each time I read one line via ReadLine(). Here is the problem: when I read this file in 10th time (roughly 15 millions reads) an except occured within the line 'while...'. sReader = new StreamReader(cl.fileStreamName); string ad = ; while ( (ad = sReader.ReadLine())!= null){ //throw exception 'Out of Memory' //parse the string ad here } sReader.Close(); Envrionment: .NET FrameWork Version 1.0 Windows 2000 Pro Can someone give me a hint? Thank you Justin

Administrator · Answer

Forgot to answer your concerns. I don't have the benchmark of reading that huge file solely because my application does lots of stuff while reading this text file. I think your way should be pretty fast if your reads doesn't lock this file.

Administrator · Answer

Thanks always. Just like you described I did create a new StreamReader every time. Also I called GC.Collect() to clean up memory right after closing the object. I remembered there are some rumors that v1.1 partially solved the problems invloved with large mount of memory management in v1.0. So I did install v1.1 today you know what it solved this issue already. Have a good day!

Administrator · Answer

Are you Creating a new stream object everytime you read the file? With files that large it is not sufficient enough to just close the object and hope the Garbage Collector disposes of the object in a reasonable amount of time. I would suggest that after each successive read close the stream and explicitly call the garbage collector to gather your stream object. System.GC.Collect(); How long does it take you to read these files? I've tried reading large text files and something as large as 156 MB would take me approximately 2.5 to 3 minutes to read using asynchronus reads and multithreaded buffer streams. I was wondering if there was a more efficient route.