How to reduce Memory Leak (Commit memory Size) ?

Question

Hello  In my application when i try to fill dataset's table with huge amount of records (2 Lakh Approx.) it gives error of outofmemory Exception . I think it is Due to increasing Virtual memory size (Commit size is increasing in Task Manager) . So how to prevent this memory link ? Thanks in Advance . Regards Yogesh

Vulpes · Answer

Depending on how big they are and what else you're doing you may well be running out of memory with 200000 records in your DataSet. Effectively you have a maximum of about 1.4 GB available in most .NET applications of any size. So is it really necessary to read all that data into memory at once? Could you read say the first 10000 records process those and then get the next 10000 etc.?

Shirsendu Nandi · Answer

if you use sqlserver 2008 then you can use clustered index or if you use sm sqlserver 2008 then you can use force seek or filtured cluster index to fetch the data

try abc · Answer

Hello  Thanks For Answers. But I have already optimized Query with Required Fields & Required Where condition so it required 50000- 2 Lakh Records in dataset at a time . I am working on Windows Application & in that for Crystal Report  i hv used dataset . to fill data for report in Dataset  it gives memory error . What i can do ? Also except for above problem  due to heavy database of my s/w in Client DB  I have to fetch atleast 25000 - 1 lakh records everywhere out of 6-7 Lakh Records. So Is there any way that we can reduce Virtual memory ?? Regards Yogesh S/W Developer

Vulpes · Answer

Sam makes a good point about only loading some of the fields if there are large ones. One thing I've noticed is that people often store images in their databases rather than a path to where the image is stored on disk. If this is the case with your database then the memory usage for 200000 records is going to be astronomic!

Sam Hobbs · Answer

In addition to what has already been said another possibility is to eliminate some of the fields taht you read in all at once. If there is a large field that you don't need then not reading it can really help. The table should have a primary key and you probably would want to read in the primary key or some key at least. Then you can retrieve additional fields later by retrieving a record based on a key. Also note that 64-bit Windows will allow you to use much more main memory.

Mahesh Chand · Answer

As Vulpes said you do not need to get all 200000 records. Get a chunk of data that you are working with usually couple of thousands. If you are doing search or any other operations on all records do that on the database side and get only results that you need. Read paging concepts in ASP.NET. Search this website for the same topic. You can also use SELECT..TOP SQL statement to get say top 1000 records.

Shirsendu Nandi · Answer

use parallel processing or threading concept...