How to reduce Memory Leak (Commit memory Size) ?

Question

Hello ,

In my application when i try to fill dataset's table with huge amount of records (2 Lakh Approx.) it gives error of outofmemory Exception .

I think it is Due to increasing Virtual memory size (Commit size is increasing in Task Manager) .

So how to prevent this memory link ?

Thanks in Advance .

Regards,
Yogesh

Vulpes · Answer

Depending on how big they are and what else you're doing, you may well be running out of memory with 200,000 records in your DataSet. Effectively, you have a maximum of about 1.4 GB available in most .NET applications of any size. So, is it really necessary to read all that data into memory at once? Could you read say the first 10,000 records, process those and then get the next 10,000 etc.?

Shirsendu Nandi · Answer

if you use sqlserver 2008 then you can use clustered index or if you use sm sqlserver 2008 then you can use force seek or filtured cluster index to fetch the data

try abc · Answer

Hello , Thanks For Answers. But I have already optimized Query with Required Fields & Required Where condition so it required 50000- 2 Lakh Records in dataset at a time . I am working on Windows Application & in that for Crystal Report , i hv used dataset . to fill data for report in Dataset , it gives memory error . What i can do ? Also except for above problem , due to heavy database of my s/w in Client DB , I have to fetch atleast 25000 - 1 lakh records everywhere out of 6-7 Lakh Records. So Is there any way that we can reduce Virtual memory ?? Regards, Yogesh S/W Developer

Vulpes · Answer

Sam makes a good point about only loading some of the fields if there are large ones. One thing I've noticed is that people often store images in their databases rather than a path to where the image is stored on disk. If this is the case with your database, then the memory usage for 200,000 records is going to be astronomic!

Sam Hobbs · Answer

In addition to what has already been said, another possibility is to eliminate some of the fields taht you read in all at once. If there is a large field that you don't need then not reading it can really help. The table should have a primary key and you probably would want to read in the primary key or some key at least. Then you can retrieve additional fields later by retrieving a record based on a key. Also note that 64-bit Windows will allow you to use much more main memory.

Mahesh Chand · Answer

As Vulpes said, you do not need to get all 200,000 records. Get a chunk of data that you are working with, usually couple of thousands. If you are doing search or any other operations on all records, do that on the database side and get only results that you need. Read paging concepts in ASP.NET. Search this website for the same topic. You can also use SELECT..TOP SQL statement to get, say top 1000 records.

Shirsendu Nandi · Answer

use parallel processing or threading concept...

How to reduce Memory Leak (Commit memory Size) ?

Pankaj Gupta

Bechir Bejaoui