Memory exception when handling very long Lists

Question

Hello Is there some technique allowing to handle very long lists. In my application when the length of a List surpasses a value about 16'000'000 the OutOfMemoryException is generated. Regards. Pavel.

Akkiraju Ivaturi · Answer

Good to know Pavel. Good luck!!!

Pavel · Answer

Thank you guys. I've changed the concept so no need in long lists.

Akkiraju Ivaturi · Answer

I just want to add to what Vulpes has mentioned. Basically the memory allocation in .NET is done in continuous blocks. If there is memory available as chunks scattered across the blocks then it will not be allocated and you will be getting the out of memory exception. Without code it is hard to follow exactly what you are doing. I think many of the values are more than 85000 byte threshold and so your objects endup on LOH and so the GC was not collecting the objects and as you continue processing objects you soon end up out of address space. FYI LOH is the normal source of OutOfMemoryException. Or you might creating a string objects over and over in your code and so it is creating memory issue. Or you might be performing many operations while adding data to list and processing it over and over. If the reason is not known and could not resolve this I would advise you to use sequences and forget about lists. Rather than List or List  try for IEnumerable or IEnumerable .

Vulpes · Answer

Do you know the approximate size in bytes of the objects you're adding to the list since with 16 million entries it's feasible that you may actually be running out of memory depending on what else you may be doing? If the objects are not too large then one thing that may help is to set the List's initial capacity (in the constructor) to whatever number of items you think will eventually be reached. If you do that then sufficient memory will be allocated on the 'large object heap' to accommodate this number of items in the List's internal array and no more allocations will be needed. If you don't do it then the List's capacity will be doubled from time to time as more space is needed by creating a new larger internal array and copying the old items into it. This means that at around 16 million items a new array of 32 million items will need to be created and even after a GC to get rid of the old one it may not be possible to find sufficient memory on the LOH to accommodate the new array because (for performance reasons) the LOH is never compacted.