The most expedient and efficient way to browse a file

Question

I have a file that contains over 3000 tab delimited entries. As of now the file is open at start time and the data is placed in a list of objects containing twelve data members, and then accessed multiple times during the execution of the program.
I always thought it was the best solution (even though the creation of that many objects take space), but isn't there a more productive way to go about it i.e. open the file each time it needs to be searched or is that process slower?
Developing environment: Visual Studio 2010, Windows Forms MonoTouch and Mono for Android, Web Service.
Thanx
Luke

Sam Hobbs · Answer

In the past main memory was very costly whereas it is now cheap. Modern computers have ample memory. You don't specify how much main memory is used. Physical hard drives spin and the computer usuallly needs to wait for the platters to spin around. Also the heads move and if the hard drive is used for anything else then the heads must move around for multiple applications. Seek time can be very expensive in terms of time. So Windows keeps a buffer of data. If it is able to retain the entire file in its buffer then there is no need to wait for the physical drive otherwise each time the file is read it is potentially very innefficient in terms of time. If Wiindows is able to retain the contents of the file in its buffer then the memory will be used up even though you don't see it. So if you read the file only once then Windows will be able to use the buffer memory for other purposes. Also I am confused; it is not possible to use Windows Forms with MonoTouch and Mono for Android.

Sam Hobbs · Answer

Smart phones are different. Most of what I said does not apply to smart phones since they don't use physical drives that have movement. I am not sure what is most efficient for smart phones. Probably for some smart phones 17 MB is significant. For smart phones you need to also consider battery life. So moving a lot of data around might take time and battery power but dedicating 17 MB for the application data might also be a problem. I think the ideal would be to design your data so that you can access pieces as needed without loading all of it in together. I realize that that might not be possible but if it possible then it would be worth designing. For example database tables usually have a primary key and you can get records using their primary key. The advantage of loading into memory is to avoid the costly delay of accessing physical data from a device that moves. Smart phones do not have devices such as that. I am not sure but I think that for many smart phones there is not a difference between main memory and permanent memory so for those loading into memory would essentially just make a second copy except the second copy would be arranged differently. Windows has something called memory mapped files. Using that you can access file contents as if the file is main memory. Windows uses the virtual memory system to load pages when needed. Windows uses the virtual memory system to decide what pages are kept in memory. If your data in permanent storage (a hard drive) is formatted in the same manner that it would be in main memory then you could let Windows take care of reading the hard drive.

jeanchal · Answer

Hi Sam my program requires 17 mg. I have tested this on an older slower Windows operated smart phone and it loads up in a fairly good amount of time not only the file in question but along with other similar larger files (about 14 secs a splash screen and progress bar is shown to the user). I have yet to try this on faster phones but no doubt it'll be faster. Thought there might have been a better leaner solution. I meant to say that I am developing for Windows PC (Windows Forms) as well as for mobile apps such as iPhone and Android using Mono. Thanx for your reply. Luke