Design help needed: Requirement to process huge number of data efficientlty

Question

Hi All We have a requirement to process a huge number of data efficiently without any idle time. The data that we need to process is related. Means we have to process some 'n' number of related data by one DLL or thread. That is we need to process the data in sets. Currently we are implementing the same using the following approach. We are having a Windows Service and 'n' number of DLLS. (Using Reflection) The service will find the 'n' sets of data and it will assign the those 'n' sets of data to the 'n' DLLS for processing. Once the DLLs are completed processing the service will assign the next set of data for processing The algorithm is like this: Service will find the number of DLLs which are free(idle) say 20 DLLs are free at a particular point of time. Service will fetch the 20 sets of data and assign those 20 sets to 20 DDLs The issue we are facing is as follows: Say the service is assigning the a set of data to 15th DLL of 20 free DLLs. Within that time there might be some n DLLS which got processed and will be idle stage till the service assigns the data to remaining 5 DLLS. So the idle DLLs has to wait till the service finishes one round of assignment. We need to avid the wait time to process the data efficiently. Could anyone suggest a better way to solve this problem. We are in a position to redesign the system (ie Service and n DLLs). We are also thinking of doing the same using threads. But the issue is the CPU usage is very high while using threading. Any design pattern will serve my purpose in a better way? Can anyone suggest any other better design? Thanks Praveen Nair

Praveen K · Answer

Hi Matt I am new to multi-threaded programming and I am having a requirement to process huge amount of data. I did lots of R&D and found Asynchronous Multithreading using Delegates is the best way to implement it. I have designed my service based on this and the logic is as follows. I have huge amount of data in MySQL Database and the Status of the data will be marked as READY before processing. I will pick only READY records for processing and once the service picks the records I will mark them as ASSIGNED. Then different threads will process these data and once the processing is done it will mark the records as PROCESSED or ERROR. The problem I am facing over here is the same record is getting picked more than once which was not supposed to. Is this because of my class design not thread-safe? If yes how I can design a thread-safe architecture. Please advise me how I can get out of this. Thanks in Advance

Praveen K · Answer

Hi Matt Thanks a lot ffor your reply. I have designed the service with the asynchronous threading. I am using delegates for assigning methods to threads and I am using AsyncCallBack to notify the main thread. I am testing the application in a dual core machine. I have tried with 10 threads as well as with 20 threads. I am almost getting double performance if I am using 20 threads instead of 10. Anyways as you said I will try runnig the service with 2 threads. I am actually connecting to MySQL and Oracle from the service and I am using ODP.NET for connecting to Oracle. If I am setting the oracle connection pool Max size as 10 or 20 initially I can see only 1 active connection made to Oracle. But after some processing (say after 1-2 min) it is using all 20 oralce connections for processing. I am wondeting why the ODP.NET is behaving like this. I have noticed that the performance is hitting when we are enabling MySQL Pooling. Please share your thoughts on these issues. Also please advice me how I can make the service thread-safe. Thanks in Advance

Matthew Cochran · Answer

First of all your best performance will be gained by having the same number of threads running on your machine as the number of processors (i.e. for a Dual core machine 2 running threads is optimal). I'm not sure if I correctly understand your situation but if you are starting 10 processes concurrently your application will run MUCH slower because your processors will be burning lots of cycles doing context switching between the threads (each process starts with it's own thread). Your best bet is to have a single process that queues the work to the thread pool because the threadpool will manage the threads and optimize allocation of work to the processors. This way your app will also scale better if you decide to scale horizontally and the machines have different hardware. The other trick is to keep your threads saturated. This means not having any i/o blocking a thread when you read a file or hit a database. You can avoid this through the async programming model in .NET. I hope this helps -Matt