If you provide a SSCCE that would be fantastic. A good place to start would be to reduce that 2 minute/task time to < 5-10 seconds if possible.
|
Hey guys,
I have a client/server application. Clients make requests to the server and the server responds with a result.
The problem
A request that a client makes can take up to 2 minutes to serve. This is because the operations are heavy and blocking. The operations are also unique and the results cannot be cached.
To tackle this, multi-threading is the best option. The operation gets done on it's own thread.
Problem A
The server receives around 100 requests every couple of seconds so I cannot create a new thread as I'd end up with hundreds. Although is work, it's not a scaleable solution and in general this is frowned upon.
Problem B
To tackle problem A - i have tried to use a Thread Pool which is what most of you guys would recommend. The issue I have with this is that because the operations take a long time, the pool runs out of threads.
As mentioned, the server receives around 100 requests every couple of seconds. Your average thread pool size is based upon CPU cores so let's pretend 4 (quad core) - The pool gives 4 threads, so that's 4 operations being processed but 96 requests waiting for a thread to become available and I end up getting blocked by the thread pool itself.
Any suggestions on what I should do? How can I serve all the requests without blocking my program and at a reasonable time?
Thanks
If you provide a SSCCE that would be fantastic. A good place to start would be to reduce that 2 minute/task time to < 5-10 seconds if possible.
So just run the threadpool itself in a seperate thread..?
Then in your main thread you can keep accepting requests and stacking up tasks for the pool.
but obviously if each task takes 2 whole minutes to compute and you are getting 100 requests a second - you will need to take a break with accepting requests at some point or you will end up with too many pent up tasks.
Also can i ask what it is that requires something that takes 2 minutes to compute to be done 100 times a second?
If each task is taking 2 minutes to compute but you are getting 100 task requests a second, you definitely need to look in to if there are any bottlenecks in your code.
You also need to definitely look in to Asynchronous programming, this lets your computation(s) not block the rest of your program.
With java, this would be... idk RXJava? Python uses its own Async stdlib. Kotlin just got Coroutines with stable status, C# uses its own async\await stuff.
Even with only 4 cores, you can have many many more software threads going. For example, as of writing this comment my 4c laptop has ~1900 software threads running.
Also what in the world is taking 2 minutes to compute? Maybe if you posted an example of the code we could possibly help identify flaws that would speed up your code? Recently had a classmate refactor and optimize his class project to run in 5 minutes down to 15 seconds.
Use a non-blocking system. Java provides non-blocking IO through their NIO (New IO) package.
Frameworks such as Netty are built ontop of this, as it allows a single thread to handle requests from different clients by polling a Selector to see which events have came in (read? write?). This will allow you to handle multiple connections from a single thread.
You can then decide how to properly balance these requests via worker threads or specialized threads. If you still feel you're running out of resources, chances are you need to chunk up the action into smaller pieces (instead of one 2 minute action, make it four 30 second actions if possible).
As mentioned above, use thread pooling, connection pooling and asynchronous operations where possible - there's no point in a thread idling while waiting for some external resource.
Have you implemented a queueing / backlogging system for requests?
Have you parallelised components of the request that are able to run independently?
Are you using a language / framework / environment in which concurrency is easy to implement and manage (i.e., C#, Kotlin, NodeJS, Go..)?
Why does your business logic take 2 minutes to complete? What portion of each request is spent idle / blocked compared to performing actual processing?
I'm curious as to the domain of this problem; I wonder whether it's the case that the code already is already optimal and the problem is just resource intensive by nature, or whether there are components of the code that are able to execute independently (and concurrently) that aren't currently doing so.
« Previous Thread | Next Thread » |
Thread Information |
Users Browsing this ThreadThere are currently 1 users browsing this thread. (0 members and 1 guests) |