Thread: I have a multi-threading problem!

Page 1 of 2 12 LastLast
Results 1 to 10 of 14
  1. #1 I have a multi-threading problem! 
    Registered Member
    Join Date
    Dec 2013
    Posts
    419
    Thanks given
    127
    Thanks received
    85
    Rep Power
    349
    Hey guys,

    I have a client/server application. Clients make requests to the server and the server responds with a result.

    The problem

    A request that a client makes can take up to 2 minutes to serve. This is because the operations are heavy and blocking. The operations are also unique and the results cannot be cached.
    To tackle this, multi-threading is the best option. The operation gets done on it's own thread.

    Problem A
    The server receives around 100 requests every couple of seconds so I cannot create a new thread as I'd end up with hundreds. Although is work, it's not a scaleable solution and in general this is frowned upon.

    Problem B
    To tackle problem A - i have tried to use a Thread Pool which is what most of you guys would recommend. The issue I have with this is that because the operations take a long time, the pool runs out of threads.
    As mentioned, the server receives around 100 requests every couple of seconds. Your average thread pool size is based upon CPU cores so let's pretend 4 (quad core) - The pool gives 4 threads, so that's 4 operations being processed but 96 requests waiting for a thread to become available and I end up getting blocked by the thread pool itself.

    Any suggestions on what I should do? How can I serve all the requests without blocking my program and at a reasonable time?

    Thanks
    Reply With Quote  
     

  2. #2  
    Donator

    Jason's Avatar
    Join Date
    Aug 2009
    Posts
    6,092
    Thanks given
    2,402
    Thanks received
    2,823
    Rep Power
    4550
    If you provide a SSCCE that would be fantastic. A good place to start would be to reduce that 2 minute/task time to < 5-10 seconds if possible.
    Reply With Quote  
     

  3. #3  
    Registered Member

    Join Date
    Oct 2014
    Posts
    192
    Thanks given
    31
    Thanks received
    15
    Rep Power
    81
    Quote Originally Posted by Kiissmyswagb View Post
    The pool gives 4 threads, so that's 4 operations being processed but 96 requests waiting for a thread to become available and I end up getting blocked by the thread pool itself.
    So just run the threadpool itself in a seperate thread..?
    Then in your main thread you can keep accepting requests and stacking up tasks for the pool.

    but obviously if each task takes 2 whole minutes to compute and you are getting 100 requests a second - you will need to take a break with accepting requests at some point or you will end up with too many pent up tasks.

    Also can i ask what it is that requires something that takes 2 minutes to compute to be done 100 times a second?
    Reply With Quote  
     

  4. #4  
    Extreme Donator

    nbness2's Avatar
    Join Date
    Aug 2011
    Posts
    692
    Thanks given
    274
    Thanks received
    139
    Rep Power
    430
    Quote Originally Posted by stav5 View Post
    So just run the threadpool itself in a seperate thread..?
    Then in your main thread you can keep accepting requests and stacking up tasks for the pool.

    but obviously if each task takes 2 whole minutes to compute and you are getting 100 requests a second - you will need to take a break with accepting requests at some point or you will end up with too many pent up tasks.

    Also can i ask what it is that requires something that takes 2 minutes to compute to be done 100 times a second?
    If each task is taking 2 minutes to compute but you are getting 100 task requests a second, you definitely need to look in to if there are any bottlenecks in your code.
    You also need to definitely look in to Asynchronous programming, this lets your computation(s) not block the rest of your program.
    With java, this would be... idk RXJava? Python uses its own Async stdlib. Kotlin just got Coroutines with stable status, C# uses its own async\await stuff.
    Even with only 4 cores, you can have many many more software threads going. For example, as of writing this comment my 4c laptop has ~1900 software threads running.

    Also what in the world is taking 2 minutes to compute? Maybe if you posted an example of the code we could possibly help identify flaws that would speed up your code? Recently had a classmate refactor and optimize his class project to run in 5 minutes down to 15 seconds.
    KT/JAVA - NBX 637 - HERE!
    KT - Drop table 4: Flexible, Powerful - HERE!
    KT - Command: Simplify writing commands - HERE
    KT - NbUtil: Make your kotlin easier - HERE
    KT - Hopping Islands: From Java to Kotlin - P1 - P2 - P3 - P4 - P5
    Reply With Quote  
     

  5. #5  
    Registered Member

    Join Date
    Jul 2013
    Posts
    119
    Thanks given
    12
    Thanks received
    50
    Rep Power
    83
    Use a non-blocking system. Java provides non-blocking IO through their NIO (New IO) package.

    Frameworks such as Netty are built ontop of this, as it allows a single thread to handle requests from different clients by polling a Selector to see which events have came in (read? write?). This will allow you to handle multiple connections from a single thread.

    You can then decide how to properly balance these requests via worker threads or specialized threads. If you still feel you're running out of resources, chances are you need to chunk up the action into smaller pieces (instead of one 2 minute action, make it four 30 second actions if possible).
    Reply With Quote  
     

  6. #6  
    Registered Member
    hc747's Avatar
    Join Date
    Dec 2013
    Age
    26
    Posts
    1,474
    Thanks given
    3,312
    Thanks received
    691
    Rep Power
    1098
    As mentioned above, use thread pooling, connection pooling and asynchronous operations where possible - there's no point in a thread idling while waiting for some external resource.

    Have you implemented a queueing / backlogging system for requests?
    Have you parallelised components of the request that are able to run independently?
    Are you using a language / framework / environment in which concurrency is easy to implement and manage (i.e., C#, Kotlin, NodeJS, Go..)?
    Why does your business logic take 2 minutes to complete? What portion of each request is spent idle / blocked compared to performing actual processing?
    Reply With Quote  
     

  7. #7  
    Registered Member
    hc747's Avatar
    Join Date
    Dec 2013
    Age
    26
    Posts
    1,474
    Thanks given
    3,312
    Thanks received
    691
    Rep Power
    1098
    Quote Originally Posted by Kiissmyswagb View Post
    Hey guys,

    I have a client/server application. Clients make requests to the server and the server responds with a result.

    The problem

    A request that a client makes can take up to 2 minutes to serve. This is because the operations are heavy and blocking. The operations are also unique and the results cannot be cached.
    To tackle this, multi-threading is the best option. The operation gets done on it's own thread.

    Problem A
    The server receives around 100 requests every couple of seconds so I cannot create a new thread as I'd end up with hundreds. Although is work, it's not a scaleable solution and in general this is frowned upon.

    Problem B
    To tackle problem A - i have tried to use a Thread Pool which is what most of you guys would recommend. The issue I have with this is that because the operations take a long time, the pool runs out of threads.
    As mentioned, the server receives around 100 requests every couple of seconds. Your average thread pool size is based upon CPU cores so let's pretend 4 (quad core) - The pool gives 4 threads, so that's 4 operations being processed but 96 requests waiting for a thread to become available and I end up getting blocked by the thread pool itself.

    Any suggestions on what I should do? How can I serve all the requests without blocking my program and at a reasonable time?

    Thanks
    Curious as to whether or not you managed to resolve this
    Reply With Quote  
     

  8. #8  
    Respected Member


    Join Date
    Jan 2009
    Posts
    5,743
    Thanks given
    1,162
    Thanks received
    3,603
    Rep Power
    5000
    Quote Originally Posted by hc747 View Post
    Curious as to whether or not you managed to resolve this
    No code could solve this, say we have 16 threads and this is an intensive task requiring all cpu. The first 2 minutes of requests would result in hours for a response... obviously if it’s not cpu or io taxing then we can scale the thread count, still it’d be hours...
    Reply With Quote  
     

  9. Thankful user:


  10. #9  
    Registered Member
    hc747's Avatar
    Join Date
    Dec 2013
    Age
    26
    Posts
    1,474
    Thanks given
    3,312
    Thanks received
    691
    Rep Power
    1098
    Quote Originally Posted by Spooky View Post
    No code could solve this, say we have 16 threads and this is an intensive task requiring all cpu. The first 2 minutes of requests would result in hours for a response... obviously if it’s not cpu or io taxing then we can scale the thread count, still it’d be hours...
    I'm curious as to the domain of this problem; I wonder whether it's the case that the code already is already optimal and the problem is just resource intensive by nature, or whether there are components of the code that are able to execute independently (and concurrently) that aren't currently doing so.
    Reply With Quote  
     

  11. #10  
    Registered Member
    Join Date
    Jan 2019
    Posts
    4
    Thanks given
    0
    Thanks received
    0
    Rep Power
    0
    Quote Originally Posted by stav5 View Post
    So just run the threadpool itself in a seperate thread..?
    Then in your main thread you can keep accepting requests and stacking up tasks for the pool.

    but obviously if each task takes 2 whole minutes to compute and you are getting 100 requests a second - you will need to take a break with accepting requests at some point or you will end up with too many pent up tasks.

    Also can i ask what it is that requires something that takes 2 minutes to compute to be done 100 times a second?
    I generally recommend running all of ur thread pools in a separate thread inside of a thread pool

    e.g all of my thread pools are run inside of a thread in another thread pool giving me access to more threads it works great for me
    Reply With Quote  
     

Page 1 of 2 12 LastLast

Thread Information
Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)


User Tag List

Similar Threads

  1. MULTI BARRAGE PROBLEM :s
    By Trock in forum Help
    Replies: 3
    Last Post: 09-05-2009, 09:48 PM
  2. Multi Zone Problem [525]
    By Nikki in forum Help
    Replies: 2
    Last Post: 05-06-2009, 04:58 AM
  3. Replies: 8
    Last Post: 04-16-2009, 03:47 PM
  4. About new Rs2e - multi thread
    By Meanz in forum RS 503+ Client & Server
    Replies: 27
    Last Post: 10-29-2008, 12:55 PM
  5. Another multi threaded tut
    By life is hopeless in forum Tutorials
    Replies: 5
    Last Post: 08-24-2007, 10:44 PM
Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •