Thread: Multi node server design, thoughts?

Page 1 of 2 12 LastLast
Results 1 to 10 of 13
  1. #1 Multi node server design, thoughts? 
    Registered Member
    Join Date
    Dec 2013
    Posts
    419
    Thanks given
    127
    Thanks received
    85
    Rep Power
    349
    I've been looking into a multi server design to serve content to my end users via REST over TCP.

    Here's my current design plan:

    Node 0 Centeral
    Node 1 (USA)
    Node 2 (Asia)
    Node 3 (EU)

    Each node, knows about each other within the system.

    The way I'm designing is that users connect to their best node.
    This is done by getting a list of online nodes from central and determining which is best via ping.
    Each node, has its own DB that it reads/writes to. After each write, it will tell the central to update its own DB. The central then notifies any other node of changes.

    If a node goes down, the clients can just connect to the next best node and service continues uninterrupted for the end user.

    If the central node goes down, each node switches to ad-hoc mode. So if node 1 does a write, it will inform node 2,3 of changes. If a node during ad-hoc goes down, it will double check with neighbouring node, then on confirmation, each node will update their internal list of available nodes.

    Once the central is back up, it will sync all the changes and normal operation is followed.

    The point of this is to ensure 100% availability.

    Could I have some suggestions for improvements and criticism or this is an acceptable approach to implement?
    Reply With Quote  
     

  2. #2  
    Bossman

    ISAI's Avatar
    Join Date
    Sep 2012
    Posts
    1,916
    Thanks given
    655
    Thanks received
    1,366
    Rep Power
    5000
    Your delay between node updates is going to be astronomical. Possibly something like a CDN which tells all nodes to update at the same time would work.

    This could also introduce dupes pretty easily if you're not careful.
    Reply With Quote  
     

  3. #3  
    Registered Member
    Join Date
    Dec 2013
    Posts
    419
    Thanks given
    127
    Thanks received
    85
    Rep Power
    349
    Quote Originally Posted by ISAI View Post
    Your delay between node updates is going to be astronomical. Possibly something like a CDN which tells all nodes to update at the same time would work.

    This could also introduce dupes pretty easily if you're not careful.
    The delay you're talking about only happens in ad-hoc mode. As each node will keep requesting live data from node 0 until it goes down.
    By current design, updates will be pushed immediately to all nodes as data is changed.

    A duplication would occur if node x goes down before it could inform all other nodes of changes whilst in ad-hoc mode. Though this will be fixed during sync (i.e. keep the latest changes).
    If a user makes a purchase via node x and that goes down (or entirely loses its data before syncing), when they switch to the next available node, their account is rolled back as the changes didn't persist. In real time, this would all have to happen seconds apart. Losing 10 seconds of your progress would be the trade-off here, so what other solution would you suggest?
    Reply With Quote  
     

  4. #4  
    Bossman

    ISAI's Avatar
    Join Date
    Sep 2012
    Posts
    1,916
    Thanks given
    655
    Thanks received
    1,366
    Rep Power
    5000
    Quote Originally Posted by Kiissmyswagb View Post
    The delay you're talking about only happens in ad-hoc mode. As each node will keep requesting live data from node 0 until it goes down.
    By current design, updates will be pushed immediately to all nodes as data is changed.

    A duplication would occur if node x goes down before it could inform all other nodes of changes whilst in ad-hoc mode. Though this will be fixed during sync (i.e. keep the latest changes).
    If a user makes a purchase via node x and that goes down (or entirely loses its data before syncing), when they switch to the next available node, their account is rolled back as the changes didn't persist. In real time, this would all have to happen seconds apart. Losing 10 seconds of your progress would be the trade-off here, so what other solution would you suggest?
    Correct me if I'm wrong, but you've gotta think about it from the client perspective;
    - Client 1 connects to Node 1
    - Client 2 connects to Node 3
    - Node 0 syncs with 1 & 3
    - Client 1 trades Client 2

    Trade hops are Client 1 -> Node 1 -> Node 0 -> Node 3 -> Client 2

    That's a lot of hops.
    Reply With Quote  
     

  5. #5  
    Registered Member
    Join Date
    Dec 2013
    Posts
    419
    Thanks given
    127
    Thanks received
    85
    Rep Power
    349
    Quote Originally Posted by ISAI View Post
    Correct me if I'm wrong, but you've gotta think about it from the client perspective;
    - Client 1 connects to Node 1
    - Client 2 connects to Node 3
    - Node 0 syncs with 1 & 3
    - Client 1 trades Client 2

    Trade hops are Client 1 -> Node 1 -> Node 0 -> Node 3 -> Client 2

    That's a lot of hops.
    This isn't for real-time applications but more for requesting data and retrieving it.

    If this was in an RSPS scenario, then each node would act as your "world" and the data that is exchanged between nodes are 1-way. Not bi-directional communication.
    Reply With Quote  
     

  6. #6  
    Bossman

    ISAI's Avatar
    Join Date
    Sep 2012
    Posts
    1,916
    Thanks given
    655
    Thanks received
    1,366
    Rep Power
    5000
    Quote Originally Posted by Kiissmyswagb View Post
    This isn't for real-time applications but more for requesting data and retrieving it.

    If this was in an RSPS scenario, then each node would act as your "world" and the data that is exchanged between nodes are 1-way. Not bi-directional communication.
    Oh! My apologies, I thought this was related to RSPS.

    If it isn't real-time, this is a decent topology.
    Reply With Quote  
     

  7. #7  
    Retired. Stop PMing me.


    Galkon's Avatar
    Join Date
    Nov 2007
    Age
    17
    Posts
    7,526
    Thanks given
    1,805
    Thanks received
    2,830
    Rep Power
    5000
    Seems overkill. Cloudflare load balancers can direct users to appropriate nodes for example. Depending on how many users you are trying to serve per region and the type of content being served, it's probably not even worth multiple nodes. Depends on if the content you are serving is static or dynamic. But if you do need regional coverage, 1 main db and a few read replicas across regions would do the trick.
    Attached image
    Reply With Quote  
     

  8. Thankful user:


  9. #8  
    Registered Member
    Join Date
    Dec 2013
    Posts
    419
    Thanks given
    127
    Thanks received
    85
    Rep Power
    349
    Quote Originally Posted by Galkon View Post
    Seems overkill. Cloudflare load balancers can direct users to appropriate nodes for example. Depending on how many users you are trying to serve per region and the type of content being served, it's probably not even worth multiple nodes. Depends on if the content you are serving is static or dynamic. But if you do need regional coverage, 1 main db and a few read replicas across regions would do the trick.
    There's not really much of a functional difference here from what cloudflare would do, though here the end client would pick the node of choice based on some deterministic value (i.e. ping in my case).

    The issue I don't want to have is that main DB going down causing service to everything to stop working.
    The OP was just one scenario I came up with. Another similar may be allowing nodes to be promoted if the current master goes down. In the OP case, if the central goes down, somebody else can take over without the service being disrupted.
    Another scenario was allowing nodes to spawn other nodes (e.g. calling AWS API to quickly spin up containers). In both cases, it would be fully ad-hoc and standalone once the initial discovery process is done.

    I'm not sure how much of an overkill this is though because they're extremely easy to program and setup.
    Reply With Quote  
     

  10. #9  
    Retired. Stop PMing me.


    Galkon's Avatar
    Join Date
    Nov 2007
    Age
    17
    Posts
    7,526
    Thanks given
    1,805
    Thanks received
    2,830
    Rep Power
    5000
    Quote Originally Posted by Kiissmyswagb View Post
    There's not really much of a functional difference here from what cloudflare would do, though here the end client would pick the node of choice based on some deterministic value (i.e. ping in my case).

    The issue I don't want to have is that main DB going down causing service to everything to stop working.
    The OP was just one scenario I came up with. Another similar may be allowing nodes to be promoted if the current master goes down. In the OP case, if the central goes down, somebody else can take over without the service being disrupted.
    Another scenario was allowing nodes to spawn other nodes (e.g. calling AWS API to quickly spin up containers). In both cases, it would be fully ad-hoc and standalone once the initial discovery process is done.

    I'm not sure how much of an overkill this is though because they're extremely easy to program and setup.
    A good practice is for the client to do as little as possible. If you depend on the client correctly pinging and choosing which server to connect to, you cannot control any of that logic until you update the client again and anyone using it updates. Maybe this is fine for a website, but if you put it behind a load balancer and the load balancer can be configured regardless of client version, you'd have more control.

    If you main DB goes down (let's say it's where you execute writes) and you have 2-3 read replicas, your site would still function 100% normally at a read-only level. Only when writes needed to go through would users experience issues.

    While I understand optimizing for being able to continuously serve your users during downtime, I would be curious why you are optimizing so much for downtime and ability to spin up new nodes and for databases to be regularly failing? It's important to know how to mitigate downtime, but not what you should be designing for imo, you've described a fairly simple service (clients connecting to REST APIs).
    Attached image
    Reply With Quote  
     

  11. Thankful user:


  12. #10  
    Jire
    Jire's Avatar
    Join Date
    Nov 2008
    Posts
    34
    Thanks given
    193
    Thanks received
    70
    Rep Power
    608
    I think this problem would best be tackled by creating a central server which is both a reverse proxy to your node servers, and also handling the actual persistence for player data. Your data would be loaded and persisted on the central node, but each "world" would be on its own node and on logout/disconnect the node would send back the player state to the central server for it to be persisted.

    1. You then can get the benefit of using memory-mapped files for persistence, increasing your I/O performance by many orders of magnitude. On your actual world servers this means that you don't need fast disks (thus cheaper nodes), because you won't be using the disk ever.

    2. Another benefit is since the central server acts as a reverse proxy, you only need to worry about DDoS protecting the central node, and can thus use much cheaper, non-protected servers for your nodes.

    3. Finally, this would potentially allow you to internally simplify your networking protocol (stripping out Jagex-specific protocol stuff), so you can support multi-revisions with codec specific implementation on your actual central server rather than each world. You can also intermix protocols, such as multiple worlds with different revisions.
    Reply With Quote  
     

Page 1 of 2 12 LastLast

Thread Information
Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)


User Tag List

Similar Threads

  1. Replies: 30
    Last Post: 12-04-2010, 05:49 PM
  2. MTP - Multi World Server 24/7
    By marcbob in forum Advertise
    Replies: 23
    Last Post: 05-03-2010, 06:58 AM
  3. First step towards my multi-client server.
    By Darryl in forum Show-off
    Replies: 16
    Last Post: 01-09-2010, 02:09 AM
  4. server designer
    By wizzyt21 in forum Tools
    Replies: 2
    Last Post: 12-14-2009, 03:26 AM
  5. ProgressScape - A Multi-World Server
    By Eleclion in forum Downloads
    Replies: 20
    Last Post: 10-20-2007, 03:07 PM
Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •