Thread: Chasing the optimal networking implementation

Page 1 of 6 123 ... LastLast
Results 1 to 10 of 51
  1. #1 Chasing the optimal networking implementation 


    Major's Avatar
    Join Date
    Jan 2011
    Posts
    2,997
    Thanks given
    1,293
    Thanks received
    3,556
    Rep Power
    5000
    This topic was raised quite frequently back in 2011, but hasn't really been mentioned since (perhaps because it's been done to death). Regardless, it's both interesting and useful to get input from others - particularly because users who contributed back in 2011 may have different/better ideas now, and because users who didn't have a clue back in 2011 may have some idea now.

    The below was initially written as a response to a private message (but ended up being too long), which is why it's written more as an instruction than a discussion (although the latter is what I am intending here):

    Use Netty. Despite what blakeman was parading around here four years ago, using "raw" NIO is not the only sane choice. Netty has a very superb abstraction layer using the interceptor pattern, and I have yet to see a better approach to writing network code in an OO language. You can use plain NIO if you wish, but you will almost certainly be writing worse code than Netty to do the same thing (which is therefore not only repeating code that already exists, but repeating it in a worse manner). This isn't intended to be an attack on you directly, but a general comment, as there are few (if any) users on here who could write a better networking library than Trustin Lee (because he has a very large amount of experience; as the original author of MINA, he knows what works and what doesn't).

    Separate your data serialization from your processed data: in other words, the decoded data in a frame should be in a separate object (and class) than the decoder itself. This is the pattern used by parabolika (who first showed it publicly afaik, but not on r-s), and also used in apollo and scape-emu (i.e. first released in a public server on r-s by Graham), but super_, saifix, Method, and some users from mopar have also described this idea both publicly and privately. Separating serialization (i.e. data-related code) from the rest of your code is pretty much always a good idea.

    The above also means your codec (i.e. serialization) classes can be written very nicely - not at all. By separating frame output (the decoded data, called a Message in apollo in scape-emu) from frame input (i.e. the raw bytes), there is only one place your codec classes will be referenced, which is in whatever handler class is used to decode data sent along the game port (all of the other code will use the Message implementations instead). This means you can build the encoders/decoders at compile time (pretty odd) or at runtime (better). There are two ways to do this (perhaps more, but these are the two I've thought of) - both of these approaches require the format of the frame be stored in some manner (a flat file is best), as would any other approach.

    The first is using reflection. This is marginally easier, but the code is significantly uglier and slower (I implemented this in a version of apollo a little over a year ago). I would strongly not recommend this approach, but it is an option. Also, I do mean significantly slower - reflection code is harder for the JIT to optimise, and lots of autoboxing is required. This approach sacrifices start-up time for running time, which is almost always a terrible decision.

    The second (and considerably better) approach is to generate the codec classes on startup using Objectweb ASM. The code for this is much nicer, there will be no performance impact (aside from slightly slower start-up times, which is effectively irrelevant), and the output can be optimised by the JIT just fine. This approach sacrifices running time for start-up time, and as such is by far the better one.

    The actual configuration format is pretty debatable, and a custom DSL is almost certainly the way you want to go. It should be easy to parse (this would be a good introduction to compile front-ends, if you are unfamiliar). When I wrote the version of apollo that used reflection, I went with XML (primarily because I think writing JSON by hand is pretty awful, and apollo had an XML parser) - the result was very unpleasant. I don't think that storing the config in another format (e.g. JSON or TOML) would be a significant improvement either, because what you are trying to express isn't simple data. This led me to the idea (also independently conceived by Graham for his game) of using a simple DSL which looks similar to C structs.
    This is nice to parse, concise, simple, and will be familiar to anyone with the most basic experience in C, C++, Rust, Nim, D (or other languages that support such structs), or easily picked up by anyone who is not. This approach is also nice because it makes expressing variable length frames nice; you can see in my original link that I attempted to add support for variable length frames using "compound attributes" (see the Walk frame definition). This is pretty gross (and the code to deal with it, because it used reflection, was even worse). Expressing it in a structural pattern is much nicer:

    Code:
    Walk decoder {
        initial_x: u16
        n times:
            first_direction: i8
            second_direction: i8
        initial_y: u16
        run: boolean
    }
    (N.b. the above is a quick example: I am unsure about the usage of the off-side rule, as well as the syntax for expressing a loop, but I haven't put much thought into the syntax yet. The above does also not include endianness/data transformation).


    Credit for the codec generation idea goes to Graham, super_, saifix, Sean, and was also done by Method (among others). Notably Graham, but also Scu11 had a direct input into some of the other ideas I've listed here.


    Please comment, critique, improve, suggest, question, or otherwise respond to this post - I have no doubt it is very possible to improve the ideas listed here.
    Last edited by Major; 03-20-2015 at 04:06 AM.
    Reply With Quote  
     

  2. Thankful users:


  3. #2  


    Major's Avatar
    Join Date
    Jan 2011
    Posts
    2,997
    Thanks given
    1,293
    Thanks received
    3,556
    Rep Power
    5000
    The above post was getting a little long, so here I wanted to expand on the above by explaining why the struct above contains no information about obfuscation of the data (that is, the endianness, the transformation, and the order of the actual data). This requires some knowledge of knowledge of older client builds (194-377):

    In all client builds, the frame opcodes are obfuscated (i.e. changed) from their original values, to make creating bots harder. As many of you will be aware, there is an array of integers in the same class as the array containing the frame length types (in client build 317 this is that array). As perhaps less will be aware, this is actually a simple map of obfuscated frame opcodes to their unobfuscated variants (this was first made aware to me on this thread by Sinisoul). Slightly differently to what that thread suggests however, this array is only a mapping for the upstream (i.e. client -> server) frame opcodes: it does not include downstream opcodes (or if it does, some transformation is applied to them first - I haven't had time to investigate this).

    This leads to the concept of a two-stage pipeline (originally suggested to me by Ryley): the first (revision-dependent) stage removes the obfuscation, and the second (revision-independent) stage decodes the now-unobfuscated data. This means there now only needs to be one decoder for the actual frame, which is used across all revision, and each revision should have code to remove the revision-dependent obfuscation.
    This idea can be coupled with the idea of runtime codec generation to make the networking implementation fully modular - changing revisions is now a simple drag-and-drop operation (as envisioned by Graham in apollo), but without the boilerplate of having ~200 codec classes. This brings additional (but solvable) problems such as how to approach content, but they are out of scope for this thread - if you wish to discuss how to solve this problem, I encourage you to make your own thread where we can all discuss that :-)

    I am told that the mapping array is present in 400+ clients, but as constants instead of an array (although I have yet to investigate this myself, and I question the likelyhood of it). Even if they do exist, that information is unfortunately useless - for later revisions (and also for downstream frames), such opcodes must be chosen by us, as we have no knowledge of the unobfuscated ones JaGex use (although we can follow a similar pattern as they did - for example, the unobfuscated opcodes for the 5 object action frames are 120-124).
    Last edited by Major; 03-20-2015 at 04:27 AM.
    Reply With Quote  
     

  4. Thankful user:


  5. #3  
    fumant viriditas quotidiana

    saifix's Avatar
    Join Date
    Feb 2009
    Age
    30
    Posts
    1,237
    Thanks given
    275
    Thanks received
    957
    Rep Power
    3304
    Embedded structures are much nicer than your for-loop syntax:
    Walk decoder {
    initial_x: u16
    n times:
    first_direction: i8
    second_direction: i8
    initial_y: u16
    run: boolean
    }
    Code:
    <message-decoder name="walking">
        <attribute name="initial_x" type="u16" />
        <embedded-object-array size="n"> <!-- where does n come from u crack pot that syntax is weird -->
            <attribute name="first_direction" type="i8" />
            <attribute name="second_direction" type="i8" />
        </embedded-object-array>        
        <attribute name="initial_y" type="u16" />
        <attribute name="run" type="boolean" />
    </message-decoder>
    The added bonus is that you can represent it in most configuration formats very easily.
    "Im so bluezd out of my box.
    Im so fkd i canr even sens makeas Smoke blunt 420hash e tizday" - A legendary guy (1993 - 2015)
    Quote Originally Posted by nmopal View Post
    I will be creating a grimy dubstep song using these lyrics and vocaloid please prepare your bodies
    Reply With Quote  
     

  6. #4  


    Major's Avatar
    Join Date
    Jan 2011
    Posts
    2,997
    Thanks given
    1,293
    Thanks received
    3,556
    Rep Power
    5000
    Quote Originally Posted by saifix View Post
    Embedded structures are much nicer than your for-loop syntax:

    Code:
    <message-decoder name="walking">
        <attribute name="initial_x" type="u16" />
        <embedded-object-array size="n"> <!-- where does n come from u crack pot that syntax is weird -->
            <attribute name="first_direction" type="i8" />
            <attribute name="second_direction" type="i8" />
        </embedded-object-array>        
        <attribute name="initial_y" type="u16" />
        <attribute name="run" type="boolean" />
    </message-decoder>
    The added bonus is that you can represent it in most configuration formats very easily.
    Did u even click my links mate. Here's how I originally did it

    Code:
    <decoder opcode="248, 164, 98" name="Walk">
    	<attribute type="short" order="little" transformation="add" placement="1">x</attribute>
    	<compound-attribute name="Walk" skip="248" placement="0">
    		<attribute type="byte" signed="true">first direction</attribute>
    		<attribute type="byte" signed="true">second direction</attribute>
    	</compound-attribute>
    	<attribute type="short" order="little" placement="2">y</attribute>
    	<attribute type="boolean" transformation="negate" placement="3">run</attribute>
    </decoder>
    XML makes this look pretty horrendous imo.
    Reply With Quote  
     

  7. Thankful user:


  8. #5  
    fumant viriditas quotidiana

    saifix's Avatar
    Join Date
    Feb 2009
    Age
    30
    Posts
    1,237
    Thanks given
    275
    Thanks received
    957
    Rep Power
    3304
    Quote Originally Posted by Major View Post
    Did u even click my links mate. Here's how I originally did it

    Code:
    <decoder opcode="248, 164, 98" name="Walk">
    	<attribute type="short" order="little" transformation="add" placement="1">x</attribute>
    	<compound-attribute name="Walk" skip="248" placement="0">
    		<attribute type="byte" signed="true">first direction</attribute>
    		<attribute type="byte" signed="true">second direction</attribute>
    	</compound-attribute>
    	<attribute type="short" order="little" placement="2">y</attribute>
    	<attribute type="boolean" transformation="negate" placement="3">run</attribute>
    </decoder>
    XML makes this look pretty horrendous imo.
    no mate didnt read the thread just saw some shit for loop syntax mate soz

    see The added bonus is that you can represent it in most configuration formats very easily.

    edit: the same idea could be applied to implementing a DSL with your language of choice. That for loop makes me cringe.

    edit2: protobuf syntax isn't bad

    Code:
    // See README.txt for information and build instructions.
    
    package tutorial;
    
    option java_package = "com.example.tutorial";
    option java_outer_classname = "AddressBookProtos";
    
    message Person {
      required string name = 1;
      required int32 id = 2;        // Unique ID number for this person.
      optional string email = 3;
    
      enum PhoneType {
        MOBILE = 0;
        HOME = 1;
        WORK = 2;
      }
    
      message PhoneNumber {
        required string number = 1;
        optional PhoneType type = 2 [default = HOME];
      }
    
      repeated PhoneNumber phone = 4;
    }
    
    // Our address book file is just one of these.
    message AddressBook {
      repeated Person person = 1;
    }
    "Im so bluezd out of my box.
    Im so fkd i canr even sens makeas Smoke blunt 420hash e tizday" - A legendary guy (1993 - 2015)
    Quote Originally Posted by nmopal View Post
    I will be creating a grimy dubstep song using these lyrics and vocaloid please prepare your bodies
    Reply With Quote  
     

  9. #6  


    Major's Avatar
    Join Date
    Jan 2011
    Posts
    2,997
    Thanks given
    1,293
    Thanks received
    3,556
    Rep Power
    5000
    Quote Originally Posted by saifix View Post
    no mate didnt read the thread just saw some shit for loop syntax mate soz

    see The added bonus is that you can represent it in most configuration formats very easily.

    edit: the same idea could be applied to implementing a DSL with your language of choice. That for loop makes me cringe.

    edit2: protobuf syntax isn't bad

    Code:
    // See README.txt for information and build instructions.
    
    package tutorial;
    
    option java_package = "com.example.tutorial";
    option java_outer_classname = "AddressBookProtos";
    
    message Person {
      required string name = 1;
      required int32 id = 2;        // Unique ID number for this person.
      optional string email = 3;
    
      enum PhoneType {
        MOBILE = 0;
        HOME = 1;
        WORK = 2;
      }
    
      message PhoneNumber {
        required string number = 1;
        optional PhoneType type = 2 [default = HOME];
      }
    
      repeated PhoneNumber phone = 4;
    }
    
    // Our address book file is just one of these.
    message AddressBook {
      repeated Person person = 1;
    }
    mate i wrote it in the reply box as an example!!!

    Yeah Protobuf is nice, although it is superseded by Cap'n Proto - using something similar to their schema is an interesting idea.

    Code:
    struct Person {
      name @0 :Text;
      birthdate @3 :Date;
    
      email @1 :Text;
      phones @2 :List(PhoneNumber);
    
      struct PhoneNumber {
        number @0 :Text;
        type @1 :Type;
    
        enum Type {
          mobile @0;
          home @1;
          work @2;
        }
      }
    }
    
    struct Date {
      year @0 :Int16;
      month @1 :UInt8;
      day @2 :UInt8;
    }
    Last edited by Major; 03-20-2015 at 04:29 AM.
    Reply With Quote  
     

  10. #7  
    Renown Programmer

    Sean's Avatar
    Join Date
    May 2007
    Age
    32
    Posts
    2,757
    Thanks given
    264
    Thanks received
    1,090
    Rep Power
    4393
    For the generation of the actual codecs I came across this library ByteBuddy, Byte Buddy - runtime code generation for Java and It could be quite useful to implement these types of ideas. Also I personally think XML or JSON is not the way to go.

    Now if we jump into what Gary has just spoken about, it comes apparent that using XML or JSON is not the best approach for such a thing where we require to generate logic for our codecs, below is an example of what the XML would possibly look like if we went down this route and used it for housing
    our conditions required. Code is taken from https://github.com/apollo-rsps/apoll...geEncoder.java

    Code:
    <decoder opcode="id" name="item_update">
    		<loop attribute="message.getSlottedItems">
    		<attribute type="smart">slottedItems.getSlot</attribute>
    		<attribute type="short" increment="1">item.id</attribute>
    		<ifelse>
    			<if condition=">" value="255">
    				<attribute type="byte">255</attribute>
    				<attribute type="int">item.amount</attribute>
    			</if>
    			<else>
    				<attribute type="byte">item.amount</attribute>
    			</else>
    		<ifelse>
    		</loop>	
    	</decoder>
    As you can see it will get very messy and implementing this would be a complete pain in the ass, Gary has the right idea because we would need to implement something like how ProtoBuf does it.
    Reply With Quote  
     

  11. Thankful user:


  12. #8  
    Registered Member
    Cadillac's Avatar
    Join Date
    Jul 2014
    Age
    9
    Posts
    336
    Thanks given
    0
    Thanks received
    228
    Rep Power
    951
    Could you possibly provide examples of

    Separate your data serialization from your data construction: in other words, the decoded data in a frame should be in a separate object (and class) than the decoder itself. This is the pattern used by parabolika (who first showed it publicly afaik, but not on r-s), and also used in apollo and scape-emu (i.e. first released in a public server on r-s by Graham), but super_, saifix, Method, and some users from mopar have also described this idea both publicly and privately. Separating serialization (i.e. data-related code) from the rest of your code is pretty much always a good idea.
    & |

    The second (and considerably better) approach is to generate the codec classes on startup using Objectweb ASM. The code for this is much nicer, there will be no performance impact (aside from slightly slower start-up times, which is effectively irrelevant), and the output can be optimised by the JIT just fine. This approach sacrifices running time for start-up time, and as such is by far the better one.
    Quote Originally Posted by Major View Post
    Data (de)serialization
    Processed data (which is what I meant instead of "data construction".

    The second one I don't have any examples for.
    Oh so what you're basically suggesting is to have an API construct these structured classes at runtime for packets data and configure them via a data storage method IE gson, xml?

    Quote Originally Posted by Major View Post
    I am only suggesting generating the serialization classes at runtime, but otherwise yes (again, this was not my idea originally and credit should go to Sean, saifix, Graham, super_, and whoever else I listed in the OP).
    So how would you go about storing the methods for encoding / decoding btw?

    I believe the method for adding a short to the buffer would be buffer.put(xxx); in netty for example.
    Last edited by Major; 03-22-2015 at 12:55 AM. Reason: Merge
    back at it.
    Reply With Quote  
     

  13. #9  


    Major's Avatar
    Join Date
    Jan 2011
    Posts
    2,997
    Thanks given
    1,293
    Thanks received
    3,556
    Rep Power
    5000
    Quote Originally Posted by Cadillac View Post
    Could you possibly provide examples of
    Data (de)serialization
    Processed data (which is what I meant instead of "data construction".

    The second one I don't have any examples for.

    Quote Originally Posted by Cadillac View Post
    Oh so what you're basically suggesting is to have an API construct these structured classes at runtime for packets data and configure them via a data storage method IE gson, xml?
    I am only suggesting generating the serialization classes at runtime, but otherwise yes (again, this was not my idea originally and credit should go to Sean, saifix, Graham, super_, and whoever else I listed in the OP).

    Quote Originally Posted by Cadillac View Post
    So how would you go about storing the methods for encoding / decoding btw?

    I believe the method for adding a short to the buffer would be buffer.put(xxx); in netty for example.
    A simple Map of Strings in the codec generation class would suffice (assuming you are asking how one would convert from e.g. the config file type "i16" to reading a signed short).

    Edit: We're writing bytecode, so the values would just be a method specification, e.g. "org/apollo/net/codec/game/GamePacketReader/getSigned(Lorg/apollo/net/codec/game/DataType;)J"

    We're getting a little off topic here though.
    Last edited by Major; 03-22-2015 at 12:55 AM.
    Reply With Quote  
     

  14. #10  
    Renown Programmer

    Sean's Avatar
    Join Date
    May 2007
    Age
    32
    Posts
    2,757
    Thanks given
    264
    Thanks received
    1,090
    Rep Power
    4393
    Oh forgot to point out something out on my earlier post because I was about to go to bed before I posted.

    I would properly not do this at Runtime but yet every time I update the protocol of the client I am using. I would prefer to just generate the classes and such once, store them into a jar and inform my server to load them as a type of Plugin. The cool thing about what is proposed in this thread is
    if we as a community built a Client Analyser for example that was intended to identify certain packets within the RS Client we could generate an output file in a way to allow us to build the packets and implement them into our server a lot damn easier than how we currently go about it.
    Reply With Quote  
     

  15. Thankful user:


Page 1 of 6 123 ... LastLast

Thread Information
Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)


User Tag List

Similar Threads

  1. The Utral Network!
    By Pehpero in forum General
    Replies: 4
    Last Post: 05-08-2011, 09:47 PM
  2. Replies: 84
    Last Post: 04-13-2011, 07:00 AM
  3. The Social Network
    By Saint in forum Chat
    Replies: 2
    Last Post: 08-31-2010, 04:46 AM
Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •