Thread: Python Server Framework

Page 4 of 5 FirstFirst ... 2345 LastLast
Results 31 to 40 of 49
  1. #31  
    Rune-Server Affiliate
    Genesis's Avatar
    Join Date
    Sep 2010
    Posts
    4,149
    Thanks given
    1,508
    Thanks received
    1,980
    Rep Power
    4944
    Good luck on the project
    Reply With Quote  
     

  2. Thankful user:


  3. #32  
    Registered Member
    hc747's Avatar
    Join Date
    Dec 2013
    Age
    26
    Posts
    1,474
    Thanks given
    3,312
    Thanks received
    691
    Rep Power
    1098
    Quote Originally Posted by Nbness2 View Post
    Update!

    • Added binary format to itemdefs
      • First binary format iteration reduced file size by 22%
      • Second binary format iteration reduced file size by a further ~25%
      • Total combined reduction of ~42% reduction of 11682 items from the JSON format (1,102,220 bytes -> 648,612 bytes, 453,608 byte reduction).

    • Binary file size reduction comes at a price of increased load times
      • Initial binary file load times were ~23.5 seconds
      • The second iteration reduced binary file load times to ~13.5 seconds.
      • This is compared to less than .1 seconds json load time using the json stdlib. I will keep attempting to optimize this process for the lowest possible item binary size.
      • Note that in a server's case, generally loading data vs saving space is very one sided towards saving space, and I will keep it this way. This is comparable to making room for ~100 more accounts (assuming ~5kb acc size) before buying another disk, or saving another 10 seconds on loading the server

    • I will be keeping the JSON format because it is much more human friendly than binary.


    Binary iteration 1 versus binary iteration 2:
    • Iteration 1 would take any item that had no bonuses and write 12 shorts equal to 0, and was LITTLE ENDIAN (this is why you might look at the lists and wonder why there is a difference when the stats are the same)
    • Iteration 2 would take any item that had no bonuses and write NO (bytes 4e and bytes 4f), and was BIG ENDIAN



    File Sizes


    File Content
    Although I'd recommend to stick to JSON, you could further optimise your binary storage format by only storing non-zero values, whereas currently, if you have the following bonuses:
    { 0, 0, 0, 0, 0, 0, 1, 2 }
    you would be storing 8 shorts, which could and should be optimised to something like this (which would only require storing 2 + 1 shorts).

    The code below could be ported from Java and arbitrarily relies upon Netty's ByteBuf implementation:
    Code:
        private static class ItemDefinition {
    
    
            private final int id;
    
    
            private final String name;
    
    
            private final String description;
    
    
            private final short[] bonuses;
    
    
            public void encode(ByteBuf buffer) {
    
    
                buffer.writeShort(id);
    
    
                buffer.writeShort(name.length());
                buffer.writeCharSequence(name, StandardCharsets.UTF_8);
    
    
                buffer.writeShort(description.length());
                buffer.writeCharSequence(description, StandardCharsets.UTF_8);
    
    
                if (bonuses == null)
                    buffer.writeShort(0);
                else {
    
    
                    short mask = 0;
    
    
                    for (short index = 0; index < bonuses.length; index++) {
                        if (bonuses[index] != 0)
                            mask |= (short) 1 << index;
                    }
    
    
                    buffer.writeShort(mask);
    
    
                    if (mask != 0) {
    
    
                        buffer.writeByte(bonuses.length);
    
    
                        for (short index = 0; index < bonuses.length; index++) {
                            if (bonuses[index] != 0)
                                buffer.writeShort(bonuses[index]);
                        }
    
    
                    }
    
    
                }
    
    
            }
    
    
            ItemDefinition(int id, String name, String description, short[] bonuses) {
                this.id = id;
                this.name = name;
                this.description = description;
                this.bonuses = bonuses;
            }
    
    
            public static ItemDefinitionBuilder builder() {
                return new ItemDefinitionBuilder();
            }
    
    
            public int getId() {
                return id;
            }
    
    
            public String getName() {
                return name;
            }
    
    
            public String getDescription() {
                return description;
            }
    
    
            public short[] getBonuses() {
                return bonuses;
            }
    
    
            public static class ItemDefinitionBuilder {
    
    
                private int id;
    
    
                private String name;
    
    
                private String description;
    
    
                private short[] bonuses;
    
    
                ItemDefinitionBuilder() {}
    
    
                public ItemDefinition.ItemDefinitionBuilder id(int id) {
                    this.id = id;
                    return this;
                }
    
    
                public ItemDefinition.ItemDefinitionBuilder name(String name) {
                    this.name = name;
                    return this;
                }
    
    
                public ItemDefinition.ItemDefinitionBuilder description(String description) {
                    this.description = description;
                    return this;
                }
    
    
                public ItemDefinition.ItemDefinitionBuilder bonuses(short[] bonuses) {
                    this.bonuses = bonuses;
                    return this;
                }
    
    
                public ItemDefinition.ItemDefinitionBuilder decode(ByteBuf buffer) {
    
    
                    id(buffer.readShort());
    
    
                    name(buffer.readCharSequence(buffer.readShort(), StandardCharsets.UTF_8).toString());
    
    
                    description(buffer.readCharSequence(buffer.readShort(), StandardCharsets.UTF_8).toString());
    
    
                    short mask = buffer.readShort();
    
    
                    if (mask != 0) {
    
    
                        bonuses = new short[buffer.readByte()];
    
    
                        for (short index = 0; index < bonuses.length; index++) {
    
    
                            if ((mask & ((short) 1 << index)) != 0)
                                bonuses[index] = buffer.readShort();
    
    
                        }
    
    
                    }
    
    
                    return this;
                }
    
    
                public ItemDefinition build() {
                    return new ItemDefinition(id, name, description, bonuses);
                }
            }
        }
    N.B. if the size of the bonuses array is a fixed length, you could remove a few additional bytes from each definition!
    Reply With Quote  
     

  4. #33  
    Extreme Donator

    nbness2's Avatar
    Join Date
    Aug 2011
    Posts
    692
    Thanks given
    274
    Thanks received
    139
    Rep Power
    430
    Quote Originally Posted by hc747 View Post
    Although I'd recommend to stick to JSON, you could further optimise your binary storage format by only storing non-zero values, whereas currently, if you have the following bonuses:
    { 0, 0, 0, 0, 0, 0, 1, 2 }
    you would be storing 8 shorts, which could and should be optimised to something like this (which would only require storing 2 + 1 shorts).

    The code below could be ported from Java and arbitrarily relies upon Netty's ByteBuf implementation:
    Code:
    	private static class ItemDefinition {
    
    
    		private final int id;
    
    
    		private final String name;
    
    
    		private final String description;
    
    
    		private final short[] bonuses;
    
    
    		public void encode(ByteBuf buffer) {
    
    
    			buffer.writeShort(id);
    
    
    			buffer.writeShort(name.length());
    			buffer.writeCharSequence(name, StandardCharsets.UTF_8);
    
    
    			buffer.writeShort(description.length());
    			buffer.writeCharSequence(description, StandardCharsets.UTF_8);
    
    
    			if (bonuses == null)
    				buffer.writeShort(0);
    			else {
    
    
    				short mask = 0;
    
    
    				for (short index = 0; index < bonuses.length; index++) {
    					if (bonuses[index] != 0)
    						mask |= (short) 1 << index;
    				}
    
    
    				buffer.writeShort(mask);
    
    
    				if (mask != 0) {
    
    
    					buffer.writeByte(bonuses.length);
    
    
    					for (short index = 0; index < bonuses.length; index++) {
    						if (bonuses[index] != 0)
    							buffer.writeShort(bonuses[index]);
    					}
    
    
    				}
    
    
    			}
    
    
    		}
    
    
    		ItemDefinition(int id, String name, String description, short[] bonuses) {
    			this.id = id;
    			this.name = name;
    			this.description = description;
    			this.bonuses = bonuses;
    		}
    
    
    		public static ItemDefinitionBuilder builder() {
    			return new ItemDefinitionBuilder();
    		}
    
    
    		public int getId() {
    			return id;
    		}
    
    
    		public String getName() {
    			return name;
    		}
    
    
    		public String getDescription() {
    			return description;
    		}
    
    
    		public short[] getBonuses() {
    			return bonuses;
    		}
    
    
    		public static class ItemDefinitionBuilder {
    
    
    			private int id;
    
    
    			private String name;
    
    
    			private String description;
    
    
    			private short[] bonuses;
    
    
    			ItemDefinitionBuilder() {}
    
    
    			public ItemDefinition.ItemDefinitionBuilder id(int id) {
    				this.id = id;
    				return this;
    			}
    
    
    			public ItemDefinition.ItemDefinitionBuilder name(String name) {
    				this.name = name;
    				return this;
    			}
    
    
    			public ItemDefinition.ItemDefinitionBuilder description(String description) {
    				this.description = description;
    				return this;
    			}
    
    
    			public ItemDefinition.ItemDefinitionBuilder bonuses(short[] bonuses) {
    				this.bonuses = bonuses;
    				return this;
    			}
    
    
    			public ItemDefinition.ItemDefinitionBuilder decode(ByteBuf buffer) {
    
    
    				id(buffer.readShort());
    
    
    				name(buffer.readCharSequence(buffer.readShort(), StandardCharsets.UTF_8).toString());
    
    
    				description(buffer.readCharSequence(buffer.readShort(), StandardCharsets.UTF_8).toString());
    
    
    				short mask = buffer.readShort();
    
    
    				if (mask != 0) {
    
    
    					bonuses = new short[buffer.readByte()];
    
    
    					for (short index = 0; index < bonuses.length; index++) {
    
    
    						if ((mask & ((short) 1 << index)) != 0)
    							bonuses[index] = buffer.readShort();
    
    
    					}
    
    
    				}
    
    
    				return this;
    			}
    
    
    			public ItemDefinition build() {
    				return new ItemDefinition(id, name, description, bonuses);
    			}
    		}
    	}
    N.B. if the size of the bonuses array is a fixed length, you could a few additional bytes from each definition!

    Interesting, didn't think too much about that. I'll save this snip and maybe use it later, but for now my focus will be mainly on getting the server up and running.
    Reply With Quote  
     

  5. #34  
    Extreme Donator

    nbness2's Avatar
    Join Date
    Aug 2011
    Posts
    692
    Thanks given
    274
    Thanks received
    139
    Rep Power
    430
    your favorite project is back (us too dont forget)



    https://gfycat.com/LeafyFemaleDiamondbackrattlesnake

    New features:
    • Logging in
    • Correct name-hashes (names used to be fucked, incorrect name -> long -> name procedure)
    • Appearance block writing
    • Movement (lol)


    Current goals:
    • Proper appearance-block writing (writes every tick, cant get it to change genders fucking cisgendered misogynistic intolerant trump supporting client)
    • Player updating
    • PROPER movement (and running/walking)
    Reply With Quote  
     

  6. #35  
    🎶 As you're falling down 🎶


    uint32_t's Avatar
    Join Date
    Feb 2015
    Posts
    1,396
    Thanks given
    6,177
    Thanks received
    776
    Rep Power
    5000
    Nice to see you're still working on this.
    Quote Originally Posted by Idiot Bird View Post
    Quote Originally Posted by Velocity View Post
    lol np mate looks like the community brought ur rep down to ur IQ
    Not too sure about that, it's at 0 . It would have to go minus to even be remotely close to his IQ.
    Reply With Quote  
     

  7. Thankful user:


  8. #36  
    Extreme Donator

    nbness2's Avatar
    Join Date
    Aug 2011
    Posts
    692
    Thanks given
    274
    Thanks received
    139
    Rep Power
    430
    Quote Originally Posted by i_pk_pjers_i View Post
    Nice to see you're still working on this.
    and why would I stop! Updates on the thread are a little slow because lots of stuff going on IRL, but if you check the bitbucket updates are much more frequent!

    Baby update (compared to what is coming soon )
    • Itemdefs moved from 459 to OSRS 145
    • Lots of params added and put into json (is_2h, is_noted, etc). This is going to be for ease, rather than handling this stuff in the code, we can just put it in itemdef! Makes python more viable for server hosting
    • JSON file size - 1077kb -> 3977kb
    • New binary pack format (8.82% size of original, retains all relevant data, with a caveat)
    • New binary is 55.4% size of the old binary and loads even faster (thanks, caveat)
    • Nulls are not fully packed, their data is all the same.


    Info on new binary packing format:
    Code:
    Item binary data length - 1 byte -- Length of the binary data for any given itemdef (will probably be changed to more if I decide to get item descriptions)
    
    Item ID - 15 bits - If the item is a null, the ID written will be 32767 and if a 32767 is read, go to the next item.
    
    Item name length - 6 bits - This is because the longest item name contains 35 characters
    
    Item Name - 7 bits per char - [Lines 88-99, 145-146, 189]
    
    CAVEAT - Nothing for item description. Right now all item descriptions are "It's a(n) xxx." so this can be eliminated while packing binary.
    
    If I ever bother to scrape the descriptions, they will be handled like item names.
    
    Item Price - 31 bits - This is for a max value of ~2147m, you won't be selling items for a negative price or for more than max-stack of an item, so last bit isnt' necessary in either scenario
    
    Item is 2h - 1 bit - True/False
    
    Item is stackable - 1 bit - True/False
    
    Item is tradable - 1 bit - True/False
    
    Item is sellable - 1 bit - True/False
    
    Item is droppable - 1 bit - True/False
    
    Item is noted - 1 bit - True/False
    
    Item can note - 1 bit - True/False
    
    If item isnt noted and item can note:
        Item noted id - 15 bits - This is the noted item id
    
    Item bonuses - 1-10 bits (first 4 for bonus id 0-13, last 9 for value -255 to 255).
        If a 1 is read, next 9 bits are read.
        If a 0 is read, the next bit is read for checking if there is a bonus
        After the 13th bit check (and possibly bonus grab), go to next item
    
    if last bit written is the last bit of the last byte:
        a 255 byte is added (11111111)
    
    if last bit written is not the last bit of the last byte (if last byte was written with 1-7 bits):
         1s are written to the rest of the byte and the byte's bits are reversed
    
    this is to maintain data integrity
    Reply With Quote  
     

  9. Thankful user:


  10. #37  
    Extreme Donator

    nbness2's Avatar
    Join Date
    Aug 2011
    Posts
    692
    Thanks given
    274
    Thanks received
    139
    Rep Power
    430
    Update - cache loading
    It's been slow because school started. (Also switched to OSRS protocol).
    Currently loading revision 152 cache (ty openrs).
    No CRC calculating yet, but want to finish it ASAP because protocol requires it.
    Link to latest Bitbucket commit 15c0671 - Renamed sector/cachepointer properties
    KT/JAVA - NBX 637 - HERE!
    KT - Drop table 4: Flexible, Powerful - HERE!
    KT - Command: Simplify writing commands - HERE
    KT - NbUtil: Make your kotlin easier - HERE
    KT - Hopping Islands: From Java to Kotlin - P1 - P2 - P3 - P4 - P5
    Reply With Quote  
     

  11. #38  
    Registered Member
    Remi's Avatar
    Join Date
    Jan 2015
    Posts
    628
    Thanks given
    572
    Thanks received
    212
    Rep Power
    574
    Nice progress man.
    Where the fuck is my cigarettes, I need my cancer. [C]44..
    Reply With Quote  
     

  12. #39  
    Banned

    Join Date
    Mar 2011
    Posts
    657
    Thanks given
    105
    Thanks received
    75
    Rep Power
    0
    Looks good man
    Reply With Quote  
     

  13. #40  
    Donator


    Join Date
    Aug 2012
    Posts
    2,462
    Thanks given
    312
    Thanks received
    459
    Rep Power
    457
    I know this is a grave dig, bug er. Any chance this framework is still being updated? Python is great
    Attached image

    Attached image
    Reply With Quote  
     

Page 4 of 5 FirstFirst ... 2345 LastLast

Thread Information
Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)


User Tag List

Similar Threads

  1. Feather - RuneScape Private Server Framework
    By Thock321 in forum Downloads
    Replies: 8
    Last Post: 08-29-2013, 04:48 AM
  2. Creating a server framework
    By Virus X3 in forum Help
    Replies: 34
    Last Post: 12-28-2009, 01:47 AM
  3. 317 Server framework
    By Stability666 in forum Downloads
    Replies: 21
    Last Post: 10-11-2009, 08:33 PM
  4. Login @ PYTHON SERVER!! AMFG!!
    By w::v::d in forum Show-off
    Replies: 11
    Last Post: 08-13-2009, 07:35 PM
  5. RSES Server Framework
    By Lazaro in forum Downloads
    Replies: 62
    Last Post: 03-14-2009, 11:32 AM
Tags for this Thread

View Tag Cloud

Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •