Thread: [634] Optimal varp handling/saving

Page 1 of 2 12 LastLast
Results 1 to 10 of 14
  1. #1 [634] Optimal varp handling/saving 
    Contributor

    clem585's Avatar
    Join Date
    Sep 2013
    Posts
    3,566
    Thanks given
    662
    Thanks received
    643
    Discord
    View profile
    Rep Power
    406
    I'm currently writing my own varp manager to store all the different configuration data sent to the client. It will implement varp (or config), varb (or config by file), global config (not sure the technical name for this one) and global strings (not sure either).

    The only thing I'm concerned about is how to optimally save the data that needs to be stored in the player save file. The most simple example of this would be the player's hitpoints which needs to be saved for obvious reasons. However, some other config like the world map blue marker can and should be ignored.

    What would be the best way to handle this issue? Here's some of the solutions I thought of, though they're all kind of bad which is why I made this thread:

    1- Duplicate the data by having a main variable (ex: "hitpoints" variable in the Entity class) and some kind of listener that updates the value in the varp manager. Seems like a pain to maintain/debug and just a terrible idea.

    2- Have 2 different data structures for transient configs and configs that have to be saved. It's not as bad as the previous solution, but it becomes complicated when you have to specify each time you update a config if you want to save the config or not.

    3- Save everything that's ever set in the varp manager. I feel like that would bloat the file saves with useless junk though...
    Project thread
    [Only registered and activated users can see links. ]
    Reply With Quote  
     

  2. #2  
    Contributor
    Kris's Avatar
    Join Date
    Jun 2016
    Age
    23
    Posts
    3,542
    Thanks given
    709
    Thanks received
    2,346
    Discord
    View profile
    Rep Power
    5000
    > (ex: "hitpoints" variable in the Entity class)
    One of the flaws in matrix; No idea why they didn't just make current hitpoints and prayer be represented as the level value in Skills class. Would be a nicer way of doing it IMO.
    Alternatively, if something is directly represented by varp, there is no need to create variables in player/entity class, instead create a method that gets/sets values in the varps directly, remove the excess variable. It's unnecessary.

    Given you already have a source written and whatnot, I think defining each existing varp to be saved or not is a bit of a hassle. Instead, I think you should save all the varps whose value isn't a 0 in a map. There is no need to save varbits individually - varbit values are calculated out of the parent varps themselves - you'd naturally have that. However, I would personally add support for defining varps that shouldn't be saved. There are some temporary varps used in RS, you probably wouldn't want their value to carry over between sessions.

    As for the other types, I don't really remember much about them so I won't answer on their behalf.
    Reply With Quote  
     

  3. Thankful user:


  4. #3  
    Registered Member
    Dread's Avatar
    Join Date
    Nov 2013
    Posts
    40
    Thanks given
    16
    Thanks received
    91
    Discord
    View profile
    Rep Power
    300
    Quote Originally Posted by clem585 View Post
    varp (or config), varb (or config by file), global config (not sure the technical name for this one) and global strings (not sure either).
    Afaik:
    "Config" = Varp
    "Config by file?" = Varbit
    Global Config = Varc
    Global Config String = VarcStr

    Quote Originally Posted by clem585 View Post
    1- Duplicate the data by having a main variable (ex: "hitpoints" variable in the Entity class) and some kind of listener that updates the value in the varp manager. Seems like a pain to maintain/debug and just a terrible idea.
    I think you should just refactor this and remove it, as it's really not needed, as Kris stated Hitpoints/Prayer should just be sent as a stat like the rest of them.

    Quote Originally Posted by clem585 View Post
    2- Have 2 different data structures for transient configs and configs that have to be saved. It's not as bad as the previous solution, but it becomes complicated when you have to specify each time you update a config if you want to save the config or not.
    Right now I've got all varps transmitting to the client when their value is changed as a default state and non-zero varps are serialized, but I've added support for overriding those flags via a YAML config (which is also used by the cache tool to create new varbits). The gradle project reads these YAML files at build time and generates code that the server can use to lookup these flags (so that I don't need to write them into the cache, or externally load any other files):
    Reply With Quote  
     

  5. Thankful user:


  6. #4  
    Contributor


    Join Date
    Dec 2009
    Posts
    764
    Thanks given
    345
    Thanks received
    420
    Discord
    View profile
    Rep Power
    834
    You only store varps. You don't store varcs (global configs) or stringcs (global strings). It's also not possible to maintain varcs or stringcs since they get manipulated by cs2. There's also a very small amount of varps that get manipulated by cs2.
    link removed
    Reply With Quote  
     

  7. Thankful users:


  8. #5  
    Contributor

    clem585's Avatar
    Join Date
    Sep 2013
    Posts
    3,566
    Thanks given
    662
    Thanks received
    643
    Discord
    View profile
    Rep Power
    406
    Quote Originally Posted by Kris View Post
    > (ex: "hitpoints" variable in the Entity class)
    One of the flaws in matrix; No idea why they didn't just make current hitpoints and prayer be represented as the level value in Skills class. Would be a nicer way of doing it IMO.
    Alternatively, if something is directly represented by varp, there is no need to create variables in player/entity class, instead create a method that gets/sets values in the varps directly, remove the excess variable. It's unnecessary.

    Given you already have a source written and whatnot, I think defining each existing varp to be saved or not is a bit of a hassle. Instead, I think you should save all the varps whose value isn't a 0 in a map. There is no need to save varbits individually - varbit values are calculated out of the parent varps themselves - you'd naturally have that. However, I would personally add support for defining varps that shouldn't be saved. There are some temporary varps used in RS, you probably wouldn't want their value to carry over between sessions.

    As for the other types, I don't really remember much about them so I won't answer on their behalf.
    Thanks for the tips on hitpoints/prayers, kind of overlooked that haha

    Quote Originally Posted by Dread View Post
    Afaik:
    "Config" = Varp
    "Config by file?" = Varbit
    Global Config = Varc
    Global Config String = VarcStr



    I think you should just refactor this and remove it, as it's really not needed, as Kris stated Hitpoints/Prayer should just be sent as a stat like the rest of them.


    Right now I've got all varps transmitting to the client when their value is changed as a default state and non-zero varps are serialized, but I've added support for overriding those flags via a YAML config (which is also used by the cache tool to create new varbits). The gradle project reads these YAML files at build time and generates code that the server can use to lookup these flags (so that I don't need to write them into the cache, or externally load any other files):
    Thanks, I ended up going for this based on your last comment. Basically, I don't serialize nor send varps/varbs by default. I have 2 DBs (one for varps, one for varbs) to specify if I want to do nothing, serialize or serialize&send the varp/varb. Added optional description too after Kris's suggestion. Might change it in the future if I see that they are saved more often than not though.

    Code:
    package com.rs.game.player.scripts;
    
    import com.common.cache.reader.loaders.ConfigDefinitions;
    import com.common.json.JsonFileLoader;
    import com.rs.json.VarbsDb;
    import com.rs.json.VarpsDb;
    import com.rs.net.game.out.game.VarpPacket;
    
    import java.util.HashMap;
    import java.util.Map;
    
    /**
     * Created by clem585 on 2021-02-14.
     */
    public class VarsManager extends PlayerScript {
    
    	private transient Map<Integer, Integer> varps;
    	private transient Map<Integer, Integer> globalInts;
    	private transient Map<Integer, String> globalStrings;
    
    	private Map<Integer, Integer> persistentVarps;
    
    	public VarsManager() {
    		varps = new HashMap<>();
    		globalInts = new HashMap<>();
    		globalStrings = new HashMap<>();
    
    		persistentVarps = new HashMap<>();
    	}
    
    	@Override
    	public void init() {
    		for (int id : JsonFileLoader.get(VarpsDb.class).fetchLoginVarps()) {
    			if (getVarp(id) != 0) {
    				player.getEncoder().write(new VarpPacket(id, getVarp(id)));
    			}
    		}
    		for (int id : JsonFileLoader.get(VarbsDb.class).fetchLoginVarbs()) {
    			if (getVarb(id) != 0) {
    				// TODO varb packet
    				// player.getEncoder().write(new ConfigByFilePacket(id, getVarb(id)));
    			}
    		}
    	}
    
    
    	public int getVarp(int id) {
    		Map<Integer, Integer> configs = JsonFileLoader.get(VarpsDb.class).isPersistent(id) ? persistentVarps : varps;
    		return configs.getOrDefault(id, 0);
    
    	}
    
    	public int getVarb(int id) {
    		ConfigDefinitions configDefinitions = ConfigDefinitions.getConfigDefinitions(id);
    		Map<Integer, Integer> configs = JsonFileLoader.get(VarbsDb.class).isPersistent(id) ? persistentVarps : varps;
    		return (configs.getOrDefault(configDefinitions.getConfigId(), 0) & getMask(id))
    				>> configDefinitions.getStartBit();
    	}
    
    	public int getGlobalInt(int id) {
    		return globalInts.getOrDefault(id, 0);
    	}
    
    	public String getGlobalString(int id) {
    		return globalStrings.getOrDefault(id, "");
    	}
    
    	public void setVarp(int id, int value) {
    		Map<Integer, Integer> configs = JsonFileLoader.get(VarpsDb.class).isPersistent(id) ? persistentVarps : varps;
    		configs.put(id, value);
    	}
    
    	public void setVarb(int id, int value) {
    		ConfigDefinitions configDefinitions = ConfigDefinitions.getConfigDefinitions(id);
    
    		int max = (int) Math.pow(2, configDefinitions.getEndBit() - configDefinitions.getStartBit());
    		if (value >= max) {
    			throw new RuntimeException(String.format("Attempted to overflow varbit, max: %d, value: %d", max, value));
    		}
    
    		value <<= configDefinitions.getStartBit();
    		Map<Integer, Integer> configs = JsonFileLoader.get(VarbsDb.class).isPersistent(id) ? persistentVarps : varps;
    		configs.put(configDefinitions.getConfigId(), varps.getOrDefault(configDefinitions.getConfigId(), 0)
    				& ~getMask(id) | value);
    	}
    
    	public void setGlobalInt(int id, int value) {
    		globalInts.put(id, value);
    	}
    
    	public void setGlobalString(int id, String value) {
    		globalStrings.put(id, value);
    	}
    
    	private int getMask(int id) {
    		ConfigDefinitions configDefinitions = ConfigDefinitions.getConfigDefinitions(id);
    
    		int mask = 0;
    		for (int i = configDefinitions.getStartBit(); i <= configDefinitions.getEndBit(); ++i) {
    			mask |= (int)Math.pow(2, i);
    		}
    
    		return mask;
    	}
    
    }
    Code:
    package com.rs.json.models;
    
    /**
     * Created by clem585 on 2021-02-14.
     */
    public enum VarStatus {
    	NORMAL, PERSISTENT, SEND_ON_LOGIN
    }
    Code:
    package com.rs.json.models;
    
    import lombok.Getter;
    import lombok.Setter;
    
    /**
     * Created by clem585 on 2021-02-14.
     */
    public class VarDetails {
    
    	@Getter @Setter
    	private String description;
    	@Getter @Setter
    	private VarStatus varStatus;
    
    	public VarDetails() {
    		varStatus = VarStatus.NORMAL;
    	}
    
    }
    Code:
    package com.rs.json;
    
    import com.common.json.JsonFile;
    import com.fasterxml.jackson.core.type.TypeReference;
    import com.rs.json.models.VarDetails;
    import com.rs.json.models.VarStatus;
    
    import java.util.HashMap;
    import java.util.List;
    import java.util.Map;
    import java.util.stream.Collectors;
    
    /**
     * Created by clem585 on 2021-02-14.
     */
    public class VarbsDb extends JsonFile<Map<Integer, VarDetails>> {
    
    	public VarbsDb() {
    		super("varbs");
    	}
    
    	public void setStatus(int id, VarStatus status) {
    		VarDetails details = data.getOrDefault(id, new VarDetails());
    		if (status == VarStatus.NORMAL && details.getDescription() == null) {
    			data.remove(id);
    		} else {
    			details.setVarStatus(status);
    			data.put(id, details);
    		}
    	}
    
    	public void setDescription(int id, String description) {
    		VarDetails details = data.getOrDefault(id, new VarDetails());
    		details.setDescription(description);
    		data.put(id, details);
    	}
    
    	public boolean isPersistent(int id) {
    		return data.containsKey(id) && data.get(id).getVarStatus() != VarStatus.NORMAL;
    	}
    
    	public List<Integer> fetchLoginVarbs() {
    		return data.keySet().stream().filter((key) -> data.get(key).getVarStatus() == VarStatus.SEND_ON_LOGIN).
    				collect(Collectors.toList());
    	}
    
    	@Override
    	protected Class getValueType() {
    		return HashMap.class;
    	}
    
    	@Override
    	protected TypeReference getTypeReference() {
    		return new TypeReference<HashMap<Integer, VarDetails>>(){};
    	}
    
    }
    Quote Originally Posted by Displee View Post
    You only store varps. You don't store varcs (global configs) or stringcs (global strings). It's also not possible to maintain varcs or stringcs since they get manipulated by cs2. There's also a very small amount of varps that get manipulated by cs2.
    Thanks, I didn't know much about varcs and stringcs since I only used them a few times in the past. In that case would it be better to not save them at all in the VarpManager since they're usually only sent once and then accessed client-sided?
    Project thread
    [Only registered and activated users can see links. ]
    Reply With Quote  
     

  9. #6  
    Contributor


    Join Date
    Dec 2009
    Posts
    764
    Thanks given
    345
    Thanks received
    420
    Discord
    View profile
    Rep Power
    834
    Quote Originally Posted by clem585 View Post
    Thanks, I didn't know much about varcs and stringcs since I only used them a few times in the past. In that case would it be better to not save them at all in the VarpManager since they're usually only sent once and then accessed client-sided?
    Yes. You don't need to save them at all. Really no reason to.
    link removed
    Reply With Quote  
     

  10. Thankful user:


  11. #7  
    Contributor

    clem585's Avatar
    Join Date
    Sep 2013
    Posts
    3,566
    Thanks given
    662
    Thanks received
    643
    Discord
    View profile
    Rep Power
    406
    What would be the best way to send the configs each tick? I was thinking of using dirty flags or I could just queue a packet each time any config's value is updated.
    Project thread
    [Only registered and activated users can see links. ]
    Reply With Quote  
     

  12. #8  
    Contributor


    Join Date
    Dec 2009
    Posts
    764
    Thanks given
    345
    Thanks received
    420
    Discord
    View profile
    Rep Power
    834
    Quote Originally Posted by clem585 View Post
    What would be the best way to send the configs each tick? I was thinking of using dirty flags or I could just queue a packet each time any config's value is updated. Also, does sending a full varp also trigger the changes on the varbs related to that varp? Let's say I send varp #340 and varb #10 refers to some bits of varp #340, will the client know varb #10 has been updated?
    Why do you want to send varps each tick? I send most of them at login, and then just do this:
    Code:
    public boolean set(int id, int value, boolean force) {
    	if (varps[id] != value) {
    		varps[id] = value;
    		send(id);
    	} else if (force) {
    		send(id);
    	} else {
    		if (GameConstants.DEVELOPER_MODE) {
    			logger.log(PrintType.ERROR, "The value for varp " + id + " was already equal to " + value + ".");
    		}
    		return false;
    	}
    	return true;
    }
    And yes. The client will update the corresponding varp. But you should never trust the client, so do it server sided.
    Code:
    public boolean setBit(int id, int value) {
    	VarBitDefinition def = CacheRepository.get(VarBitDefinition.class, id);
    	int startBit = def.startBit;
    	int endBit = def.endBit;
    	int flag = VAR_BIT_FLAGS[endBit - startBit];
    	if (value < 0 || value > flag) {
    		value = 0;
    	}
    	flag <<= startBit;
    	return set(def.getVarId(), varps[id] & ~flag | value << startBit & flag);
    }
    You don't even need to use the varbit packet.

    Edit:
    Those `VAR_BIT_FLAGS` are not really flags. They define the maximum value the varbit can have.
    link removed
    Reply With Quote  
     

  13. Thankful user:


  14. #9  
    Registered Member
    Dread's Avatar
    Join Date
    Nov 2013
    Posts
    40
    Thanks given
    16
    Thanks received
    91
    Discord
    View profile
    Rep Power
    300
    Quote Originally Posted by Displee View Post
    Why do you want to send varps each tick? I send most of them at login, and then just do this:
    Code:
    public boolean set(int id, int value, boolean force) {
    	if (varps[id] != value) {
    		varps[id] = value;
    		send(id);
    	} else if (force) {
    		send(id);
    	} else {
    		if (GameConstants.DEVELOPER_MODE) {
    			logger.log(PrintType.ERROR, "The value for varp " + id + " was already equal to " + value + ".");
    		}
    		return false;
    	}
    	return true;
    }
    And yes. The client will update the corresponding varp. But you should never trust the client, so do it server sided.
    Code:
    public boolean setBit(int id, int value) {
    	VarBitDefinition def = CacheRepository.get(VarBitDefinition.class, id);
    	int startBit = def.startBit;
    	int endBit = def.endBit;
    	int flag = VAR_BIT_FLAGS[endBit - startBit];
    	if (value < 0 || value > flag) {
    		value = 0;
    	}
    	flag <<= startBit;
    	return set(def.getVarId(), varps[id] & (~flag) | value << startBit & flag);
    }
    You don't even need to use the varbit packet.
    I do think it's a better idea to store a set of dirty varps that have changed this tick, and then flush all changed varps (I believe this is what Clem was trying to ask), for cases where a varp might have its value mutated multiple times from different varbits.
    Reply With Quote  
     

  15. Thankful users:


  16. #10  
    Contributor

    clem585's Avatar
    Join Date
    Sep 2013
    Posts
    3,566
    Thanks given
    662
    Thanks received
    643
    Discord
    View profile
    Rep Power
    406
    Quote Originally Posted by Dread View Post
    I do think it's a better idea to store a set of dirty varps that have changed this tick, and then flush all changed varps (I believe this is what Clem was trying to ask), for cases where a varp might have its value mutated multiple times from different varbits.
    Yea that's what I ended up going for. Here's the updated code:

    Code:
    package com.rs.game.player.scripts;
    
    import com.common.cache.reader.loaders.ConfigDefinitions;
    import com.common.json.JsonFileLoader;
    import com.rs.json.VarpsDb;
    import com.rs.net.game.out.game.GlobalConfigPacket;
    import com.rs.net.game.out.game.GlobalStringPacket;
    import com.rs.net.game.out.game.VarpPacket;
    
    import java.util.HashMap;
    import java.util.HashSet;
    import java.util.Map;
    import java.util.Set;
    
    /**
     * Created by clem585 on 2021-02-14.
     */
    public class VarpManager extends PlayerScript {
    
    	private transient Map<Integer, Integer> varps;
    	private Map<Integer, Integer> persistentVarps;
    	private transient Map<Integer, Integer> globalInts;
    	private transient Map<Integer, String> globalStrings;
    
    	private Set<Integer> dirtyVarps;
    	private Set<Integer> dirtyGlobalInts;
    	private Set<Integer> dirtyGlobalStrings;
    
    	public VarpManager() {
    		varps = new HashMap<>();
    		persistentVarps = new HashMap<>();
    		globalInts = new HashMap<>();
    		globalStrings = new HashMap<>();
    
    		dirtyVarps = new HashSet<>();
    		dirtyGlobalInts = new HashSet<>();
    		dirtyGlobalStrings = new HashSet<>();
    	}
    
    	@Override
    	public void init() {
    		for (int id : JsonFileLoader.get(VarpsDb.class).fetchLoginVarps()) {
    			if (getVarp(id) != 0) {
    				player.getEncoder().write(new VarpPacket(id, getVarp(id)));
    			}
    		}
    	}
    
    	@Override
    	public void tick() {
    		for (int varp : dirtyVarps) {
    			player.getEncoder().write(new VarpPacket(varp, getVarp(varp)));
    		}
    		dirtyVarps.clear();
    
    		for (int globalInt : dirtyGlobalInts) {
    			player.getEncoder().write(new GlobalConfigPacket(globalInt, getGlobalInt(globalInt)));
    		}
    		dirtyGlobalInts.clear();
    
    		for (int globalString : dirtyGlobalStrings) {
    			player.getEncoder().write(new GlobalStringPacket(globalString, getGlobalString(globalString)));
    		}
    		dirtyGlobalStrings.clear();
    	}
    
    	public int getVarp(int id) {
    		Map<Integer, Integer> configs = JsonFileLoader.get(VarpsDb.class).isPersistent(id) ? persistentVarps : varps;
    		return configs.getOrDefault(id, 0);
    
    	}
    
    	public int getVarb(int id) {
    		ConfigDefinitions configDefinitions = ConfigDefinitions.getConfigDefinitions(id);
    		Map<Integer, Integer> configs = JsonFileLoader.get(VarpsDb.class).isPersistent(configDefinitions.getConfigId())
    				? persistentVarps : varps;
    		return (configs.getOrDefault(configDefinitions.getConfigId(), 0) & getMask(id))
    				>> configDefinitions.getStartBit();
    	}
    
    	public int getGlobalInt(int id) {
    		return globalInts.getOrDefault(id, 0);
    	}
    
    	public String getGlobalString(int id) {
    		return globalStrings.getOrDefault(id, "");
    	}
    
    	public void setVarp(int id, int value) {
    		Map<Integer, Integer> configs = JsonFileLoader.get(VarpsDb.class).isPersistent(id) ? persistentVarps : varps;
    		configs.put(id, value);
    		dirtyVarps.add(id);
    	}
    
    	public void setVarb(int id, int value) {
    		ConfigDefinitions configDefinitions = ConfigDefinitions.getConfigDefinitions(id);
    
    		int max = (int) Math.pow(2, configDefinitions.getEndBit() - configDefinitions.getStartBit());
    		if (value >= max) {
    			throw new RuntimeException(String.format("Attempted to overflow varbit, max: %d, value: %d", max, value));
    		}
    
    		value <<= configDefinitions.getStartBit();
    		Map<Integer, Integer> configs = JsonFileLoader.get(VarpsDb.class).isPersistent(configDefinitions.getConfigId())
    				? persistentVarps : varps;
    		configs.put(configDefinitions.getConfigId(), varps.getOrDefault(configDefinitions.getConfigId(), 0)
    				& ~getMask(id) | value);
    		dirtyVarps.add(configDefinitions.getConfigId());
    	}
    
    	public void setGlobalInt(int id, int value) {
    		globalInts.put(id, value);
    		dirtyGlobalInts.add(id);
    	}
    
    	public void setGlobalString(int id, String value) {
    		globalStrings.put(id, value);
    		dirtyGlobalStrings.add(id);
    	}
    
    	private int getMask(int id) {
    		ConfigDefinitions configDefinitions = ConfigDefinitions.getConfigDefinitions(id);
    
    		int mask = 0;
    		for (int i = configDefinitions.getStartBit(); i <= configDefinitions.getEndBit(); ++i) {
    			mask |= (int)Math.pow(2, i);
    		}
    
    		return mask;
    	}
    
    }
    Project thread
    [Only registered and activated users can see links. ]
    Reply With Quote  
     

Page 1 of 2 12 LastLast

Thread Information
Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)


User Tag List

Similar Threads

  1. Varp for lumby bank chest 634 obects
    By Sgsrocks in forum Requests
    Replies: 0
    Last Post: 11-08-2016, 03:48 AM
  2. Better way to handle my Json Load/Save?
    By Kangaroo in forum Help
    Replies: 1
    Last Post: 09-24-2016, 04:37 AM
  3. 614/634 Appearance saving after logout?
    By zzzfishstick in forum Help
    Replies: 0
    Last Post: 01-27-2012, 10:09 PM
  4. make split private chat save [anti leech]
    By pkin3 in forum Tutorials
    Replies: 13
    Last Post: 05-20-2007, 12:36 AM
  5. Better format for saving 3D/21-Century sigs...
    By Killa Man in forum Tutorials
    Replies: 7
    Last Post: 04-05-2007, 09:35 PM
Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •