[634] Optimal varp handling/saving

I'm currently writing my own varp manager to store all the different configuration data sent to the client. It will implement varp (or config), varb (or config by file), global config (not sure the technical name for this one) and global strings (not sure either).

The only thing I'm concerned about is how to optimally save the data that needs to be stored in the player save file. The most simple example of this would be the player's hitpoints which needs to be saved for obvious reasons. However, some other config like the world map blue marker can and should be ignored.

What would be the best way to handle this issue? Here's some of the solutions I thought of, though they're all kind of bad which is why I made this thread:

1- Duplicate the data by having a main variable (ex: "hitpoints" variable in the Entity class) and some kind of listener that updates the value in the varp manager. Seems like a pain to maintain/debug and just a terrible idea.

2- Have 2 different data structures for transient configs and configs that have to be saved. It's not as bad as the previous solution, but it becomes complicated when you have to specify each time you update a config if you want to save the config or not.

3- Save everything that's ever set in the varp manager. I feel like that would bloat the file saves with useless junk though...

> (ex: "hitpoints" variable in the Entity class)
One of the flaws in matrix; No idea why they didn't just make current hitpoints and prayer be represented as the level value in Skills class. Would be a nicer way of doing it IMO.
Alternatively, if something is directly represented by varp, there is no need to create variables in player/entity class, instead create a method that gets/sets values in the varps directly, remove the excess variable. It's unnecessary.

Given you already have a source written and whatnot, I think defining each existing varp to be saved or not is a bit of a hassle. Instead, I think you should save all the varps whose value isn't a 0 in a map. There is no need to save varbits individually - varbit values are calculated out of the parent varps themselves - you'd naturally have that. However, I would personally add support for defining varps that shouldn't be saved. There are some temporary varps used in RS, you probably wouldn't want their value to carry over between sessions.

As for the other types, I don't really remember much about them so I won't answer on their behalf.

Quote:

---

Originally Posted by clem585 [Only registered and activated users can see links. Click Here To Register...]

varp (or config), varb (or config by file), global config (not sure the technical name for this one) and global strings (not sure either).

---

Afaik:
"Config" = Varp
"Config by file?" = Varbit
Global Config = Varc
Global Config String = VarcStr

Quote:

---

Originally Posted by clem585 [Only registered and activated users can see links. Click Here To Register...]

1- Duplicate the data by having a main variable (ex: "hitpoints" variable in the Entity class) and some kind of listener that updates the value in the varp manager. Seems like a pain to maintain/debug and just a terrible idea.

---

I think you should just refactor this and remove it, as it's really not needed, as Kris stated Hitpoints/Prayer should just be sent as a stat like the rest of them.

Quote:

---

Originally Posted by clem585 [Only registered and activated users can see links. Click Here To Register...]

2- Have 2 different data structures for transient configs and configs that have to be saved. It's not as bad as the previous solution, but it becomes complicated when you have to specify each time you update a config if you want to save the config or not.

---

Right now I've got all varps transmitting to the client when their value is changed as a default state and non-zero varps are serialized, but I've added support for overriding those flags via a YAML config (which is also used by the cache tool to create new varbits). The gradle project reads these YAML files at build time and generates code that the server can use to lookup these flags (so that I don't need to write them into the cache, or externally load any other files):
[Only registered and activated users can see links. Click Here To Register...]

You only store varps. You don't store varcs (global configs) or stringcs (global strings). It's also not possible to maintain varcs or stringcs since they get manipulated by cs2. There's also a very small amount of varps that get manipulated by cs2.

Quote:

---

Originally Posted by Kris [Only registered and activated users can see links. Click Here To Register...]

> (ex: "hitpoints" variable in the Entity class)
One of the flaws in matrix; No idea why they didn't just make current hitpoints and prayer be represented as the level value in Skills class. Would be a nicer way of doing it IMO.
Alternatively, if something is directly represented by varp, there is no need to create variables in player/entity class, instead create a method that gets/sets values in the varps directly, remove the excess variable. It's unnecessary.

Given you already have a source written and whatnot, I think defining each existing varp to be saved or not is a bit of a hassle. Instead, I think you should save all the varps whose value isn't a 0 in a map. There is no need to save varbits individually - varbit values are calculated out of the parent varps themselves - you'd naturally have that. However, I would personally add support for defining varps that shouldn't be saved. There are some temporary varps used in RS, you probably wouldn't want their value to carry over between sessions.

As for the other types, I don't really remember much about them so I won't answer on their behalf.

---

Thanks for the tips on hitpoints/prayers, kind of overlooked that haha

Quote:

---

Originally Posted by Dread [Only registered and activated users can see links. Click Here To Register...]

Afaik:
"Config" = Varp
"Config by file?" = Varbit
Global Config = Varc
Global Config String = VarcStr



I think you should just refactor this and remove it, as it's really not needed, as Kris stated Hitpoints/Prayer should just be sent as a stat like the rest of them.


Right now I've got all varps transmitting to the client when their value is changed as a default state and non-zero varps are serialized, but I've added support for overriding those flags via a YAML config (which is also used by the cache tool to create new varbits). The gradle project reads these YAML files at build time and generates code that the server can use to lookup these flags (so that I don't need to write them into the cache, or externally load any other files):
[Only registered and activated users can see links. Click Here To Register...]

---

Thanks, I ended up going for this based on your last comment. Basically, I don't serialize nor send varps/varbs by default. I have 2 DBs (one for varps, one for varbs) to specify if I want to do nothing, serialize or serialize&send the varp/varb. Added optional description too after Kris's suggestion. Might change it in the future if I see that they are saved more often than not though.

Code:

---

package com.rs.game.player.scripts;

import com.common.cache.reader.loaders.ConfigDefinitions;
import com.common.json.JsonFileLoader;
import com.rs.json.VarbsDb;
import com.rs.json.VarpsDb;
import com.rs.net.game.out.game.VarpPacket;

import java.util.HashMap;
import java.util.Map;

/**
 * Created by clem585 on 2021-02-14.
 */
public class VarsManager extends PlayerScript {

        private transient Map<Integer, Integer> varps;
        private transient Map<Integer, Integer> globalInts;
        private transient Map<Integer, String> globalStrings;

        private Map<Integer, Integer> persistentVarps;

        public VarsManager() {
                varps = new HashMap<>();
                globalInts = new HashMap<>();
                globalStrings = new HashMap<>();

                persistentVarps = new HashMap<>();
        }

        @Override
        public void init() {
                for (int id : JsonFileLoader.get(VarpsDb.class).fetchLoginVarps()) {
                        if (getVarp(id) != 0) {
                                player.getEncoder().write(new VarpPacket(id, getVarp(id)));
                        }
                }
                for (int id : JsonFileLoader.get(VarbsDb.class).fetchLoginVarbs()) {
                        if (getVarb(id) != 0) {
                                // TODO varb packet
                                // player.getEncoder().write(new ConfigByFilePacket(id, getVarb(id)));
                        }
                }
        }


        public int getVarp(int id) {
                Map<Integer, Integer> configs = JsonFileLoader.get(VarpsDb.class).isPersistent(id) ? persistentVarps : varps;
                return configs.getOrDefault(id, 0);

        }

        public int getVarb(int id) {
                ConfigDefinitions configDefinitions = ConfigDefinitions.getConfigDefinitions(id);
                Map<Integer, Integer> configs = JsonFileLoader.get(VarbsDb.class).isPersistent(id) ? persistentVarps : varps;
                return (configs.getOrDefault(configDefinitions.getConfigId(), 0) & getMask(id))
                                >> configDefinitions.getStartBit();
        }

        public int getGlobalInt(int id) {
                return globalInts.getOrDefault(id, 0);
        }

        public String getGlobalString(int id) {
                return globalStrings.getOrDefault(id, "");
        }

        public void setVarp(int id, int value) {
                Map<Integer, Integer> configs = JsonFileLoader.get(VarpsDb.class).isPersistent(id) ? persistentVarps : varps;
                configs.put(id, value);
        }

        public void setVarb(int id, int value) {
                ConfigDefinitions configDefinitions = ConfigDefinitions.getConfigDefinitions(id);

                int max = (int) Math.pow(2, configDefinitions.getEndBit() - configDefinitions.getStartBit());
                if (value >= max) {
                        throw new RuntimeException(String.format("Attempted to overflow varbit, max: %d, value: %d", max, value));
                }

                value <<= configDefinitions.getStartBit();
                Map<Integer, Integer> configs = JsonFileLoader.get(VarbsDb.class).isPersistent(id) ? persistentVarps : varps;
                configs.put(configDefinitions.getConfigId(), varps.getOrDefault(configDefinitions.getConfigId(), 0)
                                & ~getMask(id) | value);
        }

        public void setGlobalInt(int id, int value) {
                globalInts.put(id, value);
        }

        public void setGlobalString(int id, String value) {
                globalStrings.put(id, value);
        }

        private int getMask(int id) {
                ConfigDefinitions configDefinitions = ConfigDefinitions.getConfigDefinitions(id);

                int mask = 0;
                for (int i = configDefinitions.getStartBit(); i <= configDefinitions.getEndBit(); ++i) {
                        mask |= (int)Math.pow(2, i);
                }

                return mask;
        }

}

---

Code:

---

package com.rs.json.models;

/**
 * Created by clem585 on 2021-02-14.
 */
public enum VarStatus {
        NORMAL, PERSISTENT, SEND_ON_LOGIN
}

---

Code:

---

package com.rs.json.models;

import lombok.Getter;
import lombok.Setter;

/**
 * Created by clem585 on 2021-02-14.
 */
public class VarDetails {

        @Getter @Setter
        private String description;
        @Getter @Setter
        private VarStatus varStatus;

        public VarDetails() {
                varStatus = VarStatus.NORMAL;
        }

}

---

Code:

---

package com.rs.json;

import com.common.json.JsonFile;
import com.fasterxml.jackson.core.type.TypeReference;
import com.rs.json.models.VarDetails;
import com.rs.json.models.VarStatus;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;

/**
 * Created by clem585 on 2021-02-14.
 */
public class VarbsDb extends JsonFile<Map<Integer, VarDetails>> {

        public VarbsDb() {
                super("varbs");
        }

        public void setStatus(int id, VarStatus status) {
                VarDetails details = data.getOrDefault(id, new VarDetails());
                if (status == VarStatus.NORMAL && details.getDescription() == null) {
                        data.remove(id);
                } else {
                        details.setVarStatus(status);
                        data.put(id, details);
                }
        }

        public void setDescription(int id, String description) {
                VarDetails details = data.getOrDefault(id, new VarDetails());
                details.setDescription(description);
                data.put(id, details);
        }

        public boolean isPersistent(int id) {
                return data.containsKey(id) && data.get(id).getVarStatus() != VarStatus.NORMAL;
        }

        public List<Integer> fetchLoginVarbs() {
                return data.keySet().stream().filter((key) -> data.get(key).getVarStatus() == VarStatus.SEND_ON_LOGIN).
                                collect(Collectors.toList());
        }

        @Override
        protected Class getValueType() {
                return HashMap.class;
        }

        @Override
        protected TypeReference getTypeReference() {
                return new TypeReference<HashMap<Integer, VarDetails>>(){};
        }

}

---

Quote:

---

Originally Posted by Displee [Only registered and activated users can see links. Click Here To Register...]

You only store varps. You don't store varcs (global configs) or stringcs (global strings). It's also not possible to maintain varcs or stringcs since they get manipulated by cs2. There's also a very small amount of varps that get manipulated by cs2.

---

Thanks, I didn't know much about varcs and stringcs since I only used them a few times in the past. In that case would it be better to not save them at all in the VarpManager since they're usually only sent once and then accessed client-sided?

Quote:

---

Originally Posted by clem585 [Only registered and activated users can see links. Click Here To Register...]

Thanks, I didn't know much about varcs and stringcs since I only used them a few times in the past. In that case would it be better to not save them at all in the VarpManager since they're usually only sent once and then accessed client-sided?

---

Yes. You don't need to save them at all. Really no reason to.

What would be the best way to send the configs each tick? I was thinking of using dirty flags or I could just queue a packet each time any config's value is updated.

Quote:

---

Originally Posted by clem585 [Only registered and activated users can see links. Click Here To Register...]

What would be the best way to send the configs each tick? I was thinking of using dirty flags or I could just queue a packet each time any config's value is updated. Also, does sending a full varp also trigger the changes on the varbs related to that varp? Let's say I send varp #340 and varb #10 refers to some bits of varp #340, will the client know varb #10 has been updated?

---

Why do you want to send varps each tick? I send most of them at login, and then just do this:

Code:

---

public boolean set(int id, int value, boolean force) {
        if (varps[id] != value) {
                varps[id] = value;
                send(id);
        } else if (force) {
                send(id);
        } else {
                if (GameConstants.DEVELOPER_MODE) {
                        logger.log(PrintType.ERROR, "The value for varp " + id + " was already equal to " + value + ".");
                }
                return false;
        }
        return true;
}

---

And yes. The client will update the corresponding varp. But you should never trust the client, so do it server sided.

Code:

---

public boolean setBit(int id, int value) {
        VarBitDefinition def = CacheRepository.get(VarBitDefinition.class, id);
        int startBit = def.startBit;
        int endBit = def.endBit;
        int flag = VAR_BIT_FLAGS[endBit - startBit];
        if (value < 0 || value > flag) {
                value = 0;
        }
        flag <<= startBit;
        return set(def.getVarId(), varps[id] & ~flag | value << startBit & flag);
}

---

You don't even need to use the varbit packet.

Edit:
Those `VAR_BIT_FLAGS` are not really flags. They define the maximum value the varbit can have.

Quote:

---

Originally Posted by Displee [Only registered and activated users can see links. Click Here To Register...]

Why do you want to send varps each tick? I send most of them at login, and then just do this:

Code:

---

public boolean set(int id, int value, boolean force) {
        if (varps[id] != value) {
                varps[id] = value;
                send(id);
        } else if (force) {
                send(id);
        } else {
                if (GameConstants.DEVELOPER_MODE) {
                        logger.log(PrintType.ERROR, "The value for varp " + id + " was already equal to " + value + ".");
                }
                return false;
        }
        return true;
}

---

And yes. The client will update the corresponding varp. But you should never trust the client, so do it server sided.

Code:

---

public boolean setBit(int id, int value) {
        VarBitDefinition def = CacheRepository.get(VarBitDefinition.class, id);
        int startBit = def.startBit;
        int endBit = def.endBit;
        int flag = VAR_BIT_FLAGS[endBit - startBit];
        if (value < 0 || value > flag) {
                value = 0;
        }
        flag <<= startBit;
        return set(def.getVarId(), varps[id] & (~flag) | value << startBit & flag);
}

---

You don't even need to use the varbit packet.

---

I do think it's a better idea to store a set of dirty varps that have changed this tick, and then flush all changed varps (I believe this is what Clem was trying to ask), for cases where a varp might have its value mutated multiple times from different varbits.

Quote:

---

Originally Posted by Dread [Only registered and activated users can see links. Click Here To Register...]

I do think it's a better idea to store a set of dirty varps that have changed this tick, and then flush all changed varps (I believe this is what Clem was trying to ask), for cases where a varp might have its value mutated multiple times from different varbits.

---

Yea that's what I ended up going for. Here's the updated code:

Code:

---

package com.rs.game.player.scripts;

import com.common.cache.reader.loaders.ConfigDefinitions;
import com.common.json.JsonFileLoader;
import com.rs.json.VarpsDb;
import com.rs.net.game.out.game.GlobalConfigPacket;
import com.rs.net.game.out.game.GlobalStringPacket;
import com.rs.net.game.out.game.VarpPacket;

import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;

/**
 * Created by clem585 on 2021-02-14.
 */
public class VarpManager extends PlayerScript {

        private transient Map<Integer, Integer> varps;
        private Map<Integer, Integer> persistentVarps;
        private transient Map<Integer, Integer> globalInts;
        private transient Map<Integer, String> globalStrings;

        private Set<Integer> dirtyVarps;
        private Set<Integer> dirtyGlobalInts;
        private Set<Integer> dirtyGlobalStrings;

        public VarpManager() {
                varps = new HashMap<>();
                persistentVarps = new HashMap<>();
                globalInts = new HashMap<>();
                globalStrings = new HashMap<>();

                dirtyVarps = new HashSet<>();
                dirtyGlobalInts = new HashSet<>();
                dirtyGlobalStrings = new HashSet<>();
        }

        @Override
        public void init() {
                for (int id : JsonFileLoader.get(VarpsDb.class).fetchLoginVarps()) {
                        if (getVarp(id) != 0) {
                                player.getEncoder().write(new VarpPacket(id, getVarp(id)));
                        }
                }
        }

        @Override
        public void tick() {
                for (int varp : dirtyVarps) {
                        player.getEncoder().write(new VarpPacket(varp, getVarp(varp)));
                }
                dirtyVarps.clear();

                for (int globalInt : dirtyGlobalInts) {
                        player.getEncoder().write(new GlobalConfigPacket(globalInt, getGlobalInt(globalInt)));
                }
                dirtyGlobalInts.clear();

                for (int globalString : dirtyGlobalStrings) {
                        player.getEncoder().write(new GlobalStringPacket(globalString, getGlobalString(globalString)));
                }
                dirtyGlobalStrings.clear();
        }

        public int getVarp(int id) {
                Map<Integer, Integer> configs = JsonFileLoader.get(VarpsDb.class).isPersistent(id) ? persistentVarps : varps;
                return configs.getOrDefault(id, 0);

        }

        public int getVarb(int id) {
                ConfigDefinitions configDefinitions = ConfigDefinitions.getConfigDefinitions(id);
                Map<Integer, Integer> configs = JsonFileLoader.get(VarpsDb.class).isPersistent(configDefinitions.getConfigId())
                                ? persistentVarps : varps;
                return (configs.getOrDefault(configDefinitions.getConfigId(), 0) & getMask(id))
                                >> configDefinitions.getStartBit();
        }

        public int getGlobalInt(int id) {
                return globalInts.getOrDefault(id, 0);
        }

        public String getGlobalString(int id) {
                return globalStrings.getOrDefault(id, "");
        }

        public void setVarp(int id, int value) {
                Map<Integer, Integer> configs = JsonFileLoader.get(VarpsDb.class).isPersistent(id) ? persistentVarps : varps;
                configs.put(id, value);
                dirtyVarps.add(id);
        }

        public void setVarb(int id, int value) {
                ConfigDefinitions configDefinitions = ConfigDefinitions.getConfigDefinitions(id);

                int max = (int) Math.pow(2, configDefinitions.getEndBit() - configDefinitions.getStartBit());
                if (value >= max) {
                        throw new RuntimeException(String.format("Attempted to overflow varbit, max: %d, value: %d", max, value));
                }

                value <<= configDefinitions.getStartBit();
                Map<Integer, Integer> configs = JsonFileLoader.get(VarpsDb.class).isPersistent(configDefinitions.getConfigId())
                                ? persistentVarps : varps;
                configs.put(configDefinitions.getConfigId(), varps.getOrDefault(configDefinitions.getConfigId(), 0)
                                & ~getMask(id) | value);
                dirtyVarps.add(configDefinitions.getConfigId());
        }

        public void setGlobalInt(int id, int value) {
                globalInts.put(id, value);
                dirtyGlobalInts.add(id);
        }

        public void setGlobalString(int id, String value) {
                globalStrings.put(id, value);
                dirtyGlobalStrings.add(id);
        }

        private int getMask(int id) {
                ConfigDefinitions configDefinitions = ConfigDefinitions.getConfigDefinitions(id);

                int mask = 0;
                for (int i = configDefinitions.getStartBit(); i <= configDefinitions.getEndBit(); ++i) {
                        mask |= (int)Math.pow(2, i);
                }

                return mask;
        }

}

---