The Reference Table Structure
Table of Contents:
- Old engine history
- New engine history
- Reference tables
- Examples
Old engine history
The old engine of RuneScape operated on 'named' files by using what I call their Archive format, which is a large block composed of file entries, each with a compressed size, decompressed size, start offset, a file identifier hash (converts a variable-sized text file name to a packed 32-bit integer),
and the file's data scribed into the archive. The archives made use of BZIP2 compression on their data in order to reduce their sizes vastly. Many people who had not decoded the format had named it 'JAG Compression', after defyboy's original research in 2006-07 (?). Since then, JaGeX has abandoned their old Archive format (also called JAG, after their original RSC-era file extensions) and now have retreated back to using their raw uncompressed file system.
New engine history
Because simply using the raw file system isn't smart because file's won't be compressed, the good workers of JaGeX employed the concept of containers. Containers are simply seen as raw files to the file system I/O class, but are in a specific format that yet again allows compression. Each container consists of a header that contains it's compression type and decompressed size. Depending on the compression type, the file is either compressed
using GZIP (type 2+), BZIP2 (type 1), or not compressed at all (type 0). Every client revision comes with it's own container unpacking method. Now that compression is taken care of, what about 'naming' files? This is where the reference tables come into play.
Reference tables
The use of reference tables started at the same exact time the file system identified by 255 was dropped into your 'cache' directory. A reference table is a file within file system 255 in a specific format. Every reference table contains important information on each file in it's described file system. The data for each of these files in the reference table is called an entry. The format for a reference table is described below.
Reference tables start off with a version number (if 6, an unidentified integer is read which is believed to possible by a file system name identifier hash). Following is a flag telling if the table contains file identifier hashes. An important fact to note is that file identifiers are hashed differently than in the Archive format. After the flag comes in, the number of entries is read, and that many 'spacing' values are read. After spacing values, if file name hashes are supported, it reads a file identifier for each entry. Right after comes a set of CRC32 checksums of each entry (used internally by client to version-check it's file system data). The next set of data, dubbed 'field4' by it's position relative to the other reference table 'fields', is currently unknown. Speculations state it could possibly be the entry file's version of some sort. Later comes field 5, or children count. Every entry could potentially house several children, or 'sub-files'. After children count, the children offsets inside their parent's file data are read, later surpassed by a check to if the table contains file identifier hashes. If it does, it then reads each child's identifier hashes.
This concludes the format.
Examples
For example, file system 5 holds map and landscape related files. Map files are identified by mX_Y and landscape by lX_Y, where X and Y are the map and landscape's owner map region coordinates. Defyboy has started an attempt to bruteforce many of the file identifiers into their natural names. They can be found under 'names.txt' within the CacheEdit project's SVN folder. The update server of the official runescape server's serves requests with the raw file data in the file system. An example can be seen within the 'update keys'. 'Update keys' are actually the file data of the reference table of file system 255 itself, which are contained within the special container format.