Thread: The Reference Table structure

Results 1 to 2 of 2
  1. #1 The Reference Table structure 
    Renown Programmer
    veer's Avatar
    Join Date
    Nov 2007
    Posts
    3,746
    Thanks given
    354
    Thanks received
    1,370
    Rep Power
    3032
    The Reference Table Structure

    Table of Contents:

    1. Old engine history
    2. New engine history
    3. Reference tables
    4. Examples





    Old engine history

    The old engine of RuneScape operated on 'named' files by using what I call their Archive format, which is a large block composed of file entries, each with a compressed size, decompressed size, start offset, a file identifier hash (converts a variable-sized text file name to a packed 32-bit integer),
    and the file's data scribed into the archive. The archives made use of BZIP2 compression on their data in order to reduce their sizes vastly. Many people who had not decoded the format had named it 'JAG Compression', after defyboy's original research in 2006-07 (?). Since then, JaGeX has abandoned their old Archive format (also called JAG, after their original RSC-era file extensions) and now have retreated back to using their raw uncompressed file system.

    New engine history

    Because simply using the raw file system isn't smart because file's won't be compressed, the good workers of JaGeX employed the concept of containers. Containers are simply seen as raw files to the file system I/O class, but are in a specific format that yet again allows compression. Each container consists of a header that contains it's compression type and decompressed size. Depending on the compression type, the file is either compressed
    using GZIP (type 2+), BZIP2 (type 1), or not compressed at all (type 0). Every client revision comes with it's own container unpacking method. Now that compression is taken care of, what about 'naming' files? This is where the reference tables come into play.

    Reference tables

    The use of reference tables started at the same exact time the file system identified by 255 was dropped into your 'cache' directory. A reference table is a file within file system 255 in a specific format. Every reference table contains important information on each file in it's described file system. The data for each of these files in the reference table is called an entry. The format for a reference table is described below.

    Reference tables start off with a version number (if 6, an unidentified integer is read which is believed to possible by a file system name identifier hash). Following is a flag telling if the table contains file identifier hashes. An important fact to note is that file identifiers are hashed differently than in the Archive format. After the flag comes in, the number of entries is read, and that many 'spacing' values are read. After spacing values, if file name hashes are supported, it reads a file identifier for each entry. Right after comes a set of CRC32 checksums of each entry (used internally by client to version-check it's file system data). The next set of data, dubbed 'field4' by it's position relative to the other reference table 'fields', is currently unknown. Speculations state it could possibly be the entry file's version of some sort. Later comes field 5, or children count. Every entry could potentially house several children, or 'sub-files'. After children count, the children offsets inside their parent's file data are read, later surpassed by a check to if the table contains file identifier hashes. If it does, it then reads each child's identifier hashes.
    This concludes the format.

    Examples

    For example, file system 5 holds map and landscape related files. Map files are identified by mX_Y and landscape by lX_Y, where X and Y are the map and landscape's owner map region coordinates. Defyboy has started an attempt to bruteforce many of the file identifiers into their natural names. They can be found under 'names.txt' within the CacheEdit project's SVN folder. The update server of the official runescape server's serves requests with the raw file data in the file system. An example can be seen within the 'update keys'. 'Update keys' are actually the file data of the reference table of file system 255 itself, which are contained within the special container format.
    Reply With Quote  
     

  2. #2  
    Tool Box
    Guest
    Thanks super_

    very well explained
    Reply With Quote  
     


Thread Information
Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)


User Tag List

Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •