Thread: OpenRS2 Archive

Page 1 of 5 123 ... LastLast
Results 1 to 10 of 47
  1. #1 OpenRS2 Archive 
    Programmer, Contributor, RM and Veteran




    Join Date
    Mar 2007
    Posts
    5,072
    Thanks given
    2,624
    Thanks received
    3,567
    Rep Power
    5000
    I've been working on this service for a while now, and it's now ready to be announced publicly (though there's still lots of work to do!)

    [Only registered and activated users can see links. ]

    It's currently a collection of caches and XTEA keys for all versions of RuneScape that use JS5 to manage assets (mainline RuneScape build 402+ and all builds of OSRS). Support for archiving clients/native libraries is planned in the future.

    It automatically downloads new OSRS caches when they are released and pulls OSRS XTEA keys from various sources. Support for automatically downloading RS3 caches is also planned.

    Unlike the existing archives, it's backed by a database - not a collection of static files. The groups are stored in a content-addressable format (based on (archive, group, checksum, version) tuples), which provides several benefits:

    * The disk usage is vastly reduced. Currently it uses 11 GB to store over 300 caches for a mixture of mainline RS and OSRS builds. The database backups (which are compressed and omit the indexes) are even smaller - around 2.2 GB at the moment. That's an average of about 7 MB per cache - much between than the tens or hundreds of megabytes storing each cache individually would use. (This will probably increase when RS3 data is imported, as the cache is so much larger than it used to be.)
    * Missing groups and keys can be located from nearby caches, as long as the checksum and version number are identical.

    The internal format is converted to .dat2/.idx on demand when a cache is downloaded. An alternative flat file format will be added for caches larger than 8GB (the max size of the .dat2 file) in the future.

    XTEA keys are automatically verified and they can be downloaded as a JSON file for every cache present in the archive. I'll add support for downloading a ZIP of text files in the future too. As many of the old dumpers didn't record the region ID for dynamic regions correctly, the service verifies every candidate key against every encrypted group for every version of the cache (and all future caches when they are imported). This allows us to eke out as many validated keys as possible.

    One known problem at the moment that I'm still fixing is that there are a small number of (archive, group, checksum, version) collisions (7 across the entire database at the time of writing). Some of the GZIP-compressed containers use the stored compression level (no compression). As GZIP has a CRC-32 trailer, any changes to the uncompressed data are actually cancelled out by the trailer (due to an unfortunate property of the CRC algorithm), rendering the overall CRC-32 checksum unchanged. Unfortunately, Jagex didn't increment the version number for these changes, and this would have in fact caused problems at the time: depending on when you first downloaded the cache, you might get an old or a new variant of the group. Users with the old variant of the group wouldn't receive updates. I can actually see this in the data I imported (e.g. some newer versions of the cache containing older variants of a colliding group, with others containing the new variant and no particular pattern to it). My suspicion is that Jagex only incremented the version number if they detected a change in the CRC-32 checksum, rather than comparing the uncompressed content (or a secure hash of it) directly. Luckily I haven't found any collisions in OSRS caches yet, so hopefully they've fixed it there. At the moment, the archive doesn't deal with collisions particularly well - for colliding groups, it prefers the first variant imported. I'm thinking of ways to improve this, but it is tricky - deciding which copy of the file is correct for a particular version of the cache probably can't be automated. What I might end up doing is showing a list of colliding groups in the web interface and punting the problem to the users.

    A complete backup of the database is taken daily and made available over rsync/HTTP (rsync://archive.openrs2.org/pub/db/, [Only registered and activated users can see links. ]), in case anyone would like to maintain a mirror of the whole thing. This is much more efficient than downloading all the caches/keysets individually. The software you need to export caches from a mirrored copy is available at [Only registered and activated users can see links. ].

    I might also allow read-only public access to the underlying Postgres database in the future, but I've got to think a bit carefully about security and this would be subject to nobody abusing it (e.g. deliberately DoSing it with inefficient queries). This would allow complex queries not supported by the web interface (some data isn't exposed yet) and replication.

    I'll also be maintaining an archive of miscellaneous stuff (e.g. pre-JS5 caches) here: [Only registered and activated users can see links. ]. It's also available over rsync. It's a little bare at the moment, but more files will be added over time.

    So far in the main part of the archive I've imported OSRS/RS2 caches and keys present in Displee's/Polar's archives (so a lot of credit is due to them for doing such a good job of archiving stuff), as well as a few other sources. I'm still in the process of importing data (e.g. I haven't done any RS3 caches yet, but will soon, and I've got lots of old backup disks I need to sort through as well. I'll also trawl through old forum threads at some point too.)

    I'm very keen on receiving any old copies of the cache (unless it's bit for bit identical to a copy in Displee's/Polar's archive - if you're not sure, send it anyway, as different copies might have slightly different sets of available groups) or XTEA keys anyone might have hanging around. For XTEA keys I don't need to know the region IDs, revisions, or whether they're valid - I'd prefer just to receive a massive dump of all candidate keys you have, as the service automatically handles verification across a wide range of caches without trusting region IDs. Ideally the caches should be unmodified, and it'd be nice to know the build number or approximate timestamp (but this is not a requirement). Send me a PM on the forum or DM on Discord with a download link if you do have something to contribute.

    By pooling all our data in one place, the entire community will benefit from more complete caches and XTEA keysets, so I'm really hoping that people will see that and share rather than be selfish. For example, Sino was able to give me some XTEA keys for 550 I didn't have already, and in return my service had a few not present in the file he gave to me, so we both benefited from sharing keys.
    .
    Reply With Quote  
     


  2. #2  
    Registered Member

    Join Date
    Feb 2010
    Posts
    3,179
    Thanks given
    1,124
    Thanks received
    827
    Rep Power
    1465
    without a doubt the most important release to date for game integrity, I think I speak for more than myself when I say I appreciate the hard work that has gone into this

    also message me if you want to use this post for extra space I noticed you didn't reserve any.
    Reply With Quote  
     

  3. #3  
    🎶 As you're falling down 🎶

    uint32_t's Avatar
    Join Date
    Feb 2015
    Posts
    1,356
    Thanks given
    5,751
    Thanks received
    648
    Rep Power
    3758
    This is really cool to see, great work.
    [Only registered and activated users can see links. ]

    Quote Originally Posted by Idiot Bird View Post
    Quote Originally Posted by Velocity View Post
    lol np mate looks like the community brought ur rep down to ur IQ
    Not too sure about that, it's at 0 . It would have to go minus to even be remotely close to his IQ.
    Reply With Quote  
     

  4. #4  
    Banned

    Join Date
    Jun 2010
    Age
    23
    Posts
    4,840
    Thanks given
    1,676
    Thanks received
    1,563
    Rep Power
    0
    Seeing a release like this is a +100000 to my motivation.
    Great work, Graham! I'll be contributing to the pre-eoc archives as much as I can .



    This is a top goat release!
    Reply With Quote  
     

  5. #5  
    Community Veteran

    mige5's Avatar
    Join Date
    Aug 2008
    Posts
    5,321
    Thanks given
    546
    Thanks received
    1,127
    Rep Power
    1806
    I have some old caches, probably nothing u dont have there already though: [Only registered and activated users can see links. ]
    ...
    Reply With Quote  
     

  6. #6  
    SpawnScape Owner

    jet kai's Avatar
    Join Date
    Dec 2009
    Age
    25
    Posts
    708
    Thanks given
    315
    Thanks received
    265
    Rep Power
    2006
    My new home page!
    Spoiler for sig too large:

    [SPOIL]

    [/spoil]
    [Only registered and activated users can see links. ]
    Reply With Quote  
     

  7. Thankful user:


  8. #7  
    Programmer, Contributor, RM and Veteran




    Join Date
    Mar 2007
    Posts
    5,072
    Thanks given
    2,624
    Thanks received
    3,567
    Rep Power
    5000
    Quote Originally Posted by mige5 View Post
    I have some old caches, probably nothing u dont have there already though: [Only registered and activated users can see links. ]
    Difficult to know without importing them, so I'd be happy to import them and see if there is anything new if you send me a link to where I can download them (there doesn't seem to be a download link in the thread?)
    .
    Reply With Quote  
     

  9. #8  
    Banned

    Join Date
    Jun 2010
    Age
    23
    Posts
    4,840
    Thanks given
    1,676
    Thanks received
    1,563
    Rep Power
    0
    Quote Originally Posted by Graham View Post
    Difficult to know without importing them, so I'd be happy to import them and see if there is anything new if you send me a link to where I can download them (there doesn't seem to be a download link in the thread?)
    What value would caches that have been modified have?
    Say somebody found an old cache that has other data packed, would it be useful to still archive it?
    Reply With Quote  
     

  10. #9  
    Programmer, Contributor, RM and Veteran




    Join Date
    Mar 2007
    Posts
    5,072
    Thanks given
    2,624
    Thanks received
    3,567
    Rep Power
    5000
    Quote Originally Posted by Tyluur View Post
    What value would caches that have been modified have?
    Say somebody found an old cache that has other data packed, would it be useful to still archive it?
    Possibly useful, as it might still have unmodified groups that are missing from other copies of the cache. However, adding modified caches introduces its own risks as they'll increase the chance of (archive, group, checksum, version) collisions, so it's something I'd rather avoid if at all possible.
    .
    Reply With Quote  
     

  11. Thankful user:


  12. #10  
    Cywir RS3 - Coming soon!

    Scythe's Avatar
    Join Date
    Apr 2019
    Posts
    298
    Thanks given
    106
    Thanks received
    110
    Rep Power
    874
    Great idea Graham. Good stuff.

    The most ambitious RuneScape 3 project to date!
    Get ready for the ultimate Evolution of Combat experience.


    You can view the project thread
    [Only registered and activated users can see links. ]
    Reply With Quote  
     

Page 1 of 5 123 ... LastLast

Thread Information
Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)


User Tag List

Similar Threads

  1. My refactored Archive class
    By veer in forum Snippets
    Replies: 5
    Last Post: 07-17-2010, 06:23 AM
  2. GTA IV Archiving tool [PC]
    By Bobster in forum Application Development
    Replies: 4
    Last Post: 11-29-2008, 03:02 AM
  3. The HD client archive project
    By peterbjornx in forum Projects
    Replies: 10
    Last Post: 08-20-2008, 05:20 PM
  4. some file archiver reviews.
    By Unity in forum Downloads
    Replies: 1
    Last Post: 08-10-2008, 10:32 PM
Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •