Thread: OpenRS2 Archive

Page 1 of 15 12311 ... LastLast
Results 1 to 10 of 144
  1. #1 OpenRS2 Archive 
    Programmer, Contributor, RM and Veteran




    Join Date
    Mar 2007
    Posts
    5,147
    Thanks given
    2,656
    Thanks received
    3,731
    Rep Power
    5000
    I've been working on this service for a while now, and it's now ready to be announced publicly (though there's still lots of work to do!)

    OpenRS2 Archive

    It's currently a collection of caches and XTEA keys for all versions of RuneScape that use JS5, and its precursor, to manage assets (mainline RuneScape build 234+ and all builds of OSRS). Support for archiving clients/native libraries is planned in the future.

    It automatically downloads new OSRS and RS3 caches when they are released and pulls OSRS XTEA keys from various sources.

    Unlike the existing archives, it's backed by a database - not a collection of static files. The groups are stored in a content-addressable format (based on (archive, group, checksum, version) tuples), which provides several benefits:

    * The disk usage is vastly reduced. Currently it uses 123 GB to store over 1,500 caches for a mixture of mainline RS and OSRS builds. The database backups (which are compressed and omit the indexes) are even smaller - around 48 GB at the moment. Storing all these caches individually would use over 8.3 TB!
    * Missing groups and keys can be located from nearby caches, as long as the checksum and version number are identical.

    The internal format is converted to .dat2/.idx on demand when a cache is downloaded. An alternative flat file format will be added for caches larger than 8GB (the max size of the .dat2 file) in the future.

    XTEA keys are automatically verified and they can be downloaded as a JSON file for every cache present in the archive. I'll add support for downloading a ZIP of text files in the future too. As many of the old dumpers didn't record the region ID for dynamic regions correctly, the service verifies every candidate key against every encrypted group for every version of the cache (and all future caches when they are imported). This allows us to eke out as many validated keys as possible.

    One known problem at the moment that I'm still fixing is that there are a small number of (archive, group, checksum, version) collisions (8 across the entire database at the time of writing). Some of the GZIP-compressed containers use the stored compression level (no compression). As GZIP has a CRC-32 trailer, any changes to the uncompressed data are actually cancelled out by the trailer (due to an unfortunate property of the CRC algorithm), rendering the overall CRC-32 checksum unchanged. Unfortunately, Jagex didn't increment the version number for these changes, and this would have in fact caused problems at the time: depending on when you first downloaded the cache, you might get an old or a new variant of the group. Users with the old variant of the group wouldn't receive updates. I can actually see this in the data I imported (e.g. some newer versions of the cache containing older variants of a colliding group, with others containing the new variant and no particular pattern to it). My suspicion is that Jagex only incremented the version number if they detected a change in the CRC-32 checksum, rather than comparing the uncompressed content (or a secure hash of it) directly. Luckily I haven't found any collisions in OSRS caches yet, so hopefully they've fixed it there. At the moment, the archive doesn't deal with collisions particularly well - for colliding groups, it prefers the first variant imported. I'm thinking of ways to improve this, but it is tricky - deciding which copy of the file is correct for a particular version of the cache probably can't be automated. What I might end up doing is showing a list of colliding groups in the web interface and punting the problem to the users.

    A complete backup of the database is taken daily and made available over rsync/HTTP (rsync://archive.openrs2.org/pub/db/, Index of /pub/db/), in case anyone would like to maintain a mirror of the whole thing. This is much more efficient than downloading all the caches/keysets individually. The software you need to export caches from a mirrored copy is available at openrs2/archive at master - openrs2 - Gitea.

    I might also allow read-only public access to the underlying Postgres database in the future, but I've got to think a bit carefully about security and this would be subject to nobody abusing it (e.g. deliberately DoSing it with inefficient queries). This would allow complex queries not supported by the web interface (some data isn't exposed yet) and replication.

    I'll also be maintaining an archive of miscellaneous stuff (e.g. RS2 beta/RSC caches) here: Index of /pub/. It's also available over rsync. It's a little bare at the moment, but more files will be added over time.

    So far in the main part of the archive I've imported OSRS/RS2 caches and keys present in Displee's/Polar's archives (so a lot of credit is due to them for doing such a good job of archiving stuff), as well as a few other sources. I'm still in the process of importing data (e.g. I haven't done any RS3 caches yet, but will soon, and I've got lots of old backup disks I need to sort through as well. I'll also trawl through old forum threads at some point too.)

    I'm very keen on receiving any old copies of the cache (unless it's bit for bit identical to a copy in Displee's/Polar's archive - if you're not sure, send it anyway, as different copies might have slightly different sets of available groups) or XTEA keys anyone might have hanging around. For XTEA keys I don't need to know the region IDs, revisions, or whether they're valid - I'd prefer just to receive a massive dump of all candidate keys you have, as the service automatically handles verification across a wide range of caches without trusting region IDs. Ideally the caches should be unmodified, and it'd be nice to know the build number or approximate timestamp (but this is not a requirement). Send me a PM on the forum or DM on Discord with a download link if you do have something to contribute.

    By pooling all our data in one place, the entire community will benefit from more complete caches and XTEA keysets, so I'm really hoping that people will see that and share rather than be selfish. For example, Sino was able to give me some XTEA keys for 550 I didn't have already, and in return my service had a few not present in the file he gave to me, so we both benefited from sharing keys.
    Last edited by Graham; 09-27-2023 at 01:12 PM.
    .
    Reply With Quote  
     


  2. #2  
    Registered Member

    Join Date
    Feb 2010
    Posts
    3,253
    Thanks given
    1,145
    Thanks received
    909
    Rep Power
    2081
    without a doubt the most important release to date for game integrity, I think I speak for more than myself when I say I appreciate the hard work that has gone into this

    also message me if you want to use this post for extra space I noticed you didn't reserve any.
    Reply With Quote  
     

  3. #3  
    🎶 As you're falling down 🎶


    uint32_t's Avatar
    Join Date
    Feb 2015
    Posts
    1,396
    Thanks given
    6,177
    Thanks received
    776
    Rep Power
    5000
    This is really cool to see, great work.
    Quote Originally Posted by Idiot Bird View Post
    Quote Originally Posted by Velocity View Post
    lol np mate looks like the community brought ur rep down to ur IQ
    Not too sure about that, it's at 0 . It would have to go minus to even be remotely close to his IQ.
    Reply With Quote  
     

  4. #4  
    Registered Member
    Tyluur's Avatar
    Join Date
    Jun 2010
    Age
    26
    Posts
    5,103
    Thanks given
    1,818
    Thanks received
    1,767
    Rep Power
    2438
    Seeing a release like this is a +100000 to my motivation.
    Great work, Graham! I'll be contributing to the pre-eoc archives as much as I can .

    Attached image

    This is a top goat release!
    Quote Originally Posted by blakeman8192 View Post
    Keep trying. Quitting is the only true failure.
    Spoiler for skrrrrr:

    Attached image
    Reply With Quote  
     

  5. #5  
    Community Veteran

    mige5's Avatar
    Join Date
    Aug 2008
    Posts
    5,528
    Thanks given
    573
    Thanks received
    1,410
    Rep Power
    2114
    I have some old caches, probably nothing u dont have there already though: https://www.rune-server.ee/runescape...archiving.html
    Number of page #1 releases with most views & posts: (Updated: 2023)
    RS2 server section: 1
    RS2 client section: 2
    Reply With Quote  
     

  6. #6  
    08-13, SpawnScape Owner

    jet kai's Avatar
    Join Date
    Dec 2009
    Age
    28
    Posts
    870
    Thanks given
    630
    Thanks received
    957
    Rep Power
    5000
    My new home page!
    Reply With Quote  
     

  7. Thankful user:


  8. #7  
    Programmer, Contributor, RM and Veteran




    Join Date
    Mar 2007
    Posts
    5,147
    Thanks given
    2,656
    Thanks received
    3,731
    Rep Power
    5000
    Quote Originally Posted by mige5 View Post
    I have some old caches, probably nothing u dont have there already though: https://www.rune-server.ee/runescape...archiving.html
    Difficult to know without importing them, so I'd be happy to import them and see if there is anything new if you send me a link to where I can download them (there doesn't seem to be a download link in the thread?)
    .
    Reply With Quote  
     

  9. #8  
    Registered Member
    Tyluur's Avatar
    Join Date
    Jun 2010
    Age
    26
    Posts
    5,103
    Thanks given
    1,818
    Thanks received
    1,767
    Rep Power
    2438
    Quote Originally Posted by Graham View Post
    Difficult to know without importing them, so I'd be happy to import them and see if there is anything new if you send me a link to where I can download them (there doesn't seem to be a download link in the thread?)
    What value would caches that have been modified have?
    Say somebody found an old cache that has other data packed, would it be useful to still archive it?
    Quote Originally Posted by blakeman8192 View Post
    Keep trying. Quitting is the only true failure.
    Spoiler for skrrrrr:

    Attached image
    Reply With Quote  
     

  10. #9  
    Programmer, Contributor, RM and Veteran




    Join Date
    Mar 2007
    Posts
    5,147
    Thanks given
    2,656
    Thanks received
    3,731
    Rep Power
    5000
    Quote Originally Posted by Tyluur View Post
    What value would caches that have been modified have?
    Say somebody found an old cache that has other data packed, would it be useful to still archive it?
    Possibly useful, as it might still have unmodified groups that are missing from other copies of the cache. However, adding modified caches introduces its own risks as they'll increase the chance of (archive, group, checksum, version) collisions, so it's something I'd rather avoid if at all possible.
    .
    Reply With Quote  
     

  11. Thankful user:


  12. #10  
    Extreme Donator


    Join Date
    Apr 2019
    Posts
    332
    Thanks given
    140
    Thanks received
    167
    Rep Power
    1248
    Great idea Graham. Good stuff.
    Reply With Quote  
     

Page 1 of 15 12311 ... LastLast

Thread Information
Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)


User Tag List

Similar Threads

  1. My refactored Archive class
    By veer in forum Snippets
    Replies: 5
    Last Post: 07-17-2010, 06:23 AM
  2. GTA IV Archiving tool [PC]
    By Bobster in forum Application Development
    Replies: 4
    Last Post: 11-29-2008, 03:02 AM
  3. The HD client archive project
    By peterbjornx in forum Projects
    Replies: 10
    Last Post: 08-20-2008, 05:20 PM
  4. some file archiver reviews.
    By Unity in forum Downloads
    Replies: 1
    Last Post: 08-10-2008, 10:32 PM
Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •