Thread: Generate Reference Table Using Only Cache Indicies?

Page 1 of 2 12 LastLast
Results 1 to 10 of 11
  1. #1 Generate Reference Table Using Only Cache Indicies? 
    Community Veteran


    Join Date
    Dec 2008
    Posts
    4,263
    Thanks given
    405
    Thanks received
    436
    Rep Power
    1674
    I'm using runelites FlatStorage cache format (https://github.com/runelite/runelite...atStorage.java) which does not have a concept of index 255 or also known as the "ref table".

    In order to use the FlatStorage format with my js5 update server, I'll need to generate the entire reference table (index 255 archive 255) and all of its children reference tables (index 255 archives 0-20) from scratch. Is this possible using only the indices/archives?

    Here's what the FlatStorage cache looks like on disk:
    Attached image

    Here's what I got so far for generating the sub reference tables (index 255 and archives 0-20 or how ever many indices your cache has) but the data it gives me does not align with the data I have in the original binary cache (.dat2 file) (I assume because it does not include the main reference file 255/255 and the sectors are wrong). Does anyone have examples of generating these tables completely from scratch using the data only?

    Thanks.

    Code:
    	fun buildChildRefTables(indexId: Int, archiveId: Int, compressedData: ByteArray, refTable: ByteBuf): Int {
    		val writeBuffer = ByteArray(SECTOR_SIZE)
    		val data = ByteBuffer.wrap(compressedData)
    		
    		var sector: Int = ((refTable.readableBytes() + (SECTOR_SIZE - 1).toLong()) / SECTOR_SIZE.toLong()).toInt()
    		if (sector == 0) {
    			sector = 1
    		}
    		val startSector: Int = sector
    		
    		println("Starting sector is $sector")
    		
    		var part = 0
    		while (data.hasRemaining()) {
    			var nextSector = sector + 1 // we always just append sectors
    			var dataToWrite: Int
    			if (0xFFFF < archiveId) {
    				if (data.remaining() <= 510) {
    					nextSector = 0
    				}
    				writeBuffer[0] = (archiveId shr 24).toByte()
    				writeBuffer[1] = (archiveId shr 16).toByte()
    				writeBuffer[2] = (archiveId shr 8).toByte()
    				writeBuffer[3] = archiveId.toByte()
    				writeBuffer[4] = (part shr 8).toByte()
    				writeBuffer[5] = part.toByte()
    				writeBuffer[6] = (nextSector shr 16).toByte()
    				writeBuffer[7] = (nextSector shr 8).toByte()
    				writeBuffer[8] = nextSector.toByte()
    				writeBuffer[9] = indexId.toByte()
    				println("1. Seeking to ${SECTOR_SIZE * sector} $archiveId $part $nextSector $indexId ${writeBuffer.copyOf(10).contentToString()}")
    				refTable.writerIndex(SECTOR_SIZE * sector)
    				refTable.writeBytes(writeBuffer, 0, 10)
    				dataToWrite = data.remaining()
    				if (dataToWrite > 510) {
    					dataToWrite = 510
    				}
    			} else {
    				if (data.remaining() <= 512) {
    					nextSector = 0
    				}
    				writeBuffer[0] = (archiveId shr 8).toByte()
    				writeBuffer[1] = archiveId.toByte()
    				writeBuffer[2] = (part shr 8).toByte()
    				writeBuffer[3] = part.toByte()
    				writeBuffer[4] = (nextSector shr 16).toByte()
    				writeBuffer[5] = (nextSector shr 8).toByte()
    				writeBuffer[6] = nextSector.toByte()
    				writeBuffer[7] = indexId.toByte()
    				refTable.writerIndex(SECTOR_SIZE * sector)
    				println("2. Seeking to ${SECTOR_SIZE * sector} $archiveId $part $nextSector $indexId ${writeBuffer.copyOf(8).contentToString()}")
    				refTable.writeBytes(writeBuffer, 0, 8)
    				dataToWrite = data.remaining()
    				if (dataToWrite > 512) {
    					dataToWrite = 512
    				}
    			}
    			data[writeBuffer, 0, dataToWrite]
    			//println("3. We're at ${SECTOR_SIZE * sector} ${data.remaining()}")
    			refTable.writeBytes(writeBuffer, 0, dataToWrite)
    			sector = nextSector
    			++part
    		}
    		
    		return startSector
    	}
    Entire code can be found here:
    https://gist.github.com/Jonatino/9bb...38c45acf143be3

    You can use any osrs cache, here's the tool to convert the binary format to runelites FlatStorage format.

    Code:
    package com.pvphero.cache.packing.extractor
    
    import net.runelite.cache.fs.Store
    import net.runelite.cache.fs.flat.FlatStorage
    import java.io.File
    
    object CacheExtractor {
    	
    	@JvmStatic
    	fun main(args: Array<String>) {
    		val diskStore = Store(File("./osrs-cache"))
    		diskStore.load()
    		val out = File("./test-cache")
    		val flatStorage = FlatStorage(out)
    		flatStorage.save(diskStore)
    	}
    	
    }
    Last edited by Gluon; 01-15-2021 at 04:54 AM.
    Reply With Quote  
     

  2. Thankful user:


  3. #2  
    Registered Member
    Tyluur's Avatar
    Join Date
    Jun 2010
    Age
    26
    Posts
    5,103
    Thanks given
    1,818
    Thanks received
    1,767
    Rep Power
    2438
    Quote Originally Posted by Gluon View Post
    I'm using runelites FlatStorage cache format (https://github.com/runelite/runelite...atStorage.java)...

    In order to use the FlatStorage format with my js5 update server,
    Why? Whats the purpose in inventing a new cache format to do what the old one did? Am I misunderstanding something?
    Quote Originally Posted by blakeman8192 View Post
    Keep trying. Quitting is the only true failure.
    Spoiler for skrrrrr:

    Attached image
    Reply With Quote  
     

  4. #3  
    Community Veteran


    Join Date
    Dec 2008
    Posts
    4,263
    Thanks given
    405
    Thanks received
    436
    Rep Power
    1674
    Quote Originally Posted by Tyluur View Post
    Why? Whats the purpose in inventing a new cache format to do what the old one did? Am I misunderstanding something?
    Yes you are.
    Reply With Quote  
     

  5. Thankful users:


  6. #4  
    Registered Member
    Tyluur's Avatar
    Join Date
    Jun 2010
    Age
    26
    Posts
    5,103
    Thanks given
    1,818
    Thanks received
    1,767
    Rep Power
    2438
    Quote Originally Posted by Gluon View Post
    Yes you are.
    Are you here to flex that you spent time reading about someone else's code or are you going to expand?
    Quote Originally Posted by blakeman8192 View Post
    Keep trying. Quitting is the only true failure.
    Spoiler for skrrrrr:

    Attached image
    Reply With Quote  
     

  7. #5  
    Community Veteran


    Join Date
    Dec 2008
    Posts
    4,263
    Thanks given
    405
    Thanks received
    436
    Rep Power
    1674
    Quote Originally Posted by Tyluur View Post
    Are you here to flex that you spent time reading about someone else's code or are you going to expand?
    I've honestly never seen someone as insecure as you. Anything someone understands that you don't comes across as "flexing". If you think Jagex is sitting there working with a binary cache you're delusional. If you can't insinuate the use case from that then please stop posting as you're no help.
    Reply With Quote  
     

  8. Thankful users:


  9. #6  
    Respected Member


    Polar's Avatar
    Join Date
    Sep 2015
    Age
    28
    Posts
    420
    Thanks given
    0
    Thanks received
    418
    Rep Power
    3098
    Quote Originally Posted by Gluon View Post
    I've honestly never seen someone as insecure as you. Anything someone understands that you don't comes across as "flexing". If you think Jagex is sitting there working with a binary cache you're delusional. If you can't insinuate the use case from that then please stop posting as you're no help.
    Jagex actually does have their own binary cache format, they had the decoder for it in the TFU client. The gist of it was just a single file per archive with all of the data. Example would be client.config.pack which contains all/most of the config groups. In a sense their .pack format is probably somewhat similar to the flatcache format runelite has. But you would be right in that they most likely do not use the same format that the client uses.
    Reply With Quote  
     

  10. #7  
    Community Veteran


    Join Date
    Dec 2008
    Posts
    4,263
    Thanks given
    405
    Thanks received
    436
    Rep Power
    1674
    Quote Originally Posted by Polar View Post
    Jagex actually does have their own binary cache format, they had the decoder for it in the TFU client. The gist of it was just a single file per archive with all of the data. Example would be client.config.pack which contains all/most of the config groups. In a sense their .pack format is probably somewhat similar to the flatcache format runelite has. But you would be right in that they most likely do not use the same format that the client uses.
    Id assume jagex used some sort of extracted cache. Sprites as jpgs, models as workable model formats, etc. Thats how normal game development goes anyways I could be wrong. Binary is usually for distribution only.

    An "extracted" cache is my goal anyways. Easier to edit, visualize, index, and better for VCS. Tyluur is fine with binary because Dropbox is still his choice of VCS.

    Also would like to eventually get cs2 scripts DSL'ed as kts scripts. This is just the start to that project.

    Eventually everything will be bundled and packed by CI.
    Reply With Quote  
     

  11. #8  
    Respected Member


    Polar's Avatar
    Join Date
    Sep 2015
    Age
    28
    Posts
    420
    Thanks given
    0
    Thanks received
    418
    Rep Power
    3098
    Quote Originally Posted by Gluon View Post
    Id assume jagex used some sort of extracted cache. Sprites as jpgs, models as workable model formats, etc. Thats how normal game development goes anyways I could be wrong. Binary is usually for distribution only.

    An "extracted" cache is my goal anyways. Easier to edit, visualize, index, and better for VCS. Tyluur is fine with binary because Dropbox is still his choice of VCS.

    Also would like to eventually get cs2 scripts DSL'ed as kts scripts. This is just the start to that project.

    Eventually everything will be bundled and packed by CI.
    Based on the .pack format, I would say no. Looking up things (especially pre-ssd days) using individual files is extremely slow compared to just jumping around in an existing file that you have opened. I do believe they also have support for running based on source for development, but AFAIK they don't do that for production builds because of how much I/O that would be.

    Originally skimmed over your reply. They do what you said, everything is in individual files for version control. Once they're ready to deploy they build the "cache" (which is probably the .pack files I mention in my orig post on their side) which allows much faster lookups. The way I went about handling this, which we use for Zaros too, was essentially just applying things on top of the existing cache instead of trying to come up with a new format that version control would play nice with. I have a tool that loads a bunch of "source" files (sprites, interfaces, configs, etc) and packs them into the cache we choose to use, it also allowing overriding existing things if needed. This makes it so the only thing in our git repo is things that we actually modify, and the build task we have just links directly to an archive on my site.
    Last edited by Polar; 01-15-2021 at 10:32 AM.
    Reply With Quote  
     

  12. Thankful users:


  13. #9  
    Registered Member
    Tyluur's Avatar
    Join Date
    Jun 2010
    Age
    26
    Posts
    5,103
    Thanks given
    1,818
    Thanks received
    1,767
    Rep Power
    2438
    Quote Originally Posted by Gluon View Post
    I've honestly never seen someone as insecure as you. Anything someone understands that you don't comes across as "flexing". If you think Jagex is sitting there working with a binary cache you're delusional. If you can't insinuate the use case from that then please stop posting as you're no help.
    Did you figure it out yet?
    Quote Originally Posted by blakeman8192 View Post
    Keep trying. Quitting is the only true failure.
    Spoiler for skrrrrr:

    Attached image
    Reply With Quote  
     

  14. #10  
    Community Veteran


    Join Date
    Dec 2008
    Posts
    4,263
    Thanks given
    405
    Thanks received
    436
    Rep Power
    1674
    Quote Originally Posted by Polar View Post
    Based on the .pack format, I would say no. Looking up things (especially pre-ssd days) using individual files is extremely slow compared to just jumping around in an existing file that you have opened. I do believe they also have support for running based on source for development, but AFAIK they don't do that for production builds because of how much I/O that would be.

    Originally skimmed over your reply. They do what you said, everything is in individual files for version control. Once they're ready to deploy they build the "cache" (which is probably the .pack files I mention in my orig post on their side) which allows much faster lookups. The way I went about handling this, which we use for Zaros too, was essentially just applying things on top of the existing cache instead of trying to come up with a new format that version control would play nice with. I have a tool that loads a bunch of "source" files (sprites, interfaces, configs, etc) and packs them into the cache we choose to use, it also allowing overriding existing things if needed. This makes it so the only thing in our git repo is things that we actually modify, and the build task we have just links directly to an archive on my site.
    That's a good approach too, and probably works a bit nicer across cache revision updates since you're only ever adding stuff on rather than changing existing stuff.

    How do you handle the use case for custom sprites/configs you have with ID's that can be overwritten by new sprites/configs jagex adds. Eg you have a custom sprite with id 1001 and then jagex adds a new interface with a sprite that has ID 1001.

    Do you just add some arbitrary number to them? Eg your custom sprites are always ID + 100,000?
    Reply With Quote  
     

Page 1 of 2 12 LastLast

Thread Information
Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)


User Tag List

Similar Threads

  1. Replies: 35
    Last Post: 08-17-2011, 08:28 PM
  2. Use 377 cache with the 317 deob.
    By Tzar in forum Help
    Replies: 1
    Last Post: 07-19-2009, 12:26 AM
  3. The Reference Table structure
    By veer in forum Informative Threads
    Replies: 1
    Last Post: 06-07-2009, 08:48 PM
  4. Replies: 1
    Last Post: 03-08-2009, 09:02 AM
  5. Replies: 21
    Last Post: 10-29-2008, 03:19 PM
Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •