Thread: [EXTREMELY DETAILED][COLORCODED] How to pack multiple files in one file(Custom Cache)

Purpose: Teach reader how to make their own simple cache writing and reading system.
Difficulty: 6/10
Assumed Knowledge: Standard Java Structure, I Explain everything else in-depth
Classes Modified: CachePacker.java, CacheFile.java, CacheUnpacker.java
Refactored/Non-Refactored?: Either

In this tutorial I will be explaining in depth how to make a simple, yet effective system for packing multiple files into a single file. And then reading from that file to obtain the multiple files. The biggest advantage of using this for a RSPS client is that instead of having thousands of model and animation files which will bump up compressing, decompressing size and downloading time you can have a single file which contains all of files, and can be easily parsed during runtime.

Step 1: What kind of properties are my cached files going to need?
Well the first thing you need to do is figure out what kind of properties each file in your cache is going to need. The most common uses I could think of is the path the file would be located in within the cache. On top of these properties you will also need the byte[] Array which contains the file in byte format, which can be read by many different Java IO readers as well as a boolean which indicates whether or not to keep the file cached after loading, which may help to clear up RAM using during runtime(Always a priority)

Step 2: Creating a class to represent a cache file.
So know that you have decided what kind of properties you will need to hold for each file, you need to create a class which will represent each file within the cache and hold each of those properties for each file.

Here is my template for this class: (Contains the essentials for this usage of the class)

Code:

import java.io.ByteArrayInputStream;

public class CacheFile {

	private boolean store;
	private byte[] bytes;
	
	public CacheFile(boolean store, Arguments Here, byte[] bytes)
	{
		this.store = store;
		this.bytes = bytes;
	}

	public boolean getStore()
	{
		return store;
	}
	
	public byte[] getBytes()
	{
		return bytes;
	}
		
	public ByteArrayInputStream getNewInputStream()
	{
		return new ByteArrayInputStream(bytes);
	}
}

Here is my finished CacheFile class after I have adjusted it to support my properties of after loading storage and the path within the cache the file is located in.

Code:

import java.io.ByteArrayInputStream;

public class CacheFile {
	
	private boolean store;
	private String path;
	private byte[] bytes;
	
	public CacheFile(boolean store, String path, byte[] bytes)
	{
		this.store = store;
		this.path = path;
		this.bytes = bytes;
	}

	public boolean getStore()
	{
		return store;
	}
	
	public String getPath()
	{
		return path;
	}
	
	public byte[] getBytes()
	{
		return bytes;
	}
		
	public ByteArrayInputStream getNewInputStream()
	{
		return new ByteArrayInputStream(bytes);
	}
}

Step 3: An overview of the cache packing system you will use.
Ok so the way we are going to pack multiple files into a single cache is by first, getting the amount of files that are in the cache and writing that into the file. This way the cache unpacker can know how many files are in the cache without any troubles.

Then, for each file we are going to write, we ill write the following:
- ID : The id of the file being written, starting at 0 and incrementing for each file
- Any Properties : For each custom property the CacheFile class contains
- Bytes : The file in byte array form, which as stated earlier can be read by many of the Java IO readers.

Then we will simply make sure all the data is saved to the file and close it off. Done! The cache will then be in a single file.

Step 4: Tackling the structure of your cache packer
Now that you have an idea of how the cache packer is going to operate, you are going to have to create your CachePacker class(or whatever you chose to call it). Following this you will need to set up your imports and 3 static methods: One to take in a output file, a list of files to pack and a optional boolean of whether to stop overwriting, One to take in a byte array and write both its size and contents to the file and finally One to convert any file into a byte[] array.
Make sure to read the comments!

Code:

import java.io.DataOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.List;

public class CachePacker {
	
	// Because it is easy to put all of this in one method,
	// I have done so. I have also made it a static method
	// so that you do not need to initiate an instance of
	// this class to be able to use this method.
	public static void createCache(String cacheFile, List<String> files,
			 List<Boolean> stores, List<YourProperty> YourProperties, 
			 boolean... overwrite)
	{
	}
	
	// My method for writing bytes to the stream
	// in a way which the reader will be able to
	// parse into a byte[] array.
	private static void writeBytes(byte[] bytes, DataOutputStream out) throws IOException
	{
		out.writeShort(bytes.length);
		out.write(bytes);
	}
	
	// Simple method to read all of the bytes out of a file
	// then return with those bytes in an array.
	private static byte[] getBytes(String file) throws IOException
	{
		File f = new File(file);
		byte[] fileBytes = new byte[(int)f.length()];
		new FileInputStream(f).read(fileBytes);
		return fileBytes;
	}

}

Well you are going to notice I have highlighted a few things in this template, here are the explainations:
1. "boolean... overwrite" - Some of you will be thinking "What is this, what does it do?", so to answer that if you are asking it. When you add "..." to the end of a Class type as an argument in Java; it can make the argument optional(If it is the last argument), but is designed to make it so that you can enter multiple arguments of that Class type when executing the method, then Java will automatically turn those arguments into a Class[] array of the given type. (i.e. the method "doSomething(int anArg, String... others)" can be executed via "doSomething(1, "string1", "string2", "string3", "etc.");"

2. "List<YourProperty> YourProperties" - Remember back in step 1 you decided what properties you were going to need for your cached files, and in step 2 you implemented those properties into your CacheFile class? Well now you have to implement them into your Cache Packer, this is rather simple however. For each custom property you have, add a new List argument to the createCache method (i.e. List<Integer> prices, List<String> nicknames)

3. "out.writeShort(bytes.length)" - Your probably wondering why this is highlighted, don't worry you don't have to change anything. This is simply just to give you a tip on these sorts of things if you don't already know it. When you are using bytes to store data, eventually the amount of space the bytes(even though they are small) take up will add up. So, when using bytes to store data you should always be aware of what type of variable you are using to store data in. In this example, even though "bytes.length" is an Integer, I have it being written as a short because I am not expecting the length of that array to exceed that a Short can hold. And because an Integer takes up 4 bytes, while Shorts take up two, I am saving space(Once again it adds up!) by using a Short instead of an Integer(Hope that explains that)

Ok so now that you've hopefully read through that you should have your createCache method looking something like this:

Code:

	public static void createCache(String cacheFile, List<String> files,
			 List<Boolean> stores, List<Integer> somevalue,
			boolean... overwrite)
	{
	}

Though because my extra property is the file path, it is already given in the files List and I don't need to add any extra arguments.

Step 5: Checking and opening our cache for writing
Now just some fairly standard procedures to add to our method here, the first bit:

Code:

		// Create the File class associated with the
		// output file for the cache.
		File file = new File(cacheFile);

		// Check if our overwrite argument has been
		// given a value by checking length, then
		// checking if the first (presumably only)
		// value is true. If then the file exists
		// a new Exception will be created, traced
		// and then the method will be stopped.
		if (overwrite.length > 0)
		{
			if (overwrite[0])
			{
				if (file.exists())
				{
					new Exception("File already exists").printStackTrace();
					return;
				}
			}
		}

		// Standard try catch exception with the creation of a new
		// DataOutputStream using our previously instanced file.
		try
		{
			DataOutputStream out = new DataOutputStream(new FileOutputStream(file));
		} catch (Exception e)
		{
			e.printStackTrace();
		}

So to just summarize on all those comments, we first simply create an instance of File giving it the location of the cacheFile argument, then proceed to check if we aren't aloud to overwrite an existing file and finally create a standard TryCatch statement which opens a new DataOutputStream to write in.

Step 5: Writing files to the DataOutputStream
Ok so by now you should have a good idea of what's going on, so I'm not going to spend much time elabourating on the comments. Add this underneath the line creating a new DataOutputStream:

Code:

			// Write the amount of files going to be packed (Length of File List)
			out.writeInt(files.size());
			
			// Get the base directory for all of the cache files after making sure
			// the slashes are in sync and initialize fileID
			cacheFile = cacheFile.replaceAll("/", "\\");
			String cacheDirectory;
			if (cacheFile.contains("\\"))
			{
				cacheDirectory = cacheFile.substring(0, cacheFile.lastIndexOf("\\"));
			} else
			{
				new Exception("CacheFolder not seperate").printStackTrace();
				out.close();
				return;
			}
			int fileID = 0;
			
			// For each file listed in the File List,
			// give it an ID, set it's name and whether
			// or not it should be stored. As well as
			// get the file contents as a byte[] and
			// write that.
			for (int i=0; i<files.size(); i++) {
				out.writeInt(fileID++); // Write the file id then increment it by 1
				// If we have a list indicating whether to store
				// each file or not go by that, other wise go
				// off of a default value.
				if (stores != null)
					out.writeBoolean(stores.get(i));
				else
					out.writeBoolean(false);
				String filePath = files.get(i);
				// BEGIN CUSTOM PROPERTIES
				writeBytes(filePath.getBytes(), out);
				// END CUSTOM PROPERTIES
				if (!cacheDirectory.endsWith("\\"))
					cacheDirectory += "\\";
				writeBytes(getBytes(cacheDirectory + filePath), out);
			}

You guessed it right(I hope), after reading through the comments you should have assumed that you replace the blue code with that appropriate to the properties you decided to create.

To help you out heres a small list of common variables you can write:

Code:

	out.writeInt(INTEGER);
	out.writeBoolean(BOOLEAN);
	out.writeShort(SHORT);
	out.writeLong(LONG);
	out.writeDouble(DOUBLE);
	writeBytes((STRING).getBytes(), out); // Using my method

If you want to add more flexibility with the extra lists you added, use this template for them. This way you can set a List to null and all files will be given a default value.

Step 6: Flushing and Closing our cache
Now a the easiest part of all, flushing and closing our DataOutputStream. Add this just before the end of the try brackets:

Code:

			// And finally we flush the stream to make sure
			// all the data we have wrote to the stream gets
			// put in the file and close it so that other
			// applications may open the file for reading/writing.
			out.flush();
			out.close();

And now your AlMOST THERE!!!. Now you just need to adjust my class to read your cache.

Spoiler for Heres what your CachePacker class should look like:

Code:

mport java.io.DataOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.List;

public class CachePacker {
	
	// Because it is easy to put all of this in one method,
	// I have done so. I have also made it a static method
	// so that you do not need to initiate an instance of
	// this class to be able to use this method.
	public static void createCache(String cacheFile, List<String> files,
			 List<Boolean> stores, 
			 boolean... overwrite)
	{
		// Create the File class associated with the
		// output file for the cache.
		File file = new File(cacheFile);

		// Check if our overwrite argument has been
		// given a value by checking length, then
		// checking if the first (presumably only)
		// value is true. If then the file exists
		// a new Exception will be created, traced
		// and then the method will be stopped.
		if (overwrite.length > 0)
		{
			if (overwrite[0])
			{
				if (file.exists())
				{
					new Exception("File already exists").printStackTrace();
					return;
				}
			}
		}

		// Standard try catch exception with the creation of a new
		// DataOutputStream using our previously instanced file.
		try
		{
			DataOutputStream out = new DataOutputStream(new FileOutputStream(file));
			// Write the amount of files going to be packed (Length of File List)
			out.writeInt(files.size());
			
			// Get the base directory for all of the cache files after making sure
			// the slashes are in sync and initialize fileID
			cacheFile = cacheFile.replaceAll("/", "\\");
			String cacheDirectory;
			if (cacheFile.contains("\\"))
			{
				cacheDirectory = cacheFile.substring(0, cacheFile.lastIndexOf("\\"));
			} else
			{
				new Exception("CacheFolder not seperate").printStackTrace();
				out.close();
				return;
			}
			int fileID = 0;
			
			// For each file listed in the File List,
			// give it an ID, set it's name and whether
			// or not it should be stored. As well as
			// get the file contents as a byte[] and
			// write that.
			for (int i=0; i<files.size(); i++) {
				out.writeInt(fileID++); // Write the file id then increment it by 1
				// If we have a list indicating whether to store
				// each file or not go by that, other wise go
				// off of a default value.
				if (stores != null)
					out.writeBoolean(stores.get(i));
				else
					out.writeBoolean(false);
				String filePath = files.get(i);
				// BEGIN CUSTOM PROPERTIES
				writeBytes(filePath.getBytes(), out);
				// END CUSTOM PROPERTIES
				if (!cacheDirectory.endsWith("\\"))
					cacheDirectory += "\\";
				writeBytes(getBytes(cacheDirectory + filePath), out);
			}

			// And finally we flush the stream to make sure
			// all the data we have wrote to the stream gets
			// put in the file and close it so that other
			// applications may open the file for reading/writing.
			out.flush();
			out.close();
		} catch (Exception e)
		{
			e.printStackTrace();
		}
	}
	
	// My method for writing bytes to the stream
	// in a way which the reader will be able to
	// parse into a byte[] array.
	private static void writeBytes(byte[] bytes, DataOutputStream out) throws IOException
	{
		out.writeShort(bytes.length);
		out.write(bytes);
	}
	
	// Simple method to read all of the bytes out of a file
	// then return with those bytes in an array.
	private static byte[] getBytes(String file) throws IOException
	{
		File f = new File(file);
		byte[] fileBytes = new byte[(int)f.length()];
		new FileInputStream(f).read(fileBytes);
		return fileBytes;
	}

}