Automatic Cache Updater/Downloader - Thoroughly Explained - Using CRCs
Before we begin, if you were counting on myself picking up a spoon, scooping up some baby-food, and sticking it into your mouth, then please leave and come back when you're ready to LEARN. If you do not understand something, I want you to post and we will ALL solve your problem together. I expect no harrassment, and we will all work together to overcome any difficulties on this thread.
Table of Contents
1. Introduction/Summary
2. Helpful Before-Hand Tips (A MUST READ)
3. Checking for a Working Download
4. Checking if an Update is Required
5. Grabbing the Cache
6. Extracting Your Updated Cache
1. Introduction/Summary
Ever wanted to accomplish something useful on your client? Wanted to do it yourself with little assistance? You've come to the right place. Today I will teach you how to add an automatic cache updater and downloader. What will it do, you ask? Simple. On startup, we will have a list of URLs ready, each containing a copy of the cache you want your client to use. We will look at each URL, checking for a valid download, and if so, we will begin the work there. We will perform a Cyclic Redundancy Check, or CRC for short, and compare the uploaded cache's checksum with your current cache's checksum. If they do not match, then we will have to download that newer copy of your cache. Ready? Great, let's get started!
2. Helpful Before-Hand Tips (A MUST READ)
The following articles/links/guides will help explain everything before and after reading this tutorial. They will help explain anything you were confused on or got lost on, and should solve many, many of your questions. If one of your questions' answers can be found in one of these links, I will tell you to look at step #2 and that's pretty much it. Dont forget, more specific articles are given out throughout the tutorial. So here they are:
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]
3. Checking for a Working Download
Now we are going to loop through a number of URLs, allowing you to choose and add to that list, and check for a working and valid download. Take a look at the following snippet:
Code:
private String allMirrors[] =
{
"http://arandomdownloadsite.com/mirrored_cache.zip",
"http://anotherdownloadsite.com/mirrored_cache.zip",
"http://justgoogleit.com/mirrored_cache.zip"
};
Now lets take an even closer look. What does it do? Well, all it simply does it create a new array with the String data-type, and holds (at the moment) three URLs we could use to download the cache from. This could reduce some bandwidth usage on your primary host, etc. But, speaking technical-Java-wise now, these arent official URL instances. Dont worry though, we will get into that soon. (If at the moment you are confused about ARRAYS, please read: [Only registered and activated users can see links. Click Here To Register...] , or more specifically, [Only registered and activated users can see links. Click Here To Register...] . If you are confused about the String type, please take a look at [Only registered and activated users can see links. Click Here To Register...])
So lets move on. We have our list of URLs: what should we do now? Obviously, we will have to check for a working link. To do this, we will make use of the HttpURLConnection and URL classes located in the Java API under the java.net package. Take a look at this:
Code:
myLoop: for(String tempMirror : allMirrors)
{
HttpURLConnection ourConnection = null;
try
{
ourConnection = ((HttpURLConnection)new URL(tempMirror).openConnection());
}
catch(Exception e) { e.printStackTrace(); }
switch(ourConnection.getResponseCode())
{
case 200:
//Authorized and able to communicate
break myLoop;
case 401:
//401 Reponse Code - Unauthorized
break;
default:
//Any other response code we might not have covered
break;
}
}
Whoah! What a huge chunk of code. Dont like it? Get over it ;) It's time you learn what it does. And to do this, we'll break it up piece by piece.
Take a look at the first line, the loop statement. I have put a label on it ([Only registered and activated users can see links. Click Here To Register...] for more info.) named myLoop. We'll talk about that soon enough, and why I decided to put one there. The loop statement I put in there is designed specifically for arrays (which you should have read up on earlier in the tutorial). It will loop through every value in the array, one-by-one, and assigning the value to the temporary variable tempMirror. That way, when inside of the loop, we can use that variable to refer to that value in the array. There are plenty of other ways to loop through an array, but there are other ways to figure that out (*hint* [Only registered and activated users can see links. Click Here To Register...] *hint*).
Next, I declared the variable ourConnection. It holds a HttpURLConnection type (more info. on that could be found at [Only registered and activated users can see links. Click Here To Register...]), and will serve its purpose later on. We then, inside of a try block to stop any unintended halts in the program, assign a value to that variable by opening up a new connection using the tempMirror variable we were currently working with. Now that we have everything set up, lets check for a working connection.
To do this, we will use the switch statement. We create a new switch statement, using our HttpURLConnection assigned earlier, and get the response code. This will allow us to check what kind of response we are dealing with from the server. For example, a 401 would mean Unauthorized, a 404 would mean Not Found, and 200 would mean OK (a list of these can be found in the JavaDocs at [Only registered and activated users can see links. Click Here To Register...]) We only have a real need to handle the 200 response, so that's exactly what we'll do. We'll work on filling that part in later, and right now I want to explain the break myLoop; statement. Why didnt we just use the normal break;, you might be wondering? Well, if we used the normal break and still found a working connection, it wouldnt stop from continuing to the next value in the array. But we WANT it to stop, and work with the connection we found working.
4. Checking if an Update is Required
Now that we have found our working link, we'll have to do something with it. And that "something" is exactly what we'll do. We're about to check if an update of the cache is required. But before that, we'll have to check if we even contain a copy of that cache. So lets look at this:
Code:
String cacheFiles[] =
{
"main_file_cache.dat",
"main_file_cache.idx0",
"main_file_cache.idx1",
"main_file_cache.idx2",
"main_file_cache.idx3",
"main_file_cache.idx4",
"worldmap.dat",
"code.dat",
"uid.dat"
};
Right away, this should look familiar. If you can already identify its purpose, than congratulations. You've learned something. If you haven't, then all this does is give us a list of file names to check through. It's simply a String array containing all the file names that should belong in our cache. This'll help us later on in our program.
Code:
File cacheDirectory = new File(sign.signlink.findcachedir());
if(!cacheDirectory.isDirectory())
{
cacheDirectory.mkdir();
grabCache(); //We'll add this soon.
}
This next snippet of code is almost self-explanatory. File, a class already given to us in the Java API under the java.io package, will come in handy on this one. We have created a new File instance ([Only registered and activated users can see links. Click Here To Register...] for more info.), using the cache directory that is used in the signlink class (found in most clients; you can manually type it in, but this works too). We check if the File is a directory using the isDirectory() method found in the File class, and if not, we create it. But you see, if it is NOT a directory, then that means our cache is not properly placed. Therefore, we must grab a new copy of the cache. (We'll talk about that later on in this tutorial. Be patient. We're still going to check for an update.)
Code:
for(String tempFile : cacheFiles)
if(!new File(cacheDirectory.toString()+"/"+tempFile).exists())
grabCache(); //We'll add this soon.
else
{
//Time for our CRC check to check for updates!
//I'll be assuming the files uploaded on the webhost are in a ZIP Archive!
//While the files in our cache directory are not!
//Will hold our CRC Checksum values for us.
long currentCRC, newCRC;
//Will contain the bytes of our current file - used for CRC check.
byte currentCRCBytes[] = null;
try
{
FileInputStream inStream = new FileInputStream(cacheDirectory.toString()+tempFile);
currentCRCBytes = new byte[inStream.available()]; //Gives us the amount of bytes available for reading
inStream.read(currentCRCBytes, 0, currentCRCBytes.length); //Reads the data and outputs it into our array currentCRCBytes[]
inStream.close();
}
catch(Exception e) { e.printStackTrace(); }
//Perform CRC check for our current cache file.
CRC32 curCRCCheck = new CRC32();
curCRCCheck.update(currentCRCBytes);
currentCRC = curCRCCheck.getValue();
//Begin the CRC process again for new cache file.
//Will be different as we will be checking while its on the web.
//Assuming its also in a ZIP Archive, so things get more complicated.
byte newCRCBytes[] = null; //Holds the data of our new cache file (located on web)
URI fileURI = new URI(ourWorkingMirror);
File newFile = new File(fileURI);
try
{
ZipInputStream inStream = new ZipFile(newFile).getInputStream();
ZipEntry ze = null;
while((ze = inStream.getNextEntry()) != null
&& !ze.getName().equals(tempFile.substring));
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[10000]; int len = -1;
while((len = inStream.read(buffer, 0, buffer.length)) != -1)
baos.write(buffer, 0, len);
newCRCBytes = baos.toByteArray();
}
catch(Exception e) { e.printStackTrace(); }
//Now that we have gotten the data, or bytes, of our new cache file, we can work with it.
//But first, we have to check if no data was found. Because if so,
//That indicates there was no file matched in that ZIP Archive, meaning a new file has been added.
if(newCRCBytes == null)
{
grabCache(); //An update IS required.
break; //Stop looping through archive, we already know an update is needed.
}
//If there has been data written to the array, then we shall compare it to our current file's.
//To do this, we must obtain another CRC checksum, then compare the two we have gotten.
CRC32 newCRCCheck = new CRC32();
newCRCCheck.update(newCRCBytes);
newCRC = newCRCCheck.getValue();
//Starting comparison...
if(currentCRC != newCRC) //Checking both values
{
grabCache(); //An update IS required.
break; //Stop looping through files as we already found an update is needed.
}
//If the JVM has reached this point, then an update is not yet required.
//Therefore, we simply continue searching through the cache files.
}
Oh boy, I bet most of you just pissed your pants. There is no way we need that huge chunk of code, right? All we need are those crappy copy and paste tutorials found on some external websites I am not allowed to advertise, am I correct? No. The above snippet is very useful, and does many things for us. I will now explain what does what, how, and why we did so. (If you get lost, try searching for a similar explanation found in the comments I added in the snippet above.)
Lets start with the for loop statement. Why on Earth should we have that? The answer is quite obvious. We need to check through every value in the array, like explained earlier in the tutorial, and each time around we assign one of the cache's file's names to the variable we set as tempFile.
Next, we check if that file even exists. If it does not, then of course an update is required. So we proceed to download the cache (details later on in the tutorial). But, if the file DOES exist, we perform a procedure that to most of you will be difficult to understand, and most importantly new. New is important, because it is a door to learning. So, I suggest, before we continue, you take a look at (hopefully READ) the following articles:
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]
After reading those articles (which without reading you will be lost), we proceed to check if that individual file requires some sort of an update using the CRC32 class found in the Java API under the java.util.zip package (which you should have read about *cough*). But in order to get a value for that CRC check, we will need to retrieve the data, or sequence of bytes/byte array, of the file. To do this, I have constructed a simple method of doing so using the FileInputStream class (found at [Only registered and activated users can see links. Click Here To Register...]).
Now we simply retrieve the data, and create a new instance of the CRC32 class, and update the CRC32 stream with the byte array. Now that we have everything setup, we can retrieve the checksum using the CRC32.getValue() method (returns a long data-type) and assign that value to long currentCRC which we declared earlier in that section.
Alright, so we have our first CRC checksum. Problem is, we will have to do it again, but this time reading it off of the web, and reading individual files from a ZIP archive (assuming your cache is compressed into ZIP format and uploades as a ZIP archive). No big deal, though, so dont worry about it. Lets continue.
Next we create a new File instance (you should have read up on that earlier) using an instance of the URI class (found in the java.net package, and JavaDoc located at [Only registered and activated users can see links. Click Here To Register...]). Now using the ZipFile, ZipEntry, ZipInputStream, and ByteArrayOutputStream classes in the Java API (links located at end of this paragraph), we are able to read through the archive. We open up a new InputStream instance using the getInputStream() method found in the ZipFile class, and we start to read from it. We loop through the entries in the archive, assigning each entry to our own ZipEntry variable, and retrieving the name of each entry. This way, we can check if the name of our current cache file matches the zip entry's file name, and if so, we can retrieve the data, or byte array, of that compressed file. To do that, we use a buffer and read from the entry. (Info. on using a buffer can be found at [Only registered and activated users can see links. Click Here To Register...])
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]
Okay, cool. We have our neew file's data. What now? Well, firstly, we must check if any data was even assigned to our byte array, because if not, that indicates there was no match. Thats exactly what we did with the if statement, checking for a null value, and if there was one we would begin grabbing the cache as we obviously do not have the most up-to-date file list. But if there WAS data assigned to the variable, then it would be safe to continue with out process.
So once we get the 'all-clear' sign, we can proceed to performing another CRC check. This works exactly the same way it did earlier in the tutorial, so if you were lost simply re-read that section. Now once we have retrieved both CRC checksums, we can check if they match. Because if they dont, an update would be required as something has changed. And that's exactly what we did using another simple if statement. If they didnt match, we would start grabbing the new cache. If they DID match, we would allow the loop statement to continue iterating through the list of file names. Congratulations, you just passed the hardest part!
5. Grabbing the Cache
Okay, so we know its time to grab the cache. This is the final step in the puzzle, and it is probably the easiest one, too. All we have to do is read from the data on the server and write it to our local machine. It may sound hard, but it's not. So sit back, relax, pay attention, and most importantly, READ what I'm writing.
Code:
private void grabCache()
{
int bytesRead;
try
{
//Set the variable yourself (dont hardcore it, make it use the one that we found working)
URL yourCache = new URL(urlWeAssignedEarlier);
//Uses cache directory set in the signlink class.
BufferedOutputStream outStream = new BufferedOutputStream(sign.signlink.findcachedir());
//The input stream we will read from.
InputStream inStream = yourCache.openConnection().getInputStream();
//Only read 1024 bytes at a time.
byte ourBuffer[] = new byte[1024];
//Loop through all data, reading & writing only 1024 bytes at a time according to our buffer.
while((bytesRead = inStream.read(ourBuffer)) != -1)
outStream.write(ourBuffer, 0, bytesRead);
//Cleanup. Closed both the input stream and the output stream.
inStream.close();
outStream.close();
}
catch(Exception e) { e.printStackTrace(); }
}
Oh no, once again we find a huge chunk of code that your eyes are not used to seeing compared to the itsy bitsy lines in those copy and paste tutorials. But that's okay, its better for you in the long run. What have we done this time? Well, lets start with the declaration of the bytesRead variable. Its there to keep track of how many bytes we have read, and is used when reading from the input stream later on.
Next we setup a URL instance ([Only registered and activated users can see links. Click Here To Register...]) that will help us open up a connection to your file soon enough. We then setup a BufferedOutputStream to write the data we read to. It writes to the path of the directory obtained from the signlink class under the findcachedir() method found in most of your clients. It can be changed manually, if you would like (More info. found under [Only registered and activated users can see links. Click Here To Register...]).
Now that we have the output stream ready, we'll need something to read from as well. And to accomplish this task, we create a new instance of the InputStream class (more info. found at [Only registered and activated users can see links. Click Here To Register...]). But not just using its constructor, we do it in a somewhat special way. We first open a connection from the URL assigned earlier, creating a URLConnection instance (info. at [Only registered and activated users can see links. Click Here To Register...]) and then retrieving our input stream using the getInputStream() method found in that class.
Lets begin reading and writing. We created a buffer from a byte array, holding a max of 1024 bytes at a time (preventing possible problems later on), and used it. How? We used a 'while loop' to loop through the data in the input stream and we read only 1024 bytes at a time from it. At the same time, we wrote those 1024 bytes to our output stream, creating a part of that cache that you have long awaited. After its all done reading and writing, we clean up the two streams using the close() methods found in their corresponding classes. And that's that. We're officially done downloading.
6. Extracting Your Updated Cache
Great. Now we have the ZIP archive, but we to be able to use the actual content inside of it. You've already partially worked with the java.util.zip package in this tutorial, and that'll help you a little when it comes to this next part. I may say next, but this is also the last. Here's a little something I've had for a while:
Code:
private void extractAll(String targetDest)
{
int amountRead;
try
{
byte[] buf = new byte[1024];
ZipInputStream inStream = ZipFile(newFile).getInputStream(); //Set this variable yourself.
ZipEntry tempEntry = inZip.getNextEntry();
while(tempEntry != null)
{
String entryName = tempEntry.getName();
File newFile = new File(entryName);
String theDirectory = newFile.getParent();
if(theDirectory == null)
if(newFile.isDirectory())
{
new File(targetDest+"/"+entryName).mkdir();
continue;
}
FileOutputStream outStream = new FileOutputStream(targetDest+entryName);
while((amountRead = inStream.read(buf, 0, 1024)) > -1)
outStream.write(buf, 0, amountRead);
outStream.close();
inStream.closeEntry();
tempEntry = inStream.getNextEntry();
}
inZip.close();
}
catch (Exception e) { e.printStackTrace(); }
}
As you can see, this looks very similar to when the last time we worked with the java.util.zip package. And, infact, it is. Everything has almost the exact same format/layout, but only with a few differences. Instead of writing to a ByteArrayOutputStream, it writes to a new file using the FileOutputStream class in the java.io package.
We start off by opening a new ZipInputStream, the same way as last time. Then we get the first entry in that ZIP archive using the getNextEntry() method, and then get the entry's name using the getName method in the ZipEntry class of the java.util.zip package (you have worked with this earlier).
Now that we have our entry, we do a few checks to see if it is a directory. If it is, we make a new directory with the same name in our target destination and then continue to the next ZipEntry. Otherwise, we would read and write the same way we did last time, and after doing so, assigning a new ZipEntry value to our variable and proceeding with the loop (which checks through every entry).
******************
Well, that's all. I hope you guys did learn something today, and that this wasn't a complete waste of my time. Hopefully you took the time to understand everything here, reading the links/articles I gave out and interpreting them the right way. Have fun.
******************
