Thursday, July 19, 2007

ANN Tutorial - Forward Propagation

So far, if you've been keeping track of my blog, you'll have seen lots of stuff on Artificial Neural Networks (ANNs) and what they can do, but you might not know how they work. This is what this post is going to be about; today, I'll be talking exclusively of forward propagation, or, in layman's terms the act of getting the ANN to 'think'.

If you know much about anatomy, you'll know that the brain is made up of neurons interconnected together - It is believed that the strength of these connections is what forms a memory. Now, I'm not going to get into too much biology because it'll simply bore you out of your mind, instead, let's just tackle the real thing and I'll throw-in biological references from time to time... How's that?

First of all, one question that probably didn't pop into your head is; "what does an ANN look like" - You were probably expecting it to be some absurdly abstract concept, but it isn't quite that way, it is actually feasible, and very common might I say; to represent them using flowcharts. They say a picture means a thousand words, so here goes:

First of all, I'd like to direct you to the different 'layers' of this ANN; the input, hidden and output... This is rather simple; the input layer represents input data, the hidden layer is the one that does the processing and the output layer just outputs the result as either a 1 or a 0.

Secondly, you should take note of the 'interconnections' that connect all the nodes together. The numbers you see for every red and black connector are called 'weights' and they're used to modify output from the previous node (more on that later). You need to note that the green and blue connectors also have weights, but they were omitted from the diagram as they couldn't fit in. Typically, to begin with, each weight of every connection is initialized to a random value. Personally, I like to keep it in the range of -10 to 10.

As you can see from the diagram, in this case, the input layer is made up of 3 nodes and each of them has a value of either 1 or 0 (binary). If we were to use this ANN to identify a pattern on a 3*1 bitmap of black and white pixels (black being 0 and white being 1), each input node would hold the value of one of the pixels. Then, each of these values would be processed, this is where the weights come in; each output from the input nodes (1 or 0) is distorted in some way by the weight of each of the connectors; this is done via a simple multiplication.

Let's assume that the first pixel in our 3*1 bitmap was white, looking back at our diagram, we can say that the red input node will be responsible for it. Let's go through this process shall we; first of all, since the first pixel is white, the input node will output a 1. This value will then be propagated to each of the nodes in the hidden layer via different connections (the red ones).
The top hidden node, for example, will receive a value of -5.2 because the weight of the connector (-5.2) multiplied by the input (1) gives -5.2.

This process is done for each of the input nodes, and, once this is completed, each hidden node will evaluate the sum of all the 'weighed' values which they have received from each of the input nodes. If that sum is greater than a specified threshold (in most cases something like 0.1); then that hidden node will output a 1, otherwise, it will output a 0.

Once that's done, this output from each of the hidden nodes (1 or 0) will again be weighed by the connection between the hidden and output layer. Again, this is just a simple multiplications... Once this is completed, the output node will find the sum of all the weighed values and if it is above the threshold, then that output node will output a 1 (true) or 0 (false).

In the examples which I posted before, I designed an artificial neural network to associate certain drawings with 1 and others with 0; since pretty much anything can be represented by binary, you might want to play around with that.

For those of you interested in biology; as we've heard, the brain is made up of neurons interconnected together. Well, it is believed that the strength of each of these connections is what determines the strength of each memory unit that it holds; that's probably why scientists say that the brain is a muscle; if your neurons are buff in some areas, then these areas are going to represent the most prominent aspect of your intelligence. If only we knew what each of these memory units actually were we would probably be able to better understand, and thus exercise much more control over our ANNs.

In this case, I have showed you what is known as a step ANN because the outputs of each node is only either a 1 or 0. There is another type of ANN which performs sigmoid calculations... You might want to do some research on that. Also, you can be pretty creative with your ANNs; each layer can have variable amounts of nodes; you don't have to have the same number of hidden nodes as input nodes and you could have more than one output node if you liked, also, you could go as far as having more than one hidden layer... Think big!

Ok, this concludes the tutorial on forward propagation. Next time, I'll be talking about back-error propagation which will teach you a way to teach your ANN - Just to get you thinking; it's done by altering the weights of each interconnection by comparing the desired output with the actual final output... As the name of the algorithm suggests, it involves starting at the output node and 'correcting' the weights of each successive connector as you make your way back through the layers.

Wednesday, July 11, 2007

Java IntelligentBitmapAnalyser Released!

Hi, I have released a Java version of my ANN.
It's basically a Java implementation of my ActionScript source code.

Here are some source files and an executable JAR file:


You need the Java runtime environment to run
P.S. It is not an applet.

Wednesday, July 4, 2007

Pattern Regognition in Flash

!- Please note that you need to have a decent computer and browser to run this application or it'll crash.

A while ago, I made a pattern-recognition application in Flash and I decided to improve it by adding some sort of save and load functionality. The engine itself uses an Artificial Neural Network (ANN) to determine an output based on a series of inputs which, in this case, represents each pixel of a 50*50 bitmap. Here's screen shot:

Before you get your hands dirty, here's a run through of how it works:

What the program does is associate two different shapes with either 'yes' or 'no' but it's not that easy, you need to teach it! To start you'll need to decide two shapes that you'll use, the more different they are, the faster the machine will learn to identify them. A good set of shapes is a tick and a cross to signify 'yes' and 'no' respectively but you can use whatever you like.

To start, just draw the shape for 'yes' inside the small white square and click 'think.' Once it's done thinking it should output either 'Yes' or 'No' in the box labeled 'Response'. If it says 'No', correct the machine by pressing the 'Yes' button, the output will most likely change to 'Yes', but if it doesn't, don't keep correcting it. Then, clear the drawing area (the clear button) and draw the shape for 'No', click 'Think', and again, correct the machine if it's wrong. If it gives you the right answer, do not correct it by feeding it the right answer again. Repeat this MANY more times and, eventually, you'll notice that it gets pretty accurate.

Now, another thing which might be of interest is the big text area right under the 'Brain Data' label - this is to upload and show the neural structure of the machine. If you wish to save the machine's brain, click the 'show' button, be patient, this may take a while... Once it's done, click in the text area and press 'Ctrl+A' to select all, then copy it. Paste the text in a text file. Just for the record, if you were to paste it inside a word document, if would take-up about 50 pages when using Times New Roman at size 12.

If you want to upload your neural data, copy it from the text file and paste it inside the text area then press the 'Upload' button; again, be VERY patient and don't do anything while it's uploading, I haven't optimized the application and it can freeze very easily.

Once, that's done, you should be able to carry on with teaching your artificial brain to perfect its skill.

Ok, here's the link: ANN
Also, I've already created a brain if you don't have the patience to train your own from scratch. It associates a tick with 'Yes' and a circle with 'No.' It's about 80 to 90% accurate with me, I guess it depends on handwriting. Try to make well formed ticks that look like 'v's: Tick vs Cirle
Just open-up the word document, then, click on the text and press 'Ctrl-a', then copy and go to the application. Click inside the text box (make sure it is empty to start with) then paste and click 'Upload'.


Tuesday, July 3, 2007

Line Smoothing

A while ago, I developed a line-smoothing application in Flash.
It uses a simple line-averaging algorithm which breaks the line into a series of points which are then linked together by straight lines.

Line Smoothing

I guess I could probably make a smoother version by using intermediate points to control the bend of each line segment... I'll look into it some other day.

Flash Magic Webcam

For a while I've been carrying-out my own experiments using Flash and have accumulated a large collection of innovative and interesting apps. In the next few days, I will be posting a series of such experiments for your entertainment.

Here's a flash application which I call "Magic Web Cam" - Click to change view.

Magic Webcam

P.S. You need Flash Player 9 and a web cam to view. Please note that your video feed is not recoded and is not visible on any other computer than the one you are currently using.

Monday, July 2, 2007

Safari Browser for Windows

To kick-off my blog, I'd like to share my thoughts on the *new* Safari 3 Browser (Beta) for Windows.
As you might have heard, the Safari browser has finally made its way to the Windows operating system and to be honest, so far, I have very mixed feelings about it.

After downloading it, I was excited to try it out; judging from its excellent reputation on the Mac, I was expecting something that would completely annihilate Firefox. To my surprise, however, what I saw was rather displeasing.

What annoys me above all is the way it renders text. If you've tried IE7 before, you might have noticed that the text is a bit peculiar; this is all due to Microsoft's text-rendering technology 'ClearType'. If you've got a fine eye for detail like me, this new technology would have been more of a downfall than a benefit. Actually, my first Google search after downloading IE7 went something along the lines of "IE7 turning off blurry text." Anyway, as I started up Safari for the first time, I was shocked. Until that point, I could have sworn that font-rendering was just a one-off fad by Microsoft, I had never thought that it would actually be replicated ever again. Not only was Safari's font a replica of the ClearType disaster, but it was even worse. The text in Safari 3 Beta for Windows literally looked blurry, so much so that I'm still puzzled by why a huge company like Apple would integrate it in their very first browser for the Windows platform. Not only is it ugly, but, to my surprise, it can't be turned off!!! That was perhaps the biggest downfall... I mean, first of all; they fail to make a decent text-rendering system, then, to top it all, they will not allow me to turn it off! Why did they even release this Beta? They're just turning away potential users even before they get the real version. To me, it's just money down the drain.

Another problem that I found with Safari is the graphical user interface... Hello!!! I'm using Windows, I don't want a Mac interface! Not only is the look out of place with Windows, but it's pretty plain and boring. That being said, I like how the browser displays buttons, check-boxes and the like; Mac style does have a place in a Windows environment but it needs to have boundaries. I find that the top bar in Safari just takes it too far; it doesn't feel like Windows and on a semi-conscious level, it reminds me of the Netscape browser and that's not a very nice image.

One major aspect of this browser that I really enjoy, though, is the loading. Seriously, it's a beauty; it's like a perfect compromise between Firefox's progressive loading and IE's all-at-once approach. I find that in Firefox, for many pages, pretty much as soon as you press 'Enter' in the address bar, the content of that page starts to load, in fact, it sets-off so quickly that sometimes the content doesn't even get the time to format itself properly. IE, on the other hand is on the other side of the equation; it takes much longer to load, but at least, when it's done, the content is pretty much all ready. In brief, Safari is a bit closer to IE, but the way it loads content is more free-flowing and it's MUCH faster. I really never envisaged that it would make such a great difference in speed!

I will be looking forward to the complete version of Safari for Windows, hoping that they'll have fixed-up at least some of the major problems which I highlighted in this post.