The Professional Programmer

Neural networks can seem daunting, complicated, and impossible to explain. But in reality they are remarkably simple. In fact, they only ever require a single layer of neurons. In my previous post about the basics of neural networks , I talked about how neurons compute values. They take a set of inputs, multiply each input value by a weight, and sum the terms. An activation function is then applied to the sum of products, to yield the output value. That output value could be zero (i.e., did not activate), negative, or positive. In this post, we will go deeper down the rabbit hole. We will look at neuron layers, which layers are actually necessary for a network to function, and come to the stunning realization that all neural networks have only a single output. Organizing Neurons into Layers In most neural networks, we tend to organize neurons into layers. The reason for this comes from graph theory (as neural networks are little more than computational graphs). Each layer con...

Neural networks need to map inputs to outputs. It seems simple enough, but in most useful cases this means building a network with millions of parameters, which look at millions or billions of relationships hidden in the input data. Do we need all of these relationships? Is all of this information necessary? Probably not. That's where neural network pooling layers can help. In this post, we're going to dive into the deep end and learn how pooling layers can reduce the size of your network while producing highly accurate models.

Neural network dense layers (or fully connected layers) are the foundation of nearly all neural networks. If you look closely at almost any topology, somewhere there is a dense layer lurking. This post will cover the history behind dense layers, what they are used for, and how to use them by walking through the "Hello, World!" of neural networks: digit classification.

I used to work in HPC (as evidenced in my posts on AoS vs SoA and AoSoAs ), but my interest has always been nature-inspired algorithms, like genetic algorithms . But there's a nature-inspired algorithm that is getting lots of attention these days: neural networks. People think of them as black boxes, but in this post I'll try to peel back the top and explain what's going on inside.

Everyone will tell you something different about how to setup a git project and how to establish contribution policies. But there is a certain way I like to setup group projects in git (whether it's GitHub, BitBucket, or GitLab makes no difference) that works for me, and you are welcome to use it as well. Or not. It's okay if we disagree.

Last week I posted about Arrays of Structures vs. Structures of Arrays , and the response was pretty great. I did get some feedback that I had left out one important concept:

It's one of those things that might have an obvious answer if you have ever written scientific software for a vector machine. For everyone else, it's something you probably never even thought about: Should I write my code with arrays of structures (or classes), or structures (or classes) of arrays. Read on to see how both approaches perform, and what kind of readability you can expect from each approach.

So far in my series on Genetic Algorithms I've covered:  Basic Concepts ,  Encoding ,  Selection , and  Crossover . Those operations will get you most of the way to a functioning optimizer. However, all of the operations we've discussed exert convergence pressure  on the population. That is, selection and crossover have a tendency to draw the population together. The last operator I want to talk about -  mutation - is different. It wants to split the population apart. And as counter intuitive as it seems,  divergence  can be even more important than convergence.

So far in this series on genetic algorithms , I've talked about encoding individuals and selection methods . The next task in a generic algorithm is to mate individuals using an operator called crossover . Crossover produces two (or more) new individuals (children) from two (or more) existing individuals ( parents ). The idea behind crossover is twofold: Sharing of worthwhile traits allows for individuals to be created which inherit the best traits of their parents (optimization), and new combinations of traits may reveal as-yet-unexplored possibilities (search).

In Genetic Algorithms Basics  and my post on Encoding Individuals , I talked about the basic concepts and operations in all genetic algorithms, and considerations to keep in mind when encoding individuals for GAs. This time I want to spend some time discussing selection . Selection is the most important operation in the GA, and it performs two important functions. First, selection is how we determine who mates (and who doesn't), and as a result which traits survive to the next generation (and which do not). Second, selection acts as the major source of  convergence  in the GA. The way in which you select individuals for mating can have an enormous effect on whether you converge to a suboptimal solution, or manage to find a near-global optimum.

Genetic algorithms are a great tool for optimizing all sorts of things, from path planning and scheduling to neural network architectures. In this post, I'm going to talk about the first step in designing a genetic algorithm: Encoding Individuals . As I mentioned in my post on genetic algorithms basics , genetic algorithms borrow on the concept of phenotypic expression, which is basically the idea that specific sequences of DNA can be mapped to particular traits which are expressed in the individual. For a genetic algorithm, this means knowing what problem you want to solve and developing a succinct way of expressing solutions to that problem as chains of values.

Everyone is talking about neural networks these days. I'm glad for it. When I started working in nature-inspired algorithms nearly 20 years ago, it was an extremely niche area. In fact, few saw a future for any of these approaches outside academia. But the deep learning revolution that started in 2006 (or 2012, or 1989, depending on who you ask -- I'll have a full post on the history of neural networks later) is going strong, and shows no signs of letting up. It turns out that people like being able to interact with computers using their voice, having images automatically tagged, finding objects in pictures and movies, having TV shows and brands of toilet paper recommended to them based on their viewing and purchasing history, and self-driving cars. Who knew? I plan to write entire posts on all of the different nature-inspired techniques (neural networks, artificial immune systems, ant colony optimization, flocking/swarm algorithms, and simulated annealing) in the future. This ...

If you're like me, then you tend to follow the ultra-conservative practice of "commit early, commit often" when using git . If so, I applaud you. It's important to be sure you are keeping track of your work, and if you push your commits back to GitHub, Bitbucket, or GitLab that's even better. No one wants to do a day's worth of software development work, only to wake up the next morning to a local hard drive failure.