Natural Codes

I just completed the process of manually stitching together around 200 drone photos of my property to make a composite photograph/map. It's a really interesting and challenging problem. It's been tackled and solved many times over, of course, but still it's interesting. Google Maps, for example, is doing similar processing on satellite photos.

The drone photos of my property are immensely detailed compared to the kind of imagery that's available through something like my county's auditor web site, or through google. The flower of a dandelion, for example, is multiple pixels in diameter. Consequently, it's possible to go through the images and catalog the trees, for example, or to do something like count the flowers on strawberry plants to predict yield.

The process of stitching the photos together is really interesting. It's in the same class of problem as something like music recognition, or object recognition. The automated algorithms that stitch the photos together kind of suck, really the same way music recognition sucks. They work over a random range of conditions. They fail for various reasons and don't gradually and gracefully fall apart as there's more "noise" in an image, they just stop working for whole categories of images, like they don't work in the woods, for example. They also randomly fail even when there's large scale, really good solid features like roads and driveways.

The automated algorithms perform a simplified version of what a human does when they perform the task manually. They attempt to select unique landmarks in a photo... a human looks for unique landmarks that are present in multiple images, though, because a human can recognize them in both images. The algorithms attempt to categorize regions of the image to numerically "unique" landmarks because they can't recognize anything. Then they search for the landmarks in multiple images numerically.

For example, a human can compare a set of dandelions in two images and determine which ones are in the same group, while the algorithm just looks for something like edges or shapes that match some mathematically defined criteria.

This problem of "ex post facto" code discovery is really intriguing. It seems likely that it's really the basis of consciousness or intelligence in general. There's innumerable, really probably infinite such "codes" in the universe. There's probably infinite codes in my woods or on my property. The relationship between all the objects in every potential dimension of characterization would comprise the entire "information space".

The music recognition systems use the same process as the image stitching algorthms, and suffer from the same sort of problems. They kind of work, but then randomly fall apart, rather than gracefully decaying as noise is introduced.

Our brains can interpolate missing information and form hypotheses about landmarks in a photo, or the beats of a song that's heard in a noisy bar and can winnow down probabilities based on context... like there's probably nobody playing a classical music symphony over a noisy bar's sound system.

In the natural world, these patterns get encoded into DNA and into cultural information, and into the structures of our brains and sense organs. It's also getting encoded into entities with no obvious nervous system, e.g. the soil, or into an entire meta-entity like a forest.