Dear Neuroscience :: From Visual Effects

“To get this started, let me outline the work we do in the visual effects industry.

Our work is about abstracting the complexity of the world around us (specifically, the photographed world around us) into algorithms that can be computed, to generate a Computer Generated Image. Hence “CGI”.

Our aim is to trick the human eye into believing our CGI is real.

Because the world around us is so complex, and computing power is limited, those algorithms do not, on their own, produce an image which is good enough to meet our aim.

If the algorithms get us 70% of the way to our aim, then the remaining 30% has to be met with human craft.

That 30% is achieved by humans comparing the CGI on their computer monitor in front of them, with either:

a photograph of a real thing
an image in their mind’s eye, based on their life experiences and memories

For example, imagine we are making a CGI house. We use algorithms to define the shape of the house, the way light reflects from its roof tiles and bricks into the camera, the way ivy grows up its walls, the colour of the dirt on its windows. That gets us 70% to our aim.

Then, the artisan compares the CGI house with photographs of a real house. They notice the CGI roof tiles are too big, so they adjust the roof tile algorithm to make them smaller. They notice the ivy is growing more on the south-facing wall of the house, so they tweak the ivy algorithm to make more ivy on that wall.

Those kinds of decisions are based on visual observation skills, reasoning, leading to understanding. And then finding the right part of the algorithm to tweak.

In the end we get to 100%.

Another example. Imagine we are making a CGI dinosaur. Now there is no photograph. At best, maybe there are some photographs of some big lizards and some birds. This is where we start to see personal biases and distortions creep in to the mix. We start trying to relate to the viewer in the cinema - what’s going to make that viewer believe in our CGI dinosaur, what is the trigger for them? Can we make them believe our CGI dinosaur purely by piecing together photographic reference of lizards, birds, elephants? Will that be enough to get to 100%?

Sometimes it is, if the artisan is skilled enough in their visual observation of lizards and birds, and strong at reasoning, and understands how to use the algorithms effectively.

But sometimes that is not enough to reach the last 5%. We’ll need something else. Some special sauce. Some X factor. Often it will be something that we don’t predict or expect. We’ll try different things, trial and error, and one thing will just ‘click’ one day, and we’ll think “that’s it”.

I think that’s where there might be an overlap with your neuroscience work. Understanding what makes that special sauce.

My personal opinion is the special sauce is something that creates a personal connection between the CGI and the viewer. Something that touches a nerve.

Does that make any sense? :)”

Many thanks to Matthew Smith from Double Negative Visual Effects

A bit more about Matthew:
Outside of VFX, I have an interest in understanding the world around us, and how people interact with it. I don’t have any formal education in any science. But I do have some experience, both personal and indirect, of what happens to people when their understanding of the world, and their interaction with it, is flawed.
My hero is Leonardo da Vinci. I have some interest in Buddhism.
My ultimate aim is to build some solid framework that can help people understand more accurately the world around them. I intend to get there by using the principles of observation of the natural world, reasoning, and testing, that Leonardo used.

To learn more about the Dear Neuroscience Event Series, please visit www.everymind.online/DearNeuroscience.