Think about it, what do we call art? We call something art when it impresses us beyond our understanding of how it’s possible often comparable to magic. We call a person an artist whether they are a painter, a dancer or a martial art practitioner when they are skilled enough at something that a majority of us would not even know what to begin with to learn to do what they do. However, once an artist defines and replicates what they have done again and again it becomes a craft and if they are able to pass the structure and logic behind their craft to others it becomes knowledge.
The point is that most of the time we are craftsmen but at that moment when we use our craft to create something new and unexpected for ourselves, that first iteration, defines who we truly are as an artist.
I would like to encourage us to remember not to be afraid to experiment with our craft and keep finding new ways, our own ways and forms of art, even when we are bogged down in repetitive nature of production.
I have given the article this quirky title because I wanted to talk about procedural arts and what it means for us as CG artists. At this point, you might think: “Oh no, it’s something technical, I’m not a technical person.”, but let’s not kid ourselves, if we are digital artists we are technical artists and everyone who thinks otherwise simply excuses themselves from learning something new.
In the digital world where computers are our invaluable assistants and machine learning is a real boy now, we have an opportunity to pass our craft not only to humans but also to the machines leaving the first iteration for ourselves and delegating the repetitive tasks to our little helpers.
Think for a moment of all the possible implications of this for us digital artists. It simply enables any individual to become a maker with unlimited creative possibilities focusing on the art side of things every single day.
With this in mind, we decided to learn and utilize procedural workflows while making Soundbender VR game demo which enabled us, students from UTS Animal Logic academy, to do more with less and realize our creative vision having only a few people on board. And we would like to share our discoveries with you in this article.
Before I dive into the production process and how proceduralism helped us to achieve our goals I would like to talk a bit about how this idea originated. If you don’t know what Soundbender is about please watch this short trailer to get an idea:
The process might seem very simple and obvious to many but I have never experienced it at a scale like this before and was very impressed by the effectiveness of it, therefore, I think it’s important to mention. We had about 30 students split into groups of 3 or so given a task to pitch 5 ideas that we think would be cool to make. So the next day we would hear them all and would be told this is all not good enough, do it again. Next day again, and again… You probably can do the math. Only in a few days, we went through hundreds of ideas and many of them I would personally never think of. Nevertheless, it quickly made our creative juices flowing and made us think wider.
At this moment you start to feel the power of many diverse minds and what they can generate within a short period of time. All we need to do is listen, observe and mix.
This mechanism is depleted very quickly and when everyone runs out of ideas it is time to move on to the next stage - cooking. The approach is to use all the ideas you have heard for the past several days and combine them into several more interesting and refined ones, and then you pitch again until a few winners left. This reminds me of rule #12 of 22 rules of storytelling according to Pixar:
“Discount the 1st thing that comes to mind. And the 2nd, 3rd, 4th, 5th – get the obvious out of the way. Surprise yourself.”
I don’t know where the original article is from but after a quick google, you can find a few dozen sources. Now multiply this by 30 and you get an idea render farm :).
You might think, ok, simple enough, but no. If you have ever tried to make a product from an idea, like a movie, a game or a book, you have probably realized by now that a lot of seemingly good ideas often don’t work, don’t add up or simply actually boring. That’s where the hard part begins, you need to break the idea asap. Fail fast, fail often. It is actually very likely, assuming we stay open-minded, to discover that our awesome beloved idea doesn’t work. The first instinct is to try to fix it but you might get stuck for a long time thinking with little to no progress. And this is exactly what happened to Soundbender many times and even during production. At this point, some may suggest to give up the idea and start all over again.
I personally believe that a lot of things can still be fixed we just don’t always see how straight away.
What I have learned from my experiences that ideas do not always live in our minds they are more like fruit growing on hidden trees and waiting for us to find them and pick them up. And often we simply need to get out of your head and look around. It reminds me of one conversation I had with Kim Taylor, Art Director from Animal Logic. We were talking about colors and how to get them right. I have mentioned that I usually look at nature and observe what it does. He said something like this:
“Exactly! It’s all around us. You just need to look. It’s so simple but, for some reason, a lot of people still don’t get it.”.
I think the same goes for ideas. For me it is not simply going out and waiting for inspiration, it is a logical process. It’s more like consciously asking everyone and everything about the problem intentionally seeking a solution. It is probably clear how to do it with people - you simply talk to them. They might not give you the solution but they might trigger an idea where to look for it. Same goes for not living things. When I look at a random object like a tree or a train, for example, I ask myself how this train can help solve my problem, is there anything relevant it’s got to show me and then I look trying to see the answer. Most often you get no answer but after hundreds or thousands of iterations and maybe many daunting days spent, you eventually find what you are looking for.
If you would like to learn one thing from this section I would suggest learning this:
Ideas don’t live inside us, they live in all living and non-living things around us but we need to intentionally reach out to them otherwise we will never hear or see what we are looking for. And if we surround ourselves with a lot of diverse and creative people our ability to hear and see grows exponentially.
There is no magic, it is only logical.
What is procedural art? Simply put, and I don’t think anyone can go more complex than this,
we give step by step instructions to a computer on how to get from point A to B of our craft. Point A can be as minimal as nothing at all and point B can be as complex as a masterpiece, a finished game or a movie.
Of course, it might take some time to explain how to make a movie to a computer, that is where machine learning comes in to save the day, but it is possible and now it’s easier than ever. In modern 3D packages, the instructions are represented in a form of nodes connected with wires representing the craft material flow pipes (in CG world it’s all data flow pipes of course) which show the order in which the instructions should be applied to this material (data) in every step. Sounds familiar? If you have ever used Houdini, Nuke, Katana, Substance Designer or have ever set up a material graph in any other 3D package like Blender you have already done it! Now take a pause and go to LinkedIn and change your title to Procedural Artist, quickly, while no one sees it :). While most of the packages are blends between manual work and proceduralism, most of the modern ones are mainly focused on the latter one and we will get a better understanding why as we go through this article.
Let’s take Substance Designer, for example, a more simple 2D case but purely procedural and quite powerful. We could take an image as a starting point and use Blur node to blur the image.
The subtle difference between hitting a blur button in any other 2D package is that we give the computer an instruction to do it and the computer can do it again and again and again.
All we need to do is just simply swap input images. Now imagine we have 100 instructions - we save ourselves 100 clicks on every next iteration after the first one. Now 1000 instructions. How about your craft takes 100 000 instructions? This is the most obvious benefit of proceduralism - scalability. We show how to do it once and the computer does it all other times for us. One craftsman suddenly becomes a force of many. It reminds me of this old saying:
“Give a man a fish and you feed him for a day; teach a man to fish and you feed him for a lifetime”. In the case of computers, they will probably feed you for a lifetime :).
Seemingly impressive this is not the only benefit of procedural content creation but before we continue exploring others please take a minute to watch this quick example of how Substance Designer was used for Soundbender:
Now when we know what proceduralism is, let’s see how it was applied to the pre-production of Soundbender. There is more to making a game than procedural art but I will try to stay true to the topic and will only talk about this part. How do you approach making a 3D set? Let’s assume we are not making a photorealistic environment from a photo reference using mega scans (if you are not sure what I am talking about I would suggest to watch this super awesome Rebirth UE4 tutorial later). Building assets takes a lot of time and to stay effective one would try to minimize throw away assets by putting some effort into planning. Some would probably start from sketch thumbnails, maybe a color script or even flashed out concept art, then a map sketch, few iterations of blocking out in 3D and they are good to go. For us, there were a few challenges we were facing at that moment.
From VR prototyping, we have discovered that 2D concepts and designs are a little of use because it is totally different in VR compared to screens. The scale, the perspective, the perception of objects in space, the amount of detail required for it to look more or less natural, everything was different, everything had to be tested in VR.
We also had a team of 3 and we had to build a full VR game demo in 4 months and the majority of time had to go into game design, R&D (real-time sound processing and visualization) and programming. There was one more thing. We have never done it before and barely had an understanding of what it takes to build a real-time set so we had to learn as we made it. I wish you could see our panic faces when we started to realize what it was going to take. :)
The first right decision we have made is to go procedural as there was no other way we could pull it off. The procedural tools we have used were Houdini and UE4 with Houdini Engine as a bridge in-between, Substance Painter and Substance Designer with Substance plugin for UE4. We also used other tools but these were the ones that saved the day. The first most important thing we required was to iterate with VR very quickly during blocking out stage. Houdini Engine for UE4 has done it for us. We just had to make our setup in Houdini, hit save, click rebuild in UE4, preview in VR. It was a huge time saver for us and we could see everything in VR almost straight away.
The second major benefit of using procedural workflows is that
making changes even fundamental ones when the whole model is already complete is fairly painless as you simply make a few changes to your network and let Houdini do the rest of adjustments for you.
It reminds me of a talk from a group of environment artists from Naughty Dog where one of them mentioned that they spend a huge amount of time just moving the grass up and down every time when a level designer makes some changes to the layout. I struggle to imagine that anyone would prefer to move their assets up and down a dozen times a year instead of working on something new and more creative. I think it was this one.
We didn’t need to worry about it. We didn’t need to lock the layout down at all as at any moment we could go back and regenerate everything. It’s not a zero loss but it’s fairly quick compared to the case where we would have to do it manually again. Besides not being afraid to leave things flexible in the early stages, we also were able to test and adjust the layout and dimensions of the objects very quickly with simple setups.
Need it higher, lower, taller, shorter, straight, curved, bigger, smaller, in a different position, all of this was a matter of changing a parameter and previewing it in VR.
Instead of guessing if it’s gonna be better or not, we were able to test and compare a lot of variations at no cost. We also knew that we could add or remove details and replace instances of assets easily, later on, meaning that most of the setups are not throwaways they simply need to be enhanced as required. Take a look at this short demo to get an idea of what I’m talking about:
In fact, we had to make changes to the terrain, layout, models and textures multiple times during production and all the consequent reiterations were fairly painless. For example, we discovered that the bridge model needs to change to adapt to animation requirements. We change the model and layout and the nightmare supposedly begins: the debris should be remodelled, UVs has changed so needs retexturing, the terrain doesn’t match the layout anymore, foliage is off. But with our setup, we were able to re-simulate the debris, regenerate Substance materials and update masks, regenerate terrain, regenerate foliage instancing, recook in UE4 and be done in a half a day. Believe it or not but we had to do it more often than we would like to because of design changes, human and tech errors, polish or noice to have things that would be discarded in any other case due to the cost of changes.
Blockout isn’t everything of what procedural workflows give a boost to during pre-production. We could also make quick auto UVs and drop some smart Substance Materials using Substance Painter to get a better feel of the textured models with lighting early on. And UE4 Substance plugin was handy to test some terrain texturing as well. So we were pretty much concepting in 3D and previewing it in VR minimizing errors.
Once I was watching Zach Mandt’s talk at CG Futures. He is quite an inspiring environment generalist. He said something like this:
“I use Houdini because if my director asks me to go back and change that one pixel cause it looks a bit off. I can go back and do it without any problem.”
High five to these words, high five to Houdini and Substance :).
The bellow is a break down of the final broken bridge model created using instancing and procedural techniques with Houdini and texturing with Substance Painter. Though might not be obvious due to its layer-based workflow, Substance Painter is also mainly procedural with some additional manual paint layers. Nearly everything what you can do with Substance Designer can be done with Painter as well, having a texture generator called Smart Material as a result. We are gonna talk about generators next.
Tip. Substance Painter makes it very easy to texture, therefore, it’s very tempting to do all the texturing there but for big scale models like the bridge even having multiple 4K textures might not be enough especially for VR. Josh Lynch a Senior Environment Artist at Sony Santa Monica helped me to realize that tile-able texturing workflow is still a thing very much. Although UE4 doesn’t seem to mind a lot of 4K textures running on modern gaming laptops and some people even attempting to use 8K ones, the cheaper workflow would be to use Designer to create tile-able textures and masks to blend them and then export masks and tiles separately and blend them inside of UE4 material having unlimited resolution with only a few 4K textures which also can easily be swapped if required providing a bit more flexibility than baked textures. However, I would recommend avoiding vertex painting for blending if you like to stay procedural and avoid manual rework, besides Substance mask generators are simply amazing and texture masks give you better resolution for blending. We realized our mistake a bit too late and didn’t have time to change it. But we gonna know better the next time. There is always something to learn so we stay The Rookies forever pretty much. ;)
The third beauty of proceduralism is the ability to create reusable generators. The generators themselves are more about scalability we have discussed before. When you write the instructions once and then let the computer do the other iterations so you can produce multiple variations of a model at a cost of 1.
But the most important feature of the generators that they can be used as tools or sub-tools being a part of multiple bigger generators.
For example, you can have a bunch of rocks, plants and trees generators which are used as parts of bigger forest and mountains generators which in their own turn can be parts of event bigger world generator and so on. In the end, you can simply say, Jarvis, generate me a planet with 60% of green forest 10% of mountains and 30% of water and in several days of cooking, you will probably get your result. What? Nobody said it’s supa quick. :) Not yet at least, but hey, we have got render farms so it should be manageable.
Houdini also has the ability to create generators called Digital Assets and Substance has got Smart Masks and Smart Materials.
Houdini Engine also allows artists to use its generators directly in multiple external application like Unity, UE4 and other 3D packages. The same goes for Substance plugin. You can find more info here.
As you probably have guessed already, we also used the generators to create rocks, corals and foliage for Soundbender. Please take a look at this short demo to see one of the generators in action:
We were able to create 4 custom generators where 2 of them were quick modifications of the first one and generate 51 pieces of foliage and 7 piles of rocks within less than a week. Note, the models have procedural UVs as well.
Some of you might say that the topology could be better optimized but VR requires more modeling detail to look more natural as far as the player can get really close to anything pretty much and normal maps do not really work as well. Level of detail (LODs) handle the situation pretty well and allow us to use higher polycounts at LOD0.
It’s also possible to use instancing within a generator and save a lot of rendering time which is very important for real time. Note the tree below is made of a single branch and a single flower reducing LOD0 polycount from more than 500 000 to less than 50 000 triangles.
Texturing took a bit longer but after creating the first Smart Material the other iterations would take about 0.5-2h per model depending on its complexity. In 30 mins case, most of the time was spent to open Substance, bake and export the maps.
We have used the same techniques to create some of the mechanical creatures of Soundbender. Please take a look at this small demo if curious:
Some of the tasks would be very difficult or next to impossible to do without proceduralism like real-time sound processing, particles or various simulations due to the big quantity of actors and forces in play. FX guys use these techniques for ages delegating repetitive calculations to the computer. In the assets creation world, the techniques used to be unpopular, however, it seems the wind is changing now.
Programming is the core of proceduralism and there is not much difference honestly. It just used to be code based mainly where instructions were given in the plain text in English rather than nodes connected with wires and that’s it.
So if you are working procedurally, you can fairly claim that you are a programmer as well. Of course, it might be hard to convince anyone if you have never written a line of code. ;)
UE4 has got new early access particles system called Niagara where you can give every single particle a set of node-based instructions as well that are executed every single frame. Houdini allows to do the same but the difference is that Niagara runs real-time which is pretty awesome and in conjunction with Houdini allowed us to create this cute sound visualization:
There is one more, the less obvious benefit of proceduralism for VFX teams. I personally did not see it straight away because I was taking it for granted for many years. Because the logic of our craft is networks and sub-networks of nodes they can easily be paralleled. But not only for fast computation using render farms but also divided among people very easily.
What I mean is that any team member can create their own sub-network and connect it to yours or even a bigger one, and when they make changes to their own part, the content just flows through the pipes all the way to the finished product like a movie or a game.
It is like VFX production facebook pretty much where every page is a single shot and when everyone is done with their part you can just hit play and enjoy the show. The extensibility and collaborative implications are huge! That the main reason why Katana and Nuke thrive and it is not because they are user-friendly and easy to use. ;)
As you can see, everything we had to do to make Soundbender was only logical and we were able to give the computer step by step instructions to do most of the repetitive work leaving the creative decisions to ourselves which enabled us to make that first iteration of the game much quicker. Of course, we didn’t make the whole game demo in 4 months but rather in 6 and our team had extra help with animation and additional assets but what we created as a team of rookies we can proudly call a piece of art.
In the digital world where every digital artist is a technical one and technology is accessible as never before, it is only logical to learn to work procedurally regardless whether you making images, composing music or teaching robots to dance.
And If you are still not, you simply limiting your creativity by spending most of your energy on craft rather than art. Now ask yourself, is this what you really wanna do? What would you create if you had the power of many?
We hope you have enjoyed reading this article as we have enjoyed making Soundbender! Thank you!