تويت
So... How do we optimize our fluids? Both file sizes, but also rendering times and network bandwidth? I'll save optimizing simulation times for another day, but this is a pretty beefy and frequently asked subject...
What You Need To Know
Since Fume FX 2.0 was released, there's been DOZENS of cool new features for it, and most of them I think have been pretty overlooked. Most people associate Fume FX 2.0 and up (2.1 actually has a substantial amount of new features.
So it's not just a few bug fixes FYI) as having the new Wavelet-Turbulence (WT) and Post Processing (PP) features. And even then the PP it seems almost everyone confuses it as just being a neat way to retime your grid. Mind you, I've used the retime feature almost exclusively on a few productions - It's a great way to be able to simulate 100 frames of lets say thick billowing smoke, and then get the simulation, and get rid of the 30 or so lead up frames, and then literally get 10 or so frames of the smoke doing it's thing, and stretch that out over an entire vfx shot. No more lengthy & unnecessary pre-roll's!
But PP has a whole other lot of features, firstly it optimizes your grid down to the minimal value, as long as you have MINIMIZE GRID checkbox checked (which by default it is) then if there's a lot of excess room in your dynamic grid (that blue grid that seems to change shape inside the yellow one as you play your animation in the viewport or during a sim). It will resize your grid down to the tightest proportions possible. Which is a really great feature to have, although it's not going to revolutionize anything here! The REAL POWER, comes from the exclude channels.
What this allows, is for you to do is by clicking on all of these exclude channels before you run the PP sim, you can actually tell it to as the name suggests, exclude the channels you do not want in your final sim! Now how this work is when you're running your initial sims for use with WT, chances are you've had to include Smoke, Fire, Temperature, Velocity &, Detail. This is just so that when you run WT, you're able to use the velocity and temperature to create the additional wavelet "up-res'd" version of your sim.
Where The Main Problems Occur
The problem is, now that you have this sim, it's twice the resolution, and it still contains 5 channels of data, none of which you're really going to use. If you're really wanting to get advanced and technical, then you can use the temperature channel to render out a temperature pass, Which can be useful for comping and getting more control out of your sims. If you want to know more on how to do this, there's a chapter on it covered in Fume FX Core-Fundamentals, but it's a bit off topic for what I'm trying to cover in this news-letter.
Plus, of course you can use the vector channel exclusively with Krakatoa, as well as the Temperature Channel for plugging into KCM in Krakatoa (I'll cover this another time as well). But unless you're actually PIPING these channels into other things, you pretty much cannot see these things, or the detail channel either. They're excess channels used to get you to this point of having a cool looking WT sim.
So here's the problem - if you're doing massive simulations, with lots and lots of detail, now your sims are twice the size in resolution, and therefore your file caches are like 1gb or more per frame, so if you have a 100 frame shot, that's 100gb of data for that one sim! PLUS it's taking FOREVER to render, because it's gotta load these big ass files over the network, and then read through them, store them in memory etc. JUST TO RENDER!
This is killer, and this is the most common thing that happens on productions, you end up with massive files and the network starts to slow down to a halt, your machines need more RAM to push your renders out.
Actually, for me, it's kind of cool - because I've had a lot of consulting jobs now, where I've been brought in because studios are having problems with their Fume grids getting too big, so they want to squash them down somehow. Most of the time I'm having these discussions with IT, about setting up a hub that is off the network so the Fume FX data is being pushed locally and not slowing down the network.
Much like how compositors will be on their own hub so they can push around 2K EXR or DPX plates without absolutely killing the network (15 compositors x 50 passes per shot x however many shots are being worked on, plus rendered etc, absolutely kills the network) well Fume's doing the same thing with these big file sizes. So a lot of these studios suddenly want to set up new "Fume Artist" networks, so they can at least keep these massive collections of sims off-line. Plus you have to deal with the servers possibly running out of space. So i guess I'm shooting myself in the foot a little bit here by explaining that there's these killer features in Fume 2 now that resolve a lot of these issues, so that there's not really as much of a need for such radical solutions as moving artists off onto their own hub.
Overview
In some situations when originally testing these ideas out, we were getting files down to as low as 1% of the original size. So a 1gm file would become 10mb! Insane right?
So excluding channels is great, because you're essentially able to use all of these additional channels, and when you're done with them, strip them out, so your file sizes suddenly decrease extensively. Like I said, possibly as low as 1% of their original size. So instead of 1000mb a frame x 125 frames you're now dealing with 1.25GB instead of 125GB!! Now it's not as simple as that, but essentially that is as clear as I can make it.
The best part is that, OK you're saving space, but the fact is now your file sizes are smalle rand they have less information such as temperature and other useles sstuff, so when you're rendering your scenes, they suddenly render MUCH MUCH QUICKER!!! So suddenly rendering passes at that resolution isn't a painful process, and it means you get a lot more flexibility with your renders, because of the faster turnaround time.
So I can start to play around with changing colors and adjusting my lighting without needing to go get a coffee every time I want to render a single frame! So by having lighter file caches without all this excess info, it just means I can work at a much faster rate and my renders aren't going to take days to render (the biggest sim I performed to date is 1600x2500x1600 and it took something like 6 days to render, not to sim, just to render, at 960x540!). So PP has some great features, but this is one severly overlooked and very very important!
Additional Information
Now, this is actually a smart thing that Fume does that some people do not seem to get, when you run your default sim, it writes it to MySim_0000.fxd and then when you write your WT it writes out MySim_WT_0000.fxd and your PP is MySim_PP_0000.fxd. So you're writing out now 3 caches, so technically now you're taking up EVEN MORE SPACE. But this is a blessing in disguise, if we just overwrote our old caches there'd be a dozen things that could go wrong, what if your sim crashed, or you canceled it, it's hard for it to really resume where it was up to without it performing snapshots.
Plus what if you want to tweak your wavelet or your original sim you're essentially having to go from the very beginning, whereas this way you can tweak your wavelet and then run the retime or whatever other tasks you want to do easily. It's just all round more safe, not to overwrite your caches. So of course you then have excess files, but it's just up to you to remember to blast them away if you think you do not need them anymore.
If you're happy with your optimized PP sim, kill your WT and original fume data knowing that you can resim it if you need to, but now you have a sim that is efficiently optimized to render faster, with less memory and take up far less space on the network, so traffic bandwidth won't be as bad, and you will have no problems with running into disk space issues etc.
Sounds like a good deal to me!
-Allan
What You Need To Know
Since Fume FX 2.0 was released, there's been DOZENS of cool new features for it, and most of them I think have been pretty overlooked. Most people associate Fume FX 2.0 and up (2.1 actually has a substantial amount of new features.
So it's not just a few bug fixes FYI) as having the new Wavelet-Turbulence (WT) and Post Processing (PP) features. And even then the PP it seems almost everyone confuses it as just being a neat way to retime your grid. Mind you, I've used the retime feature almost exclusively on a few productions - It's a great way to be able to simulate 100 frames of lets say thick billowing smoke, and then get the simulation, and get rid of the 30 or so lead up frames, and then literally get 10 or so frames of the smoke doing it's thing, and stretch that out over an entire vfx shot. No more lengthy & unnecessary pre-roll's!
But PP has a whole other lot of features, firstly it optimizes your grid down to the minimal value, as long as you have MINIMIZE GRID checkbox checked (which by default it is) then if there's a lot of excess room in your dynamic grid (that blue grid that seems to change shape inside the yellow one as you play your animation in the viewport or during a sim). It will resize your grid down to the tightest proportions possible. Which is a really great feature to have, although it's not going to revolutionize anything here! The REAL POWER, comes from the exclude channels.
What this allows, is for you to do is by clicking on all of these exclude channels before you run the PP sim, you can actually tell it to as the name suggests, exclude the channels you do not want in your final sim! Now how this work is when you're running your initial sims for use with WT, chances are you've had to include Smoke, Fire, Temperature, Velocity &, Detail. This is just so that when you run WT, you're able to use the velocity and temperature to create the additional wavelet "up-res'd" version of your sim.
Where The Main Problems Occur
The problem is, now that you have this sim, it's twice the resolution, and it still contains 5 channels of data, none of which you're really going to use. If you're really wanting to get advanced and technical, then you can use the temperature channel to render out a temperature pass, Which can be useful for comping and getting more control out of your sims. If you want to know more on how to do this, there's a chapter on it covered in Fume FX Core-Fundamentals, but it's a bit off topic for what I'm trying to cover in this news-letter.
Plus, of course you can use the vector channel exclusively with Krakatoa, as well as the Temperature Channel for plugging into KCM in Krakatoa (I'll cover this another time as well). But unless you're actually PIPING these channels into other things, you pretty much cannot see these things, or the detail channel either. They're excess channels used to get you to this point of having a cool looking WT sim.
So here's the problem - if you're doing massive simulations, with lots and lots of detail, now your sims are twice the size in resolution, and therefore your file caches are like 1gb or more per frame, so if you have a 100 frame shot, that's 100gb of data for that one sim! PLUS it's taking FOREVER to render, because it's gotta load these big ass files over the network, and then read through them, store them in memory etc. JUST TO RENDER!
This is killer, and this is the most common thing that happens on productions, you end up with massive files and the network starts to slow down to a halt, your machines need more RAM to push your renders out.
Actually, for me, it's kind of cool - because I've had a lot of consulting jobs now, where I've been brought in because studios are having problems with their Fume grids getting too big, so they want to squash them down somehow. Most of the time I'm having these discussions with IT, about setting up a hub that is off the network so the Fume FX data is being pushed locally and not slowing down the network.
Much like how compositors will be on their own hub so they can push around 2K EXR or DPX plates without absolutely killing the network (15 compositors x 50 passes per shot x however many shots are being worked on, plus rendered etc, absolutely kills the network) well Fume's doing the same thing with these big file sizes. So a lot of these studios suddenly want to set up new "Fume Artist" networks, so they can at least keep these massive collections of sims off-line. Plus you have to deal with the servers possibly running out of space. So i guess I'm shooting myself in the foot a little bit here by explaining that there's these killer features in Fume 2 now that resolve a lot of these issues, so that there's not really as much of a need for such radical solutions as moving artists off onto their own hub.
Overview
In some situations when originally testing these ideas out, we were getting files down to as low as 1% of the original size. So a 1gm file would become 10mb! Insane right?
So excluding channels is great, because you're essentially able to use all of these additional channels, and when you're done with them, strip them out, so your file sizes suddenly decrease extensively. Like I said, possibly as low as 1% of their original size. So instead of 1000mb a frame x 125 frames you're now dealing with 1.25GB instead of 125GB!! Now it's not as simple as that, but essentially that is as clear as I can make it.
The best part is that, OK you're saving space, but the fact is now your file sizes are smalle rand they have less information such as temperature and other useles sstuff, so when you're rendering your scenes, they suddenly render MUCH MUCH QUICKER!!! So suddenly rendering passes at that resolution isn't a painful process, and it means you get a lot more flexibility with your renders, because of the faster turnaround time.
So I can start to play around with changing colors and adjusting my lighting without needing to go get a coffee every time I want to render a single frame! So by having lighter file caches without all this excess info, it just means I can work at a much faster rate and my renders aren't going to take days to render (the biggest sim I performed to date is 1600x2500x1600 and it took something like 6 days to render, not to sim, just to render, at 960x540!). So PP has some great features, but this is one severly overlooked and very very important!
Additional Information
Now, this is actually a smart thing that Fume does that some people do not seem to get, when you run your default sim, it writes it to MySim_0000.fxd and then when you write your WT it writes out MySim_WT_0000.fxd and your PP is MySim_PP_0000.fxd. So you're writing out now 3 caches, so technically now you're taking up EVEN MORE SPACE. But this is a blessing in disguise, if we just overwrote our old caches there'd be a dozen things that could go wrong, what if your sim crashed, or you canceled it, it's hard for it to really resume where it was up to without it performing snapshots.
Plus what if you want to tweak your wavelet or your original sim you're essentially having to go from the very beginning, whereas this way you can tweak your wavelet and then run the retime or whatever other tasks you want to do easily. It's just all round more safe, not to overwrite your caches. So of course you then have excess files, but it's just up to you to remember to blast them away if you think you do not need them anymore.
If you're happy with your optimized PP sim, kill your WT and original fume data knowing that you can resim it if you need to, but now you have a sim that is efficiently optimized to render faster, with less memory and take up far less space on the network, so traffic bandwidth won't be as bad, and you will have no problems with running into disk space issues etc.
Sounds like a good deal to me!
-Allan
تعليق