Sketchfab user Pete McNally give us the run down on how he uses Unity’s new de-lighting utility to make his scans look great in any lighting conditions.
How’s it going Sketchfab Community?! I’m Pete Mc Nally, a Senior 3D Artist based in Dublin, Ireland with a growing interest in 3D scanning, specifically photogrammetry.
Photogrammetry is such a useful tool for the 3D artist’s box, even without cumbersome or expensive equipment. As long as you have a half decent phone with you there is potential to capture a model.
Ireland generally has favourable conditions for outdoor photogrammetry: the weather is generally overcast, or at least that’s how it seems! This makes it relatively easy to correct albedo textures derived from photogrammetry for more general use in various PBR lighting setups, like this one:
Unfortunately the day I went to capture the subject of this article – a sea stack near Ireland’s most Northerly point, Malin – there were blue skies and blinding sunshine. 15 hours of sunshine that day in fact! Despite there being significant contrast between the lit and shadowed sides of the subject, I wasn’t put off, I figured this might be a good chance to try out some of the de-lighting techniques I’d been reading about.
Unity had just released an experimental de-lighting tool by Cyril Jover to help with such situations, so I captured about 120 photographs of the stack to process in RealityCapture. Here is one of the images in the sequence, you can see some artifacts like lens flare and significant darkening in the shadowed side.
Here is the resulting textured model from Reality Capture, decimated to about 16 million triangles with a 16k texture:
The goal was to create a game-ready asset from this source, removing as much lighting information as possible from the albedo without resorting to time consuming hand painting. With that in mind, I reduced polys in Meshlab to 10k and unwrapped in 3DSMax, then downloaded the Unity De-Lighting tools from here: https://github.com/Unity-Technologies/DeLightingTool
Traditionally I have used 3dSMax for texture baking, but the Unity de-lighting workflow requires a bent normals texture, which meant looking elsewhere. I tried Knald and it was pretty much a one stop shop for everything I needed, and very fast. Unity requires the original lit diffuse texture (ideally with an alpha channel around the UV islands), object space normals, bent normals and ambient occlusion for it’s de-lighting tool – Knald baked each of these in one pass at 4k.
Once you have the baked textures, it’s a simple matter to conform them for the tool by using the recommended naming convention and loading them into the Unity package:
*_DLBC.tga (Base Map, RGBA, Original color texture to unlit)
*_DLAO.tga (Greyscale, Ambient Occlusion Map)
*_DLN.tga (RGB, Object-Space Normal Map)
*_DLBN.tga (RGB, Object-Space Bent Normal Map)
(The above notes are taken from the tool’s excellent documentation on GitHub.)
The process works very quickly, in a matter of seconds it had removed a lot of the baked light, you can clearly see the effect over the top right island, which was mostly in shadow. See before and after below:
If there are areas where artifacts occur, it’s easy to paint up a mask for those parts and compute again. Unity also accepts a position map for more accurate de-lighting (a forthcoming version of Knald will also allow position map baking).
I’m impressed that it worked so well out of the box. I did paint a mask for some of the areas where there were obvious repeating detail issues, but this was super quick to do in Photoshop. It was then loaded into the “mask” area of the tool and re-computed, which gave better results.
Later, I fixed up areas with missed ray hits in Substance Painter (visible as red blotches on the texture above) by clone stamping, then adjusted the overall albedo for PBR. A roughness map was created by manipulating the corrected albedo in Photoshop and hand painting for a wetter look on the lower stack, which was being hit with sea spray constantly.
I selected some of the darker areas of the albedo on the sea facing side and used those as a basis for working out where the wave spray was hitting, then masked using AO and curvature from Knald. Next, I used the “Leaks” particle brush in Substance Painter, which filled crevices and flat areas and created believable rivulets of runoff water.
Lastly, it remained to upload it to Sketchfab and see how it behaved under different real-time lighting conditions, here are some images:
Finally, here is the scene on Sketchfab:
The location of the subject is here:
I’ll be incorporating the Unity De-lighter into my future 3D scanning workflow, it’s fast and automatic for the most part, and at the very least will give you a worthwhile head start on creating believable albedo textures from photogrammetry, at best you won’t have to re-touch it at all.
Thanks Pete! Be sure to check out more of Pete’s work on his Sketchfab Profile. If you have a go at using the delighting tool yourself, why not share the results in the comments?
If you’re nterested in this topic, be sure to check out Unity’s Photogrammetry guide as well.