"Go" "Shader Panel" Shader compilation in Unity 4.5

"Original" Shader compilation in Unity 4.5

A story in several parts. 1) How shader compilation was done in upcoming Unity 4.5; and 2) How it is developed. First one is probably interesting to Unity users; Whereas second one for the ones curious on how we work and develop stuff.

Short summary:unity 4.5 would have a "Wow, many shaders, much fast" shader importing and better error reporting.

Current state (Unity <=4.3)

When you create a new shader file (. shader) in Unity or edit existing one, we launch a "shader importer". Just like for any other changed asset. That shader importer does some parsing, and then compiles thewhole shader to all platform backends we Support.

Typically when you create a simple surface shader, it internally expands into the or so internal shader variants (classic " Preprocessor driven Uber-shader "approach). and typically there 7 or so platform backends to compile into (D3d9, D3d11, OpenGL, Gles, Gles3, d3d11_9x, Flash–mor E If you have the console licenses). This means, each time of the anything in the shader, a couple hundred shaders is being compiled. And all this assuming you has a fairly simple shader-if your throw in some multi_compile directives, you'll be looking a T thousands or tens of thousands shaders being compiled. Each. and. Every. Time.

Does it make sense to do? Not really.

Like the most of ' why is we doing this? ' situations, this one also evolved organically, and can being explained with ' it sounded Like a good idea at the time "and" it does not fix itself unless someone works on it ".

A long time ago, Unity only had one or both shader platform Backends (OpenGL and D3d9). And the amount of shader variants people were doing was much lower. With time, we got both more backends, and more variants; And it became very apparent that someone needs to solve this problem.

In addition to the above, there were other problems with shader compilation, for example:

• Errors in shaders were reported, well, "in a funny". Sometimes the line numbers does not make a sense–which is quite confusing.
• Debugging generated surface shader code involved quite some voodoo tricks ( #pragma debug etc).
• Shader Importer tried to multi-thread compilation of these hundreds of shaders, but some backend compilers (CG) has Inter NAL Global mutexes and does not parallelize well.
• Shader importer process is running out of the memory for really large Multi_compile variant counts.

So we ' re changing what shader importing works in Unity 4.5. The rest of this post is mostly dumps of our internal wiki pages.

Shader Importing in Unity 4.5

• No runtime/platforms changes compared to 4.3/4.5–all changes is editoronly.
• No shader functionality changes compared to 4.3/4.5.
• Shader Importing is much faster; Especially complex surface shaders (Marmoset skyshop etc).
  • Reimporting all shaders on graphics tests project:3 minutes with 4.3, seconds with this.
• Errors in shaders is reported on correct lines; Errors in shader include (. cginc) files is reported with the filename & line number correctly.
  • was mostly "completely broken" before, especially when include files came into play.
  • On D3d11 backend We were reporting error column as the line, hah:) At some point during D3dcompiler DLL upgrade it changed error printing syntax and we were parsing it wrong. Now added unit tests so hopefully it'll never break again.
• Surface Shader Debugging workflow is much better.
  • No more "Add #pragma debug, open compiled shader, remove tons of assembly" nonsense. Just one button in Inspector, "Show generated code".
  • Generated Surface Shader Code has some comments and better indentation. It is actually readable code now!
• Shader Inspector Improvements:
  • Errors List had scrollview when it ' s long; Can double click on errors to open correct file/line; Can copy error text via context click menu; Each error clearly indicates which platform it happened for.
  • Investigating compiled shader is saner. One button to show compiled results for currently active platform; Another button to show for all platforms.
• Misc bugfixes
  • Fixed multi_compile preprocessor directives in surface shaders sometimes producing very unexpected results.
  • UTF8 BOM markers in. Shader or. cginc files don ' t produce errors.
  • Shader include files can is at Non-ascii folders and filenames.

Overview of how it works

• Instead of compiling all shader variants for all possible platforms at import time:
  • Only do minimal processing of the shader (surface shader generation etc).
  • Actually compile the shader variants only when needed.
  • Instead of typical work of compiling 100-1000 internal shaders in import time, this usually ends up compiling just a handf Ul.
• at player build time, compile all the shader variants for that target platform
  • Cache identical Shaders und Er library/shadercache.
  • so at player build time, only not-yet-ever-compiled shaders is compiled; and always only for the platforms that need them. If you never ever use Flash, for example and then none of the shaders would be is compiled for flash  (as opposed to 4.3, whe Re all shaders is compiled to any platforms, even if you never ever, need them) .
• Shader compiler (Cgbatch) changes from being invoked for each Shader import, into being run as a "service process"
  • inter-process communication between compiler process & Unity; Using same infrastructure as for VersionControl plugins integration.
  • At player build time, the go wide and use all CPUs cores to do shader compilation. Old compiler tried to internally multithread, but couldn ' t due to some platforms not being thread-safe. Now, we just launch one compiler process per core and they can go fully parallel.
  • Helps with out-of-memory crashes as well, since shader compiler process never needs to hold bazillion of shader variants I n Memory all at Once-what it sees are one variant at a time.

How it is developed

This is mostly a one-or-two person effort, and developed in several "sprints". For the one we used our internal wiki for detailed task planning (confluence "task lists"), but we could has just as wel L use Trello or something similar. Overall this was  probably  around, months of actual work–but of out spread during much time. Initial Sprint started in March, and landed in a ' we think we can ship this tomorrow ' state to 4.5 codebase just in t IME for 1st Alpha Build (October). Minor tweaks and fixes were done during 4.5 Alpha & beta period.  should ship anyday now, fingers crossed:)

Surprisingly (or perhaps not), largest piece of work is around "how does the report errors in shaders?". Since now shader variants is imported only on demand, which means some errors can be discovered only "some time a fter initial import ". This was a by-design change, however-as the previous approach of "Let's compile all possible variants for all possible pl Atforms "Clearly does not the scale in terms of iteration time. However, this "shader seemed-like it had not had any errors, but whoops now it had" is clearly a potential downside. Oh well; As with almost everything there is upsides & downsides.

Most of development is done on a Unity 4.3-based branch, and after something is working we were sending off custom "4.3 + New Shader Importer "builds to the beta testing group. We were doing this before any 4.5 alpha even started to get early feedback. Perhaps the nicest feedback I ever got:

I ' ve now used the build for about a week and I ' m completely blown away with how it had changed how I work with shaders.

I can try out things-quicker.
I am no longer scared of making a typo in an include file.
These combine to making me play around a lot more when working.
Because of this I found off how to do fake HDR with filmic tonemapping [on my mobile target].

The thought of going back to regular beta without this [shader compiler] really scares me;)

Anyhoo, here's a dump of the tasks from our wiki (all of the them had little checkboxes that we ' d tick off if done). As usual, "it basically works and is awesome!" were achieved after first week's work (1st sprint). What is left after is "fix all the TODOs, does all the boring remaining work" etc.

March Sprint:

• Make Cgbatch a DLL
  • Run Unit Tests
  • Import Shaders from DLL
  • Don ' t use temp files
• Shader Importer Changes
  • Change surface shader-only generate source code and not-do-compilation
  • Make a "Open surface compiler Output" button
  • At import time, does surface shader generation & Cache The result (serialize in shader, editor only)
  • Also Process all cginclude blocks and actually does #includes at import time, and cache the result Program blocks, with no #include statements)
  • Shaderlab::P the needs to know it would have yet-uncompiled programs inside, and able to find appropriate Cgprogram block:
    • ADD syntax to Shaderlab, something like Pass {Gpuprogramid int}
    • Make Cgbatch no compilation, just extract Cgprogram blocks, assign IDs to them, and replace them with "Gpuprogrami D XXX "
    • "Cache the result" as editor-only data in Shader:map of snippet ID, Cgprogram block text
  • Cgbatch, add function to compile one shader variant (CG program block source + platform + keywords in, bytecode + errors O Ut
  • Remove all #include handling from actual shader compilers in Cgbatch
  • Change output of a single shader compilation to IS in Shaderlab program/subprogram/bindings syntax, and to produce data Directly. Shader code as a string, some virtual interface that would report all uniforms/textures/... for the reflection data.
• Compile Shaders on Demand
  • Data file format for GPU programs & their params
  • Shaderlab Pass has map:m_gpuprogramlookup (keywords-gpuprogram).
  • Getmatchingsubprogram:
    • Return one from M_gpuprogramlookup if found. Get from cache if found
    • Compile program snippet if not found
    • Write into cache

July Sprint:

• Pull and Merge last 3 months of trunk
• Player Build Pipeline
  • When building player/bundle, compile all shader snippets and include them
  • Exclude_renderers/include_renderers, trickle down to shader snippet data
  • Do-properly when building-a "no Target" (everything in) platforms
    • Snippets is saved in built-in resource files (needed? not?)
  • Make building built-in resource files work
    • DX11 9.x shaders aren ' t included
    • Make Building Editor resource file work
  • Multithread the "missing combinations" compilation while building the player.
    • Ensure thread safety in snippet cache
• Report Errors sensibly
• Misc
  • Each shader snippet needs to know keyword permutation possibly Needed:cgbatch extracts so, serialized in snippet (like vector< vector >)
  • Fix Glslprogram Snippets
  • Separate "compiler version" from "Cgbatch version"; Embed compiler version into snippet Data & hash
  • Fix Usepass

August Sprint:

• Move to a 4.3-based branch
• GFX Test Failures
  • Metro, failing shadow related tests
  • Flash, failing custom lightmap function test
• Error reporting:figure out how to deal with late-discovered errors. If there ' s bad syntax, typo etc.; Effectively shader is "broken". If a backend shader compiler reports an error:
  • Return Pink ' error shader ' for all programs i.e. if any of VE Rtex/pixel/had an error, we need to use the pink shaders for all of them.
  • Log the error to console.
  • ADD error to the shader, so it's displayed in the editor. Can ' t serialize shader at the time, so add shaders to some database under Library (guid>errors).
    • SQLite Database with shader GUID, set of errors.
  • Add shader to List of ' shaders with errors '; After rendering loop was done go through them and make them use Pink error shader. (Effectively this does no change current (4.2) Behavior:if A syntax error, shader is pink).
• Misc
  • Fix shader Fallback when it pulls in shader snippets
  • "Mesh components required by shader" in build time-need to figure them out! problem; Needs to compile the variants to even know it.
  • Better #include processing, now includes same files multiple times
• Make Cgbatch again to an executable (for future ...)
  • Adapt externalprocess for all communication
  • Make unit tests work again
  • Remove all Jobscheduler/mutex stuff from Cgbatch; Spawn multiple processes instead
  • Feels like is leaking memory, with to check
• Shader Inspector
  • Only show "Open surface Shader" button for surface shaders
  • "Open compiled shader" is useless now, doesn ' t display shader asm. Need to redo it somehow.

September Sprint:

• Make ready for 4.5 trunk
  • Merge with current trunk
  • Make TeamCity Green
  • Land to trunk!
• Make 4.3-based TeamCity Green
  • Build Builtin Resources, fails with shader compiler RPC errors gl-only GFX test failures (cgprops test)
  • Glslprogram preprocessing broken, add tests
  • Mobile GFX test Failures in Toonycolors
• Error reporting and #include handling
  • Fixing line number reporting once and for all, with tests.
  • Report errors on correct. cginc files and correct lines on them
  • Solve multiple includes & preprocessor affecting includes this way:at snippet extraction time, does not do include proc essing! Just hash include contents and feed that into the snippet hash.
  • UTF8 BOM in included files confusing some compilers
  • Unicode paths to files confusing some compilers
  • After shader import, immediately compile in least one variant, so, any stupid errors is caught & displayed Immedi ately.
• Misc
  • Make the flags like "Does the shader support shadows?" Work with the new GPU programs coming in
  • Check up Case 550197
  • Multi_compile vs. surface shaders, fix that
• Shader Inspector
  • Better Display of errors (lines & locations)
  • button to "exhaustively check shader" – Compiles all variants/platforms.
  • Shader snippet/total Size Stats

What ' s next?

Some Shader compilation land would go into Unity 5.0 and 5.x. Outline of our another wiki page describing 5.x Relat Ed work:

• 4.5 Fixes "Compiling shaders is slow" problem.
• Need to fix ' New standard shader produces very large shader files ' (due to lots of variants-5000 variants, 100MB) problem.
• Need to fix shader LOD with new standard shader "problem.

"Go" "Shader Panel" Shader compilation in Unity 4.5