Tangent space and MikkTSpace

Tangent space is a small coordinate frame at every point of a mesh, made of the surface normal (N), a tangent (T) that follows the U direction of the UVs, and a bitangent (B) that follows V. A tangent-space normal map stores directions relative to this frame, which is why the same map works on a mesh that moves, bends or is rotated. Blender calls tangent space "the default, and the right choice in most cases" for baked normal maps (Blender Manual: Render Baking).

#Matching tangent space in baker and engine

The baker writes colours by comparing the high poly against the low poly's tangent frame; the engine reads them back with its own tangent frame. If the two frames differ even slightly, the shading is wrong: soft gradients, seams or X-shaped smudges. MikkTSpace's source header states the requirement plainly: "the normal map sampler must use the exact inverse of the pixel shader transformation" (GitHub: mikktspace.h).

Tangent frames depend on the mesh's normals, UVs and triangulation. Marmoset calls X-shaped errors "a type of tangent space mismatch, as variance in the edge direction can cause the tangent basis to differ" (Marmoset Toolbag documentation: Baking Tips & Tricks). This is one reason to triangulate before baking (Substance 3D Bakers documentation: Triangulating before baking).

#MikkTSpace

MikkTSpace is a tangent-space algorithm written by Morten S. Mikkelsen in 2011, "designed to consistently generate the same tangent spaces, for a given mesh, in any tool in which it is used" (GitHub: mikktspace.h). It became the common standard for game art:

• Marmoset recommends "Mikk / xNormal" because "that is what Unity, the Unreal Engine, and various other applications use" (Marmoset Toolbag documentation: Tangent & Handedness).
• Substance's bakers recompute tangents with MikkTSpace by default (Substance 3D Bakers documentation: Tangent space).
• Unity's importer has a "Calculate Mikktspace" option for tangents (Unity Manual: Model tab), and Unreal's importer lets you choose "between MikkTSpace or Built-in" generation (Unreal Engine documentation: FBX Import Options Reference).

Even within MikkTSpace there are small variants. Substance Painter's per-fragment option is set off for xNormal and Unity 5.3+, and on for Unreal Engine 4, Blender and Unity HDRP (Substance 3D Bakers documentation: Tangent space). Your baker's documentation will list the right preset per engine.

#The synced workflow

A synced (or synced normals) workflow means the baker and the engine use exactly the same tangent basis, on exactly the same mesh. With it, EarthQuake says, "there is simply no drawback to using hard edges along your uv seams", and you "can often get away with less uv seams" (Polycount forum: Making sense of hard edges, uvs, normal maps…). In practice:

1. Triangulate the low poly and export it with its normals (and tangents, if your pipeline uses them).
2. Bake with the engine's tangent space; Marmoset uses whatever "the tangent space that your mesh is currently set to" is (Marmoset: The Toolbag Baking Tutorial).
3. Import into the engine without recomputing normals. In Unreal, Import Normals or Import Normals and Tangents (Unreal Engine documentation: FBX Import Options Reference). See smoothing, hard edges and smoothing groups.

#OpenGL (Y+) versus DirectX (Y−)

Separately from the tangent algorithm, programs disagree on which way the green channel points. Adobe: "OpenGL expects the first pixel to be at the bottom while DirectX expects it to be at the top" (Substance 3D Bakers documentation: What is the difference between the OpenGL and…). Marmoset: "Right-handedness is sometimes called OpenGL or Maya style, while left-handedness is called DirectX or 3DS Max style" (Marmoset Toolbag documentation: Tangent & Handedness).

To convert between them, flip the green channel. Bakers have a toggle (Toolbag's Flip Y (Marmoset: The Toolbag Baking Tutorial)), and engines can flip on import; Unity's importer option "Indicates whether to invert the green channel values of a normal map" (Unity Scripting Reference: TextureImporter.flipGreenChannel). Adobe's advice for any other program: "refer to the target application in which your textures will be used" (Substance 3D Bakers documentation: What is the difference between the OpenGL and…).

#Common mistakes

• Baking in one tangent space and rendering in another. Marmoset warns of "potentially serious shading artifacts", even on a perfect bake (Marmoset Toolbag documentation: Tangent & Handedness).
• Letting the engine recompute normals after a synced bake (Unreal Engine documentation: FBX Import Options Reference).
• Different triangulation in baker and engine (Substance 3D Bakers documentation: Triangulating before baking; GitHub: mikktspace.h).
• Wrong green channel, which makes bumps look like dents.
• Editing the low poly after baking. Any change to normals or UVs changes the tangent frame.

#Related terms

normal map · baking · smoothing, hard edges and smoothing groups · triangulation · coordinate systems and handedness · fixing baking artefacts

#Further reading and watching