Fixing baking artefacts

Most baking errors come from a short list of causes: the rays travel in the wrong direction, start in the wrong place, hit the wrong part, or the engine reads the result differently from how it was baked. This guide lists each common artefact, what causes it and how to fix it, in the order worth trying.

Before anything else, check the result on the model in the target engine, not as a flat image. Many "artefacts" are only a mismatch between baker and engine. See tangent space and MikkTSpace.

#Quick diagnosis

#Skewing

What it looks like. A round bolt bakes as an oval, or a vertical detail near an edge appears to lean toward the corner.

Cause. With an averaged cage, rays travel along the averaged vertex normals. Near a 90° corner those normals point diagonally, and across a large face they blend gradually toward straight. EarthQuake: "the normals get averaged, this means when the mesh projects out, its ray direction is skewed", and it "has nothing to do with cage distance" (Polycount: Normal map with weird angle).

Fixes, in order:

1. Add geometry to the low poly. EarthQuake: "add a little more geometry so that the rays aren't pointing in such a skewed direction", with the cuts "very close to the ends of the mesh" (Polycount: Normal map with weird angle). mikhga: "You can add in geometry in certain areas to reduce skewed details" (Polycount: Mobile Asset and Optimization. Tips Please).
2. Use hard edges with UV splits at the corner. MoP: "Harden the edges around the ends, it'll change how the rays are cast" (Polycount: Normal map with weird angle). Combine with an averaged cage to avoid gaps; see smoothing, hard edges and smoothing groups.
3. Paint skew in Marmoset, which blends "between the face normals of the low poly mesh and the cage normals" per pixel (Marmoset: The Toolbag Baking Tutorial).

#Waviness

What it looks like. Straight panel lines or bevels on a cylinder come out wobbly.

Cause. The low poly is a coarse approximation of a curved high poly, so the distance between them changes across every facet. mikhga: "you get waviness in your normal-map is because you have discrepancy between your low-poly and high-poly mesh"; the original poster was baking "a full 20 sided" high poly into "a 6 sided cylinder" (Polycount: Mobile Asset and Optimization. Tips Please).

Fixes:

1. Add sides. "Using an appropriate number of sides for curved surfaces reduces waviness", and "increasing the number of sides tends to be more effective" than bevels alone (Marmoset Toolbag documentation: Baking Tips & Tricks). Shadow2Matrix was told "a bevel should not be a quick fix for waviness" (Polycount: Mobile Asset and Optimization. Tips Please).
2. Support edges and soft edges. mikhga made a copy of the low poly, "added in some support edges and then set all edges to soft" (Polycount: Mobile Asset and Optimization. Tips Please).
3. Paint skew or edit the cage for the remaining wobble (Marmoset: The Toolbag Baking Tutorial).

#Seams

What it looks like. A visible line along a UV border, often on a hard edge.

Causes and fixes:

• Hard edge without a UV split. "You will get seam artifacts as each face will try to bake onto the hard edge line" (Marmoset Toolbag documentation: Baking Tips & Tricks). Split UVs at every hard edge (Marmoset: The Toolbag Baking Tutorial).
• Split cage. An unaveraged cage "results in gaps or seams at hard edge borders" (Marmoset: The Toolbag Baking Tutorial). Use an averaged cage.
• Tangent space mismatch between baker and engine. Sync them (Polycount forum: Making sense of hard edges, uvs, normal maps…); see tangent space and MikkTSpace.
• Aliasing and resolution. Adobe: "Often UVs are not aligned to pixels which leads to aliasing and produces seams"; increase the resolution or align UV borders to pixels (Substance 3D Bakers documentation: Seams are visible after baking a normal…).
• Not enough padding. Dilate the bake and leave gaps between islands; see UV packing and padding.

#Ray misses

What it looks like. Patches of flat colour, black areas, or random noise where the high poly should be.

Causes and fixes:

• Cage too small: parts of the high poly poke outside it. Increase the offset locally.
• Holes in the high poly. "Holes in the high poly can result in errors, as the Baker is unable to hit a sensible surface"; cap the holes or model them into the low poly (Marmoset: The Toolbag Baking Tutorial; Marmoset Toolbag documentation: Baking Tips & Tricks).
• High and low not lined up. Adobe recommends you "freeze/reset the mesh transformation" and check that both meshes overlap (Substance 3D Bakers documentation: Normal map has strange colorful gradients).

#Intersections and bleeding between parts

What it looks like. A ghost image of one part printed onto another, or smeared edges where parts touch.

Cause. "Sometimes the rays go too far and hit the wrong geometry" (Substance 3D Bakers documentation: Mesh parts bleed between each other).

Fixes: bake groups or matching by name; a cage instead of large ray distances; a lower ray distance (Substance 3D Bakers documentation: Mesh parts bleed between each other). Or explode the model: "Anything that is going to give you intersection errors should be exploded" (Polycount: Exploded Mesh or not exploded mesh?).

#AO bleeding

What it looks like. Dark contact shadows from a part that is not there in the game (a removable magazine, an opening lid), or missing shadows where parts do touch.

Fixes:

• Marmoset's Ignore Groups option "will cast AO between different bake groups"; move the groups of moving parts aside so they do not receive or cast group-to-group AO (Marmoset: The Toolbag Baking Tutorial). Individual low meshes can also be excluded from Ignore Groups (Marmoset Toolbag documentation: Bake Groups).
• In Substance, the AO baker's Self Occlusion can be set to "By Mesh Name" to "avoid matching with unwanted geometry" (Substance 3D Bakers documentation: Ambient Occlusion from Mesh). Its Ignore Backface option "should be enabled to avoid artifacts" most of the time (Substance 3D Bakers documentation: Ambient Occlusion from Mesh).
• After an exploded bake, run "a second ao bake from the lowpoly" for contact shadows that should exist (Polycount: Exploded Mesh or not exploded mesh?).

#Gradients and X-shaped errors

Rainbow gradients across a normal map usually mean "a mismatch between the high-poly and low-poly mesh", including inverted vertex normals (Substance 3D Bakers documentation: Normal map has strange colorful gradients). Check normals, reset transforms, and check name matching if you use it (Substance 3D Bakers documentation: Normal map has strange colorful gradients).

X-shaped smudges across quads come from different triangulation. Marmoset calls them "a type of tangent space mismatch" (Marmoset Toolbag documentation: Baking Tips & Tricks); Adobe shows black artefacts when the engine triangulates differently from the baker (Substance 3D Bakers documentation: Triangulating before baking). Triangulate before baking and export the triangulated mesh. See triangulation.

#Jagged edges and lost detail

• Aliasing: "Aliasing often occurs because the geometry of the UVs is not aligned with the pixel grid". Raise the resolution, raise anti-aliasing, align UVs to the pixel grid and give UVs a better texel ratio (Substance 3D Bakers documentation: Aliasing on UV seams).
• Details too small: "Details smaller than 1 pixel will not be recorded properly" (Marmoset Toolbag documentation: Baking Tips & Tricks). Exaggerate bevels for the target resolution (80 Level: Best Practices for Hard Surface Normal Map…).
• Banding: bake normal maps in 16-bit, because "Upressing from 8bit to 16bit won't help" after the fact (Warframe Steam Workshop guide: Normal Map Baking).

#Further reading and watching