Virtual texturing

Virtual texturing splits textures into small tiles and keeps in memory only the tiles the camera can currently see, at the resolution it needs. Epic describes it as "splitting all mip levels into tiles of a small, fixed size", after which the GPU determines "which of the visible tiles are accessed by all visible pixels on the screen" (Unreal Engine Documentation: Streaming Virtual Texturing). It does for textures what virtualised geometry does for meshes.

#Why it matters

Ordinary texture streaming loads whole mip levels of a texture based on distance. A large texture seen partly up close therefore loads its full top mip, even if only a corner is on screen. Virtual texturing loads only the visible tiles: "No matter the size of the texture, the fixed tile size of the SVTs only considers the ones that are visible" (Unreal Engine Documentation: Streaming Virtual Texturing). Unity describes the effect: the GPU cache needed "depends mostly on the frame resolution, and not the number or resolution of textures in the Scene" (Unity Manual: Streaming Virtual Texturing).

#Two kinds in Unreal

• Streaming Virtual Texturing (SVT) streams tiles of authored textures from disk. It suits very large textures and UDIMs: "Each UDIM image can have a different resolution enabling virtual textures that have non-uniform pixel density" (Unreal Engine Documentation: Streaming Virtual Texturing). You enable it per texture, or with "Convert to Virtual Texture", which also updates the materials (Unreal Engine Documentation: Streaming Virtual Texturing).
• Runtime Virtual Texturing (RVT) "creates its texel data on demand using the GPU at runtime" (Unreal Engine Documentation: Runtime Virtual Texturing). Its main use is terrain: it "caches shading data over large areas", so a complex landscape material is evaluated once into the cache instead of every frame, and roads, decals and meshes can write into it to blend with the ground (Unreal Engine Documentation: Runtime Virtual Texturing). See terrain and landscapes.

Unity also has Streaming Virtual Texturing, which "splits textures into tiles, and progressively uploads these tiles to GPU memory when they are needed", but labels it "experimental and not ready for production use" (Unity Manual: Streaming Virtual Texturing).

#What it costs

• Shader cost. "Sampling from a virtual texture in your materials is more expensive than sampling a traditional texture": at least two texture fetches plus some maths (Unreal Engine Documentation: Streaming Virtual Texturing). Samples that share UVs can share a stack of up to 8 textures (Unreal Engine Documentation: Streaming Virtual Texturing).
• Pop-in. Streaming is "reactive by nature": a tile is only requested after a frame needed it, so higher-resolution tiles can pop in (Unreal Engine Documentation: Streaming Virtual Texturing).
• Filtering limits. Anisotropic filtering is limited by the Tile Border (default 4), and trilinear filtering is done stochastically, which can add noise with TAA (Unreal Engine Documentation: Streaming Virtual Texturing).
• Memory budgets. RVT memory pools are set in configuration "by specifying the maximum memory budget for your project" (Unreal Engine Documentation: Runtime Virtual Texturing).

#Common mistakes

• Converting everything. Small textures gain nothing and pay the extra sampling cost (Unreal Engine Documentation: Streaming Virtual Texturing).
• Non-power-of-two sizes. SVT textures "must be a power of 2", though not necessarily square (Unreal Engine Documentation: Streaming Virtual Texturing).
• Animated meshes writing to an RVT. "Skinned and animated primitives are not good candidates" (Unreal Engine Documentation: Runtime Virtual Texturing).
• Sampling virtual textures inside materials that fill an RVT. This is not supported (Unreal Engine Documentation: Runtime Virtual Texturing).
• Shipping Unity SVT without testing. It is still experimental (Unity Manual: Streaming Virtual Texturing).

Related: texture formats, compression and mipmaps, texel density, decals.

#Further reading and watching