Composition Fundamentals

In this section will talk about fundamental concepts that we need to know before we look at individual composition arcs.

Still under construction!

As composition is USD's most complicated topic, this section will be enhanced with more examples in the future. If you detect an error or have useful production examples, please submit a ticket, so we can improve the guide!

Composition Fundamentals In-A-Nutshell
Why should I understand the editing fundamentals?
Resources
Overview
Terminology
Composition Editing Principles - What do we need to know before we start?

TL;DR - Composition Fundamentals In-A-Nutshell

Composition editing works in the active layer stack via list editable ops.
When loading a layer (stack) from disk via Reference and Payload arcs, the contained composition structure is immutable (USD speak encapsulated). This means you can't remove the arcs within the loaded files. As for what the arcs can use for value resolution: The Inherit and Specialize arcs still target the "live" composed stage and therefore still reflect changes on top of the encapsulated arcs, the Reference arc is limited to seeing the encapsulated layer stack.

Why should I understand the editing fundamentals?

Tip

This section houses terminology essentials and a detailed explanation of how the underlying mechanism of editing/composing arcs works. Some may consider it a deep dive topic, we'd recommend starting out with it first though, as it saves time later on when you don't understand why something might not work.

Resources

Overview

Before we start looking at the actual composition arcs and their strength ordering rules, let's first look at how composition editing works.

Terminology

USD's mechanism of linking different USD files with each other is called composition. Let's first clarify some terminology before we start, so that we are all on the same page:

Opinion: A written value in a layer for a metadata field or property.
Layer: A layer is an USD file on disk with prims & properties. (Technically it can also be in memory, but for simplicity on this page, let's think of it as a file on disk). More info in our layer section.
Layer Stack: A stack of layers (Hehe 😉). We'll explain it more in detail below, just remember it is talking about all the loaded layers that use the sublayer composition arc.
Composition Arc: A method of linking (pointing to) another layer or another part of the scene hierarchy. USD has different kinds of composition arcs, each with a specific behavior.
Prim Index: Once USD has processed all of our composition arcs, it builds a prim index that tracks where values can come from. We can think of the prim index as something that outputs an ordered list of [(<layer (stack)>, <hierarchy path>), (<layer (stack)>, <hierarchy path>)] ordered by the composition rules.
Composed Value: When looking up a value of a property, USD then checks each location of the prim index for a value and moves on to the next one if it can't find one. If no value was found, it uses a schema fallback (if the property came from a schema), other wise it falls back to not having a value (USD speak: not being authored).

Composition is "easy" to explain in theory, but hard to master in production. It also a topic that keeps on giving and makes you question if you really understand USD. So don't worry if you don't fully understand the concepts of this page, they can take a long time to master. To be honest, it's one of those topics that you have to read yourself back into every time you plan on making larger changes to your pipeline.

We recommend really playing through as much scenarios as possible before you start using USD in production. Houdini is one of the best tools on the market that let's you easily concept and play around with composition. Therefore we will use it in our examples below.

Composition Editing Fundamentals - What do we need to know before we start?

Now before we talk about individual composition arcs, let's first focus on these different base principles composition runs on. These principles build on each other, so make sure you work through them in order they are listed below.

List-Editable Operations
Encapsulation
Layer Stack
Edit Target

List-Editable Operations (Ops)

USD has the concept of list editable operations. Instead of having a "flat" array ([Sdf.Path("/cube"), Sdf.Path("/sphere")]) that stores what files/hierarchy paths we want to point to, we have wrapper array class that stores multiple sub-arrays. When flattening the list op, USD removes duplicates, so that the end result is like an ordered Python set().

To make it even more confusing, composition arc list editable ops run on a different logic than "normal" list editable ops when looking at the final composed value.

We take a closer look at "normal" list editable ops in our List Editable Ops section, on this page we'll stay focused on the composition ones.

Alright, let's have a quick primer on how these work. There are three sub-classes for composition related list editable ops:

Sdf.ReferenceListOp: The list op for the reference composition arc, stores Sdf.Reference objects.
Sdf.PayloadListOp: The list op for the payload composition arc, stores Sdf.Reference objects.
Sdf.PathListOp: The list op for inherit and specialize composition arcs, as these arcs target another part of the hierarchy (hence path) and not a layer. It stores Sdf.Path objects.

These are 100% identical in terms of list ordering functionality, the only difference is what items they can store (as noted above). Let's start of simple with looking at the basics:

from pxr import Sdf
# Sdf.ReferenceListOp, Sdf.PayloadListOp, Sdf.PathListOp,
path_list_op = Sdf.PathListOp()
# There are multiple sub-lists, which are just normal Python lists.
# 'prependedItems', 'appendedItems', 'deletedItems', 'explicitItems',
# Legacy sub-lists (do not use these anymore): 'addedItems', 'orderedItems'
# Currently the way these are exposed to Python, you have to re-assign the list, instead of editing it in place.
# So this won't work:
path_list_op.prependedItems.append(Sdf.Path("/cube"))
path_list_op.appendedItems.append(Sdf.Path("/sphere"))
# Instead do this:
path_list_op.prependedItems = [Sdf.Path("/cube")]
path_list_op.appendedItems = [Sdf.Path("/sphere")]
# To clear the list op:
print(path_list_op) # Returns: SdfPathListOp(Prepended Items: [/cube], Appended Items: [/sphere])
path_list_op.Clear()
print(path_list_op) # Returns: SdfPathListOp()
# Repopulate via constructor
path_list_op = Sdf.PathListOp.Create(prependedItems = [Sdf.Path("/cube")], appendedItems = [Sdf.Path("/sphere")])
print(path_list_op) # Returns: SdfPathListOp(Prepended Items: [/cube], Appended Items: [/sphere])
# Add remove items
path_list_op.deletedItems = [Sdf.Path("/sphere")]
print(path_list_op) # Returns: SdfPathListOp(Deleted Items: [/sphere], Prepended Items: [/cube], Appended Items: [/sphere])
# Notice how it just stores lists, it doesn't actually apply them. We'll have a look at that next.

# In the high level API, all the function signatures that work on list-editable ops
# usually take a position kwarg which corresponds to what list to edit and the position (front/back)
Usd.ListPositionFrontOfAppendList
Usd.ListPositionBackOfAppendList
Usd.ListPositionFrontOfPrependList
Usd.ListPositionBackOfPrependList
# We cover how to use this is our 'Composition Arcs' section.

So far so good? Now let's look at how multiple of these list editable ops are combined. If you remember our layer section, each layer stores our prim specs and property specs. The composition list editable ops are stored as metadata on the prim specs. When USD composes the stage, it combines these and then starts building the composition based on the composed result of these metadata fields.

Let's mock how USD does this without layers:

from pxr import Sdf
### Merging basics ###
path_list_op_layer_top = Sdf.PathListOp.Create(deletedItems = [Sdf.Path("/cube")])
path_list_op_layer_middle = Sdf.PathListOp.Create(prependedItems = [Sdf.Path("/disc"), Sdf.Path("/cone")])
path_list_op_layer_bottom = Sdf.PathListOp.Create(prependedItems = [Sdf.Path("/cube")], appendedItems = [Sdf.Path("/cone"),Sdf.Path("/sphere")])

result = Sdf.PathListOp()
result = result.ApplyOperations(path_list_op_layer_top)
result = result.ApplyOperations(path_list_op_layer_middle)
result = result.ApplyOperations(path_list_op_layer_bottom)
# Notice how on merge it makes sure that each sublist does not have the values of the other sublists, just like a Python set()
print(result) # Returns: SdfPathListOp(Deleted Items: [/cube], Prepended Items: [/disc, /cone], Appended Items: [/sphere])
# Get the flattened result. This does not apply the deleteItems, only ApplyOperations does that. 
print(result.GetAddedOrExplicitItems()) # Returns: [Sdf.Path('/disc'), Sdf.Path('/cone'), Sdf.Path('/sphere')]

### Deleted and added items ###
path_list_op_layer_top = Sdf.PathListOp.Create(appendedItems=[Sdf.Path("/disc"), Sdf.Path("/cube")])
path_list_op_layer_middle = Sdf.PathListOp.Create(deletedItems = [Sdf.Path("/cube")])
path_list_op_layer_bottom = Sdf.PathListOp.Create(prependedItems = [Sdf.Path("/cube")], appendedItems = [Sdf.Path("/sphere")])

result = Sdf.PathListOp()
result = result.ApplyOperations(path_list_op_layer_top)
result = result.ApplyOperations(path_list_op_layer_middle)
result = result.ApplyOperations(path_list_op_layer_bottom)
print(result) # Returns: SdfPathListOp(Appended Items: [/sphere, /disc, /cube])
# Since it now was in the explicit list, it got removed.

### Explicit mode ###
# There is also an "explicit" mode. This clears all previous values on merge and marks the list as explicit.
# Once explicit and can't be un-explicited. An explicit list is like a reset, it 
# doesn't know anything about the previous values anymore. All lists that are merged
# after combine the result to be explicit.
path_list_op_layer_top = Sdf.PathListOp.Create(deletedItems = [Sdf.Path("/cube")])
path_list_op_layer_middle = Sdf.PathListOp.CreateExplicit([Sdf.Path("/disc")])
path_list_op_layer_bottom = Sdf.PathListOp.Create(prependedItems = [Sdf.Path("/cube")], appendedItems = [Sdf.Path("/sphere")])

result = Sdf.PathListOp()
result = result.ApplyOperations(path_list_op_layer_top)
result = result.ApplyOperations(path_list_op_layer_middle)
result = result.ApplyOperations(path_list_op_layer_bottom)
print(result, result.isExplicit) # Returns: SdfPathListOp(Explicit Items: [/disc]), True
# Notice how the deletedItems had no effect, as "/cube" is not in the explicit list.

path_list_op_layer_top = Sdf.PathListOp.Create(deletedItems = [Sdf.Path("/cube")])
path_list_op_layer_middle = Sdf.PathListOp.CreateExplicit([Sdf.Path("/disc"), Sdf.Path("/cube")])
path_list_op_layer_bottom = Sdf.PathListOp.Create(prependedItems = [Sdf.Path("/cube")], appendedItems = [Sdf.Path("/sphere")])

result = Sdf.PathListOp()
result = result.ApplyOperations(path_list_op_layer_top)
result = result.ApplyOperations(path_list_op_layer_middle)
result = result.ApplyOperations(path_list_op_layer_bottom)
print(result, result.isExplicit) # Returns: SdfPathListOp(Explicit Items: [/disc]), True
# Since it now was in the explicit list, it got removed.

When working with multiple layers, each layer can have list editable ops data in the composition metadata fields. It then gets merged, as mocked above. The result is a single flattened list, without duplicates, that then gets fed to the composition engine.

Here comes the fun part:

List-Editable Ops | Getting the composed (combined) value

When looking at the metadata of a prim via UIs (USD View/Houdini) or getting it via the Usd.Prim.GetMetadata() method, you will only see the list editable op of the last layer that edited the metadata, NOT the composed result.

This is probably the most confusing part of USD in my opinion when first starting out. To inspect the full composition result, we actually have to consult the PCP cache or run a Usd.PrimCompositionQuery. There is another caveat though too, as you'll see in the next section: Composition is encapsulated. This means our edits to list editable ops only work in the active layer stack. More info below!

In Houdini the list editable ops are exposed on the reference node. The "Reference Operation" parm sets what sub-array (prepend,append,delete) to use, the "Pre-Operation" sets it to .Clear() in Clear Reference Edits in active layer mode and to .ClearAndMakeExplicit() in "Clear All References" mode.

Here is how Houdini (but also the USD view) displays the references metadata field with different layers, as this is how the stage sees it.

You can see, as soon as we have our reference list editable op on different layers, the metadata only show the top most layer. To inspect all the references that are being loaded, we therefore need to look at the layer stack (the "Scene Graph Layers" panel) or perform a compsition query.

Also a hint on terminology: In the USD docs/glossary the Reference arc often refers to all composition arcs other than sublayer, I guess this is a relic, as this was probably the first arc. That's why Houdini uses a similar terminology.

Encapsulation

When you start digging through the API docs, you'll read the word "encapsulation" a few times. Here is what it means and why it is crucial to understand.

Encapsulation | Why are layers loaded via references/payloads composition arc locked?

To make USD composition fast and more understandable, the content of what is loaded from an external file via the Reference and Payload composition arcs, is composition locked or as USD calls it encapsulated. This means that you can't remove any of the composition arcs in the layer stack, that is being loaded, via the list editable ops deletedItems list or via the explicitItems.

The only way to get rid of a payload/reference is by putting it behind a variant in the first place and then changing the variant selection. This can have some unwanted side effects though. You can find a detailed explanation with an example here: USD FAQ - When can you delete a reference?

Encapsulation | Are my loaded layers then self contained?

You might be wondering now, if encapsulation forces the content of Reference/Payload to be self contained, in the sense that the composition arcs within that file do not "look" outside the file. The answer is: It depends on the composition arc.

For Inherits and Specializes the arcs still evaluate relative to the composed scene. E.g. that means if you have an inherit somewhere in a referenced in layer stack, that inherit will still be live. So if you edit a property in the active stage, that gets inherited somewhere in the file, it will still propagate all the changes from the inherit source to all the inherit targets. The only thing that is "locked" is the composition arcs structure, not the way the composition arc evaluates. This extra "live" lookup has a performance penalty, so be careful with using Inherits and Specializes when nesting layers stacks via References and Payloads.

For Internal References this does not work though. They can only see the encapsulated layer stack and not the "live" composed stage. This makes composition faster for internal references.

We show some interactive examples in Houdini in our LIVRPS section, as this is hard to describe in words.

Layer Stack

What is the layer stack, that we keep mentioning, you might ask yourself? To quote from the USD Glossary

The ordered set of layers resulting from the recursive gathering of all SubLayers of a Layer, plus the layer itself as first and strongest.

So to summarize, all (sub)-layers in the stage that were not loaded by Reference and Payload arcs.

Now you might be thinking, isn't that the same thing as when we open a Usd file via Usd.Stage.Open? Well kind of, yes. When opening a stage, the USD file you open and its sublayers are the layer stack. USD actually calls this the Root Layer Stack (it also includes the sessions layers). So one could say, editing a stage is process of editing a layer stack. To extend that analogy, we could call a stage, that was written to disk and is being loaded via Reference and Payload arcs, an encapsulated layer stack.

These are the important things to understand (as also mentioned in the glossary):

Layer Stack | How does it affect composition?

Composition arcs target the layer stack, not individual layers. They recursively target the composed result (aka the result of all layers combined via the composition arc rules) of each layer they load in.
We can only list edit composition arcs via list editable ops in the active layer stack. The active layer stack is usually the active stage (unless when we "hack" around it via edit targets, which you 99% of the time don't do).

So to make it clear again (as this is very important when we setup our asset/shot composition structure): We can only update Reference and Payload arcs in the active layer stack. Once the active layer stack has been loaded via Reference and Payload arcs into another layer stack, it is encapsulated and we can't change the composition structure.

This means to keep our pipeline flexible, we usually have "only" three kind of layer stacks:

Asset Layer Stack: When building assets, we build a packaged asset element. The end result is a (nested) layer stack that loads in different aspects of the asset (model/materials/fx/etc.). Here the main "asset.usd" file, that at the end we reference into our shots, is in control of "final" asset layer stack. We usually don't have any encapsulation issue scenarios, as the different assets layers are usually self contained or our asset composition structure is usually developed to sidestep encapsulation problems via variants.
Shot Layer Stack: The shot layer stack is the one that sublayers in all of your different shot layers that come from different departments. That's right, since we sublayer everything, we still have access to list editable ops on everything that is loaded in via composition arcs that are generated in individual shot layers. This keeps the shot pipeline flexible, as we don't run into the encapsulation problem.
Set/Environment/Assembly Layer Stack (Optional): We can also reference in multiple assets to an assembly type of asset, that then gets referenced into our shots. This is where you might run into encapsulation problems. For example, if we want to remove a reference from the set from our shot stage, we can't as it is baked into the composition structure. The usual way around it is to: 1. Write the assembly with variants, so we can variant away unwanted assets. 2. Deactivate the asset reference 3. Write the asset reference with variants and then switch to a variant that is empty.

Edit Target

To sum up edit targets in once sentence:

Pro Tip | Edit Targets

A edit target defines, what layer all calls in the high level API should write to.

Let's take a look what that means:

An edit target's job is to map from one namespace to another, we mainly use them for writing to layers in the active layer stack (though we could target any layer) and to write variants, as these are written "inline" and therefore need an extra name space injection.

Setting the edit target is done on the stage, as this is our "controller" of layers in the high level API:

stage.SetEditTarget(layer)
# We can also explicitly create the edit target:
# Or
stage.SetEditTarget(Usd.EditTarget(layer))
# Or
stage.SetEditTarget(stage.GetEditTargetForLocalLayer(layer))
# These all have the same effect.

from pxr import Sdf, Usd
## Standard way of using edit targets
stage = Usd.Stage.CreateInMemory()
root_layer = stage.GetRootLayer()
a_layer = Sdf.Layer.CreateAnonymous("LayerA")
b_layer = Sdf.Layer.CreateAnonymous("LayerB")
root_layer.subLayerPaths.append(a_layer.identifier)
root_layer.subLayerPaths.append(b_layer.identifier)
# Direct edits to different layers
stage.SetEditTarget(Usd.EditTarget(a_layer))
bicycle_prim = stage.DefinePrim(Sdf.Path("/bicycle"), "Xform")
stage.SetEditTarget(Usd.EditTarget(b_layer))
car_prim = stage.DefinePrim(Sdf.Path("/car"), "Xform")
print(b_layer.ExportToString())
"""Returns:
#sdf 1.4.32
def Xform "car"
{
}
"""
## Reference/Payload Edit Targets
asset_stage = Usd.Stage.CreateInMemory()
cube_prim = asset_stage.DefinePrim(Sdf.Path("/root/RENDER/cube"), "Xform")
asset_layer = asset_stage.GetRootLayer()
shot_stage = Usd.Stage.CreateInMemory()
car_prim = shot_stage.DefinePrim(Sdf.Path("/set/garage/car"), "Xform")
car_prim.GetReferences().AddReference(asset_layer.identifier, Sdf.Path("/root"), Sdf.LayerOffset(10))
# We can't construct edit targets to layers that are not sublayers in the active layer stack.
# Is this a bug? According to the docs https://openusd.org/dev/api/class_usd_edit_target.html it should work.
# shot_stage.SetEditTarget(Usd.EditTarget(asset_layer))
## Variant Edit Targets
from pxr import Sdf, Usd
stage = Usd.Stage.CreateInMemory()
bicycle_prim_path = Sdf.Path("/bicycle")
bicycle_prim = stage.DefinePrim(bicycle_prim_path, "Xform")
variant_sets_api = bicycle_prim.GetVariantSets()
variant_set_api = variant_sets_api.AddVariantSet("color", position=Usd.ListPositionBackOfPrependList)
variant_set_api.AddVariant("colorA")
variant_set_api.SetVariantSelection("colorA")
with variant_set_api.GetVariantEditContext():
    # Anything we write in this edit target context, goes into the variant.
    cube_prim_path = bicycle_prim_path.AppendChild("cube")
    cube_prim = stage.DefinePrim(cube_prim_path, "Cube")
print(stage.GetEditTarget().GetLayer().ExportToString())
"""Returns:
#usda 1.0

def Xform "bicycle" (
    variants = {
        string color = "colorA"
    }
    prepend variantSets = "color"
)
{
    variantSet "color" = {
        "colorA" {
            def Cube "cube"
            {
            }

        }
    }
}
"""

For convenience, USD also offers a context manager for variants, so that we don't have to revert to the previous edit target once we are done:

from pxr import Sdf, Usd
stage = Usd.Stage.CreateInMemory()
root_layer = stage.GetRootLayer()
a_layer = Sdf.Layer.CreateAnonymous("LayerA")
b_layer = Sdf.Layer.CreateAnonymous("LayerB")
root_layer.subLayerPaths.append(a_layer.identifier)
root_layer.subLayerPaths.append(b_layer.identifier)
# Set edit target to a_layer
stage.SetEditTarget(a_layer)
bicycle_prim_path = Sdf.Path("/bicycle")
bicycle_prim = stage.DefinePrim(bicycle_prim_path, "Xform")
edit_target = Usd.EditTarget(b_layer)
with Usd.EditContext(stage, edit_target):
    print("Edit Target Layer:", stage.GetEditTarget().GetLayer()) # Edit Target Layer: Sdf.Find('anon:0x7ff9f4391580:LayerB')
    car_prim_path = Sdf.Path("/car")
    car_prim = stage.DefinePrim(car_prim_path, "Xform")
print("Edit Target Layer:", stage.GetEditTarget().GetLayer()) # Edit Target Layer: Sdf.Find('anon:0x7ff9f4391580:LayerA')
# Verify result
print(a_layer.ExportToString())
"""Returns:
#sdf 1.4.32
def Xform "bicycle"
{
}
"""
print(b_layer.ExportToString())
"""Returns:
#sdf 1.4.32
def Xform "car"
{
}
"""

Usd Survival Guide

Composition Fundamentals

Table of Contents

TL;DR - Composition Fundamentals In-A-Nutshell

Why should I understand the editing fundamentals?

Resources

Overview

Terminology

Composition Editing Fundamentals - What do we need to know before we start?

List-Editable Operations (Ops)

Encapsulation

Layer Stack

Edit Target