Siggraph Presentation

This guide will be officially introduced at Siggraph 2023 - Houdini Hive on Wednesday, 9. of August 2023 at 11:00 AM PST.

List Editable Ops (Operations)

On this page we will have a look at list editable ops when not being used in composition arcs.

Danger

As mentioned in our fundamentals section, list editable ops play a crucial role to understanding composition. Please read that section before this one, as we build on what was written there.

Table of Contents

  1. List Editable Ops In-A-Nutshell
  2. What should I use it for?
  3. Resources
  4. Overview
  5. Composition Arcs
  6. Relationships
  7. Metadata

TL;DR List Editable Ops - In-A-Nutshell

  • USD has the concept of list editable operations. Instead of having a "flat" array ([Sdf.Path("/cube"), Sdf.Path("/sphere")]) that stores data, we have wrapper array class that stores multiple sub-arrays (prependedItems, appendedItems, deletedItems, explicitItems). When flattening the list op, it merges the prepended and appended items and also removes items in deletedItems as well as duplicates, so that the end result is like an ordered Python set(). When in explicit mode, it only keeps the elements in explicitItems and ignores previous layers. This merging is done per layer, so that for example an appendedItems op in a higher layer, gets added to an explicitItems from a lower layer. This allows us to average the array data over multiple layers.
  • List editable ops behave differently based on the type:
    • Composition: When using list editable ops to define composition arcs, we can only edit them in the active layer stack. Once referenced or payloaded, they become encapsulated.
    • Relationships/Metadata: When making use of list editable ops when defining relationships and metadata, we do not have encapsulation. This means that any layer stack can add/delete/set explicit the list editable type. See the examples below for more info.

What should I use it for?

Tip

Using list editable ops in non composition arc scenarios is rare, as we often want a more attribute like value resolution behavior. It is good to know though that the mechanism is there. A good production use case is making metadata that tracks asset dependencies list editable, that way all layers can contribute to the sidecar data.

Resources

Overview

Let's first go other how list editable ops are edited and applied:

These are the list editable ops that are available to us:

  • Composition:
    • Sdf.PathListOp
    • Sdf.PayloadListOp
    • Sdf.ReferenceListOp
  • Base Data Types:
    • Sdf.PathListOp
    • Sdf.StringListOp
    • Sdf.TokenListOp
    • Sdf.IntListOp
    • Sdf.Int64ListOp
    • Sdf.UIntListOp
    • Sdf.UInt64ListOp

USD has the concept of list editable operations. Instead of having a "flat" array ([Sdf.Path("/cube"), Sdf.Path("/sphere")]) that stores data, we have wrapper array class that stores multiple sub-arrays (prependedItems, appendedItems, deletedItems, explicitItems). When flattening the list op, it merges the prepended and appended items and also removes items in deletedItems as well as duplicates, so that the end result is like an ordered Python set(). When in explicit mode, it only keeps the elements in explicitItems and ignores previous layers. This merging is done per layer, so that for example an appendedItems op in a higher layer, gets added to an explicitItems from a lower layer. This allows us to average the array data over multiple layers.

All list editable ops work the same way, the only difference is what data they can hold.

These are 100% identical in terms of list ordering functionality, the only difference is what items they can store (as noted above). Let's start of simple with looking at the basics:

from pxr import Sdf
path_list_op = Sdf.PathListOp()
# There are multiple sub-lists, which are just normal Python lists.
# 'prependedItems', 'appendedItems', 'deletedItems', 'explicitItems',
# Legacy sub-lists (do not use these anymore): 'addedItems', 'orderedItems'
# Currently the way these are exposed to Python, you have to re-assign the list, instead of editing it in place.
# So this won't work:
path_list_op.prependedItems.append(Sdf.Path("/cube"))
path_list_op.appendedItems.append(Sdf.Path("/sphere"))
# Instead do this:
path_list_op.prependedItems = [Sdf.Path("/cube")]
path_list_op.appendedItems = [Sdf.Path("/sphere")]
# To clear the list op:
print(path_list_op) # Returns: SdfPathListOp(Prepended Items: [/cube], Appended Items: [/sphere])
path_list_op.Clear()
print(path_list_op) # Returns: SdfPathListOp()
# Repopulate via constructor
path_list_op = Sdf.PathListOp.Create(prependedItems = [Sdf.Path("/cube")], appendedItems = [Sdf.Path("/sphere")])
print(path_list_op) # Returns: SdfPathListOp(Prepended Items: [/cube], Appended Items: [/sphere])
# Add remove items
path_list_op.deletedItems = [Sdf.Path("/sphere")]
print(path_list_op) # Returns: SdfPathListOp(Deleted Items: [/sphere], Prepended Items: [/cube], Appended Items: [/sphere])
# Notice how it just stores lists, it doesn't actually apply them. We'll have a look at that next.

When working with the high level API, all the function signatures that work on list-editable ops usually take a position kwarg which corresponds to what list to edit and the position (front/back):

  • Usd.ListPositionFrontOfAppendList: Prepend to append list, the same as Sdf.<Type>ListOp.appendedItems.insert(0, item)
  • Usd.ListPositionBackOfAppendList: Append to append list, the same as Sdf.<Type>ListOp.appendedItems.append(item)
  • Usd.ListPositionFrontOfPrependList: Prepend to prepend list, the same as Sdf.<Type>ListOp.appendedItems.insert(0, item)
  • Usd.ListPositionBackOfPrependList: Append to prepend list, the same as Sdf.<Type>ListOp.appendedItems.append(item)
# For example when editing a relationship:
from pxr import Sdf, Usd
stage = Usd.Stage.CreateInMemory()
yard_prim = stage.DefinePrim("/yard")
car_prim = stage.DefinePrim("/car")
rel = car_prim.CreateRelationship("locationsOfInterest")
rel.AddTarget(yard_prim.GetPath(), position=Usd.ListPositionFrontOfAppendList)
# Result:
"""
def "car"
{
    custom rel locationsOfInterest
    append rel locationsOfInterest = </yard>
}
"""
# The "Set<Function>" signatures write an explicit list:
rel.SetTargets([yard_prim.GetPath()])
# Result:
"""
def "car"
{
    custom rel locationsOfInterest = </yard>
}
"""

Now let's look at how multiple of these list editable ops are combined.

Pro Tip | List Editable OPs in Metadata

Again it is very important, that composition arc related list editable ops get combined with a different rule set. We cover this extensively in our fundamentals section.

Non-composition related list editable ops do not make use of encapsulation. This means that any layer can contribute to the result, meaning any layer can add/remove/set explicit. When getting the value of the list op for non-composition arc list ops, we get the absolute result, in the form of an explicit list editable item list.

In contrast: When looking at composition list editable ops, we only get the value of the last layer that edited the value, and we have to use composition queries to get the actual result.

This makes non-composition list editable ops a great mechanism to store averaged side car data. Checkout our Houdini example below, to see this in action.

Let's mock how USD does this (without using Sdf.Layers to keep it simple):

from pxr import Sdf
### Merging basics ###
path_list_op_layer_top = Sdf.PathListOp.Create(deletedItems = [Sdf.Path("/cube")])
path_list_op_layer_middle = Sdf.PathListOp.Create(prependedItems = [Sdf.Path("/disc"), Sdf.Path("/cone")])
path_list_op_layer_bottom = Sdf.PathListOp.Create(prependedItems = [Sdf.Path("/cube")], appendedItems = [Sdf.Path("/cone"),Sdf.Path("/sphere")])

result = Sdf.PathListOp()
result = result.ApplyOperations(path_list_op_layer_top)
result = result.ApplyOperations(path_list_op_layer_middle)
result = result.ApplyOperations(path_list_op_layer_bottom)
# Notice how on merge it makes sure that each sublist does not have the values of the other sublists, just like a Python set()
print(result) # Returns: SdfPathListOp(Deleted Items: [/cube], Prepended Items: [/disc, /cone], Appended Items: [/sphere])
# Get the flattened result. This does not apply the deleteItems, only ApplyOperations does that. 
print(result.GetAddedOrExplicitItems()) # Returns: [Sdf.Path('/disc'), Sdf.Path('/cone'), Sdf.Path('/sphere')]

### Deleted and added items ###
path_list_op_layer_top = Sdf.PathListOp.Create(appendedItems=[Sdf.Path("/disc"), Sdf.Path("/cube")])
path_list_op_layer_middle = Sdf.PathListOp.Create(deletedItems = [Sdf.Path("/cube")])
path_list_op_layer_bottom = Sdf.PathListOp.Create(prependedItems = [Sdf.Path("/cube")], appendedItems = [Sdf.Path("/sphere")])

result = Sdf.PathListOp()
result = result.ApplyOperations(path_list_op_layer_top)
result = result.ApplyOperations(path_list_op_layer_middle)
result = result.ApplyOperations(path_list_op_layer_bottom)
print(result) # Returns: SdfPathListOp(Appended Items: [/sphere, /disc, /cube])
# Since it now was in the explicit list, it got removed.

### Explicit mode ###
# There is also an "explicit" mode. This clears all previous values on merge and marks the list as explicit.
# Once explicit and can't be un-explicited. An explicit list is like a reset, it 
# doesn't know anything about the previous values anymore. All lists that are merged
# after combine the result to be explicit.
path_list_op_layer_top = Sdf.PathListOp.Create(deletedItems = [Sdf.Path("/cube")])
path_list_op_layer_middle = Sdf.PathListOp.CreateExplicit([Sdf.Path("/disc")])
path_list_op_layer_bottom = Sdf.PathListOp.Create(prependedItems = [Sdf.Path("/cube")], appendedItems = [Sdf.Path("/sphere")])

result = Sdf.PathListOp()
result = result.ApplyOperations(path_list_op_layer_top)
result = result.ApplyOperations(path_list_op_layer_middle)
result = result.ApplyOperations(path_list_op_layer_bottom)
print(result, result.isExplicit) # Returns: SdfPathListOp(Explicit Items: [/disc]), True
# Notice how the deletedItems had no effect, as "/cube" is not in the explicit list.

path_list_op_layer_top = Sdf.PathListOp.Create(deletedItems = [Sdf.Path("/cube")])
path_list_op_layer_middle = Sdf.PathListOp.CreateExplicit([Sdf.Path("/disc"), Sdf.Path("/cube")])
path_list_op_layer_bottom = Sdf.PathListOp.Create(prependedItems = [Sdf.Path("/cube")], appendedItems = [Sdf.Path("/sphere")])

result = Sdf.PathListOp()
result = result.ApplyOperations(path_list_op_layer_top)
result = result.ApplyOperations(path_list_op_layer_middle)
result = result.ApplyOperations(path_list_op_layer_bottom)
print(result, result.isExplicit) # Returns: SdfPathListOp(Explicit Items: [/disc]), True
# Since it now was in the explicit list, it got removed.

When working with multiple layers, each layer can have list editable ops data in (composition-) metadata fields and relationship specs. It then gets merged, as mocked above. The result is a single flattened list, without duplicates.

Composition Arcs

For a detailed explanation how list editable ops work in conjunction with composition arcs, please check out our composition fundamentals section.

Relationships

As with list editable metadata, relationships also show us the combined results from multiple layers. Since it is not a composition arc list editable op, we also don't have the restriction of encapsulation. That means, calling GetTargets/GetForwardedTargets can combine appended items from multiple layers. Most DCCs go for the .SetTargets method though, as layering relationship data can be confusing and often not what we want as an artist using the tools.

Pro Tip | List Editable OPs in Collections

Since collections are made up of relationships, we can technically list edit them too. Most DCCs set the collection relationships as an explicit item list though, as layering paths can be confusing.

Metadata

The default metadata fields that ship with USD, are not of the list editable type. We can easily extend this via a metadata plugin though.

Here is a showcase from our composition example file that edits a custom string list op field which we registered via a custom meta plugin.

As you can see the result is dynamic, even across encapsulated arcs and it always returns an explicit list op with the combined results.

This can be used to combine metadata non destructively from multiple layers.