Skip to content

Dynamic Objects

This Dynamic Objects documentation provides instructions on how to use the dynamic object tracking functionalities of the Cognitive3D SDK with various web game engines.

Dynamic Object Tracking allows you to record the position, rotation, and scale of any object in your scene over time. This is essential for analyzing player interactions, object manipulation, and custom gameplay events.

Overview

The general workflow for tracking a dynamic object involves three main steps:

  1. Export and Upload the Object's 3D Model

    Before the Cognitive3D dashboard can visualize data on your object, it needs to know what the object looks like. This initial, one-time step involves exporting the object's 3D model and texture files to Cognitive3D.

    Our SDK provides functions (like exportObject in the Three.js adapter) to generate the required files in the correct format (.gltf, .bin, and .png thumbnail). Please refer to c3d-upload-tools for uploading your object models to Cognitive3D.

  2. Tag the Object in Your Scene

    Once the 3D model is uploaded, you need to "tag" the corresponding object within your application's scene. This is done by adding specific metadata to the object, which allows our SDK to identify it as a trackable dynamic object. This step doesn't affect the object's appearance or behavior; it simply attaches important information that the SDK looks for.

  3. Registering the Object at runtime Tagging the object tells the SDK that it should be tracked, but registering tells the SDK when tracking should begin. This final step involves calling a specific SDK function registerObjectCustomId() when the object is initialized in your scene (e.g., when it spawns or at the start of a level). This function officially "registers" the object with our backend for the current session, linking the live, in-engine object to the 3D model you uploaded in Step 1.

Once registered, the engine-specific adapter will automatically handle the collection and transmission of transform data.

Engine-Specific Implementations

Three.js Dynamic Objects

Step 1: Mark Your Objects as Dynamic

For any THREE.Object3D you want to track, you must set two properties in its userData:

  • isDynamic: A boolean set to true. This flags the object for automatic registration.

  • c3dId: A unique string identifier for the object. This can be any unique string, but a UUID is recommended.

// Example of creating a trackable object
const myObject = new THREE.Mesh(geometry, material);
myObject.name = "MyTrackedObject";
myObject.userData.isDynamic = true;
myObject.userData.c3dId = "your-unique-object-id-12345"; // Must match the id on the c3d dashboard

Step 2: Register the Object with Cognitive3D

Next, you need to call c3d.dynamicObject.registerObjectCustomId() to officially register it with the SDK. This tells our backend what this object is.

c3d.dynamicObject.registerObjectCustomId(
    myObject.name,             // Name e.g. "Cube"
    "mesh_name_for_backend",   // A mesh identifier (e.g. "cube")
    myObject.userData.c3dId,   // Unique ID matching the id of the uploaded object 
    myObject.position.toArray(), // Initial world position
    myObject.quaternion.toArray() // Initial world rotation
);

2.1 Working with Imported Models

Access the root of the loaded model (usually gltf.scene) and treat it as you would any other THREE.Object3D.

import { GLTFLoader } from 'three/examples/jsm/loaders/GLTFLoader.js';

const loader = new GLTFLoader();
loader.load('path/to/your/model.glb', (gltf) => {
    const carModel = gltf.scene;

    // Step 1: Tag the root of the model
    carModel.name = "PlayerCar";
    carModel.userData.isDynamic = true;
    carModel.userData.c3dId = "car-unique-id-001";

    // Step 2: Register the model with the SDK
    c3d.dynamicObject.registerObjectCustomId(
        carModel.name,
        "car_mesh",
        carModel.userData.c3dId,
        carModel.position.toArray(),
        carModel.quaternion.toArray()
    );

    // Add the prepared model to the interactable group
    interactableGroup.add(carModel);
});

Step 3: Add Objects to the Interactable Group

Ensure that all your dynamic and static objects are added to a single THREE.Group. This group is what the SDK will use for tracking and raycasting.

const interactableGroup = new THREE.Group();
interactableGroup.add(myObject);
interactableGroup.add(anotherTrackedObject);
interactableGroup.add(staticSceneryObject);

scene.add(interactableGroup);

Step 4: Start Tracking with a Single Call

Use the c3dTHREEAdapter.startTracking() method. This call initializes gaze raycasting against the objects you've provided, finds all objects tagged as dynamic, and seamlessly hooks into your application's render loop to ensure data is captured on every frame.

startTracking(renderer, camera, interactableGroup, userRenderFn)

Parameters:

  1. renderer (THREE.WebGLRenderer): Required. This is your main Three.js renderer instance. The adapter uses it to hook into the animation loop (setAnimationLoop), which guarantees that transform updates are synchronized with the frame rendering.

  2. camera (THREE.Camera): Required. This is the primary camera the user sees through. The adapter uses this camera's position and orientation to perform gaze raycasting into the scene.

  3. interactableGroup (THREE.Group): Required. This is the parent THREE.Group that contains all objects you want to be available for gaze detection or dynamic tracking. The SDK will automatically scan the children of this group.

  4. userRenderFn (function): Optional. This is your application's own render function. By passing your render logic to startTracking, you allow the SDK to wrap it. The SDK will ensure your render function is called on every frame before it performs its own per-frame tasks, like updateTrackedObjectTransforms.

Example

// In your init function, after setting up the scene and objects:

function render() {
    // Your standard render logic...
    // This will be called automatically on every frame.
    renderer.render(scene, camera);
}
c3dTHREEAdapter.startTracking(renderer, camera, interactableGroup, render);

The SDK will now automatically track the position, rotation, and scale of "MyTrackedObject" and any other dynamic objects you've registered, sending updates whenever they change.

Object Engagements

Object Engagements allow you to record precise information about how a user manipulates objects in the scene. An Engagement could represent a state such as grabbing an object, proximity to an object, pointing to an object, etc. Generally we recommend calling the Engagement APIs when a user manipulates the object with their hand(s). Technically, this is an alternative workflow to record a Custom Event and associate it with a Dynamic Object.

To begin or end an Engagement, call the beginEngagement OR endEngagement function on the dynamicObject class.

beginEngagement(objectId, engagementTypeName, parentObjectId);
  • objectId: The Id of the object that is being engaged with.
  • engagementTypeName: The type of engagement that is happening. This is customizable. For example: "Grab"
  • parentObjectId: The objectId of the parent which is engaging with the object. This is generally the controller that is grabbing the object.

The example below shows how you could implement an Engagement onto a grab event:

function myStartGrab() {

    // some grabbing code for your app...

    c3d.dynamicObject.beginEngagement('1', 'grab', 'right_hand');
}

To end an engagement:

function myEndGrab() {

    // some end grabbing code for your app...

    c3d.dynamicObject.endEngagement('1', 'grab', 'right_hand');
}

intercom If you have a question or any feedback about our documentation please use the Intercom button (purple circle) in the lower right corner of any web page or join our Discord.