Quickstart
Set up the Wandelbots Isaac Sim extension with Wandelbots NOVA and NVIDIA Isaac Sim. Start simulating your first robotics use case or explore a demo to test your setup.
Quickstart
Once you have installed the simulation environment and the Wandelbots NOVA extension for NVIDIA Isaac Sim, you are ready to build your first simulation. Explore the capabilities of Wandelbots NOVA and identify potential use cases for your projects with this quickstart guide.
Prerequisites
You’ve completed the installation of the Wandelbots NOVA extension for NVIDIA Isaac Sim and your simulation environment is connected to Wandelbots NOVA.
Create a scene
To create a scene you’ll use assets, robots and tools from the stacking demo package.
- Download the Stacking demo package from the NOVA Portal.
- Open the predefined scene
main.usdfrom the package in NVIDIA Isaac Sim.
Add robot to scene
- In the stage tree, navigate to
MIR. - Right-click on
Mounting. - Select
Create→Wandelbots NOVA→Single Robot Model. - In the configuration window, select
KUKAas manufacturer andKUKA_KR10_R900_2as model. - Save the robot to a desired file location and confirm.
- To reorganize the hierarchy, in the stage tree, move the imported robot to the
workspace_kukaXform to ensure a proper environment structure.
Add tool
To pick up objects, the robot requires a gripper:
Connect gripper to robot flange
- In the stage tree, expand the
Kuka_KR10_R900_2hierarchy, navigate toLink_6and expand it too. - To position the tool, drag the gripper
schunk_coact_gripperfrom the content browser onto thetcp_flangeXform as seen in the video. The gripper aligns with the robot’s mounting flange. - To relocate the gripper into the correct layer of the robot assembly, drag the
schunk_coact_gripperfrom thetcp_flangeinto theKUKA_KR10_R900_2Xform. - To create the physical connection, select
Link_6first and then, withCtrlkey pressed, select theGripper_Bodyprim of the tool, too. - Select
Create→Physics→Joint→Fixed Joint.
Configure tool’s action graph
- Right-click on the action graph of
schunk_coact_gripperin the stage tree and selectOpen Graph. - In the action graph editor, search for the
On IO Changenode and drag it into the workspace. - Connect the
Changeoutput toInput Executionand theBooleanoutput to theConditionpin of the following logic node. - Select the
On IO Changenode and set the following properties in thePropertiespanel:\Input Robot: Select the robot from the scene.IO: Set toOUT#1.
Learn more on how to set up a parallel gripper or surface gripper as well as how to use OmniGraph nodes in the respective articles.
Add workpieces
Now that the gripper is ready to grab objects, let’s add some cubes to the scene that the robot can interact with.
- From the asset folder drag and drop the
workpiece.usdinto your scene and into theworkspace_kukaXform. Once it is imported you can place it anywhere on the table. - Right click the workpiece prim and duplicate it twice.
Create ghost objects for pick and place poses
Ghost objects act as target poses for the robot’s TCP; tool center point. They help you visualize and plan movements before execution.
-
Right-click on the gripper in the stage tree.
-
Select
Create→Wandelbots NOVA→Ghost Object.
Aposesprim is created under gripper. -
Use the transform handles in the viewport to drag the ghost object to the precise location where the robot should pick up the cube.
-
Expand the
posesprim and select the generated mesh. -
Rename the mesh to
PickPose_01. -
Position
PickPose_01where the gripper should grab the first cube.
A ghost overlay of the robot is displayed.
If the overlay is visible, the position is reachable. If it disappears, the position is out of reach. -
Duplicate
PickPose_01and position the newPickPose_02on the second cube. -
Repeat step 7 for
PickPose_03on the third cube. -
Duplicate one of the pick poses and rename the new pose to
PlacePose_01. -
Position the
PlacePose_01where the stack should start. -
Duplicate
PlacePose_01to createPlacePose_02. -
To stack the place poses, move
PlacePose_02vertically by the height of one cube. -
Repeat step 11 and 12 for
PlacePose_03, moving it up by another cube’s height to complete the stacking sequence.
Learn more about ghost objects and their use in ghost teaching.
Write robot program
To start stacking cubes, you need to write a robot program that defines the pick and place sequence using the ghost object poses as targets for the robot’s TCP.
Use the robot program stacking.py (Stacking demo package > scripts folder) from the NOVA Portal:
Show robot program
# package imports
import nova
import wandelbots_isaacsim_api as isaac_sim_api
import wandelbots_isaacsim_api.trajectory as trajectory_utils
from nova import run_program
from nova import api
from nova.actions import cartesian_ptp, io_write, linear, wait
from nova.actions.io import WriteAction
from nova.actions.mock import WaitAction
from nova.actions.motions import CartesianPTP, Linear
from nova.cell import virtual_controller
from nova.cell.cell import Cell
from nova.cell.controller import Controller
from nova.cell.motion_group import MotionGroup
from nova.types.motion_settings import MotionSettings
from nova.types.pose import Pose
from wandelbots_isaacsim_api.api.teaching_api import TeachingApi
from wandelbots_isaacsim_api.models.ghost_object import GhostObject
import logging
logger = logging.getLogger(__name__)
# parameter definitions
isaacsim_ip_address = "<add your_isaac_sim_ip_address_here>"
omniservice_host = f"http://{isaacsim_ip_address}:8011/omniservice/api/v2"
controller_name = "kuka"
robot_prim_path = "/World/cell/workspace_kuka/KUKA_KR10_R900_2"
manufacturer = api.models.Manufacturer.KUKA
robot_model = "kuka-kr10_r900_2"
gripper_signal = "OUT#1"
motion_group_num = 0
# helper function to get poses from ghost objects in isaac sim
async def get_poses_from_ghost_objects(
isaac_sim_api_url: str, robot_prim_path: str
) -> dict[str, list[Pose]]:
"""
Helper function to get pick and place poses from ghost objects in the simulation model.
Returns a dictionary mapping each unique ghost object name to a list of Pose objects.
"""
async with isaac_sim_api.ApiClient(
configuration=isaac_sim_api.Configuration(host=isaac_sim_api_url)
) as isaac_sim_api_client:
teaching_api: TeachingApi = isaac_sim_api.TeachingApi(
api_client=isaac_sim_api_client
)
ghost_objects: list[GhostObject] | None = None
ghost_objects = await teaching_api.list_ghost_objects(
relative_to_prim=robot_prim_path
)
# Create a dictionary mapping each unique ghost object name to a list of Pose objects
poses_dict: dict[str, list[Pose]] = {}
for ghost_obj in ghost_objects:
name = ghost_obj.name
pose_obj = Pose(tuple(ghost_obj.pose.pose))
if name not in poses_dict:
poses_dict[name] = []
poses_dict[name].append(pose_obj)
return poses_dict
# configure the robot program
@nova.program(
id="stacking_demo",
name="Stacking Demo",
viewer=trajectory_utils.TrajectoryViewer(
omniverse_host=omniservice_host,
motion_group_prim_paths={
f"{motion_group_num}@{controller_name}": robot_prim_path,
},
),
preconditions=nova.ProgramPreconditions(
controllers=[
virtual_controller(
name=controller_name,
manufacturer=manufacturer,
type=robot_model,
)
],
cleanup_controllers=False,
)
)
async def start(ctx: nova.ProgramContext) -> None:
"""Main robot control function."""
# get cell, controller, and motion group instances from the Nova instance
cell: Cell = ctx.cell
controller: Controller = await cell.controller(controller_name)
motion_group: MotionGroup = controller[motion_group_num]
# define motion speed and tcp
fast: MotionSettings = MotionSettings(tcp_velocity_limit=250)
tcp = "schunk_coact_gripper"
# get ghost objects from isaac sim
robot_poses: dict[str, list[Pose]] = await get_poses_from_ghost_objects(
isaac_sim_api_url=omniservice_host,
robot_prim_path=robot_prim_path,
)
# iterate over ghost objects and run pick and place motion commands
pick_poses = [pose for key, poses in robot_poses.items() if "PickPose" in key for pose in poses]
place_poses = [pose for key, poses in robot_poses.items() if "PlacePose" in key for pose in poses]
for (pick_pose, place_pose) in zip(pick_poses, place_poses):
# define robot motion sequence
actions: list[CartesianPTP | Linear | WriteAction | WaitAction] = [
cartesian_ptp(
target=pick_pose @ Pose(0, 0, -200, 0, 0, 0),
settings=fast,
),
linear(target=pick_pose, settings=fast),
io_write(key=gripper_signal, value=True),
wait(wait_for_in_seconds=2),
linear(
target=pick_pose @ Pose(0, 0, -200, 0, 0, 0),
settings=fast,
),
cartesian_ptp(
target=place_pose @ Pose(0, 0, -200, 0, 0, 0),
settings=fast,
),
linear(target=place_pose, settings=fast),
io_write(key=gripper_signal, value=False),
wait(wait_for_in_seconds=2),
linear(
target=place_pose @ Pose(0, 0, -200, 0, 0, 0),
settings=fast,
),
]
await motion_group.plan_and_execute(actions, tcp)
# execute program
if __name__ == "__main__":
run_program(program=start)You can execute the program in two setups: either directly on the NOVA Cloud instance using the Visual Studio Code app from the App Store or via a local Visual Studio Code setup on your machine.
- Open Visual Studio Code and create a new folder for your project.
- Install
uv. - Install the NOVA Python SDK and the Wandelbots Isaac Sim Extension API client:
uv init uv add wandelbots-nova wandelbots-isaacsim-api - Create a new file named
main.pywithin your project folder and paste the example code or your own code. - In
main.py, go to line 25 and enter the IP address of the host machine where Isaac Sim is currently running on. - Create a new file named
.envinmy_projectand add the following line to the file:NOVA_API = "http://<Your_NOVA_Instance_URL>" # replace `Your_NOVA_Instance_URL` with the actual URL of your running NOVA cloud instance. NOVA_ACCESS_TOKEN = "My_NOVA_Access_Token" # replace `My_NOVA_Access_Token` with the actual access token of your running NOVA cloud instance.
Learn more about how to set up a local Visual Studio Code environment and connect it to a NOVA Cloud instance here.
- Open the Wandelbots NOVA interface and install Visual Studio Code via the App Store.
- Open Visual Studio Code from the home screen.
- Open the folder or project named
your_nova_app. - Open the
start_here.pyfile and paste the example code or your own code. - Open the Visual Studio Code terminal and run the following commands to add the necessary libraries using
uv:uv add wandelbots-isaacsim-api
Start simulation
- Create a virtual controller called
kukain your NOVA Cloud instance. Manufacturer: KUKA, model:kuka-kr10_r900_2. - In NVIDIA Isaac Sim, navigate to
Wandelbots NOVA→Connected Instancesand clickRefresh.
The virtual controller is displayed in the list.
The virtual controller’s motion group is automatically matched with theArticulationin the scene. - To update the TCP on Wandelbots NOVA, in the stage tree, navigate to
KUKA_KR10_R900_2→schunk_coact_gripper→Gripper_Body→tcp_gripper. - Select
Create→Wandelbots NOVA→TCP in NOVA.
The TCP is added onto the virtual controller. - Start the NVIDIA Isaac Sim simulation with
Run. - Start the robot program in Visual Studio Code and watch the simulation execute the stacking sequence in real-time.
Congrats! You’ve successfully set up the stacking demo and simulated the stacking scenario. 🎉
To step it up a notch, try modifying the tool or the TCP position. Let us know how it went!