NVIDIA Isaac Sim, Isaac Lab and Cosmos-style pipelines can scale lighting, layouts and object variation in simulation, but they do not give you a damp sponge or a sticking drawer – those still have to be collected in a real kitchen.
That is not an argument against the stack. NVIDIA has spent the last two years making the pipeline a default for anyone training a mobile manipulator or a humanoid policy: Isaac Sim as the physically based scene, Isaac Lab as the parallel learner, Cosmos Transfer as the photorealism pass that turns RGB, depth, segmentation and edges into a video that looks like a warehouse. MobilityGen will invent trajectories. GR00T-Dreams will invent motion from an image and a prompt. GR00T-Mimic will multiply a handful of human demonstrations into a pile of synthetic ones. Isaac Lab-Arena will score the policy in simulation. None of that is vapour. All of it is variation. Variation is not a wet sponge.
The market pitch this autumn is that you can close the sim-to-real gap with more control signals and a better world model. The data pitch, if you are honest, is that the generator has never wrung out a cloth. Photorealism is a lighting problem. Contact with a soaked sponge is a physics problem that the renderer is still approximating, and that your kitchen will still fail on a Tuesday.
What the stack actually gives you
Isaac Sim is a scene. CAD, URDF, a captured room via NuRec, materials, PhysX, RTX cameras. You assemble a kitchen that is cheaper to relight than to visit. You can randomise mug colour, counter clutter, camera height, time of day. You can run thousands of those kitchens on a GPU while a real capture team is still booking a house. That is the honest use. Fill the pattern you already understand. Extra views of a class you have in real images. Extra approaches to a handle you have already grasped. A stand-in while legal reviews the real set.
Isaac Lab takes that scene and trains. Reinforcement learning, imitation, diffusion policy, thousands of parallel environments. GR00T-class models get fine-tuned there before anyone risks a real arm. Fine. The learner is only as honest as the objects it was handed. If every sponge in the USD is a rigid-ish mesh with a wet-looking texture, the policy has learned a wet-looking texture. It has not learned that the sponge compresses, slips, leaves a film on the stainless steel, and then sticks to the next plate.
Cosmos Transfer is the photorealism pass that makes the slide look finished. CosmosWriter in Isaac Sim dumps synchronised RGB, depth, segmentation, shaded segmentation and edges. Transfer’s Multi-ControlNet turns those into a clip that looks like a warehouse, or a kitchen, or whatever prompt you typed, while trying to keep structure and camera motion. You can weight the branches so the output hugs the sim or invents freely. Hugging the sim preserves a layout that was never wet. Inventing freely invents a kitchen the generator has already seen, which is not your sticking drawer. Synthetic data is a tool. It is not a human baseline. Photoreal is still synthetic.
GR00T-Dreams and GR00T-Mimic are the other temptation. Dreams: one image, a language instruction, a pile of synthetic trajectories, no new teleoperation. Mimic: a handful of human demonstrations, then scale. NVIDIA’s own pages still say real-world human demonstrations are typically used as inputs for manipulation. The blueprints exist because collecting those demonstrations is expensive. They do not exist because the demonstrations became optional. A handful of dry takes, multiplied, is a handful of dry takes with extra lighting. The long tail that made last Tuesday fail is the first thing the multiplier will not invent.
MobilityGen is honest about what it is: occupancy maps, recorded or random trajectories, replay and render, then Transfer for visual diversity. Useful for a quadruped or an AMR that has to go down a warehouse aisle it has not visited. The aisle still does not contain a tea towel that has been sitting in a puddle, or a drawer runner with grit in it. Visual diversity is not contact diversity.
The kitchen physics still refuses
A familiar Friday in a robotics lab. The Isaac Lab-Arena score is up. Transfer made the synthetic kitchens look like the ones in the product film. The policy picks a dry mug in sim, places it in a dry sink, does it again under five lighting conditions. The slide says sim-to-real. Monday is a real counter. The sponge is damp from the night before. The mug has a film of washing-up liquid. The drawer sticks two centimetres in, the way that drawer has stuck since 2019. The cloth drapes over the tap and then slides. Steam hits the wrist camera. A tea towel occludes the handle. Nothing in the USD had that combination, because nobody who built the USD had to live with that drawer.
Domain randomisation is the usual answer, and it is a good answer for the things it can randomise. Pose, lighting, texture, distractor objects, camera extrinsics. It is a weak answer for the things that are the job in a home. Cloth. Liquids. Grease. Deformable food. A handle that is wet. A bin bag that sags. A plate that is heavier than it looks because it is still full. Those are not “more variation.” They are different objects. If you only ever randomise appearance, you will ship a policy that is robust to lighting and brittle to contact.
The papers this year already said the quiet part. EgoScale, DreamDojo, the humanoid pretraining recipes: large-scale first-person human video, then a lighter robot stage. They did not say “we closed kitchens in Isaac Sim.” They said the viewpoint a wearable or a chest camera actually has is worth industrialising, because third-person demos and simulators do not cover it. What egocentric data is in physical AI is that viewpoint, not a renderer’s idea of a first-person camera. A simulated chest cam is still a simulated kitchen. Occlusion in a USD is a mesh. Occlusion in a real kitchen is a sleeve, a steam bloom, a second person reaching in.
Sim-to-real work is real. Teams that mix sim with a held-out real set outperform teams that only film, on the axes sim is good at: coverage, dangerous failures, lighting, some locomotion. That is not a licence to drop the wet take. NVIDIA’s own robot-learning write-ups still talk about combining internet-scale data, synthetic data and real robot data. The order of those three in the sentence is a product decision. If synthetic is the whole mix because generation is cheap this quarter, you have the collapse problem with extra PhysX. Rare contact is the long tail. Generators are good at the average of what they have already seen.
Eval is where the cheat is most tempting. Isaac Lab-Arena is a serious object for scoring a policy in simulation. It is still a rehearsal in a room that will not exist at launch. Synthetic eval of a kitchen policy is the student writing the exam, then sitting it, with nicer lighting. Frozen human eval that looks like production – this drawer, this sponge, this camera mount, people held out – is the unfashionable object that still answers “did this release help?” A Transfer clip of a mug is not that object, however photoreal.
What you still have to collect on purpose
Start from the skill, not from the renderer. “Wring a damp sponge over a stainless sink, chest camera, indoor daylight, start to finish including the slip, faces out.” That sentence decides what sim can do for you and what it cannot. Sim can give you extra approaches to a dry sponge you already collected. It cannot invent the dampness. If you cannot write the sentence, you are shopping for hours of USD, not collecting a skill.
The fields that change the files are ordinary. Device and mount: chest, head, glasses, wrist, third-person. A simulated camera at the “right height” is not a GoPro that slipped. Environment: this kitchen, not a warehouse USD with a fridge asset. Locale is not a language name and not a texture pack. A British kitchen with a sticking 1980s drawer is a different object from a US island with soft-close runners. Who may appear: faces in or out, whether a second person can walk through. Rights for the real product: training a policy is not a marketing still, and a simulated face is still a face problem if the seed was an uncleared photo.
Write reject codes for the real capture while everyone is still calm. Task not completed. Camera on the ceiling. Dry sponge when you asked for damp. Drawer that never sticks when you asked for the failure. Beauty filters. Digital zoom that destroys the hands. Non-consented faces. Screens. Automation can flag black frames. A person still has to keep the mess that is the job. If gold items do not exist before you generate a million Transfer clips, the first month of generation is a very expensive prior you will then have to unlearn.
Split eval by person and by house, not by random clip. A teleoperator who is good in the lab will dominate the real hours you do collect. If their hands sit in train and test, the policy has seen that household. The Isaac Lab graph will look like transfer. The next kitchen will not. Hold those people out. Rights to keep the eval set and not train on it are a separate line from “we recorded a demo for Mimic.”
Label the mix. Human: this skill, this mount, this kitchen, rights-cleared for training and for keeping the eval set. Robot: this embodiment, this camera, these failures. Synthetic: this generator, this seed, this control-net weighting, not used as the frozen eval, lineage on the card. If you cannot write the third without the first two, you are not ready to generate. You are hoping Cosmos will invent a behaviour you have not collected.
Teleoperation is not the kitchen either, even though it is real. A person driving a robot is useful. It is a different object: robot cameras, robot dynamics, a different rights conversation. Third-person demonstration is useful for checking whether the task finished. It does not show occlusion the way a chest camera does. Found YouTube is first-person-shaped and almost never labelled for the sponge. Sim is cheap variation. Mix sources if you must. Label them. When the mug fails, you need to know which object to collect more of.
Do not let the renderer become the seed
A one-off real collection at the start of the project, then a year of Mimic and Transfer, is not a seed you held onto. It is a vintage. Models change. Products change. The kitchen in the seed is last year’s kitchen. 2026 practice on synthetic-only runs is crude and unfashionable: keep a real slice in every cycle. Robotics does not get a pass because the generator is now called Cosmos 3. A seed you never re-open is how the policy becomes more average, and average is a dry mug in a dry sink.
If a catalogue human set already matches the skill and the mount, look at it with the actual trainer before you stand up a sim factory. Generate around it. Do not generate instead of it. Custom when the object set, the kitchen, or the dampness is yours. Starting in Isaac Sim because the licence is already on the cluster is how briefs balloon and recapture becomes a line item after the first real house visit.
The models can look like a kitchen. Your files still cannot feel like one, unless you recorded the contact on purpose and kept it out of the fine-tune. Simulation scales the variation you already understand. It does not wet the sponge, grit the runner, or steam the lens. Collect those, labelled, with people held out, or stop claiming the policy works in a home. A Transfer video of a drawer is a lighting story. A sticking drawer is the product.
People4.ai collects consented human behaviour for robot training and evaluation. Send the wet skill, not a USD. hello@people4.ai · Contact
FAQ
Can we fine-tune on Isaac Sim and Cosmos clips and only collect a little real data at the end?
You can try. You will be teaching dry variation and then serving contact. Keep a real slice in every cycle, rights-cleared, labelled separately, and use it as the frozen eval. A handful of end-stage takes will not unlearn a year of dry priors.
Is Isaac Lab-Arena enough as a release gate for a home skill?
It is a better shared exam than a demo hall. It is still simulation. Freeze a private set that looks like your kitchens, with real contact, the actual mount, and people who are not in training. Score sim for coverage. Score the house for the product.
Does GR00T-Mimic mean we no longer need human demonstrations?
No. Mimic multiplies a handful of human takes. Dreams still wants an image seed. If the seed is dry, the pile is dry. Collect the contact you claim to handle, then generate around it.
Related reading
Synthetic data is a tool. It is not a human baseline. · What is egocentric data in physical AI?