Synthetic-to-Real Perception for Agricultural Robotics

Introduction to the series Synthetic-to-Real Perception on NVIDIA Jetson
by Konsulko Group Software Engineer Andjela Danilovic Nedic

A robot working in a sugar beet field has to make a decision for every plant in front of its camera: leave it in place, or treat it as a weed. Before any tool moves, that decision depends on perception, which means locating each plant in the image and classifying it correctly. Building that capability involves two separate engineering problems. The detector must be trained on images that represent the field, and the trained model must run on embedded hardware that can be deployed, updated, and maintained on a working machine.

At Konsulko Group, we built a crop and weed detection workflow that addresses both problems end to end. We generated synthetic training images in NVIDIA Isaac Sim, trained and evaluated RT-DETR detectors with NVIDIA TAO Toolkit, and deployed the selected model on an NVIDIA Jetson Orin Nano 8 GB running Konsulko Orca OS™, our Yocto Project-based Linux distribution for Jetson platforms. Konsulko has experience supporting customers in the agricultural sector, including work on embedded operating systems. This project applied that platform experience to a complete perception application, from training data to inference on the device.

The Training Data Challenge

Selective weed control depends on distinguishing crops from weeds accurately enough to guide treatment. A missed weed may remain untreated, while a crop mistaken for a weed may be damaged. Reliable detection therefore requires training data that captures the visual conditions a system will encounter in the field.

Crop and weed plants can look similar, overlap, or occupy only a small area of an image. Differences in plant size, soil texture, plant residue, shadows, and lighting add further variation. A useful dataset needs to represent these conditions, including small and partially visible plants that are difficult to distinguish from their surroundings.

Collecting representative images can require repeated field visits within limited seasonal windows. Preparing and checking annotations adds further effort, as individual plants must be located and assigned the correct class. These demands make training data a significant part of developing an agricultural perception system.

This led to the central question of this project: could synthetic images reduce the amount of labeled real data needed to train a crop and weed detector?

From Simulation to Real Images

Synthetic data moves much of this effort from the field to the workstation. We used NVIDIA Isaac Sim to build a configurable agricultural scene containing sugar beet plants, weeds, soil, and unlabeled background objects such as straw and stones. A generator script varied plant selection, placement, and size, along with soil appearance, lighting, and camera pose. It used Omniverse Replicator to export each rendered image together with its bounding-box annotations. Because the simulator knows where every plant is, labels are produced as part of rendering rather than drawn afterward.

Figure 1. The synthetic-to-real perception workflow, from scene generation in NVIDIA Isaac Sim to inference on NVIDIA Jetson 

Rendered images are not photographs. Differences in plant geometry, textures, lighting, and object distribution create a domain gap between synthetic and real data, and a model can learn cues that exist only in the rendered images. The value of the synthetic data therefore had to be measured on real images, not judged by how realistic the renders looked.

We trained RT-DETR, a transformer-based object detector, using NVIDIA TAO Toolkit under three approaches: synthetic images only, real images only, and synthetic pretraining followed by fine-tuning on real images. The real images came from a sugar beet subset of the public CropAndWeed dataset, with labels mapped to the same two classes. To see how the benefit of synthetic pretraining changed with the amount of real data, the real-image experiments used nested training subsets of 50, 100, 200, and 512 images. Every model was evaluated on the same held-out real test set, drawn from recording sessions that were not used for training or validation.

Deployment on Jetson with Orca OS

NVIDIA’s ecosystem provided tools for each stage of the workflow: Isaac Sim and Replicator for synthetic data generation, TAO Toolkit for detector training, and TensorRT and DeepStream for deployment on Jetson. This gave the project a path from simulated training images to GPU-accelerated inference on an embedded device.

Orca OS provided the embedded Linux foundation for that deployment. Built on the Yocto Project, with NVIDIA Jetson platform support through meta-tegra, it allowed us to combine a configurable operating system with NVIDIA’s standard application containers. With Docker and the NVIDIA Container Toolkit providing GPU access, we could use NVIDIA’s published DeepStream container and add our trained model, inference configuration, and custom parser without integrating the DeepStream framework into the Yocto image. This kept the perception frameworks and their dependencies within the application containers while the underlying system remained managed through the Yocto build.

We converted the selected model into a TensorRT FP16 engine on the Jetson itself and integrated it into two applications: a DeepStream video pipeline whose annotated output can be viewed in a web browser, and a ROS 2 image-processing pipeline that publishes annotated detection output.

The DeepStream and ROS 2 applications ran in separate containers on the same Orca OS system, sharing the underlying Jetson platform and its GPU resources.

Figure 2. Deployment architecture on NVIDIA Jetson Orin Nano

This separation establishes a clear boundary between the underlying operating system and the perception applications. Platform components can be managed through the Yocto build, while application dependencies remain within their respective containers. This provides flexibility to maintain and update the operating system and applications separately, provided that compatibility between the host and container environments is preserved.

Results and Findings

In our experiments, synthetic pretraining provided the greatest benefit when labeled real training data was limited. With 50 real training images, the synthetic-to-real approach achieved 32.42 AP on the held-out real test set, compared with 8.15 AP for the real-only approach. The advantage narrowed as more real images became available.

On the NVIDIA Jetson Orin Nano running Orca OS, the selected model achieved 39.77 FPS in a standalone TensorRT FP16 benchmark, while the ROS 2 application processed an average of 11.20 FPS. We also demonstrated the DeepStream pipeline with browser-based viewing of annotated video. These measurements describe the tested implementation; performance under live field conditions and integration with a weeding tool remain to be assessed.

Security and Product Maintenance

Agricultural perception systems carry models and algorithms that represent substantial investment in data, training, and engineering. Deploying them on machines introduces a need to protect that intellectual property against unauthorized extraction and to protect the software against tampering. Security also needs to support continued operation as vulnerabilities are discovered and software changes over time.

Orca OS provides capabilities that support these requirements. Secure boot and filesystem integrity verification help protect system software against unauthorized modification, while encrypted storage can protect model files and application data at rest. Over-the-air updates with rollback support software maintenance and recovery from unsuccessful updates. These capabilities require configuration for the target product and were not utilized as part of this perception project.

For products within its scope, the EU Cyber Resilience Act introduces cybersecurity requirements covering product development and maintenance, including vulnerability handling. A software bill of materials supports component tracking and vulnerability assessment, while an update mechanism enables fixes to reach deployed devices. These capabilities support a manufacturer’s compliance work; meeting the requirements also depends on the complete product and the processes used to develop and support it.

In This Series

The three technical articles follow the workflow in order, and each answers one question raised here.

Part 1 – Generating Synthetic Crop and Weed Images with NVIDIA Isaac Sim explains how to design a synthetic scene that a detector can learn from: plant assets for different growth stages, crop rows and weed placement, camera geometry and plant scale, domain randomization, and automated checks on the generated data.

Part 2 – Training RT-DETR with Synthetic and Real Data Using NVIDIA TAO examines when synthetic pretraining helps: dataset preparation, the three training approaches, model selection, and the full evaluation on real images.

Part 3 – Deploying an RT-DETR Perception Model on NVIDIA Jetson with Orca OS covers what it takes to run the model on the device: TensorRT conversion, the DeepStream and ROS 2 integrations, browser viewing, and the performance measurements on Orca OS.

Konsulko provides end-to-end NVIDIA Jetson engineering services: custom carrier board design, board bring-up, Orca OS integration, application deployment, and secure over-the-air updates. Orca OS is delivered as part of an engineering engagement and adapted to each customer’s hardware and product requirements. Contact Konsulko to discuss bringing a perception application from training to a maintainable embedded product.

Konsulko Group

Founded in 2012 by embedded Linux industry veterans, Konsulko Group is a unique services company. Unique in the experience provided by 20+ years of commercial development leveraging open source software. Unique in the decades of open source community work as active contributors and sometimes maintainers. Unique in the insight and precision we bring to every customer product.

See Full Bio