Agricultural Robots Explained: Weeding, Harvesting, and Monitoring

Agricultural robots replace repetitive field labor with sensing, planning, and actuation loops that run at 10-500Hz. A weeding robot classifies plants in under 50ms, a harvester solves inverse kinematics for a 6-DOF arm in real time, and a monitoring drone fuses GNSS, IMU, and multispectral data to build crop health maps. This guide covers the hardware stack, the kinematics and control math, a ROS2 weeding pipeline, and tuning workflows you can reproduce on a prototype platform.

Quick Takeaways

• Weeding robots pair an RGB-D camera with a CNN classifier and a delta-arm or laser actuator, reaching 95%+ weed-detection accuracy at 3-6 km/h.
• Harvesting robots are the hardest class: they need a 6-DOF manipulator, a soft end effector, and visual servoing with cycle times of 4-10 seconds per fruit.
• Monitoring platforms (UAVs and UGVs) rely on RTK-GNSS (2cm accuracy), IMU fusion, and multispectral NDVI imaging.
• ROS2 with Nav2 and robot_localization is the dominant open middleware for field navigation.

System Architecture of a Field Robot

Every agricultural robot runs the same four-layer stack, regardless of task.

1. Perception: cameras, LiDAR, IMU, RTK-GNSS, wheel encoders.
2. State estimation: an Extended Kalman Filter (EKF) fusing odometry, IMU, and GNSS.
3. Planning and control: coverage planning, row following, manipulator trajectories.
4. Actuation: BLDC wheel hub motors, servo joints, spray valves, or laser diodes.

Typical Hardware Layout

• Compute: NVIDIA Jetson Orin (up to 275 TOPS) for vision inference, plus a microcontroller (STM32/Teensy) for 1kHz motor loops.
• Power: 48V LiFePO4 pack, with a DC-DC converter stepping down to 24V (actuators) and 12V (sensors).
• Comms: CAN bus at 500 kbit/s for motor drivers, Ethernet for LiDAR and cameras, micro-ROS over serial for the MCU bridge.

Sensor Selection for Field Conditions

Dust, direct sunlight, and vibration break many lab-grade sensors. Choose accordingly.

For row crops, a 2D LiDAR mounted 0.3-0.5m above ground often beats a 3D unit for row following because it is cheaper and gives a clean cross-section of the plant rows.

Weeding Robots

How Robotic Weeding Works

1. The camera captures a frame at 30fps.
2. A segmentation network (e.g., a lightweight U-Net or YOLOv8-seg) labels each pixel as crop, weed, or soil.
3. Pixel coordinates convert to world coordinates using the camera intrinsics and robot pose.
4. A delta arm, micro-spray nozzle, or laser targets the weed center.

Pixel-to-World Projection

With intrinsic matrix K and depth Z at pixel (u, v):

X_c = (u - c_x) * Z / f_x
Y_c = (v - c_y) * Z / f_y
Z_c = Z

P_world = T_world<-base * T_base<-cam * [X_c, Y_c, Z_c, 1]^T

Here f_x, f_y are focal lengths in pixels, c_x, c_y are the principal point, and each T is a 4×4 homogeneous transform.

Timing Budget

At 4 km/h (1.11 m/s) with the actuator 0.4m behind the camera, the robot has 0.36s from detection to action. The pipeline must fit inside it:

ROS2 Weed-Targeting Node (Python)

import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image, CameraInfo
from geometry_msgs.msg import PointStamped
from cv_bridge import CvBridge
import numpy as np
import tf2_ros
import tf2_geometry_msgs  # registers PointStamped transform support

class WeedTargeter(Node):
    def __init__(self):
        super().__init__('weed_targeter')
        self.bridge = CvBridge()
        self.K = None  # 3x3 intrinsic matrix, filled from CameraInfo
        self.tf_buffer = tf2_ros.Buffer()
        self.tf_listener = tf2_ros.TransformListener(self.tf_buffer, self)

        # Topics: aligned depth, weed mask (mono8, 255 = weed), camera intrinsics
        self.create_subscription(CameraInfo, '/camera/color/camera_info', self.info_cb, 10)
        self.create_subscription(Image, '/camera/depth/aligned', self.depth_cb, 10)
        self.mask = None
        self.create_subscription(Image, '/weed_mask', self.mask_cb, 10)

        # Output: weed centroid in the base_link frame
        self.target_pub = self.create_publisher(PointStamped, '/weed_target', 10)

    def info_cb(self, msg):
        self.K = np.array(msg.k).reshape(3, 3)  # [fx 0 cx; 0 fy cy; 0 0 1]

    def mask_cb(self, msg):
        self.mask = self.bridge.imgmsg_to_cv2(msg, 'mono8')

    def depth_cb(self, msg):
        if self.K is None or self.mask is None:
            return
        depth = self.bridge.imgmsg_to_cv2(msg, 'passthrough').astype(np.float32) / 1000.0  # mm -> m
        ys, xs = np.nonzero(self.mask)          # pixel coords of weed pixels
        if len(xs) < 50:                        # reject tiny blobs (noise)
            return
        u, v = xs.mean(), ys.mean()             # centroid in pixels
        Z = np.median(depth[ys, xs])            # robust depth estimate
        if not np.isfinite(Z) or Z <= 0:
            return

        fx, fy, cx, cy = self.K[0, 0], self.K[1, 1], self.K[0, 2], self.K[1, 2]
        pt = PointStamped()
        pt.header = msg.header
        pt.point.x = (u - cx) * Z / fx
        pt.point.y = (v - cy) * Z / fy
        pt.point.z = Z

        try:
            out = self.tf_buffer.transform(pt, 'base_link', timeout=rclpy.duration.Duration(seconds=0.05))
            self.target_pub.publish(out)
        except Exception as e:
            self.get_logger().warn(f'TF failed: {e}')

def main():
    rclpy.init()
    rclpy.spin(WeedTargeter())

if __name__ == '__main__':
    main()

Actuation Comparison

Harvesting Robots

Kinematic Foundations

A 6-DOF arm needs a closed-form or numerical inverse kinematics (IK) solution. Forward kinematics with Denavit-Hartenberg (DH) parameters:

T_{i-1}^i = Rot_z(theta_i) * Trans_z(d_i) * Trans_x(a_i) * Rot_x(alpha_i)

T_0^6 = T_0^1 * T_1^2 * T_2^3 * T_3^4 * T_4^5 * T_5^6

For numerical IK, use the damped least-squares (Levenberg-Marquardt) Jacobian update:

delta_q = J^T * (J * J^T + lambda^2 * I)^-1 * e

where:
  e      = [position error; orientation error]  (6x1)
  J      = geometric Jacobian (6xn)
  lambda = damping factor (0.01-0.1), avoids singularity blow-up

Visual Servoing

Position-based visual servoing (PBVS) closes the loop on the fruit pose estimated from the camera:

v_cam = -lambda_g * (P_target - P_ee)

with gain lambda_g = 0.5-2.0 s^-1 and a velocity limit of 0.25 m/s near the fruit

Image-based servoing (IBVS) is more robust to calibration error but can drive the arm through singular configurations.

Gripper and End Effector Trade-offs

Harvest Cycle State Machine

IDLE -> DETECT -> PLAN -> APPROACH -> GRASP -> DETACH -> PLACE -> RETRACT -> IDLE
                     |        |          |        |
                     +--------+----------+--------+--> RECOVER (on timeout / force fault)

Implement this with BehaviorTree.CPP or py_trees_ros, and use MoveIt2 for collision-aware planning around stems and trellis wires.

Monitoring Robots (UAV and UGV)

Vegetation Index Computation

Multispectral cameras output NIR and Red bands. The standard NDVI formula:

NDVI = (NIR - Red) / (NIR + Red)

Typical interpretation:
  < 0.2   bare soil / stressed
  0.2-0.5 sparse or moderately healthy vegetation
  > 0.6   dense healthy canopy

Calibrate with a reflectance panel before each flight. Without it, lighting changes shift NDVI by 10-20%.

State Estimation: EKF Fusion

The EKF state vector for a ground rover:

x = [px, py, theta, v, omega]^T

Predict: x_k|k-1 = f(x_{k-1}, u_k)
Update:  x_k = x_k|k-1 + K_k * (z_k - h(x_k|k-1))
         K_k = P_k|k-1 * H^T * (H * P_k|k-1 * H^T + R)^-1

Sensor noise trade-offs: RTK-GNSS gives low-drift position at 5-10Hz, the IMU gives high-rate yaw rate at 100-200Hz, and wheel odometry fills the gaps but slips on wet soil.

robot_localization Configuration (ROS2)

# ekf_filter.yaml
ekf_filter_node:
  ros__parameters:
    frequency: 30.0            # filter output rate in Hz
    two_d_mode: true           # ignore z, roll, pitch for flat fields
    odom_frame: odom
    base_link_frame: base_link
    world_frame: odom

    # Wheel odometry: use vx and yaw rate only
    odom0: /wheel/odometry
    odom0_config: [false, false, false,   # x, y, z
                   false, false, false,   # roll, pitch, yaw
                   true,  false, false,   # vx, vy, vz
                   false, false, true,    # roll rate, pitch rate, yaw rate
                   false, false, false]   # ax, ay, az

    # IMU: use yaw rate and forward acceleration
    imu0: /imu/data
    imu0_config: [false, false, false,
                  false, false, false,
                  false, false, false,
                  false, false, true,
                  true,  false, false]
    imu0_remove_gravitational_acceleration: true

    # GNSS (via navsat_transform_node) in odom frame
    odom1: /gps/odometry
    odom1_config: [true,  true,  false,
                   false, false, false,
                   false, false, false,
                   false, false, false,
                   false, false, false]

Control Design: Tuning a PID Loop for a Skid-Steer Field Robot

Wheel speed control is the base layer of every ground platform.

u(t) = Kp * e(t) + Ki * integral(e) dt + Kd * de/dt

Discrete form at sample time Ts:
u[k] = Kp*e[k] + Ki*Ts*sum(e) + Kd*(e[k] - e[k-1])/Ts

Practical Tuning Workflow

1. Run the loop at 1kHz on the MCU with encoder feedback at 2000+ counts/rev.
2. Set Ki = Kd = 0. Raise Kp until the step response oscillates, then record Ku (ultimate gain) and Tu (oscillation period).
3. Apply Ziegler-Nichols PID: Kp = 0.6*Ku, Ki = 1.2*Ku/Tu, Kd = 0.075*Ku*Tu.
4. Reduce Kp by 20-30% to cut overshoot in loose soil.
5. Add anti-windup (clamp the integral term to ±30% of max output) because wheels stall in mud.
6. Add a low-pass filter on the derivative term (cutoff ~50Hz) to suppress encoder noise.

Row-Following Controller

Use a pure-pursuit or Stanley controller on the lateral error e_y and heading error e_theta:

delta = e_theta + atan(k * e_y / (v + eps))

Typical values: k = 1.0-2.5, eps = 0.1 m/s to prevent division by zero at standstill.

Real-World Workflow: Autonomous Row Navigation with ROS2 Nav2

Goal: Drive a UGV down 0.75m-spaced crop rows with 5cm lateral tolerance.

1. Mount a 2D LiDAR 0.4m above ground, facing forward.
2. Launch the sensor driver and EKF (robot_localization).
3. Extract rows: fit a line to LiDAR points with RANSAC (inlier threshold 3cm) on each side of the robot.
4. Compute the centerline between fitted lines and publish it as a nav_msgs/Path.
5. Configure Nav2 with the RegulatedPurePursuitController:

controller_server:
  ros__parameters:
    controller_frequency: 20.0
    FollowPath:
      plugin: "nav2_regulated_pure_pursuit_controller::RegulatedPurePursuitController"
      desired_linear_vel: 1.0          # m/s, field-safe speed
      lookahead_dist: 1.2              # m, shorter for tight rows
      min_lookahead_dist: 0.6
      max_lookahead_dist: 2.0
      use_velocity_scaled_lookahead_dist: true
      use_regulated_linear_velocity_scaling: true
      regulated_linear_scaling_min_speed: 0.25
      transform_tolerance: 0.1

6. Handle headlands: switch to GNSS waypoint following at row ends and execute a 180° U-turn with a minimum radius equal to the vehicle’s turning limit.
7. Validate in Gazebo with a procedurally generated crop-row world before field trials.

Simulation and Testing

Generate synthetic training images with domain randomization (lighting, soil texture, weed density) to cut labeled-data needs by 40-60%.

Deployment Checklist

• IP67 rating on enclosures, connectors, and sensors.
• E-stop hardware loop independent of software (and a 2m safety bubble using LiDAR).
• Thermal limits: Jetson throttles above 80°C, so add a heat pipe and shade.
• Time sync: use PTP or chrony so camera, LiDAR, and IMU timestamps align within 1ms.
• Calibration: redo camera-to-arm hand-eye calibration after any mechanical shock.

FAQ

How accurate are agricultural weeding robots?

Commercial systems reach 90-98% weed-detection accuracy and ±3-5mm actuation precision under normal lighting. Accuracy drops with heavy shadows, dust, and overlapping crop-weed leaves, which is why many units use shaded enclosures and artificial illumination.

What sensors do agricultural robots use for navigation?

Most combine RTK-GNSS (±2cm), an IMU, wheel encoders, and a 2D or 3D LiDAR. GNSS handles global positioning, while LiDAR and cameras handle local row following where canopy blocks satellite signals.

Can ROS2 be used for agricultural robots?

Yes. ROS2 supplies Nav2 for navigation, robot_localization for sensor fusion, MoveIt2 for manipulation, and DDS-based communication with configurable QoS for lossy field networks.

Why is robotic harvesting harder than weeding?

Harvesting requires 3D fruit localization under occlusion, collision-free arm planning through foliage, and gentle grasping. Cycle times of 4-10 seconds per fruit still trail human pickers, so economics depend on 24-hour operation.