Reproducing and Training the ACT Neural Network from the LeRobot Project

Team Members: Jiawei Xia~, Xianglei Dong~

• The robotic arm involved in the project was 3d printed as well as physically assembled. We also attempted to train the dataset ourselves to allow the robotic arm to autonomously organize the desktop

• Set up the project environment and dependencies, including configuring frameworks. Conducted in-depth research on the ACT algorithm from the LeRobot project, implemented and reproduced its core functionalities based on the original paper. Designed and executed training pipelines, monitored performance metrics, and conducted multiple experiments to validate results against the paper’s benchmarks. Debugged training issues (e.g., gradient vanishing and hyperparameter tuning) and optimized model performance.

RoboCup@Home China Open 2024

Team Members: Jiawei Xia, Xianglei Dong, Wengcong Zhang, Boyi Zhang, Yutong Sun

• Developed an intelligent home service robot using Kinova Gen2 and Kinect Azure camera, with functions including guest reception and guidance, voice interaction, object recognition, and room cleaning
• Used the End-to-end grasping algorithm Graspnet to obtain the 6D pose of the object, which will be sent to the end of the robotic arm by ‘TF listener, Controlled the arm using Actionlib and Movelt!, used the RR’T* algorithm for efficient trajectory generation, and TRAC-IK for an accurate inverse kinematics solver. Adopted Azure Al Speech SDK for speech recognition and voice output, combined with a finite state machine to enable simple human-computer voice interaction. For object recognition, integrated YOLOv8 3d detection and Kinect DK. Applied Cartographer for 2D indoor mapping, used IRIS LaMa for robot localization, achieved local path planning based on the DWA and global path planning based on the A* algorithm

ICRA RoboMaster University Sim2real Challenge 2024

Team Members: Jiawei Xia, Xianglei Dong, *Wengcong Zhang, Boyi Zhang

• ICRA 2024 RoboMaster University Sim2Real Challenge: Utilized ArUco markers to determine the poses of boxes. Extracted the transformation and rotation matrices of boxes from detected marker poses. Deployed an Extended Kalman Filter (EKF) alongside an omnidirectional robot to enhance localization and pose estimation accuracy.

Waypoint_nav_plugin

Team Members: Xianglei Dong

• Implemented functionality for outputting waypoints to a waypoints.txt file.

• Enabled multi-point navigation for robots, significantly reducing the need for manual input of coordinate points in RViz.

• Contributed to the improvement of robot path planning and navigation efficiency.

YOLOv8 GraspNet ROS

Team Members: Xianglei Dong

• An integrated ROS (Robot Operating System) based system combining YOLOv8 target detection and GraspNet grasping pose generation, mainly for robot visual grasping tasks.

Yolov8 3d detection based on ROS

Team Members: Xianglei Dong

• Based on ROS Noetic, YOLO 3D Detection is a real-time 3D target detection technology based on YOLO (You Only Look Once) series of target detection algorithms, designed for autonomous driving, robot navigation and other scenarios, and is able to predict the 3D Bounding Box and spatial attitude of an object from 2D images or point cloud data.