Presented by Dr. Fang Wan @ Chicago, USA, during WAFR2024. 
In industrial settings, swiftly and accurately moving fabrics to target positions for downstream tasks is a common operation. Workers often employ dynamic flinging motions to enhance efficiency. This observation raises our first challenge: Can we train robots to achieve precise fabric positioning in one fling? 
Previous research often involves robots making multiple attempts to learn the delta dynamics between each trial, continuously optimizing the fling trajectory to reach the target position. However, multiple trials reduce efficiency and delta dynamics are difficult to generalize to complex environments, typically only being effective in flat scenarios. 
Furthermore, fabric properties significantly influence the outcome of the fling. For example, With the same trajectory, more elastic fabrics tend to produce more wrinkles, needs a lighter Fling. This diversity leads to our second challenge: How can we adjust the fling trajectory when fabric properties are unknown? 
Beyond industrial applications, fabric manipulation tasks are commonly-used in daily life, where environments are more complex and often involve interactions between the fabric and surrounding objects. This complexity introduces our third challenge: Can we encode the rich interaction between fabric and environment for robotic manipulation? 
To address these challenges, we propose 1Fling2Goal. Our objective is to manipulate the fabric dynamically into the target state with only one fling in complex environments. We utilize a dual-arm robot setup and an RGB-D camera to capture real-time state of the fabric. Our method is tested across various complex scenarios, as shown on the right, where environmental objects are randomly placed within the workspace, and corresponding fabric target states are defined. 
We begin by training a dynamic model of the fabric where the state is represented as a point cloud. We employ a Graph Neural Network (GNN) to predict the fabric’s point-wise velocities at the next time step, capturing the complex dynamics involved in fabric manipulation. 
To enhance the dynamic model’s performance during interactions with the environment, we integrate environmental information into the node features of our GNN. We term this approach Environment-Aware State Representation. For each node vi, we calculate the distance between the fabric xi and the nearest object point qj in the environment. Later we will show our model more accurately predicts motions by involving fabric-environment contact. 
We collected training data from the PyFlex simulation environment. During data collection, we randomized key parameters related to fabric properties, such as weight and size. Which allows our model to have good generalization capabilities across different fabrics. 
With the dynamic model, we implement a two-stage model-based control strategy to optimize the fling trajectory. In stage one, we acquire the fabric’s current state using the RGB-D camera. 
In this work, we simplify potential trajectory to a fling-then-pull movement, which can be parameterized by a single turning point. To get an initial estimation of the fling trajectory, we randomly sample 50 turning points. 
And then utilize the dynamic model to infer the final state for each sampled trajectory. We select the optimal trajectory whose final state has the minimum distance to the target state. 
Despite this initial estimation, unknown fabric properties may prevent accurate task completion. Therefore, we implement real-time refinement of the fling trajectory. At every ten time steps, we sample 9 delta actions to adjust the current action of the initial estimation. Here delta actions include faster, slower flinging movements, as well as no change to the initial estimation. 
We select the best delta actions based on the dynamic model’s inference. For instance, if we detect that the fabric is highly elastic, we adjust by reducing the fling speed appropriately. 
At each control step, we update the fling trajectory and the fabric’s current state, ensuring that adjustments are responsive to real-time observations and predictions. 
Through this iterative process, the fabric is eventually manipulated to reach the target state. 
We evaluate the performance of the proposed GNN dynamic model through a series of comparative experiments. The results show that incorporating environmental awareness into the fabric representation and collecting data across multiple scenarios significantly improve the model’s accuracy and generalization. The video on the right illustrates our model’s ability to accurately predict the fabric’s motion, even during contact with environmental objects. 
For the goal-conditioned fabric manipulation task, we compared our method with state-of-the-art approaches such as FlingBot and IRP. We also conducted ablation studies on key components of our algorithm, namely environmental awareness and Model predictive control. The experimental results indicate that our method exhibits notable stability and generalization, especially in complex environments. 
Here, we present simulation results across different scenarios, including Flat Scenario without environment awareness, and Blue particles represent the fabric’s state, while red arrows indicate velocity of the fabric’s motion. The green rectangles show the fabric’s target state for visualization purpose; We use the fabric’s point cloud to compute the distance between current state and the target state. 
Platform Scenario, 
Hemisphere Scenario, and 
Pole Scenario with Environment Awareness. 
The proposed two-stage strategy transfers seamlessly to real-world robotic experiments. we randomly place the fabric within the workspace and allow the robot to plan an initial trajectory to fling the fabric to the target position and do real-time adjustment according the RGBD observation. Our method proves to be effective across various settings, including flat surfaces. 
platforms, 
hemispheres, 
poles, and 
stools. We also validate our approach on 
unseen fabrics with novel shapes and 
unseen folded target states. It works well since our dynamic model is generalized and target-free. And the control strategy is applicable to any feasible target. 
In summary, we propose a method that encodes environmental information into the dynamic model of fabric, achieving precise fabric manipulation in one fling. Our approach addresses the challenges of unknown fabric properties and complex environmental interactions. 
There are some limitations. Inaccurate perception due to fling speed and partial observation can sometimes cause the fabric to contact the environment too abruptly, leading to task failure. Additionally, the model’s effectiveness can be limited when interacting with unseen fabric configurations that differ drastically from the training data. Future work will focus on improving the robustness to perception accuracy and extending the model to more diverse task configurations. 
Presented by Dr. Fang Wan @ Chicago, USA, during WAFR2024.
CopGNN: Learning End-to-End Cloth Coverage Prediction via Graph Neural Networks. 2024, (Extended Abstract accepted to IROS 2024 Workshop on Benchmarking via Competitions in Robotic Grasping and Manipulation).
Abstract
Cloth manipulation in robotics, such as folding or unfolding fabrics, remains challenging due to deformable materials' complex and nonlinear dynamics, which can adopt infinite configurations. As Team Greater Bay, we participated in the ICRA 2024 Cloth Competition and scored an Average Coverage of 0.53 (the 1st place team scored 0.60). This extended abstract presents our Coverage Prediction Graph Neural Network (CopGNN) approach implemented for this competition. Instead of directly estimating the cloth's configuration, our method implicitly infers the unknown state using a Graph Neural Network (GNN). It predicts the resultant coverage area from multiple grasping points using a second GNN without relying on an explicit dynamics model. Contributions of this work include: (1) Developed a comprehensive simulation pipeline to generate a large-scale dataset tailored to the cloth manipulation task. (2) Proposed an end-to-end approach to predict the coverage area using only the hanging cloth's depth image. (3) Introduced a heuristic-based sampling strategy to enhance the robustness of zero-shot sim-to-real transfer.
Links
BibTeX (Download)
@workshop{Sun2024CopGNN,
title = {CopGNN: Learning End-to-End Cloth Coverage Prediction via Graph Neural Networks},
author = {Haoran Sun and Linhan Yang and Zeqing Zhang and Ning Guo and Lei Yang and Fang Wan and Chaoyang Song and Jia Pan},
url = {https://sites.google.com/view/iros2024-workshop-bench-in-rgm/},
year = {2024},
date = {2024-10-13},
urldate = {2024-10-13},
abstract = {Cloth manipulation in robotics, such as folding or unfolding fabrics, remains challenging due to deformable materials' complex and nonlinear dynamics, which can adopt infinite configurations. As Team Greater Bay, we participated in the ICRA 2024 Cloth Competition and scored an Average Coverage of 0.53 (the 1st place team scored 0.60). This extended abstract presents our Coverage Prediction Graph Neural Network (CopGNN) approach implemented for this competition. Instead of directly estimating the cloth's configuration, our method implicitly infers the unknown state using a Graph Neural Network (GNN). It predicts the resultant coverage area from multiple grasping points using a second GNN without relying on an explicit dynamics model. Contributions of this work include: (1) Developed a comprehensive simulation pipeline to generate a large-scale dataset tailored to the cloth manipulation task. (2) Proposed an end-to-end approach to predict the coverage area using only the hanging cloth's depth image. (3) Introduced a heuristic-based sampling strategy to enhance the robustness of zero-shot sim-to-real transfer.},
note = {Extended Abstract accepted to IROS 2024 Workshop on Benchmarking via Competitions in Robotic Grasping and Manipulation},
keywords = {Abstract - Extended Abstract, Authorship - Corresponding, Conf - IROS, Special - Workshop},
pubstate = {published},
tppubtype = {workshop}
}
