RPMArt: Towards Robust Perception and Manipulation for Articulated Objects

Our primary contribution is the robust articulation network, which is carefully designed to be robust and sim-to-real, by local feature learning, point tuple voting, and an articulation awareness scheme. First, a collection of point tuples are uniformly sampled from the point cloud. For each point tuple, we predict several voting targets with a neural network from the local context features of the point tuple. Further, an articulation score is applied to supervise the neural network so that the network is aware of the articulation structure. Then, we can generate multiple candidates using the predicted voting targets, given the one degree-of-freedom ambiguity constraint. The candidate joint origin, joint direction and affordable point with the most votes, from only point tuples with high articulation score, are selected as the final estimation.

We evaluate RPMArt in both noise-added simulation and real-world environments. In the simulation environment, we add different levels of noise to the point clouds to test the robustness of models. And we also test the sim-to-real transfer ability of RPMArt by directly applying the model trained on synthetic data to real-world scenarios.

Joint origin estimation results

Joint direction estimation results

Affordable point estimation results

We also add different levels of noise to the input point clouds, and report the manipulation success rates. We run around 100 trials per object instance for each task. Higher is better. Results are averaged across six tasks. We also run manipulation experiments using the ground truth joint parameters and affordable points, and the average success rates are 96.694% and 99.627% for pulling and pushing respectively. It demonstrates that RPMArt can still stably manipulate articulated objects with inaccurate perception results. We also show the twelve task examples finished by RPMArt under noise level 2. We can observe that the perception results are still robust to noise but actually not perfect, but RPMArt can still successfully manipulate the articulated objects.

Manipulation results

Pull Microwave

Pull Refrigerator

Pull StorageFurniture (Prismatic)

Pull Safe

Pull StorageFurniture (Revolute)

Pull Drawer

Push Microwave

Push Refrigerator

Push StorageFurniture (Prismatic)

Push Safe

Push StorageFurniture (Revolute)

Push Drawer

We present perception results on real-world point clouds of six articulated objects, using the models trained only on synthetic data. We collect data by capturing depth images for each object with 5 different joint states, each from 20 different camera views, and throw away some bad views which cannot capture enough area of the target part. Here shows the mean estimation errors and standard deviations under the condition of excluding backgrounds and distractors. Lower is better. We can find that RoArtNet demonstrates more stable performance compared to other baselines. However, some performance degradation is found in the StorageFurniture and Drawer categories for RoArtNet, as well as for ANCSH and GAMMA. This could possibly be attributed to the relatively small size of parts in these two objects, where all three models somewhat rely on part segmentation to complete the estimation. Another noteworthy observation pertains to the performance on WashingMachine. Specifically, only RoArtNet successfully estimates targets accurately, while the other three baselines exhibit significantly large estimation errors. We find a potential reason that we take a relatively small washing machine toy as the object, then the influence of noisy points is relative significant.

We also visualize some qualitative results of the performance on real-world articulation perception, with both background and distractors included. Color is used only for visualization here. Red arrows represent the estimated articulation joints, and blue points represent the estimated affordable points. RoArtNet can still generate robust joint parameters and affordable points estimation even with the presence of some unrelated points.

We also apply the models to manipulate the real articulated objects, under the condition of excluding backgrounds and distractors. We run 10 trials for each task, and count the number of successful, half-successful and failed trials. Here, half-successful trials include behaviors like detaching during pulling and pushing forcefully. RPMArt outperforms other counterparts, especially for Refrigerator and WashingMachine. Refrigerator has a glossy surface, while WashingMachine is relatively small, which makes the noise more prominent.

Some half-successful example trials.

Due to inaccurate joint parameters estimation, the robot occasionally detaches the object during manipulation. And due to incomplete point cloud and inaccurate affordable point estimation, the selected grasp pose may not be suitable for manipulation.