Motion Planning for Robotic Manipulation of Deformable Linear

Motion Planning for Robotic Manipulation of Deformable Linear Objects (DLOs) Mitul Saha and Pekka Isto Artificial Intelligence Lab Stanford University Research Institute for Technology University of Vaasa, Finland Research supported by NSF

Manipulation Planning Research so far… • The ability to autonomously manipulate objects is one of most desirable features in a robot. Hence manipulation planning has been an active area of research for the last many decades • So far, manipulation planning research has mainly focused on manipulating rigid objects • We have been interested in manipulation planning for deformable objects, because a large number of objects that we handle in our daily lives are deformable to some extent • There has not been much development in manipulation planning for deformable objects because –it is difficult to model and predict the deforming nature of deformable objects –struggle in basic motion planning

Manipulation Planning Research so far… • The ability to autonomously manipulate objects is one of most desirable features in a robot. Hence manipulation planning has been an active area of research for the last many decades • So far, manipulation planning research has mainly focused on manipulating rigid objects • We have been interested in manipulation planning for deformable objects, because a large number of objects that we handle in our daily lives are deformable to some extent • There has not been much development in manipulation planning for deformable objects because –it is difficult to model and predict the deforming nature of deformable objects –struggle in basic motion planning

Manipulation Planning Research so far… • The ability to autonomously manipulate objects is one of most desirable features in a robot. Hence manipulation planning has been an active area of research for the last many decades • So far, manipulation planning research has mainly focused on manipulating rigid objects • We have been interested in manipulation planning for deformable objects, because a large number of objects that we handle in our daily lives are deformable to some extent • There has not been much development in manipulation planning for deformable objects because –it is difficult to model and predict the deforming nature of deformable objects –struggle in basic motion planning

Manipulation Planning Research so far… • The ability to autonomously manipulate objects is one of most desirable features in a robot. Hence manipulation planning has been an active area of research for the last many decades • So far, manipulation planning research has mainly focused on manipulating rigid objects • We have been interested in manipulation planning for deformable objects, because a large number of objects that we handle in our daily lives are deformable to some extent • There has not been much development in manipulation planning for deformable objects because –it is difficult to model and predict the deforming nature of deformable objects –struggle in basic motion planning

Manipulation Planning for Deformable Linear Objects (DLOs) GOAL: to develop a motion planner that would enable robots to autonomously manipulate Deformable Linear Objects (ropes, cables, sutures) in various settings. knot tying in daily/recreational life bowline knot figure-8 knot sailing knot laying/loading cables in industrial settings suturing in medical surgery robot dress autonomous robotic DLO manipulation

Manipulation Planning for Deformable Linear Objects (DLOs) Challenging • The DLO manipulation problem is extremely challenging for robotics because o being highly deformable, they can exhibit a much greater diversity of behaviors, which are hard to model and predict o identifying topological states of DLOs is coupled with some unsolved problems in knot-theory/ mathematics Interesting • The DLO manipulation problem has a nice structure. It brings together robotics, knot theory, and computational mechanics.

Previous Related Work “Planning of One-Handed Knotting/Raveling Manipulation of Linear Objects”, IEEE ICRA 2004, Wakamatsu, et. al. - knot simplified using Reidemeister moves (RM) from knot theory -one robot used to execute the RMs -assumes DLO resting on a plane

Previous Related Work “Planning of One-Handed Knotting/Raveling Manipulation of Linear Objects”, IEEE ICRA 2004, Wakamatsu, et. al. - knot simplified using Reidemeister moves (RM) from knot theory -one robot used to execute the RMs -assumes DLO resting on a plane Our contribution: -DLO need not be in a plane -We use more than one robot in coordination -We consider collision constraints (robot-DLO, robot-obstacle) -We consider the physical behavior of the DLO while planning -We consider interaction of the DLO with other objects

The Manipulation Problem available robot arms How do we define goal configurations?

Defining Goal Configurations • Goal configurations are defined in terms of topology instead of exact geometry Geometrically different but topologically same: Bowline knot while winding, number of wounds more important

Defining Goal Configurations • In knot theory, crossing configuration of a curve is used to characterize its topology planar projection of the DLO central axis

Defining Goal Configurations • In knot theory, crossing configuration of a curve is used to characterize its topology planar projection of the DLO central axis crossing: local self-intersections C 1: C 2: (-2, 5)C 3: (3, -8)- (1, -6)C 4: (-4, 7)- sign of a crossing Configuration: (C 1, C 2, C 3, C 4): ((1, -6)-, (-2, 5)-, (3, -8)-, (-4, 7)-) how to account for interactions with other objects? make them part the DLO semi-deformable linear object (s. DLO)

Physical modeling of the DLO We take as input the physical model of the DLO in the form of a state transition function f: Recent successes in computational mechanics: Suture model: [Brown, et al. , 04] Elastic thread model: [Wang, et al. , 05] Nylon thread model: [Dhanik, 05]

Manipulation Tools • Manipulation using 2 cooperating robot arms

Manipulation Tools • Manipulation using 2 cooperating robot arms • Use of static sliding supports (“tri-needles”) to provide structural support

Basis of our Planning Approach • Defining “Forming Sequence” walk along the DLO; crossing “formed” when encountered the second time Forming Sequence: C 2, C 1, C 4, C 3

Basis of our Planning Approach • Defining “Forming Sequence” walk along the DLO; crossing “formed” when encountered the second time Forming Sequence: C 2, C 1, C 4, C 3 C 2 C 3 C 1 C 4 A DLO topology or knot can be tied, crossing-by-crossing, in the order defined by its “forming sequence”

Basis of our Planning Approach • Defining “Forming Sequence” walk along the DLO; crossing “formed” when encountered the second time Forming Sequence: C 2, C 1, C 4, C 3 C 2 C 3 C 1 C 4 A DLO topology or knot can be tied, crossing-by-crossing, in the order defined by its “forming sequence” • Defining “loop hierarchy” used to determine the placementof static sliding supports (“tri-needles”)

Our Manipulation Planning Algorithm search tree forbidden region -search the configuration-space using a sampling-based tree -use physical model to sample new DLO shapes -use forming sequence to bias search -use the loop hierarchy to place static sliding supports (tri-needles)

Our Manipulation Planning Algorithm search tree forbidden region -search the configuration-space using a sampling-based tree -use physical model to sample new DLO shapes -use forming sequence to bias search -use the loop hierarchy to place static sliding supports (tri-needles) Robot A DLO grasping robot fails Robot A Robot B

Our Manipulation Planning Algorithm search tree forbidden region -search the configuration-space using a sampling-based tree -use physical model to sample new DLO shapes -use forming sequence to bias search -use the loop hierarchy to place static sliding supports (tri-needles)

Our Manipulation Planning Algorithm search tree forbidden region -search the configuration-space using a sampling-based tree -use physical model to sample new DLO shapes -use forming sequence to bias search -use the loop hierarchy to place static sliding supports (tri-needles) loop hierarchy tri-needles

Results -Planner implemented in C++ -Took 15 -20 minutes on a 1 GB, 1 GHz processor to generate manipulation plans for tying popular knots: bowline, neck-tie, bow (shoe-lace), and stunsail -Videos: http: //ai. stanford. edu/~mitul/dlo bowline knot neck-tie bow sailing knot

Results

Results neck-tie

Results In the real-life, we have tested the ability of the planner to generate robust plans by tying the popular Bowline knot with various household ropes on a hardware platform with two PUMA robots, using the manipulation plan generated by the planner. bowline knot robustness dues to tri-needles

Conclusion • We have developed a motion planner for manipulating deformable linear objects (such as ropes, cables, sutures) in 3 D using cooperating robots. - it can tie self-knots and knots around rigid objects - unlike in traditional motion planning, goals are topological and not geometric - we account for the physical behavior of the DLO - it is robust to imperfections in the physical model of the DLO - it is first of its kind (we not aware of any other planner for computing collisionfree robot motions to manipulate a DLO in environments with obstacles) - the implemented planner has been tested both in graphic simulation and in real-life on a dual-PUMA-560 hardware platform • Future Plans suturing in medical surgery collaboration with General Motors

Motion Planning for Robotic Manipulation of Deformable Linear Objects (DLOs) Acknowledgement: Advisory: Jean-Claude Latombe PUMA experiments: Oussama Khatib, Irena, Jaehueng Park, Jin Sung Physical models of ropes: Etienne Burdet, Wang Fei (EPFL) Useful comments: anonymous reviewers

We focus on two types of common knots: - Tight knots under over Crossing Configuration: ((1, -6)-, (-2, 5)-, (3, -8)-, (-4, 7)-) - Semi-tight knots over

Needle Placement