RePAIR
is a project funded by the Horizon 2020 Research and Innovation
program of the European Union. The main goal of the project is to
develop an automated solution for the physical reconstruction of
fragmented
fresco artworks.
More precisely, we aim to leverage
advancements in robotics and computer vision to support the
archaeological site of Pompeii in the digitization of fresco
fragments and their subsequent restoration to their original forms.
This automated solution will virtually eliminate one of the most
labor-intensive challenges: the processing and reassembling of thousands
of fresco pieces, thus significantly accelerating the job of archaeologists.
The final system is expected to autonomously handle
fragile fresco artifacts. This system will be able to digitize
the artifacts, extract their 3D structure using a network of
camera sensors, digitally reconstruct them into their original
configuration by solving the 3D puzzle, and ultimately physically
assemble the currently accessible pieces. Different aspects
of this process have been assigned to different partners.
Our lab is responsible for developing the grasping and motion
planning systems that will run on the robot in order to handle
the frescos.
In our lab, we have built a mock-up environment using the
Kinova Kortex Gen3 robotic arm and the Pisa/IIT
SoftHand
(QB Hand).
A main aspect of the project is to use soft robotic hands
that can adapt to the shape of the object and prevent
damaging the objects and the hardware.
Our workflow is divided into two stages. Firstly,
the perception part, in which we detect the frescos through
vision, select which one to grasp, and complete the partial
point cloud. Secondly, the motion planning part, in which we
use a geometric heuristic to propose a candidate grasp, and
the motion planner generates a feasible trajectory to execute it.
In our current pipeline, we begin by acquiring a point cloud of the
scene, which consists of fresco parts on top of a flat surface.
We, then, employ Open3D's
plane segmentation
algorithm to remove the supporting surface from the point cloud. We use the
DBSCAN algorithm
to cluster the point cloud containing
all the frescos and effectively segment the frescos with each other.
We select the biggest fresco as our first target for grasping.
The fresco point cloud is a partial view of the entire fresco which
can result in the generation of imprecise grasp candidates.
To avoid this, we employ a point cloud competition method (Mohammadi et al. 2023) that takes
as input the partial view and predicts a complete point cloud of
the target object. This complete point cloud is then used in the
subsequent grasp planning stage.
After detecting the target fresco and predicting a complete point
cloud, the next step is to compute a candidate grasp pose and execute
it. We employ a simple geometric heuristic to propose grasp poses.
More specifically, we use the center of mass of the object as the
target grasp position, which corresponds roughly to the middle point
of the palm. This way, when the fingers start closing the object
lies inside the hand. To decide the hand’s rotation, we need to take into
account the fact that the fingers of the hand should not come in
contact with the upper part of the fresco to avoid damaging the painting.
So, we want the fingers to grasp the object by the side. The rotation
of the hand is then fixed such that the palm is always parallel to
the table and looking downward. This way, when the hand starts closing,
the fingers tend to enclose the object and grasp it from its sides.
Finally, we use MoveIt
to plan the trajectory of the robot and execute it.
You can see an example of the vision and grasping pipeline in
the video below.
Using this pipeline, we can execute more complex tasks. For example, in the video below y ou can see an example task of detecting two frescos and transferring them to another table for further processing.
Currently, we are working to improve the grasping system by designing a more intelligent method to propose candidate grasp poses to the planner. Our new method is again based on the geometric structure of the object and tries to detect the antipodal sides of the object in order to use them as candidate contact points for the fingers. We are also working on a reinforcement learning solution where we have an agent performing the entire grasping and placing tasks without needing to compute grasp poses.