Working at Richard Andersen's lab at Caltech, we designed and implemented a shared control robotic arm framework that takes decoded neural intent and translates it to actions that are executed by the arm. The framework consists of several modules:  

​

• Objection detection powered by Microsoft Kinect

• JACO arm controller to send commands and receive sensory data

• Trajectory planner to avoid collisions and determine the path for a specified action

• Action handler to determine the available commands and execute them

​

Let's walk through an example!

1. The system is tasked with finding bottles and apples in the environment for the patient to interact with.

2. The object detection module detects an apple and a bottle.

3. The patient is presented with a list of potential actions: grab bottle, grab apple.

4. The neural decoder decodes the patient imagining "grasping a bottle" and selects "grab bottle"

5. The trajectory planner calculates a route to pick up the bottle and the arm controller executes it

6. Once the bottle is grasped, the action handler generates a new list of actions: bring bottle to mouth, return bottle, give bottle to caregiver

7. The neural decoder decodes the patient imagining "drinking" and selects "bring bottle to mouth"

8. The trajectory planner routes the robot arm to bring the bottle to the patient, allowing them to drink

​

Objection detection was built with algorithms from PCL.

The modules were developed using ROS.

For robot arm actuation we used Kinova's JACO API.