2.1. Kinematics analysis of CDM

In order to control CDM, we establish and analyze its kinematics model and mapping relationships by using a product of exponentials (POEs) formula method. As shown in Fig. 1(d), the robotic manipulator has 8 sections. Each section is composed of a link and a universal joint. Mathematical symbols used in this paper are listed in Table I.

Table I. Mathematical notation.

Figure 1. Design and model of a 17-degree of freedom cable-driven manipulator (CDM). (a) The designed hyper-redundant CDM. (b) A simplified geometric model of the CDM. (c) Model of a joint. (d) Geometric model.

The kinematics analysis of CDM includes relationships among task space, joint space and driving space. Since the focus of this work is path planning of CDM, we mainly derive kinematics between task space and joint one. For the task-joint kinematics, a mutual transformation between rotational joints’ angles and end effector’s posture is necessary.

As defined in our previous work [Reference Ju, Zhang, Xu, Yuan, Miao, Zhou and Cao29], $f_1$ and $f_{1}^{-1}$ are the forward and inverse kinematics between task space and joint space, respectively. We have:

(1) \begin{equation} f_1= T_{\mathbb{S},\mathbb{E}}= T_{\mathbb{S},\mathbb{B}}(dB)T_{ \mathbb{B},\mathbb{E}}(\theta _1,\theta _2,\theta _3,\ldots,\theta _{16}). \end{equation}

The twist $\hat{\xi }_{i}$ can be obtained from:

(2) \begin{equation} \hat{\xi }_{i}={ \left [ \begin{array}{cc} \hat{\omega }_{i}&\nu _{i}\\ 0&0\\ \end{array} \right ]}\in SE(3), \end{equation}

where $\hat{\omega }_{i}$ is a skew symmetric matrix of $\omega _{i}$ , $\xi _{i}=[\nu ^{T}_{i}, \omega ^{T}_{i}]^{T}$ , $\nu _{i}=-\omega _{i}\times q_{i}$ , $\omega _{i}$ and $q_{i}$ are set as Table II.

Table II. Product of exponential parameters of cable-driven manipulator.

In a POE formula method, a rotation motion of CDM can be described in the form of an exponential product as:

(3) \begin{equation} e^{\hat{\xi }\theta }={ \left [ \begin{array}{cc} R&\rho \\ 0&1\\ \end{array} \right ]}\in SE(3), \end{equation}

where $R=I+\sin \theta \hat{\omega }+(1-\cos \theta )\hat{\omega }^{2}$ , $\rho =(I\theta +(1-\cos \theta )\hat{\omega }+(\theta -\sin \theta )\hat{\omega }^{2})\nu$ and $I\in R^{3\times 3}$ is an identity matrix. The posture of the end effector is:

(4) \begin{equation} T_{\mathbb{B},\mathbb{E}}(\theta _{1},\theta _{2},\ldots,\theta _{16})=e^{\hat{\xi }_{1}\theta _{1}}e^{\hat{\xi }_{2}\theta _{2}}\ldots e^{\hat{\xi }_{16}\theta _{16}}T_{\mathbb{B},\mathbb{E}}(0), \end{equation}

where $T_{\mathbb{B},\mathbb{E}}(0)$ is an initial end posture of CDM.

2.2. Kinematics analysis of a joint

As shown in Fig. 1 (c), position and orientation of each section are determined by rotation of its joints. Rotation angles are controlled by corresponding three cables. To analyze the relationship between cable lengths and rotation angles of joints, a geometric model of a joint is simplified as shown in Fig. 1 (d). In the schematic diagram of the first universal joint, $i=1$ , the transformation relationship between coordinate systems is:

(5) \begin{equation} \begin{split} _{O}^{M}T=Trans\left ( 0,0,h \right ) Rot\left ( X,\theta _{2} \right ) \end{split}, \end{equation}

(6) \begin{equation} \begin{split} _{N}^{O}T=Rot\left ( Y,\theta _{1} \right ) Trans\left ( 0,0,h \right ) \end{split}. \end{equation}

The transformation relationship between $\{M\}$ and $\{N\}$ is:

(7) \begin{equation} \begin{split} _{M}^{N}T=_{N}^{O}T_{O}^{M}T =\left [ \begin{matrix} C_{\beta _i}& S_{\alpha _i}S_{\beta _i}& C_{\alpha _i}S_{\beta _i}& 2S_{\beta _i}h\\ 0& C_{\alpha _i}& -S_{\alpha _i}& 0\\ -S_{\beta _i}& C_{\beta _i}S_{\alpha _i}& C_{\alpha _i}C_{\beta _i}& 2C_{\beta _i}h\\ 0& 0& 0& 1\\ \end{matrix} \right ] \end{split}. \end{equation}

In addition, coordinates of cables’ holes in the corresponding disk can be computed.

The driven cables of the first joint can be computed as:

(8) \begin{equation} \begin{split} L_{c,j}=\left [\begin{array}{c} 2hS_{\theta _{1}}-r C_{\varphi +(j-1)\frac{2\pi }{3}}+rC_{\varphi +(j-1)\frac{2\pi }{3}}C_{\theta _{1}}+rS_{\theta _{2}}S_{\theta _{1}}S_{\varphi +(j-1)\frac{2\pi }{3}}\\[5pt] r C_{\theta _{2}}S_{\varphi +(j-1)\frac{2\pi }{3}}-rS_{\varphi +(j-1)\frac{2\pi }{3}}\\[5pt] 2hC_{\theta _{1}}-rC_{\varphi +(j-1)\frac{2\pi }{3}}S_{\theta _{1}}+rC_{\theta _{1}}S_{\theta _{2}}S_{\varphi +(j-1)\frac{2\pi }{3}}\\[5pt] 0\\ \end{array} \right ] \mathcal{} \end{split},\ j=1,2,3. \end{equation}

Their lengths are:

(9) \begin{equation} \begin{split} |L_{c,j}|&=((rC_{\varphi +(j-1)\frac{2\pi }{3}}(C_{\theta _{1}}-1)+2h S_{\theta _{1}}C_{\theta _{2}} +r S_{\varphi +(j-1)\frac{2\pi }{3}}S_{\theta _{1}}S_{\theta _{2}})^{2}\\[5pt] &\quad +(2h(C_{\theta _{1}}C_{\theta _{2}}+1)-rC_{\varphi +(j-1)\frac{2\pi }{3}}S_{\theta _{1}} +r S_{\varphi +(j-1)\frac{2\pi }{3}}C_{\theta _{1}}S_{\theta _{2}})^{2} \quad \quad j=1,2,3.\\[5pt] &\qquad \qquad \qquad\qquad\quad +(r S_{\varphi +(j-1)\frac{2\pi }{3}}(C_{\theta _{2}}-1)-2hS_{\theta _{2}})^{2})^{1/2} \end{split} \end{equation}

where $r$ is radius of the circle where the holes are distributed on the disk.

Using above kinematics model, desired motion of other section can be realized with given pitch and yaw angles $\theta _{2i-1}$ and $\theta _{2i}$ .

3.1. DDPG

DDPG is a model-free algorithm that can learn competitive policies for many tasks by using same hyper-parameters and network structures [Reference Lillicrap, Hunt, Pritzel, Heess, Erez, Tassa, Silver and Wierstra30]. It is illustrated in Fig. 2 and realized in Algorithm 1.

Algorithm 1: The DDPG Algorithm.

Figure 2. The deep deterministic policy gradient path-planning algorithm.

DDPG maintains a parameterized actor function $\mu (s\mid \theta ^{\mu })$ that specifies the current policy by deterministically mapping states to a specific action. A critic $Q(s, a)$ is learned by using a Bellman equation. An actor is updated by using a chain rule to an expected return from a start distribution $J$ according to parameters of the actor:

(10) \begin{equation} \begin{split} \nabla _{\theta _{\mu }}J\approx \mathbb{E}_{s_{t}\sim \rho ^{\beta }}[\nabla _{\theta ^{\mu }}Q(s,a\mid \theta ^{Q}))\mid _{ s=s_{t}, a=\mu (s_{t}\mid \theta ^{\mu })}] \\ =\mathbb{E}_{s_{t}\sim \rho ^{\beta }}[\nabla _{a}Q(s,a\mid \theta ^{Q})\mid _{s=s_{t},a=\mu (s_{t})}\nabla _{\theta _{\mu }}\mu (s\mid \theta ^{\mu })\mid s=s_{t}].\\ \end{split} \end{equation}

To accelerate the convergence of DDPG, weights of the network are updated by a back propagation algorithm based on an experience replay technology. Transitions are sampled from environments according to the exploration policy and a tuple $(s_t, a_t, r_t, s_{t+1})$ is stored in a replay buffer $\mathfrak{R}$ . The oldest samples are discarded when $\mathfrak{R}$ is full. The actor and critic networks are updated by sampling a minibatch uniformly from $\mathfrak{R}$ at each timestep. Since DDPG is an off-line policy algorithm, $\mathfrak{R}$ should be large enough to improve its benefit.

3.2. Reward function

A proper reward function is key to a successful reinforcement learning algorithm. This work proposes the following reward function to plan a safe path toward the target for the CDM, i.e.,

(11) \begin{equation} R= \left \{ \begin{array}{lr} 500, \qquad \qquad case \, 1\\ c_{1}R_{1}+c_{2}R_{2}, \quad case \, 2\\ -50, \qquad \qquad case \, 3\\ \end{array} \right . \end{equation}

where $case 1$ is the robot arrives at the target, $case 2$ is the robot moves toward the target, $case 3$ is the robot collides with obstacles, $R_{1}$ is the distance reward between the end effector and the target, $R_{2}$ is a distance reward between the end effector and the obstacle and $c_{1}$ and $c_{2}$ are weights that decide priority during training. The distance reward $R_{1}$ is computed by using a Huber-Loss function:

(12) \begin{equation} R_{1}= \left \{ \begin{array}{lr} \frac{1}{2}d^{2}, \quad |d|\lt \delta \\ \delta (|d|-\frac{1}{2}\delta ), \quad otherwise\\ \end{array}, \right . \end{equation}

where $d$ is the Euclidean distance between CDM’s end effector and the target and $\delta$ is a parameter that determines the smoothness. The second reward $R_{2}$ relates to the distance between the robot and the nearest obstacle. It is computed as:

(13) \begin{equation} R_{2}=\left(\frac{d_{o}}{ \skew{10}\check{d}+ d_{o}}\right)^{p}, \end{equation}

where $d_{o}$ is set as a constant to ensure $R_{2}\in (0, 1)$ , $\skew{10}\check{d}$ is the minimum distance between the end effector and the obstacle and $p$ is an exponential decay of the negative reward. Parameters of DDPG used in this work are listed in Table III.

Table III. Parameters of deep deterministic policy gradient.

3.3. Path following of CDMs

To control CDM to move along a planned path, a tip-following method as shown in Fig. 3 is adopted. Joints and links of CDM should be matched with the planned path. The planned path is discretized into some waypoints. As shown in Fig. 3 (a), $k_{i}$ is a key point located at the center of mass of universal joint $i$ , the pedestal is fixed with $k_{1}$ and $M_{i}$ is the $i$ - $th$ mark point located on the path. In this work, a set of points denoted as $\boldsymbol{M}$ are generated by discretizing the path. The length of a curve between adjacent points is set as a constant $s_{d}$ .

When $k_{1}$ is located at a discrete point of the path, the point is defined as $p_{k_{1}}$ . Other key points $p_{k_{i}}$ need satisfy the constraint:

(14) \begin{equation} \min |\overrightarrow{p_{k_{i}}p_{k_{i-1}}}|-d_{l}, k_{i}\in (k_{i-1}, n\times p), \end{equation}

where $n\times p$ is the number of all discretization points.

To find $p_{k_{i}}$ , this work adopts an iteratively sequential searching method proposed in refs. [Reference Wang, Tang, Gu and Zhu19, Reference Tang, Zhu, Zhu and Gu20]. As shown in Fig. 3 (a), $p_{^{t}k_{i}}$ is the nearest mark point to the key point $k_{i}$ at time $t$ . In the next time step, key points move forward along the direction of discrete points:

(15) \begin{equation} ^{t+1}k_{i}-^{t+1}k_{i-1} \approx ^{t}k_{i}-^{t}k_{i-1} = \Delta k, \end{equation}

where $\Delta k= d_{l}/s_{d}$ . Index $^{t+1}k_{i}$ can be predicted.

Figure 3. The tip-following method of cable-driven manipulators. (a) Iteration of links. (b) The interpolation algorithm.

Algorithm 2: The Key Point Searching Algorithm.

Figure 4. Simulations of path planning by using the proposed deep deterministic policy gradient algorithm. (a) The planned path. (b) The reward converges to a stable state.

With these predictions, the points $p_{^{t+1}k_{i}}$ corresponding to $^{t+1}k_{i}$ need to be verified whether satisfy (14) by using Algorithm 2. Although $p_{^{t}k_{i}}$ is the nearest mark point to the key point $k_{i}$ , the distance between $p_{^{t}k_{i}}$ and $p_{^{t+1}k_{i-1}}$ is unequal to the link’s length. A point certifies Equation (14) needs to be found around $k_{i}$ . As shown in Fig. 3 (b), $M_{t+1_{k_{i}}}$ is the predicted $p_{^{t+1}k_{i}}$ , $M_{t+1_{k_{i-1}}}$ and $M_{t+1_{k_{i+1}}}$ are its adjacent mark points. The expected $p_{^{t+1}k_{i}}$ locates the blue arc. The center and the radius of the arc are $p_{^{t+1}k_{i-1}}$ and $d_{l}$ , respectively. The expected $p_{^{t+1}k_{i}}$ can be computed by using the geometric relationships. Positions of all joints can be computed based on the planned path after this iteration.

4.1. Simulations

In our simulations, Simulink, CoppeliaSim and Pycharm are used. Communication among different platforms is implemented through a remote API. Two groups of simulation are conducted in Simulink and CoppeliaSim, respectively. In Simulink and CoppeliaSim, a virtual robot is built according to the size of our prototype and a multi-obstacle environment is conducted.

The passable path is planned by DDPG algorithm and CDM is controlled by the tip-following method to move toward the target. As shown in Fig. 4 (a), DDPG algorithm can plan a passable path (black curve) in a multi-obstacle environment. The reward can converge to a stable state as shown in Fig. 4 (b). Angles and positions of links are computed by using the tip-following method. As shown in Fig. 5 (a) and (b), the CDM can move along the planned path (black curve) without collision in Simulink and CoppeliaSim. Joint angles are measured and saved to analyze the motion controlled by the proposed methods. As shown in Fig. 5 (c), all the angles are always less than 40 degrees while moving from the start point to the target point. They are smooth enough and satisfy to mechanical constraints of the joints.

4.2. Physical experiments

Prototype experiences are used to test our path planning and following methods of CDMs. As shown in Fig. 6 (a), the robotic manipulator is connected by 8 sections. The platform is powered by a stepper motor (57TB6600). Each section can rotate between $[{-}45^{\circ }, +45^{\circ }]$ . Motors communicate with a PC via an analyzer (Kvaser Leaf Light v2). Geometric parameters of our CDM are summarized in Table IV.

Table IV. Geometric parameters of cable-driven manipulator.

Figure 5. Simulations of path planning and following of a cable-driven manipulator by using the proposed methods. (a) Simulations in Simulink. (b) Simulations in CoppeliaSim. (c) Joint angles.

To control CDM move toward the target along a desired path, a closed-loop control framework is designed. Links and obstacles are wrapped by tapes to fix markers. To measure states of CDM accurately, each link attached with 3 markers is defined as a rigid body. Positions of all markers can be measured by a 3-D OptiTrack motion capture system. States of CDM can be computed by using these data and its kinematics model. Passable paths are planned by using DDPG. Positions of all links are computed by the tip-following method. Joint angles and cable lengths are computed according to multi-level relationships among motors, joints and the end effector of the robot. As shown in Fig. 6 (b), our CDM controlled by the proposed methods can plan a possible path and move toward a target position smoothly without collision in a multi-obstacle environment.

Figure 6. The cable-driven manipulator (CDM) prototype and experiments. (a) The structure of CDM. (b) Experiments of path planning and following of CDM by using the proposed methods.