3.1. Problem Formulation

In order to model our optimization problem as an RL problem, we define the test field consisting of same area unit square, whose side length is 30 cm. K = 8 harvested energy-enabled IoT devices are deployed in the test field, whose orders are 0, 1, 2, 3, 4, 5, 6, and 7, respectively. The map is shown in Figure 2. The system state sⁿ at time slot n is defined as the position of a particular square where the robot is currently located at in the test field, which is specified as $s^n=pos\left(h,v\right)$ , where h is the distance between the present square and the leftmost edge, which is counted by the number of squares. v indicates the distance between the present square and the upmost edge, which is counted by the number of squares. For example, the No. 5 IoT device can be denoted as $o_5=pos\left(\mathrm{2,0}\right)$ . The shadow area adjacent to No. k IoT devices indicates the effective charging area for the respective IoT devices, which is denoted as eff_k. For example, the boundary of effective charging areas for No. 6 IoT device is highlighted in red. We define the direction of movement in a particular time slot n as the actions aⁿ. The set of possible actions A consists of 4 different $A=\left(U,D,L,R\right)$ , where U is moving upward one unit, D is moving downward one unit, L is moving left one unit, and R is moving right one unit.

FIGURE 2: The entire test field consists of same space unit square. K = 8 harvested energy-enabled IoT devices are deployed in the test field. The shadow area adjacent to each IoT device indicates the effective charging area for the respective IoT devices. For example, the boundary of effective charging areas for No. 6 IoT device is highlighted in red.

Given the above, the mobile wireless charging problem can be formulated as minimizing the time duration T for the robot to complete running one loop at the same time the robot has to pass through one of the effective areas of each IoT device.

(4) $\begin{array}{}{P}_1:\begin{array}{cc}\underset{\left(a^n\right)}{\text{minimize}}& T,\\ \text{subject to}& \begin{array}{c}{s}^0=s^T,\\ \exists s^n\in ef{f}_k,\forall k\in K,n=\mathrm{1,2},\dots ,T.\end{array}\end{array}\end{array}$

Time duration for the robot to complete running one loop is defined as T. The starting position is the same as the last position, since the robot cruises in a loop. In the loop, the robot has to pass through at least one of the effective charging areas of each IoT device.

Adapting to the different positions, the agent chooses different action at each time slot. Henceforth, we can model our proposed system as a Markov chain. In the system, we use the current position to specify a particular state s. S denotes the system state set. The starting state s ⁰ and final state s^T are the same, since the robot needs to move and return to the starting point. The MDP process can be described as the agent chooses an action a from A at a specific system state s. After that, a new system state s′ will be transit into. $p_{s,s^{\prime }}\left(a\right)$ , $s,s^{\prime }\in S$ and $a\in A$ , denotes the probability that system state transits from s to s′ with a.

The reward of the MDP is denoted as $w\left(s,a,s^{\prime }\right)$ , which is defined for the condition that system state transits from s to state s′. The optimization problem is formulated as reaching s^T in the fewest transmission time slots; henceforth, the reward has to be defined to motivate the mobile robot that does not repeatedly pass through any effective charging area of any IoT devices. Besides, the rewards at different positions are interconnected with each other, since the goal of the optimization is to pass through the effective charging areas of all the IoT devices. We assume that the optimal order to pass through all the IoT devices is defined as $o_0,o_1,\dots ,o_7$ . $o_k=\mathrm{0,1},\dots ,7$ . Specifically, the reward function can be expressed as

(5) $\begin{array}{}w\left(s,a,s^{\prime }\right)=\left(\begin{array}{cc}{o}_k\zeta ,& s^{\prime }\in ef{f}_{o_k},ac{c}_{o_{k-1}}=1,\\ -1,& \text{otherwise}.\end{array}\right)\end{array}$

In the above equation, $ac{c}_{o_{k-1}}=1$ if the robot has already passed through as effective area of the $o_{k-1}$ th IoT device; and ζ denotes the unit price of the harvested energy.

As we have defined all the necessary elements for MDP, we can characterize the formulated problem as a stochastic shortest path search that starts at s ⁰ and ends at s^T. At each system state s, we derive the best action $a^{\ast }\left(s\right)$ which can generate the maximum reward. The optimal policy sets are defined as $\pi =\left(a\left(s\right):s\in S\right)$ .

3.2. Optimal Path Planning with Reinforcement Learning

If the systematic dynamics obey a specific transition probability, reinforcement learning will be the perfect match to solve the optimization problems. In this section, Q-learning [Reference Barto, Bradtke and Singh23] is first introduced to solve the proposed problem. After that, to address the large states and actions sets, the DQN algorithm [Reference Mnih, Kavukcuoglu and Silver14] is utilized to determine the optimal action for each particular system state.

3.2.1. Q-Learning Method

The traditional Q-learning method is widely used to solve the dynamic optimization problem provided that the number of the system states is moderate. Corresponding to each particular system state, the best action can be determined to generate the highest reward.

Q(s, a) denotes the cost function, which uses a numerical value to describe the cost of taking action a at state s. At the beginning of the algorithm, all the cost function is zero since no action has ever been taken to generate any consequence $Q\left(s,a\right)=0$ . All the Q values are saved in the Q table. Only one cost function is updated in each time slot as the action is taken and the reward function is calculated. The cost function is updated as

(6) $\begin{array}{}Q\left(s,a\right)=\left(1-\sigma \left(s,a\right)\right)Q\left(s,a\right)+\sigma \left(s,a\right)\left(w\left(s,a,s^{\prime }\right)+\gamma f\left(s^{\prime },a\right)\right),\end{array}$

where

(7) $\begin{array}{}f\left(s^{\prime },a\right)=\underset{a\in A}{\max }Q\left(s^{\prime },a\right).\end{array}$

The learning rate is defined as $\sigma \left(s^{\prime },a\right)$ .

When the algorithm initializes, the Q table is empty since no exploration has been made to obtain any useful cost function to fill the Q table. Since the agent has no experience about the environment, the random action selection is implemented at the beginning of the algorithm. A threshold ${ϵ}_c\in \left(0.5,1\right)$ is designed to start the exploration. In each time slot, a numerical value $p\in \left(\mathrm{0,1}\right)$ is generated and compared with the threshold. If $p\ge {ϵ}_c$ , action a is picked as

(8) $\begin{array}{}a=\underset{a\in A}{\max }Q\left(s,a\right).\end{array}$

However, provided that $p<{ϵ}_c$ , an action is randomly selected from the action set A.

After iteratively updating the value in the Q table, the Q value converges. We can calculate the best action corresponding to each action and state by

(9) $\begin{array}{}{\pi }^{\ast }\left(s\right)=\arg \underset{a\in A}{\max }{Q}^{\ast }\left(s,a\right),\end{array}$

which corresponds to finding the optimal moving direction for each system state explored during the charging process.

3.2.2. DQN Algorithm

The increase in the number of IoT devices has led to an increase in the number of system states. Suppose that Q-learning algorithm is used; a very large Q table has to be created and the convergence speed is too slow. DQN algorithm is more compatible since there is a deep neural network in the structure that can be well trained and take immediate action to determine the best action that is going to be taken.

The deep neural network in the structure has the system state as the input and the Q value for each action is defined as the output. Henceforth, the function of the neural network is to generate the cost function for particular state and action. We can describe the cost function as $Q\left(s,a,\theta \right)$ , where θ is the weight on the neuron nodes in the structure. As we collect the data when different actions are taken in different time slot, the neural network is trained to update the weight of the neural network, which can output a more precise Q value:

(10) $\begin{array}{}Q\left(s,a,\theta \right)\approx Q^{\ast }\left(s,a\right).\end{array}$

There are two identical neural networks existing in the structure of DQN [Reference Van Hasselt, Guez and Silver24]: one is called the evaluation network eval_net, and the other is called the target network target_net. Since these two deep neural networks have the same structure, multiple hidden layers are defined for each network. We use the current system state s and the next system state s′ as the input to eval_net and target_net, respectively. We use $Q_e\left(s,a,\theta \right)$ and $Q_t\left(s,a,{\theta }^{\prime }\right)$ to define the output of two deep neural networks eval_net and target_net. In the structure, in order to update the value of the weight of neuron nodes, we only continuously train the evaluation network eval_net. The target network is not trained. It periodically duplicates the weights of the neurons from the evaluation network (i.e., ${\theta }^{\prime }=\theta$ ). The loss function is described as follows, which is used to train eval_net:

(11) $\begin{array}{}\text{Loss}\left(\theta \right)=E\left({\left(y-Q_e\left(s,a,\theta \right)\right)}^2\right).\end{array}$

We use y to represent the real Q value, which can be expressed as

(12) $\begin{array}{}y=w\left(s,a,s^{\prime }\right)+ϵ\underset{a^{\prime }\in A}{\max },Q_t\left(s^{\prime },a^{\prime },{\theta }^{\prime }\right).\end{array}$

We denote the learning rate as ϵ. The idea of back-propagation is utilized to update the weight of eval_net; as a result, the neural network is trained.

The experience reply method is utilized to improve the training effect, since it can effectively eliminate the correlation among the training data. Each single experience includes the system state s, the action a, and the next system state s′, together with the reward function $w\left(s,a,s^{\prime }\right)$ . We define the experience set as $\text{ep}=\left(s,a,w\left(s,a,s^{\prime }\right),s^{\prime }\right)$ . In the algorithm, D individual experiences are saved and, in each training epoch, only D_s (with D_s < D) experiences are selected from D. As the training process is completed, target_net copies the weight of the neurons from the evaluation network (i.e., ${\theta }^{\prime }=\theta$ ). D different experiences are generated from ep, while only D_s are picked to train the evaluation network eval_net. The total number of training iterations can be denoted as U. Both evaluation network and target network share the same structure, in which the deep neural networks have N_l hidden layers.

3.2.3. Dueling Double DQN

In order to leverage the performance of DQN, which can effectively select the optimal action to charge multiple harvesters in a time-varying channel conditions, we redesign the structure of the deep neural network by using Dueling Double DQN. Doubling DQN is an advanced version of DQN which can prevent the overestimating problem appearing throughout the training [Reference Van Hasselt, Guez and Silver24]. Dueling Double DQN can efficiently solve the overestimating problem throughout the training process. In the same training epochs, Dueling Double DQN is proved to outperform the original DQN in learning efficiency.

In traditional DQN, as shown in equation (12), the target network target_net is designed to derive the cost function for a particular system state. Nevertheless, because we do not update the weight of the target network target_net in each training epoch, the training error will increase while training, hence prolonging the training procedure. In Doubling DQN, both the target network target_net and the evaluation network eval_net are used to calculate the cost functions. We use evaluation network eval_net to calculate the best action for system state s′.

(13) $\begin{array}{}y=w\left(s,a,s^{\prime }\right)+ϵ\underset{a^{\prime }\in A}{\max }{Q}_e\left(s^{\prime },\underset{a\in A}{\arg }\max Q\left(s^{\prime },a,\theta \right),{\theta }^{\prime }\right).\end{array}$

The latest research proves that the training error can be dramatically reduced using the Doubling DQN structure [Reference Van Hasselt, Guez and Silver24].

In traditional DQN, we only define the cost function Q value as the output of the deep neural network. The Dueling DQN is invented to speed up the convergence of the deep neural network by designing 2 individual streams of the output for the deep neural network. We use the output value $V\left(s,\theta ,\beta \right)$ to represent the first stream of the neural network. It denotes the cost function for a specific system state. We name the second stream of the output as advantage output $A\left(s^{\prime },a,\theta ,\alpha \right)$ , which is utilized to illustrate the advantage of using a specific action to a system state [Reference Wang, Schaul, Hessel, Van Hasselt, Lanctot and De Freitas25]. We define α and β as the parameters to correlate the output of two streams and the neural network. The cost function can be denoted as

(14) $\begin{array}{}Q\left(s,a,\theta ,\alpha ,\beta \right)=V\left(s,\theta ,\beta \right)+\left(A\left(s^{\prime },a,\theta ,\alpha \right)-\frac{1}{\left(A\right)}\sum _{a^{\prime }}A\left(s^{\prime },a,\theta ,\alpha \right)\right).\end{array}$

The latest research proves that Dueling DQN can speeds up the training procedure by efficiently annihilating the additional freedom while training the deep neural network [Reference Wang, Schaul, Hessel, Van Hasselt, Lanctot and De Freitas25].

3.3. Area Division Deep Reinforcement Learning

In this paper, the optimization problem can be seen as calculating the optimal close-loop path which generates the maximum accumulated reward. However, the traditional DQN shows the difficulty converging to the optimal path because of the complicated experimental field. In order to leverage the performance of traditional DQN, we invent an AD-DQN in this paper. At first, the experimental field is divided into multiple separate parts. DQN is run on each part individually to obtain the optimal path for the robot, respectively. Finally, the entire close-loop path is formulated using the path on each part. In area division, the whole area is defined as W. The whole area is divided at multiple specific locations. $p_i\in P$ .

The criterion to pick p_i is finding the squares, which exist in more than one effective charging area of the IoT devices.

(15) $\begin{array}{c}\forall p_i\in P,p_i\in ef{f}_m,\\ p_i\in ef{f}_n,\\ m,n=\mathrm{0,1},\dots ,K-1,m\ne n.\end{array}$

For each p_i, we define $N_i=\left(p_i\right)$ . We define set $K_e=\left(o_{\arg p_i\in ef{f}_j,j=\mathrm{0,1},\dots ,K-1}\right)$ . In the clockwise direction, we find that the IoT device o_i has the shortest distance to p_i, and then add both o_i and the effective charging area of o_i to N_i. The new area can be expressed as

(16) $\begin{array}{}{N}_i=N_i\cup \left(o_i\right)\cup \left(ef{f}_i\right).\end{array}$

Next, we find the IoT device having the shortest distance to the IoT device o_i that is just added to set N_i, and then add both the new IoT device and the effective charging area of it to N_i. Iteratively, we find that all the IoT devices besides the ones in $K_e$ are included in one N_i. Finally, classify all the rest squares to the nearest N_i. $\left(N_i\right)=W$ .

In each area, the DQN is run to determine the optimal path for the robot. In each area, the starting point is the same as the position of p_i; the end point is one of the effective charging squares of the furthest IoT device from the starting point in the same area. After the optimal path is calculated for each individual area, the close-loop optimal path for the entire area can be synthesized. The algorithm is shown in Algorithm 1.

1. (i) Define $E=\left(ef{f}_k,k=\mathrm{0,1},\dots ,K-1\right)$ . Among E, find all the area division points p_i by $\left(p_i\right)=\left(pos\left(h,v\right)|pos\left(h,v\right)\in ef{f}_m,pos\left(h,v\right)\in ef{f}_n,m,n\in K\right)$

2. (ii) The number of area divison points is defined as $\left(P\right)$ .

3. (iii) $i=1,\dots ,\left(P\right)$ . The number of the area to be divided is $\left(P\right)+1$ .

4. (iv) $K_e=\left(o_{\arg p_i\in ef{f}_j,j=\mathrm{0,1},\dots ,K-1}\right)$ .

5. (v) $fori=1,\dots ,\left(P\right)$ :

6. (vi) $r_1=p_i$ . $r_2=p_i$ .

7. (vii) $while\nexists o_g\in N_i,o_g\in N_{ℐ\backslash \left(i\right)}$

8. (viii) $ifi<=\left(P\right)$

9. (ix) In the clockwise direction, find the the IoT devices, that has the shortest distance to r₁. The order of the IoT device is: $g=\arg {\min }_{o_i\notin K_e}\left(o_i-r_1\right)$ . $N_i$ is updated as: $N_i=N_i\cup \left(o_g\right)\cup \left(ef{f}_g\right)$ . $r_1=o_g$ .

10. (x) else

11. (xi) In the counterclockwise direction, find the the IoT devices, that has the shortest distance to r₂. The order of the IoT device is: $g=\arg {\min }_{o_i\notin K_e}\left(o_i-r_2\right)$ . N_i is updated as: $N_i=N_i\cup \left(o_g\right)\cup \left(ef{f}_g\right)$ . $r_2=o_g$ .

12. (xii) en d

13. (xiii) en dw hile

14. (xiv) en d

15. (xv) $fori=1,\dots ,\left(P\right)$ :

16. (xvi) Define set $W_i$

17. (xvii) $W_i=N_i\cup \left(pos\left(h,v\right)\left({\arg }_{h,v}{\min }_{pos\left(h,v\right)\in W}\right)pos\left(h,v\right)-o_k|,k\notin K_e\right)$

18. (xviii) en d

19. (xix) $fori=\mathrm{1,2},\dots ,\left(P\right)+1$ :

20. (xx) $forj=\mathrm{1,2},\dots ,\left(J\right)$ :

21. (xxi) The starting point is defined as p_i. The end point is defined as $e_j\in ef{f}_c,j\in J$ .

22. (xxii) The weight of the neuron nodes θ are randomly generated for the eval_net and the weights are copied by $\text{target}\_\text{net}{\theta }^{\prime }=\theta$ . u = 1. D = d = 1.

23. (xxiii) $whileu<Us=s^0$ . $t=1$ .

24. (xxiv) A probability is generated as a numerical parameter $p\in \left(\mathrm{0,1}\right)$ .

25. (xxv) $ifD>200andp\ge {ϵ}_{ch}$

26. (xxvi) $a={\max }_{a\in A}Q\left(s,a\right)$

27. (xxvii) else

28. (xxviii) Randomly choose the action from action set A.

29. (xxix) en d

30. (xxx) $while{s}^{\prime }\ne s^T$

31. (xxxi) The state transit into s′ after taking the action. d = d + 1. $\text{ep}\left(d\right)=\left(s,a,w\left(s,a,s^{\prime }\right),s^{\prime }\right)$ . Suppose D keeps unchanged if it goes over the experience pool’s limitation, d = 1; otherwise, $\text{\$}D=d\text{\$}$ . $t=t+1$ . $s=s^{\prime }$ . After enough data has been collected in experience pool, eval_net is trained using D of D_s experiences. Minimize the loss function Loss(θ) using Back-propagation. target_net copies the weight from eval_net periodically.

32. (xxxii) en dw hile

33. (xxxiii) en dw hile

34. (xxxiv) The optimal path of the entire test field is synthesized with the optimal path in each $W_i$ .