![References [1] Richard S. Sutton and Andrew G. Barto, Reinforcement Learning—An Introduction [2] Chengan Guo, Applications of Reinforcement Learning in Sequence Detection and Network Routing [3] Simon Haykin, Neural Networks– A Comprehensive Foundation](https://image.slidesharecdn.com/applying-reinforcement-learning-for-network-routing4818/85/Applying-Reinforcement-Learning-for-Network-Routing-29-320.jpg)
Applying Reinforcement Learning for Network Routing
- 1. Application of Reinforcement Learning in Network Routing By Chaopin Zhu
- 2. Machine Learning Supervised Learning Unsupervised Learning Reinforcement Learning
- 3. Supervised Learning Feature: Learning with a teacher Phases Training phase Testing phase Application Pattern recognition Function approximation
- 4. Unsupervised Leaning Feature Learning without a teacher Application Feature extraction Other preprocessing
- 5. Reinforcement Learning Feature: Learning with a critic Application Optimization Function approximation
- 6. Elements of Reinforcement Learning Agent Environment Policy Reward function Value function Model of environment (optional)
- 8. Markov Decision Process (MDP) Definition: A reinforcement learning task that satisfies the Markov property Transition probabilities
- 9. An Example of MDP
- 10. Markov Decision Process (cont.) Parameters Value functions
- 11. Elementary Methods for Reinforcement Learning Problem Dynamic programming Monte Carlo Methods Temporal-Difference Learning
- 13. Dynamic Programming Methods Policy evaluation Policy improvement
- 14. Dynamic Programming (cont.) E ---- policy evaluation I ---- policy improvement Policy Iteration Value Iteration
- 15. Monte Carlo Methods Feature Learning from experience Do not need complete transition probabilities Idea Partition experience into episodes Average sample return Update at episode-by-episode base
- 16. Temporal-Difference Learning Features (Combination of Monte Carlo and DP ideas) Learn from experience (Monte Carlo) Update estimates based in part on other learned estimates (DP) TD( ) algorithm seemlessly integrates TD and Monte Carlo Methods
- 17. TD(0) Learning Initialize V(x) arbitrarily to the policy to be evaluated Repeat (for each episode): Initialize x Repeat (for each step of episode) a action given by for x Take action a; observe reward r and next state x’ x x’ until x is terminal
- 18. Q-Learning Initialize Q(x,a) arbitrarily Repeat (for each episode) Initialize x Repeat (for each step of episode): Choose a from x using policy derived from Q Take action a, observe r, x’ x x’ until x is terminal
- 19. Q-Routing Q x (y,d)----estimated time that a packet would take to reach the destination node d from current node x via x’s neighbor node y T y (d) ------y’s estimate for the time remaining in the trip q y ---------queuing time in node y T xy --------transmission time between x and y
- 20. Algorithm of Q-Routing Set initial Q-values for each node Get the first packet from the packet queue of node x Choose the best neighbor node and forward the packet to node by Get the estimated value from node Update Go to 2.
- 22. Network Model
- 24. Node Model
- 26. Initialization/ Termination Procedures Initilization Initialize and / or register global variable Initialize routing table Termination Destroy routing table Release memory
- 27. Arrival Procedure Data packet arrival Update routing table Route it with control information or destroy the packet if it reaches the destination Control information packet arrival Update routing table Destroy the packet
- 28. Departure Procedure Set all fields of the packet Get a shortest route Send the packet according to the route
- 29. References [1] Richard S. Sutton and Andrew G. Barto, Reinforcement Learning—An Introduction [2] Chengan Guo, Applications of Reinforcement Learning in Sequence Detection and Network Routing [3] Simon Haykin, Neural Networks– A Comprehensive Foundation