Markov Chains: Theory and Applications, Volume 52 in the Handbook of Statistics series, highlights new advances in the field, with this new volume presenting interesting chapters on topics such as Markov Chain Estimation, Approximation, and Aggregation for Average Reward Markov Decision Processes and Reinforcement Learning, Ladder processes: symmetric functions and semigroups, Continuous-time Markov Chains and Models: Study via Forward Kolmogorov System, Analysis of Data Following Finite-State Continuous-Time Markov Chains, Computational applications of poverty measurement through Markov model for income classes, and more.
Other sections cover Estimation and calibration of continuous time Markov chains, Additive High-Order Markov Chains, The role of the random-product technique in the theory of Markov chains on a countable state space., On estimation problems based on type I Longla copulas, and Long time behavior of continuous time Markov chains.
1. 1 Introduction - p. 30 2. 2 Symmetric functions - p. 34 3. 3 Semigroups - p. 61 4. References - p. 85 5. Chapter 3: Long time behaviour of continuous time Markov chains - p. 87 6. Abstract - p. 87 7. 1 Introduction - p. 88 8. 2 Weighted graphs, Q-matrices, Markov chains - p. 89 9. 3 Function theoretic characterizations - p. 105 10. 4 Estimates for the heat equation - p. 108 11. 5 Discrete Ricci curvature and stochastic completeness - p. 113 12. 6 Volume growth criteria for stochastic completeness - p. 116 13. 7 Volume growth criteria for upper escape rate - p. 119 14. References - p. 123 15. Chapter 4: Generalized steady-state distributions of somewhat stochastic matrices with eigenvalues 1, λ2, λ3, ⋯ , λn and modulus ∣λi∣ < 1 for i = 2, 3, ⋯ , n: Duality applications for calculating Gambler's ruin problems - p. 127 16. Abstract - p. 128 17. 1 Introduction and summary - p. 128 18. 2 Background material - p. 132 19. 3 Visualizing low dimensional, somewhat stochastic birth-death transition matrices that have steady-state distributions - p. 139 20. 4 Applications of finite dimensional siegmund duality and algebraic power duality - p. 164 21. References - p. 176 22. Chapter 5: Continuous-time Markov chains and models: Bounding via forward Kolmogorov system - p. 179 23. Abstract - p. 179 24. 1 Introduction - p. 180 25. 2 Forward Kolmogorov system - p. 182 26. 3 Logarithmic norm as the main estimation method - p. 184 27. 4 Main classes of Markov chains and corresponding types of transformations - p. 186 28. 5 Birth-death processes, finite state space, homogeneous situation - p. 192 29. 6 Birth-death processes, general situation - p. 195 30. 7 Numerical example. Queue-length process for the Mt∣Mt∣3 queue - p. 202 31. References - p. 207 32. Part II: Applications - p. 211 33. Chapter 6: Markov chain estimation, approximation, and aggregation for average reward Markov decision processes and reinforcement learning - p. 213 34. Abstract - p. 214 35. 1 Introduction - p. 214 36. 2 Prologue: Markov reward processes - p. 217 37. 3 Markov decision processes - p. 220 38. 4 Markov chain and MDP approximation - p. 226 39. 5 Estimating the stationary distribution from trajectories - p. 236 40. 6 Markov chain and MDP aggregation - p. 238 41. 7 Beyond aggregation: (εr, εp)-structured MDPs and restless bandits - p. 243 42. References - p. 248 43. Chapter 7: The role of the random-product technique in the theory of Markov chains on a countable state space - p. 251 44. Abstract - p. 251 45. 1 Introduction - p. 251 46. 2 Discrete-time Markov chains - p. 252 47. 3 Continuous-time Markov chains - p. 262 48. 4 Stochastic networks - p. 275 49. References - p. 281 50. Chapter 8: Computational applications of poverty measurement through Markov model for income classes - p. 283 51. Chapter 9: Additive high-order Markov chains - p. 311 52. Index - p. 339
Markov Chains: Theory and Applications, Volume 52 in the Handbook of Statistics series, highlights new advances in the field, with this new volume presenting interesting chapters on topics such as Markov Chain Estimation, Approximation, and Aggregation for Average Reward Markov Decision Processes and Reinforcement Learning, Ladder processes: symmetric functions and semigroups, Continuous-time Markov Chains and Models: Study via Forward Kolmogorov System, Analysis of Data Following Finite-State Continuous-Time Markov Chains, Computational applications of poverty measurement through Markov model for income classes, and more.
Other sections cover Estimation and calibration of continuous time Markov chains, Additive High-Order Markov Chains, The role of the random-product technique in the theory of Markov chains on a countable state space., On estimation problems based on type I Longla copulas, and Long time behavior of continuous time Markov chains.
1. 1 Introduction - p. 30 2. 2 Symmetric functions - p. 34 3. 3 Semigroups - p. 61 4. References - p. 85 5. Chapter 3: Long time behaviour of continuous time Markov chains - p. 87 6. Abstract - p. 87 7. 1 Introduction - p. 88 8. 2 Weighted graphs, Q-matrices, Markov chains - p. 89 9. 3 Function theoretic characterizations - p. 105 10. 4 Estimates for the heat equation - p. 108 11. 5 Discrete Ricci curvature and stochastic completeness - p. 113 12. 6 Volume growth criteria for stochastic completeness - p. 116 13. 7 Volume growth criteria for upper escape rate - p. 119 14. References - p. 123 15. Chapter 4: Generalized steady-state distributions of somewhat stochastic matrices with eigenvalues 1, λ2, λ3, ⋯ , λn and modulus ∣λi∣ < 1 for i = 2, 3, ⋯ , n: Duality applications for calculating Gambler's ruin problems - p. 127 16. Abstract - p. 128 17. 1 Introduction and summary - p. 128 18. 2 Background material - p. 132 19. 3 Visualizing low dimensional, somewhat stochastic birth-death transition matrices that have steady-state distributions - p. 139 20. 4 Applications of finite dimensional siegmund duality and algebraic power duality - p. 164 21. References - p. 176 22. Chapter 5: Continuous-time Markov chains and models: Bounding via forward Kolmogorov system - p. 179 23. Abstract - p. 179 24. 1 Introduction - p. 180 25. 2 Forward Kolmogorov system - p. 182 26. 3 Logarithmic norm as the main estimation method - p. 184 27. 4 Main classes of Markov chains and corresponding types of transformations - p. 186 28. 5 Birth-death processes, finite state space, homogeneous situation - p. 192 29. 6 Birth-death processes, general situation - p. 195 30. 7 Numerical example. Queue-length process for the Mt∣Mt∣3 queue - p. 202 31. References - p. 207 32. Part II: Applications - p. 211 33. Chapter 6: Markov chain estimation, approximation, and aggregation for average reward Markov decision processes and reinforcement learning - p. 213 34. Abstract - p. 214 35. 1 Introduction - p. 214 36. 2 Prologue: Markov reward processes - p. 217 37. 3 Markov decision processes - p. 220 38. 4 Markov chain and MDP approximation - p. 226 39. 5 Estimating the stationary distribution from trajectories - p. 236 40. 6 Markov chain and MDP aggregation - p. 238 41. 7 Beyond aggregation: (εr, εp)-structured MDPs and restless bandits - p. 243 42. References - p. 248 43. Chapter 7: The role of the random-product technique in the theory of Markov chains on a countable state space - p. 251 44. Abstract - p. 251 45. 1 Introduction - p. 251 46. 2 Discrete-time Markov chains - p. 252 47. 3 Continuous-time Markov chains - p. 262 48. 4 Stochastic networks - p. 275 49. References - p. 281 50. Chapter 8: Computational applications of poverty measurement through Markov model for income classes - p. 283 51. Chapter 9: Additive high-order Markov chains - p. 311 52. Index - p. 339
