Reinforcement learning (RL) has driven machine learning from basic data-fitting to the new era of learning and planning through interacting with complex environments. Equipped with deep learning, RL has achieved tremendous successes in many applications, including autonomous driving, recommendation systems, wireless communications, robotics, gaming, etc. The success of RL is largely based on the foundational developments of RL algorithms, which were not thoroughly understood until recently, especially their finite-time convergence rates and sample complexities. This tutorial will provide a comprehensive overview of the recent advances on theoretical understanding of fundamental RL algorithms, which leverage stochastic approximation/optimization theory and exploit the Markovian structures of RL problems. The tutorial will also introduce some advanced RL algorithms and their recent developments.