Bellman expectation equation for MDP value functions

The Bellman expectation equation for MDPs provides a way to calculate a policy-indexed state-value function $v_{\pi }$. We can compute $v_{\pi }$ by doing a one-step lookahead on the action values of all possible $a$ and the value of the policy function for that action at the current state. The sum obtained should be a summary of how advantageous it is at the state—if all actions have high action values at this state then the state itself must also be advantageous to be in. We can then expand the definition of $q_{\pi }$ to define $v_{\pi }$ recursively.

$$\begin{array}{rl}{v}_{\pi }(s)& ={\mathbb{E}}_{\pi }\left[R_{t+1}+\gamma v_{\pi }(S_{t+1}|S_t=s)\right]\\ & =\sum _{a\in \mathcal{A}}\pi (a|s)q_{\pi }(s,a)\\ & =\sum _{a\in \mathcal{A}}\pi (a|s)\left({\mathcal{R}}_s^a+\gamma \sum _{s^{\prime }\in S}{\mathcal{P}}_{s{s}^{\prime }}^a{v}_{\pi }(s^{\prime })\right)\end{array}$$

We can also compute the policy-indexed action-value function $q_{\pi }$ by doing a one-step lookhead, this time using the discounted state-value of all subsequent states. Expanding the definition of $v_{\pi }$ produces a recursively defined $q_{\pi }$.

$$\begin{array}{rl}{q}_{\pi }(s,a)& ={\mathbb{E}}_{\pi }[G_t|S_t=s,A_t=a]\\ & ={\mathcal{R}}_s^a+\gamma \sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^a{v}_{\pi }(s^{\prime })\\ & ={\mathcal{R}}_s^a+\gamma \sum _{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}^a\sum _{a^{\prime }\in \mathcal{A}}\pi (a^{\prime }|s^{\prime })q_{\pi }(s^{\prime },a^{\prime })\end{array}$$

Say if we decide to calculate $v_{\pi }$, we can convert the value function to matrix and turn it into a closed-form solution:

$$\begin{array}{rl}{v}_{\pi }& ={\mathcal{R}}^{\pi }+\gamma {\mathcal{P}}^{\pi }{v}_{\pi }\\ & =(I-\gamma {\mathcal{P}}^{\pi }{)}^{-1}{\mathcal{R}}^{\pi }\end{array}$$