Assignments

HW1

1. Work out the details of gradient ascent on the preference function found on pages 38+. Justify each step.

HW2

1. Derive a formula for $v_\pi$ in terms of $q_\pi$ and $\pi$.
2. Derive a formula for $q_\pi$ in terms of $v_\pi$ and $p$.
3. Use these two relations to derive recursions for $v_\pi$ and $q_\pi$.
4. Substute the greedy policy into these recursions to derive recursions for $v_\ast$ and $q_\ast$.

HW3

1. Work the gamblers problem in example 4.2 on page 84. Use value iteration. Turn in your code and plots of the value function and the policy like those in the book.
2. Exercise 4.9 on page 84.
3. Exercise 4.10 on page 84.

HW 4

1. Write code for policy iteration for the grid world example 3.5 and compare your results to those in example 3.8 on page 65. For this part, use the state value function $v_\pi$ in your policy iteration.
2. Repeat using $q_\pi$ in your policy iteration.