Jekyll2020-11-25T18:32:34+00:00https://brandinho.github.io/feed.xmlBrandon Da SilvaMachine Learning BlogBrandon Da Silvabrandasilva9@gmail.comA Bayesian Perspective on Q-Learning2020-10-21T00:00:00+00:002020-10-21T00:00:00+00:00https://brandinho.github.io/bayes-q<p>Please redirect to the following link: <a href="https://brandinho.github.io/bayesian-perspective-q-learning/">HERE</a></p>
<div id="disqus_thread"></div>
<script>
/**
* RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS.
* LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables*/
/*
var disqus_config = function () {
this.page.url = https://brandinho.github.io; // Replace PAGE_URL with your page's canonical URL variable
this.page.identifier = /bayes-q; // Replace PAGE_IDENTIFIER with your page's unique identifier variable
};
*/
(function() { // DON'T EDIT BELOW THIS LINE
var d = document, s = d.createElement('script');
s.src = 'https://brandinho-github-io.disqus.com/embed.js';
s.setAttribute('data-timestamp', +new Date());
(d.head || d.body).appendChild(s);
})();
</script>
<noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>Brandon Da Silvabrandasilva9@gmail.comReinforcement LearningDeep Learning Presentation at Quant World 20182019-04-02T00:00:00+00:002019-04-02T00:00:00+00:00https://brandinho.github.io/quantworld<iframe width="560" height="315" src="https://www.youtube.com/embed/Zy6mp_pRD94" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<div id="disqus_thread"></div>
<script>
/**
* RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS.
* LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables*/
/*
var disqus_config = function () {
this.page.url = https://brandinho.github.io; // Replace PAGE_URL with your page's canonical URL variable
this.page.identifier = /quantworld; // Replace PAGE_IDENTIFIER with your page's unique identifier variable
};
*/
(function() { // DON'T EDIT BELOW THIS LINE
var d = document, s = d.createElement('script');
s.src = 'https://brandinho-github-io.disqus.com/embed.js';
s.setAttribute('data-timestamp', +new Date());
(d.head || d.body).appendChild(s);
})();
</script>
<noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>Brandon Da Silvabrandasilva9@gmail.comDeep LearningThe Math of Loss Functions2019-02-06T00:00:00+00:002019-02-06T00:00:00+00:00https://brandinho.github.io/cost-function-gradients<h2 id="overview">Overview</h2>
<p>In this post we will go over some of the math associated with popular supervised learning loss functions. Specifically, we are going to focus on linear, logistic, and softmax regression. We show that the derivatives used for parameter updates are the same for all of those models! Most people probably won’t care because they use automatic differentiation libraries like TensorFlow, but I find it cool.</p>
<p>Each section in this blog is going to start out with a few lines of math that explain how the model works. Then we are going to dive into the derivative of the loss function with respect to \(z\). I go into a lot of detail when calculating the derivatives - probably more than necessary. I do this because I want everybody to completely understand how the math works.</p>
<p>We will layout the math for each of the models in the following way:</p>
<ul>
<li>Define a linear equation which will be denoted \(z\)</li>
<li>Define an activation function if there is any</li>
<li>Define a prediction (transform of the linear equation) which will be denoted \(\hat{y}\)</li>
<li>Define a loss function which will be denoted \(\mathcal{L}\)</li>
</ul>
<p>I am laying it out this way to maintain consistency between each of the models. Keep this in mind when you realize how silly it is to move from \(z\) to \(\hat{y}\) for linear regression.</p>
<p>Before diving into the math it is important to note that I shape the input \(X\) as \((\text{# instances}, \text{# features})\), and the weight matrix \(w\) as \((\text{# features}, \text{# outputs})\). This is an important distinction because the equations would look slightly different if you used \(X^\boldsymbol{\top}\), which a lot of people use for some reason. I personally dislike using rows for features and columns for instances, but if it floats your boat then go for it (you’ll just need to make minor changes to the math notation).</p>
<h2 id="linear-regression">Linear Regression</h2>
<p>To start, let’s define our core functions for linear regression:</p>
\[\begin{align*}
&\text{Linear Equation}: &&z = Xw + b \\[1.5ex]
&\text{Activation Function}: &&\text{None} \\[1.5ex]
&\text{Prediction}: &&\hat{y} = z \\[0.5ex]
&\text{Loss Function}: &&\mathcal{L} = \frac{1}{2}(\hat{y} - y)^2
\end{align*}\]
<p>We can also define the functions in python code:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">weights</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">normal</span><span class="p">(</span><span class="n">size</span> <span class="o">=</span> <span class="n">n_features</span><span class="p">).</span><span class="n">reshape</span><span class="p">(</span><span class="n">n_features</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">bias</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">def</span> <span class="nf">linear_regression_inference</span><span class="p">(</span><span class="n">inputs</span><span class="p">):</span>
<span class="k">return</span> <span class="n">np</span><span class="p">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">weights</span><span class="p">)</span> <span class="o">+</span> <span class="n">bias</span>
<span class="k">def</span> <span class="nf">calculate_error</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
<span class="c1">### Mean Squared Error (I know I'm not taking an average, but you get the point)
</span> <span class="n">y_hat</span> <span class="o">=</span> <span class="n">linear_regression_inference</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="k">return</span> <span class="mf">0.5</span> <span class="o">*</span> <span class="p">(</span><span class="n">yhat</span> <span class="o">-</span> <span class="n">y</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span>
</code></pre></div></div>
<p>We are interested in calculating the derivative of the loss with respect to \(z\). Throughout this post, we will do this by applying the chain rule:</p>
\[\frac{\partial \mathcal{L}}{\partial z} = \frac{\partial \mathcal{L}}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial z}\]
<p>First we will calculate the partial derivative of the loss with respect to our prediction:</p>
\[\frac{\partial \mathcal{L}}{\partial \hat{y}} = \hat{y} - y\]
<p>Next, although silly, we calculate the partial derivative of our prediction with respect to the linear equation. Of course since the linear equation is our prediction (since we’re doing linear regression), the partial derivative is just 1:</p>
\[\frac{\partial \hat{y}}{\partial z} = 1\]
<p>When we combine them together, the derivative of the loss with respect to the linear equation is:</p>
\[\frac{\partial \mathcal{L}}{\partial z} = \hat{y} - y\]
<p>Although this was pretty straight forward, the next two sections are a bit more involved, so buckle up. Get ready to have your mind blown as you learn that \(\frac{\partial \mathcal{L}}{\partial z} = (\hat{y} - y)\) for logistic regression and softmax regression as well!</p>
<h2 id="logistic-regression">Logistic Regression</h2>
<p>Like linear regression, we will define the core functions for logistic regression:</p>
\[\begin{align*}
&\text{Linear Equation}: &&z = Xw + b \\[0.5ex]
&\text{Activation Function}: &&\sigma(z) = \frac{1}{1 + e^{-z}} \\[0.5ex]
&\text{Prediction}: &&\hat{y} = \sigma(z) \\[1.5ex]
&\text{Loss Function}: &&\mathcal{L} = -(y\log\hat{y} + (1-y)\log(1-\hat{y}))
\end{align*}\]
<p>We can also define the functions in python code:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">weights</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">normal</span><span class="p">(</span><span class="n">size</span> <span class="o">=</span> <span class="n">n_features</span><span class="p">).</span><span class="n">reshape</span><span class="p">(</span><span class="n">n_features</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">bias</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">def</span> <span class="nf">sigmoid</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">1</span> <span class="o">/</span> <span class="p">(</span><span class="mi">1</span> <span class="o">+</span> <span class="n">np</span><span class="p">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="n">x</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">logistic_regression_inference</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
<span class="k">return</span> <span class="n">sigmoid</span><span class="p">(</span><span class="n">np</span><span class="p">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">weights</span><span class="p">)</span> <span class="o">+</span> <span class="n">bias</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">calculate_error</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
<span class="c1">### Binary Cross-Entropy
</span> <span class="n">y_hat</span> <span class="o">=</span> <span class="n">logistic_regression_inference</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="k">return</span> <span class="o">-</span><span class="p">(</span><span class="n">y</span> <span class="o">*</span> <span class="n">np</span><span class="p">.</span><span class="n">log</span><span class="p">(</span><span class="n">y_hat</span><span class="p">)</span> <span class="o">+</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">y</span><span class="p">)</span> <span class="o">*</span> <span class="n">np</span><span class="p">.</span><span class="n">log</span><span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">y_hat</span><span class="p">))</span>
</code></pre></div></div>
<p><strong>NOTE</strong>: When calculating the error for logistic regression we usually add a small constant inside the \(\log\) calculation to prevent taking the log of 0.</p>
<p>Again, we use the chain rule to calculate the partial derivative of interest:</p>
\[\frac{\partial \mathcal{L}}{\partial z} = \frac{\partial \mathcal{L}}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial z}\]
<p>The partial derivative of the loss with respect to our prediction is pretty simple to calculate:</p>
\[\frac{\partial \mathcal{L}}{\partial \hat{y}} = -\frac{y}{\hat{y}} + \frac{1-y}{1-\hat{y}}\]
<p>Next we will calculate the derivative of our prediction with respect to the linear equation. We can use a little algebra to move things around and get a nice expression for the derivative:</p>
\[\begin{align*}
\frac{\partial \hat{y}}{\partial z} &= \frac{\partial}{\partial z}\left[\frac{1}{1 + e^{-z}}\right] \\[0.75ex]
&= \frac{e^{-z}}{(1 + e^{-z})^2} \\[0.75ex]
&= \frac{1 + e^{-z} - 1}{(1 + e^{-z})^2} \\[0.75ex]
&= \frac{1 + e^{-z}}{(1 + e^{-z})^2} - \frac{1}{(1 + e^{-z})^2} \\[0.75ex]
&= \frac{1}{1 + e^{-z}} - \frac{1}{(1 + e^{-z})^2} \\[0.75ex]
&= \frac{1}{1 + e^{-z}} \left(1 - \frac{1}{1 + e^{-z}}\right) \\[0.75ex]
&= \hat{y}(1 - \hat{y})
\end{align*}\]
<p>Isn’t that awesome?! Anyways, enough of my love for math, let’s move on. Now we’ll combine the two partial derivatives to get our final expression for the derivative of the loss with respect to the linear equation.</p>
\[\begin{align*}
\frac{\partial \mathcal{L}}{\partial z} &= \left(-\frac{y}{\hat{y}} + \frac{1-y}{1-\hat{y}}\right)\hat{y}(1 - \hat{y}) \\[0.75ex]
&= -\frac{y}{\hat{y}}\hat{y}(1 - \hat{y}) + \frac{1-y}{1-\hat{y}}\hat{y}(1 - \hat{y}) \\[0.75ex]
&= -y(1 - \hat{y}) + (1-y)\hat{y} \\[0.75ex]
&= -y + y\hat{y} + \hat{y} - y\hat{y} \\[0.75ex]
&= \hat{y} - y
\end{align*}\]
<p>Would you look at that, it’s the exact same!! If you think that is cool (which you should), then just wait for the next section where we go through softmax regression.</p>
<h2 id="softmax-regression">Softmax Regression</h2>
<p>Once again, we will define the core functions for softmax regression:</p>
\[\begin{align*}
&\text{Linear Equation}: &&z = Xw + b \\[0.5ex]
&\text{Activation Function}: &&\varphi(z_i) = \frac{e^{z_i}}{\sum_n e^{z_n}} \\[0.5ex]
&\text{Prediction}: &&\hat{y_i} = \varphi(z_i) \\[1.5ex]
&\text{Loss Function}: &&\mathcal{L} = -\sum_i y_i\log\hat{y_i}
\end{align*}\]
<p>We can also define the functions in python code:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">weights</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">normal</span><span class="p">(</span><span class="n">size</span> <span class="o">=</span> <span class="n">n_features</span><span class="p">).</span><span class="n">reshape</span><span class="p">(</span><span class="n">n_features</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">bias</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">def</span> <span class="nf">softmax</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
<span class="k">return</span> <span class="n">np</span><span class="p">.</span><span class="n">exp</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">/</span> <span class="n">np</span><span class="p">.</span><span class="nb">sum</span><span class="p">(</span><span class="n">np</span><span class="p">.</span><span class="n">exp</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">1</span><span class="p">).</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">softmax_regression_inference</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
<span class="k">return</span> <span class="n">softmax</span><span class="p">(</span><span class="n">np</span><span class="p">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="bp">self</span><span class="p">.</span><span class="n">weights</span><span class="p">)</span> <span class="o">+</span> <span class="bp">self</span><span class="p">.</span><span class="n">bias</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">calculate_error</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
<span class="c1">### Categorical Cross-Entropy
</span> <span class="n">y_hat</span> <span class="o">=</span> <span class="n">softmax_regression_inference</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="k">return</span> <span class="o">-</span><span class="n">np</span><span class="p">.</span><span class="n">mean</span><span class="p">(</span><span class="n">np</span><span class="p">.</span><span class="nb">sum</span><span class="p">(</span><span class="n">y</span> <span class="o">*</span> <span class="n">np</span><span class="p">.</span><span class="n">log</span><span class="p">(</span><span class="n">yhat</span><span class="p">),</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">1</span><span class="p">))</span>
</code></pre></div></div>
<p><strong>NOTE</strong>: With softmax regression, we also typically add a small constant inside \(\log\) for the same reason as logistic regression.</p>
<p>For the last time, we will restate the partial derivative using the chain rule:</p>
\[\frac{\partial \mathcal{L}}{\partial z_j} = \frac{\partial \mathcal{L}}{\partial \hat{y_i}} \frac{\partial \hat{y_i}}{\partial z_j}\]
<p>Let’s calculate the first partial derivative of the loss with respect to our prediction:</p>
\[\frac{\partial \mathcal{L}}{\partial \hat{y_i}} = -\sum_i \frac{y_i}{\hat{y_i}}\]
<p>That was pretty easy! Now let’s tackle the monster… the partial derivative of our prediction with respect to the linear equation:</p>
\[\frac{\partial \hat{y_i}}{\partial z_j} = \frac{\sum_n e^{z_n} \frac{\partial}{\partial z_j}[e^{z_i}] - e^{z_i} \frac{\partial}{\partial z_j}\left[\sum_n e^{z_n}\right]}{\left(\sum_n e^{z_n}\right)^2}\]
<p>It is important to realize that we need to break this down into two parts. The first is when \(i = j\) and the second is when \(i \neq j\).</p>
<p>if \(i = j\):</p>
\[\begin{align*}
\frac{\partial \hat{y_i}}{\partial z_j} &= \frac{e^{z_j}\sum_n e^{z_n} - e^{z_i}e^{z_j}}{\left(\sum_n e^{z_n}\right)^2} \\[0.75ex]
&= \frac{e^{z_i}\sum_n e^{z_n}}{\left(\sum_n e^{z_n}\right)^2} - \frac{e^{z_i}e^{z_j}}{\left(\sum_n e^{z_n}\right)^2} \\[0.75ex]
&= \frac{e^{z_i}}{\sum_n e^{z_n}} - \frac{e^{z_i}e^{z_j}}{\left(\sum_n e^{z_n}\right)^2} \\[0.75ex]
&= \frac{e^{z_i}}{\sum_n e^{z_n}} - \frac{e^{z_i}}{\sum_n e^{z_n}} \frac{e^{z_j}}{\sum_n e^{z_n}} \\[0.75ex]
&= \frac{e^{z_i}}{\sum_n e^{z_n}} \left(1 - \frac{e^{z_j}}{\sum_n e^{z_n}}\right) \\[0.75ex]
&= \hat{y_i}(1 - \hat{y_j})
\end{align*}\]
<p>if \(i \neq j\):</p>
\[\begin{align*}
\frac{\partial \hat{y_i}}{\partial z_j} &= \frac{0 - e^{z_i}e^{z_j}}{\left(\sum_n e^{z_n}\right)^2} \\[0.75ex]
&= - \frac{e^{z_i}}{\sum_n e^{z_n}} \frac{e^{z_j}}{\sum_n e^{z_n}} \\[0.75ex]
&= - \hat{y_i}\hat{y_j}
\end{align*}\]
<p>We can therefore combine them as follows:</p>
\[\frac{\partial \mathcal{L}}{\partial z_j} = - \hat{y_i}(1 - \hat{y_j})\frac{y_i}{\hat{y_i}} - \sum_{i \neq j} \frac{y_i}{\hat{y_i}}(-\hat{y}_i\hat{y_j})\]
<p>The left side of the equation is where \(i = j\), while the right side is where \(i \neq j\). You will notice that we can cancel out a few terms, so the equation now becomes:</p>
\[\frac{\partial \mathcal{L}}{\partial z_j} = - y_i(1 - \hat{y_j}) + \sum_{i \neq j} y_i\hat{y_j}\]
<p>These next few steps trip some people out, so pay close attention. The first thing we’re going to do is change the subscript on the left side from \(y_i\) to \(y_j\) since \(i = j\) for that part of the equation:</p>
\[\frac{\partial \mathcal{L}}{\partial z_j} = - y_j(1 - \hat{y_j}) + \sum_{i \neq j} y_i\hat{y_j}\]
<p>Next, we are going to multiply out the left side of the equation to get:</p>
\[\frac{\partial \mathcal{L}}{\partial z_j} = - y_j + y_j\hat{y_j} + \sum_{i \neq j} y_i\hat{y_j}\]
<p>We will then factor out \(\hat{y_j}\) to get:</p>
\[\frac{\partial \mathcal{L}}{\partial z_j} = - y_j + \hat{y_j}\left(y_j + \sum_{i \neq j} y_i\right)\]
<p>This is where the magic happens. We realize that inside the bracket \(y_j\) can become \(y_i\) since it is from the left side of the equation. Since \(y\) is a one-hot encoded vector:</p>
\[y_j + \sum_{i \neq j} y_i = 1\]
<p>So our final partial derivative equals:</p>
\[\frac{\partial \mathcal{L}}{\partial z_j} = \hat{y_j} - y_j = \hat{y} - y\]
<h2 id="partial-derivative-to-update-parameters">Partial Derivative to Update Parameters</h2>
<p>As you may have noticed, the equation for \(z\) is the same for all of the models mentioned above. This means that the derivative for the parameter updates will also be the exact same, since the only other step is to chain together \(\frac{\partial \mathcal{L}}{\partial z}\) and \(\frac{\partial z}{\partial w}\):</p>
\[\frac{\partial \mathcal{L}}{\partial w} = \frac{\partial \mathcal{L}}{\partial z} \frac{\partial z}{\partial w}\]
<p>Since \(\frac{\partial z}{\partial w} = X\), to get the partial derivative of the loss with respect to the weights, we simply take the dot product between the transpose of the input and \((\hat{y} - y)\). We transpose \(X\) to make the shapes line up nicely for matrix multiplication. Thus, we get:</p>
\[\frac{\partial \mathcal{L}}{\partial w} = X^\boldsymbol{\top}(\hat{y} - y)\]
<h2 id="concluding-remarks">Concluding Remarks</h2>
<p>After reading this post it might be temping to say that you can use Mean Squared Error (MSE) for logistic regression since the derivatives for linear and logistic regression are the same. However, this is incorrect. It is important to realize that the derivative only works out to be the same because there is no activation function for linear regression. If you now have a sigmoid activation function in the output, then \(\frac{\partial \mathcal{L}}{\partial z} \neq (\hat{y} - y)\) for \(\mathcal{L}_{MSE}\).</p>
<p>I hope you enjoyed learning about the math behind some supervised learning loss functions! In the future I might make another blog post about loss functions, except with less math and more visuals.</p>
<div id="disqus_thread"></div>
<script>
/**
* RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS.
* LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables*/
/*
var disqus_config = function () {
this.page.url = https://brandinho.github.io; // Replace PAGE_URL with your page's canonical URL variable
this.page.identifier = /cost-function-gradients; // Replace PAGE_IDENTIFIER with your page's unique identifier variable
};
*/
(function() { // DON'T EDIT BELOW THIS LINE
var d = document, s = d.createElement('script');
s.src = 'https://brandinho-github-io.disqus.com/embed.js';
s.setAttribute('data-timestamp', +new Date());
(d.head || d.body).appendChild(s);
})();
</script>
<noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>Brandon Da Silvabrandasilva9@gmail.comGradient DescentAccelerated Proximal Policy Optimization2018-12-29T00:00:00+00:002018-12-29T00:00:00+00:00https://brandinho.github.io/APPO<h2 id="overview">Overview</h2>
<p>This post is going to be a little different than the other ones that I’ve made (and probably quite different than most blog posts out there) because I’m not going to be showcasing a finished algorithm. Rather, I’m going to show some of the progress I’ve made in developing a new algorithm that builds off of <a href="https://arxiv.org/pdf/1707.06347.pdf">Proximal Policy Optimization (PPO)</a> and <a href="http://proceedings.mlr.press/v28/sutskever13.pdf">Nesterov’s Accelerated Gradient (NAG)</a>. The new algorithm is called Accelerated Proximal Policy Optimization (APPO). The reason I’m making a post about an incomplete algorithm is so other researchers can help <strong>accelerate</strong> its development. I only ask that you cite this blog post if you use this algorithm in a research paper.</p>
<h2 id="nesterovs-accelerated-gradient">Nesterov’s Accelerated Gradient</h2>
<p>We already know how PPO works from my <a href="https://brandinho.github.io/mario-ppo/">previous blog post</a>, so now the only background information we need is NAG. In this post I will not be explaining how gradient descent works, so for those who are not familiar with gradient descent and want a comprehensive explanation, I highly recommend <a href="http://ruder.io/optimizing-gradient-descent/">Sebastian Ruder’s post</a>. I actually used that post to first learn gradient descent a couple years ago.</p>
<p>Below is the update rule for vanilla gradient descent. We have our parameters (weights), \(\theta\), which we update with our gradients \(\nabla_{\theta}J(\theta)\). If you are not familiar with this notation, the \(\nabla_\theta\) refers to a vector of partial derivatives with respect to our parameters (also called the gradient vector). \(J(\theta)\) represents the cost function given our parameters, while \(\eta\) represents our learning rate.</p>
\[\theta = \theta - \eta \nabla_{\theta}J(\theta)\]
<p>The problem with vanilla gradient descent however, is that progress is quite slow during training (shown on the left side of the image below). You will notice a large amount of oscillations across the error surface. To prevent overshooting, we use a small learning rate, which ultimately makes training slow. To help solve this problem, we use the momentum algorithm (shown on the right side of the image below), which is basically just an exponentially weighted average of the gradients.</p>
<p style="text-align: center;"><img src="https://brandinho.github.io/images/momentum.png" alt="alt" /></p>
\[v_t = \gamma v_{t-1} + \eta \nabla_{\theta}J(\theta)\]
\[\theta = \theta - v_t\]
<p>The two new terms introduced in the equations above are \(\gamma\), which is the decay factor, and \(v_t\), which is the exponentially weighted average for the current update. Let’s consider two scenarios to see why momentum helps move our parameters in the right direction faster, while also dampening oscillations. In scenario 1, imagine our previous update \(v_{t-1}\) was a positive number for one of our parameters. Now imagine the current partial derivative for that parameter is also positive. With the momentum update rule, we will be accelerating our parameter update in that direction by adding \(v_{t-1}\) to the already positive partial derivative. The same logic works for negative partial derivatives if \(v_{t-1}\) is negative. In scenario 2, imagine if \(v_{t-1}\) and \(\nabla_{\theta}J(\theta)\) had opposite directions (i.e. one is positive and the other is negative). In that case they will somewhat cancel each other out, which ultimately makes the gradient update smaller (dampening the oscillation).</p>
<p>While this sounds great, there is one pretty obvious flaw in its design - what happens when we have a large momentum term \(v_{t-1}\) and we reached a local minima (i.e. the current gradient is \(\sim 0\))? Well, if we use the momentum update rule, then we will overshoot the local minima because we have to add \(\gamma v_{t-1}\) to the current gradient. To prevent this from happening, we can anticipate the effect of \(\gamma v_{t-1}\) on our parameters and calculate that gradient vector to come up with \(v_t\). So if we used our previous example and assume \(v_{t-1}\) had a large positive value for most of the parameters, then after anticipating what our parameters will be, the gradient vector will consist of mostly negative numbers since we overshot the local minima. Now when we add the two together, they cancel each other out (for the most part). This is know as Nesterov’s Accelerated Gradient and is shown below:</p>
\[v_t = \gamma v_{t-1} + \eta \nabla_{\theta}J(\theta - \gamma v_{t-1})\]
\[\theta = \theta - v_t\]
<p>You might be wondering how the concept behind NAG can be used to improve PPO. To understand this, we first need to understand some of the drawbacks of PPO.</p>
<h2 id="areas-of-improvement-for-ppo">Areas of Improvement for PPO</h2>
<p>Even though I think PPO is an awesome algorithm, upon examining it more closely I noticed a few things that I would like to improve. I want to change the following aspects of PPO:</p>
<ol>
<li>It is a reactive algorithm and I want to make it a proactive algorithm</li>
<li>Using a ratio to measure divergence from the old policy handicaps updates for low probability actions</li>
</ol>
<p>Before diving into these points, I want to define a word I will be using going forward: <strong>training round</strong> is defined as the series of updates to our policy network after collecting a certain number of experiences. For example, we can define one training round to be 4 epochs after playing two episodes of a game.</p>
<h3 id="1-reactive-vs-proactive">1) Reactive vs Proactive</h3>
<p>Clipping only takes effect after the policy gets pushed outside of the range (it also depends on the sign of advantage). As such, it is a reactive algorithm because it only restricts movement once the policy has already moved outside of the specified bounds. This means that PPO does not ensure the new policy is proximal to the old policy because \(r_t(\theta)\) can easily move well below \(1 - \epsilon\) or above \(1 + \epsilon\). As we will see later in this post, by using the accelerated gradient concept behind NAG, we can design a proactive algorithm which anticipates how much the policy will move. We can then use the anticipated move to better control our policy and keep it roughly within the bounds.</p>
<h3 id="2-ratio-vs-absolute-difference">2) Ratio vs Absolute Difference</h3>
<p>The bounds that we set for the policy in PPO is based off of a ratio, which I do not like. The denominator \(\pi_{\theta_\text{old}}\) matters a lot because if it is too small, then learning is severely impaired. Let’s use an example to show you what I mean. Imagine two scenarios:</p>
<ol>
<li>Low probability action: \(\pi_{\theta_\text{old}}(s_t, a_t) = 2\%\)</li>
<li>High probability action: \(\pi_{\theta_\text{old}}(s_t, a_t) = 70\%\)</li>
</ol>
<p>Now let’s assume that \(\epsilon = 0.2\). This means that we restrict our new policy to be \(1.6\% \leq \pi_\theta \leq 2.4\%\) for the first scenario and \(56\% \leq \pi_\theta \leq 84\%\) for the second scenario. Wait a second… under the first scenario the policy can only move within a range that is \(0.8\%\) wide, while in the second scenario the policy can move within a range that is \(28\%\) wide. If the low probability action should actually have a high probability, then it will take forever to get it to where it should be. However, if we use an absolute difference, then the range in which the new policy can move is the exact same regardless of how small or large the probability of taking an action under our old policy is.</p>
<p><strong>NOTE</strong> - There is the obvious exception when the probability of taking an action is near \(0\%\) or \(100\%\). In those cases the lower and upper bound on the new policy is bounded at \(0\%\) and \(100\%\) respectively. However, I don’t consider this a drawback because it is the same when using the ratio.</p>
<h2 id="initial-attempt-to-improve-ppo">Initial Attempt to Improve PPO</h2>
<p>Let’s deal with the ratio first. In order to get rid of the denominator problem, we define \(\hat{r}_t(\theta)\) as the absolute difference between the new policy and the old policy:</p>
\[\hat{r}_t(\theta) = \left|\pi_\theta(a_t \mid s_t) - \pi_{\theta_\text{old}}(a_t \mid s_t)\right|\]
<p>Next, we need to make the algorithm proactive instead of reactive. To do this, we create an additional neural network that is responsible for telling us how much the policy would change if we implement a policy gradient update. We will denote its parameters as \(\theta_\text{pred}\). At the start of each mini-batch, we reset \(\theta_\text{pred}\) to be equal to \(\theta\). As a result, we can define:</p>
\[\hat{r}_t(\theta_\text{pred}) = \left|\pi_{\theta_\text{pred}}(a_t \mid s_t) - \pi_{\theta_\text{old}}(a_t \mid s_t)\right|\]
<p>Now we can see exactly how much our policy will change if we apply a policy gradient update. In order to constrain the amount our policy can change to be within \(\pi_{\theta_\text{old}}(a_t \mid s_t) \pm \epsilon\), we can calculate the following <strong>shrinkage factor</strong>:</p>
\[shrinkage = \frac{\epsilon}{\max(\hat{r}_t(\theta_\text{pred}), \epsilon)}\]
<p>I thought that applying this shrinkage factor to the gradients when updating \(\pi_\theta\) will constrain \(\hat{r}_t(\theta) \leq \epsilon\). Boy was I wrong. I was making a linear extrapolation on a function approximation that is non-linear… It was clear that I needed a better way to ensure our policy stays within the specified range per training round.</p>
<h2 id="current-state-of-appo">Current state of APPO</h2>
<p>Okay so the shrinkage factor didn’t work, what else can we do? Let’s take a page out of the supervised learning book! After calculating \(\pi_{\theta_\text{pred}}\), we can see if it moves outside of the bounds \(\pi_{\theta_\text{old}}(a_t \mid s_t) \pm \epsilon\). If so, then we can use Mean Squared Error (MSE) as the loss function for those samples and move \(\pi_\theta\) towards the bound that it crossed. On the other hand, if it is within the range, then you can update \(\pi_\theta\) with the regular policy gradient method.</p>
<p>There is one important nuance that we should keep in mind. If we consider a neural network update in isolation, then the method above works great. However, given that we train on multiple mini-batches afterwards, the proceeding change in neural network weights can easily push our policy well beyond our specified range. To prevent this, I found that increasing the number of epochs during training, while also shuffling samples between each epoch significantly reduces the probability of this occurring. I use 10 epochs, but you can probably get away with a smaller number. Empirically, this method has been shown to constrain the new policy to be within the specified bound, with an occasional small deviation outside of the range after each training round.</p>
<p>You will notice that by using the method above, an if statement splits the mini-batch into two smaller batches: one to be trained with MSE and the other to be trained with a policy gradient loss. If you don’t want to split up your mini-batch with an if statement during training, then you can update the whole mini-batch with the following loss function:</p>
\[\mathcal{L} = \frac{1}{n}\sum^n_{i=1}\left(\pi^{(i)}_{\theta} - \text{clip}(\pi^{(i)}_{\theta_\text{pred}}, \pi^{(i)}_{\theta_\text{old}} - \epsilon, \pi^{(i)}_{\theta_\text{old}} + \epsilon)\right)^2\]
<p>This is more computationally expensive because you no longer train a portion of the mini-batch using the policy gradient method (which requires less epochs than the MSE portion). Nonetheless, it is still an option for those who don’t like breaking up the loss function with an if statement.</p>
<h2 id="concluding-remarks">Concluding Remarks</h2>
<p>Results currently look promising, but I don’t think the algorithm is complete. I will continue to work on it and I welcome any feedback!</p>
<div id="disqus_thread"></div>
<script>
/**
* RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS.
* LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables*/
/*
var disqus_config = function () {
this.page.url = https://brandinho.github.io; // Replace PAGE_URL with your page's canonical URL variable
this.page.identifier = /APPO; // Replace PAGE_IDENTIFIER with your page's unique identifier variable
};
*/
(function() { // DON'T EDIT BELOW THIS LINE
var d = document, s = d.createElement('script');
s.src = 'https://brandinho-github-io.disqus.com/embed.js';
s.setAttribute('data-timestamp', +new Date());
(d.head || d.body).appendChild(s);
})();
</script>
<noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>Brandon Da Silvabrandasilva9@gmail.comReinforcement Learning, Neural Networks, Policy GradientPlaying Super Mario Bros with Proximal Policy Optimization2018-12-02T00:00:00+00:002018-12-02T00:00:00+00:00https://brandinho.github.io/mario-ppo<h2 id="overview">Overview</h2>
<p>In this post, our AI agent will learn how to play Super Mario Bros by using Proximal Policy Optimization (PPO). We want our agent to learn how to play by only observing the raw game pixels so we use convolutional layers early in the network, followed by dense layers to get our policy and state-value output. The architecture of our model is shown below.</p>
<p style="text-align: center;"><img src="https://brandinho.github.io/images/mario-model-architecture.png" alt="alt" /></p>
<p>To find the code, please follow this <a href="https://github.com/brandinho/Super-Mario-Bros-PPO">link</a>.</p>
<p>Throughout this post, I’m going to explain each of the model’s components. First we start with the convolutional layers.</p>
<h2 id="convolutional-neural-network">Convolutional Neural Network</h2>
<p>Convolutional neural networks (CNNs) are widely used in image recognition, and have achieved very impressive results to date. They have their own set of issues, such as the inability to take important spatial hierarchies into account, which <a href="https://arxiv.org/pdf/1710.09829.pdf">capsule networks</a> attempt to address. However, we don’t think that this significantly impacts an agent’s ability to play a video game from raw pixels so convolutional layers will be just fine for our algorithm.</p>
<p>Unfortunately I’m not going to fully explain CNNs because that would take a whole post on its own. Rather, I’m going to explain some of the most important concepts for our model. If you want a more detailed explanation, I highly recommend <a href="https://colah.github.io">Chrisopher Olah’s blog</a> - all his posts are incredible. Also <a href="https://www.coursera.org/learn/convolutional-neural-networks">Andrew Ng’s course</a> is awesome!</p>
<p>The first thing to understand is that every image is comprised of pixels, and every pixel is represented as a numerical value (or combination of values). The images from the game screen use the RGB color model, which means that for each pixel in the picture, there are going to be 3 numbers associated with it. The numbers correspond to how much red, green, and blue light to add to the image. An example of the RGB codes for the Mario picture are shown below:</p>
<p style="text-align: center;"><img src="https://brandinho.github.io/images/mario-color-codes.png" alt="alt" /></p>
<p>Okay great, now what do we do with these pixel values? Convolutional layers are a great way to deal with raw pixel inputs into a neural network. Each convolutional layers consists of multiple filters, which extract important information about an image. You can think of each convolutional layer as a building block for the next. For example, the first layer can put together the pixels to form edges, the second layer can put together the edges to form shapes, the third layer can put the shapes together to form objects, etc.</p>
<p>The filters work by performing an operation called convolution, shown in the image below. The operation works by taking the sum of the element-wise product between a portion of the image and the filter (also called a kernel). It focuses on a portion of the image because we need the two matrices to be the same size. In our example, we perform convolution on the bottom-right portion of the image. The filter shown below is specifically designed to detect vertical edges in an image. However, in practice we don’t preset the filter weights to perform a specific task - instead the neural network will learn the weights that it deems the best with backpropagation.</p>
<p style="text-align: center;"><img src="https://brandinho.github.io/images/convolution-operation.png" alt="alt" /></p>
<p>I know I said that the operation being performed above is convolution, but that is not completely true… We’re technically performing cross-correlation, but everyone refers to this operation in the neural network context as convolution. Let me explain why. To actually perform convolution, you need to either flip the source image or the kernel. The reason why we don’t do this for CNNs is because it adds unnecessary complexity. Why is it unnecessary? Because the neural network learns the weights for the kernel anyways, so if you needed to flip the kernel, the CNN will automatically learn the flipped kernel weights, making the actual flipping pointless. Since flipping does not make a difference, cross-correlation is equivalent to convolution in this context.</p>
<p>As mentioned before, the kernel is applied to a portion of the image, so we have to slide the kernel over the whole image to account for all the portions. Below we show an example of the filter in action! We used some different numbers - they don’t actually mean anything, I just made them up:</p>
<p style="text-align: center;"><img src="/images/ConvNet.gif" alt="Alt Text" /></p>
<p>The last concept that I want to introduce for CNNs is stride. The stride determines how many pixels the filter jumps over between convolution operations. For example, in our animation above, the stride was 1 because it moved one pixel at a time. But if we specify a stride of 2, then it will move two pixels at a time (skipping over one pixel). The larger the stride, the smaller the output from the convolutional layer. Below we show what a stride of 2 looks like for the same input and kernel:</p>
<p style="text-align: center;"><img src="/images/ConvNet2.gif" alt="Alt Text" /></p>
<p>Now that we understand how the neural network is able to deal with pixelated inputs, we will move onto the feed-forward (dense) portion of our model - it splits into a value estimation stream and a policy stream. Below we show the implementation of convolutional layers followed by a flattening layer in TensorFlow:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c1"># Convolutional Layers
</span> <span class="n">conv1</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">layers</span><span class="p">.</span><span class="n">conv2d</span><span class="p">(</span><span class="n">inputs</span> <span class="o">=</span> <span class="n">inputs</span><span class="p">,</span> <span class="n">filters</span> <span class="o">=</span> <span class="n">n_filters</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">kernel_size</span> <span class="o">=</span> <span class="n">kernel_size</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span>
<span class="n">strides</span> <span class="o">=</span> <span class="p">[</span><span class="n">n_strides</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">n_strides</span><span class="p">[</span><span class="mi">0</span><span class="p">]],</span> <span class="n">padding</span> <span class="o">=</span> <span class="s">"valid"</span><span class="p">,</span> <span class="n">activation</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">nn</span><span class="p">.</span><span class="n">elu</span><span class="p">,</span> <span class="n">trainable</span> <span class="o">=</span> <span class="n">trainable</span><span class="p">)</span>
<span class="n">conv2</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">layers</span><span class="p">.</span><span class="n">conv2d</span><span class="p">(</span><span class="n">inputs</span> <span class="o">=</span> <span class="n">conv1</span><span class="p">,</span> <span class="n">filters</span> <span class="o">=</span> <span class="n">n_filters</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">kernel_size</span> <span class="o">=</span> <span class="n">kernel_size</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span>
<span class="n">strides</span> <span class="o">=</span> <span class="p">[</span><span class="n">n_strides</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">n_strides</span><span class="p">[</span><span class="mi">1</span><span class="p">]],</span> <span class="n">padding</span> <span class="o">=</span> <span class="s">"valid"</span><span class="p">,</span> <span class="n">activation</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">nn</span><span class="p">.</span><span class="n">elu</span><span class="p">,</span> <span class="n">trainable</span> <span class="o">=</span> <span class="n">trainable</span><span class="p">)</span>
<span class="c1"># Flatten the last Convolutional Layer
</span> <span class="n">first_dimension</span> <span class="o">=</span> <span class="nb">round</span><span class="p">((((</span><span class="n">image_height</span> <span class="o">-</span> <span class="n">kernel_size</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span> <span class="o">/</span> <span class="n">n_strides</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="o">-</span> <span class="n">kernel_size</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span> <span class="o">/</span> <span class="n">n_strides</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="n">second_dimension</span> <span class="o">=</span> <span class="nb">round</span><span class="p">((((</span><span class="n">image_width</span> <span class="o">-</span> <span class="n">kernel_size</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span> <span class="o">/</span> <span class="n">n_strides</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="o">-</span> <span class="n">kernel_size</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span> <span class="o">/</span> <span class="n">n_strides</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="n">dimensionality</span> <span class="o">=</span> <span class="n">first_dimension</span> <span class="o">*</span> <span class="n">second_dimension</span> <span class="o">*</span> <span class="n">n_filters</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">conv2_flat</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">reshape</span><span class="p">(</span><span class="n">conv2</span><span class="p">,</span> <span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">dimensionality</span><span class="p">])</span>
</code></pre></div></div>
<h2 id="the-value-function">The Value Function</h2>
<p>In reinforcement learning, we often care about value functions - specifically, the state-value function \(V(s)\) and the action-value function \(Q(s,a)\). Before diving into some math, I want to explain these concepts intuitively. \(V(s)\) tells us how good it is to be in a particular state. In Super Mario Bros, the goal is to go all the way to the right side of the map, as fast as possible. Thus, we get a positive (negative) reward if we move to the right (left), while getting a negative reward every time the clock ticks. Let’s let M1 and M2 represent two of Mario’s possible positions. If we define \(V(s)\) as the expectation of \(G_t\), which is the cumulative discounted reward from time step \(t\), then we realize that \(V(s_{M2}) > V(s_{M1})\).</p>
<p style="text-align: center;"><img src="https://brandinho.github.io/images/mario-value.png" alt="alt" style="max-width: 300px; height: auto;" /></p>
<p>If my previous statement did not completely make sense, let’s make it a bit more concrete with some math. Let’s let \(R_t\) represent the reward from time step \(t\). We will define the cumulative discounted reward from time step \(t\) as:</p>
\[G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \ldots = \sum_{k=0}^{\infty} \gamma^kR_{t+k+1}\]
<p>where \(\gamma\) is a discount factor that we apply to future rewards. This is a math trick that makes an infinite sum finite since \(0 \leq \gamma \leq 1\). Although technically if \(\gamma = 1\) then the sum is still infinite because all future rewards have an equal weight. However, we generally use \(\gamma < 1\).</p>
<p>Now that we know how \(G_t\) is defined mathematically, let’s revisit our previous statement: \(V(s_{M2}) > V(s_{M1})\). The farther Mario is from the right, the longer it takes to get to the end of the map. If it takes longer to get to the end of the map, then we have to add up more negative rewards to our cumulative sum (since we get a negative reward every time the clock ticks). Thus, it makes sense that \(V(s_{M2}) > V(s_{M1})\).</p>
<p>Great, now that we have an intuition into how the state-value function works, let’s do some algebra to get a very important equation in reinforcement learning:</p>
\[\begin{align*}
V(s) &= \mathbb{E}[\, G_t \, | \, S_t=s \,] \\
&= \mathbb{E}[\, R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \ldots \, | \, S_t=s \,] \\
&= \mathbb{E}[\, R_{t+1} + \gamma (R_{t+2} + \gamma R_{t+3} + \ldots) \, | \, S_t=s \,] \\
&= \mathbb{E}[\, R_{t+1} + \gamma G_{t+1} \, | \, S_t=s \,] \\
&= \mathbb{E}[\, R_{t+1} + \gamma V(S_{t+1}) \, | \, S_t=s \,]
\end{align*}\]
<p>The equation that we end up with is know as the Bellman equation. If we think about it, it’s actually quite intuitive: the value for being in a particular state is equal to the expected reward we will receive from that state plus the discounted expected value of being in the next state. Let’s break this down a bit more. If the value for being in a state is equal to the sum of discounted future rewards, then \(V(s_{t+1})\) is the sum of discounted rewards after \(t+1\). So if we add \(R_{t+1}\) to \(\gamma V(s_{t+1})\), then we get the sum of discounted rewards after \(t\), which is \(V(s)\).</p>
<p>Alternatively, we can write the Bellman equation as,</p>
\[V(s)=\mathcal{R}_s + \gamma \sum_{s^\prime \in \mathcal{S}} \mathcal{P}_{ss^\prime} V(s^\prime)\]
<p>where \(\mathcal{P}_{ss^\prime}\) refers to the probability transition matrix (i.e. the probability of moving from \(s\) to \(s^{\prime}\) for all \(\mathcal{S}\)).</p>
<p>Up until now, we were talking about the state-value function, but what about \(Q(s,a)\)? Most times, people actually care more about \(Q\) than \(V\). The reason is because they want to know how to act in a given state, rather than the value of being in a state. This is exactly what \(Q(s,a)\) helps you determine because it tells you the value for taking a specific action in a given state. Thus, if you calculate the Q-value for all actions you can take (assuming the action space is discrete), then you can choose the action that has the maximum value. The super popular Q-learning algorithm learns the mapping from states to Q-values, so that an agent knows which actions will yield the highest cumulative discounted reward.</p>
<p>Let’s solidify our understanding of state-value and action-value. There is going to be a bit more math in this part, so get ready! First, let’s define a new term: the mapping from states to actions is defined as the policy and is denoted as \(\pi(a \mid s)\). Although policies can be deterministic, we are going to read \(\pi(a \mid s)\) as “the probability of taking an action given the state”. I find that reading equations out loud in plain english helps solidify my understanding, so that’s what I’m going to do for the next few equations.</p>
<p>First we show the value of being in state \(s\) by following policy \(\pi\). It is equal to the sum of Q-values, which correspond to particular actions, multiplied by the probability of taking those actions according to policy \(\pi\).</p>
\[v_{\pi}(s)=\sum_{a \in \mathcal{A}}\pi(a \mid s)q_{\pi}(s,a)\]
<p>Let’s break down \(v_{\pi}(s)\) in english:</p>
<ul>
<li>In any state, there are multiple actions that we can take</li>
<li>We take each action according to a probability distribution</li>
<li>Each action has a different value associated with it</li>
<li>Thus, the value of being in a state is equal to the weighted average of the action-values, in which the weights are the probabilities of taking each action</li>
</ul>
<p>Next we show the value of taking action \(a\) in state \(s\) by following policy \(\pi\). It is equal to the expected reward from taking an action plus the discounted expected value of being in the next state.</p>
\[q_{\pi}(s,a)=\mathcal{R}_s^a + \gamma \sum_{s^\prime \in \mathcal{S}}\mathcal{P}_{ss^\prime}^{a}v_{\pi}(s^\prime)\]
<p>Let’s break down \(q_{\pi}(s,a)\) in english:</p>
<ul>
<li>For any action an agent takes, it receives a reward</li>
<li>When an agent takes an action, it can end up in a different state
<ul>
<li>Image if your action was to move to the right - your agent is now in a new state</li>
</ul>
</li>
<li>Sometimes environments have randomness embedded in them
<ul>
<li>Imagine if you try to move to the right, but wind pushes you back and you end up to the left of your original position</li>
</ul>
</li>
<li>Thus, by taking an action in a given state, there is a probability that the agent will end up in various new states</li>
<li>As a result, the value of taking an action in a given state is equal to the immediate reward from taking that action plus the weighted average of state-values for the next state multiplied by a discount factor.
<ul>
<li>The weights are the probabilities of ending up in the next states.</li>
</ul>
</li>
</ul>
<p>The previous two equations shown were half-step lookaheads. To show the full one-step lookaheads, we can plug in the previous equations to obtain the following:</p>
\[v_{\pi}(s)=\sum_{a \in \mathcal{A}}\pi(a \mid s)\left(\mathcal{R}_s^a + \gamma \sum_{s^\prime \in \mathcal{S}}\mathcal{P}_{ss^\prime}^{a}v_{\pi}(s^\prime)\right)\]
\[q_{\pi}(s,a)=\mathcal{R}_s^a + \gamma \sum_{s^\prime \in \mathcal{S}}\mathcal{P}_{ss^\prime}^{a}\sum_{a^\prime \in \mathcal{A}}\pi(a^\prime|s^\prime)q_{\pi}(s^\prime,a^\prime)\]
<p>If you understood the intuition for the first two equations, then you should have no problem with the two equations above - they are simply an extension using the exact same logic.</p>
<h2 id="policy-gradient">Policy Gradient</h2>
<p>What if we want to skip the middle part and just learn a mapping from states to actions without estimating the value of taking an action? We can do this with the policy gradient method, in which we explicitly learn \(\pi\)! Well sort of… we will soon see why we will actually need to incorporate the value function, but until then, let’s walk through a simple implementation of a policy gradient. Let’s consider the loss function:</p>
\[\mathcal{L} = r \times \log \pi(s,a)\]
<p>We want to maximize \(\mathcal{L}\), which is equivalent to minimizing \(-\mathcal{L}\) (we usually perform gradient descent, so minimizing a loss function is the convention). By minimizing \(-\mathcal{L}\), we ensure that we increase the probability of taking an action that gives us a positive reward, and decrease the probability of taking an action that gives us a negative reward. That seems like a good idea, right? Not really… let’s go through an example to understand why. Imagine there are 3 actions that an agent can take with rewards of \([-1,3,20]\) in particular state. There are two main problems with this approach:</p>
<ol>
<li>Credit Assignment Problem</li>
<li>Multiple “Good” Actions</li>
</ol>
<p>The credit assignment problem refers to the fact that rewards can be temporally delayed. For example, if an agent takes an action in time step \(t\), the reward might come well after \(t+1\). An example in Super Mario Bros is when our agent has to jump over a tube; multiple frames elapse from the time it presses the jump button to the time it actually makes it over the tube. The number of time steps that can possibly elapse between actions and rewards differ for each situation, so how do we solve this problem? Although this is not a perfect solution, we can use value functions, specifically \(q_{\pi}(s,a)\). Since \(q_{\pi}(s,a)\) sums all future discounted rewards from taking action \(a\) and following policy \(\pi\), our agent can take into account rewards that are temporally delayed. Our loss function now becomes:</p>
\[\mathcal{L} = q_{\pi}(s,a) \times \log \pi(s,a)\]
<p>Let’s now assume that \([-1,3,20]\) represents Q-values instead of rewards. We still have an issue because there are multiple actions that have a positive expected value. Imagine if we sample the second action, which has a positive Q-value. Based on our new policy gradient loss function, the parameter update would increase the probability of taking that action since \(q_{\pi}(s,a)\) is positive. But what about action 3? It had a much higher Q-value than action 2, so instead we need a way to tell the model to decease the probability of selecting action 2 and instead select action 3. That is what advantage helps us do.</p>
<h2 id="the-advantage-function">The Advantage Function</h2>
<p>Rather than looking at how good it is to take an action, advantage tells us how good an action is relative to other actions. This subtlety is important because we want to select actions that are better than average, as opposed to any action that has a positive expected value. To do this, we have to strip out the state-value from the action-value to get a pure estimate of how good an action is. We define advantage as:</p>
\[A(s,a) = Q(s,a) - V(s)\]
<p>If we assume that our policy follows a uniform distribution (equal probability for each action), then \(V(s) = 7.33\), which means that \(A(s,a) = [-8.3,-4.3,12.7]\). Using our new loss function for policy gradients,</p>
\[\mathcal{L} = A_{\pi}(s,a) \times \log \pi(s,a)\]
<p>we see that after selecting action 2, our agent will decrease the probability of selecting that action again in the same state because it has a negative advantage (its value is worse than the average). This is great, it does exactly what we want it to do! However, we don’t know the true advantage function (much like the value functions), so we have to estimate it. Luckily, there are a few ways to do this, but I’m going to focus on one method - using the temporal difference error (\(\delta_{TD}\)) from our value estimation.</p>
<p>Let me back up a little to explain what temporal difference error is. Remember when we saw this somewhat complicated equation earlier:</p>
\[q_{\pi}(s,a)=\mathcal{R}_s^a + \gamma \sum_{s^\prime \in \mathcal{S}}\mathcal{P}_{ss^\prime}^{a}\sum_{a^\prime \in \mathcal{A}}\pi(a^\prime|s^\prime)q_{\pi}(s^\prime,a^\prime)\]
<p>Well it turns out that it will come in handy after all! Just a refresher - the equation above considers all possible paths. But what if we just sample one action from our policy and sample the next state from the environment? Well then it becomes:</p>
\[q_{\pi}(s,a) = r + \gamma v_{\pi}(s^{\prime})\]
<p>Keep this in mind while I explain \(\delta_{TD}\). As the name implies, temporal difference error refers to the difference between the one-step lookahead and the current estimate. We can calculate \(\delta_{TD}\) for either the state-value or action-value, but in this example we’re using the state-value. When we sample, the one-step lookahead equation for state-value becomes \(v_{\pi}(s) = r + \gamma v_{\pi}(s^{\prime})\). You’ll notice that the left side is a pure estimate, while the right side is a mix of estimation and actual data from the environment. This means that the right side contains more information about the environment than the left! By taking the difference between the two we obtain:</p>
\[\delta_{TD} = r + \gamma v_{\pi}(s^{\prime}) - v_{\pi}(s)\]
<p>and by minimizing \(\delta_{TD}^2\), we move our value estimation closer to the actual value function. This is because we are continually moving our estimate closer to a target that contains more data from the actual environment. In addition to using \(\delta_{TD}\) to optimize our value network, it turns out that we can also use it to estimate advantage. Wait, what? How? Let’s bring back \(q_{\pi}(s,a)\):</p>
\[q_{\pi}(s,a) = r + \gamma v_{\pi}(s^{\prime})\]
<p>Recall what advantage is defined as:</p>
\[A = Q - V\]
<p>Now let’s take a look at \(\delta_{TD}\) again:</p>
\[\delta_{TD} = \underbrace{r + \gamma v_{\pi}(s^{\prime})}_{q_{\pi}(s,a)} - v_{\pi}(s)\]
<p>which means that \(\delta_{TD} \approx A\)</p>
<h2 id="generalized-advantage-estimation">Generalized Advantage Estimation</h2>
<p>The <a href="https://arxiv.org/pdf/1506.02438.pdf">paper</a> we are referencing in this section was used for continuous control, but it can also be used for a discrete action space, like the one we are working with.</p>
<p>We will denote our advantage estimate as \(\hat{A}_t\). Like any other estimate, \(\hat{A}_t\) is subject to bias (although it has low variance). To get an unbiased estimate, we need to get rid of the value estimate completely and sum all future rewards in an episode. This is known as the Monte Carlo return, and it has high variance. As with most things in machine learning, there is a tradeoff - this one is known as the bias-variance tradeoff in reinforcement learning. Generalized Advantage Estimation (GAE) is a great solution that significantly reduces variance while maintaining a tolerable level of bias. It is parametereized by \(\gamma \in [0,1]\) and \(\lambda \in [0,1]\), where \(\gamma\) is the discount factor mentioned earlier in this blog, and \(\lambda\) is the decay parameter used to take an exponentially weighted average of k-step advantage estimators. It is analogous to <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.132.7760&rep=rep1&type=pdf">Sutton’s TD(\(\lambda\))</a>.</p>
<p>Before we get into some of the math, I want to note that \(\gamma\) and \(\lambda\) serve different purposes. To determine the scale of the value function, we use \(\gamma\). In other words, the value of \(\gamma\) determines how nearsighted (\(\gamma\) near 0) or farsighted (\(\gamma\) near 1) we want our agent to be in its value estimate. No matter how accurate our value function is, if \(\gamma < 1\), we introduce bias into the policy gradient estimate. On the other hand, \(\lambda\) is a decay factor and \(\lambda < 1\) only introduces bias when the value function is inaccurate.</p>
<p>I’m going to spare you the details on the derivation of GAE because I feel like we’ve gone through enough math for one post. However, if you have any questions just let me know in the comments section below and I’ll explain it in-depth. As mentioned before, GAE is defined as the exponentially weighted average of k-step advantage estimators. The equation is shown below:</p>
\[\hat{A}^{GAE(\gamma,\lambda)}_t = \sum^{\infty}_{l=0}(\gamma \lambda)^l\delta_{t+l}\]
<p>Below we show an implementation of GAE in python:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="nf">get_gaes</span><span class="p">(</span><span class="n">rewards</span><span class="p">,</span> <span class="n">state_values</span><span class="p">,</span> <span class="n">next_state_values</span><span class="p">,</span> <span class="n">GAMMA</span><span class="p">,</span> <span class="n">LAMBDA</span><span class="p">):</span>
<span class="n">deltas</span> <span class="o">=</span> <span class="p">[</span><span class="n">r_t</span> <span class="o">+</span> <span class="n">GAMMA</span> <span class="o">*</span> <span class="n">next_v</span> <span class="o">-</span> <span class="n">v</span> <span class="k">for</span> <span class="n">r_t</span><span class="p">,</span> <span class="n">next_v</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">rewards</span><span class="p">,</span> <span class="n">next_state_values</span><span class="p">,</span> <span class="n">state_values</span><span class="p">)]</span>
<span class="n">gaes</span> <span class="o">=</span> <span class="n">copy</span><span class="p">.</span><span class="n">deepcopy</span><span class="p">(</span><span class="n">deltas</span><span class="p">)</span>
<span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="nb">reversed</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">gaes</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)):</span>
<span class="n">gaes</span><span class="p">[</span><span class="n">t</span><span class="p">]</span> <span class="o">=</span> <span class="n">gaes</span><span class="p">[</span><span class="n">t</span><span class="p">]</span> <span class="o">+</span> <span class="n">LAMBDA</span> <span class="o">*</span> <span class="n">GAMMA</span> <span class="o">*</span> <span class="n">gaes</span><span class="p">[</span><span class="n">t</span> <span class="o">+</span> <span class="mi">1</span><span class="p">]</span>
<span class="k">return</span> <span class="n">gaes</span><span class="p">,</span> <span class="n">deltas</span>
</code></pre></div></div>
<p>If you understand the equation above, then you might find this next part pretty cool, otherwise you can just skip over it. There are two special cases of the formula above, when \(\lambda=0\) and \(\lambda=1\):</p>
\[GAE(\gamma,0): \hat{A}_t := \delta_t = r_t + \gamma V(S_{t+1}) - V(S_t)\]
\[GAE(\gamma,1): \hat{A}_t := \sum^{\infty}_{l=0}\gamma\delta_{t+l} = \sum^{\infty}_{l=0}\gamma^lr_{t+l} - V(S_t)\]
<p>When we have \(0 < \lambda < 1\), our GAE is making a compromise between bias and variance. From now on, our loss function for the policy gradient becomes:</p>
\[\mathcal{L} = \hat{A}^{GAE(\gamma,\lambda)} \times \log \pi(s,a)\]
<p>Going forward, when you see \(\hat{A}_t\), we are actually referring to \(\hat{A}^{GAE(\gamma,\lambda)}_t\).</p>
<h2 id="proximal-policy-optimization">Proximal Policy Optimization</h2>
<p>We’re finally done catching up on all the background knowledge - time to learn about Proximal Policy Optimization (PPO)! This algorithm is from <a href="https://arxiv.org/pdf/1707.06347.pdf">OpenAI’s paper</a>, and I highly recommend checking it out to get a more in-depth understanding after reading my blog.</p>
<p>PPO takes inspiration from <a href="https://arxiv.org/pdf/1502.05477.pdf">Trust Region Policy Optimization</a> (TRPO), which maximizes a “surrogate” objective function:</p>
\[L^{CPI}(\theta) = \hat{\mathbb{E}}_t\big[r_t(\theta)\hat{A}_t\big]\]
<p>where \(r_t(\theta)\) represents the probability ratio of our current policy versus our old policy:</p>
\[r_t(\theta) = \frac{\pi_{\theta}(a_t \mid s_t)}{\pi_{\theta_\text{old}}(a_t \mid s_t)}\]
<p>TRPO also has constraints that I’m not going to get into, but if you’re interested, I highly recommend reading the paper. While TRPO is quite impressive, it is complex and computationally expensive to run. As a result, OpenAI came up with a simpler, more general algorithm that has better sample complexity (empirically). The idea is to limit how much our policy can change during each round of updates by clipping \(r_t(\theta)\) between a range determined by \(\epsilon\):</p>
\[L^{CLIP}(\theta) = \hat{\mathbb{E}}_t\big[\min(r_t(\theta)\hat{A}_t, \text{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon)\hat{A}_t)\big]\]
<p>The reason we do this is because conventional policy gradient methods are very sensitive to your choice of step size. If the step size is too small then the training progresses too slowly. If the step size is too large then your policy can overshoot the optimal policy during training, making it too noisy. By limiting how much our policy can change, we reduce the sensitivity to the step size. An implementation of \(L^{CLIP}\) in python is shown below:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">with</span> <span class="n">tf</span><span class="p">.</span><span class="n">variable_scope</span><span class="p">(</span><span class="s">'actor_loss'</span><span class="p">):</span>
<span class="n">action_probabilities</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">reduce_sum</span><span class="p">(</span><span class="n">policy</span> <span class="o">*</span> <span class="n">tf</span><span class="p">.</span><span class="n">one_hot</span><span class="p">(</span><span class="n">indices</span> <span class="o">=</span> <span class="n">actions</span><span class="p">,</span> <span class="n">depth</span> <span class="o">=</span> <span class="n">output_dimension</span><span class="p">),</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">old_action_probabilities</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">reduce_sum</span><span class="p">(</span><span class="n">old_policy</span> <span class="o">*</span> <span class="n">tf</span><span class="p">.</span><span class="n">one_hot</span><span class="p">(</span><span class="n">indices</span> <span class="o">=</span> <span class="n">actions</span><span class="p">,</span> <span class="n">depth</span> <span class="o">=</span> <span class="n">output_dimension</span><span class="p">),</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">ratios</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">exp</span><span class="p">(</span><span class="n">tf</span><span class="p">.</span><span class="n">log</span><span class="p">(</span><span class="n">action_probabilities</span><span class="p">)</span> <span class="o">-</span> <span class="n">tf</span><span class="p">.</span><span class="n">log</span><span class="p">(</span><span class="n">old_action_probabilities</span><span class="p">))</span>
<span class="n">clipped_ratios</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">clip_by_value</span><span class="p">(</span><span class="n">ratios</span><span class="p">,</span> <span class="n">clip_value_min</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">_clip_value</span><span class="p">,</span> <span class="n">clip_value_max</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">+</span> <span class="n">_clip_value</span><span class="p">)</span>
<span class="n">clipped_loss</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">minimum</span><span class="p">(</span><span class="n">tf</span><span class="p">.</span><span class="n">multiply</span><span class="p">(</span><span class="n">GAE</span><span class="p">,</span> <span class="n">ratios</span><span class="p">),</span> <span class="n">tf</span><span class="p">.</span><span class="n">multiply</span><span class="p">(</span><span class="n">GAE</span><span class="p">,</span> <span class="n">clipped_ratios</span><span class="p">))</span>
<span class="n">actor_loss</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">reduce_mean</span><span class="p">(</span><span class="n">clipped_loss</span><span class="p">)</span>
</code></pre></div></div>
<p>You will notice in the image below (taken from the PPO paper) that there are certain values of \(r_t(\theta)\) where the gradient is 0. When the advantage is positive, the cutoff point is \(1 + \epsilon\). When the advantage is negative, the cutoff point is \(1 - \epsilon\). By taking the minimum of the clipped and unclipped objective, as demonstrated below, we are creating a lower bound on the unclipped objective. In other words, we ignore a change in \(r_t(\theta)\) when it makes the objective improve, which is why the lower bound is also known as the pessimistic bound.</p>
<p style="text-align: center;"><img src="https://brandinho.github.io/images/PPO-objective.png" alt="alt" /></p>
<p>Our implementation has a unique feature that I haven’t mentioned yet: after the convolutional layers, we concatenate a series of one hot encodings that correspond to previous actions that our agent took. The reason we do this is because there are a few cases in which a combination of buttons need to be pressed in a sequential order. By taking previous actions into account, we allow our agent to learn such sequences. The video shown below was created after relatively little training using PPO on a Macbook Pro. I plan on running the algorithm for longer and updating the video sometime in the near future:</p>
<video controls="controls" allowfullscreen="true">
<source src="/images/mario.avi" type="video/mp4" />
</video>
<h2 id="concluding-remarks">Concluding Remarks</h2>
<p>In this post, we covered a lot of reinforcement learning background and learned how PPO works. We see that using GAE with PPO is a clever way to deal with the credit assignment problem, while keeping bias in check. We also learned a little bit about convolutional neural networks as a way to deal with pixelated inputs. I hope you can take what you learned in this post and apply it to your favorite games!</p>
<div id="disqus_thread"></div>
<script>
/**
* RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS.
* LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables*/
/*
var disqus_config = function () {
this.page.url = https://brandinho.github.io; // Replace PAGE_URL with your page's canonical URL variable
this.page.identifier = /mario-ppo; // Replace PAGE_IDENTIFIER with your page's unique identifier variable
};
*/
(function() { // DON'T EDIT BELOW THIS LINE
var d = document, s = d.createElement('script');
s.src = 'https://brandinho-github-io.disqus.com/embed.js';
s.setAttribute('data-timestamp', +new Date());
(d.head || d.body).appendChild(s);
})();
</script>
<noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>Brandon Da Silvabrandasilva9@gmail.comReinforcement Learning, Neural Networks, Policy GradientLearning How to Run with Genetic Algorithms2018-11-18T00:00:00+00:002018-11-18T00:00:00+00:00https://brandinho.github.io/genetic-algorithm<h2 id="overview">Overview</h2>
<p>When most people think of Deep Reinforcement Learning, they probably think of Q-networks or policy gradients. Both of these methods require you to calculate derivatives and use gradient descent. In this post, we are going to explore a derivative-free method for optimizing a policy network. Specifically, we are going to be using a genetic algorithm on DeepMind’s <a href="https://arxiv.org/pdf/1801.00690.pdf">Control Suite</a> to allow the “cheetah” physical model to learn how to run. You can find the complete code on my <a href="https://github.com/brandinho/Genetic-Algorithm-Control-Suite">github repo</a>.</p>
<h2 id="genetic-algorithm-background">Genetic Algorithm Background</h2>
<p>Genetic algorithms (GAs) are inspired by natural selection, as put forth by Charles Darwin. The idea is that over generations, the heritable traits of a population change because of <em>mutation</em> and the concept of <em>survival of the fittest</em>.</p>
<p>Similar to natural selection, GAs iterate over multiple generations to evolve a population. The population in our case is going to consist of a bunch of neural network weights, which define our cheetah agents. You can think of each set of neural network weights as an individual agent in the population - usually called a chromosome or genotype. Chromosomes are usually encoded as binary strings, but since we want to optimize neural networks weights, we will adapt it for continuous numbers. Each neural network weight in our chromosome can be referred to as a gene. After iterating through all the generations, and continually improving the cheetah’s chromosome (its neural network weights), we hope that it learns how to run.</p>
<h3 id="initialization">Initialization</h3>
<p>To begin the process, we need to initialize our population of agents. We sample the initial neural network weights from a normal distribution with a scaling factor outlined in <a href="http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf">Glorot and Bengio’s paper</a>:</p>
\[Var[W^i] = \frac{2}{n_i + n_{i+1}}\]
<p>where \(W^i\) refers to the weight matrix in the \(i^\text{th}\) layer, while \(n_i\) and \(n_{i+1}\) refer to the input and output dimensionality of that layer. Below you’ll see python code to implement the population initialization, where <code class="language-plaintext highlighter-rouge">scaling_factor</code> is a vector of variances calculated according to the equation above:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">population</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">multivariate_normal</span><span class="p">(</span><span class="n">mean</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">*</span><span class="n">scaling_factor</span><span class="p">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span>
<span class="n">cov</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">diag</span><span class="p">(</span><span class="n">scaling_factor</span><span class="p">),</span>
<span class="n">size</span> <span class="o">=</span> <span class="n">population_size</span><span class="p">)</span>
</code></pre></div></div>
<h3 id="selection">Selection</h3>
<p>Now that we have a population, we can have the agents within the population compete against each other! The agents that are the most “fit” have the highest probability of passing their genes onto the next generation. We will define fitness as the cumulative reward of our agent over the span of an episode. As you might have guessed by the way we defined it, fitness refers to how good an agent is at performing the task we want it to learn. Those that are better at performing the task will have a better chance of being selected as parents to breed a new generation. There are two primary methods for parent selection - <strong>Roulette</strong> and <strong>Tournament</strong>.</p>
<p>The roulette method selects parents with a probability proportional to their fitness score. This is why it is also called <em>Fitness Proportionate Selection</em>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c1"># Roulette Wheel Selection
</span> <span class="n">position</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">2</span><span class="p">):</span>
<span class="n">random_number</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">uniform</span><span class="p">(</span><span class="n">low</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="n">high</span> <span class="o">=</span> <span class="n">scores_cumulsum</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
<span class="n">position</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="nb">next</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">scores_cumulsum</span><span class="p">)</span> <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">></span> <span class="n">random_number</span><span class="p">))</span>
<span class="n">parent_1</span> <span class="o">=</span> <span class="n">population</span><span class="p">[</span><span class="n">population_index_sorted</span><span class="p">[</span><span class="n">position</span><span class="p">[</span><span class="mi">0</span><span class="p">]]]</span>
<span class="n">parent_2</span> <span class="o">=</span> <span class="n">population</span><span class="p">[</span><span class="n">population_index_sorted</span><span class="p">[</span><span class="n">position</span><span class="p">[</span><span class="mi">1</span><span class="p">]]]</span>
</code></pre></div></div>
<p>The tournament method runs two tournaments in parallel with different subsets of the total population. The competitors for each tournament are chosen at random. The winners from each tournament are selected as the parents to breed the next generation.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c1"># Tournament Selection
</span> <span class="n">k</span> <span class="o">=</span> <span class="n">population_size</span> <span class="o">//</span> <span class="mi">2</span>
<span class="n">tournament_population</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">zeros</span><span class="p">((</span><span class="n">k</span><span class="p">,</span> <span class="mi">2</span><span class="p">))</span>
<span class="n">total_competitors</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">choice</span><span class="p">(</span><span class="n">np</span><span class="p">.</span><span class="n">arange</span><span class="p">(</span><span class="n">population_size</span><span class="p">),</span> <span class="n">k</span> <span class="o">*</span> <span class="mi">2</span><span class="p">,</span> <span class="n">replace</span> <span class="o">=</span> <span class="bp">False</span><span class="p">)</span>
<span class="n">tournament_population</span><span class="p">[:,</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="n">competition_scores</span><span class="p">[</span><span class="n">total_competitors</span><span class="p">[:</span><span class="n">k</span><span class="p">]]</span>
<span class="n">tournament_population</span><span class="p">[:,</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">competition_scores</span><span class="p">[</span><span class="n">total_competitors</span><span class="p">[</span><span class="n">k</span><span class="p">:]]</span>
<span class="n">parent_indexes</span> <span class="o">=</span> <span class="n">total_competitors</span><span class="p">[</span><span class="n">np</span><span class="p">.</span><span class="n">argmax</span><span class="p">(</span><span class="n">tournament_population</span><span class="p">,</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">0</span><span class="p">)</span> <span class="o">+</span> <span class="n">np</span><span class="p">.</span><span class="n">array</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="n">k</span><span class="p">])]</span>
<span class="n">parent_1</span> <span class="o">=</span> <span class="n">population</span><span class="p">[</span><span class="n">parent_indexes</span><span class="p">[</span><span class="mi">0</span><span class="p">],]</span>
<span class="n">parent_2</span> <span class="o">=</span> <span class="n">population</span><span class="p">[</span><span class="n">parent_indexes</span><span class="p">[</span><span class="mi">1</span><span class="p">],]</span>
</code></pre></div></div>
<h3 id="elitism">Elitism</h3>
<p>One thing we can do to improve performance in our algorithm is introduce the concept of elitism. This refers to the act of carrying over the most fit agents to the next generation without altering their chromosomes through crossover or mutation (which we will explore very soon). We do this because we always want to preserve the best agents from one generation to the next; it is not guaranteed that any of the children will be more fit than their parents.</p>
<h3 id="crossover">Crossover</h3>
<p>Now that we know how to select the parents from the population, let’s talk breeding! Crossover, also called recombination, takes the chromosomes of two parents and combines them to form children in the next generation. Here are a few ways you can combine two chromosomes:</p>
<p>The first and easiest way is to perform <strong>One Point</strong> crossover. You randomly select a partition in the chromosome, as indicated by the red line below. The child gets the left side of the partition from one parent and the right side from the other parent.</p>
<p style="text-align: center;"><img src="https://brandinho.github.io/images/one_point_crossover.png" alt="alt" /></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">partition</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">parent_1</span><span class="p">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="c1"># Select which parent will be the "left side"
</span> <span class="k">if</span> <span class="n">which_parent</span> <span class="o">==</span> <span class="s">"Parent 1"</span><span class="p">:</span>
<span class="n">child</span> <span class="o">=</span> <span class="n">parent_1</span>
<span class="n">child</span><span class="p">[</span><span class="n">partition</span><span class="p">:]</span> <span class="o">=</span> <span class="n">parent_2</span><span class="p">[</span><span class="n">partition</span><span class="p">:]</span>
<span class="k">elif</span> <span class="n">which_parent</span> <span class="o">==</span> <span class="s">"Parent 2"</span><span class="p">:</span>
<span class="n">child</span> <span class="o">=</span> <span class="n">parent_2</span>
<span class="n">child</span><span class="p">[</span><span class="n">partition</span><span class="p">:]</span> <span class="o">=</span> <span class="n">parent_1</span><span class="p">[</span><span class="n">partition</span><span class="p">:]</span>
</code></pre></div></div>
<p>Building on the previous method is <strong>Two Point</strong> crossover. This is conceptually the same, except you randomly select two points, which serve as a lower and upper bound. The child gets the elements outside of the bounds from one parent, and the elements within the bounds from the other parent.</p>
<p style="text-align: center;"><img src="https://brandinho.github.io/images/two_point_crossover.png" alt="alt" /></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">lower_limit</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">parent_1</span><span class="p">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">upper_limit</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="n">lower_limit</span><span class="o">+</span><span class="mi">1</span><span class="p">,</span> <span class="n">parent_1</span><span class="p">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="c1"># Select which parent will be the "outside bounds"
</span> <span class="k">if</span> <span class="n">which_parent</span> <span class="o">==</span> <span class="s">"Parent 1"</span><span class="p">:</span>
<span class="n">child</span> <span class="o">=</span> <span class="n">parent_1</span>
<span class="n">child</span><span class="p">[</span><span class="n">lower_limit</span><span class="p">:</span><span class="n">upper_limit</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">parent_2</span><span class="p">[</span><span class="n">lower_limit</span><span class="p">:</span><span class="n">upper_limit</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span>
<span class="k">elif</span> <span class="n">which_parent</span> <span class="o">==</span> <span class="s">"Parent 2"</span><span class="p">:</span>
<span class="n">child</span> <span class="o">=</span> <span class="n">parent_2</span>
<span class="n">child</span><span class="p">[</span><span class="n">lower_limit</span><span class="p">:</span><span class="n">upper_limit</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">parent_1</span><span class="p">[</span><span class="n">lower_limit</span><span class="p">:</span><span class="n">upper_limit</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span>
</code></pre></div></div>
<p>Unlike the previous two methods, which required the swapped genes to be in a sequence, the <strong>Uniform</strong> crossover does not. Rather, it randomly selects, with a uniform distribution, the indexes to be swapped during crossover.</p>
<p style="text-align: center;"><img src="https://brandinho.github.io/images/uniform_crossover.png" alt="alt" /></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">random_sequence</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">choice</span><span class="p">(</span><span class="n">np</span><span class="p">.</span><span class="n">arange</span><span class="p">(</span><span class="n">parent_1</span><span class="p">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]),</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">parent_1</span><span class="p">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]),</span> <span class="n">replace</span> <span class="o">=</span> <span class="bp">False</span><span class="p">)</span>
<span class="k">if</span> <span class="n">which_parent</span> <span class="o">==</span> <span class="s">"Parent 1"</span><span class="p">:</span>
<span class="n">child</span> <span class="o">=</span> <span class="n">parent_1</span>
<span class="n">child</span><span class="p">[</span><span class="n">np</span><span class="p">.</span><span class="n">sort</span><span class="p">(</span><span class="n">random_sequence</span><span class="p">)]</span> <span class="o">=</span> <span class="n">parent_2</span><span class="p">[</span><span class="n">np</span><span class="p">.</span><span class="n">sort</span><span class="p">(</span><span class="n">random_sequence</span><span class="p">)]</span>
<span class="k">elif</span> <span class="n">which_parent</span> <span class="o">==</span> <span class="s">"Parent 2"</span><span class="p">:</span>
<span class="n">child</span> <span class="o">=</span> <span class="n">parent_2</span>
<span class="n">child</span><span class="p">[</span><span class="n">np</span><span class="p">.</span><span class="n">sort</span><span class="p">(</span><span class="n">random_sequence</span><span class="p">)]</span> <span class="o">=</span> <span class="n">parent_1</span><span class="p">[</span><span class="n">np</span><span class="p">.</span><span class="n">sort</span><span class="p">(</span><span class="n">random_sequence</span><span class="p">)]</span>
</code></pre></div></div>
<p>For the last crossover method, we’ll switch it up a little bit with the <strong>Arithmetic</strong> crossover. Like the name implies, rather than swapping genes to form a new chromosome, we will do some arithmetics to make a new chromosome. We will perform a simple weighted average on the chromosomes, where the weight is randomly generated.</p>
<p style="text-align: center;"><img src="https://brandinho.github.io/images/arithmetic_crossover.png" alt="alt" /></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">random_weight</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">rand</span><span class="p">()</span>
<span class="n">child</span> <span class="o">=</span> <span class="n">parent_1</span> <span class="o">*</span> <span class="n">random_weight</span> <span class="o">+</span> <span class="n">parent_2</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">random_weight</span><span class="p">)</span>
</code></pre></div></div>
<p>Personally, I like using all of the crossover methods, so each time my algorithm performs crossover I randomly select one of the above methods with equal probability.</p>
<p>When setting up a genetic algorithm we define a probability of performing crossover, \(p_\text{cross}\). Thus, with \(1 - p_\text{cross}\) probability, we carry over the parent chromosomes to the next generation without crossover. Since we are going to use elitism in our algorithm, we will probably want to set \(p_\text{cross}\) to be close to 1 because otherwise there is a high probability that we will have duplicate chromosomes in the next generation.</p>
<h3 id="mutation">Mutation</h3>
<p>After reviewing some of the crossover methods, you might be thinking that we’re just combining genes together without changing their order (with the exception of the arithmetic operator). This means that our chromosomes will be bounded by the initialized values from the first generation, which limits how much our agents can evolve. To ensure this doesn’t happen, we need to maintain genetic diversity - we do this with the mutation operator.</p>
<p>Similar to crossover, there are multiple ways to perform mutation. For my implementation I randomly select a gene with \(p_\text{mutate}\) probability and add gaussian noise to it:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">noise</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">standard_normal</span><span class="p">()</span> <span class="o">*</span> <span class="n">noise_scale</span>
<span class="n">mutation_position</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">population</span><span class="p">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="n">child</span><span class="p">[</span><span class="n">mutation_position</span><span class="p">]</span> <span class="o">=</span> <span class="n">child</span><span class="p">[</span><span class="n">mutation_position</span><span class="p">]</span> <span class="o">+</span> <span class="n">noise</span>
</code></pre></div></div>
<p>Even though I remained relatively simple with my implementation, you can get a bit fancier by implementing some of the mutation methods outlined below. The first is the <strong>Swap</strong> mutation, which selects two random positions in the chromosome and swaps their genes:</p>
<p style="text-align: center;"><img src="https://brandinho.github.io/images/swap_mutation.png" alt="alt" style="max-width: 300px; height: auto;" /></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">random_positions</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">choice</span><span class="p">(</span><span class="n">np</span><span class="p">.</span><span class="n">arange</span><span class="p">(</span><span class="n">child</span><span class="p">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]),</span> <span class="mi">2</span><span class="p">,</span> <span class="n">replace</span> <span class="o">=</span> <span class="bp">False</span><span class="p">)</span>
<span class="n">value_1</span><span class="p">,</span> <span class="n">value_2</span> <span class="o">=</span> <span class="n">child</span><span class="p">[</span><span class="n">random_positions</span><span class="p">[</span><span class="mi">0</span><span class="p">]],</span> <span class="n">child</span><span class="p">[</span><span class="n">random_positions</span><span class="p">[</span><span class="mi">1</span><span class="p">]]</span>
<span class="n">child</span><span class="p">[</span><span class="n">random_positions</span><span class="p">[</span><span class="mi">0</span><span class="p">]],</span> <span class="n">child</span><span class="p">[</span><span class="n">random_positions</span><span class="p">[</span><span class="mi">1</span><span class="p">]]</span> <span class="o">=</span> <span class="n">value_2</span><span class="p">,</span> <span class="n">value_1</span>
</code></pre></div></div>
<p>Another method you can implement is the <strong>Inversion</strong> mutation, which selects two random positions and inverts/reverses the substring of genes between them:</p>
<p style="text-align: center;"><img src="https://brandinho.github.io/images/inversion_mutation.png" alt="alt" style="max-width: 300px; height: auto;" /></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">lower_limit</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">child</span><span class="p">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">upper_limit</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="n">lower_limit</span><span class="o">+</span><span class="mi">1</span><span class="p">,</span> <span class="n">child</span><span class="p">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="n">child</span><span class="p">[</span><span class="n">lower_limit</span><span class="p">:</span><span class="n">upper_limit</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">child</span><span class="p">[</span><span class="n">lower_limit</span><span class="p">:</span><span class="n">upper_limit</span><span class="o">+</span><span class="mi">1</span><span class="p">][::</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
</code></pre></div></div>
<p>Lastly, you can implement the <strong>Scramble</strong> mutation, which selects two random positions and scrambles the positions of the genes within them:</p>
<p style="text-align: center;"><img src="https://brandinho.github.io/images/scramble_mutation.png" alt="alt" style="max-width: 300px; height: auto;" /></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">lower_limit</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">child</span><span class="p">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">upper_limit</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="n">lower_limit</span><span class="o">+</span><span class="mi">1</span><span class="p">,</span> <span class="n">child</span><span class="p">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="n">scrambled_order</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">choice</span><span class="p">(</span><span class="n">np</span><span class="p">.</span><span class="n">arange</span><span class="p">(</span><span class="n">lower_limit</span><span class="p">,</span> <span class="n">upper_limit</span><span class="o">+</span><span class="mi">1</span><span class="p">),</span> <span class="n">upper_limit</span> <span class="o">+</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">lower_limit</span><span class="p">,</span> <span class="n">replace</span> <span class="o">=</span> <span class="bp">False</span><span class="p">)</span>
<span class="n">child</span><span class="p">[</span><span class="n">lower_limit</span><span class="p">:</span><span class="n">upper_limit</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">child</span><span class="p">[</span><span class="n">scrambled_order</span><span class="p">]</span>
</code></pre></div></div>
<h2 id="deepminds-control-suite">DeepMind’s Control Suite</h2>
<p>Great, now that we have all the pieces to make a genetic algorithm, let’s put them together to train the “cheetah” domain from DeepMind’s Control Suite. For those who are not familiar with the library, it is powered by the MuJoCo physics engine and provides you with an environment to train agents on a set of continuous control tasks. For our experiment we want the cheetah to learn how to run.</p>
<p>The thing that I really like about this library is that it has a standardized structure. For example, the library provides you with an observation of the environment and a reward for every action you take. The state observation for our domain task is a combination of the cheetah’s position and velocity. The reward, \(r\), is a function of the forward velocity, \(v\), up to a maximum of \(10 m/s\):</p>
\[r(v) = max(0, min(v/10, 1))\]
<p>We run each episode for 500 frames and calculate the fitness, \(f\), as:</p>
\[f = \sum_{i=1}^{500}r_i\]
<p>At each time step, our agent has to make 6 actions in parallel - the movement of each of its limbs. The action vector for our cheetah has the following property: \(\boldsymbol{a} \in \mathcal{A} \equiv [-1,1]^{6}\). Thus, for our policy, we are going to use a neural network with a 6-dimensional \(\tanh\) output. We flatten all of the neural network weights to a one dimensional array in order to implement the crossover and mutation operators mentioned above. After the child chromosome is created, we reshape the weights to be used in a neural network for the next generation. Overall, we used 1000 generations with a population size of 40 to train our cheetah.</p>
<p>When the training process starts (Generation 1), we see that the cheetah doesn’t know how to move and end up falling backwards:</p>
<p style="text-align: center;"><img src="/images/cheetah_start.gif" alt="Alt Text" /></p>
<p>As training progresses (Generation 250), the cheetah learns how to run forward. However, we see that near the end of the episode it loses control of its stride and falls flat on its face:</p>
<p style="text-align: center;"><img src="/images/cheetah_mid.gif" alt="Alt Text" /></p>
<p>At the end of the training process (Generation 1000), we see that the cheetah learns how to run, while also maintaining its center of gravity during large strides:</p>
<p style="text-align: center;"><img src="/images/cheetah_end.gif" alt="Alt Text" /></p>
<p>Awesome, we did it!</p>
<h2 id="concluding-remarks">Concluding Remarks</h2>
<p>In this post we learned how genetic algorithms can be used to optimize parameters of a neural network for a continuous control task. In a future post we will explore an application where we mix genetic algorithms (derivative-free method) and policy gradients (derivative-based method) for better training.</p>
<div id="disqus_thread"></div>
<script>
/**
* RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS.
* LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables*/
/*
var disqus_config = function () {
this.page.url = https://brandinho.github.io; // Replace PAGE_URL with your page's canonical URL variable
this.page.identifier = /genetic-algorithm; // Replace PAGE_IDENTIFIER with your page's unique identifier variable
};
*/
(function() { // DON'T EDIT BELOW THIS LINE
var d = document, s = d.createElement('script');
s.src = 'https://brandinho-github-io.disqus.com/embed.js';
s.setAttribute('data-timestamp', +new Date());
(d.head || d.body).appendChild(s);
})();
</script>
<noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>Brandon Da Silvabrandasilva9@gmail.comReinforcement Learning, Neural Networks, Genetic AlgorithmLearning Probability Distributions in Bounded Action Spaces2018-11-12T00:00:00+00:002018-11-12T00:00:00+00:00https://brandinho.github.io/bayesian-policy<h2 id="overview">Overview</h2>
<p>In this post we will learn how to apply reinforcement learning in a probabilistic manner. More specifically, we will be looking at some of the difficulties in applying conventional approaches to bounded action spaces, and provide a solution. This blog assumes you have knowledge in deep learning. If not, check out <a href="http://neuralnetworksanddeeplearning.com/chap1.html">Michael Nielsen’s book</a> - it is very comprehensive and easy to understand.</p>
<h2 id="reinforcement-learning-background">Reinforcement Learning Background</h2>
<p>I am not going to provide a complete background on Reinforcement Learning (RL) because there are already some excellent resources online such as <a href="https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-0-q-learning-with-tables-and-neural-networks-d195264329d0">Arthur Juliani’s blogs</a> and <a href="https://www.youtube.com/watch?v=2pWv7GOvuf0&list=PLzuuYNsE1EZAXYR4FJ75jcJseBmo4KQ9-">David Silver’s lectures</a>. I highly recommend going through both to get a solid understanding of the fundamentals. With that said, I will explain some concepts that are important for this blog post.</p>
<p>At the most basic level, the goal of RL is to learn a mapping from states to actions. To understand what this means, I think it is important to take a step back and understand the RL framework more generally. Cue the overused RL diagram:</p>
<p style="text-align: center;"><img src="https://brandinho.github.io/images/AgentEnvironment.jpg" alt="alt" /></p>
<p>The first thing to notice is that there is a feedback loop between the agent and the environment. For clarity, the agent refers to the AI that we are creating, while the environment refers to the world that the agent has to navigate through. In order to navigate through an environment, the agent has to take actions. The specific actions will depend on the domain - we will describe a few fairly soon. After the agent takes an action, it receives an observation of the environment (the current state) and a reward (assuming we don’t have sparse rewards).</p>
<p>After interacting with the environment for long enough, we hope that our agent learns how to take actions that maximize its cumulative reward over the long-term. It is important to realize that the best action in one state is not necessarily the best action in another state. So going back to our statement about mapping states to actions, this simply means that we want our agent to learn the best actions to take in each environment state. The function that maps states to actions is called a policy and is denoted as \(\pi(a \mid s)\). Usually we read \(\pi(a \mid s)\) as: <em>probability of taking action \(a\), given we are in state \(s\)</em>. However, just as a side note, your policy does not have to be defined probabilistically - you can define it deterministically as well.</p>
<p>Now let’s talk a bit about actions an agent can take. The first distinction I would like to make is between discrete actions and continuous actions. When we refer to discrete actions, we simply mean that there is a finite set of possible actions an agent can take. For example, in pong an agent can decide to move up or down. On the other hand, continuous actions have an infinite number of possibilities. An example of a continuous action, although kind of silly, is the hiding position of an agent if it is playing hide and seek.</p>
<p>Given enough time, the agent can theoretically hide anywhere - so the action space is unbounded. In contrast, we can have a continuous action space that is bounded. An example close to my heart is position sizing when trading a financial asset. The bounds are -1 (100% Short) and 1 (100% Long). To map states to that bounded action space, we can use \(\tanh\) in the final layer of a neural network. That seems pretty easy… so why am I writing a blog post about it? Often times we need more than just a deterministic output, especially when the underlying data has a low signal-to-noise ratio. The additional piece of information that we need is <em>uncertainty</em> in our agent’s decision. We will use a Bayesian approach to model a posterior distribution and sample from this distribution to estimate the uncertainty. Don’t worry if that doesn’t completely make sense yet - it will by the end of this post!</p>
<h2 id="probability-distributions">Probability Distributions</h2>
<p>For a great introduction to Bayesian statistics I suggest reading <a href="https://www.countbayesie.com">Will Kurt’s blog</a> - Count Bayesie. It’s awesome.</p>
<p>Distributions can be thought of as representing beliefs about the world. Specifically as it relates to our task at hand, the probability distributions represent our beliefs in how good an action is, given the state. In the financial markets context, where the action space is continuous and bounded between -1 and 1, a mean close to 1 represents a belief that it is a good time to buy that asset, so we should long it. A mean close to -1 represents the opposite, so we should short the asset. Building on this example, if the standard deviation of our distribution is large (small) then our agent is uncertain (certain) in its decision. In other words, if the agent’s policy has a large standard deviation, then it has not developed a strong belief yet.</p>
<p>Whenever you hear anyone talking about Bayesian statistics, you always hear the terms “prior” and “posterior”. Simply put, a prior is your belief about the world <em>before</em> receiving new information. However, once you receive new information, then you update your prior distribution to form a posterior distribution. After that, if you receive more information, then your posterior becomes your prior, and the new information gets incorporated to form a new posterior distribution. Essentially, there is this feedback loop of continual learning that happens as more and more new information gets processed by your agent. Below we visually show one iteration of this loop:</p>
<p style="text-align: center;"><img src="https://brandinho.github.io/images/prior_posterior.png" alt="alt" /></p>
<p>Our goal is to learn a good posterior distribution on actions, conditioned on the state that the agent is in. If you are familiar with <a href="https://arxiv.org/pdf/1506.02142.pdf">this paper</a>, then you might be thinking that we can just use Monte Carlo (MC) dropout with a \(\tanh\) output layer. For those who are not familiar with this concept, let me explain. Dropout is a technique that was originally used for neural network regularization. With each pass, it will randomly “drop” neurons from each hidden layer by setting their output to 0. This reduces the output’s dependency on any one particular neuron, which should help generalization. However, researchers at Cambridge found that using dropout during inference can be used to approximate a posterior distribution. This is because each time you pass inputs through the network, a different set of neurons will be dropped, so the output is going to be different for each run - creating a distribution of outputs.</p>
<p>The great thing about this architecture is that you can easily pass gradients through the policy network. The loss function that we are minimizing throughout this blog is \(\mathcal{L} = - r \times \pi(s)\), where \(r\) denotes the reward and \(\pi(s)\) denotes the policy output given the states (i.e. the action). We wanted to demonstrate how the distribution changes in a controlled environment. So we use the same state input throughout all our experiments and continually feed it a positive reward to view the changes during training. Below is the first example using the MC Dropout method and a \(\tanh\) output layer.</p>
<p><img src="/images/MC_dropout_posterior.gif" alt="Alt Text" /></p>
<p>I omitted a kernel density estimation (KDE) plot on top of the histogram because as training progressed, the KDE became much more jagged and not representative of the actual probability density function (PDF). I was using <code class="language-plaintext highlighter-rouge">sns.kdeplot</code>, if anyone knows how to fix this, please let me know in the comments section!</p>
<p>There are two things that I don’t particularly like about this approach. The first is that it is possible to have multiple peaks in the distribution, as seen when the neural network is first initialized. I realize that as training went on, only one peak emerged. However, the fact that an agent can potentially learn such a distribution (with multiple peaks) makes me uncomfortable. If we go back to our example in the financial markets, an action of -1 will have the exact opposite reward compared to an action of 1 (because it is the other side of the trade), so having peaks at both ends of the spectrum is quite confusing. I would much rather just have one peak near 0 with a large standard deviation if the agent is uncertain which action to take. The second is that it becomes overly optimistic in its decision when compared to a gaussian output (we will see this later), which could possibly indicate that it is understating the uncertainty.</p>
<p>I will digress for a moment to state that a multimodal distribution (a distribution with multiple peaks) is not always bad. For example if you imagine an agent trying to navigate through a room and their policy dictates the angle at which they will move, then there could be two different angles that, while momentarily will send them in different directions, will ultimately lead them to the same end location. However, for this post, we will stick to the example in the financial markets, where a multimodal distribution doesn’t make sense.</p>
<p>Instead of using MC dropout, we can try using a normal distribution in the output and see if things improve. The architecture of our neural network now becomes:</p>
<p style="text-align: center;"><img src="https://brandinho.github.io/images/gaussian_output.png" alt="alt" /></p>
<p>If our neural network parameters are denoted by \(\theta\), then we can define \(\mu_{\theta}\) and \(\sigma_{\theta}\) as outputs of the neural network, such that:</p>
\[\pi \sim \mathcal{N}(\mu_{\theta}(s), \sigma_{\theta}(s))\]
<h2 id="reparameterization-trick">Reparameterization Trick</h2>
<p>We want to update the policy network with backpropagation (similar to what we did with the MC dropout architecture), but you’ll notice that we have a bit of a problem - a random variable is now part of the computation graph. This is a problem because backpropagation cannot flow through a random node. However, by using the reparameterization trick, we can move the random node outside of the computation graph and then feed in samples drawn from the distribution as constants. Inference is the exact same, but now our neural network can perform backpropagation.</p>
<p>To do this, we define a random variable \(\varepsilon\), which does not depend on \(\theta\). The new architecture becomes:</p>
<p style="text-align: center;"><img src="https://brandinho.github.io/images/gaussian_reparameterized.png" alt="alt" /></p>
\[\varepsilon \sim \mathcal{N}(0,I)\]
\[\pi = \mu_{\theta}(s) + \sigma_{\theta}(s) \cdot \varepsilon\]
<p>Python code to take the random variable outside of the computation graph is shown below (I’m only showing the relevant portion of the computation graph):</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kn">import</span> <span class="nn">tensorflow</span> <span class="k">as</span> <span class="n">tf</span>
<span class="n">policy_mu</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">nn</span><span class="p">.</span><span class="n">tanh</span><span class="p">(</span><span class="n">tf</span><span class="p">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">previous_layer</span><span class="p">,</span> <span class="n">weights_mu</span><span class="p">)</span> <span class="o">+</span> <span class="n">bias_mu</span><span class="p">)</span>
<span class="n">policy_sigma</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">nn</span><span class="p">.</span><span class="n">softplus</span><span class="p">(</span><span class="n">tf</span><span class="p">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">previous_layer</span><span class="p">,</span> <span class="n">weights_sigma</span><span class="p">)</span> <span class="o">+</span> <span class="n">bias_sigma</span><span class="p">)</span>
<span class="n">epsilon</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">random_normal</span><span class="p">(</span><span class="n">shape</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">shape</span><span class="p">(</span><span class="n">policy_sigma</span><span class="p">),</span> <span class="n">mean</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="n">stddev</span> <span class="o">=</span> <span class="mi">1</span><span class="p">,</span> <span class="n">dtype</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">float32</span><span class="p">)</span>
<span class="n">policy</span> <span class="o">=</span> <span class="n">policy_mu</span> <span class="o">+</span> <span class="n">policy_sigma</span> <span class="o">*</span> <span class="n">epsilon</span>
</code></pre></div></div>
<p>Now to get the neural network to work in a bounded space, we can clip outputs to be between -1 and 1. We simply change the last line of code in our network to:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">policy</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">clip_by_value</span><span class="p">(</span><span class="n">policy_mu</span> <span class="o">+</span> <span class="n">policy_sigma</span> <span class="o">*</span> <span class="n">epsilon</span><span class="p">,</span> <span class="n">clip_value_min</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">clip_value_max</span> <span class="o">=</span> <span class="mi">1</span><span class="p">)</span>
</code></pre></div></div>
<p>The resulting distribution is shown below:</p>
<p><img src="/images/clipped_posterior.gif" alt="Alt Text" /></p>
<p>There is one obvious flaw in this approach - all of the clipped values get a value of either -1 or 1, which creates a very unbalanced distribution. To fix this, we will sample \(\varepsilon\) from a truncated normal distribution.</p>
<h2 id="truncated-normal-solution">Truncated Normal Solution</h2>
<p>A truncated normal distribution is similar to a normal distribution, in that it is defined by a mean (\(\mu\)) and standard deviation (\(\sigma\)). However, the key distinction is that the distribution’s range is limited to be within a lower and upper bound. Typically the lower bound is denoted by \(a\) and the upper bound is denoted by \(b\), but I’m going to use \(L\) and \(U\) because I think it is easier to follow.</p>
<p>One might think that the bounds we define for the distribution should be the same as the bounds of our policy, but that won’t work if we want to use reparameterization. This is because the bounds apply to \(\varepsilon\) and not \(\pi\). Since we expand \(\varepsilon\) by \(\sigma\) and shift it by \(\mu\), then applying bounds of -1 and 1 will result in a \(\pi\) that extends beyond the bounds. To make this point more clear, let’s say we defined our bounds \(-1 \leq \varepsilon \leq 1\), and \(\mu = 0.5 , \, \sigma = 1\). If we generate a sample \(\varepsilon = 0.9\), then after you apply the transformation \(\mu + \sigma \cdot \varepsilon\), you get \(\pi = 0.5 + 1 \cdot 0.9 = 1.4\), which is beyond the upper bound.</p>
<p>To generate the proper upper and lower bounds, we will use the equations below:</p>
\[L = \frac{-1 - \mu_{\theta}}{\sigma_{\theta}}\]
\[U = \frac{1 - \mu_{\theta}}{\sigma_{\theta}}\]
<p>Using our previous example, we find that \(U = 0.5\), which means that the largest \(\varepsilon\) we can sample is 0.5. Plugging this into our reparameterized equation, we see that the largest \(\pi\) we can generate is 1. Similarly, \(L = -1.5\), which means that the lowest \(\pi\) we can generate is -1. Perfect, we figured it out!</p>
<p>Given the PDF for a normal distribution:</p>
\[p(\varepsilon) = \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\left(\frac{\varepsilon - \mu}{\sigma}\right)^2}\]
<p>We will let \(F(\varepsilon)\) denote our cumulative distribution function (CDF). Our truncated density now becomes:</p>
\[p(\varepsilon \mid L \leq \varepsilon \leq U) = \frac{p(\varepsilon)}{F(U) - F(L)} \, \, \text{for} \, L \leq \varepsilon \leq U\]
<p>The denominator, \(F(U) - F(L)\), is the normalizing constant that allows the truncated density to integrate to 1. The reason we do this is because, as shown below, we are only sampling from a portion of \(p(\varepsilon)\).</p>
<p style="text-align: center;"><img src="https://brandinho.github.io/images/truncated_distribution.png" alt="alt" /></p>
<p>You can import <code class="language-plaintext highlighter-rouge">scipy</code> and use the following function to generate samples from a truncated normal distribution:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kn">import</span> <span class="nn">scipy.stats</span> <span class="k">as</span> <span class="n">stats</span>
<span class="n">mu_dims</span> <span class="o">=</span> <span class="mi">3</span> <span class="c1"># Dimensionality of the Mu generated by the Neural Network
</span>
<span class="n">n_samples</span> <span class="o">=</span> <span class="mi">10000</span>
<span class="n">sn_mu</span> <span class="o">=</span> <span class="mi">0</span> <span class="c1"># Standard Normal Mu
</span> <span class="n">sn_sigma</span> <span class="o">=</span> <span class="mi">1</span> <span class="c1"># Standard Normal Sigma
</span>
<span class="n">generator</span> <span class="o">=</span> <span class="n">stats</span><span class="p">.</span><span class="n">truncnorm</span><span class="p">((</span><span class="n">lower_bound</span> <span class="o">-</span> <span class="n">sn_mu</span><span class="p">)</span> <span class="o">/</span> <span class="n">sn_sigma</span><span class="p">,</span> <span class="p">(</span><span class="n">upper_bound</span> <span class="o">-</span> <span class="n">sn_mu</span><span class="p">)</span> <span class="o">/</span> <span class="n">sn_sigma</span><span class="p">,</span> <span class="n">loc</span> <span class="o">=</span> <span class="n">sn_mu</span><span class="p">,</span> <span class="n">scale</span> <span class="o">=</span> <span class="n">sn_sigma</span><span class="p">)</span>
<span class="n">epsilons</span> <span class="o">=</span> <span class="n">generator</span><span class="p">.</span><span class="n">rvs</span><span class="p">([</span><span class="n">n_samples</span><span class="p">,</span> <span class="n">mu_dims</span><span class="p">])</span>
</code></pre></div></div>
<p><img src="/images/posterior.gif" alt="Alt Text" /></p>
<p>This distribution looks a lot nicer than both of the previous approaches, and has some nice properties:</p>
<ul>
<li>It only has one peak at all times</li>
<li>Outputs do not need to be clipped</li>
<li>The policy doesn’t look overly optimistic.</li>
</ul>
<h2 id="concluding-remarks">Concluding Remarks</h2>
<p>In this post, we examined a few approaches to approximating a posterior distribution over our policy. Ultimately, we feel that using a neural network with a truncated normal policy is the best approach out of those examined. We learned how to reparameterize a truncated normal, which allows us to train the policy network using backpropagation.</p>
<h2 id="acknowledgments">Acknowledgments</h2>
<p>I would like to thank <a href="https://www.linkedin.com/in/alek-riley-609073110/">Alek Riley</a> for his feedback on how to improve the clarity of certain explanations.</p>
<div id="disqus_thread"></div>
<script>
/**
* RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS.
* LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables*/
/*
var disqus_config = function () {
this.page.url = https://brandinho.github.io; // Replace PAGE_URL with your page's canonical URL variable
this.page.identifier = /bayesian-policy; // Replace PAGE_IDENTIFIER with your page's unique identifier variable
};
*/
(function() { // DON'T EDIT BELOW THIS LINE
var d = document, s = d.createElement('script');
s.src = 'https://brandinho-github-io.disqus.com/embed.js';
s.setAttribute('data-timestamp', +new Date());
(d.head || d.body).appendChild(s);
})();
</script>
<noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>Brandon Da Silvabrandasilva9@gmail.comReinforcement Learning, Neural Networks, Bayesian