Description
1. (4 points) Analyzing gradient descent. Consider a simple function having two weight variables:
L(w1, w2) = 0.5(aw2
1 + bw2
2
).
a. Write down the gradient ∇L(w), and using this, derive the weights w
∗
that achieve the
minimum value of L.
b. Instead of simply writing down the optimal weights, let’s now try to optimize L using
gradient descent. Starting from some randomly chosen (non-zero) initialization point
w1(0), w2(0), write down the gradient descent updates. Show that the updates have the
form:
w1(t + 1) = ρ1w1(t), w2(t + 1) = ρ2w2(t)
where wi(t) represent the weights at the t
th iteration. Derive the expressions for ρ1 and
ρ2 in terms of a, b, and the learning rate.
c. Under what values of the learning rate does gradient descent converge to the correct
minimum? Under what values does it not?
d. Provide a scenario under which the convergence rate of gradient descent is very slow.
(Hint: consider the case where a/b is a very large ratio.)
2. (6 points) Open the (incomplete) Jupyter notebook provided as an attachment to this homework
in Google Colab (or other cloud service of your choice) and complete the missing items. Save
your finished notebook in PDF format and upload along with your answers to the above theory
questions in a single PDF.
