{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Name: Your name here\n",
    "UID: Your student ID num here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework 4:  Gradients \n",
    " \n",
    "Use either the slides, or Boyd $\\S$3.2 as a reference.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem 1\n",
    "For this problem, you may use any of the results/rules from the lecture slides.\n",
    "\n",
    "### (a) Consider the function\n",
    " $$f(x) = \\begin{cases}\n",
    "0, \\text{ if } x<0\\\\\n",
    "\\infty , \\text{ if } x\\ge 0\\\\\n",
    "\\end{cases}.$$\n",
    "\n",
    "Is this function closed and proper?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work here*\n",
    "\n",
    "No. It is proper but not closed because the epigraph does not contain the point (0,0) or anything above it.  This means the epigraph has an open edge.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (b) Consider the set of $n\\times n$ low-rank matrices $\\{A\\in\\mathbb{R}^{n\\times n} | \\, \\text{rank}(A)\\le k \\},$ where $0<k<n$.   Is this set convex?  Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work here*\n",
    "\n",
    "No.  Here's a counter-example for $n=2,$ $k=1$:\n",
    " $$\n",
    "\\left ( \n",
    "\\begin{array}{cc}\n",
    "1& 0  \\\\\n",
    "0 & 0 \n",
    "\\end{array}\n",
    "\\right)\n",
    "+\n",
    "\\left ( \n",
    "\\begin{array}{cc}\n",
    "0& 0  \\\\\n",
    "0 & 1\n",
    "\\end{array}\n",
    "\\right)\n",
    "=\n",
    "\\left ( \n",
    "\\begin{array}{cc}\n",
    "1& 0  \\\\\n",
    "0 & 1 \n",
    "\\end{array}\n",
    "\\right).\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (c) Suppose $\\{A_i\\}$ are matrices of identical size.  Define the set $$\\{X \\,\\,|\\,\\,  \\text{trace}(A_i^T X A_i) \\le 0, \\,\\, \\forall i \\}.$$\n",
    "Is this set of matrices convex?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work here*\n",
    "\n",
    "Yep. $\\text{trace}(A_i^T X A_i) $ is a linear operator, so this is an intersection of half spaces."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (d) Let $C \\subset \\mathbb{R}^n$ be a compact (closed and bounded), non-empty convex set.  For a scalar $x,$ let C(x) be the set of all points in $C$ whose first coordinate is closer to x than any other point. More precisely: $C(x)=\\{a\\in C \\, |\\,  \\forall b \\in C , |a_1-x|\\le |b_1-x|  \\}.$  Is $C(x)$ a convex set for any choice of $x$? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work here*\n",
    "\n",
    "Choose some $C$ and x.\n",
    "Define $f(a) = | a_1 - x |.$  This is a convex function.  The set $C(x)$ is the set of minimizers of $f$ restricted to the convex set $C$.  The set of minimizers to this convex optimization problem is therefore convex.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (e)  Let $\\|\\cdot\\|$ be the vector 2-norm.  Is the set $C = \\{(x,y)\\, |\\,\\, \\|x\\| \\le  y \\}$ convex?  Is it a cone?  Why?  What about the set $C = \\{(x,y)\\, |\\,\\, \\|x\\|^2 \\le  y \\}?$  Is this convex? Is this a  cone?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work here*\n",
    "\n",
    "Let $(x,y)$ and $(x',y')$ be in $C$.  The first set is convex because\n",
    "$$\\|\\theta x'+(1-\\theta)x\\| \\le \\theta \\| x'\\|+(1-\\theta)\\|x\\| \\le  \\theta y'+(1-\\theta)y.$$\n",
    "It is a cone because $$\\|x\\| \\le y  \\longrightarrow \\|\\alpha x\\|  = \\alpha \\| x\\|\\le \\alpha y$$\n",
    "for non-negative $y.$\n",
    "\n",
    "The second set is convex by a similar argument (using Jensens inequality on the convex function $\\|\\cdot\\|^2$). However, this particular set is not always a cone.  Look in 1 dimension and assume the 2-norm.   Then $(1,1)$ is in the set, but $(2,2)$ is not.   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem 2\n",
    "### (a) Suppose the function $f(x,y)$ is convex.  Prove that \n",
    "  $$g(x) = \\min_y f(x,y)$$\n",
    "  is also convex.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work here*\n",
    "\n",
    "Choose $x_1$ and $x_2.$  Let $y_i=\\arg\\min_y f(x_i,y),$ for $i=1,2.$   Then\n",
    "$$g(\\theta x_1 +(1-\\theta) x_2) = \\min_y f(\\theta x_1 +(1-\\theta) x_2,y)$$\n",
    "$$\\le f(\\theta x_1 +(1-\\theta)x_2,\\theta y_1 +(1-\\theta) y_2) \\le \\theta f( x_1,y_1) +(1-\\theta)f(x_2,y_2).$$\n",
    "Note, we used Jensen's inequality to form the last inequality.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (b)  Suppose $f$ is strongly convex.  Show that the gradient of $f$ is strongly monotone, i.e., there is a constant $m>0$ with \n",
    "  $$ \\langle x-y,\\nabla f(x) - \\nabla f(y) \\rangle \\ge m\\|x-y\\|^2.$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work here*\n",
    "\n",
    "By the definition of strong convexity\n",
    "$$f(y) \\ge f(x) + (y-x)^T\\nabla f(x) + \\frac{m}{2}\\|y-x\\|^2.$$\n",
    "We also have\n",
    "$$f(x) \\ge f(y) + (x-y)^T\\nabla f(y) + \\frac{m}{2}\\|x-y\\|^2.$$\n",
    "Adding these together and cancelling terms yields\n",
    "\n",
    "$$0 \\ge  (x-y)^T(\\nabla f(y)-\\nabla f(x)) + m\\|x-y\\|^2,$$\n",
    "which re-arranges to yield the result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem 3\n",
    " Suppose that $g(x)$ is convex and $h(x)$ is concave (i.e. $-h(x)$ is convex).  Suppose we restrict both functions into a closed, convex set $C$ such that both $g(x)$ and $h(x)$ are always positive when $x \\in C.$  Prove that the function\n",
    "  $f(x) = g(x)/h(x)$ \n",
    "  is quasi-convex.  (note: It follows that every local minimum is also global)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work here*\n",
    "\n",
    "Choose some $\\alpha \\ge 0$ and consider the sub-level set $S_\\alpha = \\{x| f(x) \\le \\alpha \\}.$  If $x\\in S_\\alpha,$ then \n",
    "$$g(x)/h(x) \\le \\alpha  \\quad \\longrightarrow  \\quad g(x) \\le \\alpha h(x) \\quad \\longrightarrow \\quad g(x) - \\alpha h(x) \\le 0.$$\n",
    "But $-h(x)$ is convex and $\\alpha \\ge 0,$ and so  $g(x) - \\alpha h(x)$ is convex.  Because the set $S_\\alpha$ is a sub-level set of the convex function $g(x) - \\alpha h(x),$ it is convex."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}