Nonlinear and Stochastic Optimization

Computational optimization is essential to many subdomains of engineering, science, and business such as design, control, operations, supply chains, game theory, data science/analytics, and machine learning.

This course provides a practical introduction to models, algorithms, and modern software for large-scale numerical optimization. Topics include (nonconvex) nonlinear programming, deterministic global optimization, integer programming, dynamic optimization, and stochastic programming. Multi-objective optimization and mathematical programs with complementarity constraints may be covered based on time and student interests. The class is design for advanced undergraduate/graduate students from engineering, science, and mathematics who wish to incorporate optimization methods into their research. The course will begin with an introduction to modeling and the Python-based Pyomo computational environment. Optimization theory and algorithms are emphasized.

What am I going to get out of this class?

At the end of the semester, you should be able to…

• Mathematically formulate optimization problems relevant to your discipline (and research!)

• Program optimization models in Pyomo and compute numerical solutions with state-of-the-art (e.g., commercial) solvers

• Explain the main theory for nonlinear constrained and unconstrained optimization

• Describe basic algorithm elements in pseudocode and implement them in Python

• Analyze results from an optimization problem and communicate key finds in a presentation

• Write and debug 200 lines of Python code using best practices (e.g., publication quality figures, doc strings)

Content

Organization

• Welcome
• Assignments

Optimization Modeling in Pyomo

• 1. Getting Started with Pyomo
  • 1.1. Local Installation
  • 1.2. Optimization Modeling with Applications
  • 1.3. Your First Optimization Problem
  • 1.4. Continuous Optimization
  • 1.5. Integer Programs
  • 1.6. 60 Minutes to Pyomo: An Energy Storage Model Predictive Control Example
• 2. Logical Modeling
  • 2.1. Logical Modeling and Generalized Disjunctive Programs
  • 2.2. Modeling Disjunctions through the Strip Packing Problem
• 3. Dynamic Optimization
  • 3.1. Differential Algebraic Equations (DAEs)
  • 3.2. Numeric Integration for DAEs
  • 3.3. Dynamic Optimization with Collocation and Pyomo.DAE
  • 3.4. Dynamic Optimization with Pyomo.DAE: An Example
• 4. Optimization Under Uncertainty
  • 4.1. Stochastic Programming
  • 4.2. Risk Measures and Portfolio Optimization
• 5. Data Science and Applied Statistics
  • 5.1. Parameter estimation with parmest
  • 5.2. Supplementary material: data for parmest tutorial
  • 5.3. Reactor Kinetics Example for Pyomo.DoE Tutorial

Algorithms and Theory

• 6. Unconstrained Nonlinear Optimization
  • 6.1. Linear Algebra Review and SciPy Basics
  • 6.2. Mathematics Primer
  • 6.3. Unconstrained Optimality Conditions
  • 6.4. Newton-type Methods for Unconstrained Optimization
  • 6.5. Quasi-Newton Methods for Unconstrained Optimization
  • 6.6. Descent and Globalization
• 7. Constrained Nonlinear Optimization
  • 7.1. Convexity Revisited
  • 7.2. Local Optimality Conditions
  • 7.3. Analysis of KKT Conditions
  • 7.4. Constraint Qualifications
  • 7.5. Second Order Optimality Conditions
  • 7.6. NLP Diagnostics with Degeneracy Hunter
  • 7.7. Simple Netwon Method for Equality Constrained NLPs
  • 7.8. Inertia-Corrected Netwon Method for Equality Constrained NLPs
• 8. Special Topics
  • 8.1. Integer Programming with Simple Branch and Bound
  • 8.2. MINLP Algorithms
  • 8.3. Deterministic Global Optimization

Student Contributions

• Derivative-Free Optimization
  • Bayesian Optimization Tutorial 1
  • Bayesian Optimization Tutorial 2
• Stochastic Gradient Descent
  • Stochastic Gradient Descent Tutorial 1
  • Stochastic Gradient Descent Tutorial 2
  • Functions and Utilities
  • Implementation of the Algorithm
  • Regression Example: Fitting a Quadratic Function to a 2D Data Set
  • Binary Classification: Logistic Regression Example
  • Conclusions
  • References
  • Stochastic Gradient Descent Tutorial 3
• Machine Learning and Applied Statistics
  • Expectation Maximization Algorithm and MAP Estimation