Optimization for Decision Science

Computational optimization is essential to many subdomains of engineering, science, and business such as design, control, operations, supply chains, game theory, data science/analytics, and machine learning.

This course provides a practical introduction to models, algorithms, and modern software for large-scale numerical optimization, especially for decision-making in engineering and business contexts. Topics include (nonconvex) nonlinear programming, deterministic global optimization, integer programming, dynamic optimization, and stochastic programming. Multi-objective optimization, optimization with embedded machine learning models as constraints, optimal experiment design, optimization for statistical inference, and mathematical programs with complementarity constraints may be covered based on time and student interests. The class is designed for advanced undergraduate/graduate engineering, science, mathematics, business, and statistics students who wish to incorporate computation optimization methods into their research. The course begins with an introduction to modeling and the Python-based Pyomo computational environment. Optimization theory and algorithms are emphasized throughout the semester.

What am I going to get out of this class?

At the end of the semester, you should be able to…

• Mathematically formulate optimization problems relevant to decision-making within your discipline (and research!)

• Program optimization models in Pyomo and compute numerical solutions with state-of-the-art (e.g., commercial) solvers

• Explain the main theory for nonlinear constrained and unconstrained optimization

• Describe basic algorithmic elements in pseudocode and implement them in Python

• Analyze results from an optimization problem and communicate key findings in a presentation

• Write and debug 200 lines of Python code using best practices (e.g., publication quality figures, doc strings)

What do I need to know to take the class?

Graduate students (60499): A background in linear algebra and numerical methods is strongly recommended but not required. Students must be comfortable programming in Python (preferred), MATLAB, Julia, C, or a similar language. Topics in EE 60551 and ACMS 60880 are complementary to CBE/ACMS 60499. These courses are not prerequisites for CBE/ACMS 60499.

Undergraduate students (40499): Students must be comfortable with linear algebra and programming in Python (preferred), MATLAB, Julia, C, or a similar language. A course on these topics, such as CBE 20258, AME 30251 (concurrent is okay), CE 30125 (concurrent is okay), PHYS 20451, or ACMS 20220, and the standard undergraduate curriculum in computer science or electrical engineering should satisfy this prerequisite. Please contact the instructor for any preparation questions.

Content

Organization

• Welcome
• Assignments
• Archive

Optimization Modeling in Pyomo

• 1. Getting Started with Pyomo
  • 1.1. Local Installation
  • 1.2. Optimization Modeling with Applications
  • 1.3. Your First Optimization Problem
  • 1.4. Continuous Optimization: Linear Programming
  • 1.5. Continuous Optimization: Nonlinear Programming
  • 1.6. Integer Programs
  • 1.7. 60 Minutes to Pyomo: An Energy Storage Model Predictive Control Example
• 2. Logical Modeling
  • 2.1. Logical Modeling and Generalized Disjunctive Programs
  • 2.2. Modeling Disjunctions through the Strip Packing Problem
• 3. Dynamic Optimization
  • 3.1. Pyomo.DAE Example: Race Car
  • 3.2. Pyomo.DAE Example: Temperature Control Lab
  • 3.3. Differential Algebraic Equations (DAEs)
  • 3.4. Numeric Integration for DAEs
  • 3.5. Dynamic Optimization with Collocation and Pyomo.DAE
  • 3.6. Pyomo.DAE: Racing Example Revisited
• 4. Optimization Under Uncertainty
  • 4.1. Stochastic Programming
  • 4.2. Blocks and Other Pyomo Best Practices
  • 4.3. Advanced Topics in Stochastic Programming
  • 4.4. Risk Measures and Portfolio Optimization
• 5. Data Science and Applied Statistics
  • 5.1. Parameter estimation with parmest
  • 5.2. Supplementary material: data for parmest tutorial
  • 5.3. Reactor Kinetics Example for Pyomo.DoE Tutorial

Algorithms and Theory

• 6. Unconstrained Nonlinear Optimization
  • 6.1. Linear Algebra Review and SciPy Basics
  • 6.2. Mathematics Primer
  • 6.3. Unconstrained Optimality Conditions
  • 6.4. Newton-type Methods for Unconstrained Optimization
  • 6.5. Quasi-Newton Methods for Unconstrained Optimization
  • 6.6. Descent and Globalization
• 7. Constrained Nonlinear Optimization
  • 7.1. Convexity Revisited
  • 7.2. Local Optimality Conditions
  • 7.3. Analysis of KKT Conditions
  • 7.4. Constraint Qualifications
  • 7.5. Second Order Optimality Conditions
  • 7.6. NLP Diagnostics with Degeneracy Hunter
  • 7.7. Simple Netwon Method for Equality Constrained NLPs
  • 7.8. Inertia-Corrected Netwon Method for Equality Constrained NLPs
• 8. Special Topics
  • 8.1. Integer Programming with Simple Branch and Bound
  • 8.2. MINLP Algorithms
  • 8.3. Deterministic Global Optimization

Student Contributions

• More Pyomo Examples
  • Semiconductor Production Planning
  • Optimization of Daily Diet Using Pyomo
  • Blending Under Uncertainty
  • Vehicle Routing
  • Learning Objectives
  • Risk Measures and Portfolio Optimization: Expanded
  • Learning Objectives
  • The Base Problem
  • Extended Race Car Optimization Models
  • Hot Air Balloon Dynamic Control
  • Chemical Reactor Design
• Derivative-Free Optimization
  • Bayesian Optimization Tutorial 1
  • Bayesian Optimization Tutorial 2
• Stochastic Gradient Descent
  • Stochastic Gradient Descent Tutorial 1
  • Stochastic Gradient Descent Tutorial 2
  • Functions and Utilities
  • Implementation of the Algorithm
  • Regression Example: Fitting a Quadratic Function to a 2D Data Set
  • Binary Classification: Logistic Regression Example
  • Conclusions
  • References
  • Stochastic Gradient Descent Tutorial 3
• Machine Learning and Applied Statistics
  • Expectation Maximization Algorithm and MAP Estimation