Optimization Modeling with Applications

Recommended Reading¶

Please review the following book chapters (in order):

1. Chapter 1 of Biegler (2010) introduces classes of optimization problems motivated by applications.

2. Chapters 1 and 2 in Bynum et al. (2021) provide an overview of Pyomo and optimization modeling.

3. Chapters 3 and 4 in Bynum et al. (2021) describe core Pyomo features through examples.

4. Chapter 7 in Bynum et al. (2021) describes special considerations for nonlinear programs.

5. Chapter 8 in Bynum et al. (2021) describes structured modeling with blocks.

6. Chapter 11 in Bynum et al. (2021) describes generalized disjunctive programming (logical decisions).

7. Chapter 12 in Bynum et al. (2021) describes optimization with differential algebraic equations (DAEs).

8. Chapter 10 in Biegler (2010) provides mathematical background for DAE-constrained optimization.

For stochastic programming, read Birge and Louveaux instead of the Pyomo book. The third edition dropped the PySP chapter, and PySP is no longer shipped with Pyomo; the capability now lives in the separate mpi-sppy package.

Reference: Bynum, M. L., Hackebeil, G. A., Hart, W. E., Laird, C. D., Nicholson, B. L., Siirola, J. D., Watson, J.-P., and Woodruff, D. L. Pyomo — Optimization Modeling in Python, Third Edition. Springer Optimization and Its Applications, Vol. 67, 2021.

From a situation to a mathematical model¶

An optimization model specifies decision variables, a feasible set defined by equations/inequalities/domains, and one scalar objective measuring performance. For $f:S\subseteq\mathbb R^n\to\mathbb R$, a global minimizer $\mathbf x^*\in S$ satisfies $f(\mathbf x^*)\le f(\mathbf x)$ for every $\mathbf x\in S$. A minimizer need not exist; the definition does not guarantee one.

Over the same feasible set,

Negation reverses the objective ordering. This is a mathematical equivalence, not a restriction that software must always minimize: Pyomo explicitly supports either objective sense.

“The process of identifying objective, variables, and constraints for a given problem is known as modeling. Construction of an appropriate model is the first step — sometimes the most important step — in the optimization process.”

Nocedal and Wright (2006), p. 2. Biegler (2010), §§1.1–1.2, pp. 1–4, motivates the same components through applications. A predictive model links adjustable quantities to outcomes; constraints encode physical laws, resource limits and admissible actions. Domain expertise and stakeholder priorities determine what belongs in those expressions. A technically accurate solve can still answer an unhelpful question if the objective or feasible alternatives were chosen poorly.

Implementation tools, including AI assistants, can help with the mechanics once a model is stated. Domain knowledge, stakeholder priorities and checking the formulation remain part of the analyst’s work.

Vessel sizing: the full elimination argument¶

Choose the diameter $D$ and length $L$ of a closed cylindrical vessel. Given $V>0$ and positive side/end area costs $c_s,c_t$,

There are two circular ends, whose total area is $\pi D^2/2$. Positive fixed volume forces $D,L>0$. Eliminating length gives $L=4V/(\pi D^2)$ and

Differentiate and solve:

The intermediate length substitution is

Dividing by $D^*=(4V/\pi)^{1/3}(c_s/c_t)^{1/3}$ gives $L^*/D^*=c_t/c_s$. More expensive ends favor a relatively longer, narrower vessel under this model. This ratio does not account for manufacturing, pressure-vessel design, structural, packaging or other omitted constraints.

The positive critical point is the unique positive-domain global minimum: $C''(D)=8c_sV/D^3+\pi c_t>0$ and $C(D)\to+\infty$ as $D\downarrow0$ or $D\to\infty$. The right panel plots the actual normalized expression $C(D)/C(D^*)=(2/q+q^2)/3$, $q=D/D^*$.

Source: Biegler (2010), Eqs. (1.34)–(1.36), pp. 13–14. The additional positive-domain argument states why the critical point solves the model rather than merely satisfying a derivative equation.

Communities and three kinds of decisions¶

These are overlapping interests, not exclusive professional boundaries. Biegler (2010), §1.1, p. 2, supplies the introductory community perspective; the broader application list reflects this cross-listed course.

A battery’s control schedule is an action decision; estimating efficiency is inference; choosing an informative charge/discharge experiment is an information decision. For an application in your discipline, name the adjustable quantities, the objective and where the data come from. The same mathematical machinery can support all three roles.

Taxonomy of Optimization Problems¶

Reference: Chapter 1 in Biegler (2010).

The chart below organizes optimization problems by two questions: are any of the decision variables discrete, and are the objective and constraints differentiable? Those two answers largely determine which algorithms apply, which is why we start here.

Three things in this chart are worth pausing on.

Each branch ends in a chain of inclusions, read downward from the most general. On the convex side, the arrows express representability after a suitable reformulation: second-order cones admit semidefinite representations; convex QPs with affine constraints, and convex quadratically constrained QPs, admit second-order-cone epigraph formulations; and LPs are QPs with a zero quadratic term. The QP rung in this convex chain excludes nonconvex QPs. SOCP and SDP are the conic programs, and this chain is where they fit: they are the natural continuation of LP and QP, not a separate kingdom. We will not cover them, but it is worth knowing the ladder exists — a surprising number of engineering problems (robust design, some control synthesis, relaxations of nonconvex QPs) turn out to be conic once written the right way.

The mixed-integer side nests the same way. MINLP $\supseteq$ MIQP $\supseteq$ MILP $\supseteq$ IP. An MIQP is just an MILP that has been allowed a quadratic objective; it earns a box because it is by far the most common MINLP you will meet in practice (portfolio selection with a cardinality limit, best-subset regression, hybrid MPC with a quadratic cost). We solve MIQPs in this course but do not study algorithms specific to them.

The bottom band is not part of the tree. This is the point students most often miss, and the reason the band is drawn as a different kind of object. Dynamic optimization, parameter estimation and experimental design, stochastic programming, logical models, decision support, and robust optimization are modeling paradigms — ways of writing a problem down. They are orthogonal to the LP/QP/NLP/MINLP axis. Concretely, each one is realized as one of the classes above:

Application labels do not determine the optimization class. Write the expressions and domains first, then classify the resulting finite-dimensional formulation. The robust examples above require the stated static linear structure; the uncertainty-set shape alone does not classify every robust problem. Robust optimization is a further-reading topic in this course.

Adapted from Figure 1.1 of Biegler (2010). The shading, the conic and mixed-integer inclusion chains, and the modeling-paradigm band are additions for this course.

Formulation templates and dimensions¶

These are representative forms; auxiliary/slack variables allow other equivalent forms. Set $\mathbf x\in\mathbb R^n$, $\mathbf A\in\mathbb R^{m\times n}$, $\mathbf b\in\mathbb R^m$ and $\mathbf f\in\mathbb R^n$. Bounds may be infinite.

LP:

To represent $x_1+x_2\le c$, introduce either (i) an auxiliary variable $S_1$ with both $x_1+x_2=S_1$ and $S_1\le c$, or (ii) a conventional nonnegative slack $s$ with $x_1+x_2+s=c$, $s\ge0$. The equality alone, with an unrestricted new variable, does not impose the original inequality. See Nocedal and Wright (2006), p. 357.

QP: use the same affine constraints and a quadratic objective:

The factor $1/2$ is a coefficient convention; a formulation written without it uses half the coefficient matrix to represent the same objective. Convexity tests and their implications belong to the continuous-optimization guide and the later theory references. Specialized LP/QP solvers exist; the algorithms portion of this course concentrates on general nonlinear methods.

NLP: continuous variables with potentially nonlinear functions:

Here $f:D\subseteq\mathbb R^n\to\mathbb R$, $\mathbf h:D\to\mathbb R^m$, and $\mathbf g:D\to\mathbb R^q$ are defined on the stated domain. Bounds can be included in $\mathbf g$. Not every expression must be nonlinear; LPs and QPs are structured special cases. Smooth NLP methods require the corresponding differentiability assumptions.

MILP: a common binary form is

$\mathbf B\in\mathbb R^{m\times r}$ and $\mathbf c\in\mathbb R^r$ are data. General integer variables are also allowed in MILP/MINLP; binary variables are the common on/off special case. Integrality changes the domain without requiring nonlinear expressions.

MINLP: the general binary form allows $f,\mathbf h,\mathbf g$ to depend nonlinearly on both kinds of decisions:

A structured form instead uses $f(\mathbf x)+\mathbf c^\top\mathbf y$, affine coupling $\mathbf A\mathbf x+\mathbf B\mathbf y=\mathbf b$, continuous-only nonlinear rows $\mathbf h(\mathbf x)=0$, $\mathbf g(\mathbf x)\le0$, and bounds. Integer decisions then appear only affinely in the expressions. Decomposition methods can exploit this structure, subject to further convexity, feasibility and duality assumptions; it is not a general promise that every such instance is easy. See Biegler (2010), Eq. (1.1), p. 3, and the MINLP algorithms reference.

Dynamic optimization is a modeling paradigm¶

For a horizon $T>0$, choose state, control and algebraic trajectories $\mathbf x(t)$, $\mathbf u(t)$ and $\mathbf z(t)$ to minimize

$\ell$ is a scalar running cost; $\mathbf f$ has the state dimension; $\mathbf a$ represents algebraic equations and $\mathbf r$ path restrictions, with their stated row dimensions. Initial conditions must be consistent with the algebraic model. Terminal costs or constraints are added when the application requires them. The original Biegler notation uses different letters/signs for the algebraic and path rows; this reference uses $\mathbf a=0$, $\mathbf r\le0$ to distinguish them from the general NLP’s equality/inequality functions. See Biegler (2010), p. 5 and Eqs. (8.5b)–(8.5c), p. 216.

The decisions here are functions, not just a vector of scalar values. Discretization produces a finite-dimensional optimization problem whose class depends on the expressions and variable domains. Examples include reactor/building/vehicle MPC, batch temperature/feed recipes, energy-storage schedules, spacecraft or robot trajectories, moving-horizon estimation and fitting kinetic parameters from dynamic measurements. Detailed DAE properties and numerical methods have their teaching homes in Dynamic Optimization.

Finding a good candidate and proving global optimality are different computational tasks. Global certification can be expensive; a local solution of a nonconvex problem is not automatically a global certificate.

Where These Problem Classes Show Up¶

The table below maps common application areas onto the five problem classes above. Read a row as “problems of this kind are routinely posed as one of these”; most rows have several marks, because the same application admits models at several levels of fidelity, and choosing that level is the modeling decision.

A few rows are worth a comment:

• Parameter estimation is a QP when the model is linear in the parameters and the objective is least squares; it is an NLP as soon as the model is not; and it is an MINLP when you are also choosing which model to fit. This is the subject of its own lecture.

• Machine learning is optimization. Training a support vector machine is a QP; training a neural network with smooth activation and loss functions gives a large, generally nonconvex NLP; verifying a trained ReLU network, or fitting an optimal decision tree, is an MILP; best-subset regression is a mixed-integer quadratic program.

• Power grid dispatch is the one row that touches every column. Economic dispatch with a linear cost curve is an LP and with a quadratic one a QP; unit commitment adds on/off decisions and becomes an MILP; AC optimal power flow is a nonconvex NLP; AC unit commitment is an MINLP. One application, five formulations, chosen by how much physics you keep.

• Hybrid MPC is “MPC with discrete actuators” — a valve that is open or shut, a compressor that is on or off — with linear dynamics and linear cost, this can give an MILP; a quadratic cost gives an MIQP, while nonlinear physics can give a general MINLP.

Some additional examples clarify the modeling choices. A diet LP chooses a least-cost menu subject to nutrient requirements. A classical portfolio QP minimizes $\mathbf x^\top\boldsymbol\Sigma\mathbf x$ subject to a required expected return, $\sum_i x_i=1$ and affine holding bounds. Adding a cardinality limit gives a mixed-integer model: linear return/cost expressions give an MILP, whereas a covariance-risk objective gives an MIQP. Scheduling can choose crew/job assignments for airlines, hospitals or classrooms; unit commitment adds start-up and minimum up/down restrictions. Circle packing and pooling illustrate nonlinear geometric constraints and products of composition and flow; their complete models remain in Continuous Optimization: Nonlinear Programming and Milk Pooling and Blending. Nonlinear alternatives, on/off streams and an economy-of-scale term such as capacity$^{0.6}$ can give MINLPs. These distinctions supplement the table; they do not imply every application has only one valid formulation.

Responses to uncertainty¶

A deterministic optimization run treats its input data as fixed. That does not mean the inputs or the model are known accurately. In the battery example, future prices require forecasts; efficiency and usable capacity can change with age/temperature; state of charge is estimated; and omitted degradation or market rules can matter.

Nonanticipativity means a decision cannot depend on information not yet available. Independent scenario solves may choose different first actions because they have hindsight. Their average optimized profit is an information benchmark, not an implementable schedule. The full mathematical models, seeded historical inputs, 1/6/24-hour commitments, evaluated RP/EEV/WS/VSS/EVPI and rolling-policy results are preserved in How Much Does a Better Forecast Help?. Its fixed weekly experiment and causal rolling experiment deliberately use different information patterns and parameters; use each result with its own stated model.

The deterministic battery reference preserves all model/units/code/results detail for the introductory example. There is no need to duplicate either full builder here.

Statistical inference as optimization¶

Observe $y_i$ under experimental conditions $x_i$, $i\in\mathcal I$, and predict $m(x_i,\boldsymbol\theta)$. The prediction may be science-based, data-driven or hybrid: a kinetic rate law, a smooth neural network, or a physical model with a learned correction. Parameters $\boldsymbol\theta$ that are data when predicting become decision variables when fitting:

The observations/conditions and parameter bounds are given inputs. The residual is measurement minus prediction. The model is a QP when the prediction is affine in the fitted parameters and the remaining constraints are affine; nonlinear dependence can give an NLP. “Least squares” alone does not determine the class or guarantee a unique estimate. See Nocedal and Wright (2006), Eq. (10.1), p. 245.

The illustration uses $x=(0,1,2,3,4)$ and $y=(1.1,1.8,3.2,3.7,5.2)$, with trial prediction $1+x$. The residuals are $(.1,-.2,.2,-.3,.2)$ and their squared sum is .22. For the unconstrained affine fit, the full-rank design gives intercept .98, slope 1.01 and squared residual sum .219. Those are a separate computed illustration, not claimed as a fit to the course’s experimental BOD data.

Two uncertainty questions follow: how does measurement uncertainty propagate into the fitted parameters, and which experimental condition would reduce that uncertainty? Their full treatments stay in Parameter estimation with parmest and Optimizing Experiments with Pyomo.DoE; the introductory role is to distinguish fitting a model from controlling the system or choosing measurements.

Decision science and the limits of a model optimum¶

In this course, decision science means studying how models, data and analytical methods support decisions under uncertainty. This is a course working definition. Optimization is one family of analytical methods; a mathematical optimum alone does not settle the broader decision process.

Driscoll et al. (2023), pp. 148–149, describe the four-phase systems decision process as iterative and centered on decision-maker and stakeholder values. The labels are not an automatic four-step algorithm for an ill-defined situation. The older three-phase shorthand attributed to Herbert Simon, The New Science of Management Decision (1960), is intelligence, design and choice: understanding the problem, developing possible actions and choosing among them. The four-phase systems view makes implementation explicit.

Wong-Parodi et al. (2020) uses the coastal-planning example to examine how quantitative analysis supports deliberation about tradeoffs. The model, institutional process, evidence and stakeholder input work together. The closing decision-support guide gives the explicitly labelled prose paraphrase of the review’s contribution and the historical case’s limits.

Check the framing as well as the computation¶

Ackoff (1979), p. 97, criticizes assessing an optimized model through the same model assumptions: managers’ real choices and the modeled optimum can differ for reasons missing from the formulation. His statement that models simplify reality occurs in that critique, not as proof that any simplified optimum is useful. Check representation and consequences independently of the solver’s internal certificate.

The familiar paired aphorism “All models are wrong, but some are useful” is attributed to George E. P. Box. The precise printed source of that paired wording remains uncertain in the course record; do not treat the held 1976 article as verification of both halves. The methodological issue is whether a model is adequate for the decision and evidence at hand.

Rittel and Webber (1973), pp. 161–162, distinguish public-planning problems with no definitive formulation, no inherent stopping rule and solutions judged good/bad rather than simply true/false. Defining alternatives, constraints and the performance measure is part of that framing work. The claim concerns these planning contexts; it does not imply that feasibility or an optimality certificate for an already specified mathematical model is a matter of opinion. Their argument also challenges prematurely treating a wicked problem as though its goals were settled.

For a battery revenue model that ignores aging, an accurately optimized high-cycle schedule can expose a missing consequence. Discuss the operating priorities and data, then add a justified degradation cost or throughput restriction. Tightening tolerances does not add an omitted cost. A model can fail through poor numerical scaling, violated algorithm assumptions, or a mismatch between the mathematical objective and the actual decision; these are different diagnoses.

Course orientation¶

Local or asymptotic convergence results require hypotheses. They do not alone establish practical performance on an arbitrary model. The final Optimization for Decision Support returns to objective construction, represented alternatives and implementation.

Part I covers continuous, integer/logical, dynamic/DAE/collocation, inference/design and stochastic modeling. Part II supplies linear algebra and real analysis, unconstrained tests and Newton/quasi-Newton methods, line-search/trust-region globalization, constrained optimality and multipliers, constraint qualifications, SQP/interior-point methods, and integer/global algorithms. The mathematical material explains modeling choices rather than replacing domain knowledge. The two midterms and final draw on the open course pack; students are responsible for working the examples and using these full reference pages alongside the shorter lecture activities. Consult the course organization/calendar pages for offering-specific details.

References: Biegler (2010), Chapter 1; Nocedal and Wright (2006), Chapters 1 and 10; Driscoll, Parnell and Henderson (eds., 2023), Decision Making in Systems Engineering and Management, 3rd ed.; Wong-Parodi, Mach, Jagannathan and Sjostrom (2020), Current Opinion in Environmental Sustainability 42, 52–59; Ackoff (1979), Journal of the Operational Research Society 30, 93–104; Rittel and Webber (1973), Policy Sciences 4, 155–169. Full introductory battery market context: Dowling, Kumar and Zavala (2017), as cited in the deterministic battery notebook.