Calculus of Variations and Optimal Control Theory: A Concise Introduction - Hardcover

Liberzon, Daniel

Hardcover

ISBN 10: 0691151873 ISBN 13: 9780691151878

Publisher: Princeton University Press, 2012

Synopsis

This textbook offers a concise yet rigorous introduction to calculus of variations and optimal control theory, and is a self-contained resource for graduate students in engineering, applied mathematics, and related subjects. Designed specifically for a one-semester course, the book begins with calculus of variations, preparing the ground for optimal control. It then gives a complete proof of the maximum principle and covers key topics such as the Hamilton-Jacobi-Bellman theory of dynamic programming and linear-quadratic optimal control. Calculus of Variations and Optimal Control Theory also traces the historical development of the subject and features numerous exercises, notes and references at the end of each chapter, and suggestions for further study. * Offers a concise yet rigorous introduction * Requires limited background in control theory or advanced mathematics * Provides a complete proof of the maximum principle * Uses consistent notation in the exposition of classical and modern topics * Traces the historical development of the subject * Solutions manual (available only to teachers) Leading universities that have adopted this book include: * University of Illinois at Urbana-Champaign ECE 553: Optimum Control Systems * Georgia Institute of Technology ECE 6553: Optimal Control and Optimization * University of Pennsylvania ESE 680: Optimal Control Theory * University of Notre Dame EE 60565: Optimal Control

About this title

About the Author

Daniel Liberzon is associate professor of electrical and computer engineering at the University of Illinois, Urbana-Champaign. He is the author of "Switching in Systems and Control".

From the Back Cover

"A very scholarly and concise introduction to optimal control theory. Liberzon nicely balances rigor and accessibility, and provides fascinating historical perspectives and thought-provoking exercises. A course based on this book will be a pleasure to take."--Andrew R. Teel, University of California, Santa Barbara

From the Inside Flap

"A very scholarly and concise introduction to optimal control theory. Liberzon nicely balances rigor and accessibility, and provides fascinating historical perspectives and thought-provoking exercises. A course based on this book will be a pleasure to take."--Andrew R. Teel, University of California, Santa Barbara

Excerpt. © Reprinted by permission. All rights reserved.

Calculus of Variations and Optimal Control Theory

A Concise IntroductionBy Daniel Liberzon

PRINCETON UNIVERSITY PRESS

Copyright © 2012 Princeton University Press
All right reserved.
ISBN: 978-0-691-15187-8

Contents

Preface..............................................................xiii1 Introduction.......................................................12 Calculus of Variations.............................................263 From Calculus of Variations to Optimal Control.....................714 The Maximum Principle..............................................1025 The Hamilton-Jacobi-Bellman Equation...............................1566 The Linear Quadratic Regulator.....................................1807 Advanced Topics....................................................200Bibliography.........................................................225Index................................................................231

Chapter One

Introduction

1.1 OPTIMAL CONTROL PROBLEM

We begin by describing, very informally and in general terms, the class of optimal control problems that we want to eventually be able to solve. The goal of this brief motivational discussion is to fix the basic concepts and terminology without worrying about technical details.

The first basic ingredient of an optimal control problem is a control system. It generates possible behaviors. In this book, control systems will be described by ordinary differential equations (ODEs) of the form

[??] = f(t, x, u), x(t₀) = _x0 (1.1)

where x is the state taking values in Rⁿ, u is the control input taking values in some control set U [subset] R^m, t is time, t₀ is the initial time, and x₀ is the initial state. Both x and u are functions of t, but we will often suppress their time arguments.

The second basic ingredient is the cost functional. It associates a cost with each possible behavior. For a given initial data (t₀; x₀), the behaviors are parameterized by control functions u. Thus, the cost functional assigns a cost value to each admissible control. In this book, cost functionals will be denoted by J and will be of the form

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (1.2)

where L and K are given functions (running cost and terminal cost, respectively), t_f is the final (or terminal) time which is either free or fixed, and xf := x(t_f) is the final (or terminal) state which is either free or fixed or belongs to some given target set. Note again that u itself is a function of time; this is why we say that J is a functional (a real-valued function on a space of functions).

The optimal control problem can then be posed as follows: Find a control u that minimizes J(u) over all admissible controls (or at least over nearby controls). Later we will need to come back to this problem formulation and fill in some technical details. In particular, we will need to specify what regularity properties should be imposed on the function f and on the admissible controls u to ensure that state trajectories of the control system are well defined. Several versions of the above problem (depending, for example, on the role of the final time and the final state) will be stated more precisely when we are ready to study them. The reader who wishes to preview this material can find it in Section 3.3.

It can be argued that optimality is a universal principle of life, in the sense that many|if not most|processes in nature are governed by solutions to some optimization problems (although we may never know exactly what is being optimized). We will soon see that fundamental laws of mechanics can be cast in an optimization context. From an engineering point of view, optimality provides a very useful design principle, and the cost to be minimized (or the profit to be maximized) is often naturally contained in the problem itself. Some examples of optimal control problems arising in applications include the following:

• Send a rocket to the moon with minimal fuel consumption.

• Produce a given amount of chemical in minimal time and/or with minimal amount of catalyst used (or maximize the amount produced in given time).

• Bring sales of a new product to a desired level while minimizing the amount of money spent on the advertising campaign.

• Maximize throughput or accuracy of information transmission over a communication channel with a given bandwidth/capacity.

The reader will easily think of other examples. Several specific optimal control problems will be examined in detail later in the book. We briey discuss one simple example here to better illustrate the general problem formulation.

Example 1.1 Consider a simple model of a car moving on a horizontal line. Let x [member of] R be the car's position and let u be the acceleration which acts as the control input. We put a bound on the maximal allowable acceleration by letting the control set U be the bounded interval [-1; 1] (negative acceleration corresponds to braking). The dynamics of the car are [??] = u. In order to arrive at a first-order differential equation model of the form (1.1), let us relabel the car's position x as x₁ and denote its velocity [??] by x₂. This gives the control system [??]₁ = x₂, [??]₂ = u with state [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. Now, suppose that we want to "park" the car at the origin, i.e., bring it to rest there, in minimal time. This objective is captured by the cost functional (1.2) with the constant running cost L [equivalent to] 1, no terminal cost (K [equivalent to] 0), and the fixed final state [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. We will solve this optimal control problem in Section 4.4.1. (The basic form of the optimal control strategy may be intuitively obvious, but obtaining a complete description of the optimal control requires some work.)

In this book we focus on the mathematical theory of optimal control. We will not undertake an in-depth study of any of the applications mentioned above. Instead, we will concentrate on the fundamental aspects common to all of them. After finishing this book, the reader familiar with a specific application domain should have no difficulty reading papers that deal with applications of optimal control theory to that domain, and will be prepared to think creatively about new ways of applying the theory.

We can view the optimal control problem as that of choosing the best path among all paths feasible for the system, with respect to the given cost function. In this sense, the problem is infinite-dimensional, because the space of paths is an infinite-dimensional function space. This problem is also a dynamic optimization problem, in the sense that it involves a dynamical system and time. However, to gain appreciation for this problem, it will be useful to first recall some basic facts about the more standard static finitedimensional optimization problem, concerned with finding a minimum of a given function f : Rⁿ -> R. Then, when we get back to infinite-dimensional optimization, we will more clearly see the similarities but also the differences.

The subject studied in this book has a rich and beautiful history; the topics are ordered in such a way as to allow us to trace its chronological development. In particular, we will start with calculus of variations, which deals with path optimization but not in the setting of control systems. The optimization problems treated by calculus of variations are infinite-dimensional but not dynamic. We will then make a transition to optimal control theory and develop a truly dynamic framework. This modern treatment is based on two key developments, initially independent but ultimately closely related and complementary to each other: the maximum principle and the principle of dynamic programming.

1.2 SOME BACKGROUND ON FINITE-DIMENSIONAL OPTIMIZATION

Consider a function f : Rⁿ -> R: Let D be some subset of Rⁿ, which could be the entire Rⁿ. We denote by | ? | the standard Euclidean norm on Rⁿ.

A point x^* [member of] D is a local minimum of f over D if there exists an ε > 0 such that for all x [member of] D satisfying |x - x^*| < ε we have

f(x^*) ≤ f(x). (1.3)

In other words, x^* is a local minimum if in some ball around it, f does not attain a value smaller than f(x^*). Note that this refers only to points in D; the behavior of f outside D is irrelevant, and in fact we could have taken the domain of f to be D rather than Rⁿ.

If the inequality in (1.3) is strict for x [not equal to] x*, then we have a strict local minimum. If (1.3) holds for all x [member of] D, then the minimum is global over D. By default, when we say "a minimum" we mean a local minimum. Obviously, a minimum need not be unique unless it is both strict and global.

The notions of a (local, strict, global) maximum are defined similarly. If a point is either a maximum or a minimum, it is called an extremum. Observe that maxima of f are minima of -f, so there is no need to develop separate results for both. We focus on the minima, i.e., we view f as a cost function to be minimized (rather than a profit to be maximized).

1.2.1 Unconstrained optimization

The term "unconstrained optimization" usually refers to the situation where all points x sufficiently near x^* in Rⁿ are in D, i.e., x* belongs to D together with some Rⁿ-neighborhood. The simplest case is when D = Rⁿ, which is sometimes called the completely unconstrained case. However, as far as local minimization is concerned, it is enough to assume that x* is an interior point of D. This is automatically true if D is an open subset of Rⁿ.

First-order necessary condition for optimality

Suppose that f is a ITLITL¹ (continuously differentiable) function and x* is its local minimum. Pick an arbitrary vector d [member of] Rⁿ. Since we are in the unconstrained case, moving away from x* in the direction of d or -d cannot immediately take us outside D. In other words, we have x^* + αd [member of] D for all α [member of] R close enough to 0.

For a fixed d, we can consider f(x^* + αd) as a function of the real parameter α, whose domain is some interval containing 0. Let us call this new function g:

g(α) := f(x* + αd): (1.4)

Since x* is a minimum of f, it is clear that 0 is a minimum of g. Passing from f to g is useful because g is a function of a scalar variable and so its minima can be studied using ordinary calculus. In particular, we can write down the first-order Taylor expansion for g around α = 0:

g(α) = g(0) + g'(0)α + o(α) (1.5)

where o(α) represents "higher-order terms" which go to 0 faster than α as α approaches 0, i.e.,

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (1.6)

We claim that

g'(0) = 0: (1.7)

To show this, suppose that g'(0) [not equal to] 0. Then, in view of (1.6), there exists an ε > 0 small enough so that for all nonzero α with |α| < ε, the absolute value of the fraction in (1.6) is less than |g'(0)|. We can write this as

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

For these values of α, (1.5) gives

g(α) - g(0) < g'(0)α + |g'(0)α|. (1.8)

If we further restrict α to have the opposite sign to g'(0), then the right-hand side of (1.8) becomes 0 and we obtain g(α) - g(0) < 0. But this contradicts the fact that g has a minimum at 0. We have thus shown that (1.7) is indeed true.

We now need to re-express this result in terms of the original function f. A simple application of the chain rule from vector calculus yields the formula

g'(α) = ∇f(x^* + αd) · d (1.9)

where

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

is the gradient of f and · denotes inner product. Whenever there is no danger of confusion, we use subscripts as a shorthand notation for partial derivatives: [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]. Setting α = 0 in (1.9), we have

g'(0) = ∇f(x^*) · d (1.10)

and this equals 0 by (1.7). Since d was arbitrary, we conclude that

∇ f(x^*) = 0 (1.11)

This is the first-order necessary condition for optimality.

A point x* satisfying this condition is called a stationary point. The condition is first-order" because it is derived using the first-order expansion (1.5). We emphasize that the result is valid when f [member of] ITLITL¹ and x^* is an interior point of D.

We now derive another necessary condition and also a sufficient condition for optimality, under the stronger hypothesis that f is a ITLITL² function (twice continuously differentiable).

First, we assume as before that x^* is a local minimum and derive a necessary condition. For an arbitrary fixed d [member of] Rⁿ, let us consider a Taylor expansion of g(α) = f(x^* + αd) again, but this time include second-order terms:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (1.12)

where

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (1.13)

We know from the derivation of the first-order necessary condition that g0(0) must be 0. We claim that

g"(0) ≥ 0: (1.14)

Indeed, suppose that g"(0) < 0. By (1.13), there exists an ε > 0 such that

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

For these values of α, (1.12) reduces to g(α) - g(0) < 0, contradicting that fact that 0 is a minimum of g. Therefore, (1.14) must hold.

What does this result imply about the original function f? To see what g"(0) is in terms of f, we need to differentiate the formula (1.9). The reader may find it helpful to first rewrite (1.9) more explicitly as

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

Differentiating both sides with respect to α, we have

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

where double subscripts are used to denote second-order partial derivatives. For α = 0 this gives

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

or, in matrix notation,

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (1.15)

where

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

is the Hessian matrix of f. In view of (1.14), (1.15), and the fact that d was arbitrary, we conclude that the matrix ∇²f(x^*) must be positive semidefinite:

∇²f(x^*) ≥ 0 (positive semidefinite)

This is the second-order necessary condition for optimality.

Like the previous first-order necessary condition, this second-order condition only applies to the unconstrained case. But, unlike the first-order condition, it requires f to be ITLITL² and not just ITLITL¹. Another difference with the first-order condition is that the second-order condition distinguishes minima from maxima: at a local maximum, the Hessian must be negative semidefinite, while the first-order condition applies to any extremum (a minimum or a maximum).

Strengthening the second-order necessary condition and combining it with the first-order necessary condition, we can obtain the following second- order sufficient condition for optimality: If a C2 function f satis_es

∇f(x^*) = 0 and ∇²f(x^*) > 0 (positive definite) (1.16)

on an interior point x^* of its domain, then x^* is a strict local minimum of f. To see why this is true, take an arbitrary d [member of] Rn and consider again the second-order expansion (1.12) for g(α) = f(x^* + αd). We know that g'(0) is given by (1.10), thus it is 0 because ∇f(x^*) = 0. Next, g"(0) is given by (1.15), and so we have

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (1.17)

(Continues...)

Excerpted from Calculus of Variations and Optimal Control Theoryby Daniel Liberzon Copyright © 2012 by Princeton University Press. Excerpted by permission of PRINCETON UNIVERSITY PRESS. All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.

Edition details

• Publisher: Princeton University Press
• Publication date: 2012
• ISBN 10: 0691151873
• ISBN 13: 9780691151878
• Binding: Hardcover
• Language: English
• Number of pages: 256