History of Optimization

What comes to mind when you think of the word “optimize?”

The Merriam-Webster defines it as to make a situation or use of a resource as perfect, effective, or functional as possible.

Human beings have been optimizing for centuries, as it can apply to almost any situation that eases day-to-day life. The idea of optimization can be traced to hunter-gatherer societies where efficient survival was important. This could take form maximizing the harvest given a certain amount of land or a predator minimizing the exerted energy while hunting. Nowadays, we are more concerned with issues like traveling the furthest distance using the least amount of gas.

From these examples and definitions, the concept of optimization can be narrowed down to determining the minimum (or maximum) of some set of requirements to fulfill another set of requirements.

Definition: Minimum

The minimum is usually thought of as the least something. However, the least amount of anything in the real world must be 0 since it is impossible to have a negative amount of a physical quantity. For example, the least amount of candy you can get on Halloween is zero pieces (unless you have a sneaky sibling who steals your candy). Therefore, we must place limits or constraints on our parameters to determine what a feasible minimum or maximum can be. We’ll review these concepts in more detail later.

Test Your Knowledge: Think of some ways you have potentially used optimization in your life, they don’t necessarily have to be complex situations

  • Taking the shortest route to a destination

  • Making the cheapest meal with the most nutritional value

  • Fencing in the greatest area with least amount of material

  • For a more complex examples take note of the following:

    During WWII, engineers were deciding where to add armor to planes so that they would survive attacks. Planes that returned from often had bullet holes in the fuselage and wings so one would naturally assume that is where they are most often hit and should be most protected. However, a statistician Abraham Wald disagreed, determining that the armor should be placed on the engine and nose where there were little too few bullets. Before revealing the reason, think about why Wald was correct. Simply, the planes that had bullet holes in the engine and nose were getting shot down and never returned. The sample size that the engineers were looking at did not consider the planes that never survive battle.

Although the basis of the idea of optimization is simple and almost human nature, it is difficult to precisely say when the concept was fully realized as an application. It can be traced back to as early as the Greeks with relatively simple geometric problems like optimizing the largest area given a set perimeter. Others give credit to Newton for his study in minimal resistance in which he sought a shape that minimized drag with potential applications to ship designs.

The origins of optimization were put into text by Leonis Vitalyevich in 1939 with his monograph “Mathematical Methods for Organization and Planning of Production.”

However, the application of optimization was used on a wide scale during World War II and linear programming began to grow in popularity. Essentially, officials realized that a key factor in fighting the war was the coordination of energy and resources. The calculations required to solve the problems posed by the US military often required many resources and time. With the advent of electronic computing, formal optimization problems with constraints became feasible.

Definition: Linear Programming

Linear programming views a system as a series of inputs and outputs. These can include labor and materials as inputs and goods or services as outputs. What are some other input-output scenarios you can think of?

Optimization is an ever-growing field with numerous practical applications. A current and widely known example of a system using optimization to function is ChatGPT. The AI uses various optimization techniques to comb through data and output answers promptly. If we were to restate how ChatGPT operates using optimization it would be along the lines of reducing time taken to provide an answer. This is achieved through methods including caching, pruning, and scaling horizontally.

The idea of a fully-fledged optimization problem and constraints may seem unfamiliar at first, but we’ll delve deeper into specifics in later sections.

Today, optimization is used in a variety of fields from economics to mathematics to engineering.

If you’d like to learn more about current techniques and applications of optimization, feel free to explore the other sections of this website to learn more.