Notation for growth rates

In both ICS 23 and CS 161, a major goal is the design and analysis of efficient solutions to problems. At first it might appear that it would be hard to make precise statements about the amount of time used by our solutions without knowing the details of the machine on which they will be running; on one machine a problem might take an hour to solve, while on a faster machine the same problem might take only a few minutes. Although we may not be able to say exactly how long an algorithm will take, we can often say something about how its time usage will depend on the size of the problem. For example, a procedure to traverse a list might take an amount of time roughly proportional to the length of the list, while an inefficient sorting procedure might take time proportional to the square of the number of elements to be sorted. There are a variety of notations available to enable us to make precise statements about the rate of growth of a function even when we do not know its exact value.

One of the most commonly used notations for growth rate is O-notation. Let f and g be two functions of n. (In this discussion, we'll assume we're only dealing with functions whose domain is the nonnegative integers. Also, we'll always assume that f assumes only nonnegative values.) Formally, when we write g(n) = O(f(n)) this means that

there exists some positive constant c, and

there exists some value n₀, such that

for all n ≥ n₀, |g(n)| ≤ c f(n).

Informally this means that f(n) gives an upper bound on g(n), provided that we ignore constant factors (this is what the c does for us), we ignore finitely many values of n (this is what the n₀ does for us), and we ignore the sign of the function g (because of the absolute value signs).

As an example of the formal use of this notation, let's show that 3n² + 7n = O(n²). Note that in view of the formal definition, this means we will need to show that there exist values of c > 0 and n₀ such that for all n ≥ n₀, 3n²+7n ≤ c n². Note that we don't have any hope of making this true unless we pick c to be greater than 3, so let's try 4. Then we'll need to show that (for large n) 3n² + 7n ≤ 4n², which is equivalent to 7n ≤ n², which is certainly true if n ≥ 7. So, putting it all together, we have that if we pick c=4 > 0 and n₀=7, then for all n ≥ n₀, we have

|3n² + 7n| = 3n² + 7n ≤ 3n² + n² = 4n² = cn², which is just what we need to match the definition.

There are certain types of statements we might like to make which are difficult to make with O-notation. For example, we might want to say that a certain type of sorting algorithm must, in the worst case, use an amount of time at least proportional to n log n. Saying that the worst-case time is "at least O(n log n)" is not appropriate, since O-notation only provides an upper bound. Ω-notation provides a solution to this problem. When we write g(n)=Ω(f(n)) this means that

there exists c > 0 and n₀ such that for all n ≥ n₀, g(n) ≥ c f(n). Informally this means that f(n) gives a lower bound on g(n), provided that we ignore constant factors and ignore small values of n. Note that with this notation the sign of g does matter -- we don't take absolute values. So, now we can say that an algorithm uses Ω(n log n) time in the worst case, and this will mean that the worst-case time usage grows at a rate at least proportional to n log n.

Finally, it is often useful to be able to make both statements of the sort described above at once, i.e., that g is both at least proportional to f and that it is at most proportional to f. When we write g(n) = Θ(f(n)), this means that

there exists c₁ > 0, c₂ > 0, n₀ such that for all n ≥ n₀, c₁ f(n) ≤ g(n) ≤ c₂ f(n).

As an example of the use of this notation, suppose we wanted to describe the growth rate of ∑_{i =1 to n}( i⁴ ). It probably isn't obvious how to obtain an exact formula for this, but if we will be satisified with correct Θ-notation the solution is relatively easy. In view of the definition of Θ-notation, we need to find c₁, c₂, and n₀ such that

for all n ≥ n₀, c₁ f(n) ≤ ∑_{i =1 to n}( i⁴ ) ≤ c₂ f(n). First let's work on the upper bound. Note that ∑_{i =1 to n}( i⁴ ) ≤ ∑_{i =1 to n}( n⁴ ) = n⁵, so if we just let c₂ =1 then the right inequality will hold for any n ≥ 0 if we let f(n)=n⁵. To get a lower bound, we can add up the last half of the terms. Note that

∑_{i =1 to n}( i⁴ )	≥ ∑_{i =[(n+1)/2] to n}( i⁴ )
	≥ ∑_{i =[(n+1)/2] to n}( [(n+1)/2]⁴ )
	≥ [ (n+1)/2 ]⁵
	≥ n⁵/2⁵ = n⁵/32.

Thus if we let c₁ = 1/32, the lower bound will hold as well for n ≥ 0. Thus we can let c₁ = 1/32, c₂ = 1, and n₀ = 0 in the definition to obtain ∑_{i =1 to n}( i⁴ ) = Θ(n⁵).

One final bit of notation is more accurate in that it indicates the appropriate value of the proportionality constant for large n. We write g(n) ~ f(n) if lim_{n to ∞} ( g(n)/f(n) ) = 1. If you review the definition of a limit, you will see that (assuming f(n) > 0 for all n beyond some point) this is equivalent to

for all ε > 0 there exists n₀ such that for all n ≥ n₀, (1 - ε) f(n) ≤ g(n) ≤ (1 + ε) f(n). Note that this is similar to Θ-notation, but makes a much stronger statement since we let ε become arbitrarily small.

Note that O-notation (as well as Ω- and <Θ-notation) has a serious problem. Ordinarily if x=z and y=z we can conclude that x=y. But this clearly doesn't work when we are dealing with O-notation. For example, we have n=O(n) and 2n+9=O(n), but obviously it is not true that n=2n+9. The problem is that the common use of the equality symbol with O-notation is not really correct. A better way to think about it (as in Knuth) is to interpret O(f(n)) as referring to a set of functions, namely, all those g which satisfy the definition. Thus it would be better to write n is a member of O(n) and 2n+9 is a member of O(n). Now you see the above problem goes away: we are just saying that both n and 2n+9 are in the set O(n). Unfortunately, many books and articles (including our text) use the equality notation, so I will follow that convention in class.

The use of these notations varies somewhat from book to book; the definitions we are using are based on Knuth.

Acknowledgement. This discussion is derived from a handout developed by Professor G. Lueker.

Dan Hirschberg
Last modified: Jan 29, 2006