Operator Overloading I Now that we have reviewed the fundamentals of classes, we are going to cover some new material about them: how we can write certain class methods that are used by Python in a special way: we typically don't call these methods directly, but instead Python calls them automatically based on our use of []s, operators, keywords, conversion functions, etc. All these special methods have their names written between double underscores (just like __init__ which is also one of these special methods that we have seen how Python calls automatically). We can call them by their double-underscore name (but more often than not, don't). Most of these methods (but not all) are invoked by Python operators (whose first arguments are instances of the class that they are defined in) so we call this technique "operator overloading": it means overloading (giving more than one meaning to) the standard operators, applying to objects constructed from different programmer-defined classes (which really is an application of "polymorphism", which translates to "many forms": the same operators work on many forms of data). Overloading has a good, not bad, connotation here. By studying operator overloading, we will better understand how Python executes our programs, and be able to write classes that are more powerful, by exploiting our understanding of this mechanism. While learning this material, we illustrate each overloaded operator simply with methods defined in a "simple" class, Vector. Later we will apply this technique to a bigger examples; but, a typical class has only some of its operators overloaded: not all operators are meaningfully overloaded for every class. ------------------------------------------------------------------------------ __init__ The example that we will use throughout this lecture is storing and manipulating Vectors: a mathematical quantity that represents a point in N-dimensional space. We will represent a Vector by a tuple of N coordinates; the length of the tuple (N, its number of coordinates) specifies the dimension of the space in which the vector resides: 2-dimesional, 3-dimensional, etc. We'll start with __init__ because you have probably already written many of these methods. Recall that when you "call the class" (write the class name followed by parentheses), you can supply it with some arguments. Python creates an object (something that has an empty namespace/__dict__) for the instance of the class and then calls __init__ with self as the first parameter (a reference to the empty object it just created) followed by all the other arguments in the "call of the class". So writing Vector(a,b,c) means that Python will call Vector.__init__(self,*args) where self refers to the empty object Python just created and the *args combines all its matching positional arguments into a single tuple. class Vector: def __init__(self,*args): self.coords = args Note that self.coords = ... establishes a new attribute name in the namespace of the constructed object and initializes it to the tuple args. In light of the last lecture, I should have named this self/instance variable _coords or __coords (to indicate no programmer should access that attribute: e.g., not rebind it to a string!), but for simplicity's sake, I'll leave it as coords (I do access that attribute outside the class methods, but only, for illustration purposes, as I do directly below). Like the methods that we will discuss below, we don't call __init__ directly, but Python does, automatically, when we "call the Vector class" to construct a new object from that class. Note that we can also use __init__ another way in Python: we can call __init__ explicitly on any existing Vector object, to reinitialize its coords attribute. v = Vector(0,0) print(v.coords) v.coords = (1,1) print(v.coords) v.__init__(5,5,5,5,5) # We can call __init__ explicitly, like any other method print(v.coords) prints (0, 0) (1, 1) (5, 5, 5, 5, 5) When we discuss inheritance later in the quarter, we will see more examples of explicit calls to the __init__ method. ------------------------------------------------------------------------------ __len__ We can call the len function on any object. It translates into a call of the parameterless __len__ method on the object. The len function is defined, and automatically imported from the builtins module, as approximately the following code: def len(x): return x.__len__() Note that some objects (e.g., those of class/type int) do not define a __len__ method, for example the int class. So, if we call len(1), Python raises an exception and reports: TypeError: object of type 'int' has no len() This is exactly because len(1) calls/returns 1.__len__() which FEOOP translates into type(1).__len__(1) which is int.__len__(1) but the int class defines no __len__ method, so Python reports an error. It might actually be useful for the int class to define len for ints: maybe returning the number of digits (in base 10) for that int. But in Python, len is not defined for ints. In fact, Python requires that the len function always return an integer value, so a more accurate definition of len is def len(x): answer = x.__len__() if type(answer) is not int: raise TypeError(str(type_as_str(x))+' object cannot be interpreted as an integer') return answer Note that str(type(1)) returns the string ""; likewise str(type(Vector(0,0)) returns the string "". The function type_as_str (defined in the goody module) slices this string [8,-2] (from 8 to one before 2nd to last), producing 'int' and '__main__.Vector', respectively . The following code extends our Vector class to illustrate this method. The len method returns the dimension of the vector: the number of values in the tuple used to represent the vector. class Vector: def __init__(self,*args): self.coords = args def __len__(self): print('Calling __len__') return len(self.coords) v = Vector(0,0) print(len(v)) This script prints 'Calling __len__' and then 2. What do you think would happen if we changed the last statement in __len__ from return len(self.coords) to return len(self)? Check your intuition by running this code and justifying the result it produces. Generally we should not call dunder methods explicitly (not always, but generally; __init__ is an exception). For built-in types (like str) calling len(s) is faster than calling x.__len__(); also, as we have just seen, sometimes len does extra processing not done by the dunder method it calls. ------------------------------------------------------------------------------ __bool__ Whenever Python needs to interpret some object as a boolean value, it calls the parameterless method __bool__ on the object. For example, if we write an "if" or "while" statement whose boolean test is just an object, Python attempts to call the __bool__function on that object to determine whether its boolean equivalent represents True or False. Using the definition of Vector above (defining just __init__ and __len__) and the test function below def test(x): print('x\'s boolean equivalent is ') if x: print(True) else: print(False) when we execute the script v = Vector(0,0) test(v) v = Vector(2,2) test(v) v = Vector() test(v) it prints Calling __len__ True Calling __len__ True Calling __len__ False This leads to two questions: 1) How did it compute the boolean equivalent of a Vector? 2) Why is __len__being called? The actual rule in Python for evaluating an object as if it were a boolean value is to first try to call __bool__ on the object (which cannot be done above, because a __bool__ method is not defined for the Vector class yet) and return its result; if that method is not present, Python instead returns whether the object's len is != 0. If there is no __len__ function Python just returns True (the mechanism to do this is actually related to inheritance, a topic we will cover later in the quarter). This rule explains why we can test str, list, tuple, set, and dict objects as booleans: all define __len__, and if any is empty (len = 0) it represents False; if any is not empty, it represents True. Also, the object None has a boolean value of False (the NoneType class specifies a __bool__ method that always return False). So, when v = Vector(0,0) or v = Vector(2,2), len(v) is 2 which is != 0, so it is treated as True. When v = Vector(), len(v) is 0 which is not != 0 so it is treated as False. But suppose that we want to interpret vectors as boolean values differently. We will define Vector so that an instance is True if self.coords is not the origin, regardless of the vector's dimension: so, it is True if any of the coordinates in self.coords is not 0. class Vector: def __init__(self,*args): self.coords = args def __len__(self): print('Calling __len__') return len(self.coords) def __bool__(self): print('Calling __bool__') return any( v!=0 for v in self.coords ) Here we call the any function applied to a tuple of bool values, created by a comprehension, each specifying whether one coordinate in self.coords isn't zero. So the boolean value for Vector(0,0) would return False, Vector() would return False too (because any(()) is False: there are no True values because there are no values! But Vector(2,2) would return True. All three of these examples return their values because Python uses the __bool__ function supplied above, and doesn't have to (nor does it) call the len function to compute an answer. Note, if we wrote return not all( v==0 for v in self.coords ) we would get the same results as above. Note that for Vector(), all(())returns True: not one value is False, because there are no values. This is how the all and any functions work with arguments containing no/0 values. Another possible implementation of len would compute distance of the tip of the Vector from the origin. def __len__(self): return math.sqrt( sum( v**2 for v in self.coords ) ) If we wrote the __len__ method this way, could omit the __bool__ method? Would Python then compute bool(x) by evaluating len(x) != 0; len(x) != 0 is True only when any of its coords are not equal to 0. Unfortunately, this __len__ returns a float value, not the require int, so we cannot use it in our class as a stand-in for __bool__. There is no single "right" __len__ definition for Vector; as the implementers of the Vector class we can define __len__ in almost any way that we want (subject to the constraint of returning an int). The bool function also calls __bool__, but unlike the len function, it does not require a certain return type: we can write a __bool__ function that returns any type (see the "Going really deep into and/or" section below). Finally, if we write a class that defines neither __bool__ nor __len__ then the boolean equivalent of any object constructed from that class is always True. -------------------- Interlude: Short-Circuit Logical Operators: and their real meanings The operators and/or are called short-circuit operators, because unlike other operators (which evaluate both of their operands before being applied), short- circuit operators evaluate only their left operand first, and sometimes don't need to evaluate their right operand: True or ... always evaluates to True and False and ... always evaluates to False. So if the "or" operator's left operand is True, it doesn't need to evaluate its right operand: the result is always True; and if the "and" operator's left operand is False, it doesn't need to evaluate its right operand: the result is always False. This rule sometimes saves execution time (avoids the cost of evaluating the right argument), but it is more important for another reason (discussed below). If the left operands of and/or is True/False it must evaluate the right operand in order to compute its result. So, for example, if we wrote the following if/test (assume d is a dict) if k in d.keys() and d[k] == 0: .... Python would first evaluate the expression: k in d.keys(). When False it would determine that the value of the "and" operator is already known: it is False. It would not have to evaluate d[k] == 0, which is a good thing, because evaluating d[k] would raise a KeyError exception, because k is not a key in d: just what i in d.keys() is checking. So, while short-circuit operators can save a little time, that is not their most important purpose; avoiding raising exceptions is the primary reason that and/or operators are short-circuit. Even without short-circuit operators, we could write this same statement as if k in d.keys(): if d[k] == 0: .... But that requires nested if statements and is much more cumbersome to write. It is better to spend some time learning about and understanding short-circuit logical operators now, and then be able to use them appropriately later, to write simpler and cleaner code. ---------- Going really deep into and/or Finally, the exact meaning of and/or is a bit more subtle, and concerns the boolean interpretation of values (i.e., the __bool__ methods). Let's look at "and" first. If the boolean interpretation (computed by __bool__) of the left operand of "and" is False, we know that we don't have to evaluate the right operand. We said that the result is False, but really the result is the actual value of left operand, not its boolean interpretation. So [] and ... evaluates to [] (a value whose boolean interpretation is False); likewise "" and ... evaluates to "" (a value whose boolean interpretation is False). Likewise if the boolean interpretation (computed by __bool__) of the left operand of "or" is True, we know that we don't have to evaluate the right operand. We said that the result is True, but really the result is the actual value of left operand, not its boolean interpretation. So [1] or ... evaluates to [1] (a value whose boolean value is True); likewise "abc" or ... evaluates to "abc" (a value whose boolean interpretation is True). Similarly, if the boolean interpretation of the left operand of "or" is False, we need to evaluate the right operand to compute the result of "or", and the result will be the value of just the right operand (not its boolean interpretation): so (any value considered False) or [] evaluates to [], and (any value considered False) or [1] evaluates to [1]. Of course Python can determine if this value is interpreted as True or False, if it needs to. Likewise, if the boolean interpretation of the left operand of "and" is True, we need to evaluate the right operand to compute the result of "and", and the result will just be the value of the right operand (not its boolean interpretation): so (any value considered True) and [] evaluates to [], and (any value considered True) and [1] evaluates to [1]. Of course Python can determine if this value is interpreted as True or False, if it is needs to. Most programmers use logical operators where boolean values are needed (like in "if" and "while" loops), so regardless of their strange results, we are interested only in the boolean interpretation of the result (which corresponds to our intuition about and/or). But the reality in Python is a bit subtler, and sometimes programmers do rely on understanding the exact meaning of the and/or operators. Sometimes programmers find interesting uses for these semantics. For example, Suppose last stores the last name of a person, and first their first name, but for some people we don't know their first name (it is stored as either None or ''). Suppose we want to call someone by their first name, but if we don't know it we want to call them by their last name prefaced by Mr. We could write if first != '' and first != None: call_me = first else: call_me = 'Mr. ' + last In fact, because both '' and None have False as their boolean value we can write if first: call_me = first else: call_me = 'Mr. ' + last And if we really want to use our knowledge of Python we can write call_me = first or 'Mr. ' + last Note that because of the "or" operator, if first is not equivalent to the boolean False, it will be assigned to call_me; if it is equivalent to the boolean False, then 'Mr.'+last will be assigned to call_me. Some would argue the single assignment is a bit cryptic: but you should know what it means in Python. Some would prefer call_me = (first if first != '' else 'Mr. '+last) which executes equivalently ---------- Note that "not" has only one operand, so there is no way to short-circuit it. Also, note that "not" always returns a bool value (unlike what we learned above). That is, not [] evaluates to True; not [1] evaluates to False. ------------------------------------------------------------------------------ __repr__ and __str__ (and other conversion functions) Python can call two methods that should return string representations of an object. The __str__ method is called when the conversion function str is called: so again, this is like len: Python translates str(x) into type(x).__str__(x). For example, the print and format functions automatically calls str on all their arguments. If there is no __str__ method in the argument's class, Python tries calling the __repr__ method as a backup to produce the string. If we call repr(x) Python returns x.__repr__() for this method, similarly to what it does for len(x) and str(x)). If there is no __repr__method, Python reverts to its standard method for computing a string value for objects: a bracketed string with the name of the object's class and the location of this object in memory (as a hexadecimal number). In the last lecture we saw that with x = C(), print(x) prints like <__main__.C object at 0x02889C50>. When we cover inheritance, we will see more details about how this happens So in the Vector class as it was defined above, without defining __repr__ or __str__, if we write v = Vector(0,0) print(v) Python prints something like <__main__.Vector object at 0x027CE470> The convention for __repr__ is that it returns a string, which if passed as the argument to the eval function, would produce an object with the same state. So if v = Vector(0,0) the repr(v) should return 'Vector(0,0)'. For Vector, we can define __repr__ as follows def __repr__(self): return 'Vector('+','.join(str(c) for c in self.coords)+')' or def __repr__(self): return 'Vector({})'.format(','.join(str(c) for c in self.coords)) Note that join expects an iterable as its argument, and it expects each object produced to be a string. So, we specify a tuple comprehension that uses the str conversion on every coordinate in self.coords. Now, if we executed the print code above, Python prints the following (there is still no definition of __str__, so __repr__ is ultimately called by print), printing a string value. Vector(0,0) And sure enough, writing x = eval(repr(v)) print(type(x),x) would print Vector(0,0) Here x refers to a different object than v, whose state is the same as v's state: x is v evaluates to False. In fact, similar to the len function, Python requires that the str function always return a str value, so a more accurate definition of str is def str(x): answer = x.__str__() if type(answer) is not str: raise TypeError('__str__ returned non-string (type '+type_as_str(answer)+')) return answer __str__ should return a string that nicely formats interesting aspects (maybe all attribute names, maybe not) of the object. Some objects have more attribute names than those needed to construct the object. The __str__ below just returns the Vector as a list of its coordinates, prefaced by its dimension. So if v = Vector(0,0) then str(v) returns '(2)[0,0]' def __str__(self): return '('+str(len(self))+')'+str(list(self.coords)) # using + #return '({d}){c}'.format(d=len(self),c=list(self.coords)) # using format #return f'({len(self)}){list(self.coords)}' # using f string Unlike __repr__, Python there is no convention for the result that __str__ returns (except that it must be a string), we can use our judgement as to how best to show the string representation of objects. Typically we use __str__ for debugging, when calling the print function, which automatically calls the str function on all its positional arguments. Note that print(repr('abc')) prints 'abc' (including the quotes); print('abc') or print(str('abc')) prints abc (no quotes); eval("'abc'") is the string 'abc'; eval("abc") is whatever object the name abc is bound to (and raises an exception if there is no binding for this attribute name). len(repr('abc')) is 5 (two quotes and 3 characters); len(str('abc')) is 3 (3 characters). When we call print on a list (and other Python types), it shows the repr values of the list contents: print(['a', 'b', 'c']) prints ['a', 'b', 'c']. Finally the other type conversion functions: int, float, complex, oct, hex, and trunc call the Python methods __int__ __float__ __complex__ __oct__ __hex__ __trunc__. So, we could define def __int__(self): return len(self.coords)) If we did, and assumed v = Vector(0,0), then int(v) would return 2; if we did not define any __int__ function, then calling int(v) would raise the following exception: TypeError: int() argument must be a string or a number, not 'Vector' The actual int conversion function looks more like the following, because the int function is required to return an int. def int(x): answer = x.__int__() if type(answer) is not int: raise TypeError('__int__ returned non-string (type '+type_as_str(answer)+')) return answer ------------------------------------------------------------------------------ Relational operators: __lt__ (__gt__, __le__, __ge__, __eq__, __ne__): < (>, <=, >=, ==, and !=) In this section, the overloaded operators that we discuss are really operators, and we overload their meaning to compute values on new classes/types. The most important thing to know is that Python translates any occurrence of a relational operator into a call on the appropriate method for its LEFT operand: x < y is translated to x.__lt__(y) which by the Fundamental Equation of Object Oriented Programming (FEOOP) is translated into type(x).__lt__(x,y) or, assuming Vector is a class and x is of type Vector, Vector.__lt__(x,y). Understanding this translation and application of FEOOP is very important. Understanding this translation and application of FEOOP is very important. Understanding this translation and application of FEOOP is very important. Please note that int is just a class in Python, so it translates 12 < x into 12.__lt__(x) and then into type(12).__lt__(12,x) which is equivalent to int.__lt__(12,x). That is really what is happening in Python when we use the < operator: using < with an integer first/left argument calls the method __lt__ defined in the class the first/left object 12 is constructed from (and 12 is an int). If we wrote "abc" < x it would get translated to "abc".__lt__(x) and then by the FEOOP into str.__lt__("abc",x). So, the type of first/left operand of < determines which class calls its __lt__ method. So < does NOT treat its left and right arguments symmetrically! When studying relational operators, we will first look at comparing objects from the same class, and then we will extend our understanding to how Python compares objects from different classes (which is a bit more subtle). -------------------- 1) Comparing objects from the same class: We will start our discussion by defining these relational operators to work when comparing two objects from the Vector class. Later we will cover how to expand them to compare Vectors against other types (any numeric type). If we wrote x = Vector(0,0) y = Vector(2,2) print(x < y) Python raises the exception: TypeError: unorderable types: Vector() < Vector() Likewise, it raises the same exception for print(x < 10) Python raises the exception: TypeError: unorderable types: Vector() < int And finally, it raises the same exception for print(10 < x) Python raises the exception: TypeError: unorderable types: int < Vector() Unorderable here means that there is no way known to Python to determine whether one Vector object is less than another Vector object, nor whether a Vector is less than an int, nor or whether an int is less than a Vector(). Note that the cause of the problem in the first two cases is that the Vector class defines no __lt__ method. Python translate x < ... into x.__lt__(...) and then into type(x).__lt__(x,...) which is equivalent to Vector.__lt__(x,...). But there is no __lt__ method defined in the Vector class (yet). In the third case the int method does define a __lt__ method, but it does not know how to compare integers to Vectors. Vectors are a type defined by us, so there is no way the int class, which has been built into the Python system, can know about our Vector class nor how we want to compare them to ints. Python translates 10 < x into 10.__lt__(x) and then into type(10).__lt__(10,x) which is equivalent here to int.__lt__(10,x). So, there is no built-in meaning for comparing an object of class/type Vector with any object of any type. There is no Vector.__lt__ method. But, by defining a __lt__ method in Vector, there is a method for Python to call when it needs to compute the < operator on two operands, whose left/first operand is an instance of the Vector class. So to start, let's add the following two definitions to the Vector class. Also assume that we have imported the math module (by import math). Here, distance computes the distance from the origin to the coordinate specified by the Vector; we then implement "less than" such that one Vector is less than another if its distance from the origin is smaller. def distance(self): return math.sqrt( sum( v**2 for v in self.coords ) ) def __lt__(self,right): return self.distance() < right.distance() I named the second parameter here right because it is the value on the right- hand side of the < operator; it can be named anything (e.g., qq17), so long as that same name is used for the second parameter in the body of the function. Notice the < operator in __lt__ is NOT RECURSIVE! Python calls the __lt__ method above when comparing two objects constructed from the Vector class; but inside this method the < operator is called on two float value returned by the sqrt function in distance, so it calls the __lt__ method defined in the float class: self.distance() < right.distance() assuming self.distance() evaluates to self_dist_float and right.distance() evaluates to right_dist_float self_dist_float < right_dist_float self_dist_float.__lt__(right_dist_float) type(self_dist_float).__lt__(self_dist_float , right_dist_float) float.__lt__(self_dist_float , right_dist_float) So, to compute this relational operator for Vectors, Python calls a __lt__ method that computes the same relational operators on floats computed from the Vectors. Thus, the < operator is overloaded to work for different types of operands. The __lt__ method above would be recursive if its body were return self < right (causing infinite recursion, a topic that we will cover in depth later). So now, when evaluating x < y in the example below, the method call Vector.__lt__ is found and it is called with arguments (x,y): as in Vector.__lt__(x,y). x = Vector(0,0) y = Vector(2,2) print(x < y) Python prints: True What if we changed the print to print(x > y) and executed the code? Would Python raise an exception because there is no __gt__ method defined in class Vector? It turns out Python is able to compute the correct answer. How Python evaluates > in this case is a bit more complicated, but by knowing how we can gain extra insight into how Python works and how we can make it work correctly for us. It first tries to call Vector.__gt__(x,y) but an exception is raised internally because __gt__ is not defined in Vector. Mathematically, x > y is true exactly when y < x is true: so, Python tries to use the __lt__ method to compute the __gt__ method, with reversed operands: Vector.__gt__(x,y) is the same as Vector.__lt__(y,x). So Python tries to evaluate y < x, which means Vector.__lt__(y,x) which we defined above and successfully computes its answer: False. So, we can let Python implicitly call __lt__ with the operands reversed, as above, or we can explicitly implement __gt__ either directly (as we did for __lt__) or we can implement __gt__ by having it explicitly call __lt__ with the arguments reversed. We can write either def __gt__(self,right): return self.distance() > right.distance() def __gt__(self,right): return right < self # or right.__lt__(self) or Vector.__lt__(right,self) So there are a variety of implicit and explicit ways in which we can use the __gt__ method to compute the correct result. Finally, note that we can get the same behavior by defining only __gt__, which Python will call with reversed operands if it needs to compute the __lt__ but that method is undefined. Thus, Python can use __lt__ to compute __gt__ and __gt__ to compute __lt__ as needed. So, writing one of these pairs allows Python to compute both relations. In the next section we will compare Vectors to ints/floats and see why it is useful to define both __lt__ and __gt__ explicitly. What if we changed the print to print(x <= y) and executed the code? Python again raises the exception: TypeError: unorderable types: Vector() <= Vector() In fact, the __le__ and __ge__ pair of methods are opposites, like __lt__ and __gt__. If we define the __le__ method, and Python needs to compute x >= y, it will try the call to Vector.__le__(y,x), because x >= y is true exactly when y <= x is true. And if we define the __ge__ method and Python needs to compute x <= y it will translate the call to Vector.__ge__(y,x), because x <= y when y >= x. Finally, for the == operator, if we don't define an __eq__ method, Python substitutes the "is" operator (which always returns a boolean value); if we don't define a __ne__ method Python calls __eq__ and negates the result. So unlike the other pairs, in this pair, an undefined __ne__ method calls __eq__ but an undefined __eq__ WILL NOT CALL __ne__, it uses the "is" operator. For example, with neither __eq__ nor __ne__ defined x=Vector(0,0) y=Vector(0,0) z=x print(x == y, x is y, x == z, x is z) print(x != y, x is not y, x != z, x is not z) Prints False False True True True True False False Here ==/is and !=/is not produce the same results because we have not defined the __eq__ nor __ne__ methods. So even though x and y represent the origin, x == y computes the same result as x is y, which is False because "is" is comparing two different Vector objects (although each stores the same coordinates). ---- is vs. == interlude Recall that the "is" operator is called the object-identity operator. It evaluates to True when its left operand refers to the SAME OBJECT as its right operand. In the pictures that we have been drawing, this means the left/right references (arrows) refer to the same object. When "x is y" is True, id(x)==id(y) and vice versa. The == operator (when present) should compare objects by their state. It is possible that two different objects (objects for which "is" evaluates to False) store the same state, so for them == should evaluate to True. This is the case for the objects x and y shown above: they refer to different objects, but each of the objects referred to has the same state: the self.coord reference of each refers to a tuple containing (0,0). ---- So, if we define the __eq__ method as follows (True when the tuples are same regardless of identity) def __eq__(self,right): return self.coords == right.coords print(x == y, x is y, x == z, x is z) print(x != y, x is not y, x != z, x is not z) now prints True False True True False True False False So in summary, to get all six relational operators to work correctly for comparing two objects of the same class, we can define all six or choose one of __lt__/__gt__, one of __le__/__ge__, and always __eq__ to define, and Python will call one of these to compute the correct value for any of the undefined relational operators. Note that although Python doesn't do it, we can implement all the relational operators using only < (or > or <= or >= we show only < below): although this approach might not be efficient in execution, the last 5 methods would be the same in every class, all depending on the meaning of __lt__. In fact, here is the equivalence (each proven by the law of trichotomy: x < y, x == y, or x > y) Relation | < only + logical operators ---------+------------------------------ x < y | x < y x > y | y < x x <= y | not (y < x) x >= y | not (x < y) x == y | not (x < y or y < x) x != y | x < y or y < x, or we could write not(x == y), using == So, we could write each of these operators as illustrated above and be able to compare values from the Vector class in all possible ways. We will later see how to use a mix-in via class inheritance to simplify this process. You can also look in the functools module in Python's library for the total_ordering decorator, which supplies the other operators so long as the class defines one of __lt__, __le__, __gt__, or __ge__ and also supplies the __eq__ method. But for now, this is only of theoretical interest. -------------------- 2) Comparing objects from different classes/types: Now we will discuss what we must do if we also want Python to be able to compare objects of different types: here, we compare objects from the Vector type with objects of a numeric type (int or float). In the example below, again, assume x refers to an object constructed from the Vector class. Writing the code print(x < 15) won't magically work. As you would suspect, this will not end well, because Python raises the exception: AttributeError: 'int' object has no attribute 'distance' (in the body of the __lt__ method). The current definition assumes that we can call the distance function on the right parameter, which works if type(right) is Vector; but here it is an int. Now, maybe it makes NO sense to do such a comparison, so raising this exception is as good as doing anything else; or, maybe we should test whether type(right) in (int,float) and raise a different exception with a better message. The accepted exception to raise is TypeError, which with the correct string would produce the same result as if the method were not defined (see below). But if it makes sense to compare a Vector and an int/float, we can write more code to fix the "problem", expressing how to compare Vectors to integers/floats. Suppose that we decide for __lt__ to compare the distance of the Vector value with the int/float. For these semantics, we can rewrite __lt__ as def __lt__(self,right): if type(right) is Vector: return self.distance() < right.distance() elif type(right) in (int,float): return self.distance() < right else: return NotImplemented Here, if the type of right object is Vector, we do the standard comparison we did before, but if the type is either int or float, we perform the necessary call (on the distance method only for the Vector) before doing the standard numeric comparison with the right, which we know is a numeric value. If type(right) is neither Vector nor int nor float, we return the value NotImplemented, meaning trying to do the comparison is not valid (signalling Python to see if it can try to compute this relational operator another way). Now it is time to explore an interesting asymmetry when Python evaluates relational operators, and how to avoid any possible problems. ----- (1) If we extend __lt__ to cover a right argument that is an int (as we did above) and do not define an explicit __gt__ method similarly, then Python will try, but fail to compute print(x > 15) correctly: remember that Python would first translate this expression into x.__gt__(15) and then into type(x).__gt__(x,15) which is equivalent to Vector.__gt__(x,15); if Vector.__gt__ is not defined, Python tries to compute x > 15 equivalently by computing 15 < x. But this expression won't work correctly, because Python will translate 15 < x into 15.__lt__(x) and then into type(15).__lt__(15,x) which is equivalent to int.__lt__(15,x); but the int class does not (and cannot -see the next paragraph) know how to compare an int object to a Vector object. Python would raise the exception: TypeError: unorderable types: int() > Vector() (2) If we define a __gt__ method (which WON'T work) def __gt__(self,right): return right < self # or right.__lt__(self) or Vector.__lt__(right,self) then 15 < x would call int.__lt__(15,x) and fail to produce a result, so Python would translate it to x > 15 and call Vector.__gt__(x,15), but that would be computed as 15 < x, which would get us back where we started and cause infinite recursion. Technically, an exception is raised: RuntimeError: maximum recursion depth exceeded. (3) If we define a __gt__ method (which WILL work) def __gt__(self,right): if type(right) is Vector: return self.distance() > right.distance() elif type(right) in (int,float): return self.distance() > right else: return NotImplemented Now Python translates 15 < x into 15.__lt__(x) and into type(15).__lt__(15,x) which is equivalent to int.__lt__(15,x); but the __lt__ method defined in the int class returns NotImplemented or raises an exception when it tries to process the second argument: an object constructed from class Vector which the int class doesn't know about. But in this case, Python next tries to evaluate 15 < x by evaluating x > 15, which translates into x.__gt__(15) and then into type(x).__gt__(x,15) which is equivalent to Vector.__gt__(x,15), which now computes the correct answer (assuming we have defined __gt__ properly in the Vector, as illustrated above). ----- The int/float classes are built into Python and were written BEFORE we wrote the Vector class, so they know nothing about comparing ints/floats to objects constructed from the Vector class. And, we cannot change the int/float class to compare against objects constructed from the Vector class (because they are built into Python, and need to run very efficiently, we cannot modify them as we modified classes written in the previous lecture) . So, it is the Vector class that must know how to compare ints/floats against objects constructed from the Vector class. Thus, if it makes sense to compare objects of the Vector class against ints/floats (and objects of any other already defined classes) we must define all 6 relational operators in class Vector, each one explicitly performing its comparison. Recall that chained relational operators are translated into pairwise comparisons joined implicitly by "and": e.g., a < b < c == d is translated into a < b and b < c and c == d. Most programming languages don't define chained relational operators, so programmers must to this translation themselves. Finally note that we can make a < b do whatever we want. We could make it print values, change the state of a or b, do whatever we can do with Python code. Here is where beginners might go wild by adding all sorts of strange meanings to using the < operator; but more experienced programmers will ensure that all relational operators are pure accessors/queries and probably just return a bool value (not even one that just can be interpreted as a bool value using the __bool__ method) without making any state changes (mutation) in the arguments. We can summarize the rule Python uses to determine how to compute x < y: (1) Try to call x.__lt__(y), which is translated into type(x).__lt__(x,y) (2) If this produces a result, that is the result of the comparison (3) If there is no such __lt__ method, or calling it returns NotImplemented or raises an exception, (which is handled by Python internally), then try to compute y > x: call type(y).__gt__(y,x); if this produces a result, that is the result of the expression (4) If there is no such __gt__ method Python raises a TypeError; if there is a method, but calling it returns NotImplemented or raises an exception, Python propagates the exception (Python doesn't handle it internally) For the main example above, here is how Python does this process. 1> & | ^ Now let's move from unary to binary arithmetic operators, where there are more operators and they are it bit harder to write correctly. We will start by discussing one method in particular __add__ and also discuss the need for another related method __radd__. What we learn about these methods applies identically to all the arithmetic operators, so we will not discuss them in detail here (generally we are concerned with the arithmetic operators + - * / // % **: if you want you can read about the more obscure "arithmetic" operators on Python's documentation. See 6.7 and beyond. Binary arithmetic operators, like relational operators, are written in between their two operands. Python translates the call x + y into x.__add__(y) and then by FEOOP into type(x).__add__(x,y). As with the relational operators and unary arithmetic operators, neither operand should be mutated, and the method should return a new object initialized with the correct state. Here is an example of the + operator overloaded for Vector: to work correctly, the right operand must also be of type Vector and both must have the same number of coordinates (len of the coords attribute) and the resulting Vector object has that common length, with coordinates that are the pairwise sum of the coordinates in the two Vectors. Later we will write code so that + adds Vectors and the values of the int/float numeric types as well. def __add__(self,right): if type(right) is not Vector: return NotImplemented assert len(self) == len(right), 'Vector.__add__: operand self('+str(self)+') has different dimension that operand right('+str(right)+')' return Vector( *(c1+c2 for c1,c2 in zip(self.coords,right.coords)) ) and an example of it running v1 = Vector(0,1) v2 = Vector(2,2) print(v1+v2) which produces the result (2)[2, 3] Note that we know that type(self) is Vector: that is why the __add__ method in the Vector class is called. Note also that executing print(v1+1) raises and prints the exception TypeError: unsupported operand type(s) for +: 'Vector' and 'int' (Python translates returning NotImplemented into raising the TypeError exception) and if we define v3 = Vector(2,2,2), then print(v1+v3) results in the exception AssertionError: Vector.__add__: operand self((2)[0, 0]) has different dimension that operand right((3)[2, 2, 2]) Now recall that we allowed objects from class Vector to compare to int/float using relational operators: let's also allow addition between Vectors and these numeric types. We will defined adding a Vector and a numeric value to add that numeric value to EACH COORDINATE in the Vector. As with the implementation of relational operators, we check types in the __add__ method. def __add__(self,right): if type(right) not in (Vector,int,float): return NotImplemented if type(right) in (int,float): return Vector( *(c+right for c in self.coords) ) else: assert len(self) == len(right), 'Vector.__add__: operand self('+str(self)+') has different dimension that operand right('+str(right)+')' return Vector( *(c1+c2 for c1,c2 in zip(self.coords,right.coords)) ) Note that the assertion about lengths is now moved into the else: it shouldn't be checked when type(right) is an int or float. An an example of it running is v = Vector(0,0) print(v+1) print(v+1.) which produces the results (2)[1, 1] (2)[1., 1.] We can write this code equivalently as follows, puting the Vector case first, then the (int,Float) cases, and finally the error case. Either code is reasonable. def __add__(self,right): if type(right) is Vector: assert len(self) == len(right), 'Vector.__add__: operand self('+str(self)+') has different dimension that operand right('+str(right)+')' return Vector( *(c1+c2 for c1,c2 in zip(self.coords,right.coords)) ) elif type(right) in (int,float): return Vector( *(c+right for c in self.coords) ) else: return NotImplemented What if we also wanted to allow the expression 1+v? If we try to execute this code, unsurprisingly Python responds by raising an exception: TypeError: unsupported operand type(s) for +: 'int' and 'Vector' because Python calls int.__add__(1,v) and the int class has no clue about objects of the Vector class (and we cannot change the int class to process Vectors correctly). Recall that we covered this problem when discussing __lt__: because the class of the left operand was used there too, to call the method (sometimes called "left-operand dispatch"), we would have to change the definition of __lt__ in the int class to know about class Vector, which we cannot do. This problem was solved for relational operators by Python automatically reversing the operands and calling the __gt__ method: 1 < v always has the same value as v > 1, where the __gt__ method for v (defined in class Vector) can include code that checks whether its right operand it an int and process it accordingly. But for binary arithmetic operators, Python cannot always find an equivalent operator to transpose the operands, so it uses a different mechanism to solve this problem. For example, 1-v is not the same as v-1. ----- This is because in mathematics, although operators on some operands (ints and floats with + and *) are COMMUTATIVE, they aren't always. Matrix multiplication is the first example of non-commutivity that mathematics students often see. For matrices m1 and m2, m1*m2 does not produce the same value as m2*m1. Likewise + as concatenation is not commutative on strings: 'ab' + 'cd' is not equal to 'cd' + 'ab'. Because of non-commmutivity, Python solves this problem for arithmetic operators by using a different mechanism than the one it uses for relational operators. ----- For every binary arithmetic operator, Python also allows us to define a "right" (sometimes known as "reversed") version of it: where the method name is prefixed by an r: so __add__ has an related __radd__ method ("right/reverse add"). Here is how we could define __radd__ in the Vector class to successfully compute expressions of the form int() + Vector(). def __radd__(self,left): if type(left) not in (int,float): # see note (1) below return NotImplemented return Vector( *(left+c for c in self.coords) ) # see note (2) below When Python evaluates 1+v, it translates it into 1.__add__(v) and then by FEOOP into tries int.__add(1,v); it doing so returns NotImplmemented or raises an exception because the int class doesn't know how to operate on Vector operands. Then, Python translates the + into v.__radd_(1) and into type(v).__radd__(v,1) using "right/reversed-operand dispatch". This methods determines what to do if the left operand is an int/float. In the method below, relating to the + operator, the self parameter is the right operand and the left parameter is the left operand. We could rewrite the headers as def __add__(left,right) and def __radd__(right,left) and replacing self appropriately in the method bodies. An example of __radd__ running is v = Vector(0,0) print(1+v) which produces the result (2)[1, 1] Note: Two interesting questions are, (1) why doesn't the __radd__ method need to also check whether type(left) is a Vector as does __add__, and (2) why does neither __add__'s nor __radd__'s body check whether self is a Vector. (1) If left were a Vector, then __radd__ would never be called, because the left (Vector) argument would result in Python calling the equivalent of Vector.__add__(left,self) which knows how to compute a Vector result, so it would never call __radd__ when left is a Vector. (2) self must be a Vector because Python is calling __add__ or __radd__ in the Vector class, so the other argument must be a Vector. That is, for Vector v, v+anything first gets translated to Vector.__add__(v,anything) so self here is a Vector; anything+v (if anything doesn't know how to add Vectors) gets translated into Vector.__radd__(v,anything) and again self here is a Vector. A Note on Commutivity to Simplify __radd__: For arithmetic types/operators that are commutative (e.g., a+b == b+a, which is true for Vectors) we can write __radd__ by simply calling __add__ with the arguments reversed: e.g., def __radd__(self,left): return self + left # or self.__add__(left) So 1+v is translated into 1.__add__(v) and then into int.__add(1,v) which fails, so is translated into v.__radd__(1) and into type(v).__radd__(v,1) and into Vector.__radd__(v,1), which returns the result of Vector.__add__(v,1) which is v+1, which has the same value as 1+v because of commutivity. Again, there are many mathematical structures where + and * are commutative, although operators like - and / are typically not commutative. In fact, some interpretations of + and/or * are not commmutative (e.g., multiplying matrices). So, we must be careful to write the normal and right versions of all binary arithmetic operators correctly. We now get a point where we can really test our understanding. Here is the rule Python uses to determine how to compute x + y: (1) Try to call x.__add__(y), which is translated into type(x).__add__(x,y) (2) If this produces a result, that is the result of the expression (3) If there is no such __add__ method, or calling it returns NotImplemented or raises an exception (which is handled by Python internally) then call type(y).__radd__(y,x); if this produces a result, that is the result of the expression (4) If there is no such __radd__ method Python raises a TypeError; if there is a method, but calling it returns NotImplemented or raises an exception, Python propagates the exception (Python doesn't handle it internally) For the main example above, here is how Python sees this process 1+v which is translated by Python to call a method 1.__add__(v) which by FEOOP is translated to type(1).__add__(1,v) which Python translates to int.__add__(1,v) which returns NotImplmented: int doesn't know about Vector; so Python tries v.__radd__(1) which by FEOOP is translated to type(v).__radd__(v,1) which Python translates to Vector.__radd__(v,1) which correctly computes the result for Vector and int You should completely understand this example for the upcoming quiz and midterm. We have now explored (in about 150 lines of text and code) the relationship between the + operator and __add__ and __radd__ methods. Here is the complete list of operators and their related methods: + - * / // % divmod ** << >> & | ^ translate to __add__ __sub__ __mul__ __truediv__ __floordiv__ __mod__ __divmod__ __pow__ __lshift__ __rshift__ __and__ __or__ __xor__; also there is a right form for each of these method: __radd__, __rsub__, __rmul__, etc. One final comment: for the binary arithmetic operator -, if one has defined unary - (__neg__) correctly, one can often implement the __sub__ method as self + -right, which uses the __neg__ and __add__ methods to compute subtraction: again, this can be a bit inefficient, but it is very easy to code. If efficiency is an issue, we can rewrite __sub__ more directly, to execute more quickly. ------------------------------------------------------------------------------ Incrementing Arithmetic Delimiters: += -= *= /= //= %= **= <<= >>= &= |= ^= Again in this section we will look at one incrementing arithmetic delimiter, +=. Technically this is not an operator, because we cannot compose bigger expressions with it: += is more like = (a delimiter) than + (an operator). This same discussion applies similarly to all the other delimiters. For every arithmetic operator method, there is another method prefaced by i that we can write and Python will try to call (e.g. __add__ and __iadd__). Recall that the meaning of x += y is similar to x = x + (y). We parenthesize y in case it is an expression that contains any operators whose precedence is lower than +. When Python executes x += y, it tries to execute x = type(x).__iadd__(x,y): if that method is available and doesn't raise an exception, that is the result; if it cannot find that method or it raises an exception, Python executes the code x = x + (y), which also can fail if __add__ is not defined for the types of x and y. Here is an example of the __iadd__ method for the Vector class, which works for incrementing objects of the Vector type/class by a Vector or an int. Note that Python automatically will bind x to the result returned by this method. def __iadd__(self,right): if type(right) not in (Vector,int,float): return NotImplemented if type(right) in (int,float): return Vector(*(c+right for c in self.coords)) else: assert len(self) == len(right), 'Vector.__add__: operand self('+str(self)+') has different dimension that operand right('+str(right)+')' return Vector(*(c1+c2 for c1,c2 in zip(self.coords,right.coords))) Because Vectors are immutable, this method goes through exactly the same code as executing x = x + y (computing a new Vector, with x losing its reference to its old Vector, so there is no point/advantage of writing it (same for the builtin str type). But what if x is mutable, like a list? The list class specifies the following __iadd__ def __iadd__(self,right): self.extend(right) return self So if x =[0] then x += [1,2] extends to the list x, and then x is rebound to the returned reference to this list. So x is now bound to a list whose state is [0,1,2]. Although the result is the same as x = x + [1,2] the process is different. In the case of x = x + [1,2], Python builds a completely new list containing all of x's values followed by 1 and 2. Then it binds x to refer to this new list, throwing away the old list. In the implementation of __iadd__ above, it just mutates list x and rebinds x to this mutated list, which is faster. Perform this experiment x = [] y = x # x and y share the same empty list x = x + [1,2] print(x, y, x is y) # prints [1, 2] [] False and compare it to the results of this x = [] y = x # x and y share the same empty list x += [1,2] print(x, y, x is y) # prints [1, 2] [1, 2] True Can you draw pictures for these two separate cases? So for mutable objects, we might want to implement __iadd__ to produce an equivalent result more efficiently. If the object is immutable, or there is no faster way to compute it, we can omit defining __iadd__ and let Python compute the result using the __add__ method__. Here is the complete list of arithmetic incrementing delimiters and their related methods: += -= *= /= //= %= **= <<= >>= &= |= ^= translate to __iadd__ __isub__ __imul__ __itruediv__ __ifloordiv__ __imod__ __idivmod__ __ipow__ __lishift__ __irshift__ __iand__ __ior__ __ixor__. ------------------------------------------------------------------------------ Other overloaded operators (coming up) In the next lecture we will discuss container operators (including [index]), the call operator (using ()), the context manager protocol, and managing simple attributes. Next week we will have a lengthy discussion about the intricacies of implementing the iterator protocol (and the __iter__ and __next__ methods). ------------------------------------------------------------------------------ FYI, here is the entire Vector class with all the methods described above. We can put this class in a script and experiment calling its methods. import math from goody import type_as_str class Vector: def __init__(self,*args): self.coords = args def __len__(self): return len(self.coords) def __bool__(self): return all( v==0 for v in self.coords ) def __repr__(self): return 'Vector('+','.join(str(c) for c in self.coords)+')' def __str__(self): return '('+str(len(self))+')'+str(list(self.coords)) # using + #return '({d}){c}'.format(d=len(self),c=list(self.coords)) # using format def distance(self): return math.sqrt( sum( v**2 for v in self.coords ) ) def __lt__(self,right): if type(right) is Vector: return self.distance() < right.distance() elif type(right) in (int,float): return self.distance() < right else: return NotImplemented def __gt__(self,right): if type(right) is Vector: return self.distance() > right.distance() elif type(right) in (int,float): return self.distance() > right else: return NotImplemented def __eq__(self,right): return self.coords == right.coords def __le__(self,right): return self < right or self == right def __ge__(self,right): return self > right or self == right def __neg__(self): return Vector( *(-c for c in self.coords) ) def __pos__(self): return self def __abs__(self): return Vector( *(abs(c) for c in self.coords) ) def __add__(self,right): if type(right) not in (Vector,int,float): return NotImplemented if type(right) in (int,float): return Vector( *(c+right for c in self.coords) ) else: assert len(self) == len(right), 'Vector.__add__: operand self('+str(self)+') has different dimension that operand right('+str(right)+')' return Vector( *(c1+c2 for c1,c2 in zip(self.coords,right.coords)) ) def __radd__(self,left): if type(left) not in (int,float): # see note below return NotImplemented return Vector( *(left+c for c in self.coords) ) # def __iadd__(self,right): # if type(right) not in (Vector,int,float): # return NotImplemented # if type(right) in (int,float): # return Vector( *(c+right for c in self.coords)) # else: # assert len(self) == len(right), 'Vector.__add__: operand self('+str(self)+') has different dimension that operand right('+str(right)+')' # return Vector( *(c1+c2 for c1,c2 in zip(self.coords,right.coords))) ------------------------------------------------------------------------------ Problems: 1a) Using your knowledge of the or operator, and the boolean interpretation of string values, explain what the following statement x = string1 or string2 assigns to x in each of the following cases (of empty and non-empty strings) string1 | string2 ---------+---------- '' | 'a' '' | '' 'a' | '' 'a' | 'b' Rewrite this statement as an equivalent (a) conditional statement and (b) conditional expression. 1b) Write a bool function that takes some object as an argument. If that object is None, return False; if that object's class contains a __bool__ method, call it to return a result; if not, if that object's class contains a __len__ method,, call it to return result (True if the len is not 0); otherwise return True. Hint: recall we can check whether a class object defines a method by checking whether it is stored in the class object's __dict__. 2) Assume that we define x = C(['0']) where the class C is defined by class C: def __init__(self,los): self.los = los def __lt__(self,right): if type(right) is C: return self.los < right.los elif type(right) is int: return self.los < [d for d in str(right)] else: return NotImplemented def __gt__(self,right): return right < self Explain in detail (as I did in this lecture) how Python attempts to evaluate 12 < x and whether it succeeds or fails, and if it fails -hint: it fails- how and why it fails. Answer the same question, if we replaced the defintion of __gt__ above by def __gt__(self,right): return C.__lt__(right,self) 3) Assume that we define the class C below class C: def __init__(self,los): self.los = los def __add__(self,right): if type(right) is C: return C(self.los + right.los) elif type(right) is int: return C(self.los + [d for d in str(right)]) else: return NotImplemented In what cases can we guarantee that when __add__ is called, self is guaranteed to be an object constructed from class C? Explain how we can call this method with a self object that is not constructed from class C.