Asimov's Zeroth Law
We continue this discussion from where we left off a few weeks ago: robot-ethics and how Asimov's robots resolved conflicts. A key update to Asimov's original three laws is the inclusion of the zeroth law (from Wikipedia):
"A robot may not harm humanity, or, by inaction, allow humanity to come to harm."
A fundamental difference between this law and the others is its abstract specification, with little clue on how it will be implemented. Furthermore, by giving this law the highest priority, Asenion robots are designed to first and foremost safeguard 'humanity', while also minimizing injury to individual humans and themselves as secondary and tertiary objectives. A robot is given the difficult computational task of proving that the cost of hurting a human is less than humanitarian benefit derived from an alternative action, and must do so within a finite amount of time.
The Universal Conflict-Resolution Model
Immanuel Kant's 'categorical imperative' is an example of an universal conflict resolution model that furiously strives to be context-free, and one that has greatly influenced western thought. Similarly, the Ten-Commandments' "Thou shall not kill" is absolute. Any machine that includes this hard constraint would be unable to kill, even defensively in order to protect a large number of humans under threat. Kantian rules are easy to 'encode-and-forget' within machines and systems since one does not have to ever worry about the context of its application. As we saw in the previous post, the robotic laws are context-free and Kantian in design. The original three laws operated as hard, must-satisfy constraints and necessary conditions. Per Wikipedia, Kant's
"perfect duties are those that are blameworthy if not met, as they are a basic required duty for a human being."
The problem of course is that (see prior post), the rigidity of all-hard rules is not practical and later versions of Asenion robots appear to additionally operate based on the concept of Kant's imperfect duty (again from Wikipedia):
"unlike perfect duties, you do not attract blame should you not complete
an imperfect duty but you shall receive praise for it should you
complete it, as you have gone beyond the basic duties and taken duty
upon yourself"
Thus, imperfect duties are soft, rather than hard constraints, and the aim is to maximally satisfy (minimally violate) requirements. However, the optimization weights that trade off the degree of importance assigned to each of the Asenion robot's 'imperfect duties' are hard-coded, and additional context-specific inputs are required from humans to achieve a satisfactory result in resolving a dilemma. The robots have no authority to perform context-specific conflict resolutions on their own.
How then does this zeroth law practically work in Asimov stories? It's Kantian abstraction is incomprehensible to all except a couple of enlightened robots (with telepathic ability, no less). In general, the zeroth law remains useful only on paper for most of the robots.
Contextual Ethics: The Indian Way
Rajiv Malhotra's path-breaking book "Being Different: Indian Challenge to Western Universalism" provides a fascinating contrast between the traditional Indian approach of 'contextual ethics' (CE) that arose from its Dharmic thought system, and the 'context free ethics' that largely guides the western approach (for an earlier post based on his work, see here. Some of the methods in this post are applications of ideas in this book). From an optimization perspective, we can think of a CE-embedded robot as one that maximally satisfies a combination of hard, soft, and firm constraints, where a 'firm' constraint refer to a hard constraint that is minimally and temporarily relaxed, depending on the specific context of the dilemma, and for the benefit of the 'greatest good', at the expense of incurring a context-specific penalty. This flexibility must not be confused with moral relativism - where a set of soft constraints are tactically and optimally manipulated according to context to maximize some convenient self-serving objective. 'Optimally timing an apology' can be thought of as an example of a non self-serving, contextual optimization model. It is worth doing a deep dive into this concept, by reviewing some passages in the aforementioned book:
"The frequently leveled charge of moral relativism against this contextual morality is inaccurate, because the conduct and motive are considered consequential in judging the ultimate value of statements.
.... Dharmic ethics are formulated in response to the situation and context of the problem in a way that makes Western ethics seem unduly codified, rigid, monolithic and even simplistic. A.K. Ramanujan, in his influential essay 'Is There an Indian Way of Thinking?', uses the terms 'context-free' and 'context-sensitive' to contrast the West and India in their respective approaches to ethics: "Cultures may be said to have overall tendencies to idealize, and think in terms of, either the context-free or the context-sensitive kind of rules. Actual behavior may be more complex, though the rules they think with are a crucial factor in guiding the behavior. In cultures like India's, the context-sensitive kind of rule is the preferred formulation" ....
.... Dharmic traditions, on the other hand, have long sought to arrive at truth by balancing universal truths and acts with those that can be determined only in the context in which they occur. Dharmic cultures have thus evolved to become comfortable with complexity and nuance, rejecting notions of the absolute and rigid ideals of morality and conduct....
....dharmic thought offers both universal and contextual poles – not
just the latter, as that would be tantamount to moral relativism."
The dharmic approach lies in between an "all-soft constraint" and the Kantian "hard and soft constraint" approach to decision optimization.
Applying contextual optimization
Asimov's telepathic robot Giskard formulates and solves a probabilistic optimization problem where it trades off the opportunity cost (in terms of human lives) against the expected benefit to humanity. However the degree of uncertainty in this conflict-resolution model is too high and the robot eventually crashes. This episode comes across as an example of applying the CE approach to resolve a dilemma. The great Indian epics - the Ramayana and the Mahabharata, contain several brilliantly narrated instances of contextual conflict-resolution. Indian sci-fi movie buffs would not be surprised to know that George Lucas' Star Wars was inspired by the Ramayana.
Contextual optimization in the specific area of 'mathematical decision support software' would mean: allowing the rules of engagement to be configurable depending on the context. For regular users, advanced settings are greyed out, with only universal (default) rules enabled. Only super users, who are well-trained and comprehend the nature and consequences of the beast, get to work with 'firm' constraints, and on rare occasions. For example, an airline crew schedule optimization system should be configured to satisfy contractual and FAA rules, except during emergencies (e.g. post 9/11 recovery) where 'crew welfare' is only achievable by overriding one or more of these rules. Practical decision support systems should be carefully designed to allow such controlled contextual optimization.
Amending the Zeroth Law: The Dharmic Robot
The four laws do not quite protect the rest of the cosmos (e.g., from humanity) given their anthropocentric nature. From an Indian point of view, this gap can be closed by modifying the zeroth law based on the contextual ethics of dharma. Rajiv Malhotra, in his book, provides the etymology and a working definition of dharma:
"Dharma has the Sanskrit root dhri, which means 'that which upholds' or 'that without which nothing can stand' or 'that which maintains the stability and harmony of the universe'. Dharma encompasses the natural, innate behaviour of things, duty, law, ethics, virtue, etc. For example, the laws of physics describe current human understanding of the dharma of physical systems. Every entity in the cosmos has its particular dharma – from the electron, which has the dharma to move in a certain manner, to the clouds, galaxies, plants, insects, and of course, man. Dharma has no equivalent in the Western lexicon."
In such a framework, Asimov's laws would delineate a robot's various dharmas. At the highest level, we can require that a robot abide by the following fundamental law, that is based on an ancient Indian text:
"Non-harming is a robot's highest priority, except in the defense of dharma"
Conflict-resolution is always performed by first applying this highest dharmic principle and customizing it to the specific context. Note that by operating on the fundamental dharmic principle of least harm, a robot would usually satisfy Asimov's zeroth law, albeit in a context-specific manner, while also being in harmony with the original laws, as well as any new laws that get written in the future. Interestingly, the Hippocratic oath of medical doctors is based on a similar idea that represents a non-negative bound: "do good, or at least no harm". If complex systems, new drugs, etc., are designed by always keeping this fundamental principle in context, it may well minimize the risk of catastrophic failure.
Sunday, March 10, 2013
Monday, February 4, 2013
Optimally Shoveling Snow
This post attempts to compare the efficiency of various approaches used to clear a rectangular area (L > B) of snow, using a good old manual shovel of width w, always moving in straight lines. We assume that the gradient of the area is zero, and that h inches of snow has fallen uniformly over the field. We also ignore the capacity of the shovel for now. The total snow that must be banked at the borders of the field is constant = LBh. Also total working distance traversed ~ LB/w.
Snow shoveling can be a grueling exercise for some, and public health departments always publish safe shoveling tips during the winter season. This is especially important for people with potential heart problems or back problems. Hopefully analyses like these will help (I'm sure people have looked at this before, but just in case ...).
Our objectives are to minimize:
1. Total work done = Total snow moved X distance moved.
2. Number of Stop-Starts with a fixed unit cost k incurred
3. Total work time in cold weather.
Model:
Given an area bx, and assuming we are moving along x, the snow nearest the starting point must be moved the maximum distance, and this distance linearly reduces to near-zero at the end of x. Thus (introducing an integral sign in this blog, yikes):
a) total work done = ∫wxdx = wx2/2
b) Unit stop-start cost = k
Approach 1: Shoveling east-to-west (along L, banking at the west end) or bi-directional traversing:
total number of passes = B/w
total work done = (B/w) * wL2/2 + kB/w = BL2/2 + kB/w
Note that the effort level is independent of the shovel width. A wider shovel requires fewer passes but more effort per pass.
Approach 2: Shoveling south-to-north (along B, banking at north end) or traversing in both directions:
total cost = LB2/2 + kL/w
Thus, the second approach reduces the total work done by a factor L/B. However, this reduction is achieved at the expense of more stops (which may not be a bad thing from a health perspective; k may be negative or zero for some). If L = 2B, the second approach requires 50% less effort, but requiring twice as many passes.
Approach 3: I observed that my Canadian neighbor divides up the rectangle into two areas, always starts from the middle, and creates banks on both sides. If we bank the snow at the south and north ends:
total number of passes = 2L/w
total cost = LB2/4 + 2kL/w
The third approach is twice as effort-efficient as the second approach, at the expense of making twice as many banks, plus an additional penalty of traveling to the mid point after each pass. On the other hand, you are also less likely to run into capacity problems, since the material handled per pass is the lowest. We can also verify that among all starting points chosen for approach-3, the mid-point (B/2) yields the minimal effort. In general, any of these approaches become preferable in terms of total cost, depending on the chosen objective function. For example, we could modify approach-3 and bank snow close to the ends the field (< B/2) at their respective (west or east) ends at the expense of adding more stop-starts.
Time Considerations
If the shape of the field changes, the approaches may change, but it appears that a good principle to follow is to limit the material moved between successive bankings. In particular, we want to identify the shortest Euclidean path between every point in the field to the nearest feasible bank, and purely from an effort perspective, shovel snow along a path to a feasible bank that is optimal for all points on that path. This seems to reduce the working duration taken as well:
Suppose we maintain constant momentum (i.e. mass X velocity = constant) during a pass, our velocity will inversely proportional to the amount of snow accumulated (e.g. 1/wx). Total time per pass ∝ ∫wxdx = wx2/2, which is proportional to the effort.
For such a velocity model, approach-3 also appears to be optimal from the time perspective. If we assume constant velocity, then the three approaches take the same working time, ignoring the shovel-free travel time required for approach-3. Among these alternative time-optimal approaches, the third approach does the best on the effort-objective.
Lawn Mowing
Non-propelled, walking lawn mowers with bags? Regardless of the traversing method adopted, our effort (per our chosen model) increases as the square of distance until we empty our cut-grass. If we adopt approach-2 or 3, we have more banking opportunities. However, approach-3 may result in a mowing pattern that won't look so nice, so approach 2 may be more preferable. Operating a self-propelled mower with built-in mulching may reduce our effort considerably.
Update 1: Post "Nemo" Shoveling, February 9, 2013
A truly optimal solution is to have a kind and wonderful neighbor. After three hours of efficient shoveling (h = 24 inches) and 40% completion, Kevin zipped in with his snow plough and finished the remainder in 5 minutes before going off to help others. I now have to find the best way to thank him.
Update 2: (WSJ, via @ThaddeusKTSim), February 10
(source link: online.wsj.com)
Snow shoveling can be a grueling exercise for some, and public health departments always publish safe shoveling tips during the winter season. This is especially important for people with potential heart problems or back problems. Hopefully analyses like these will help (I'm sure people have looked at this before, but just in case ...).
Our objectives are to minimize:
1. Total work done = Total snow moved X distance moved.
2. Number of Stop-Starts with a fixed unit cost k incurred
3. Total work time in cold weather.
Model:
Given an area bx, and assuming we are moving along x, the snow nearest the starting point must be moved the maximum distance, and this distance linearly reduces to near-zero at the end of x. Thus (introducing an integral sign in this blog, yikes):
a) total work done = ∫wxdx = wx2/2
b) Unit stop-start cost = k
Approach 1: Shoveling east-to-west (along L, banking at the west end) or bi-directional traversing:
total number of passes = B/w
total work done = (B/w) * wL2/2 + kB/w = BL2/2 + kB/w
Note that the effort level is independent of the shovel width. A wider shovel requires fewer passes but more effort per pass.
Approach 2: Shoveling south-to-north (along B, banking at north end) or traversing in both directions:
total cost = LB2/2 + kL/w
Thus, the second approach reduces the total work done by a factor L/B. However, this reduction is achieved at the expense of more stops (which may not be a bad thing from a health perspective; k may be negative or zero for some). If L = 2B, the second approach requires 50% less effort, but requiring twice as many passes.
Approach 3: I observed that my Canadian neighbor divides up the rectangle into two areas, always starts from the middle, and creates banks on both sides. If we bank the snow at the south and north ends:
total number of passes = 2L/w
total cost = LB2/4 + 2kL/w
The third approach is twice as effort-efficient as the second approach, at the expense of making twice as many banks, plus an additional penalty of traveling to the mid point after each pass. On the other hand, you are also less likely to run into capacity problems, since the material handled per pass is the lowest. We can also verify that among all starting points chosen for approach-3, the mid-point (B/2) yields the minimal effort. In general, any of these approaches become preferable in terms of total cost, depending on the chosen objective function. For example, we could modify approach-3 and bank snow close to the ends the field (< B/2) at their respective (west or east) ends at the expense of adding more stop-starts.
Time Considerations
If the shape of the field changes, the approaches may change, but it appears that a good principle to follow is to limit the material moved between successive bankings. In particular, we want to identify the shortest Euclidean path between every point in the field to the nearest feasible bank, and purely from an effort perspective, shovel snow along a path to a feasible bank that is optimal for all points on that path. This seems to reduce the working duration taken as well:
Suppose we maintain constant momentum (i.e. mass X velocity = constant) during a pass, our velocity will inversely proportional to the amount of snow accumulated (e.g. 1/wx). Total time per pass ∝ ∫wxdx = wx2/2, which is proportional to the effort.
For such a velocity model, approach-3 also appears to be optimal from the time perspective. If we assume constant velocity, then the three approaches take the same working time, ignoring the shovel-free travel time required for approach-3. Among these alternative time-optimal approaches, the third approach does the best on the effort-objective.
Lawn Mowing
Non-propelled, walking lawn mowers with bags? Regardless of the traversing method adopted, our effort (per our chosen model) increases as the square of distance until we empty our cut-grass. If we adopt approach-2 or 3, we have more banking opportunities. However, approach-3 may result in a mowing pattern that won't look so nice, so approach 2 may be more preferable. Operating a self-propelled mower with built-in mulching may reduce our effort considerably.
Update 1: Post "Nemo" Shoveling, February 9, 2013
A truly optimal solution is to have a kind and wonderful neighbor. After three hours of efficient shoveling (h = 24 inches) and 40% completion, Kevin zipped in with his snow plough and finished the remainder in 5 minutes before going off to help others. I now have to find the best way to thank him.
Update 2: (WSJ, via @ThaddeusKTSim), February 10
Thursday, January 24, 2013
Conflict Resolution - 2: Asimov's Laws of Robotics
Asimov's Laws of Robotics
The introductory discussion on prioritized business rules can be read here. This sequel treats Asimov's sci-fi laws of robotics as a set of prioritized business rules within a decision support system (DSS). We try to learn some 'field lessons' from stories that revolve around violations of one or more of these laws. The three (original) laws postulated in the 1940s and the stories cited in the Wikipedia page are used in this post:
(Trivia: Robot Chitti in the Indian movie Endhiran is obviously not Asenion. A portion of the plot in the movie bears some resemblance to a story-line in 'Robots of Dawn')
Inadequacy of rules for complex systems
Hard Constraint Violation
Stories around initial designs of Asenion robots appear to revolve around hard-constraint satisfaction. Humans forget that a telepathic robot also protects humans from mental harm. It often prefers to tell the questioner what she/he wants to hear, violating the second law by not responding with facts, in order to satisfy the higher priority constraint of not injuring human minds 'by dashing hopes'. In the end, it runs into an irreconcilable constraint violation when it can neither speak nor be silent without hurting somebody mentally. It 'dies', taking its telepathic secrets with it.
An expensive decision support system that provides a 'null' response (or '42') after running for a long time can be irritating. There once was a user who loved a DSS when it worked, but would question slow-failures, asking why the application was designed to go through bugs and not around them.
Soft Constraint Violation
A more sophisticated robotic design in another story employs an optimization model. The laws are encoded as soft constraints using potential functions that take into account the level of importance of the order given, the priority of the law, and the degree of constraint 'violation'. Consequently, a 'truant' robot that is faced with a conflict between the second and third laws determines an optimal (equilibrium) solution to the corresponding weighted cumulative violation minimization problem that causes it to continually run around in a circle centered at the origin of the conflict. Doing so prevents the robot from completing a routine task it was assigned. Unfortunately, the robot is uninformed that the delay in the completion of this task is increasingly endangering human lives on the planet. An initial idea of the engineers to break this cycle by perturbing the potentials in the objective function merely shifts the optimum and alters the radius of the Robot's circle, but doesn't help them one bit. Finally, an engineer put his own life in visible jeopardy to inject high-intensity counter-potential (associated with the first law) into the objective function, and is ultimately able to resolve the conflict without loss of life.
This story reminds us of the problems associated with careless softening of 'hard' constraints that attenuates or randomizes the sensitivity of the model's response to priorities and scarce resources ("noisy duals"). The user is left with the unpleasant task of figuring out how to tweak the various 'potential' coefficients for a given problem instance in order to achieve the best results. In practice, an artful mixture of hard and soft constraints usually results in a more manageable and usefully responsive system.
Side-effects of Violation
Users often turn a few knobs in a DSS, look at the results, and wonder "why did this weird thing happen?". This leads to discussions that often unlocks hidden benefits for the customer. In an Asimov story, a robot determines that it is optimal for it to take over the management of delicate instrumentation that is vital to human survival on a planet. It deliberately disobeys humans and gangs up with other robots to permanently banish humans from the control room. The engineers are initially exasperated by this behavior ("is it a bug or a modeling issue?") before a data-driven realization of the life-saving benefit of 'violating' the lower-priority rule calms them. In practice, it is useful to identify unnecessary legacy constraints, if any, that are employed.
Rule violations can also result in undesirable side effects. In the world of airline scheduling, careless rule-changes put in to fix one problem can result in bizarre (hilariously alternative-optimal) schedules for pilots and planes that send them on a wild-goose chase all over the continent. In Asimov stories, engineers discover that a conflict-stressed robot's positronic brain turns it into a practical joker, a prophet, or a wacky friend who plays tag in a life-or-death situation.
Conflict resolution is not just about figuring out what went wrong in the model - a relatively easy task given the excellent solver tools that are available in the market today. It also involves the task of usefully and legibly mapping this change in the model's response back to real world causals. The more interlinked the decision variables and business rules in the system, the more difficult this latter task can become.
In memory of the late Tamil writer Sujatha.
The introductory discussion on prioritized business rules can be read here. This sequel treats Asimov's sci-fi laws of robotics as a set of prioritized business rules within a decision support system (DSS). We try to learn some 'field lessons' from stories that revolve around violations of one or more of these laws. The three (original) laws postulated in the 1940s and the stories cited in the Wikipedia page are used in this post:
- A robot may not injure a human being or, through inaction, allow a human being to come to harm.
- A robot must obey the orders given to it by human beings, except where such orders would conflict with the First Law.
- A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws.
(Trivia: Robot Chitti in the Indian movie Endhiran is obviously not Asenion. A portion of the plot in the movie bears some resemblance to a story-line in 'Robots of Dawn')
Inadequacy of rules for complex systems
Anticipating 'bugs', and identifying/resolving potential conflicts (low-probability high consequence (LPHC) events in particular) within and between interacting components of a complex, synthesized system can be quite challenging. Recent examples: Boeing 787 Dreamliners, offshore oil rigs, or the western financial system. The positronic robot is no exception. Roger Clarke notes: "Asimov's Laws of Robotics have been a very successful literary device. Perhaps
ironically, or perhaps because it was artistically appropriate, the sum of
Asimov's stories disprove the contention that he began with: It is not possible
to reliably constrain the behavior of robots by devising and applying a set of
rules."
(picture link source: Wikipedia)
Hard Constraint Violation
Stories around initial designs of Asenion robots appear to revolve around hard-constraint satisfaction. Humans forget that a telepathic robot also protects humans from mental harm. It often prefers to tell the questioner what she/he wants to hear, violating the second law by not responding with facts, in order to satisfy the higher priority constraint of not injuring human minds 'by dashing hopes'. In the end, it runs into an irreconcilable constraint violation when it can neither speak nor be silent without hurting somebody mentally. It 'dies', taking its telepathic secrets with it.
An expensive decision support system that provides a 'null' response (or '42') after running for a long time can be irritating. There once was a user who loved a DSS when it worked, but would question slow-failures, asking why the application was designed to go through bugs and not around them.
Soft Constraint Violation
A more sophisticated robotic design in another story employs an optimization model. The laws are encoded as soft constraints using potential functions that take into account the level of importance of the order given, the priority of the law, and the degree of constraint 'violation'. Consequently, a 'truant' robot that is faced with a conflict between the second and third laws determines an optimal (equilibrium) solution to the corresponding weighted cumulative violation minimization problem that causes it to continually run around in a circle centered at the origin of the conflict. Doing so prevents the robot from completing a routine task it was assigned. Unfortunately, the robot is uninformed that the delay in the completion of this task is increasingly endangering human lives on the planet. An initial idea of the engineers to break this cycle by perturbing the potentials in the objective function merely shifts the optimum and alters the radius of the Robot's circle, but doesn't help them one bit. Finally, an engineer put his own life in visible jeopardy to inject high-intensity counter-potential (associated with the first law) into the objective function, and is ultimately able to resolve the conflict without loss of life.
This story reminds us of the problems associated with careless softening of 'hard' constraints that attenuates or randomizes the sensitivity of the model's response to priorities and scarce resources ("noisy duals"). The user is left with the unpleasant task of figuring out how to tweak the various 'potential' coefficients for a given problem instance in order to achieve the best results. In practice, an artful mixture of hard and soft constraints usually results in a more manageable and usefully responsive system.
Side-effects of Violation
Users often turn a few knobs in a DSS, look at the results, and wonder "why did this weird thing happen?". This leads to discussions that often unlocks hidden benefits for the customer. In an Asimov story, a robot determines that it is optimal for it to take over the management of delicate instrumentation that is vital to human survival on a planet. It deliberately disobeys humans and gangs up with other robots to permanently banish humans from the control room. The engineers are initially exasperated by this behavior ("is it a bug or a modeling issue?") before a data-driven realization of the life-saving benefit of 'violating' the lower-priority rule calms them. In practice, it is useful to identify unnecessary legacy constraints, if any, that are employed.
Conflict resolution is not just about figuring out what went wrong in the model - a relatively easy task given the excellent solver tools that are available in the market today. It also involves the task of usefully and legibly mapping this change in the model's response back to real world causals. The more interlinked the decision variables and business rules in the system, the more difficult this latter task can become.
In memory of the late Tamil writer Sujatha.
Sunday, January 13, 2013
Being Optimally Sorry: When to Apologize?
A Delayed Apology
This post examines modeling ideas related to the timing of an apology in a two-person scenario that results in a maximally effective 'sorry'. We optimize timing here not to maximize own benefits (user optimal), but on the basis of mutual respect, to express regret and maximally repair the damage in a timely manner that most helps the subject (recipient optimal). We start with the findings in Frank Partnoy's book "Wait: the Art and Science of Delay". It's one of the many useful books in the last couple of years that analyze human decision making. We introduce a mental decision support model for a timely apology that is derived from decision analytical methods employed in an industrial setting.
Objectives and Constraints
Justice delayed may be justice denied, but an apology that is optimally delayed may not be such a bad thing. The 'Wait' book recognizes the existence of a suitable time to apologize, and notes that the fastest apology in not necessarily the most effective. Given that we may have to apologize more than once, in general we have to determine an optimal trajectory of timed apologies. Thus, our goals are to:
i) apologize at least once,
ii) in a timely manner, and
iii) within a finite time horizon, such that
iv) a measure of the recipient's benefits is maximized
'Wait' notes:
"... Saying you are sorry is always better than not apologizing at all. But as with the first study, the students felt better about a delayed apology: “Improvement in the late apology condition was significantly greater than improvement in the early apology condition.” In fact, a statistically significant improvement in the students’ reactions occurred only in the late apology condition, when there was a chance for them to discuss what had happened and why. Overall, these studies suggest that the relationship between apologies and timing follows a “bell curve” distribution: effectiveness is low at first, then rises, peaks, and ultimately declines."
(It seems that these ideas are related to the complementary 'problem' of delivering the most time-effective 'Thank You')
We can see that the timing-effectiveness curve described in the book extract above is related to the subject's level of distress/angst (which we represent as 'entropy') that follows a similar trajectory of rise, cruise, and a gradual demise. Depending on the person, the 'cruise' and 'demise' portions can last long and result in a very fat-tailed distribution. But before we get into 'when', a quick comment from the book on the what/why/how questions:
".... effective apologies typically contain four parts:
1. Acknowledge that you did it.
2. Explain what happened.
3. Express remorse.
4. Repair the damage, as much as you can."
Searching for the Optimal Timing
'Wait' notes:
"The art of the apology centers on the management of delay. For most of us, the lesson is that the next time we do something wrong to a close friend or family member, or say something at work we wish we could take back, we should try to imagine how the victim might react to an apology tomorrow instead of today, or in a few hours instead of right now. If delay will give a friend or relative or coworker a chance to react, to voice a response and prepare themselves to hear our regret, the apology will mean more later than right away."
In other words, the timing has to take into account where the subject is located in their entropic life cycle: is the person likely to be getting angrier by the hour now (positive entropic gradient), or has reached the peak and is calming down (negative entropic gradient). To formulate a model based on these observations, we borrow ideas from a classical inventory management problem analyzed in retail operations research: Markdown Optimization (MDO).
An Optimization Model
MDO is employed to manage an inventory of short-life cycle (SLC) products that are manufactured pre-season, with the (sunk) costs paid up-front. Thus MDO typically focuses on total revenue earned in-season. Analogously, we already messed up in the beginning incurring an irreversible cost, and thereafter it costs relatively little to issue a sincere apology. Retailers employ a cadence of optimally delayed price cuts to smartly boost the end-of-season demand rate so as to maximize revenue over the remaining life of the product. Like MDO, we eventually have to solve an entropic inventory depletion problem: optimally alter the entropic gradient via one or more carefully timed apologies, which will (ideally) reduce the inventory level to zero within a finite time period.
Disclaimer: The postulated model is not assumed to be the most suitable or even a "correct" one for this problem, but merely a useful starting point. Some brief comments on the modeling elements, next.
a. Life-cycle of the entropy
SLC products (like designer fashion apparel) often have little to no historic data early in the season, and retailers may borrow results for a comparable historical "like-item" to produce an initial prediction and then continually update their sales projection based on in-season demand. Here, we play the role of a 'like-item' and place ourselves in the recipient's shoes to better appreciate the degree of distress caused and the impact it will have on the recipient over time. The entropy level is an uncertain quantity that must be learned, but its 'mean value' is assumed to representable using an approximately concave function like the one shown in the figure below. Note that unlike the MDO case where inventory is always non-increasing, entropic inventory initially increases before gradually decreasing.
b. Elasticity of the entropy with respect to an apology
Elasticity ~ % change in entropy / % change in regret and effort, as perceived by the subject
A simple model like the inverse square law that abounds in nature (elasticity = -2) may be a good starting point. Ill-timed and empty-sounding apologies may have zero elasticity and do little to reduce entropic inventory. A careless apology can result in an entropic spike ("adding insult to injury"). On the other hand, an apology that is 'deep and sincere' and well-timed can be expected to have a calming effect.
c. Timings
Optimally timing a single apology requires impeccable timing. On the other hand, randomly distributed, and incessant apologies may not be helpful either. A premature apology (e.g. around an increasing entropic gradient) that kicks the can down the road is a greedy approach that may be counter-productive. Thus, optimally timing multiple apologies can require a degree of coordination between decisions. Today's apologize-or-delay decision will impact the timings of future decisions, so we have to holistically manage the impact on the entropic life-cycle.
Often, despite our best efforts, the damage can never be fully repaired. Note that our objective function was setup to be indifferent to personal benefits. To paraphrase a profound Indian saying: "You have the right to optimize, but not to the fruits of your actions". Regardless of the outcome, a sincere and optimally timed apology is good Karma.
This post examines modeling ideas related to the timing of an apology in a two-person scenario that results in a maximally effective 'sorry'. We optimize timing here not to maximize own benefits (user optimal), but on the basis of mutual respect, to express regret and maximally repair the damage in a timely manner that most helps the subject (recipient optimal). We start with the findings in Frank Partnoy's book "Wait: the Art and Science of Delay". It's one of the many useful books in the last couple of years that analyze human decision making. We introduce a mental decision support model for a timely apology that is derived from decision analytical methods employed in an industrial setting.
Objectives and Constraints
Justice delayed may be justice denied, but an apology that is optimally delayed may not be such a bad thing. The 'Wait' book recognizes the existence of a suitable time to apologize, and notes that the fastest apology in not necessarily the most effective. Given that we may have to apologize more than once, in general we have to determine an optimal trajectory of timed apologies. Thus, our goals are to:
i) apologize at least once,
ii) in a timely manner, and
iii) within a finite time horizon, such that
iv) a measure of the recipient's benefits is maximized
'Wait' notes:
"... Saying you are sorry is always better than not apologizing at all. But as with the first study, the students felt better about a delayed apology: “Improvement in the late apology condition was significantly greater than improvement in the early apology condition.” In fact, a statistically significant improvement in the students’ reactions occurred only in the late apology condition, when there was a chance for them to discuss what had happened and why. Overall, these studies suggest that the relationship between apologies and timing follows a “bell curve” distribution: effectiveness is low at first, then rises, peaks, and ultimately declines."
(It seems that these ideas are related to the complementary 'problem' of delivering the most time-effective 'Thank You')
We can see that the timing-effectiveness curve described in the book extract above is related to the subject's level of distress/angst (which we represent as 'entropy') that follows a similar trajectory of rise, cruise, and a gradual demise. Depending on the person, the 'cruise' and 'demise' portions can last long and result in a very fat-tailed distribution. But before we get into 'when', a quick comment from the book on the what/why/how questions:
".... effective apologies typically contain four parts:
1. Acknowledge that you did it.
2. Explain what happened.
3. Express remorse.
4. Repair the damage, as much as you can."
Searching for the Optimal Timing
'Wait' notes:
"The art of the apology centers on the management of delay. For most of us, the lesson is that the next time we do something wrong to a close friend or family member, or say something at work we wish we could take back, we should try to imagine how the victim might react to an apology tomorrow instead of today, or in a few hours instead of right now. If delay will give a friend or relative or coworker a chance to react, to voice a response and prepare themselves to hear our regret, the apology will mean more later than right away."
In other words, the timing has to take into account where the subject is located in their entropic life cycle: is the person likely to be getting angrier by the hour now (positive entropic gradient), or has reached the peak and is calming down (negative entropic gradient). To formulate a model based on these observations, we borrow ideas from a classical inventory management problem analyzed in retail operations research: Markdown Optimization (MDO).
An Optimization Model
MDO is employed to manage an inventory of short-life cycle (SLC) products that are manufactured pre-season, with the (sunk) costs paid up-front. Thus MDO typically focuses on total revenue earned in-season. Analogously, we already messed up in the beginning incurring an irreversible cost, and thereafter it costs relatively little to issue a sincere apology. Retailers employ a cadence of optimally delayed price cuts to smartly boost the end-of-season demand rate so as to maximize revenue over the remaining life of the product. Like MDO, we eventually have to solve an entropic inventory depletion problem: optimally alter the entropic gradient via one or more carefully timed apologies, which will (ideally) reduce the inventory level to zero within a finite time period.
Disclaimer: The postulated model is not assumed to be the most suitable or even a "correct" one for this problem, but merely a useful starting point. Some brief comments on the modeling elements, next.
a. Life-cycle of the entropy
SLC products (like designer fashion apparel) often have little to no historic data early in the season, and retailers may borrow results for a comparable historical "like-item" to produce an initial prediction and then continually update their sales projection based on in-season demand. Here, we play the role of a 'like-item' and place ourselves in the recipient's shoes to better appreciate the degree of distress caused and the impact it will have on the recipient over time. The entropy level is an uncertain quantity that must be learned, but its 'mean value' is assumed to representable using an approximately concave function like the one shown in the figure below. Note that unlike the MDO case where inventory is always non-increasing, entropic inventory initially increases before gradually decreasing.
Elasticity ~ % change in entropy / % change in regret and effort, as perceived by the subject
A simple model like the inverse square law that abounds in nature (elasticity = -2) may be a good starting point. Ill-timed and empty-sounding apologies may have zero elasticity and do little to reduce entropic inventory. A careless apology can result in an entropic spike ("adding insult to injury"). On the other hand, an apology that is 'deep and sincere' and well-timed can be expected to have a calming effect.
c. Timings
Optimally timing a single apology requires impeccable timing. On the other hand, randomly distributed, and incessant apologies may not be helpful either. A premature apology (e.g. around an increasing entropic gradient) that kicks the can down the road is a greedy approach that may be counter-productive. Thus, optimally timing multiple apologies can require a degree of coordination between decisions. Today's apologize-or-delay decision will impact the timings of future decisions, so we have to holistically manage the impact on the entropic life-cycle.
Often, despite our best efforts, the damage can never be fully repaired. Note that our objective function was setup to be indifferent to personal benefits. To paraphrase a profound Indian saying: "You have the right to optimize, but not to the fruits of your actions". Regardless of the outcome, a sincere and optimally timed apology is good Karma.
Tuesday, January 8, 2013
Business Rule Conflict Resolution: Needles in a Haystack
This post uses the retail industry as a sketchpad to illustrate some simple but useful conflict resolution ideas employed within optimization methods in practice.
Least Infeasible Answers
Retailers have to address a myriad of business rules while optimizing merchandising, pricing, shelving, replenishing, and assortment planning decisions. Operations Researchers build mathematical models that translate these rules into a combination of soft constraints (goals, objectives, Key Performance Indicators (KPI)) that have to be maximally satisfied, and hard constraints that must be absolutely satisfied.
Conflicting hard rules can prevent the optimization application from returning a recommendation. The retailer would like to know which among their long list of rules is/are the root cause of this conflict so that they can address them appropriately and re-evaluate. Commercial solvers like CPLEX (and GUROBI, among others) can perform an infeasibility analysis to provide useful information pertaining to IIS (Irreducibly Inconsistent Sets), and based on optimizing a user-parameterized penalty function, can also find a minimally infeasible answer. These are non-trivial problems. (Try explaining to a senior pilot why a more junior pilot was assigned that plum schedule filled with all those long Hawaii layovers!) JW Chinneck has an entire book dealing with such issues.
User Experience
The liberal-arts degree armed retail store manager (RSM) has a business to operate and sales targets to achieve. Many a RSM care little for iterative conflict refinement and resolution and have even lesser patience to set up paramterized penalty functions. He/She often desires that the analytics software product automatically flag problematic rules and always return the "best possible" answer(s) and be done with it. 'Null' or 'Try, try again' is not an option. A positive user experience combined with the ability to "help me earn a bonus or you are toast" is correlated with continued patronage. In such situations, the decision analytics plumbing must be designed in such that it can, with little to no external guidance, produce practically useful diagnostics that pinpoints infeasibility and generate 'best fit' solutions that are readily business-comprehensible to the average user. After all, the business application interface and functionality is not primarily meant to be a analytics-tutorial warp drive thingy.
(picture link source: anandtech.com)
Actions Express Priorities - Mohandas Gandhi
A store manager may have a high-level goal in mind such as: find a minimally business disruptive resolution of conflicting business rules. Toward this, several retailers employ prioritized business rules that result in a special haystack structure that invariably makes the conflict identification and resolution process more transparent and easy to manage. What would this decision problem structure look like? Borrowing computer-ware terminology (for example), it would be akin to an ordered collection of hard constraints, soft constraints, and firm constraints. Patented methods that gainfully exploit related ideas have been used to build retail optimization products that turned out to be successful in the global marketplace. On the other hand, the efficacy of such resolution methods can be context-specific and is not necessarily designed to yield universally scalable solutions. If your job is to solve only your company's or industry's specific problems and not that of all current and future corporations in the galaxy and beyond, such methods can be handy.
Least Infeasible Answers
Retailers have to address a myriad of business rules while optimizing merchandising, pricing, shelving, replenishing, and assortment planning decisions. Operations Researchers build mathematical models that translate these rules into a combination of soft constraints (goals, objectives, Key Performance Indicators (KPI)) that have to be maximally satisfied, and hard constraints that must be absolutely satisfied.
Conflicting hard rules can prevent the optimization application from returning a recommendation. The retailer would like to know which among their long list of rules is/are the root cause of this conflict so that they can address them appropriately and re-evaluate. Commercial solvers like CPLEX (and GUROBI, among others) can perform an infeasibility analysis to provide useful information pertaining to IIS (Irreducibly Inconsistent Sets), and based on optimizing a user-parameterized penalty function, can also find a minimally infeasible answer. These are non-trivial problems. (Try explaining to a senior pilot why a more junior pilot was assigned that plum schedule filled with all those long Hawaii layovers!) JW Chinneck has an entire book dealing with such issues.
User Experience
The liberal-arts degree armed retail store manager (RSM) has a business to operate and sales targets to achieve. Many a RSM care little for iterative conflict refinement and resolution and have even lesser patience to set up paramterized penalty functions. He/She often desires that the analytics software product automatically flag problematic rules and always return the "best possible" answer(s) and be done with it. 'Null' or 'Try, try again' is not an option. A positive user experience combined with the ability to "help me earn a bonus or you are toast" is correlated with continued patronage. In such situations, the decision analytics plumbing must be designed in such that it can, with little to no external guidance, produce practically useful diagnostics that pinpoints infeasibility and generate 'best fit' solutions that are readily business-comprehensible to the average user. After all, the business application interface and functionality is not primarily meant to be a analytics-tutorial warp drive thingy.
Actions Express Priorities - Mohandas Gandhi
A store manager may have a high-level goal in mind such as: find a minimally business disruptive resolution of conflicting business rules. Toward this, several retailers employ prioritized business rules that result in a special haystack structure that invariably makes the conflict identification and resolution process more transparent and easy to manage. What would this decision problem structure look like? Borrowing computer-ware terminology (for example), it would be akin to an ordered collection of hard constraints, soft constraints, and firm constraints. Patented methods that gainfully exploit related ideas have been used to build retail optimization products that turned out to be successful in the global marketplace. On the other hand, the efficacy of such resolution methods can be context-specific and is not necessarily designed to yield universally scalable solutions. If your job is to solve only your company's or industry's specific problems and not that of all current and future corporations in the galaxy and beyond, such methods can be handy.
Thursday, January 3, 2013
Priceless Pricing
Listening to Jazz feels like time well spent, despite a limited understanding of this beautiful American art-form. John Coltrane and Miles Davis are especially close to heart given their affection for India's native music that also influenced how they played. Miles is recognized for paying attention to the 'space between the notes' as noted in this blog:
"Among Miles’ trademarks was his emphasis on the space between the notes as much as the notes themselves. Silence was key to his music. He was so cool he didn’t feel the need to jam up every measure with noise. Sometimes it’s better to just stand there and not do something."
It is well known that silence is an important tool in the hands of an expert stand-up comedian or a novelist. In the world of ORMS, optimally scheduling working hours for airline crews is really about paying close attention to their silent periods of rest. In the world of pricing optimization, it is imperative to recognize that a customer always values something else in the product in addition to price. To usefully optimize price, we must accurately identify its priceless attributes.
Update: January 11, 2013
Richard Marcus, former CEO of Neiman Marcus: “Price is important, but quality is remembered long after price is forgotten, as are qualities of uniqueness and originality.”
Update: February 06, 2013
Competing beyond price to maximize customer loyalty: "...“Attractive prices are an effective means to get people in the door, but it’s not enough to maintain loyalty, drive future purchases, and generate customer recommendations..."
Update: May 30, 2013
Price Perception: " ....it’s not about price, it’s about what am I getting for that price"
"Among Miles’ trademarks was his emphasis on the space between the notes as much as the notes themselves. Silence was key to his music. He was so cool he didn’t feel the need to jam up every measure with noise. Sometimes it’s better to just stand there and not do something."
It is well known that silence is an important tool in the hands of an expert stand-up comedian or a novelist. In the world of ORMS, optimally scheduling working hours for airline crews is really about paying close attention to their silent periods of rest. In the world of pricing optimization, it is imperative to recognize that a customer always values something else in the product in addition to price. To usefully optimize price, we must accurately identify its priceless attributes.
Update: January 11, 2013
Richard Marcus, former CEO of Neiman Marcus: “Price is important, but quality is remembered long after price is forgotten, as are qualities of uniqueness and originality.”
Update: February 06, 2013
Competing beyond price to maximize customer loyalty: "...“Attractive prices are an effective means to get people in the door, but it’s not enough to maintain loyalty, drive future purchases, and generate customer recommendations..."
Update: May 30, 2013
Price Perception: " ....it’s not about price, it’s about what am I getting for that price"
Monday, December 31, 2012
My Teacher, My Guru
Just found out an hour ago that my PhD adviser retires tonight. Operations Research was the least I learnt from Prof. HDS - despite the fact I used to be clueless enough to think that 'simplex' was probably some kind of vinyl plastic before I enrolled in his graduate program. If Prof. HDS was teaching archaeology I'd be blogging about digging techniques now. He infused a magic into ORMS that has not gone missing yet. His brilliance is stunning and his research contributions, staggering; his students are moved by his compassion, his classes were masterpieces, and many of his colleagues across the university and the world are in respectful awe... Thankfully, it is much simpler for me. He is my teacher, my Guru. Happy retirement, sir. Go easy on the red ink now, maybe?
-----
Wishing readers of dualnoise from around the world a very happy, safe, and healthy new year! Thank you for taking the time to visit this small 'toolers' corner. Feel free to message dualnoise at gmail on topics you would like to read about in 2013.
-----
Wishing readers of dualnoise from around the world a very happy, safe, and healthy new year! Thank you for taking the time to visit this small 'toolers' corner. Feel free to message dualnoise at gmail on topics you would like to read about in 2013.
Friday, December 21, 2012
The Scamster versus the Statesman (or why the Indian National Congress fancies its chances in 2014 and 2019)
A Scam Response Model
The loss due to government-blessed scams and failed populist schemes in India run in the order of Billions and Trillions of Indian Rupees in a country where the impoverished still form a sizable percentage and earning under or around the government designated poverty level of ₹20+ a day (50₹ ~ 1$). To many Indians, such twelve-zero numbers seem fantastic. Thus a person caught stealing thousands in a small town may receive a sound thrashing because people more clearly recognize the utility of amounts that actually show up in their historical experience. However, as the size of the theft increases beyond imaginable sums, the magnitude of the public response does not appear to proportionally increase. The law of diminishing returns seems to kick in, resulting in a concave response function. We postulate that:
Change in Public Response (ΔR) ∝ Change in # of Zeros (ΔZ) in Scam amount S , i.e
R = k log S, where k is a constant to be estimated using historical data.
The assumption of a logarithmic response model turns to be quite useful in the Indian context. For example, if political party A pulls off a 12-zero scam, and party B a 6-zero scam (using a log-10 base):
response(A) = 12k, and response(B) = 6k.
The response is only a factor of 2 more for the scam that was a million times bigger. This makes it easier for party-A to equate itself to party-B in the eyes of the public. It can have its cake and perhaps eat it too. Most Indians would not find it hard to map A & B to their real-life representatives.
Statesman versus Scamster - Round 1
A subjugated, under-informed electorate may exhibit a strong concave response that is characterized by relatively high sensitivity to small thefts at one end, and a disregard for mega-scams at the other end. We postulate that such a population's mental decision making model tends to be (1) myopic, (2) local optimality seeking, and (3) pessimistic (MYLOP). It accepts its terrible local living conditions provided the situation doesn't get perceptibly worse, and votes in polls based on this mental model. The immediate cause-and-effect tied to a local ₹100 theft represents a clear and present risk to current survival and is likely to elicit a swift and violent response, just like a populist cash-equivalent freebie elicits an equally enthusiastic response. However, a gigantic national scam that all but ensures that its future generations will see no improvement is shrugged off. MYLOP behavior is akin to a frog that allows itself to boiled alive because it does not register the gradual increase in water temperature until it is too late.
On the flip side, a government that is largely scam-free and claims to work on long-term growth, but is perceived to have marginally worsened the status quo can get booted out of power in the next election. Asking such a population to endure temporary 'hard choices' in order to be rewarded with medium-to-long term improvement ("no free lunch", "it will get worse before it gets better") implies a non-convex decision model - a hard sell since it clashes with the MYLOP attitude of the population. Scamsters + freebies trumps the Statesman here. Let's dig a little deeper into scams.
How to Hide a Scam?
Answer: A new scam that is an order of magnitude bigger. This can be explained via a mathematical model based on an interesting technical report by John Cook, M. D. Anderson Cancer Center (and author of the Endeavor blog):
The soft maximum of two variables is the function
g(x; y) = log(exp(x) + exp(y))
Given scams S1 and S2, the cumulative response from the public per our postulated model is log(S1 + S2). For example, the Commonwealth Games of India (CWG) plus a few other scams put together was a 9-zero scam, and this was followed by the 2G bandwidth sell-off, which was a thousand times bigger, i.e., a 12-zero scam. The cumulative response is:
log(10^12 + 10^9) = log(10.001 ^ 12) = 12.0005211273k,
a value that is barely more than the response to just the 2G scam itself (12k). In other words, the cumulative response to scams is approximately its soft maximum. The 2G scam simply eclipses the CWG scam in the mind of the public and the media. Even if there were 10 CWG-level scams, they'd be cumulatively eclipsed by a new 2G-level scam for all practical purposes and drop off the radar.
Scamster versus Scamster
Suppose a generally honest political party-B gets greedy seeing the muted public response and its forgiveness of prior scams. If it imitates party-A and pulls off a CWG-level scam, the public's anger would not be a 1000 times smaller. It would only be (12-9)/12 = 25% smaller. Having a clean prior record is not very helpful. Furthermore, if Party-B campaigned on a plank of 'transparent party that is different', it will be perceived as being not very different from party-A (75% alike) despite being 99.9% less corrupt compared to party-A, measured in terms of money siphoned off.
A MYLOP population that is forced to choose between two scamsters (even if their scam sizes are different) will pick the one that offers better freebies. If both provide equal freebies, the MYLOPs are likely to alternate between the two choices, every electoral cycle (e.g state of Tamil Nadu or Uttar Pradesh).
Statesman v/s Scamster - Round 2
Getting out a terrible locally-optimal situation may require a statesman to bring out and maintain a series of organic (sustainable and permanent) and highly visible improvements to first earn the trust of its MYLOP people over a sufficiently long period before unveiling any risky long-term development models. This means a lot of very hard infrastructure work: providing reliable 24x7 electricity, good roads, clean water, primary schools for girls, security for women, etc., to fundamentally alter the decision model of the population from MYLOP to a more optimistic and forward-looking model where people begin to see the possibilities within their grasp. An electorate that is moving along such a positive gradient is more likely to vote to continue along this path of incremental improvement without settling for "dont rock the old boat" and "here and now" populist freebies provided by the scamster. So the statesman can win (e.g. State of Gujarat), but it takes a lot of heavy lifting to be done upfront.
The two seemingly contradictory outcomes from the two state elections that were announced yesterday (Gujarat and Himachal Pradesh), as well as many other Indian elections in the recent past can be plausibly explained using these simple models.
Update 1: 01/21/2013
Interesting quote from 'Steve Jobs' top 5 mistakes" that directly relates to the above idea of hiding scams using bigger scams:
"... The lesson that I take from these defunct products is that people will soon forget that you were wrong on a lot of smaller bets, so long as you nail big bets in a major way (in Jobs's case, the iPod, iPhone, iPad, etc) .."
The loss due to government-blessed scams and failed populist schemes in India run in the order of Billions and Trillions of Indian Rupees in a country where the impoverished still form a sizable percentage and earning under or around the government designated poverty level of ₹20+ a day (50₹ ~ 1$). To many Indians, such twelve-zero numbers seem fantastic. Thus a person caught stealing thousands in a small town may receive a sound thrashing because people more clearly recognize the utility of amounts that actually show up in their historical experience. However, as the size of the theft increases beyond imaginable sums, the magnitude of the public response does not appear to proportionally increase. The law of diminishing returns seems to kick in, resulting in a concave response function. We postulate that:
Change in Public Response (ΔR) ∝ Change in # of Zeros (ΔZ) in Scam amount S , i.e
R = k log S, where k is a constant to be estimated using historical data.
The assumption of a logarithmic response model turns to be quite useful in the Indian context. For example, if political party A pulls off a 12-zero scam, and party B a 6-zero scam (using a log-10 base):
response(A) = 12k, and response(B) = 6k.
The response is only a factor of 2 more for the scam that was a million times bigger. This makes it easier for party-A to equate itself to party-B in the eyes of the public. It can have its cake and perhaps eat it too. Most Indians would not find it hard to map A & B to their real-life representatives.
Statesman versus Scamster - Round 1
A subjugated, under-informed electorate may exhibit a strong concave response that is characterized by relatively high sensitivity to small thefts at one end, and a disregard for mega-scams at the other end. We postulate that such a population's mental decision making model tends to be (1) myopic, (2) local optimality seeking, and (3) pessimistic (MYLOP). It accepts its terrible local living conditions provided the situation doesn't get perceptibly worse, and votes in polls based on this mental model. The immediate cause-and-effect tied to a local ₹100 theft represents a clear and present risk to current survival and is likely to elicit a swift and violent response, just like a populist cash-equivalent freebie elicits an equally enthusiastic response. However, a gigantic national scam that all but ensures that its future generations will see no improvement is shrugged off. MYLOP behavior is akin to a frog that allows itself to boiled alive because it does not register the gradual increase in water temperature until it is too late.
On the flip side, a government that is largely scam-free and claims to work on long-term growth, but is perceived to have marginally worsened the status quo can get booted out of power in the next election. Asking such a population to endure temporary 'hard choices' in order to be rewarded with medium-to-long term improvement ("no free lunch", "it will get worse before it gets better") implies a non-convex decision model - a hard sell since it clashes with the MYLOP attitude of the population. Scamsters + freebies trumps the Statesman here. Let's dig a little deeper into scams.
How to Hide a Scam?
Answer: A new scam that is an order of magnitude bigger. This can be explained via a mathematical model based on an interesting technical report by John Cook, M. D. Anderson Cancer Center (and author of the Endeavor blog):
The soft maximum of two variables is the function
g(x; y) = log(exp(x) + exp(y))
Given scams S1 and S2, the cumulative response from the public per our postulated model is log(S1 + S2). For example, the Commonwealth Games of India (CWG) plus a few other scams put together was a 9-zero scam, and this was followed by the 2G bandwidth sell-off, which was a thousand times bigger, i.e., a 12-zero scam. The cumulative response is:
log(10^12 + 10^9) = log(10.001 ^ 12) = 12.0005211273k,
a value that is barely more than the response to just the 2G scam itself (12k). In other words, the cumulative response to scams is approximately its soft maximum. The 2G scam simply eclipses the CWG scam in the mind of the public and the media. Even if there were 10 CWG-level scams, they'd be cumulatively eclipsed by a new 2G-level scam for all practical purposes and drop off the radar.
Scamster versus Scamster
Suppose a generally honest political party-B gets greedy seeing the muted public response and its forgiveness of prior scams. If it imitates party-A and pulls off a CWG-level scam, the public's anger would not be a 1000 times smaller. It would only be (12-9)/12 = 25% smaller. Having a clean prior record is not very helpful. Furthermore, if Party-B campaigned on a plank of 'transparent party that is different', it will be perceived as being not very different from party-A (75% alike) despite being 99.9% less corrupt compared to party-A, measured in terms of money siphoned off.
A MYLOP population that is forced to choose between two scamsters (even if their scam sizes are different) will pick the one that offers better freebies. If both provide equal freebies, the MYLOPs are likely to alternate between the two choices, every electoral cycle (e.g state of Tamil Nadu or Uttar Pradesh).
Statesman v/s Scamster - Round 2
Getting out a terrible locally-optimal situation may require a statesman to bring out and maintain a series of organic (sustainable and permanent) and highly visible improvements to first earn the trust of its MYLOP people over a sufficiently long period before unveiling any risky long-term development models. This means a lot of very hard infrastructure work: providing reliable 24x7 electricity, good roads, clean water, primary schools for girls, security for women, etc., to fundamentally alter the decision model of the population from MYLOP to a more optimistic and forward-looking model where people begin to see the possibilities within their grasp. An electorate that is moving along such a positive gradient is more likely to vote to continue along this path of incremental improvement without settling for "dont rock the old boat" and "here and now" populist freebies provided by the scamster. So the statesman can win (e.g. State of Gujarat), but it takes a lot of heavy lifting to be done upfront.
The two seemingly contradictory outcomes from the two state elections that were announced yesterday (Gujarat and Himachal Pradesh), as well as many other Indian elections in the recent past can be plausibly explained using these simple models.
Update 1: 01/21/2013
Interesting quote from 'Steve Jobs' top 5 mistakes" that directly relates to the above idea of hiding scams using bigger scams:
"... The lesson that I take from these defunct products is that people will soon forget that you were wrong on a lot of smaller bets, so long as you nail big bets in a major way (in Jobs's case, the iPod, iPhone, iPad, etc) .."
Monday, December 17, 2012
Collection of Notes on Gun Control
Notes update 1: Dec 18.
Notes update 2: Dec 19.
Notes update 3: Jan 02.
Introduction: The US Outlier
This post is merely an extension of the discussion initiated by Professor Rubin on how not to debate gun control - a discussion that is well worth reading, and one that I have referred to multiple times to try and comprehend a situation that is unique to the US while also sharing certain tragic commonalities with other countries. Much of this post are 'note to self' type remarks for future reference and is a largely unstructured collection of pointers to various articles and personal inferences drawn, which may change as more data becomes available.
The country-independent issues are brought out well in the blog cited in Prof. Rubin's post. That the issue of gun-ownership & deaths also has a US-specific dimension is apparent in this picture presented by a comment on Dr. Rubin's blog.
(link source: https://dl.dropbox.com/u/38668/deaths-vs-guns.png, Courtesy 'Christian')
Macro-Trends
Note however that Newtown was a specific incident (a 'sample of one', as Dr. Rubin notes), whereas this picture above captures macro-trends. Examine these six time-series based facts published by the Washington Post. Assuming this report is factual, the first graphic reconfirms the existence of a unique US situation shown in the first scatter plot. We will focus our attention on just the first plot in this post. There are other interesting plots that show:
- a correlation (not necessarily causality) between stricter gun-controlled US states and lower rates of gun-related deaths
- Southern US has the highest gun-driven fatality rates, while the Northeast has the lowest,
- Gun ownership rates are declining, etc.
The above graphic reveals that overall gun-related fatality rates in the US have monotonically dropped over the last twenty years. Neither the economic boom of the 1990s or the relative economic decline in the last decade, or demographic shifts, etc. has managed to cause a significant inflection at the macro-level. Does that mean that the violence in the society has dropped overall (reduction in market size), and/or have people found alternatives to guns to express their violence (reduction in market-share)? We may need several other charts to answer that. However, purely based on this time-series, it appears that what we have in place has apparently worked relatively well at a macro-level so far. We may be tempted to think that by maintaining the status-quo we are on course to achieve parity with the rest of the pack with respect to this specific metric, but to predict the future is difficult, and requires some knowledge of causality (or at least correlations that can give us a tiny hint).
Significant Incremental Improvement
To motivate the next discussion, let us recall this very useful comment by Dr. Rubin:
"Many (including, I'm willing to bet, Mr. Cassingham) would consider a reduction in mass shootings, or even a reduction in the body counts, as a significant improvement (satisficing)."
From the point of view of the person who wants to act violently, they may well use the first (or most suitable) 'weaponizable' object they can first lay their hands on, and thus a gun or a knife may well represent alternative optimal solutions to a perpetrator. The time durations for:
a. formation of violent intent
b. searching for optimal method to express intent
c. translating intent into reality
seem to be quite important. These time durations may range from a few milliseconds to years. Mr. Cassingham's blog rightly notes in more than one place (and I paraphrase) 'if not guns, then people will always find something else' (in stage (b). True. Addressing the root cause of violence (that arises in a person's mind, and incubated over varying time durations in stage (a)) is very important to the discussion, so that we never get to see the realization of later stages. Perhaps addressing each of these stages (a-c) in some order of priority may help.
Addressing (a) is likely to reduce the probability of future incidents, and is something that may have to be addressed in from the short-term p.o.v (e.g. using medical science), and all the way to the long term where the world can move beyond mere tolerance for differences to mutual respect that celebrates difference. Addressing (b) can reduce the expected consequence of a violent incident, given that a tragic incident does occur (ideally to zero). Addressing (c) appears to be tied to 'last line of defense' mechanisms in place.
One of the key contributions of Operations Research is to recognize alternative optimal solutions. Dr. Rubin's idea of 'optimality versus 'satisficing' seems pretty valuable. From the point of view of expected casualties | given an attempt to mass-injure, a smaller gun, a knife or a baseball bat will probabilistically have a lower casualty rate. An explosive device or a WMD-type gun (like that used in Newtown) is likely to kill more per unit time, on average. Thus from the perspective of a group of trapped victims, horrific as it may be, there may exist a reasonably well-defined ranking of what weapons we least prefer to face (obviously 'facing no weapons' would be ideal). It would beneficial if those who already in stage (b) are always forced to switch to a weapon that is far less deadlier than what was available before. We could think of the deadliness of the weapons permitted as a time-variable 'upper bound' on weapons control. There may be a variety of such bounds along each dimension of the weapon and ammunition, as well as lower bounds on re-training frequency, life-event triggers for re-evaluation of permits, etc. Doing so may provide a gradient along which this discussion can take constructive steps toward reaching the most satisfactory solution by iteratively adjusting these bounds. Again, there are likely to be multiple goals having different priorities as already mentioned in Prof. Rubin's post. The presence of constraints may result in 'satisficing' rather than achieving the desired 'global optimality'. One also wonders about the disruptive impact of 3D printing of weapons on such discussions.
(pic linked from popular science)
What do we do when people have passed through stages (a)-(b), and ready to move to stage (c)? This is the stage, where their intent, as well as degree of planning starts to become visible to others, but the duration may be extremely limited...
Average versus Distribution
The next discussion borrows from a very recent post of another ORMS professor. Dr. Trick wrote about the tricky issue of averages recently in his post "which average do you want?". In addition to the amazing and heroic teachers and staff of Sandy Hook Elementary School, Newton, we just lost eighteen of our most precious and littlest citizens in a matter of minutes after near-complete peace and quiet in that area during the previous 364 days. Such sudden, scarcely comprehensible mass murders (low-probability, catastrophic consequence events) suck the oxygen out of a nation's morale compared to an alternative and equal cumulative tragedy that is well spread out over time and space. Dr. John Cook's blog today talks about the nonlinear effect of 'batch size'. In either of these equally tragic scenarios, from every individual victim's family p.o.v, the anguish and price paid over time is likely to be the same - too high to imagine. Picture 1 of the WaPo article only reveals a reduction in the overall rate, but not the changes in the distribution (victim's age, motive, ..). The nation's response last week leaves little doubt that the distribution of this statistic is just as much if not more important.
Sam Harris notes in his "The Riddle of the gun" :
"... As my friend Steven Pinker demonstrates in his monumental study of human violence, The Better Angels of Our Nature, our perception of danger is easily distorted by rare events. Is gun violence increasing in the United States? No. But it certainly seems to be when one recalls recent atrocities in Newtown and Aurora. In fact, the overall rate of violent crime has fallen by 22 percent in the past decade (and 18 percent in the past five years).
We still have more guns and more gun violence than any other developed country, but the correlation between guns and violence in the United States is far from straightforward...
..... Seventy mass shootings have occurred in the U.S. since 1982, leaving 543 dead. These crimes were horrific, but 564,452 other homicides took place in the U.S. during the same period. Mass shootings scarcely represent 0.1 percent of all murders. When talking about the problem of guns in our society, it is easy to lose sight of the worst violence and to become fixated on symbols of violence ... "
LPHC Events in the Life cycle of a Gun
Finally, to the gun itself. Like a pair of scissors, a kitchen knife, or a car, a gun is a tool that has associated with it a probability for helping and hurting simultaneously whenever it is is handled. Associated with each of these tools is an expected frequency of usage and a risk profile that is context dependent (such as geographical location). The intended target of the primary benefit is largely limited to self and family (who bear the cost of maintenance), whereas the cascading and probabilistic liability is necessarily borne by self, family, and others. Every additional gun, its type, and associated inventory of ammunition, increases this probabilistic liability to the owner, his/her family, and public in a certain way, while altering the expected probabilistic benefit to self in another way. In many instances, both the benefit as well as the liability of acquiring guns appear to be tied to Low-Probability, High-consequence events. In extremely peaceful places, the expected liability may exceed the expected benefit over the lifetime of the weapon, whereas in lawless places, the opposite may hold true.
Ahimsa: Notes from Books
Louis L'Amour noted: "When guns are outlawed, only the outlaws have guns". One of his best books, where violence is almost a living character is 'The Daybreakers" that ultimately ends with a note on the self-realization of the main protagonist leading to his rejection of senseless violence: "We found our home, and we graze and work our acres, and since that day in the street of Mora when I killed Tom Sunday, I have never drawn a gun on any man. Nor will I ...". King Asoka of India fought and won the bloody battle of Kalinga, but was so shocked by the carnage that he gave up further violent conquests and turned toward meditation and the Dharmic way of life. The multi-millennium old Sanskrit Shloka on Ahimsa (non-harming) in Hindu Philosophy says (ref: Hindupedia):
अहिंसा परमो धर्मः
धर्म हिंसा तथीव च
Translation: Non-harming is the ultimate Dharma (very loosely translated as 'righteous way of life'). Harming in service of Dharma (only) is equally virtuous. Consequently, enduring or ignoring, rather than opposing tyranny is not a virtue. The spirit of the second amendment (arguably) and Asimov's three (plus zeroth) laws of robotics come across as examples of the application of this profound Shloka. Some of Gandhi's more head-scratching ideas are apparently due to his ambivalent treatment of the second line of the Shloka, although the Mahatma did say (ref: http://www.mkgandhi.org)
"I do believe that, where there is only a choice between cowardice and violence, I would advise violence... I would rather have India resort to arms in order to defend her honour than that she should, in a cowardly manner, become or remain a helpless witness to her own dishonor. But I believe that nonviolence is infinitely superior to violence, forgiveness is more manly than punishment"
In Remembrance: Dr.L (2007) and Little Maddy (2012). Om Shanti.
Notes update 2: Dec 19.
Notes update 3: Jan 02.
Introduction: The US Outlier
This post is merely an extension of the discussion initiated by Professor Rubin on how not to debate gun control - a discussion that is well worth reading, and one that I have referred to multiple times to try and comprehend a situation that is unique to the US while also sharing certain tragic commonalities with other countries. Much of this post are 'note to self' type remarks for future reference and is a largely unstructured collection of pointers to various articles and personal inferences drawn, which may change as more data becomes available.
The country-independent issues are brought out well in the blog cited in Prof. Rubin's post. That the issue of gun-ownership & deaths also has a US-specific dimension is apparent in this picture presented by a comment on Dr. Rubin's blog.
(link source: https://dl.dropbox.com/u/38668/deaths-vs-guns.png, Courtesy 'Christian')
Macro-Trends
Note however that Newtown was a specific incident (a 'sample of one', as Dr. Rubin notes), whereas this picture above captures macro-trends. Examine these six time-series based facts published by the Washington Post. Assuming this report is factual, the first graphic reconfirms the existence of a unique US situation shown in the first scatter plot. We will focus our attention on just the first plot in this post. There are other interesting plots that show:
- a correlation (not necessarily causality) between stricter gun-controlled US states and lower rates of gun-related deaths
- Southern US has the highest gun-driven fatality rates, while the Northeast has the lowest,
- Gun ownership rates are declining, etc.
The above graphic reveals that overall gun-related fatality rates in the US have monotonically dropped over the last twenty years. Neither the economic boom of the 1990s or the relative economic decline in the last decade, or demographic shifts, etc. has managed to cause a significant inflection at the macro-level. Does that mean that the violence in the society has dropped overall (reduction in market size), and/or have people found alternatives to guns to express their violence (reduction in market-share)? We may need several other charts to answer that. However, purely based on this time-series, it appears that what we have in place has apparently worked relatively well at a macro-level so far. We may be tempted to think that by maintaining the status-quo we are on course to achieve parity with the rest of the pack with respect to this specific metric, but to predict the future is difficult, and requires some knowledge of causality (or at least correlations that can give us a tiny hint).
Significant Incremental Improvement
To motivate the next discussion, let us recall this very useful comment by Dr. Rubin:
"Many (including, I'm willing to bet, Mr. Cassingham) would consider a reduction in mass shootings, or even a reduction in the body counts, as a significant improvement (satisficing)."
From the point of view of the person who wants to act violently, they may well use the first (or most suitable) 'weaponizable' object they can first lay their hands on, and thus a gun or a knife may well represent alternative optimal solutions to a perpetrator. The time durations for:
a. formation of violent intent
b. searching for optimal method to express intent
c. translating intent into reality
seem to be quite important. These time durations may range from a few milliseconds to years. Mr. Cassingham's blog rightly notes in more than one place (and I paraphrase) 'if not guns, then people will always find something else' (in stage (b). True. Addressing the root cause of violence (that arises in a person's mind, and incubated over varying time durations in stage (a)) is very important to the discussion, so that we never get to see the realization of later stages. Perhaps addressing each of these stages (a-c) in some order of priority may help.
Addressing (a) is likely to reduce the probability of future incidents, and is something that may have to be addressed in from the short-term p.o.v (e.g. using medical science), and all the way to the long term where the world can move beyond mere tolerance for differences to mutual respect that celebrates difference. Addressing (b) can reduce the expected consequence of a violent incident, given that a tragic incident does occur (ideally to zero). Addressing (c) appears to be tied to 'last line of defense' mechanisms in place.
One of the key contributions of Operations Research is to recognize alternative optimal solutions. Dr. Rubin's idea of 'optimality versus 'satisficing' seems pretty valuable. From the point of view of expected casualties | given an attempt to mass-injure, a smaller gun, a knife or a baseball bat will probabilistically have a lower casualty rate. An explosive device or a WMD-type gun (like that used in Newtown) is likely to kill more per unit time, on average. Thus from the perspective of a group of trapped victims, horrific as it may be, there may exist a reasonably well-defined ranking of what weapons we least prefer to face (obviously 'facing no weapons' would be ideal). It would beneficial if those who already in stage (b) are always forced to switch to a weapon that is far less deadlier than what was available before. We could think of the deadliness of the weapons permitted as a time-variable 'upper bound' on weapons control. There may be a variety of such bounds along each dimension of the weapon and ammunition, as well as lower bounds on re-training frequency, life-event triggers for re-evaluation of permits, etc. Doing so may provide a gradient along which this discussion can take constructive steps toward reaching the most satisfactory solution by iteratively adjusting these bounds. Again, there are likely to be multiple goals having different priorities as already mentioned in Prof. Rubin's post. The presence of constraints may result in 'satisficing' rather than achieving the desired 'global optimality'. One also wonders about the disruptive impact of 3D printing of weapons on such discussions.
(pic linked from popular science)
What do we do when people have passed through stages (a)-(b), and ready to move to stage (c)? This is the stage, where their intent, as well as degree of planning starts to become visible to others, but the duration may be extremely limited...
Average versus Distribution
The next discussion borrows from a very recent post of another ORMS professor. Dr. Trick wrote about the tricky issue of averages recently in his post "which average do you want?". In addition to the amazing and heroic teachers and staff of Sandy Hook Elementary School, Newton, we just lost eighteen of our most precious and littlest citizens in a matter of minutes after near-complete peace and quiet in that area during the previous 364 days. Such sudden, scarcely comprehensible mass murders (low-probability, catastrophic consequence events) suck the oxygen out of a nation's morale compared to an alternative and equal cumulative tragedy that is well spread out over time and space. Dr. John Cook's blog today talks about the nonlinear effect of 'batch size'. In either of these equally tragic scenarios, from every individual victim's family p.o.v, the anguish and price paid over time is likely to be the same - too high to imagine. Picture 1 of the WaPo article only reveals a reduction in the overall rate, but not the changes in the distribution (victim's age, motive, ..). The nation's response last week leaves little doubt that the distribution of this statistic is just as much if not more important.
Sam Harris notes in his "The Riddle of the gun" :
"... As my friend Steven Pinker demonstrates in his monumental study of human violence, The Better Angels of Our Nature, our perception of danger is easily distorted by rare events. Is gun violence increasing in the United States? No. But it certainly seems to be when one recalls recent atrocities in Newtown and Aurora. In fact, the overall rate of violent crime has fallen by 22 percent in the past decade (and 18 percent in the past five years).
We still have more guns and more gun violence than any other developed country, but the correlation between guns and violence in the United States is far from straightforward...
..... Seventy mass shootings have occurred in the U.S. since 1982, leaving 543 dead. These crimes were horrific, but 564,452 other homicides took place in the U.S. during the same period. Mass shootings scarcely represent 0.1 percent of all murders. When talking about the problem of guns in our society, it is easy to lose sight of the worst violence and to become fixated on symbols of violence ... "
LPHC Events in the Life cycle of a Gun
Finally, to the gun itself. Like a pair of scissors, a kitchen knife, or a car, a gun is a tool that has associated with it a probability for helping and hurting simultaneously whenever it is is handled. Associated with each of these tools is an expected frequency of usage and a risk profile that is context dependent (such as geographical location). The intended target of the primary benefit is largely limited to self and family (who bear the cost of maintenance), whereas the cascading and probabilistic liability is necessarily borne by self, family, and others. Every additional gun, its type, and associated inventory of ammunition, increases this probabilistic liability to the owner, his/her family, and public in a certain way, while altering the expected probabilistic benefit to self in another way. In many instances, both the benefit as well as the liability of acquiring guns appear to be tied to Low-Probability, High-consequence events. In extremely peaceful places, the expected liability may exceed the expected benefit over the lifetime of the weapon, whereas in lawless places, the opposite may hold true.
Ahimsa: Notes from Books
Louis L'Amour noted: "When guns are outlawed, only the outlaws have guns". One of his best books, where violence is almost a living character is 'The Daybreakers" that ultimately ends with a note on the self-realization of the main protagonist leading to his rejection of senseless violence: "We found our home, and we graze and work our acres, and since that day in the street of Mora when I killed Tom Sunday, I have never drawn a gun on any man. Nor will I ...". King Asoka of India fought and won the bloody battle of Kalinga, but was so shocked by the carnage that he gave up further violent conquests and turned toward meditation and the Dharmic way of life. The multi-millennium old Sanskrit Shloka on Ahimsa (non-harming) in Hindu Philosophy says (ref: Hindupedia):
अहिंसा परमो धर्मः
धर्म हिंसा तथीव च
Translation: Non-harming is the ultimate Dharma (very loosely translated as 'righteous way of life'). Harming in service of Dharma (only) is equally virtuous. Consequently, enduring or ignoring, rather than opposing tyranny is not a virtue. The spirit of the second amendment (arguably) and Asimov's three (plus zeroth) laws of robotics come across as examples of the application of this profound Shloka. Some of Gandhi's more head-scratching ideas are apparently due to his ambivalent treatment of the second line of the Shloka, although the Mahatma did say (ref: http://www.mkgandhi.org)
"I do believe that, where there is only a choice between cowardice and violence, I would advise violence... I would rather have India resort to arms in order to defend her honour than that she should, in a cowardly manner, become or remain a helpless witness to her own dishonor. But I believe that nonviolence is infinitely superior to violence, forgiveness is more manly than punishment"
In Remembrance: Dr.L (2007) and Little Maddy (2012). Om Shanti.
Thursday, December 13, 2012
The King and the Vampire - 2: Cricketing Conundrum
This is the second episode in our 'King Vikram and the Vetaal' (Vampire) series, based on the simple but nice Indian story-telling format. Read the first K&V story here.
Dark was the night and weird the atmosphere. It rained from time to time. Eerie laughter of ghosts rose above the moaning of jackals. Flashes of lightning revealed fearful faces. But King Vikram did not swerve. He climbed the ancient tree once again and brought the corpse down. With the corpse lying astride on his shoulder, he began crossing the desolate cremation ground. "O King, it seems that you are generous in your appreciation for the analytics-based Duckworth-Lewis method used in weather-interrupted cricket matches. But it is better for you to know that there are situations where the D/L method invariably results in complaints, especially in T20 cricket. Let me cite an instance. Pay your attention to my narration. That might bring you some relief as you trudge along," said the vampire that possessed the corpse.
The cricketing vampire went on:
In a recent Australian T20 match, the D/L method did not just perform badly, it actually failed. Here's why:
Team-A batted first, played terribly and made just 69 runs.
Team-B chasing 70 for a win, got off to a flyer and scored 29 of 2 overs before rain interrupted the match. However, the D/L revised target for Team-B only comes into play when at least 5 overs have been completed by both teams... Anyway, here's what transpired, per cricinfo:
(pic source link: omshivam.files.wordpress.com)
Dark was the night and weird the atmosphere. It rained from time to time. Eerie laughter of ghosts rose above the moaning of jackals. Flashes of lightning revealed fearful faces. But King Vikram did not swerve. He climbed the ancient tree once again and brought the corpse down. With the corpse lying astride on his shoulder, he began crossing the desolate cremation ground. "O King, it seems that you are generous in your appreciation for the analytics-based Duckworth-Lewis method used in weather-interrupted cricket matches. But it is better for you to know that there are situations where the D/L method invariably results in complaints, especially in T20 cricket. Let me cite an instance. Pay your attention to my narration. That might bring you some relief as you trudge along," said the vampire that possessed the corpse.
(pic source link: pryas.wordpress.com)
The cricketing vampire went on:
In a recent Australian T20 match, the D/L method did not just perform badly, it actually failed. Here's why:
Team-A batted first, played terribly and made just 69 runs.
Team-B chasing 70 for a win, got off to a flyer and scored 29 of 2 overs before rain interrupted the match. However, the D/L revised target for Team-B only comes into play when at least 5 overs have been completed by both teams... Anyway, here's what transpired, per cricinfo:
".. Under the Duckworth/Lewis method the target for the Stars (Team-B) was
recalculated. The calculation, which itself has been disputed, ensured
that the Stars required just six runs from five overs. Even though the
Stars had already reached and exceeded the target, given the D/L target
had changed when overs were lost play needed to resume to set the
revised target. Play resumed at 7.52pm after a minor delay. Hilton Cartwright bowled one
ball to Rob Quiney, who allowed it to pass through to the keeper, and
the match was over as the Stars had reached their revised target after
2.1 overs."
So tell me King Vikram, What is the correct result? Did the Stars win because they achieved the revised target, or should the points be shared because the required five overs were not completed? Answer me if you can. Should you keep mum though you may know the answers, your head would roll off your shoulders!"
King Vikram was silent for a while, and then spoke: "Vetaal, unlike the last time, this is a tough one, so first consider this counter-factual:
King Vikram was silent for a while, and then spoke: "Vetaal, unlike the last time, this is a tough one, so first consider this counter-factual:
The Stars continue to play for another 1.4 overs, and are bowled out for 29. If the revised target when they were 29/9 was 30, then Team-A (Scorchers) would have won the match. Therefore, even though the Stars were temporarily ahead of the revised target, that target is not static since the minimum overs were not completed. The D/L based revised target is computed based on two resource constraints, taking into account the runs to be scored and wickets lost. It can change over time and the Scorchers still had a theoretical chance, however small, of winning the match by taking wickets.
However, here's another situation. Suppose Team-B was 29/9 after 2 overs, and the revised 5-over D/L target was 52. Team-B gets to 57/9 in 4.5 overs, hitting the last ball for a six before rains come down and and stop the game permanently. In this case, the D/L target cannot increase further, given that Team-B is already 9-down. In this case, Team-A has zero chance of winning the match. Team-B should be declared the winner even though the minimum overs have not been bowled.
1. For the current game, the points have to shared. This answers your specific question.
2. If a match ends before the minimum overs are completed, the chasing team can be declared the winner only if they have achieved a score that equals or exceeds the highest of all possible revised D/L targets that can occur for the fixed number of overs possible. In this example, Team-B would have been declared the winner if they scored at least 52.
3. However, I suspect that the cricket council will simply enforce the minimum-over rule as a hard-constraint that must be satisfied before the match can be decided in favor of one team over the other.
3. However, I suspect that the cricket council will simply enforce the minimum-over rule as a hard-constraint that must be satisfied before the match can be decided in favor of one team over the other.
No sooner had King Vikram concluded his answer than the vampire, along with the corpse, gave him the slip.
(pic source link: omshivam.files.wordpress.com)
The Vetaal apparently agreed with Vikram's answer, but do you? If not, explain why. There's no penalty for trying!
Analytics and Cricket - X : Reader's Response to DRS Debate
It's getting increasingly difficult to post on cricket given that the Indian cricket team is getting ripped to shreds by half-decent opposition despite home-ground advantage. Of course, as noted in an
earlier post,
home courts can significantly increase the chance of a choke, and this may well be happening. Mahendra Singh Dhoni (if
by some chance, still remains the captain of the Indian team
after the current cricket series) can win a few more tosses if he can exploit this idea. Desperate times call for analytical measures!
Meanwhile, an astute reader emailed a detailed response to the Bayes-theorem based analysis of the Decision Review System (DRS) used in cricket, which was posted on this blog a few months ago. He made some very pertinent points along with some brilliant comments on the game, which led to an informative exchange that will be carried in the next couple of cricket-related posts. Here is our the 2x2 color coded DRS matrix again for reference.
Raghavan notes:
".... I must question some of the steps in your analysis:
1. In your derivation you use P(RED|OUT) = 0.95. I think this is true only if all decisions are left to DRS. You have considered only those decisions that are deemed not out by the umpire and referred. The 95% number does not hold for these selected cases. It would be lower. Here's the rationale:
There is a high degree of correlation between DRS and umpires decisions; understandably so, since all those "plumb" decisions are easy for both, the umpire and DRS. Bowlers would rarely review these decisions. For the 10% or so cases when the umpire rules the batsman not out incorrectly, the DRS would very likely have a lower accuracy than its overall 95%.
2. If you assume the "red zone" in the picture is sufficiently small compared to 10%, you would get the accuracy of DRS being about 50% for the cases when the umpire incorrectly rules not out. Now, this needs a bit of explanation.
Let's assume that whenever the umpire rules out correctly, the DRS also rules correctly (well, at least close to 100% of the time). Note that this does not include just the referrals, but also all the "easy and obvious" decisions that are not referred. Since the overall accuracy of DRS is 95%, of the 10% that the umpire incorrectly rules not out, DRS also gets it wrong for half of those 10% cases giving an overall 95% accuracy. In case the "red zone" corresponding to incorrect OUT decisions of the DRS is not close to zero, but say 2% (which is large in my opinion), the DRS accuracy in the bowler referred cases we are talking of would by 70% rather than 50%. Still way lower than the 95% overall accuracy. [I have made some approximations here, but the overall logic hold]
3. Now, if you plug 70% instead of 95% in your next steps, you get P(OUT|RED) = 88.6%. Nothing wrong with this number, except when you compare it with the 90% accuracy of umpires. It's not apples to apples. P(OUT|Umpire says OUT) is not 90% if you only considered referred cases. It's actually a conditional probability:
P(OUT|Umpire says OUT, BOWLER REFERS). I don't have enough information to estimate this, but I'm sure you'll agree it's lower than 90% since bowlers don't refer randomly.
4. I think the right comparison is between the what the final decision would be with and without DRS. There is no doubt that umpire + DRS referrals improve overall accuracy of decisions. I admit that false positives would increase marginally, which affects batsmen more than bowlers because of the nature of the game (a batsman has no chance of a comeback after a bad decision, while a bowler does). But I think it is because of the way Hawk-eye is used today.
5. In my opinion, the main problem with DRS is that its decision are made to be black and white. There should be a reliability measure used. A very rudimentary form of this currently used in LBW decisions. For example, if the umpire has ruled out, to be ruled not out by DRS the predicted ball path has to completely miss the stumps. But if the umpire has ruled not out, the predicted path should show that at least half the ball is within the stumps for the decision to be over-turned. Eventually, I feel Hawk eye would be able to estimate the accuracy of it's decision. I'm sure Hawk eye has statistics on it's estimates. The standard deviation of the estimate would depend on several factors - (1) how far in from of the stumps has the ball struck the pads (2) How close to the pads has the ball pitched (hawk-eye needs at least a couple of feet after the bounce to track the changed trajectory), (3) Amount of turn, swing or seam movement observed.
If a standard deviation (sigma) can be estimated, then a window of say +/- 3*sigma could be used as the "region of uncertainty". If the ball is predicted to hit the stumps within this region of uncertainty then the decision should be out. Of course the more complicated it gets to explain to the viewer, the more resistance there would be to be accepted. But if it is the right way, it will eventually get accepted. Take DL method for example. A vast majority of viewers don't understand it fully, but most of them know that it is fair.
6. There's another aspect that needs to be investigated. It's about how the decision of the on-field umpire is affected by the knowledge that DRS is available. "
Followups will be carried in a subsequent post. Blog-related emails can be sent to: dual[no space or dot here]noise AT gmail dot com, or simply send a tweet.
Meanwhile, an astute reader emailed a detailed response to the Bayes-theorem based analysis of the Decision Review System (DRS) used in cricket, which was posted on this blog a few months ago. He made some very pertinent points along with some brilliant comments on the game, which led to an informative exchange that will be carried in the next couple of cricket-related posts. Here is our the 2x2 color coded DRS matrix again for reference.
Raghavan notes:
".... I must question some of the steps in your analysis:
1. In your derivation you use P(RED|OUT) = 0.95. I think this is true only if all decisions are left to DRS. You have considered only those decisions that are deemed not out by the umpire and referred. The 95% number does not hold for these selected cases. It would be lower. Here's the rationale:
There is a high degree of correlation between DRS and umpires decisions; understandably so, since all those "plumb" decisions are easy for both, the umpire and DRS. Bowlers would rarely review these decisions. For the 10% or so cases when the umpire rules the batsman not out incorrectly, the DRS would very likely have a lower accuracy than its overall 95%.
2. If you assume the "red zone" in the picture is sufficiently small compared to 10%, you would get the accuracy of DRS being about 50% for the cases when the umpire incorrectly rules not out. Now, this needs a bit of explanation.
Let's assume that whenever the umpire rules out correctly, the DRS also rules correctly (well, at least close to 100% of the time). Note that this does not include just the referrals, but also all the "easy and obvious" decisions that are not referred. Since the overall accuracy of DRS is 95%, of the 10% that the umpire incorrectly rules not out, DRS also gets it wrong for half of those 10% cases giving an overall 95% accuracy. In case the "red zone" corresponding to incorrect OUT decisions of the DRS is not close to zero, but say 2% (which is large in my opinion), the DRS accuracy in the bowler referred cases we are talking of would by 70% rather than 50%. Still way lower than the 95% overall accuracy. [I have made some approximations here, but the overall logic hold]
3. Now, if you plug 70% instead of 95% in your next steps, you get P(OUT|RED) = 88.6%. Nothing wrong with this number, except when you compare it with the 90% accuracy of umpires. It's not apples to apples. P(OUT|Umpire says OUT) is not 90% if you only considered referred cases. It's actually a conditional probability:
P(OUT|Umpire says OUT, BOWLER REFERS). I don't have enough information to estimate this, but I'm sure you'll agree it's lower than 90% since bowlers don't refer randomly.
4. I think the right comparison is between the what the final decision would be with and without DRS. There is no doubt that umpire + DRS referrals improve overall accuracy of decisions. I admit that false positives would increase marginally, which affects batsmen more than bowlers because of the nature of the game (a batsman has no chance of a comeback after a bad decision, while a bowler does). But I think it is because of the way Hawk-eye is used today.
5. In my opinion, the main problem with DRS is that its decision are made to be black and white. There should be a reliability measure used. A very rudimentary form of this currently used in LBW decisions. For example, if the umpire has ruled out, to be ruled not out by DRS the predicted ball path has to completely miss the stumps. But if the umpire has ruled not out, the predicted path should show that at least half the ball is within the stumps for the decision to be over-turned. Eventually, I feel Hawk eye would be able to estimate the accuracy of it's decision. I'm sure Hawk eye has statistics on it's estimates. The standard deviation of the estimate would depend on several factors - (1) how far in from of the stumps has the ball struck the pads (2) How close to the pads has the ball pitched (hawk-eye needs at least a couple of feet after the bounce to track the changed trajectory), (3) Amount of turn, swing or seam movement observed.
If a standard deviation (sigma) can be estimated, then a window of say +/- 3*sigma could be used as the "region of uncertainty". If the ball is predicted to hit the stumps within this region of uncertainty then the decision should be out. Of course the more complicated it gets to explain to the viewer, the more resistance there would be to be accepted. But if it is the right way, it will eventually get accepted. Take DL method for example. A vast majority of viewers don't understand it fully, but most of them know that it is fair.
6. There's another aspect that needs to be investigated. It's about how the decision of the on-field umpire is affected by the knowledge that DRS is available. "
Followups will be carried in a subsequent post. Blog-related emails can be sent to: dual[no space or dot here]noise AT gmail dot com, or simply send a tweet.
Subscribe to:
Posts (Atom)