The One-Dollar Haircut

Every year, I travel more than 9K miles to my native place in Tamil Nadu, India, to get my $1 haircut.

(pic source: http://jhingu.com)

The local barber is an expert and does it the old-fashioned way. A comb to sift, and super-quick scissors acting as efficient cutting planes to produce a nice, convex-hull hairdo. No fancy machinery, seating, lighting, or A/C in his environment-friendly shop, and not a single nick, or hair out of place when he's done. This keeps his customers happy and operating cost low. It's a dollar regardless of crop density. Perhaps the argument is that for the sparse-headed, the cost of searching goes up even as the actual cutting time reduces. If the shop is full, spill-over customers can sit in makeshift chairs outside the shop, and sip chai from the adjoining 'tea-kadai'.

The Indian barber shop is also an instance of the 'elastic' capacity model that is key to understanding how the Indian economy has chugged along. Here is what Prof. R. Vaidyanathan, Professor at the Indian Institute of Management, Bangalore, in his superb book 'India, Uninc' has to say:

Data shows that it this 'unincorporated' economy built from the ground-up by entrepreneurs that is responsible for much of India's GDP and employment, and not the stock market that grabs headlines. Despite the recent crash of the global markets, the Indian elephant is likely to remain solid, like it did after 2008.

The traditional Indian way of doing business may be initially bewildering to the external observer, which may have led to this 'Uninc' being categorized by Nehruvian India's west-educated officials as "unorganized". Actually, it is anything but, and appears to be based on balancing constraint-enforcing 'order' and constraint-relaxing 'chaos'. This traditional Indian approach, which I personally view as dharmic optimization, yields astonishingly high and sustainable levels of quality and efficiency when done right, but can also be disastrous when either order or chaos is excessive.

End Note: Apparently, the global wig supply-chain sources much of its hair from India.

Traveling Surveyor: Contributions of Mahalanobis to Analytics - 2

Bengal
This post is the second in a series of blogs based on the work done by P. C. Mahalanobis in the area of statistics, analytics, and operations research between 1930-1960. We move north-east from the location of our prior post on storm-flood forecasting (Orissa) to Bengal: a land of science, wisdom, and dharma that gave to the world a Vivekananda whose thoughts deeply affected Gandhi's contribution to India's freedom struggle, who in turn shaped the work of Martin Luther King, Jr., and Nelson Mandela, and thus the civil liberties of a significant population of the world.

Much of discussion here is gleaned from ISI archives, websites, and various papers from Sankhya, ISI's flagship journal. Those interested in a more detailed and accurate analysis of this work are referred directly to ISI's journal material.

Jute
Bengal (including Bangladesh, formerly East Bengal) produces much of the world's jute. India is the largest producer and consumer of Jute today, followed by Bangladesh. Jute is an incredibly useful crop and has been a significant contributor to Bengal's revenue for a long time.  Here are some contemporary pictures of standing Jute crop in Bengal.

(pics source: informedfarmers.com)

Why this work is important
Prior to 1947, when India was still occupied by the British Raj (there's a very relevant reason for bringing this up, but we'll get to that later), forecasting the supply of this valuable and lucrative crop in Bengal was largely a product of bad guesswork. Like most other sectors in India, the agricultural sector too is highly decentralized, which means a myriad of tiny farms growing jute and other crops, all of which had to be surveyed if one wanted to get an exact, enumerated production number. Mahalanobis came up with an alternative in the 1930-40s using methods derived from statistics and a field that is now termed 'Operations Research': a scarce-resource optimized method for accurate crop forecasting. Today, the Government of India employs sophisticated remote sensing including a Satellite Survey System to improve crop forecasts, but the methods developed then are still relevant and valuable. The seminal work of Mahalanobis in developing an optimal sample-based survey is also interesting to read from a practitioner's perspective. The combination of ideas employed in the work done in the 1930s include data analysis, statistical modeling, pilot study, scarce resource allocation, and mathematical optimization, and ranks among the great achievements in the practice of Operations Research and Analytics.

A map of undivided Bengal, circa 1850 C. E. (source: http://jrahman.files.wordpress.com)

Motivation
Jute and cotton were two of the most important exports out of India after the manufacturing sectors was crippled by the British Raj - 24% of the total revenue between 1927-37 was from Jute. Estimating the total Jute produce in Bengal up until the 1940s was largely a product of guesswork and ad-hoc estimates provided by the administrative chain of the British Raj produced wildly varying numbers. Like other parts of India, cultivation in Bengal was decentralized and spread over nearly 100 Million small farms, which were on average less than half-an-acre in area, spread over more than 60, 000 sq. miles. Jute was grown in a subset of these farms. Furthermore, the cultivation lifecycle of Jute is very short - about two months from planting to harvesting. Consequently, even if the administration was willing to cough up the expenses for an enumeration survey, covering all these farms within 8-9 weeks would be extremely expensive, if not impossible. Add to the fact, that many plots (30%) that cultivated Jute also cultivated other crops in parallel. Thus, while in theory, we can expect a total enumeration to give us near-zero error, in practice, allotting multi-crop areas to Jute and other Human-induced errors would introduce noise. In fact, the report states that the biggest negative associated with an enumerative survey was not the prohibitive cost but its unreliability, and this motivated Mahalanobis to develop and implement an alternative approach that accomplished the task at a fraction of the cost and time, and at a higher level of accuracy using random sampling.

Random Sampling
The nearly 100M jute farms were spread over Bengal in a non-homogeneous manner. Some areas were densely cultivated, some sparsely. The approach was to partition the total area into zones, i = 1, ..., n (area A_i) whose area was internally homogeneous (kinda like the way finite element analysis is used in structural engineering). Within each zone, a number of areas or grids were selected and sampled at random. If a sufficient number of such grids were sampled, the average proportion of area under Jute within a zone (J_i) can be obtained, which allows us to predict the total area under Jute  = sum(i) A_i * J_i.

Decision variables
1. The partition of the total area into approximately homogeneous zones
2. The number of random samples within a zone
3. The area of a sample

For simplicity, we assume that the first decision of partitioning the area within Bengal is an external input and thus our focus is on optimizing the remaining two decisions.

Constraints
1. The cost of the whole operation depends on the second and third set of decision variables.  For a given budget, if the area of an individual sample is large, then the number of samples has to be reduced, and thus the samples would be more spread out and further away from each other.

2. The achievable precision (variance) varies similarly. If the sample area is large, the per-sample variance is smaller, but cost considerations limited the number of such large-samples, and this can hurt the overall variance accumulated across the zone. On the other hand, a smaller area in tandem with a larger number of such small-samples affects precision in an opposite manner.

Nonlinear Optimization Problem
Given either cost or precision as a hard constraint, select the sampling area and the number of random samples to maximize precision, or minimize cost.

Mahalanobis' approach attempts to model the change in variance and cost as continuous functions of the two decision variable sets. Once these functions are at hand, a local optimum is obtained using a derivative based Lagrange-multiplier method. Mahalanobis used this approach to tabulate the achievable precision for a range of cost levels.

An exploratory, small-scale (pilot) survey was initially conducted at a small expense as a proof-of-concept and proof-of-technology validation of the methodology prior to embarking on a full-scale project. This type of an approach is now widely adopted in many business analytics projects.

The effect of the decision variables on variance can be calculated relying on theoretical methods. However, human-induced errors were also common, and Mahalanobis used the idea of interpenetrating half-sample pairs, where two groups independently arrived at Jute area estimates for a given location.  There are many important details here that are left out for brevity. The cost calculation is detailed and empirical and depends on the nature of the survey, and among things, include:
a. cost of staying and surveying at a given site - this depends on the size of the sampled area and time spent
b. cost of traveling from sample to sample - this depends on the distances between the chosen samples and the sequence of visiting.

Again, we have left out a humongous amount of cost calculations that were done. Reading the reports that came out of this work, one is amazed by the time and effort devoted to meticulously tabulating the various costs that go beyond 'ball-park' estimates, to produce an accurate cost function. For example cost calculation (b) depends on the solution to the corresponding traveling salesman problem.

The TSP
One of the many reports that came out of this this project notes:

(source: Sankhya journal, 1940)

This cost calculation is reviewed by Applegate, Bixby, et al. in their book on TSP and in Bill Cook's 'In Pursuit of the Traveling Salesman'.  A literature review of this TSP in these books mention that researchers later showed that the expected length of the optimal tour was approximately between (0.707, 1.27) times the square root of the number of samples visited in a unit square, so Mahalanobis' 1930s estimate was a remarkably good choice.

Results and Business Impact
The cost- and precision-controlled random sampling approach proved to be revolutionary. It achieved greater precision at a fraction of the cost.  Specifically, the margin of error was +/- 2%, and the cost was 1/15 of an enumeration census that was performed the same year and found to be less accurate compared to the random sampling approach. Thus the benefit and return-on-investment of this analytical approach was successfully demonstrated in practice, which received widespread recognition and was later embraced by the Government of independent India for many nationwide surveys.

Prelude to Part-3: The Bengal Holocaust
Within a couple years of the successful demonstration and publication of this work, Mahalanobis' Bengal lost between 2-6 million people due to starvation and disease between 1942-1945, triggered in part possibly by a failure of rice crop. The British Raj, locked in an grim Atlantic battle during WW2, may have suppressed reports and figures. It appears that most of the world, and even a vast majority of Indians, to this day, remain unaware of the reality behind this event.  How to obtain a reasonable estimate of casualties due to this disaster? Who was responsible and how? A recent book has brought this controversy into the open, and it appears that Mahalanobis (and his statistical sampling method) may have played a critical part in solving this puzzle.

To be continued.

Jugaad Innovation: Stuck in a Local Optimum

Updated July 5, 2013:
CNN link on Jugaad Innovation.

The book cover blurb sounded exciting: Do more with less. An alternative to risky expenditure-driven, resource-hungry growth using a "bottom up approach to frugal and flexible innovation". Endorsement from a salesforce.com CEO. No doubt, Jugaad is a useful concept - it's a Hindi word that implies a improvised and clever work-around, but there is such a thing as stretching an idea too far. Jugaad arises from the Indian way of doing "more with less" - possibly a public response to artificial scarcity induced by genocidal British colonization since the late 18th century when a resource-rich India's share of world GDP plummeted from 25% to a negligible quantity in a short time to turn it into from a knowledge and manufacturing economy into an agrarian, impoverished nation. A scarcity-driven economy that has been nurtured by Nehruvian socialist politics over the last 60+ years. However, the word itself ties in nicely with ideas in Operations Research that deal with the optimal allocation and utilization of scarce resources, and hence this post.

The book has some heart-warming and splendid examples of Indian innovation, which are nice to read. The remarkably decentralized and entrepreneurial nature of the workforce, and their seeming comfort in operating effectively within what appears as "chaos" to the external observer is one of the salient features of the native Indian economy. A discussion of the Indian practice of the "missed call" (the closest that mankind has come to 'half a bit' of information) adds humor. The authors could have lent depth to the book by exploring the deep-rooted cultural origins of the decentralization, and entrepreneurial spirit that drives the Indian way of doing things. They do make some useful comparisons between the profligacy and rigidity of some Western CEOs with the adaptive nature of the "jugaad entrepreneur". However, celebrating every success from Bharti airtel to PepsiCo's direct seeding to Mitticool (now that is really cool) as a win for "Jugaad" dilutes the message. When the book hyped the crappy 'Aakash' tablet, the populist, Indian government's tax-payer subsidized tablet for the unwashed masses (a cheaper and cynically sub-optimal alternative to constructing and maintaining primary schools), it was time to cry halt. Celebrating artificial scarce resource allocation, however efficiently done, feels like a cop-out.

Operations Research practice is not just about optimizing within constraints and declaring victory. Value can be unlocked by using OR to expose the more expensive bottlenecks in the system. Constraints that turn out to be artifacts must be eliminated whenever possible. I'm all for (also) 'thinking small', but not at the expense of losing the context.

The "Chaos" of India

Thanks to the person who pointed me to an interesting article in some online magazine called "livemint" : "Why Rahul Gandhi has understood the chaos of India"

As the 2014 General Election in India approaches, and pro-establishment journos crank out essays defending their favorite candidates, the author of the above article attempts something truly brave: resurrect a universally panned speech by a leading politician of the incumbent ruling political party. In particular, this paragraph caught my attention:
"... There are two things that stand out in the speech. The first is Rahul’s recognition of the chaos of India, for which he used the image of the beehive. I think this is wonderful realism, and excellent imagery."

As a keen India observer and commentator pointed out on twitter: "....beehive is nt chaotic"

Is India really chaotic?
To many a visitor from the west, it has appeared so. A vast section of India's English media who have worshiped at the altar of Western Universalism and return to lecture India's unwashed masses, may readily concur.

India, for a long, long time, has functioned as a highly decentralized system - in virtually every aspect of life. It's workforce contains among the highest percentage of entrepreneurs in the world, and traditionally possesses a remarkably decentralized economic model. It's indigenous religions are decentralized and base on empirical self-realization. There is no single holy book or authority (there is an entire reference library, with scope for new books to be written). However, this lack of a central authority does not imply chaos or that India was never a nation. It is more likely that what is perceived as chaos is in fact a cognitive overload on the part of the casual observer - as noted in Rajiv Malhotra's book "Being Different". The best example for this apparent chaos that actually produces incredibly effective results is the Kumbh Mela (the largest human gathering on earth, which has been recurring for thousands of years now) that recently concluded in India. Several observers from Harvard and other big-brand western institutions showed up this time to analyze this amazing event. Here's what Rajiv Malhotra says about the Mela:

"....India's Kumbha Mela amply demonstrates that diversity can be self-organized and not anarchic, even on a very large scale. Held every twelve years, this is the world's largest gathering of people, attracting tens of millions of individuals from all corners of India, from all strata of society, and from all kinds of traditions, ethnicities and languages. Yet there is no central organizing body, no 'event manager' to send out invitations or draw up a schedule, nobody in charge to promote it, no centralized registration system to get admitted. Nobody has official authority or ownership of the event, which is spontaneous and 'belongs' to the public domain. Since time immemorial, numerous groups have put up their own mini-townships and millions go as individuals just to participate in the festivities."

How and Why? Rajiv argues in his introduction to the chapter "Order and Chaos":
"Indians tend to be more relaxed in unpredictable situations than westerners. Indians indeed find it natural to engage in non-linear thinking, juxtaposing opposites and tackling complexities that cannot be reduced to simple concepts or terms. They may be said even to thrive on ambiguity, doubt, uncertainty, multitasking, and in the absence of centralized authority and normative codes. .... In the vast canon of classical writings in Sanskrit, we see many context-sensitive and flexible ways of dealing with chaos and difference. The search here is always for balance and equilibrium with the 'rights' of chaos acknowledged. On the other hand, in the creation stories in Genesis and in the Greek classics, there is a constant zero-sum battle between the two poles in which order must triumph."

Interestingly, the socio-biological systems of beehives and ant colonies, which may appear chaotic to the casual observer has an underlying order. In his brilliant book 'Traffic: why we drive the way we do', Tom Vanderbilt quotes studies that note how vast numbers of ants move without colliding (without a centralized traffic cop ant), which is certainly worthy of emulation by human car drivers. The author notes "looking at the swarm as a whole, one might not see what is driving the movement". The insects have their own Kumbha Mela going, perhaps.

Rahul Gandhi's 'beehive' metaphor may not be unreasonable, but it appears to have been chosen based on an entirely shallow reasoning due to his inability to decipher India's apparent "complexity", which he contrasts with China's 'simplicity'. India can be plausibly compared in certain contexts to a beehive or an ant colony not because of 'chaotic frenzied activity', but because of an certain well-functioning harmonic order underlying the apparent chaos.

The lesson here for those of us wishing to properly study India is to look beyond the obvious, look deeper, pay attention to context, and there are amazing discoveries waiting to be made. When Rahul Gandhi, the 4th edition of the Nehru dynasty and potential contender for the post of prime-ministership in 2014 compares India to a beehive for all the wrong reasons; when sections of the Indian English media jump in to spin this into some 'deep thought', it only betrays a fundamental lack of understanding of the essence of India: pluralism, diversity, and cooperative decentralization, blissfully unaware that at its deepest philosophical levels, dharmic India is bound by what Rajiv Malhotra terms an 'Integral Unity'.