Introduction
The Vedas are timeless. Indic scholars do not ascribe a particular time to its origin because it has always "been there" - the Vedas are considered to be a constantly evolving treasure of knowledge, wisdom, science, math, and 'best practices' since time immemorial. Modern researchers' requirement to assign a measurable date to an event and slot it into their finite history model results in a 'start date' for some random snapshot of the Vedas that is anywhere between 2-5K years prior to the common era.
The Vedas are in Sanskrit
The Vedas contains a wealth of useful guidelines and ethical principles to ensure a productive lifestyle that is in harmony with the surroundings. Indeed, the word as well as the religion 'Hinduism' that is in use today is an inaccurate artifact created by colonizing Europeans (update, 2013: Term 'Hindu' is much more ancient). The correct term is Sanathana Dharma, a phrase that very roughly translates as "the right way for good guys (and gals) to lead their life". There is no equivalent for Dharma in English, and there is no word for organized religion in Sanskrit to the best of my knowledge (update, 2013: highly limited knowledge).
Fidelity of the Vedas
From an algorithmic and modeling perspective, an amazing fact is that the Vedas have been orally transmitted and remembered using Vedic Chants across millennia, and what we hear today is probably what was first chanted a very, very, long time ago. In a world that is so accustomed to written proof, this approach and it's implications can be a bit tough to grasp initially.
Vedic Chants and its Impact
Per Wikipedia "The UNESCO proclaimed the tradition of Vedic chant a Masterpiece of the Oral and Intangible Heritage of Humanity on November 7, 2003." The Vedas are to be listened to, and not primarily intended to be a reading text. The textual meaning of the Sanskrit words in the Vedas are but a part of its benefit, but the way it is combined together and chanted in a specific manner is equally if not more important. It is this entire structure that has been remarkably preserved across eras and onslaughts including Greek, Mongolian, Central Asian, and European invaders. There were no "heretic" writings for them to easily destroy and thanks to these error codes, India has been able to maintain an unbroken chain of culture and thus retain the most ancient of its connections to its roots.
Here's a digital sample of a Vedic chant. Some may notice a passing resemblance to mystic pagan, Gaelic, or Gregorian Latin chants.
Combinatorial Aspects
I'm betting those wise old Rishis (Indian sages) would have dug OR for its practicality, so they sure would have loved to chant:
"operations research is the science and engineering of better". We'll drop the 'engineering' part for the sake of brevity.
The meaning and pronunciation of each Sanskrit word stands unambiguously by itself and is written just like it sounds (unlike, say English). Sanskrit is characterized by precision. It requires no punctuation marks. Orally, this is accomplished by chanting each word in the passage distinctly, one after the other (a "/" indicates a brief pause).
Code 1: operations/research/is/the/science/of/better
However, even a few hundred cycles is likely to induce errors (In English: where's the comma, pause, syllable emphasis, etc), so they must have decided to add a second layer that chains a word to the next one:
Code 2: operations research/research is/is the/the science/science of/of better
Each word is now spoken twice in order, thereby introducing some redundancy, while also explaining how pairs of words are to be spoken in combination. Remember, the pitch and tone for a particular word may depend on one or more words that precede or follow any given word. Sure enough, another Rishi decided to take this a step further by chaining three words to introduce further redundancy and handle cases where the ebbs and flows in the chants persist a bit longer.
Permutations
Yet another Rishi put words together in an apparently strange way to create:
Braid Code: operations research research operations operations research/research is is research research is /is the the is is the/the science science the the science/science of of science science of/of better better of of better
The number of times a particular word is repeated is 6. The phrases seem to be chanted in a forward/reverse/ forward format. However, this is a moot point in Sanskrit because switching the order of the words in a sentence does not change the meaning! In other words, each section in Code 2 is chanted thrice in different orders, but they reinforce the exact same meaning each time. Indeed, this also ensures that Rishis who loved to swap words around could now do so (even inadvertently) without fear of messing up the cadence. Pretty cool. Those who've followed this tab for a while would have noticed that the words in this tab are often hopelessly out of order. It's not easy to downgrade to English and write acceptable OR journal papers :)
There are a few more codes, but the pièce de résistance is this particular "bell code" which also happens to be the most pleasing to the ear:
Bell code: operations research research operations operations research is is research operations operations research is/
research is is research research is the the is research research is the/
is the the is is the science science the is is the science/
the science science the the science of of science the the science of/... and so on
The number of times a particular word is spoken in this code is 10, i.e. a factor of 10 redundancy. A sample of the Bell-code chant (the example chosen is the "peace mantra") can be found toward the bottom of this site. Search for the phrase "Ghana Paatha". The first 2.5 minutes is the same mantra that was chanted in the first example, but is now done by a different individual. Comparing the variations in the two performances gives us an idea of the challenge faced by those Rishis eons ago.
Complexity
If the order of words doesn't matter, there's going to be an exponential number of ways of speaking a sentence correctly without altering the meaning (although some choices are more preferable and popular). But this also serves as a natural fault-tolerant mechanism. Each ordering is an alternative feasible solution that results in at least the correct decoding of the meaning of the passage; yet another reason why the treasures of the Vedas would have been lost to the world without the magical medium of Sanskrit. The Rishis appear to have managed these permutations by carefully designing the chants in sync with phrases to make it easier to orally encode and decode a massive amount of information across generations without a single written note. Indeed, they may have thought of this as an infinite horizon problem given the cyclical time concept of Indian philosophy. They appear to have converged on a limited number of simple encoding rules that appear to be quite effective as well as practically robust. It is estimated that it would take a month or two of continuous chanting to cover the entire bell-code version of the Vedas!
Present Day
We know that there's an inherent connection between linguistics and computer science. If Jeopardy were to be played in Sanskrit, would IBM's Watson have been more accurate and less dependent on guesswork?
These ideas also seem to be used in reinforcing learning among kids. Songs and rhymes with repetitive themes appear to be relatively easier to remember over long time periods. The holiday song in the previous post is a good example. If we ever wondered what drives otherwise fun-loving Indian kids to memorize entire lexicons to collect endless Spelling Bee trophies, and why there is so much emphasis on rote learning in India, now we know its because of Shruti and her dual, Smriti.
It would be nice if our business customers were to present all their business rules, priorities, and goals in bell-code Sanskrit verse on a CD rather than Word text and Power-point charts to avoid the endless headaches arising from the inevitable losses in translation!
Credits and Disclaimer
The sources for this post include numerous websites and individuals. It's tough to list each one, so a big thanks to all who are working hard to preserve this part of the world heritage in its original form and context. Being neither a Sanskrit or an English (or pretty much any language) expert, errors, inadequacies, and misinterpretations, if any, in this post are of course, entirely mine.
update: some typos fixed.
2nd update: January 2013
Saturday, July 30, 2011
Wednesday, July 27, 2011
A tough MIP for the winter
The 7.33 Days of C.Xmas
On the first day of C.Xmas,
mipsolver sent to me
A node-zero bound and a tiny branch-and-bound tree.
On the second day of C.Xmas,
mipsolver sent to me
two depth-first searches that dove ...,
and came back empty to a growing branch-and-bound tree.
On the third day of C.Xmas,
mipsolver sent to me
3 French-invented cuts,
2 'doves',
and a tighter bound, trimming the branch and bound tree.
On the fourth day of C.Xmas,
mipsolver sent to me
4 call-back birds,
3 French cuts,
2 doves,
and a still-infeasible branch-and-bound tree.
On the fifth day of C.Xmas,
mipsolver sent to me
5 hi-flying rounding heuristics,
4 call-back birds,
3 French cuts,
2 doves,
all tossed into a gigantic branch-and-bound tree.
On the sixth day of C.Xmas,
mipsolver sent to me
6 more goose eggs,
5 rounding heuristics,
4 call-back birds,
3 French cuts,
2 doves,
and still no incumbent in that branch-and-bound tree.
On the seventh day of C.Xmas,
mipsolver sent to me
7 types of warnings,
6 more goose eggs,
5 rounding heuristics,
4 call-back birds,
3 French cuts,
2 doves,
all mocking me from the overflowing branch-and-bound tree.
8am after my last day of C.Xmas,
manager said to me,
(this song is exponentially grating)
7 days of computing,
6 paying customers a-waiting,
5 days of test-match cricket watching,
4 P-series server-snatching,
3 more tough MIPs in your job batching,
2 minutes for u to clear your desk of every bird-dropping,
and climb that branch-and-bound tree.
Disclaimer: Purely in jest, to state the obvious. Claims of any statistically significant correlation to any real MIP solvers, real managers, real holidays, and virtually anything else that is real is just as unimaginative as this song.
On the first day of C.Xmas,
mipsolver sent to me
A node-zero bound and a tiny branch-and-bound tree.
On the second day of C.Xmas,
mipsolver sent to me
two depth-first searches that dove ...,
and came back empty to a growing branch-and-bound tree.
On the third day of C.Xmas,
mipsolver sent to me
3 French-invented cuts,
2 'doves',
and a tighter bound, trimming the branch and bound tree.
On the fourth day of C.Xmas,
mipsolver sent to me
4 call-back birds,
3 French cuts,
2 doves,
and a still-infeasible branch-and-bound tree.
On the fifth day of C.Xmas,
mipsolver sent to me
5 hi-flying rounding heuristics,
4 call-back birds,
3 French cuts,
2 doves,
all tossed into a gigantic branch-and-bound tree.
On the sixth day of C.Xmas,
mipsolver sent to me
6 more goose eggs,
5 rounding heuristics,
4 call-back birds,
3 French cuts,
2 doves,
and still no incumbent in that branch-and-bound tree.
On the seventh day of C.Xmas,
mipsolver sent to me
7 types of warnings,
6 more goose eggs,
5 rounding heuristics,
4 call-back birds,
3 French cuts,
2 doves,
all mocking me from the overflowing branch-and-bound tree.
8am after my last day of C.Xmas,
manager said to me,
(this song is exponentially grating)
7 days of computing,
6 paying customers a-waiting,
5 days of test-match cricket watching,
4 P-series server-snatching,
3 more tough MIPs in your job batching,
2 minutes for u to clear your desk of every bird-dropping,
and climb that branch-and-bound tree.
Disclaimer: Purely in jest, to state the obvious. Claims of any statistically significant correlation to any real MIP solvers, real managers, real holidays, and virtually anything else that is real is just as unimaginative as this song.
Saturday, July 16, 2011
Where are the new OR innovations coming from?
The telephone appears to be increasingly rejected in favor of returning to Morse-code like telegraphic tweets and talk-free 'radio' text messages. Is OR going through a similar cycle? In a prior tab, empirical evidence was provided to suggest that the so-called age of analytics did not really start at Y2K; like the telegraph and the radio, its always been there and only gone digital now.
Our airline customers who pioneered the construction of scalable OR-embedded infrastructure to manage complex revenue and cost issues have seemingly run out of similar low-hanging fruit. Attending a recent INFORMS conference felt like reading a classifieds ad: "thoroughly impressive solutions seek unanswered practical questions to justify time and expense". This does not mean that we are not innovating - far from it; it's just not from within the OR community, where we continue to indulge in our dual laundering cycle of model building and tool polishing. Occasionally we hunt for non-existent (or worse, gullible and real) customers to pay us good money to take the resultant code-scrap off our hands. After all, OR is merely the science of 'better'. It is our customer who did all the hard work of taking it all the way from 'nothing' to 'good'.
The world of retail is one example of a margin-starved, data-rich industry that is driven by a realistic necessity to innovate. Our retail customer has been gratefully but carefully adapting practically useful resource optimization techniques from the airline world. By carefully refining these methods in the demanding retail context, they are generating a bunch of new analytical 'best practices'. Combined with some good old sales techniques, these approaches appear to be on the verge of reigniting similar innovations in other industrial sectors.
On a related note, OR resembles the fast-drawing gunman of the wild west legend whose niche skills are desperately sought after and bid for by a town threatened by outlaws, but one who also becomes a liability for the town once peace has been established. We should avoid outliving our welcome and be more proactive in seeking new 'towns'.
Our airline customers who pioneered the construction of scalable OR-embedded infrastructure to manage complex revenue and cost issues have seemingly run out of similar low-hanging fruit. Attending a recent INFORMS conference felt like reading a classifieds ad: "thoroughly impressive solutions seek unanswered practical questions to justify time and expense". This does not mean that we are not innovating - far from it; it's just not from within the OR community, where we continue to indulge in our dual laundering cycle of model building and tool polishing. Occasionally we hunt for non-existent (or worse, gullible and real) customers to pay us good money to take the resultant code-scrap off our hands. After all, OR is merely the science of 'better'. It is our customer who did all the hard work of taking it all the way from 'nothing' to 'good'.
The world of retail is one example of a margin-starved, data-rich industry that is driven by a realistic necessity to innovate. Our retail customer has been gratefully but carefully adapting practically useful resource optimization techniques from the airline world. By carefully refining these methods in the demanding retail context, they are generating a bunch of new analytical 'best practices'. Combined with some good old sales techniques, these approaches appear to be on the verge of reigniting similar innovations in other industrial sectors.
On a related note, OR resembles the fast-drawing gunman of the wild west legend whose niche skills are desperately sought after and bid for by a town threatened by outlaws, but one who also becomes a liability for the town once peace has been established. We should avoid outliving our welcome and be more proactive in seeking new 'towns'.
Tuesday, July 5, 2011
Goodbye, Maine.
There's much forest in Maine, more than any other state in the US. If you randomly parachute into ME, the odds of landing in mother earth's lap is about 1:8, which improves marginally in the winter. People love the outdoor life. This is the land of Acadia, a place for camps and hikes, and is so full of summer life. Yet ME is also the resting place for fallen heroes who have always arrived quietly from distant lands - Korea, Vietnam and Iraq. On rare and lucky occasions at Bangor Airport, you get to see the tired but elated troops walk safely home after a tough tour of duty in another one of those quagmires. And there are other days, when the same faces wait to cross the pond on the orders issued by generals and politicians. Bangor, is an important logistical point for such overseas operations. You can't get a decent runway more north-east in the mainland US.
Nothing sensational happens in Maine, and yet the first events at dawn on 9/11 unfolded at the Portland airport, and at dusk, a small airport-town in Newfoundland, just north of Maine improvised splendidly as they hosted several international flights that were ordered to land as soon as they touched North America. We border just one US state, and I cheered in vain for the 2010 Canadian Edelman team who turned out to be from the 'hood (New Brunswick).
Maine is a strong blue state which elected not one but two red senators, grumbled about it, and then decided to elect a red governor and grumbled even more. ME actually voted against FDR in 1936. Tough times makes for tough decisions. ME always scrapes the bottom in terms of the other green - business-friendliness, and most Mainers struggle through these tough times, as people seem to put up more and more of their possessions 'for sale'. Yet it's hard to get angry at people here after a bad day at the office - even the cable TV operators here are so damn friendly and helpful. The post-woman delivers mail flawlessly in her USPS car on her route, and come winter, very politely asks me to do something about her postbox, even though she knows the answer. Her nemesis is the ubiquitous snow-plow truck whose demon drivers perfectly take out every postbox on their TSP routes for fun. Its a game where Stephen King's creatures rule the streets of Bangor at night, and blissful peace returns at daybreak. There is this duality about Maine that will be missed.
Nothing sensational happens in Maine, and yet the first events at dawn on 9/11 unfolded at the Portland airport, and at dusk, a small airport-town in Newfoundland, just north of Maine improvised splendidly as they hosted several international flights that were ordered to land as soon as they touched North America. We border just one US state, and I cheered in vain for the 2010 Canadian Edelman team who turned out to be from the 'hood (New Brunswick).
Maine is a strong blue state which elected not one but two red senators, grumbled about it, and then decided to elect a red governor and grumbled even more. ME actually voted against FDR in 1936. Tough times makes for tough decisions. ME always scrapes the bottom in terms of the other green - business-friendliness, and most Mainers struggle through these tough times, as people seem to put up more and more of their possessions 'for sale'. Yet it's hard to get angry at people here after a bad day at the office - even the cable TV operators here are so damn friendly and helpful. The post-woman delivers mail flawlessly in her USPS car on her route, and come winter, very politely asks me to do something about her postbox, even though she knows the answer. Her nemesis is the ubiquitous snow-plow truck whose demon drivers perfectly take out every postbox on their TSP routes for fun. Its a game where Stephen King's creatures rule the streets of Bangor at night, and blissful peace returns at daybreak. There is this duality about Maine that will be missed.
Thursday, June 30, 2011
Worst practices: gender and literacy shaping
Demand shaping is an OR-rich technique of using control levers (e.g. pricing in a supply chain) to induce a demand shift away from scarce items, or toward products in excess supply. If you know the shadow price of constrained inventory (e.g. an airline seat in a particular market), you can use that information to make useful quantitative control decisions that maximize utilization and drive profits.
We look at a couple of related examples of demand shaping done for all the wrong reasons. While I present a couple of egregious examples from India, this is most certainly not a unique Indic problem. Hopefully such examples with motivate a deeper OR study into this worldwide and systemic human rights violation pattern in the human biological chain.
Example 1: gender shaping
Urban India is in the midst of a pop-culture-driven left-liberal movement for men and women that addresses a lot of new-age social issues (e.g. right to fashion, right to a fairer skin tone), while almost completely neglecting the more fundamental 'right to life' issues. This isn't a pro-life v pro-choice discussion. While there's valid arguments on both sides in that debate, this is about the nihilistic practice of gender shaping practiced in several parts of the world, including, but not limited to the English-speaking literati of India, where the sex of the unborn can seal its fate. Since a positive identification is possible only after many months into pregnancy, the term used in India for such gender shaping is close to the truth: female infanticide, i.e. murdering baby girls (for monetary reasons). The gender shapers have succeeded in reducing 'demand for excess inventory' by more than 160 million in just the last few years. In fact, as researchers have argued, since the viability of the female fetus is probabilistically higher, that 'lost-sales number' is merely a lower bound. India has banned sex-determination tests to reverse the tide, but the practice is still rampant.
One of Russel Ackoff's OR consulting projects in India was focused on the primal problem of over-population which motivates the second example. A section in his classic book examines why demand-shaping there (in the benign form of family planning) was relatively unsuccessful. Filtering through all the noise, it now appears that gender-shaping is merely it's evil dual twin.
Example 2: literacy shaping
President Obama has more than once pleaded for US students to shift toward increased enrollment in science and math "STEM" courses to be able to effectively compete with their Indian and Chinese counterparts (A quick look at the recent results of the national spelling-bee and national geographic math/geography contests will tell u why). This was widely published, but where? A Google search shows that this was carried prominently by many newspapers in Asia (a classic example of a complementary "cross elasticity" effect in pricing analytics). Not that students in those countries need any more motivation. The utter bankruptcy in governance exhibited by the dynastic ruling political party (which ironically calls itself "the congress party") that has ruled India for more than 75% of the time post-independence, has driven the country to near ruin. The number of good-quality higher and primary education institutions has been practically stagnant for decades. Rather than do the actual work of building more schools and colleges with all that money pouring in from the liberalized economy, the education ministry has adopted an easier, but dastardly three-pronged literacy shaping approach:
a. decrease the rate of construction of primary and high schools to control the resultant demand for higher education
b. Increase the score threshold for acceptance into the existing higher education institutions.
c. Misuse the well-intentioned affirmative action 'quotas' to drive demand toward politically influential families.
(a) effectively ensures that the poorest and marginalized communities of India cannot really avail of the illusory affirmative action-fenced inventory segments. The literacy shapers further shift demand toward those who wield power via (b) and (c). For the last 3-4 years, we have had farcical situations where, in some parts of India, the score threshold (i.e. the shadow price for a scarce university seat) was adjusted to 100%! In US terms, it means that only those who have a picture perfect SAT score are eligible to apply to a reasonably good college.
That brilliant lady professor from India who shined at your local OR conference is one of the lucky ones who managed to evade the dual traps of gender and literacy shaping.
We look at a couple of related examples of demand shaping done for all the wrong reasons. While I present a couple of egregious examples from India, this is most certainly not a unique Indic problem. Hopefully such examples with motivate a deeper OR study into this worldwide and systemic human rights violation pattern in the human biological chain.
Example 1: gender shaping
Urban India is in the midst of a pop-culture-driven left-liberal movement for men and women that addresses a lot of new-age social issues (e.g. right to fashion, right to a fairer skin tone), while almost completely neglecting the more fundamental 'right to life' issues. This isn't a pro-life v pro-choice discussion. While there's valid arguments on both sides in that debate, this is about the nihilistic practice of gender shaping practiced in several parts of the world, including, but not limited to the English-speaking literati of India, where the sex of the unborn can seal its fate. Since a positive identification is possible only after many months into pregnancy, the term used in India for such gender shaping is close to the truth: female infanticide, i.e. murdering baby girls (for monetary reasons). The gender shapers have succeeded in reducing 'demand for excess inventory' by more than 160 million in just the last few years. In fact, as researchers have argued, since the viability of the female fetus is probabilistically higher, that 'lost-sales number' is merely a lower bound. India has banned sex-determination tests to reverse the tide, but the practice is still rampant.
One of Russel Ackoff's OR consulting projects in India was focused on the primal problem of over-population which motivates the second example. A section in his classic book examines why demand-shaping there (in the benign form of family planning) was relatively unsuccessful. Filtering through all the noise, it now appears that gender-shaping is merely it's evil dual twin.
Example 2: literacy shaping
President Obama has more than once pleaded for US students to shift toward increased enrollment in science and math "STEM" courses to be able to effectively compete with their Indian and Chinese counterparts (A quick look at the recent results of the national spelling-bee and national geographic math/geography contests will tell u why). This was widely published, but where? A Google search shows that this was carried prominently by many newspapers in Asia (a classic example of a complementary "cross elasticity" effect in pricing analytics). Not that students in those countries need any more motivation. The utter bankruptcy in governance exhibited by the dynastic ruling political party (which ironically calls itself "the congress party") that has ruled India for more than 75% of the time post-independence, has driven the country to near ruin. The number of good-quality higher and primary education institutions has been practically stagnant for decades. Rather than do the actual work of building more schools and colleges with all that money pouring in from the liberalized economy, the education ministry has adopted an easier, but dastardly three-pronged literacy shaping approach:
a. decrease the rate of construction of primary and high schools to control the resultant demand for higher education
b. Increase the score threshold for acceptance into the existing higher education institutions.
c. Misuse the well-intentioned affirmative action 'quotas' to drive demand toward politically influential families.
(a) effectively ensures that the poorest and marginalized communities of India cannot really avail of the illusory affirmative action-fenced inventory segments. The literacy shapers further shift demand toward those who wield power via (b) and (c). For the last 3-4 years, we have had farcical situations where, in some parts of India, the score threshold (i.e. the shadow price for a scarce university seat) was adjusted to 100%! In US terms, it means that only those who have a picture perfect SAT score are eligible to apply to a reasonably good college.
That brilliant lady professor from India who shined at your local OR conference is one of the lucky ones who managed to evade the dual traps of gender and literacy shaping.
Friday, June 24, 2011
Duality in music
Regular dual noise service is interrupted to provide some dual music for a change. Check out this piece by the music band 'Tirtha', aptly titled 'Duality', I kid you not:
Being a bit biased toward instruments with strings attached, fast-forward to about 5:50 and listen. That is India's most versatile guitarist, Prasanna, playing. To listeners from India (South India, specifically), Carnatic music's distinctive 'microtone' notes ring through pretty clearly, whereas Jazz aficionados will enjoy the 'swing' i suppose. He seems to be playing both of these, but then, it is not what radio-stations label either as 'fusion' (not quite imaginative), or 'new age' (out of ignorance?). Both Carnatic and Jazz music are original, classical art forms with a rich history and a dedicated fan-base. The former is native to S.India and the latter, apparently is America's only true native art-form. Equally, he is playing neither - purists are more likely come to this conclusion after detecting some 'noise' intertwined with the specific form of music they swear by.
From an OR perspective, i would like to think that the two seemingly unrelated musical forms in this piece achieve the same objective, follow certain well-defined rules and satisfy certain well-understood and pleasing properties, and depending on how you 'look' at it, you can call it Carnatic or Jazz. Duality - there is more than one way of getting things done. Whether you are a South-Indian who is just beginning to discover Jazz, or a Jazz-fan who is touched by the Kalyani Ragam for the first time, it's a great two-for-one deal.
And there is one more connection that might explain the name of the musical piece. One ancestor of Prasanna was the mathematical genius Srinivasa Ramanujan, the 'man who knew infinity'. Signing off with an old solo piece by Prasanna, composed when he was an undergrad at the Indian Institute of Technology, Madras:
Being a bit biased toward instruments with strings attached, fast-forward to about 5:50 and listen. That is India's most versatile guitarist, Prasanna, playing. To listeners from India (South India, specifically), Carnatic music's distinctive 'microtone' notes ring through pretty clearly, whereas Jazz aficionados will enjoy the 'swing' i suppose. He seems to be playing both of these, but then, it is not what radio-stations label either as 'fusion' (not quite imaginative), or 'new age' (out of ignorance?). Both Carnatic and Jazz music are original, classical art forms with a rich history and a dedicated fan-base. The former is native to S.India and the latter, apparently is America's only true native art-form. Equally, he is playing neither - purists are more likely come to this conclusion after detecting some 'noise' intertwined with the specific form of music they swear by.
From an OR perspective, i would like to think that the two seemingly unrelated musical forms in this piece achieve the same objective, follow certain well-defined rules and satisfy certain well-understood and pleasing properties, and depending on how you 'look' at it, you can call it Carnatic or Jazz. Duality - there is more than one way of getting things done. Whether you are a South-Indian who is just beginning to discover Jazz, or a Jazz-fan who is touched by the Kalyani Ragam for the first time, it's a great two-for-one deal.
And there is one more connection that might explain the name of the musical piece. One ancestor of Prasanna was the mathematical genius Srinivasa Ramanujan, the 'man who knew infinity'. Signing off with an old solo piece by Prasanna, composed when he was an undergrad at the Indian Institute of Technology, Madras:
Wednesday, June 8, 2011
Open-source Java tools for OR/BA projects
A list of open-source Java-based resources for optimization and business analytics is presented in this post. There's surprisingly few of these going around. COIN-OR has relatively few or no pure Java tools, but if you have heard of any, please post.
Object-oriented, algorithmic, decision-driven analysis of a complex business system having many moving parts, is an effective approach, and Java, a solid delivery option. Add to this the inherent cross-platform interoperability and performance improvements in recent times, and Java becomes a pareto-optimal deployment choice in many project instances.
This is by no means an exhaustive list, and yes, it's missing all those heavy-duty MINLP and SOCP Cadillac solvers, but there's always some solid mileage in your old Chevy if you're a good driver (whatever that means)
Optimization:
LP solver: "Gooplex": Google's 'toy' implementation that is well-written and thus extensible (Apache commons) to a more sophisticated DIY simplex implementation. An old post on this topic here. Great for solving one or a million tiny LPs.
NLP: L-BFGS for large-scale unconstrained nonlinear optimization. This is a translation of the original FORTRAN version written by Jorge Nocedal. The license for this tool appears to be fairly relaxed (LGPL), but do your own checks. The bound-constrained L-BGFS-B is not 'free' for commercial use.
Trivia: Which is supposedly the fastest pure Java LP solver 'in the world'?
Predictive analytics
Colt: high-performance tools for linear algebra, matrix methods, OLS, etc.
Jama: A light-weight package for linear algebra
OpenForecast: Moving averages, multivariate regression, etc.
Wikipedia's meta list of numerical Java tools
Libsvm: SVM for classification, etc.
Object-oriented, algorithmic, decision-driven analysis of a complex business system having many moving parts, is an effective approach, and Java, a solid delivery option. Add to this the inherent cross-platform interoperability and performance improvements in recent times, and Java becomes a pareto-optimal deployment choice in many project instances.
This is by no means an exhaustive list, and yes, it's missing all those heavy-duty MINLP and SOCP Cadillac solvers, but there's always some solid mileage in your old Chevy if you're a good driver (whatever that means)
Optimization:
LP solver: "Gooplex": Google's 'toy' implementation that is well-written and thus extensible (Apache commons) to a more sophisticated DIY simplex implementation. An old post on this topic here. Great for solving one or a million tiny LPs.
NLP: L-BFGS for large-scale unconstrained nonlinear optimization. This is a translation of the original FORTRAN version written by Jorge Nocedal. The license for this tool appears to be fairly relaxed (LGPL), but do your own checks. The bound-constrained L-BGFS-B is not 'free' for commercial use.
Trivia: Which is supposedly the fastest pure Java LP solver 'in the world'?
Predictive analytics
Colt: high-performance tools for linear algebra, matrix methods, OLS, etc.
Jama: A light-weight package for linear algebra
OpenForecast: Moving averages, multivariate regression, etc.
Wikipedia's meta list of numerical Java tools
Libsvm: SVM for classification, etc.
Sunday, May 29, 2011
April 1963 discussion: OR v Analytics
While digging through old scientific articles trying to ascertain when O.R gained a solid foothold in India, I stumbled on this gem from 48 years ago. The pdf version of this article that was published by "the defense documentation center for scientific and technical information, Cameron Station, Alexandria, Virginia" can be found here. The article is interesting for a variety of reasons. Among other things, it critically appraises the work of Russel Ackoff and Mahalanobis (about whom this non-blog will comment on in a later post) on using OR models for solving nation-planning problems in developing countries.
Words from the original document are in italics and any emphasis below in 'bold' is mine.
Ackoff asserted that a large role was both feasible and desirable. He predicted that extremely high returns would result from addressing national planning problems with operations research techniques in these countries. In striking contrast to this viewpoint, the ORSA president (Dr. Charles Hitch) at that time, stressed the risks of over-selling what OR has to offer the underdeveloped countries at the level of national planning, and apparently wanted OR'ers to focus on tactical and commercial applications at the project and industry level since 'OR is the art of suboptimizing' On the other hand, the document also notes that since then, Hitch has been a pioneer in adapting OR to national defense problems in the United States as Assistant Secretary of Defense (Comptroller). Hmm.
Other dissenters similarly commented on the characteristics of problems to which operations research can be most successfully applied, e.g., abundant and reliable data, a well-structured model, and a clear, reduceable objective function. [Dorfman] concluded that the conditions that are most propitious for the use of operations research tend to occur in "routine and technical problems" at lower and middling levels. The document includes examples that apparently point out the hazards of applying operations research techniques too quickly and broadly.
Salient features of this document include:
1. An open and frank conversational style of writing
2. The presence of constructive and sharp dissent without disparaging the worth or the author of that prior contribution
3. A strong emphasis on the practical method, which really distinguishes OR from other disciplines
- all three of which is mostly missing in the 'sterile' articles that we see published in recent times. When we arrive at the last paragraph that precedes the main body of work in this document (which consists of copious amounts of what we would almost certainly classify as 'analytics' today, e.g., causal statistical regression models, etc), we read this:
First, I am not particularly concerned with whether it might be more appropriate to apply the labels "econometrics," or "systems analysis," rather than "operations research," to one or both of these examples. Methodological purists may find it preferable to fit the examples into one or the other of these categories, but for my purpose, what we are concerned with is the application of quantitative analytical techniques to decision problems in the underdeveloped countries. From this standpoint, econometrics applied to practical, policy problems Is operations research.
At least for me, this argument amicably settles the non-debate on the dual noises emanating from our tribe on 'OR v analytics', but of course, it is wishful thinking that this practical discussion will fix the larger contemporary issue that is centered on 'brand labeling' and 'image perception' than anything that practically and directly affects our customer base, which should have been our #1 priority. An inclusive dual identity for a person is fast becoming the norm in this non-homogenous, 'globalized' planet, and to paraphrase Shakespeare, OR by any another name would add the same value to our customers.
The author of this informative 1963 document is Dr. Charles Wolf, Jr. A brief bio of this distinguished gentleman, and another one can be found here. By sheer coincidence, the May Informs blog challenge appears to be about 'OR and Analytics'.
Words from the original document are in italics and any emphasis below in 'bold' is mine.
Ackoff asserted that a large role was both feasible and desirable. He predicted that extremely high returns would result from addressing national planning problems with operations research techniques in these countries. In striking contrast to this viewpoint, the ORSA president (Dr. Charles Hitch) at that time, stressed the risks of over-selling what OR has to offer the underdeveloped countries at the level of national planning, and apparently wanted OR'ers to focus on tactical and commercial applications at the project and industry level since 'OR is the art of suboptimizing' On the other hand, the document also notes that since then, Hitch has been a pioneer in adapting OR to national defense problems in the United States as Assistant Secretary of Defense (Comptroller). Hmm.
Other dissenters similarly commented on the characteristics of problems to which operations research can be most successfully applied, e.g., abundant and reliable data, a well-structured model, and a clear, reduceable objective function. [Dorfman] concluded that the conditions that are most propitious for the use of operations research tend to occur in "routine and technical problems" at lower and middling levels. The document includes examples that apparently point out the hazards of applying operations research techniques too quickly and broadly.
Salient features of this document include:
1. An open and frank conversational style of writing
2. The presence of constructive and sharp dissent without disparaging the worth or the author of that prior contribution
3. A strong emphasis on the practical method, which really distinguishes OR from other disciplines
- all three of which is mostly missing in the 'sterile' articles that we see published in recent times. When we arrive at the last paragraph that precedes the main body of work in this document (which consists of copious amounts of what we would almost certainly classify as 'analytics' today, e.g., causal statistical regression models, etc), we read this:
First, I am not particularly concerned with whether it might be more appropriate to apply the labels "econometrics," or "systems analysis," rather than "operations research," to one or both of these examples. Methodological purists may find it preferable to fit the examples into one or the other of these categories, but for my purpose, what we are concerned with is the application of quantitative analytical techniques to decision problems in the underdeveloped countries. From this standpoint, econometrics applied to practical, policy problems Is operations research.
At least for me, this argument amicably settles the non-debate on the dual noises emanating from our tribe on 'OR v analytics', but of course, it is wishful thinking that this practical discussion will fix the larger contemporary issue that is centered on 'brand labeling' and 'image perception' than anything that practically and directly affects our customer base, which should have been our #1 priority. An inclusive dual identity for a person is fast becoming the norm in this non-homogenous, 'globalized' planet, and to paraphrase Shakespeare, OR by any another name would add the same value to our customers.
The author of this informative 1963 document is Dr. Charles Wolf, Jr. A brief bio of this distinguished gentleman, and another one can be found here. By sheer coincidence, the May Informs blog challenge appears to be about 'OR and Analytics'.
Friday, May 27, 2011
Memorable battle scenes on film - spot the O.R connection
This post consists of 'tragic battle' scenes from old classics. The aim is to find a popular O.R theme (however flimsy) in the clip, as we honor those who fought for a free world and laid down their lives so that we could continue to build O.R models, unconstrained. There are certainly many other war-movies with stronger military and civilian-OR themes, but these are five sentimental favorites. Enjoy the memorial day weekend as you think of other OR-themed movies!
5. Dirty Dozen (1967) - Jim Brown's incomplete TSP. What a ripper. A politically incorrect movie by today's "lofty" standards.
4. Von Ryan's Express (1965) Sinatra finds a feasible train route through a hostile network, but doesn't make it.
3. Haqeeqat (1964) - Final scene. The Indian army fails to build a robust supply-chain that is required for high-altitude warfare against the Maoists. Still remains the greatest Indian war movie ever made. An incredibly touching rendition of 'Kar Chale' by the great Mohammad Rafi that tears listeners up every time.
2. Enigma (2001) . Alan Turing, tragic genius, WW2 Bletchley park. OK, this is a relatively new movie, but the point is that critical 'analytics' battles have been fought by humble OR-types who saved countless lives and certainly did not nit-pick over whether it was 'OR' or 'analytics'.
1. Sholay (1975) Amitabh's fatal attempt to sever the critical link in the network and save lives succeeds. Favorite all-time movie scene in the most popular Indian movie ever made to date.
5. Dirty Dozen (1967) - Jim Brown's incomplete TSP. What a ripper. A politically incorrect movie by today's "lofty" standards.
4. Von Ryan's Express (1965) Sinatra finds a feasible train route through a hostile network, but doesn't make it.
3. Haqeeqat (1964) - Final scene. The Indian army fails to build a robust supply-chain that is required for high-altitude warfare against the Maoists. Still remains the greatest Indian war movie ever made. An incredibly touching rendition of 'Kar Chale' by the great Mohammad Rafi that tears listeners up every time.
2. Enigma (2001) . Alan Turing, tragic genius, WW2 Bletchley park. OK, this is a relatively new movie, but the point is that critical 'analytics' battles have been fought by humble OR-types who saved countless lives and certainly did not nit-pick over whether it was 'OR' or 'analytics'.
1. Sholay (1975) Amitabh's fatal attempt to sever the critical link in the network and save lives succeeds. Favorite all-time movie scene in the most popular Indian movie ever made to date.
Monday, May 16, 2011
Prescriptive Analytics: Optimize the history you are about to create
An important part of business analytics / OR practice is to assess the impact of a prediction-driven decision-support (PDS) system on:
a) the end-users aka consumers
Data-driven analytical prescriptions are based on perturbing a predictive model, which in turn is (usually) based on the observed collective consumer response to the same or similar products offered in the past. If the PDS recommends a clearly obvious pattern of decisions that differ from the past, it can change the behavior of even non-savvy customers, the cascading effects of which can be disruptive to the product provider's business. With all these mobile apps, the population of non-savvy customers is shrinking every day. Therefore, being proactive in designing the PDS to account for this feedback can be important.
b) the PDS
Some times, the decisions that the PDS recommends today (based on yesterday's history), become part of tomorrow's history, which in turn drives the predictor. However, we can be proactive in designing a PDS that is more likely to 'make' a friendlier history down the line. Furthermore, the 'inventory' of history you need to stock up on to calibrate your predictor can also be minimized. History, like gasoline, is often a scarce resource in OR practice.
a) the end-users aka consumers
Data-driven analytical prescriptions are based on perturbing a predictive model, which in turn is (usually) based on the observed collective consumer response to the same or similar products offered in the past. If the PDS recommends a clearly obvious pattern of decisions that differ from the past, it can change the behavior of even non-savvy customers, the cascading effects of which can be disruptive to the product provider's business. With all these mobile apps, the population of non-savvy customers is shrinking every day. Therefore, being proactive in designing the PDS to account for this feedback can be important.
b) the PDS
Some times, the decisions that the PDS recommends today (based on yesterday's history), become part of tomorrow's history, which in turn drives the predictor. However, we can be proactive in designing a PDS that is more likely to 'make' a friendlier history down the line. Furthermore, the 'inventory' of history you need to stock up on to calibrate your predictor can also be minimized. History, like gasoline, is often a scarce resource in OR practice.
Monday, May 9, 2011
Will the associates of #OBL move east?
Will the discovery of OBL in a posh hill-station in Pakistan increase the probability of finding the rest of his crew members? If so, will it be optimal for the other members to relocate from their current hideout to a safer place to minimize their chances of discovery but at the risk of briefly surfacing? What is the conditional probability that they are also in eastern Pakistan, given that OBL was found in that area? Does it increase or decrease? If Al-Q initially wanted to avoid an 'all eggs in one basket' situation, then they must have spread out and chosen to hide in places comfortably far away from each other. But on the other hand, if it was 'every rat is on his own', then they may have in fact have ended up, either independently or in collusion, gravitating toward the same geographical area.
It's my 2 cents worth of arguments based on just a flimsy data point that the latter case seems likely now. That they are hiding in places where they least expect the US to pop-in unannounced (Pakistan was never a concern, it now seems). A few hundred yards from a major military academy so far away from Afghanistan must have seemed pretty safe and for just for this reason alone, the raid achieved ultimate surprise. If this is the case, then there's probably a couple more in the vicinity. Pakistan-occupied Kashmir seems like another safe haven. It's further east, and most of the terrorist camps, whose recruits so predictably hit India after the winter snow melts to open up the passes, are located there. Furthermore, the element of surprise is gone for most part. Pakistan is unlikely to welcome further chopper incursions unless they are extremely well compensated.
So an Al-Q card-carrying member who lives west of Abottabad may feel the need to head further east. If he gets into India (which is just 60 miles away from Abbotabad!) through the porous border, there's 1.2 billion people to blend into.
It's my 2 cents worth of arguments based on just a flimsy data point that the latter case seems likely now. That they are hiding in places where they least expect the US to pop-in unannounced (Pakistan was never a concern, it now seems). A few hundred yards from a major military academy so far away from Afghanistan must have seemed pretty safe and for just for this reason alone, the raid achieved ultimate surprise. If this is the case, then there's probably a couple more in the vicinity. Pakistan-occupied Kashmir seems like another safe haven. It's further east, and most of the terrorist camps, whose recruits so predictably hit India after the winter snow melts to open up the passes, are located there. Furthermore, the element of surprise is gone for most part. Pakistan is unlikely to welcome further chopper incursions unless they are extremely well compensated.
So an Al-Q card-carrying member who lives west of Abottabad may feel the need to head further east. If he gets into India (which is just 60 miles away from Abbotabad!) through the porous border, there's 1.2 billion people to blend into.
Saturday, May 7, 2011
Informs Northeast Conference 2011 - summary
The INFORMS Northeast conference 2011 was held at U Mass, Amherst yesterday and today. There were many wonderful presentations from students and practitioners alike and it did not feel too crowded either. There was good representation from the OR-strong schools in the Northeast. In terms of industry participation, bigwigs IBM, Oracle (Retail), GE, the US Military, and a host of other interesting LLCs participated, giving students some valuable exposure to some real world business analytics and OR.
My personal favorites:
- The poster session: One example: Using OR to optimally do vehicle routing and sequencing to expedite power restoration after a major power loss in the network at multiple locations. Pretty innovative. and cool.
- Dr. Simchi Levi's plenary on the effectiveness of long chains in improving flexibility - Innovation in supply chain optimization continues...
- Optimal use of Airspace/runway capacity at congested airports - There were three talks featuring young OR/MS faculty members - I suspect we will hear a lot more about them in the coming years
- Agent based simulation by GE corporate research to optimally and practically manage a terribly complex 10-year project involving river dredging. There is something intuitively appealing about ABS and its utterly object-oriented approach to modeling and I do hope I get to use that idea somewhere.
Other comments:
- students (and some others as well) need to be doing more scientific-graphical presentations. More pictures - not the clip-art clutter / video junk type, but a well-thought out, informative visualization of scientific results that makes your approach more transparent to an audience that doesn't necessarily share your background. Mind numbing equation after equation is a kill-joy. Why tell when you can show?
- Of course, great work by the main chair, Dr. Hari J. of U Mass (that's thirty letters in a name, like his blogspot address). The 'health care and OR' sessions were quite packed (there were even some bonafide MD's presenting analytical stuff!) and hopefully this means that a lot more innovative work is on the way in this important area.
Nice work, and hope for an encore next year.
p.s: GPS sub-optimal response reconfirmed
This issue was mentioned on this non-blog a few months before, and now we have reconfirmation that for a brief time period, the GPS routing algorithm (Garmin, my favorite brand) does not always recalculate the optimal path if you deviate from the chosen route. The best route from Elmsford, NY to Amherst MA per google maps appears to be any one of three arc-disjoint pareto-optimal paths (practically). For a period of 10 minutes when I was off the GPS-recommended route in pursuit of an alternative optimal route (that was less congested historically), I was asked to get back to plan and every rejected exit increased my ETA. Finally, the GPS thingy gave up and recalculated from scratch and suddenly the ETA plunged to a value pretty close to the original value, as expected.
This strategy to 'get back to plan' is not uncommon and I don't consider this a defect (not yet). At large airlines, real-time aircraft routers and dispatchers (aka irregular ops) typically utilize such a scheme to minimize any unintended cascading disruptions introduced by wholesale re-optimization.
My personal favorites:
- The poster session: One example: Using OR to optimally do vehicle routing and sequencing to expedite power restoration after a major power loss in the network at multiple locations. Pretty innovative. and cool.
- Dr. Simchi Levi's plenary on the effectiveness of long chains in improving flexibility - Innovation in supply chain optimization continues...
- Optimal use of Airspace/runway capacity at congested airports - There were three talks featuring young OR/MS faculty members - I suspect we will hear a lot more about them in the coming years
- Agent based simulation by GE corporate research to optimally and practically manage a terribly complex 10-year project involving river dredging. There is something intuitively appealing about ABS and its utterly object-oriented approach to modeling and I do hope I get to use that idea somewhere.
Other comments:
- students (and some others as well) need to be doing more scientific-graphical presentations. More pictures - not the clip-art clutter / video junk type, but a well-thought out, informative visualization of scientific results that makes your approach more transparent to an audience that doesn't necessarily share your background. Mind numbing equation after equation is a kill-joy. Why tell when you can show?
- Of course, great work by the main chair, Dr. Hari J. of U Mass (that's thirty letters in a name, like his blogspot address). The 'health care and OR' sessions were quite packed (there were even some bonafide MD's presenting analytical stuff!) and hopefully this means that a lot more innovative work is on the way in this important area.
Nice work, and hope for an encore next year.
p.s: GPS sub-optimal response reconfirmed
This issue was mentioned on this non-blog a few months before, and now we have reconfirmation that for a brief time period, the GPS routing algorithm (Garmin, my favorite brand) does not always recalculate the optimal path if you deviate from the chosen route. The best route from Elmsford, NY to Amherst MA per google maps appears to be any one of three arc-disjoint pareto-optimal paths (practically). For a period of 10 minutes when I was off the GPS-recommended route in pursuit of an alternative optimal route (that was less congested historically), I was asked to get back to plan and every rejected exit increased my ETA. Finally, the GPS thingy gave up and recalculated from scratch and suddenly the ETA plunged to a value pretty close to the original value, as expected.
This strategy to 'get back to plan' is not uncommon and I don't consider this a defect (not yet). At large airlines, real-time aircraft routers and dispatchers (aka irregular ops) typically utilize such a scheme to minimize any unintended cascading disruptions introduced by wholesale re-optimization.
Tuesday, April 19, 2011
Is your query being queried?
In the world of OR practice, it is almost always the case that a solution approach for a particular analytical problem in one industry is mathematically equivalent to something that was solved in a completely different context three decades ago. To avoid re-inventing the wheel, a first and basic step to rule this in or out would be to Google this (for example). However, there are some unexpected occasions when a query that combines some known well-documented methods and ideas returns very few hits. These situations are interesting because it hints at a probability that the attempted method/approach could be something that is practically new or rare. Perhaps you are asking an original and specific question in a context that doesn't have a well-documented answer. Could this potentially be an 'intellectual property' ? I get excited, but ...
On the other hand, these are exactly the queries that could also interest a search engine provider - technical words/phrases that on their own generate a shipload of responses, but in combination, yield no more than a handful. I get a bit worried. Will my query be flagged and put in a 'queue in a cloud' somewhere, waiting to be inspected by man and machine? Am I going overboard here? Two decades of explaining OR to non-OR people can do that. But didn't that Watson thingy provide the right questions to answers in a jiffy without even using the Internet? Maybe I'm just better off reinventing the wheel and doing 'original research'. Let the lawyers sort it out.
After three weeks of blissful soaking in the joy of India's successful world cup campaign that culminated in the biggest victory party in the world atop cloud nine, it's back to square one.
On the other hand, these are exactly the queries that could also interest a search engine provider - technical words/phrases that on their own generate a shipload of responses, but in combination, yield no more than a handful. I get a bit worried. Will my query be flagged and put in a 'queue in a cloud' somewhere, waiting to be inspected by man and machine? Am I going overboard here? Two decades of explaining OR to non-OR people can do that. But didn't that Watson thingy provide the right questions to answers in a jiffy without even using the Internet? Maybe I'm just better off reinventing the wheel and doing 'original research'. Let the lawyers sort it out.
After three weeks of blissful soaking in the joy of India's successful world cup campaign that culminated in the biggest victory party in the world atop cloud nine, it's back to square one.
Subscribe to:
Posts (Atom)