October 20, 2009

For Gabriel Tarde, the notion that one can cleanly separate economics from sociology is pure nonsense. The current financial crisis has given some credence to his belief. Chrystia Freeland may well be right to write "the boom - and bust - happened because investors obeyed the logic of financial markets" (*). But the logic underlying herd behavior runs counter to the rationality of microeconomic agents and it will be a while before Andrew Lo's adaptive markets hypothesis can synthesize a more realistic account of homo economicus.

This should not discourage us. But why not tackle first a narrower task, the study of recommendation systems? Though limited in scope, this topic is of paramount importance. Could our Information Age exist, let alone prosper, without them? It generates far too many lies for us to uncover by ourselves and recommendations are our only way to save time, our most precious resource.

Faithful readers know of course these fillips have taken up the challenge early. Beyond classic cases such as Netflix, they have for instance analyzed Google as a unique combination of two recommendation systems. Beyond individual cases, they have looked at the business models which may, or not, sustain them. And overall they have stressed how difficult it is to build a good system, neither crooked by insiders, nor abused by outsiders.

There is no lack of examples. Although they execute the transactions themselves and commingle their results, hedge fund managers may still be considered to be what we call interested recommenders, who directly and legally benefit from the transactions they recommend to their clients. One of them, Michael de la Merced reports (**), was charged last week of illegal insider trading. On the other end, the general media derives no direct benefit in recommending specific events as news worthy of our attention. Unfortunately Brian Stelter says the police suspects the recent scare about a balloon bound boy was a hoax staged by a fame famished father, eager to trick ever gullible editors (***).

But perhaps we have missed the forest for the tree. Meeting recently with Glenn Ellison (****), the TNIT interviewer gently pokes fun of this economics professor for ignoring Walmart, after preaching this company "deserves more attention than it gets in our profession". Indeed "the retailer's website, Walmart.com, is already the second-busiest retail website after Amazon and has annual revenues of several billion dollars" writes Jonathan Birchall (*****). But even in its brick and mortar stores, Walmart may be found just as close to Google as, online, to Amazon.

By design, a large store has plenty of available space left once the variety of products on offer is wide enough to satisfy the consumer. So it displays selected items in redundant repetition on its shelves if not piled high in its aisles. This is nothing but a recommendation system based on an interested business model. Ideally the retailer will want to maximize profit computed as its margin on the recommended item times the number of sales per period. Divide by the number of visitors per period, assuming it is fixed for a given store and the goal becomes to maximize the item margin times the probability the item is picked by a consumer. Replace pick by click and you have the Google AdWords business model.

The irony is, online, Walmart loses this powerful recommendation engine. Duplicate listings would look downright silly and may repell consumers.

Thus recommendation systems transcend any border erected between traditional industries and the information industry. For a Twitter in search of a sustainable model, there is a Louis Vuitton fighting for an increasingly outdated implementation of its business. Both are missing the opportunity.

Some trees of course do guide us through the forest. Turn to Glenn Ellison and Sarah Fisher Ellison's past study of online sales of computer memory chips (******). Looking at search sites specific to this domain, it finds price-based recommendations to the buyers are routinely "obfuscated" by the sellers. Notice this effect does not involve any illegal manipulation by insiders nor outsiders. It teaches us a new, important lesson.

Some recommenders only issue positive recommendations, such as normally the case with professors sollicited by former students applying for a job or with credit rating agencies paid by banks selling structured derivative products. Others deliver both positive and negative recommendations. The latter may be explicit, as can be read on popular feedback providers, or implicit, as inferred from the lower ranks given by search sites. It is only natural that those who give out negative recommendations face underhanded efforts if not relentless hostility from those who receive them.

One tactic documented by Glenn Ellison and Sarah Fisher Ellison on the search site Pricewatch is to take advantage of "contractual terms [which] are multidimensional". To a really low price, which will trigger a favorable rank, the seller for instance associates sub-standard delivery speed, hoping to switch baited consumers to a higher priced substitute. Pricewatch seems to have responded in a Whack-a-mole approach.

Might not the fault lie with the recommender? If price is thus proven to be one criteria of choice among many, shouldn't Pricewatch switch to a value-based market? Besides why ignore sellers' wishes to rank buyers too, e.g. for their credit worthiness? To search engines delivering automatic, public recommendations, I prefer matching engines enabling mutual, confidential introductions backed with personal recommendations, sollicited and verified according to the rule of three (1). A better model perhaps for credit report agencies for persons and their equivalent for businesses?

Negative recommendations should not always be avoided. Looking at "airworthiness", John Kay presents "that [civil aviation] must be regulated" as self evident (*******). Few will argue that a centralized, governmental agency is not an efficient solution to keep unsafe planes on the ground. His point is that "regulation" should not inflate into "supervision", i.e. "a shadow management" "both extensive and intrusive" and "rarely a success".

John Kay's warning concerns the financial service industry. As it happens, Aline Van Duyn gives us an up-to-date assessment on the aftermath of the failure by credit rating agencies to issue valid recommendations on CDO's (********). The danger here actually lies in the opposite direction, a status quo which protects a tight oligopoly of for profit recommenders from meaningful competition, tempered by a potential for more lawsuits.

This would be the worst of both worlds. Not enough decentralization for the current recommendation system to decrease the influence of individual recommenders' shortcomings, not enough central authority to impose a system design against the participants' self interest.

Bad as it is, it is just a symptom. Under cover of minimizing the administrative burden, the forces of pronaocracy want to wrest from the sovereign government the responsibility for designing sound recommendation systems where the welfare of the nation and of the people are concerned. Sure the administration needs to act as the lone recommender in some instances and, as John Kay illustrated, will be tempted to take a power trip. More often though its role is to promote rules to ensure such mechanisms are fair, well decentralized and sustainable. That's a Nobel Prize to shoot for.

Healthcare reform in the US is not solely about recommendation systems. Yet doesn't this theory provide us a test to assess the underlying disease, the dissolution of democracy?

Philippe Coueignoux

• (*) ............... Investors had little choice but to keep on dancing, by Chrystia Freeland (Financial Times) - October 9, 2009
• (**) ............. Hedge Fund Chief Is Charged With Fraud, by Michael de la Merced (New York Times) - October 17, 2009
• (***) ........... Calling Story of Boy and Balloon a Hoax, a Sheriff Will Seek Felony Charge, by Brian Stelter (New York Times) - October 19, 2009
• (****) ......... Interview with Glenn Ellison, (TNIT newsletter), October, 2009
• (*****) ....... Walmart offers personal products online, by Jonathan Birchall (Financial Times) - October 14, 2009
• (******) ..... Search, Obfuscation and Price Elasticities on the Internet, by Glenn Ellison and Sarah Fisher Ellison (MIT and NBER) - October 2007
• (*******) ... How the skies proved the limits of regulation, by John Kay (Financial Times) - October 14, 2009
• (********) . When top marks no longer count for much, by Aline Van Duyn(Financial Times) - October 14, 2009
• (1) in declaring this preference, I am acting as an interested recommender myself as ePrio happens to propose software to run value-based markets