False Positives & the Limits of Predictive Analysis

Analytic systems share system limits with financial markets.

Correspondent Lew G. recently sent me a thought-provoking commentary on the limits of "total information awareness" in terms of any information system's intrinsic rate of generating false positives.

In essence, the rate of false positives limits the effectiveness of any predictive system. The process of attempting to eliminate false positives is inherently one of diminishing return: even with no expense spared, the effort to eliminate false positives runs into boundaries of signal noise and generation of false positives.

To the degree that financial markets are ultimately predictive systems, this suggests a systemic cause of "unexpected" market crashes: signal noise and the intrinsic generation of false positives lead to a false sense of confidence in the system's stability and its ability to predict continued stability.

Here are Lew's comments:

Resources to deal with reality are inherently limited by that reality. 

Information, to the contrary, is inherently infinite, because of the fractal nature of reality. 

A property of that information reality is that 'meaning' is relative to other items of info, and that any single item can change the interpretation of a big set of facts. E.g., "Muslim, bought pipes, bought gun powder, visits jihadi sites, attends the Mosque weekly, tithes ..." can be completely changed in meaning by a fact such as 'belongs to the Libertarian Party', even 'is a plumber, 'is a target shooting enthusiast'". 

This will continue to be true no matter how much info the NSA gathers: it will be a small subset of the information needed to answer the question 'possible terrorist?'. 

Thus NSA's tradeoff of privacy vs security is inconsistent with reality: no matter how much info they gather, no matter how sophisticated their filters, they can never detect terrorists without a false positive rate so high that there will be insufficient resources to follow up on them.

In other words, if the system's lower boundary is one false positive per million, no additional amount of information gathering or predictive analysis will lower that rate of false positive generation to zero.

Why does this matter? It matters because it reveals that large-scale analytic systems are limited by their very nature. It isn't a matter of a lack of political will or funding; there are limits to the practical effectiveness of information gathering and predictive analysis.

Though Lew applied this to the NSA's "total information awareness" program, couldn't it also be applied to other large-scale information gathering and analysis projects such as analyzing financial markets?

This was the conclusion drawn by the father of fractals, Benoit Mandelbrot, in his bookThe (Mis)Behavior of Markets. As Mandelbrot observed: "When the weather changes, nobody believes the laws of physics have changed. Similarly, I don't believe that when the stock market goes into terrible gyrations its rules have changed."

All this should arouse a sense of humility about our ability to predict events, risks and crashes of one kind or another. In other words, risk cannot be entirely eliminated. Beyond a certain point, we're sacrificing treasure, civil liberties and energy for not just zero gain but negative return, as the treasure squandered on the quixotic quest for zero risk carries a steep opportunity cost: what else could we have accomplished with that treasure, effort and energy?

This entry was drawn from the Musings Reports, which are sent weekly to subscribers and major contributors.

Posts and email responses will be sporadic in October due to family commitments. Thank you for your understanding. 

---

The Nearly Free University and The Emerging Economy:
The Revolution in Higher Education

Reconnecting higher education, livelihoods and the economy

With the soaring cost of higher education, has the value a college degree been turned upside down? College tuition and fees are up 1000% since 1980. Half of all recent college graduates are jobless or underemployed, revealing a deep disconnect between higher education and the job market.

It is no surprise everyone is asking: Where is the return on investment? Is the assumption that higher education returns greater prosperity no longer true? And if this is the case, how does this impact you, your children and grandchildren?


We must thoroughly understand the twin revolutions now fundamentally changing our world: The true cost of higher education and an economy that seems to re-shape itself minute to minute.

The Nearly Free University and the Emerging Economy clearly describes the underlying dynamics at work - and, more importantly, lays out a new low-cost model for higher education: how digital technology is enabling a revolution in higher education that dramatically lowers costs while expanding the opportunities for students of all ages.

The Nearly Free University and the Emerging Economy provides clarity and optimism in a period of the greatest change our educational systems and society have seen, and offers everyone the tools needed to prosper in the Emerging Economy.

Read the Foreword, first section and the Table of Contents.

print edition (list $20, now $18)

Kindle edition: list $9.95 

---

Things are falling apart--that is obvious. But why are they falling apart? The reasons are complex and global. Our economy and society have structural problems that cannot be solved by adding debt to debt. We are becoming poorer, not just from financial over-reach, but from fundamental forces that are not easy to identify. We will cover the five core reasons why things are falling apart:

1. Debt and financialization
2. Crony capitalism
3. Diminishing returns
4. Centralization
5. Technological, financial and demographic changes in our economy

Complex systems weakened by diminishing returns collapse under their own weight and are replaced by systems that are simpler, faster and affordable. If we cling to the old ways, our system will disintegrate. If we want sustainable prosperity rather than collapse, we must embrace a new model that is Decentralized, Adaptive, Transparent and Accountable (DATA).

We are not powerless. Once we accept responsibility, we become powerful.

Kindle: $9.95       print: $24

Thank you, readers, for your much-appreciated generous contributions to this site-- I am greatly honored by your support and readership.