B-SidesDC 2018

Choice. The Problem Is Choice

October 2018
Dan Geer

nominal delivery draft

Good morning, all. If you were to come to my office, you would see on the wall these four rules:

Your invitation encompasses all four, and I thank you for it.

There is a lot to say and some of it has the kind of subtle characteristics all of you know only too well. Nevertheless, I will endeavor to be crisp. I must disclaim one thing, and that is that I am speaking for myself and not for In-Q-Tel. At the same time, outreach is part of IQT's core mission and we appreciate the opportunity to give back to the community wherever we can. I would be happy to discuss IQT one-on-one at some other time or connect you to one of my colleagues who are milling around BSides today.

When I was younger, I used to say that I lived in the future. Most people who would say that are thinking William Gibson's line, "The future is already here -- it's just not very evenly distributed". I did not mean it that way; what I meant was staring into the fog of the future as hard as I can stare and so to pick a path through it.

Some of you will recognize a verse from 1st Corinthians, Chapter 13, "For now we see through a glass, darkly; but then face to face: now I know in part; but then shall I know even as also I am known". In the original Greek, the "dark glass" is a word with substantial cybersecurity import; it is "enigma". I will return to this point shortly, but in the meantime let me be clear: Cybersecurity and the future of humanity are conjoined now. Each of cybersecurity's core realities has policy freight, but full analysis is too subtle for this morning's short exercise.

However you feel about the origins of the world, it is now our world. We are evolving it. Our intervention in evolution is just more evolution, but at a faster clock rate. Changes that depend on the favorable alignment of random perturbations take geologic time, including time to test for side effects. Changes that are designed may have just as many unforeseen consequences, but the equilibria they punctuate are of much shorter duration, often a duration short enough that there is little real stability between episodes of punctuating rapid change. By analogy, as wind strengthens at sea, waves stop getting higher but become closer together.

In my career, I have seen enough of these cycles of stability and surprise to find the description apt even if its cadence is arrythmic. The most challenging security Request For Proposal I ever received was three bullets long:

and, despite its brevity and clarity, it was simply impossible to meet -- keeping at constant risk was, and remains, the bugaboo.

The root source of risk is dependence, especially dependence on the expectation of stable system state. Dependence is not only individual but mutual, not only am I dependent or not but rather a continuous scale asking whether we are dependent or not; we are, and it is called interdependence. Interdependence is transitive, hence the risk that flows from interdependence is transitive, i.e., if you depend on the digital world and I depend on you, then I, too, am at risk from failures in the digital world. If individual dependencies were only static, they would be eventually evaluable, but we regularly and quickly expand our dependence on new things, and that added dependence matters because we each and severally add risk to our portfolio by way of dependence on things for which their very newness confounds risk estimation and thus risk management. Interdependence within society is today absolutely centered on the Internet beyond all other dependencies excepting climate, and the Internet has a time rate of change five orders of magnitude faster. Remember, something becomes "a critical infrastructure" as soon as it is widely enough adopted; adoption is the gateway drug to criticality.

One thing we see is that there is a kind of oscillation going on; thirty years back, you would take your data to where the computing was -- a university's central computing facility perhaps. Then inventions made it possible to move the computing to where the data was and every worker came to have something on their desk with which to do that processing. Then come the twin innovations of virtualization and software-defined networks, so the distal end is once again a display tool while the data has again gone to where the computing is (only now we can't tell where "where" is). The next oscillation has already begun but it is unevenly distributed; computing is going back to where the data resides -- in the sensor fabric, per se.

At each stage of evolution, the best X ever produced is also the last X ever produced when it is itself rendered irrelevant by the leading edge of the next phase. Unless we have become as Gods, some mix of the unexpected and our adaptations to it will be that of which the future is made. Every invention we make proves either non-viable, which is to say a non-functional mutation within our human-designed ecosystem, or viable, which is to say it seizes an hitherto unoccupied niche either by creating that niche or by stealing it from whatever occupies it now. Whether creating or stealing niches matters not, since, over time, the number of niches increases, at least until a catharsis, which in the original Greek meant "purification" or "cleansing".

The important thing to realize here is that to the biome, to the occupier of a niche that was here yesterday and will be gone tomorrow, extinction is a surprise. No tree can say "Time to move uphill" and no salamander can think "I need to mutate." Evolutionary change depends on this unpredictability; otherwise yesterday's winners are tomorrow's winners, yesterday's dominant species only get more dominant tomorrow. This is true both for Nature as wetware and nature as software.

I trust that some of you have read Nassim Taleb's work for which a, if not the, central point is that as the tails of a distribution get fatter, the mean, the average, the predictable becomes a function of the distribution's extreme values alone and all the rest of the distribution is so much window dressing. A fat-tailed setting inherently resists prediction, but for that very reason makes prediction ever more compelling to pursue. Taleb synopsizes the present technosocial momentum:

[We are] undergoing a switch between [continuous low grade volatility] to ... the process moving by jumps, with less and less variations outside of jumps.

Note that I am not conjuring up nation state actors or Divine intervention, though I personally believe that both are at work and at all times. What I am suggesting is that change is what evolution is about, that change is rarely steady but rather tends to be abrupt, that change is event driven, that the amount of change an event engenders is proportional to the surprise with which that event arrives, and that we cannot make this otherwise. Our preaching to this topic wastes airtime; the more we say "It is coming" for some cybersecurity value of "it", the more those who live in the moment will have good and marketable reason to ignore us. Speaking as a person with some training in probability, the point to remember is that probabilistic events occur eventually.

If we look at Nature in the form of the equations of ecology, we see two alternative games for survival, r-selection and K-selection. R-selected species produce many offspring, each of whom has a relatively low probability of surviving to adulthood while K-selected species are strong competitors in crowded niches. K-selected species invest more heavily in many fewer offspring, each of whom has a relatively high probability of surviving to adulthood. If we change the term from "produce many offspring" to "re-image frequently" you now have precisely the world of VMs. Or, to be more current still, the kind of container components in a DevOps setting where it is arguable whether moving target defense or minimizing new product introduction latency is the paramount goal.

This idea of "punctuated equilibrium" has a hold on me. I trace the birth of the cybersecurity industry to Microsoft's introduction of a TCP/IP stack as a freebie in the Windows 95 platform thereby taking an operating system designed for a single owner/operator on, at most, a private net and connecting it to a world where every sociopath is your next door neighbor. That event was the birth of our industry, though the fact was unnoticed at the time.

The second of these moments occurred somewhere around 2006 when our principal opponents changed over from adventurers and braggarts to professionals. From then on, mercenaries, some armed with zero-days, have dominated. The defense response has been varied, but the rise of bug-bounty programs and software analysis companies are the two most obvious. An Internet of Things with a compound annual growth rate of 35% will be like anabolic steroids for at least those two.

Two Augusts ago we passed a third such moment. The DARPA Cyber Grand Challenge showed that vulnerability finding and fixing, which has heretofore required human experts, may shortly come within the ken of fully automatic programs, or, shall we say, algorithms that are today at the upper level of skill with intelligence, in some sense, soon to follow. As with both of the previous two turning points, the effects will reverberate for the indefinite future. I have long argued that all security technologies are dual use, and the day after the Cyber Grand Challenge Mike Walker, its DARPA program manager, said as much: "I cannot change the reality that all security tools are dual-use".

Those who wrote "[W]henever any Form of Government becomes destructive..., it is the Right of the People to alter or to abolish it" also wrote "[T]he right of the people to keep and bear arms shall not be infringed", and they wrote both at a time when the weapons of the yeoman farmer were on par with the weapons of the infantryman. In the intervening centuries, weapons of infantries so surpassed those of the yeomen that any right of the people to abolish destructive government could not rely on weapons kept at home, but relative might between state and non-state is today closer than it has been at any time since 1791. This oscillation in the balance of power may be peaking, but never before could a dozen guys and gals in their pajamas meaningfully annul the State's use of force.

Just as hunters must lead their target, we must lead ours. Much of the time, however, we first have to argue -- arguing what our target actually is in what is a target-rich environment. But let me step back just a moment before contributing to that argument.

First, what is security? My definition, which underlies my analysis today, is that a state of security is the absence of unmitigatable surprise. Let me repeat that; a state of security is the absence of unmitigatable surprise. A state of security is not the absence of surprise, it is the absence of unmitigatable surprise. In other words, we will fail from time to time. Second, coming from the first, is that the highest goal of security engineering is "no silent failure" -- not no failure, but no silent failure. You cannot mitigate something that you are not aware exists.

All well and good. As you nonetheless know directly, our hardest problem is not that of deploying mitigations but rather deciding how to spend our inherently limited deployment capacity. What mitigations do we actually deploy? Which malware do we counter? What vulnerabilities do we patch? What alarms do we investigate? You know this as well as I do -- the true task of the security worker is to pick what to do based on an informed sense of which task selects for the better future. There is always too much to do. There is always a top-down demand for perfection.

I could now start making predictions, per se, but I won't. They might be right, and they might be wrong, but unless we want to start putting real money on the table, those predictions would be worth what you paid for them.

Instead, for the purpose of this short talk I am going describe some choices we have to make, choices we *will* make one way or another. They are choices we have to make not in the sense of what to have for breakfast, but, rather, in the sense of coming to a fork in the road and picking one of turnings, realizing full well that we will never come back to that fork as unencumbered as we are now.


Much has been written on security metrics, but we are still short on what constitutes the minimum essential set for cybersecurity. Charles Darwin said "All observation must be for or against some view if it is to be of any service". At the macro level, the fork in the road we face is which of cybersecurity's two possible goals our metrics are our observations to be for or against. Is it driving the mean time between failures (MTBF) to infinity or is it driving the mean time to repair (MTTR) to zero? Both can be said to achieve the same goal, 100% availability, but which do we steer by? Don't say "both."

Embedded systems:

You can call this the Internet of Things is you prefer. Qualcomm's Swarm Lab at UC Berkeley predicts 1000 radios per human by 2025. Pete Diamandis' book _Abundance_ calls for 45x10^12 networked sensors by 2035. These kinds of scale cannot be supervised, they can only be deployed and left to free-run. If any of this free-running is self-modifying, the concept of measurable attack surface is just plain over. So, too, is the concept of trustworthy computing, at least as is presently understood. But to make clear the fork in the road, embedded systems must have either a remote management interface or they must have a finite lifetime. A remote management interface delivers serious risk *to* us almost as much as it delivers us *from* serious risk. Enforcing a finite lifetime means culpability for those who fail to implement it, but that means culpability for whom and how? Either way, immortal and unfixable is anathema.


As mentioned earlier, the automated discovery of vulnerabilities is a classic dual use technology. Our choice, our fork in the road, is EITHER to move toward proving that a body of code implements its spec and no more than its spec -OR- to embrace moving target defense. Code proving is a, if not the, gold standard, but it is also difficult enough to do that after a successful proof is in hand you must then adopt brutal change control to protect your investment. Moving target defense may be good enough to reverse the defender/attacker strategic asymmetry that today favors attackers, but it also guarantees that how any fielded instance of code really works at any given moment requires an intermediary to explain. Taking either path is a serious choice supported by serious results obtained by serious scientists; it is also a choice that is hard to later rescind.


Three years ago, my colleagues at a leading software security assessment firm said that the sizes of applications being submitted for analysis were so large that they had to be machine written code; so they were able to say that even machines write vulns. Today, they say that the average code blob is very small, consistent with the trend to micro-services. This is more of the same -- but now we throw in small modules that might each be provably correct and ask whether security can be made composable. Can we find a way to ensure that connecting two secure modules results in a secure composite? Probably not, but DARPA Program Manager Sergey Bratus is formulating an effort to see if parsing, at least, can be made to be composable. Until the composability question is answered, the choice of module size and interconnection strategy presents fork after fork after fork from which you have to choose.

To Fix or Not to Fix:

In a 2014 Atlantic Monthly article, Bruce Schneier asked a cogent, first-principles question: Are vulnerabilities in software dense or sparse? Treating that as a policy question, if they are sparse, then every one you find and fix meaningfully lowers the number of avenues of attack that are extant. If they are dense, then finding and fixing one more is essentially irrelevant to security and a waste of the treasure spent finding it. Six-take-away-one is a 15% improvement. Six-thousand-take-away-one has no detectable value. My policy answer is that for any body of stable code under competent leadership, vulnerabilities become sparse over time, but as code volume explodes the total number of extant vulnerabilities must rise, which again leads us to Nassim Taleb and the heavy tails of power law distributions (and the black swan). Ongoing work by various researchers on the rediscovery rate for vulnerabilities is instructive in this regard. In any case, this is another fork in the road: defense by hardening versus moving target strategies, dance like a butterfly or sting like a bee.


In the real world, abandoned things that matter -- a car, a house, a baby, a bank account -- get seized for the public good. Abandoned code should be no different, and abandoned code seized "for the public good" means open-sourcing it. This fork in the road is a Hobson's choice, a take it or leave it proposition. Note that in the sense used here, "code" means not just source but the build environment, too. Reproducibility is the issue; reproducibility is the requirement. Tell that to your average router maker. But if seizing an abandoned code base is too big a stretch for you before breakfast, then start with a Certifying Authority that goes bankrupt -- who gets the keys?

Machine learning:

Algorithms that are self-modifying are either blindly trusted or they must be interrogatable, by which I mean that you must be able to get an understandable, coherent, checkable answer to the question "Hey, Algorithm, why did you just do that?" Yet as data volumes grow, algorithm efficiency grows more crucial, but the more efficient the algorithm, the less interrogatable it is. That was the exact theme of a workshop held by Morgan Stanley and the Santa Fe Institute in October, 2014, entitled "Are Optimality and Efficiency the Enemies of Robustness and Resilience?" If we choose the blind-trust fork, then that is what we do, we blindly trust (thus setting in place the pre-condition for a silent failure). If we choose the fork that requires interrogatability, then we have a research grade problem to solve before there is any more deployment, and we give up optimality and efficiency for robustness and resilience. Because we have to. This fork presents itself anew every single day we put anything in production.


A public safety argument mandated that we geocode all cellular devices in real time. Do we use a public safety argument to mandate geocoding the Internet? If so, we have a change to the dynamic of both attack and service provision in addition to a new duty for ISPs. Of course, if cellular endpoints come to overwhelmingly dominate the client-side Internet, which now seems likely to be soon true, parts of this question may become largely moot. Nevertheless, since 1648 Westphalian States have had a "monopoly on the legitimate use of force within their territory". For that to have continuing relevance, cyberspace must be balkanized and each State must amass the means to project cyberforce. Yet, as we know, with enough interconnections physical boundaries and cyber boundaries lose all correlation, so States increasingly define cyber-territory by where their subjects electronically go, whether by destination control in the Chinese style or by data control in the EU style. And so appears a second impetus to geocode the Internet -- the merger of national security with domestic tranquility.


Amongst the classic triad of confidentiality, integrity, and availability, we have heretofore prioritized confidentiality, especially in the military sector. That will not be the case going forward. In the civilian sector, integrity will supplant confidentiality as the highest goal of cybersecurity, if for no other reason than a growing dependence on data-driven algorithms. Confidentiality may not even be position number two -- that may be held by availability. What we get to choose at this fork in the road is a body of procedural niceties as to when a need for integrity trumps a need for confidentiality, which is, of course, where anonymity and provenance have their long delayed head-on collision. This leads us to...

Data mitigation:

We have to soon choose what we want to happen when stolen data is, for example, not just exposed but also put on a blockchain from which it cannot be erased. Put differently, assured data deletion is far harder than permanent data retention, yet many civilized goals, including but hardly limited to a right to be forgotten, require the sealing of records or their outright destruction. Which do we give up, the slick usefulness of immutability or information crime being unmitigatable? What does consent of the governed mean when a technology trumps a Court Order? Might we say that the fork in the road is whether a technology that trumps a Court Order is a choice between variants of libertarianism or a characteristic of what technology must be found to be inherently unlawful? What *is* to be done when contraband information appears in an otherwise innocent blockchain? If an algorithm is in charge of something we care about, what *is* to be done if the data from which that algorithm was derived becomes suspect or, even, is made to disappear?

Names and Reachability:

The fraction of the world's endpoints that have names is falling. If you combine an estimated three billion endpoint addresses that are globally reachable and a global routing table that contains about 750,000 entries, then a routing table entry represents, on average, approximately 4000 potential endpoints -- in CIDR terms that's a "/20". Notably, many of the smallest routing table entries are NAT gateways and so might represent a vast population of endpoints. In the sparseness of IPv6, a named end point is discoverable and hence reachable whereas an un-named end point is not. Are the nameless constrained to initiate outbound transactions but never to accept inbound ones? What is a key-management model for a trillion small devices that don't have resolvable names? In the name of security, do we require nameless, outbound-only entities to ask for an update from time to time and, if so, only from entities that do have names? Momentum says that, soon, the majority of Internet end points will not be describable by name nor discoverable by scanning. Another layer of indirection will, as ever, solve some problems and create others. Provenance and forensics will all but surely be affected. So what, really, is the meaning of taking the fork that leads to namelessness? More to the point, do we dare take it if we can't answer that question?


*The* most telling fork in the road of them all is whether we retain an ability to operate our world, or at least the parts we would call critical, by analog means. Analog means, and only analog means, do not share a common mode failure with the digital world at large. But to preserve analog means requires that they be used, not left to gather dust on some societal shelf in the hope that when they are needed they will work. This requires a base load, a body of use and users that keep the analog working. There is a genuinely substantial debate going on now in Sweden over cash-less-ness and exactly what regulatory mechanism for keeping a base load of cash processing in place is societally necessary. As the difference between Sweden and Puerto Rico demonstrates, being cashless by design or cashless by accident is equivalent. Sen. Angus King's S.79 is pending; it would require that the electric grid be operable by analog means. It needs your support. But, entirely uniquely, keeping the analog working is also a civil rights issue.

What we have here is an historic anomaly, an anomaly where the most readily available counter to an otherwise inexorable drift into a vortex of singleton technology risk *and* the preservation of a spectrum of non-trivial civil rights is one and the same counter: the guarantee, by force of law where necessary, that those who choose to not participate in the digital melange can nevertheless fully enjoy life, liberty, and the pursuit of happiness, that to opt out of the digital vortex does not thereby require that they live in medieval conditions, and, by doing so, we reap a national security benefit in the bargain as those opting out *are* the base load for the analog alternative. This is a fork in the road worthy of Robert Frost,