Striking The Balance Between Man And Machine In IT Management

A keynote presented in Paris at CNSM 2011 on 27th October 2011.

How shall we use humans and technology to manage IT resources? During the past twenty years there have been large shifts in the technologies used to manage IT systems, but very few shifts in the underlying principles of management. I believe that we need a better understanding of this collaboration between man and machine to understand management in general. In this keynote, I will use the theme of human knowledge and comprehension to drive a picture of what is needed of IT management in the future.

Choosing man or machine

You enter a restaurant and move to the wardrobe to deposit your coat, where

1. You back into a cubicle, allowing a robot to crudely lift your arms and remove the outer garments and place them in storage ready for dinner.
2. A tall well-dressed gentleman or hostess smiles and greets you softly while helping you out of your coat.

You are driving in your car, trying to find the way but you're lost, so you

1. Call a personal service on your mobile to a single mother working from home with a map that helps you with directions by talking you through the journey (or tune into your mother-in-law in the back seat).
2. You program an GPS with a few touches of the screen and a robot voice guides you calmly into port with pictures.

The balance between human guidance and automation has never been completely intuitive to anyone involved in technology. Technologists err on the side of technology, while almost everyone else assumes that the human touch is needed to solve a problem.

In the 1960s the first robotized factories raised fears that replacing humans with machines was dehumanizing. Futurist and author Alvin Toffler, channeling Lewis Mumford, countered that what is truly dehumanizing is asking humans to behave like machines in the first place, in production lines and sweat shops, to build pyramids and massive infrastructure.

I have written often about this over the past ten years or more, arguing that all of the lessons learnt in industrialization are now having to be learnt again for IT. But what have we really learnt from 40 years of industrialization?

Mistakes

The claim is that IT is basically following the same path that society followed during the industrial revolution. For example, the introduction of simple minded machines with great power that can enact change by brute force was indicative of industrialization. One thinks of the simple minded tools that have been scaled up by brute-force to cope with the scale of variation: MIB, CIM, rdist, etc.

A second mistake is in the segregation of ownership -- between the infrastructure and those who work on it. The division into land owners and slaves - or in IT, we have network management and server management as almost completely unrelated activities, with different models and very different ideas, in spite of facing essentially identical problems. Network management has become a kind of rich landowner, populated by engineers and Cisco University graduates -- whereas system administration (the domain of the server) has a much lower status, and sysadmins are generally thought of a `mere mechanics' who are trained at community colleges, or learn by apprenticeship. So where do we draw the line between these strata, between workforce and infrastructure, and how does automation help?

Today, the new awareness of commerce in IT, and the need to align IT services more directly with operations, has placed a service oriented spin on technology. This is healthy mainstream productization. We turn space rockets into 747s. This is how humans mature technology.

Today, developers are getting into the game with "DevOps" - impatient with the level of service they are getting from manual human system administrators, who have not really advanced in step with commerce.

When technologists design systems for this or any other area of society, they often miss the target. Steve Jobs' passing has reminded us of the skill with which Apple has very effectively brought IT to the masses - but what about the larger problems of management? It is very hard for technologists to put themselves in the frame of mind of human reaction because it is so complex and so fickle.

I have always argued that we should not merely study computer systems, but in fact human-computer systems -- as these two components are strongly coupled.

IT configuration

In this talk, I want to re-examine what we are doing wrong in managing human-computer systems from a number of perspectives and at a number of levels.

There are several themes I want to talk about

• How we design management configurations to support different purposes
• Knowledge of identified patterns to express and compress management information
• Organizing, learning and using domain expertise for humans and machines
• The role of autonomous behaviour in modelling and solving constraints

Configuring systems is not a new thing, it is only a re-invention of an old problem. Managing a datacentre is not that different from landscape gardening, only the materials are different. So this deals with a few things that I can say something about:

• Efficiency and rational thinking.
• Human motivation and irrational thinking.
• Knowledge as an enabler and as a ranker.

Since leaving academia to pursue technology, my work has been focused on understanding the following themes, which I believe are the way to understand what is currently happening in technology, and what needs to happen going forward.

• Knowledge engineering
• Autonomous cooperation - Promise Theory
• Desired States
• Self-healing

I don't want to talk about these things explicitly here today, as they are now well documented -- but I do want to talk about how they fit together into a larger picture of human-computer systems.

What are machines good at - and do we need it?

After 2000 years of civilization from agriculture to the information age, there can hardly be much doubt that we need tools and machines.

Machines are good at being consistent and at being able to inflict greater force on a problem than humans. They are not very good at identifying problems or adapting to changing situations. Machines are impersonal -- sometimes that is good and sometimes it is bad. If you are looking to engage, it is bad, If you want the peace of mind to concentrate, it is good. In the examples above, a robot wardrobe attendant would be just a little creepy and you probably wouldn't trust it to not break your arms to remove your coat. On the other hand, sometimes machines give us peace and quiet from the fickle demands of a human interaction. The GPS is an example of that.

Sometimes a machine can be a mirror in which to look at our own humanity. The ELIZA experiment where people would share their innermost feelings to a lisp program, that gave the feeling that its impartial responses might lead them to some kind of revelation. Monitoring systems are somewhat like this too: they show us something about our situation that is hard to see without help, but they need our help to interpret what is there -- machines don't have our abilities to recognize.

What are humans good at - and do we need it?

Can machines replace humans? I hope that the examples from the beginning show that they cannot do so in all situations. But there are two sides to this. It is neither technically possible nor desirable to ask machines to do what humans excel at, because machines exist for our purpose/desires, not the other way around. On the other hand, machines can achieve feats of concentrated force that no humans can muster.

We as humans have to remember this, as one thing we are good at is falling into patterns of behaviour -- habits that are entertaining but addictive. We sometimes enjoy victimizing ourselves and becoming slaves to the machines we create.

Something else humans are good at is being emotional. Although science fiction movies have somehow taught us a false cliche that emotions do not go together with technology, it is funny how all those stories end up having an emotional resolution. Emotions are the reason we do everything, and trying to suppress them simply makes no sense. We need to understand how emotions enter into systems, not brush their existence under the rug and hope for the best.

When some of us, myself included, favour an approach to solving problems that is based on rational methods, we do so out of an emotional desire to see that agenda fulfilled, not because we are rational. The day we are no longer happy with the outcome it is emotional instincts that are involved in telling our guts that the answer is wrong.

Human goals are the primus motor for what happens in society. As soon as we start asking humans to service machines, we have gone too far. And yet we do this all the time -- expect humans to adapt to technology.

What drives technology?

Pursuing that theme, it is worth examining what is really going on in technological progress. Which inventions are successes and which are failures -- and why?

Technologists often imagine that users choose technology because it is the best tool for the job, and that if we simply make the best tool everyone will flock to use it. But that is not the reason we use tools. We use tools if they give pleasure or remove pain.

Since the second world war, there has been a change of vision. If you look at the future dreams prior to the second world war, there was a lot of talk of big civilisation. Perhaps this stemmed from the vestiges of colonialism in Europe .. and perhaps it was the a result of the great depression during which capitalism seemed to fail. Either way, you see dreams of great galaxy spanning civilizations that persist until the 1950s (The Day The Earth Stood Still etc). Then suddenly the movie makers caught up with what was going on. Repressed by ten years of hardship, people began to think less about society and more about themselves -- their freedom and comfort. The sugar and fat generation - helping themselves.

Instead of ushering in a new chapter of the Roman Empire, Italians bought scooters. Instead of developing Lunar real-estate and prosperity for all, Americans invested in refrigerators and cars.

The real cloud

Today, freedom has exploded in every corner of the world, spurred on by the ease of communication, with electronic devices. The latest symptom of this freedom is the development of commerce within the Cloud -- that insubstantial, indefinable pink elephant of IT today, as inflated to godly status by the Amazons and Googles of the Internet. So what is it?

During the vigil for the victims of the bombing and massacre in Oslo this summer, we saw the real cloud condense in the public square in Oslo, testing the infrastructure to its very limits. There was very little mobile reception with this density of people. No middleware or ontology could rescue us from the simple reality that centralized services do not scale to peak demand. The rigid infrastructure did not cope even though it was based on a state of the art design.

The real cloud is not happening in Amazon's datacentre, it is happening in our back pockets. The real cloud is the gaseous mass of users using IT to exercise their personal freedom. It takes commerce to make this happen, arranging for freedom by financing buried infrastructure.

It's not a Swiss watch...

This kind of freedom does not lead to a beautiful crystalline ontology of neatly typed objects or entities. It is a much more organic and natural structure. Not a diamond, but a muddy ecosystem. Yet we always try to overlay some kind of spanning tree over it, with all of its points of failure and limitations.

This actual beautiful mess is the kind of thing that scientists get excited about, but technologists reject immediately. We study ad hoc networks and structures, but no one really trusts them to scale up predictably. That makes sense. Ad hoc networks are not really useful for large scale reliable service. It's the difference between a bike and a high speed train. They are suited to different problems at different scales.

The Architecture of Computing to Come

What does this mean for IT? It means we need to make models that capture this kind of complexity -- both in order to hide the complexity in the walls, and to handle what goes on behind those walls. In other words, ee need honest knowledge models of sufficient power -- we need a third wave of technology -- that uses its eyes and its ears better.

As IT becomes pervasive, like the basic utilities of society, we expect a seamless, ubiquitous experience. We need to reproduce not only the water taps and electricity sockets but also the garbage collection bins and recycling points...and this is where the Cloud comes in.

The point of the Cloud is not low cost rental where users avoid the need for capital investment - rather it is the application of proper resource management (paging algorithms) at the level of machine computational resources, and the flexibility to enable even small businesses to deploy apps the support the freedom we crave from mobility. It is about reducing the cost barrier of adoption.

The future of computing is something like Cloud Computing API today, not because of the commercial model, but because of the flexible and transparent reuse of resources under a common API.

Cloud computing does for platforms what paging did for disks, and packets did for networks.

That is nothing to do with outsourcing, but everything to do with rational integration and resource management. I believe that the point of the cloud infrastructure phenomenon is two-fold:

• Moving towards to integration of network and server management
• Viewing IT services as resources that support end users in a much more dynamic way

And underneath this API there needs to be a complex pattern-based configuration manager.

The genius that commerce or business brings to technology in a mundane form -- it lies in turning rocket science into daily bread and butter. This is why commerce and individual freedom drive technology.

Risky expertise or packaged dependability?

Why does science need this commerce? The answer is simple when you think about it. The purpose of science is to understand things in isolation, to gradually build up a story about something that we can accept and retell to to others.

The purpose of business is to get people interested in something so that they are willing to trade or buy. Science invests a lot of effort in clarity and invention of new ideas, new models. It is looking for connections between similar things and to distinguish uniqueness. It looks for symmetries and patterns that allow us to see the "general in the particular" and identify principles that can then be used elsewhere.

Business wants trust and predictability above all else. Predictability makes it easy to make a simple story to sell to someone, it brings trust in what you are selling, and trust is the basis of all human interaction.

Science is looking for a tidy classification of its understanding (a simply structured ontology) - to create "knowledge" -- something we'll come back to, as this is a cultural phenomenon. Business does not really have any incentive to care about the global civilization impact of its wares -- it just wants to survive -- by selling -- and this leads to competition, which in turn leads to an evolution and an it becomes a muddy ecosystem.

The key is modelling

What has always struck me as being curious is that science and business have become something of enemies, competing for recognition. But it is science that has the skills to get business out of trouble when things get out of control.

Business generally does not want to invest in science because it adds short-term cost, and science tends to eschew business because it muddies the clear thinking that can only be achieved with a distance from complicating reality. But it is business that keeps society alive.

Now technologists try to live in the chasm between these worlds, and find the balance between the cultures difficult to navigate. The way to balance imperfect understanding with simplicity is to work through a number of models that are acceptable to the different players in the game. Models are a representation, i.e. a "good enough story" or as we say in science a "suitably idealized approximation" to something. We can never have a perfect understanding of something, as Karl Popper pointed out. At best we can know when there is something wrong. So the issue then becomes when is a model good enough, and what specifically are its limitations?

Models are difficult things to create -- and I think we have not always done a very good job. Being people, we tend to start from what we know rather than asking the tough questions and thinking things through -- as this lowers the cost barrier to creating a model. Even scientists do this, which is why we have some superstars like Einstein, who shook off the shackles of convention. Often our most cherished truths are simply wrong.

But a good model will save us in the long run. It needs to start from a representation of reality, where we know what variables describe the system and what freedom we have to change them. What are the freedoms and constraints in the problem?

I think we need to ask, are we really dealing with these issues properly? Or are we imposing traditional old ideas about management and technology onto the current situation and trying to fit a square peg into a round hole?

I believe this is the case with the use if logic as a paradigm for classification in systems (ontology or roles), or knowledge, for instance -- it is a classic case of trying to put humans to work for machines.

The habitual mistake - the dogma of hierarchy

Mutual exclusivity lies at the heart of logic, and it is an idea that engineers instinctively run to. I have tried to ask myself why this should be the case, and I do not have a very good answer. But is has some simple flaws that come back to bite us. Being not very smart Pavlovian dogs, technologists come back and burn themselves on this again and again.

Human thinking is simply not logical. We do not divide and subdivide things in our minds - we mix everything together and even look for connections that are not there. For some reason we believe we are imposing some divine discipline on ourselves by creating taxonomies and hierarchical classification systems, but in fact we are digging a hole for ourselves that very often causes more problems than solutions.

An example in security is trying to separate applications by Chinese walls. It leads to an inefficient an inflexible approach to management.

MIB, CIM, Species, subject classifications in journals -- show me one of these where humans do not find it actually ambiguous to find a general category, or which requires a new category for every new instance (a simple enumeration).

This arrived with libraries. Before that, there was never any need for this approach.

Hierarchies can be understood rather simply as follows: they are spanning trees -- arbitrary viewpoints on a more general associative graph of relationships. We should attribute to them any more authority than this!

Comprehension of complexity

So this brings up the issue of meaning, another key difference between business and science. In science, we emphasize information, and are somewhat embarrassed by interpretation as it compromises a presumed objectivity.

In business, it is the interpretation or significance one is interested in because that is what humans instinctively look for: what does it mean to ME... that is what builds trust. That is what forges a relationship. So our computer science methods need to see the paradox of not modelling significance -- we need this in the real world, so we'd better model it!

Technologists should not worry. Humans have an innate ability to see through informational complexity, because what we are good at is seeing patterns in noisy environments. The complexity that is a headache for rational methods is not a problem for irrational humans because what humans do is to look for low entropy patterns -- to see things that stand out from a particular filter viewpoint.

We do not need the kind of oversimplifications that are often inflicted upon us by technologists. A classic example of this is in organizational structure. These oversimplified approaches might save some initial cost of modelling, but increase the long-term cost of using the model.

People will see what they want to see in whatever mess we create. All we need to do is make it stable and predictable. Management is really about arranging parts in a functional or aesthetic pattern.

It is DESIRED STATE, not optimal state we want.

In other words, it is about human desires, which includes human emotions. And when faced with technology, human desire is often fuelled by a desire to maintain human dignity.

Config = information = pattern

Instead of taking one pattern that we know and trying to force the world into this mold, why not be good scientists and use our eyes to discover what the actual pattern is.

The first place to start would be in creating a system of coordinates to address things that happen in the world of servers. The basic kind of model we need must then answer the questions:

• What,
• When,
• Where,
• How
• Why....

These are the basic questions we must answer in order for a pattern to be maintained. They are the basic elements of a CFEngine `promise'.

What about information? Gathering more and more information is not really helpful -- what we want to do to minimize the amount of work is to identify patterns that simplify the amount of information into a few simple, clearly identifiable items.

Information is defined as the amount of variability in a signal. Things have significance or meaning precisely because they are different from their surroundings, which means less information.

It is not about gathering more information, but about how we reduce information into compressible patterns that yields insight.

You can trade this skill for the brute force of number-crunching in some situations, up to a point, but this is a delaying tactic, to avoid an initial cost of coming up with a better model. For technologists and end-users, this cost is not only computational but emotional.

Untethered `agents' - atoms go anywhere, be anything

To model the kinds of freedoms that commerce it driving us towards, we need something gaseous, flexible, non-rigid - something that avoids the mistakes of hierarchy. This applies to hardware, software -- and to the thing that describes all of these: knowledge.

Rigidly tied together hierarchies limit the way we think and apply ideas, they limit the way we apply resources. If we want to manage systems, we can herd them as loosely coupled entities. We can always add tethers, but to take them away is hard. This is what one does in "annealing" methods to try to relax prematurely applied constraints to later make them fit. But we have no annealing methods for hierarchies, as they are the most.

This was the idea of Promise Theory - to start from these basic atoms, with individual intentions that can be combined into a larger picture .. with their own copy of the script, much like the players in an orchestra.

We need the hidden infrastructure buried in the walls

Although we have a free and personalized experience of computing today, it is illusory because it builds on the existence of a hidden infrastructure, buried in the ground or the walls that some Morlocks have had to put together. Not everyone is free. In order to make this smart environment support the mobile agents that also IT managers want to work with, we also need to look at the dignity of those gardeners and maintainers of the infrastructure.

So, do the same models apply to the hidden infrastructure and to the clients? Traditionally, the answer is no, because we separate network management from host management -- this is the approximate dividing line between the two. Alas, that divide line is changing completely now with on-line services taking over many functions that were traditionally done on PCs.

We want simplicity but not the the extent that there is no longer any control, any power to influence fate. Again, human dignity needs to be respected. This is the main reason trains and planes have manual controls.

Modelling goals -- We turn this into desired states

Technology is not about discovering the truth about the world, it is about accomplishing goals that humans want to accomplish. Human desire comes into our modelling at a fundamental level. Sometimes we pretend that it doesn't, and we talk about optimization and bounded rationality, etc. But these are just covers for applying mathematical methods to the economics of our approach.

Can we train humans and technology to work together?

Approaches to management need to respect the emotions, desires and unreliability of the people or agents working in this area, otherwise we will not be successful at implementing a sensible division of labour between humans and machines.

We have to think of the whole computer system as part of an extended society, including the administrators and end users -- as whole parts of a larger society. It is the economics of this whole virtual society (measured not in dollars but in joy and pain, dignity and satisfaction) that governs success or failure.

Making the best of emergent patterns

We also have to respect what humans are good at, and enjoy doing, their basic qualities. That means that we should start from the kinds of structures and processes that humans generate spontaneously (these things are going to happen like it or not) and use that as our template. Science should always begin with observation.

However messy, such processes will stabilize by some kind of evolutionary pruning, and they represent a point of stability in the environment around which we should base everything else we do.

We see the same problem in the organization of knowledge. Directories like Yahoo that try to pout information into a common ontology have been a complete failure, but brute force searches through structureless data like Google are much preferred because they are ranked according to their social importance -- it is this cost of relevance to the end user that is being compared.

For years the best answer people had to scaling up systems was "make everything the same". We have seen throughout history what happens when one tries this approach in a human society -- it is a failure because it demotivates, destroys creative freedom, etc. The mistake technologists have made is in failing to realize that IT systems are a society.

What would happen if you let everyone do exactly what they wanted?

You would not end up with a perfect diamond, but would the result converge more quickly into something that humans would not fight against -- bringing stability?

Would the result be maximal entropy? Every system is a balance of freedoms and constraints. Evolution would eventually drive the system to search out the space within the immutable boundaries -- behaving like a gas to fill up the available possibilities. But what if this gas has forces of attraction? Eventually it might condense into a structure. This is just as true of an artificial structure as it is of a physical one. Things will start to cluster together.

Humans are the glue in this = trust, like - they generate structures like a muddy ecology

Conclusion

Perhaps it is time to stop ignoring the human part of IT systems, to stop imposing idealized structures on models we expect humans to live with in order to solve technical problems. We can instead look at the actual behaviour, boundary conditions and dynamics of a problem and ask what structures would naturally emerge. We can work with the environment rather than against it. This is not much harder -- CFEngine is proof of that after 18 years of dealing with the most complex environments.

Humans don't want to be constrained - they want, freedom, pleasure and comfort.