The Business Value of System Administration
MB with Carolyn Rowland (blog version)
The role of the traditional system administrator is changing. It is being packaged, commoditized, and standardized, just like the hardware and the software it relies on. Even the name `system administrator' is being replaced. This should not come as any surprise. It is the inevitable process of evolution at work, guiding a fledgling profession from improvised origins into mainstream commerce.
In many ways, the changes are aftershocks from the arrival of the PC and Microsoft Windows, where commercialization began the transformation, starting with the basic tools. The currency of progress in the world of Windows is tied, of course, to business goals: the PC was created for businesses. By contrast, those who came to system administration from the research culture of mainframes and Unix, had mastery of the system as their prize. PCs like minicomputers ushered in standardized programs and business recipes, so that repeatable simplicity could streamline success.
Today many basic tasks of system administration have been simplified by technologies, such as automated Configuration Management, Web Servers, Content Management Systems and Package Managers. Where does this leave the system administrator as we understand the term? Today the challenges of IT specialists include new issues like massive scale, service orientation, business agility and knowledge management.
With ever more dependence on IT, Business can no longer rely on a few tried-and-true technical people to maintain an infrastructure as if by magic. IT permeates organizations, and Business live or die through its services. This means a dependence on system administration expertise, but not necessarily on a large number of traditional system administrators. The system administrator of the future is going to have to demonstrate new skills and lean business value by steering systems on the fine line between agility and stability.
Business vs Tech - entrenched and under fire
A traditional organization has layers, or departments. Some do management, sales, business development, etc., and some do the technical work of the business (whether that be IT services, carpentry or brickwork). This separation of concerns makes a lot of sense, as the skills and personality types needed to perform sales and management are quite different from those needed for technical work.
Who are these people? The business layers, which traditionally include management, deal with the raison d'etre of the company -- where is the money coming from, and what should the company really be about? There is a lot of thinking and soul searching at this level, planning and brainstorming. It is a predatory level, characterized by conflict, going head-to-head with challenges. There is talk of "business models" and "value chains", and the bottom line is that someone has to give the company money to keep on going. It is helm and navigation. Technical levels are a more sedate environment, governed typically by predictable processes of engineering and resource management. IT people might have to weather a storm now and again, but they don't do hand-to-hand combat.
Irksome perhaps (for dedicated IT staff), a business succeeds only if the business departments are successful. The technical worker is a helper, and sometimes an enabler, but technical work alone does not a business make. Moreover, we use the term `business' here, but the world is not all that different if you are a university or government institution.
The system administrator of today runs customer call centers, web sites, e-commerce store fronts, maintains data accessibility, IT security, compliance with laws and concerns about privacy. They enable responsive IT services to maintain company standing. Both sides need one another but they do not realize how closely tied they are. It is not unusual to find friction between Business and technical layers, just as there is between any parties who don't fully understand each other. Working together across the cultural divide has become so important, it is now the subject of research.
Business-IT alignment
A few years ago, Mark got interested in the question of how IT serves business needs, driven originally by researchers at Hewlett Packard. This quickly manifested as a series of workshops known as BDIM (Business Driven IT Management), hosted under the IM and NOMS conference series. In HP's words
"BDIM is a set of design principles and control systems that enable IT services to adapt to changes in business requirements, thereby achieving synchronization between Business and IT functions. Decisions on how resources, physical, human and financial, available to the IT function can best be utilized are complex decisions as business requirements frequently change and the IT function is expected to rapidly adapt to these changes." [HPweb]
This is not a bad description of the problem, but researchers are not business people, and the answers they came up with made some assumptions:
- There is already a successful business that just needs tuning a little.
- All processes and relationships are in place and need no alteration.
As often happens, researchers scrabbled to find something they could say about this and took words like "business alignment" literally. Some looked for a way of mapping key performance indicators onto a compass needle direction so that one could compare two angles on a compass to speak of alignment (this is perhaps reminiscent of the skier who claims to make perfectly stylized parallel turns by claiming -- `well, one of the skis was parallel!').
Others began with specific technical issues they understood (such as transaction queue processing) and tried to optimize this in terms of collectable revenue, essentially assuming that all the business layer had to do was to collect a predictable amount of money from each transaction.
To learn more about the wider issues, Mark jumped the divide and started a business, which became the Cfengine company, attracting experienced business people and providing invaluable experience. To them, business was about the following:
- The perception of confidence,
- Rapid turnaround of ideas, or time to market and
- The unique ("business") value of what is being sold.
Such concepts have been basically absent from technical discussions. Now, anxious to reconcile the technical and business views, Mark organized two workshops at USENIX/LISA 2008 and 2009 to ask attendees, from different levels of their workplaces, how they viewed alignment. Carolyn saw immediate value in discussing these issues among an audience of system administrators and participated in both workshops. A variety of questions were put forward:
- Do best practices exist?
- What metrics do we have for alignment?
- How does one define business processes?
- Where does Research and Development fit in?
- Does this only apply to E-commerce?
- What role is played by communications (phone/mail etc)
- What does mission critical mean?
Sysadmins speak
The attendees of the BDIM LISA workshops had plenty to say about the IT-Business barrier. Here is a lightning summary of some of the raw points raised in the sessions, paraphrased by us:
- Better communication is needed between Business and IT (this point was repeated again and again, in many different ways).
- The IT staff, aka Sysadmins, are often the last to hear about needs and changes for the business, as they have their heads down. Advance warning of upcoming issues helps. Having something like a 5-year plan helps everyone to overcome a day-to-day regimen of fire-fighting.
- Sysadmins should become trusted partners rather than grumpy slaves -- offering expert advice. They need communication and people skills to have their advice taken seriously.
- Upward visibility of sysadmins is needed - Sysadmins need to document the impact of their work in relation to the business. One way is to charge for services in an in-sourcing model.
- Management should provide incentives to document and simplify processes to prevent `king of the hill' scenarios from building up and crippling from within.
- Organizations need headroom (`provisioning margins') to meet new challenges. Some departments are optimized to the point of rigidity which leads to brittle bones and breakages.
We want to address some of these points below as we attempt to paint a broader picture of the Business-IT relationship.
Talk, talk
How do we achieve better communication between Business and the IT? Why are IT people often the last to hear about needs for change in the business? Responsibility surely lies on both sides, but one answer is cultural. Rather than working `heads down', IT needs to take an active role in analyzing and advising Business. IT needs to be a `trusted partner' in business -- increasing the level of confidence and ensuring the rapid turnaround of ideas.
If Business learns that the IT department has valuable input, it will respond by turning to the IT department for advice. Strong communication and `people skills' are important but often missing. Add to this an understanding of how Business communicates. IT people need to learn to talk to Business using the right terminology:
- Use Business language instead of IT jargon.
- Talk about impact of the work, the mission, and the costs.
It is easier for IT to learn business-speak than for Business to learn the technical side, so system administrators have to be the ones to bridge the communications gap. The ultimate cooperation occurs when IT is invited to the table by Business to discuss upcoming issues. A measure of success is when Business values the opinions of IT enough to make them part of the management team. This relationship could be formalized by developing a 5-year plan for IT. This relationship allows the IT department to be involved at the problem definition level before the business layer has made a decision about a direction that will impact the technical people.
Ultimately, sysadmins need to develop a relationship of trust with Business. A lack of trust means lack of business credibility and low status for sysadmins. This lack of trust will cost the business too, as an IT department that is constantly surprised may be unable to respond when company has to shift course overnight.
Another way that IT raises management awareness is visibility of the work of the system administrators. Documenting the impact of their work on the business, not just the work itself, provides Business with measurable accomplishments from the IT layer. A blind way to achieve this is to charge for services using an insourcing model so the business can see the actual cost using specific technologies. Being generous rather with than protective of expertise shows the Return On Investment (ROI) from IT; such visibility is essential if IT personnel are not to be replaced by a lowest common denominator workforce.
External IT requirements (e.g. SOX, HIPAA, FISMA, STIGs, etc.) can be confusing to Business, and expertise is needed to communicate the impact and costs of compliance on the business, plus obtain necessary budgetary support to implement it.
In her work, Carolyn found that prior to attempting these communications improvements, the system administrator needed to better understand the core mission of the business. What does the organization do and how does IT support Business? Is it just a tool or is it critical to the image of the company? Understanding the big picture of the business and how IT fitted into it gave system administrators clarity to discuss business IT issues effectively, and Business appreciated that. A good exercise is trying to write a 5-year IT plan.
Simplicity vs. Complexity: any colour you like as long as it's black
Another gap in the Business/IT relationship is in understanding the complexity of the infrastructure. Business does not usually have the time or wherewithall to comprehend complex technicalities, so a simple environment seems preferable. Often there is a perception that a simpler environment costs less to maintain: fewer hardware and software requirements, fewer IT staff and fewer skill-sets for those IT staff.
For IT specialists, the mood can be to swim upstream. The latest technology might be irresistible but installing and configuring costs time. Coding open source modifications rather than buying a solution might seem cool, but is it costing the business in the long run? When is it ethical to explore on company time? Does the new item meet a business need? Did anyone ask for that capability? What is the impact? Does it cost more to maintain as the infrastructure becomes more complex or alternatively does it aid in compliance with external requirements or closes an IT security hole? What is the ROI?
Many organizations distrust change and variation. They expect heterogeneity to be a problem. A major issue is therefore comprehension: how can IT explain to Business what the system will do if it seems overtly complicated to them? Doesn't a complex system cost too much? IT must better understand business needs in order to communicate the need for heterogeneity. That said, oversimplification is not a foreign concept in system administration either.
At the earliest USENIX/LISA conferences from the 1980s there were papers being written about how to make all machines in an organization identical in order to save work. Thirty years later, it is still a prevalent strategy to create one or two standard "images" and to force these on all machines in the organization to avoid dealing with necessary variation. But as Alvin Toffler pointed out in the late 1960s, for production lines, technology exists precisely to make variations cheap[FutureShock].
Cloud computing is even making a business model out of selling people incomplete machines, just blank slates, what Cfengine calls stem-cell hosts, to be configured individually. Alva Couch, Associate Professor at Tufts University, referred to system homogeneity as the "nuclear weapon" of predictability in server management, implying that it is too heavy handed an approach to managing expectations. The counter argument is that consistency has advantages if there is no need for variation, because it reduces the amount of knowledge that needs to be maintained. It boils down to the difference between intended or controlled variation and unintended variation (i.e. out of control). A useful middle ground between complete homogeneity and rampant heterogeneity is to foster enclaves of uniformity.
System administrators could turn complexity vs simplicity into a conversation with Business, on the best way to support the core mission of the organization.
A way to manage complexity-knowledge is to provide incentives to IT to simplify and document the infrastructure, immunizing against silos and `king of the hill' scenarios. This goes hand-in-hand with better tools and communication. If IT people know what the business objectives are (beyond lowering internal operating costs), they may understand better how to reduce unecessary complexities and reduce costs without sacrificing capability. Indeed, sometimes a simplistic environment causes problems when homogeneity is imposed through lack of understanding (such as in a research environment, where freedom to `play' brings long term value). IT needs to sell this to Business because it could cost more in immediate outlay, but it serves Business better in the long run. Change may require concessions on both ends.
Leaving Room For Change
Business likes to think that it is the driver of change, and everyone else is dragging their feet. This is not the case. Usually both sides drag their feet. Business leaders generally like the word `innovation' but sneer at words like `research', so there is a paradox to be faced -- as noted above. People tend to stick with what they know, even when it is harmful to them.
Agility in IT or in Business requires us to be able to face sudden challenges. Reactive fire-fighting approaches to IT management hinder this. Creative thinking requires "headroom" or "slack". IT has to plan for this, and Business has to fund it. Business commonly sees IT support as a tax on their organization. If one could create a lean-and-mean IT organization, then there would be more cash to spend on main mission-specific objectives. However, bogged down with fire-fighting, or optimized to the point of rigidity makes an organization brittle and breakages occur during sudden change. Some organizations, fearful of allowing change, intentionally throw such syrup over their staff in the form of red tape and bureaucracy, which makes it hard for them to be agile. Workloads can be distilled into digestible business-relevant morsels to maintain the confidence of Business. One way to do that is to modularize, but how is Business supposed to have insight into these issues without input from the IT department?
IT Services and the SLA keyhole
Modularity of systems is universal lore today. We understand that building systems from `off the shelf' standard parts has many advantages, including economies of scale and the possibility of outsourcing. The service paradigm is part of this phenomenon, and the culture has edged its way steadily into IT since the 1980s.
The term "SLA" (Service Level Agreement) was originally conceived as a legal agreement between service providers and service consumers, documenting promised operational levels and repercussions if service could not be delivered. It was a genuine contract, with guarantees that were binding. The SLA was associated with QoS or Quality of Service, which was originally only considered to be an issue in network service provision by Internet Service Providers (ISPs). Over time, the term has been picked up on in a broader context, and it has been used in an increasingly casual way to talk about what Mark prefers to call promises. A promise is a documentation of intent, agreed upon by both parties, where the outcome matters to an organization.
Many IT departments use SLAs as a way to talk about Business goals, because it is a concept that both can understand. Jacques Sauvé, Associate Professor of Computer Science Universidade Federal de Campina Grande, Brasil, has pointed out that this view of performance is too simplistic, or `too small a keyhole' through which to view the Business, but it is a natural thing for technical people to cling to.
Another problem with SLAs concerns internal IT shops. Business and IT find the SLA to be common ground so it becomes a tool used by both to attempt to set expectations for the internal IT department. The weakness in this plan, the SLA has no teeth. The Business end is typically not going to fire the IT department if they don't meet the documented expectations specified in the SLA so the agreement becomes meaningless.
A more fundamental question is: are we providing the right services in the first place? Are the promises being made the right ones? In other words, what about the catalogue of services and the way they are composed? What about services that cannot easily be quantified? Everyone is looking for "Key Performance Indicators" or "metrics", i.e. rationally measurable signals that allow us to evaluate the success or failure of our actions.
Best practices -- do they exist?
Strategic enablement and unnecessary cost avoidance could be viewed as some kind of `recommended practice', if these pie-in-the-sky ideals could be turned into technical concrete. Thus the complexity and fragmentation of IT operations has motivated another kind of standard over the past 25 years. Enter COBIT and ITIL which present themselves as 'best practice' frameworks.
These standards employ various levels of recommended practices for IT governance and service delivery. Individuals can even get certified in the ways of ITIL. Controversy surrounds the efficacy of ITIL and critics decry the added beauracracy and lack of agility that comes with a full ITIL implementation. They are unable to explain why they deserve the accolate "best". What is interesting is that, once again, their very existence suggests a need and hence something wanting in the industry. In spite of their paucity of technical content, they exist to provide Business with a simple language with which to define, describe and manage IT.
The reality is that these practices apply classic human-management practices to the field of IT services. They are not solutions for technical personnel, but for `managers' (in whatever sense you care for). Sometimes the clash of culture leads to questionable technical practices (such as unjustified homogeneity); other times, they reveal insights that the technical side often misses.
Another significant challenge for Business lies in `scaling up', or transforming from a small startup or department into a larger enterprise. Bureaucracy is the main tool of predictability in human management, and it is an easy trap which can smother an enterprise and stifle innovation.
Indeed, this dependence on bureaucracy has led to `database' approaches to management technology in the past, such as SNMP, CIM, DEN, DAP, etc, which some feel have held systems back for decades. The problem with bureaucracy is that is was designed to abhor change -- databases have fixed data formats that are not supposed to change significantly. The fact that there is natural and desirable system evolution is a problem that was never solved by these approaches as the data formats cannot adapt to drift without continual redesign. What we see with SNMP and CIM are data formats growing out of control through patching, striving to capture a taxonomy of natural variations by brute force.
In search of certainty: promises and intentions
What of theory? Mark has rallied this cause for several years. At the keynote of the 2008 BDIM workshop in Salvadore Brasil, he suggested Promise Theory as a way of modelling organizations from Business to IT, using the model originally constructed for Cfengine[KeyNote]. One attendee of the 2009 LISA BDIM workshop indicated that the voluntary cooperation methodology of Promise Theory had helped his teams to work together politically as well as technically [RoseGarden].
Without going into the details, the formalization of promises provides a simple methodology: formulate all actions as promises to be kept (best effort), and ask: is there a realistic expectation that this promise could be kept? If not, there are serious design flaws. Alignment then becomes a matter of logically determining whether promises are compatible or incompatible with one another across all levels of the organization. The value of promises is not that they offer guarantees, quite the opposite. They help to document responsibilities and outcomes.
Simplistic marketing promises like "five nine's reliability", i.e. 99.999% uptime, attempt to push people's buttons to spend money on verification. Do such slogans make any rational connection with Business goals? What is the cost of keeping such a promise? Would the Business rather be happy with a promise of two nines at a tenth of the cost? To be able to make a promise is a powerful confidence builder, but it has to be one that it can afford to keep.
What did you do for Business today?
When all is said and done, consider the question: what did you contribute to the success of your business today? We'd wager that most system administrators, junior or senior, would falter if asked this question, mumble something about technologies and help tickets transacted, rather than business impact. What about: what does your organization do? What is its primary goal? Few organizations reflect on their own activities in this way, and it is easy to over-simplify: universities do not just do teaching, Microsoft does not just do software development. Organizations have diverse departmental activities, all contributing to their day to day business.
This task of reflecting on goals is usually shoved into a `manager' tray. `Who cares? That's not my job to think about.' Perhaps herein lies the root of a problem. Merely planting the question could provoke an act of infectious cultural awareness.
Top ten things to do
A very business-like thing to do is to create Top Ten lists of priorities. Prioritization is an economic imperative. Such lists ask us to make value judgements, and confront pressing issues. You should make your own.
- Answer the question: 'What does your Business do?' Then ask 'How does IT fit into the big picture?'
- Show leadership in Business-IT decision making. Arrange regular meetings between Business and IT, even appoint a permanent liaison who can translate.
- Communicate effectively. IT should learn the language of Business; drop the jargon and speak in terms Business will understand.
- IT should start thinking and talking about the impact of the work, rather than the details (think ROI). Find out how others assess IT's efforts and use that knowledge to increase visibility.
- Understand the business in order to explain complexities (heterogeneity).
- Cache knowledge. Make procedures as simple as they can possibly be, so that you can scale them in a crisis, without having to rely on expertise always being present.
- IT should know when to spend company time (e.g. money) researching new technology. What does it do for the business? What problem does it solve? Be prepared to defend the time and effort.
- Create buffers. Weak coupling of modular roles offers much needed slack or headroom for the IT department.
- Formulate all changes and strategies in terms of promises (who, what, when, where, why) and make it your business to keep them. Identify and understand the stakeholders of those promises and ask them how they measure success.
- Be open to personal as well as institutional change.
Summary
Thinking about the interface between Business and IT, it is tempting to place oneself in the trenches and to blame `middle management' from above or below, some poor person that is responsible for oiling the gears. That is merely avoiding the issue. The issue is rather the whole organization's collective role in its own management.
If we ask how to align Business and IT, it makes sense to find the common ground. Science and Business are not all that different. Both do `uncertainty management'. A scientist tries to reduce possible misunderstanding to a minimum and document lasting principles for the future. Sysadmins and engineers try to bring that predictablity to users. Business-folk are trying to engineer predictable streams of revenue in a fickle environment. Are these views compatible? Surely they are, by seeking to enable:
- Superior capabilities through expertise.
- Resources efficiency (with headroom) with guile.
- Clearer cost justification through communication.
- Agile adaptation to meet strategic promises, by all of the above.
What will be the professional shape of system administrators to come? They will have to be increasingly in-tune with their organization's diverse goals. They will ask: what are the core promises of my organization, and what did I do to keep these promises today?
References
[KeyNote] M. Burgess, Business Alignment and Promises, NOMS 08, Salvadore Brasil, (2008)
[FutureShock] A. Toffler, Future Schock, (1969)
[HPweb] http://www.hpl.hp.com/research/ssrc/services/adaptive/management.html
[RoseGarden] M. Burgess, Promise you a Rose Garden, http://research.iu.hio.no/papers/rosegarden.pdf
Glossary
ITIL - IT Infrasture Library
SNMP - Simple Network Management Protocol
CIM Common Information Model
DAP Directory Access Protocol
DENng Directory Enabled Networks (next generation)
This work is supported by the EC IST-EMANICS Network of Excellence (#26854)