The Business Value of System Administration

MB with Carolyn Rowland (login version)
The status of system administrators as experts is at stake as both technology and businesses evolve. To evolve in step, professionals need to become more business aware. In this article, we summarize discussions on Business alignment that took place at the LISA BDIM (Business Driven IT Management) workshops over the past two years, and try to place them into the context of the wider view. The outcome points to some straightforward tips to improve sysadmin business value.

The role of the traditional system administrator is changing. It is being packaged, commoditized, and standardized, just like the hardware and the software it relies on. Even the name `system administrator' is being forgotten and replaced by a generation that doesn't know its history. This should not come as any surprise. It is the inevitable process of evolution at work, forcing improvised origins into mainstream commerce.

In many ways, the changes we see are aftershocks from the arrival of the PC and Microsoft Windows, where commercialization began the transformation, starting with the basic tools. The currency of progress in the world of Windows is tied, of course, to business goals: the PC was created for businesses. By contrast, those who came to system administration from the research culture of mainframes and Unix had mastery of the system as their prize. PCs, like minicomputers, ushered in standardized programs and business recipes so that repeatable simplicity could streamline success.

Today many basic tasks of system administration have been simplified by technologies, such as automated Configuration Management, Web Servers, Content Management Systems and Package Managers. Where does this leave the system administrator as we understand the term? Today the challenges of IT specialists include new issues like massive scale, service orientation, business agility and knowledge management, but the system administrator of the future is going to have to demonstrate new skills and lean business value by steering systems on the fine line between agility and stability.

Business vs Tech - entrenched and under fire

A traditional organization has layers, or departments. Some do management, sales, business development, etc., and some do the technical work of the business (whether that be IT services, carpentry or brickwork). This separation of concerns makes a lot of sense, as the skills and personality types needed to perform sales and management are quite different from those needed for technical work.

Who are these people? The business layers, which traditionally include management, deal with the raison d'etre of the company -- where is the money coming from, and what should the company really be about? There is a lot of thinking and soul searching at this level, planning and brainstorming. Technical levels are governed typically by the more predictable processes of engineering and resource management.

Irksome perhaps (for dedicated IT staff), a business succeeds only if the business departments are successful. The technical worker is a helper, and sometimes an enabler, but technical work alone does not a business make.

In commerce, one needs an edge to succeed:

  • The perception of confidence,
  • Rapid turnaround of ideas, or time to market and
  • The unique ("business") value of what is being sold.

Such concepts have been basically absent from discussions on system administration, before the BDIM workshops at USENIX/LISA 2008 and 2009, but we believe that to develop as a profession, system administrators must confront the IT-Business divide.

Sysadmins speak

The attendees of the BDIM LISA workshops had plenty to say about the IT-Business barrier. We wanted to know how people in the field perceived the relationship between IT and business. We began with some questions:

  • Do best practices exist?
  • What metrics do we have for alignment?
  • How does one define business processes?
  • Where does Research and Development fit in?
  • Does this only apply to E-commerce?
  • What role is played by communications (phone/mail etc)
  • What does mission critical mean?

Not all of these questions were really answered satisfactorily, but the emergent dialogues were a valuable source of insight into the Business-IT relationship at different workplaces. We heard the following issues cited repeatedly:

  • Better communication is needed between Business and IT.
  • Sysadmins, are often the last to hear about needs and changes for the business. Advance warning of upcoming issues helps. Having something like a 5-year plan helps everyone to overcome a day-to-day regimen of fire-fighting.
  • Sysadmins should be trusted partners not grumpy slaves offering expert advice. They need communication and people skills.
  • Upward visibility of sysadmins is needed -- sysadmins need to document the impact of their work in relation to the business. One way is to charge for services in an in-sourcing model.
  • Management should provide incentives to document and simplify processes to prevent development of `king of the hill' scenarios, crippling from within.
  • Organizations need headroom to meet new challenges. Some departments are optimized to the point of rigidity which makes an organization brittle.

We expand on some of these points below as we attempt to paint a broader picture of the Business-IT relationship.

Talk between experts

How do we achieve better communication between Business and IT? Why are IT people often the last to hear about the need for change in the business? Responsibility surely lies on both sides, but one answer is cultural. Rather than working `heads down', IT needs to take an active role in analyzing and advising Business. IT needs to be perceived as a `trusted partner' by Business -- increasing that perception of confidence and ensuring the rapid turnaround of ideas alluded to above.

If Business learns that the IT department has valuable input, it will respond by turning to them for advice. Strong communication and `people skills' are important here. Sysadmins need to learn Business language and avoid IT jargon, as well as talk much more about impact of their work, the mission, and the costs.

A milestone of success is when Business values the opinions of IT enough to make them part of the management team. One way to formalize the relationship is the creation of a 5-year plan for IT. This allows the IT department to be involved at the problem definition level. Ultimately, sysadmins need to develop a relationship of trust with Business. A lack of trust means lack of business credibility and low status for sysadmins. This lack of trust will cost the business too, as it is unable to get the most from IT.

Visibility of the system administrators' work is an important education for Business. Documenting the impact of the work, not just the work itself, shows Business measurable accomplishments from the IT layer. A blind way to achieve this is to simply charge for services using an insourcing model so that Business can see the actual costs of using specific technologies. Being generous with, rather than protective of, expertise shows the Return On Investment (ROI) from IT; such visibility is essential if IT personnel are not to be replaced by a lowest common denominator workforce. External IT requirements (e.g. SOX, HIPAA, FISMA, STIGs, etc.) can be confusing to Business, and expertise is needed to communicate the impact and costs of compliance on the business, plus obtain necessary budgetary support to implement it.

Simplicity vs. Complexity: any colour you like as long as it's black

Business does not usually have the time or wherewithal to comprehend complex technicalities, so a simple environment intuitively seems preferable. Often there is a perception that a simpler environment costs less to maintain: fewer hardware and software requirements, fewer IT staff and fewer skill-sets for those IT staff.

For sysadmins, the mood can be to swim upstream. The latest technology might be irresistible but installing and configuring costs time. Coding private software rather than buying a solution might seem cool, but is it costing the business in the long run? When is it ethical to explore on company time? Does the new item meet a business need? Did anyone ask for that capability? What is the impact? Does it cost more to maintain as the infrastructure becomes more complex or alternatively does it aid in compliance with external requirements or close an IT security hole? What is the ROI?

Many organizations distrust change and variation. They expect heterogeneity to be a problem. A major issue is therefore comprehension: how can IT explain to Business what the system will do if it seems overtly complicated to them? Doesn't a complex system cost too much? IT must better understand business needs in order to communicate the need for heterogeneity to Business. That said, oversimplification is not a foreign concept in system administration either.

At the earliest USENIX/LISA conferences from the 1980s there were papers being written about how to make all machines in an organization identical in order to save work. Thirty years later, it is still a prevalent strategy to create one or two standard "images" and to force these on all machines in the organization to avoid dealing with necessary variation. Cloud computing is even making a business model out of selling people incomplete machines, blank slates, to be configured individually. Alva Couch, Associate Professor at Tufts University, referred to system homogeneity as the "nuclear weapon" of predictability in server management, implying that it is too heavy handed an approach to managing expectations. The counter argument is that consistency has advantages, if there is no need for variation, because it reduces the amount of IT knowledge required. It boils down to the difference between intended or controlled variation and unintended variation (i.e. out of control). A useful middle ground between complete homogeneity and rampant heterogeneity is to foster enclaves of uniformity. System administrators could turn complexity vs simplicity into a conversation with Business on the best way to support the core mission of the organization.

Business can offer incentives to IT to simplify and document the infrastructure. However, forcing a simplistic environment will cause problems when homogeneity is imposed through lack of understanding (such as in a research environment, where freedom to `play' brings long term value). IT needs to sell this to Business; it could cost more in immediate outlay, but serve Business better in the long run.

One way Business has sought to simplify their understanding of IT is through the SLA. The term "SLA" (Service Level Agreement) was originally conceived as a legal agreement between service providers and service consumers, documenting promised operational levels and repercussions if service could not be delivered. Over time, the term has been used in an increasingly casual way to talk about promised objectives. But a more fundamental question is: are we providing the right catalogue of services in the first place?

Simplistic marketing promises like "five nine's reliability", i.e. 99.999% uptime, attempt to push people's fear-buttons to spend money on verification. Do such slogans make any rational connection with Business goals? What is the cost of keeping such a promise? Would the Business rather be happy with a promise of two nines at a tenth of the cost? To be able to make a promise is a powerful confidence builder, but it has to be one that it can afford to keep. Sysadmins need to bring a rationality to this discussion on behalf of Business.

Leaving Room For Change

Business likes to think that it is the driver of change, and everyone else is dragging their feet. This is not the case. Usually both sides drag their feet. Business leaders generally like the word `innovation' but sneer at words like `research', so there is a paradox to be faced -- as noted above. People tend to stick with what they know, even when it is harmful to them.

Agility in IT or in Business requires us to be able to face sudden challenges. Reactive fire-fighting approaches to IT management hinder this. Creative thinking requires "headroom" or "slack". IT has to plan for this, and Business has to fund it. Business commonly sees IT support as a tax on their organization. If one could create a lean-and-mean IT organization, then there would be more cash to spend on main mission-specific objectives. However, bogged down with fire-fighting, or optimized to the point of rigidity makes an organization brittle and breakages occur during sudden change. Some organizations, fearful of allowing change, intentionally throw syrup over their staff in the form of red tape and bureaucracy, which makes it hard for them to be agile.

One way to improve agility is to modularize. Modularity of systems is universal lore today. We understand that building systems from `off the shelf' standard parts has many advantages, including economies of scale and the possibility of outsourcing. The service paradigm is part of this phenomenon, and the culture has edged its way steadily into IT since the 1980s.

Best practices -- do they exist?

Complexity and fragmentation of IT operations has motivated another kind of standard over the past 25 years. Enter COBIT and ITIL which present themselves as 'best practice' frameworks. These de-facto standards employ various levels of recommended practices for IT governance and service delivery. Individuals can even get certified in the ways of ITIL. Controversy surrounds the efficacy of ITIL and critics decry the added beauracracy and lack of agility that comes with a full ITIL implementation. Moreover, the frameworks are unable to explain why they deserve the accolate "best". What is interesting is that, once again, their very existence suggests a need and hence something wanting in the industry. In spite of their paucity of technical content, they exist to provide Business with a simple language with which to define, describe and manage IT.

What did you do for Business today?

When all is said and done, consider the question: what did you contribute to the success of your business today? We'd wager that most system administrators, junior or senior, would falter if asked this question, mumbling something about technologies and help tickets transacted, rather than business impact. What about: what does your organization do? What is its primary goal? Few organizations reflect on their own activities clearly, and it is easy to over-simplify: universities do not just do teaching, Microsoft does not just do software development. Organizations have diverse departmental activities, all contributing to their day to day business.

Reflecting on goals is usually shoved into a `manager' tray. `That's not my job to think about!' Perhaps herein lies the root of a problem. Planting this question could provoke an act of infectious cultural awareness.

Here is a very business-like thing to do: a Top Ten list of priorities. Prioritization is an economic imperative. Such lists ask us to make value judgements, and confront pressing issues. You can make your own.

  1. Answer the question: 'What does your Business do?' Then ask 'How does IT fit into the big picture?'
  2. Show leadership in Business-IT decision making. Arrange regular meetings between Business and IT, even appoint a permanent liaison who can translate.
  3. Communicate effectively. IT should learn the language of Business; drop the jargon and speak in terms Business will understand.
  4. IT should start thinking and talking about the impact of the work, rather than the details (think ROI). Find out how others assess IT's efforts and use that knowledge to increase visibility.
  5. Understand the business in order to explain complexities (heterogeneity).
  6. Cache knowledge. Make procedures as simple as they can possibly be, so that you can scale them in a crisis, without having to rely on expertise always being present.
  7. IT should know when to spend company time (e.g. money) researching new technology. What does it do for the business? What problem does it solve? Be prepared to defend the time and effort.
  8. Create buffers. Weak coupling of modular roles offers much needed slack or headroom for the IT department.
  9. Formulate changes and strategies in terms of promises (who, what, when, where, why) and make it your business to keep them.
  10. Be open to personal as well as institutional change.

Summary

Thinking about the interface between Business and IT, it is tempting to place oneself in the trenches and to blame `middle management' from above or below, some poor person that is responsible for oiling the gears. That is merely avoiding the issue. The issue is rather the whole organization's collective role in its own management.

If we ask how to align Business and IT, it makes sense to find the common ground. Sysadmins and engineers try to bring that predictablity to users. Business-folk are trying to engineer predictable streams of revenue in a fickle environment. Are these views compatible? Surely they are, with openness and focus.

We have seen conference discussions that spend hours arguing over sysadmin self-image rather than bite the bullet of change and adapt. What will be the professional shape of system administrators to come? They will have to be increasingly in-tune with their organization's diverse goals. They will ask: what are the core promises of my organization, and what did I do to keep these promises today?

References

An extended version of this article is available at: http://www.iu.hio.no/~mark/blog.html

[KeyNote] M. Burgess, Business Alignment and Promises, NOMS 08, Salvadore Brasil, (2008)

[FutureShock] A. Toffler, Future Schock, (1969)

[HPweb] http://www.hpl.hp.com/research/ssrc/services/adaptive/management.html

[RoseGarden] M. Burgess, Promise you a Rose Garden, http://research.iu.hio.no/papers/rosegarden.pdf

Glossary

ITIL - IT Infrasture Library
SNMP - Simple Network Management Protocol
CIM Common Information Model
DAP Directory Access Protocol
DENng Directory Enabled Networks (next generation)

This work is supported by the EC IST-EMANICS Network of Excellence (#26854)