906 Chapter 36 Firing System Administrators 36.2.2 System File Changes If someone suspects that he may be fired, he may create a back door—a secret way to get into the system—or plant a logic bomb, or software that causes damage once he has gone. Ideally, you can take a snapshot of all software before the person becomes suspicious and compare it to the running system on a regular basis. However, that is time consuming, requires a lot of storage, and would easily tip off the person that something is about to happen. However, no suspicions can be raised if such a thing is always done. Programs that checksum system filesand report changes are commonly found. The earliest to achieve popularity is named Tripwire. If this process is an automated system that is used regularly to notice external intruders, system failures, or other problems, it will be much easier to use it without raising suspicion. However, care must be taken to make sure that that person being fired doesn’t update the database so that his changes aren’t noticed. Such a system is an excellent measure to detect any kind of intrusions. However, it is time consuming to process all the false-positives. The issue becomes scaling it to too many machines. 36.3 Conclusion Firing SAs isn’t fun or easy, but sometimes it has to happen. The basics are very simple. The most important rule is to follow the policies of your HR de- partment. HR personnel are the experts, and you are supporting their process. Three tiers of access must be removed: physical access, remote access, and service access. The primary benefit of the three-tier model is that it provides a structured, rather than ad hoc, approach and is tolerant of mistakes made at any one level. Architectures that seek to minimize the number of access databases and a well-maintained inventory ease the process considerably. In creating a checklist for all the manners of access to be disabled, one might begin with the new-hire procedure as a starting point: Whatever is done for a new hire must be undone for a termination. Although no checklist is complete, we have assembled several checklists of things to disable in the event of termination: • Physical access. Change combination locks, all applicable safe combi- nations, and locks on doors with keys, even if they are returned. Re- move access for all buildings: for example, remote locations, shacks, and utility buildings. 36.3 Conclusion 907 • Property surrender. Have the ex-employee turn in keys, card-keys, badges, HHAs, PDAs, and any company-owned equipment at home. • Remote access. Modem pools, ISDN pool, VPN servers, in-bound net- work access—that is, ssh, telnet, rlogin—cable modem access, xDSL X.25 access. • Service access. Remove access from database servers, NIS domains, NT domains, superuser access IDs, Netnews IDs, password files, and RADIUS servers. The icing is a set of design and operational factors that better prepare a site for these unlikely but important tasks. The fewer the administrative databases, the easier the task will be, but if they are all tied to a single au- thentication database, the entire process becomes much simpler. Regularly maintained file checksum histories provide a way to detect and prevent back doors and logic bombs. Dividing the process into HR policy and physical, remote, and service access brings clarity to the process. The process can be explained easily. The staff can be divided into a physical team, a remote team, and a service team. Each team can then work with complete focus because it has only one task. This process works best when one can leverage the infrastructure that should be in any system. A solid security infrastructure keeps the wrong people out. Having a single (or few) administrative databases, such as a well- implemented HHA architecture, makes disabling all access from a central place a snap. Properly documented environments and well-maintained in- ventory improve one’s ability to disable all access quickly. Routine Tripwire runs and system-monitoring processes are some of the automation that may already be in place at a site. The better the infrastructure is, the easier this process becomes. The process described in this chapter handles the extreme case of ter- minating an SA but is also a useful model to consider when anyone leaves a company, simply leaves your domain of support, or changes jobs within a company and should no longer have privileged access to the systems she previously administered. We don’t cover those topics directly. We felt that it would be more interesting to cover one extreme case and leave the others as an exercise to the reader. Our discussion of this topic has been restricted to the technical side of the process. The nontechnical side, the human side, is equally important. You are changing this person’s life in a very profound way. The person has bills to pay, 908 Chapter 36 Firing System Administrators a family to support, and a life to live. Corporate policies range from “get them out the door immediately” to “we’re laying you off in six months.” There are potential problems with both, but from our point of view, the latter not only works best but also shows trust and respect. This issue is not so much for the benefit of the person being laid off as for the benefit of those remaining. Exercises 1. When someone is fired, does HR know whom to contact in the IT orga- nization to have the person’s access disabled? 2. In your current environment, what must be disabled if you were to be fired? Outside of checking individual hosts for local accounts, how many individual administrative systems did you have to touch? 3. What improvements to your system could make it easier to disable your access when you are fired? 4. A system like Tripwire causes periodic points of filesystem I/O. How does that affect the planning and deployment of such a system? How is this different for a file server, an e-commerce server, and a database server? Epilogue We began this book asking for a concise definition of system administration. Now we’re no closer to an answer. If anything, we’ve broadened the defi- nition. Rather than building a crisper definition of system administration, we’ve discussed customer support, repair, operations, architecture definition, deployment, disaster planning, and even management skills. System admin- istration is an extremely broad field, and no simple definition can cover it all. We hope that you’ve learned a lot from reading this book. We’ve certainly learned a lot by writing it. Having to put into words things that had become second nature has forced us to think hard about everything we do, every habit we’ve developed. The peer-review process stands us naked in front of our mentors and comrades to receive criticism of our fundamental beliefs. We’re better for writing this book, and we hope you are better for reading it. We hope that some day, you write a book and enjoy the same exhilaration. The most exciting part of this book has been to record, in such a permanent form, the rants and anecdotes that we have accumulated over our careers. We respond to certain technical and nontechnical issues by getting on our soapboxes to expound our opinions. These monologues are refined every time we repeat them, until we find ourselves repeating them word for word, over and over again. We can honestly say that this book includes every tub-thumping rant we authors blurt out with Pavlovian predictability. With a little bit of luck, these rants will stand the test of time. This book also captures every useful anecdote in our library of experience. Each anecdote teaches an important lesson or two. We can rest assured that these anecdotes will not be lost, and we can safely look forward to all the new anecdotes we will accrue in the future. System administration is a culture. Every culture has its anecdotes, myths, and stories. It is how we pass our history to new generations and propagate 909 910 Epilogue the lessons and values that are important to us. We learn best from hearing our culture’s stories and anecdotes. We enrich the culture every time we share a new one. We’d like to share with you one final anecdote. A Concise Definition A facility had several researchers from a variety of universities visiting for the summer. That autumn, after they left, the SAs had to decommission their computers and clean the large room they had been sharing. The SAs found a scrap of paper that had been taped near the phone. It simply said, “Makes things work,” followed by the phone number of the SAs. It was absolutely the highest compliment they had ever received. Appendixes This page intentionally left blank Appendix A The Many Roles of a System Administrator This appendix is heavy on philosophy. If that turns you off, you can skip it, but we think that it will help you think about your place in the universe or at least your role within your company, organization, or SA team. Examining your own role within an organization helps you focus, which helps you do a better job. It can give you a long-term perspective on your career, which can help you make the big career decisions necessary for having a happy and successful life. This can also give your organization a framework for thinking about what roles they want you to play. Each of these roles in some way affects your organization. This is by no means a complete list; however, it is a very good starting point. You should use this list to consider what roles are missing in your organization and perhaps to start on a quest to fill them. It is interesting to think about which and how many of these roles you are asked to play as your career moves forward. The list can help you plan your career. Some entry-level SAs are asked to play single roles and grow into more roles as they gain experience. Sometimes, SAs start out flooded with many roles and specialize as time goes on. A small site may require its single SA to take on many roles. As the organi- zation grows, certain roles can be transferred to newly hired SAs. Sometimes, you discover that you don’t enjoy a particular role and look to avoid it when you change jobs. Thinking about these roles may also help guide your career with respect to what kinds of companies you decide to work for: Small com- panies tend to require people to fill multiple roles, larger companies tend to require people to specialize, and megacorporations have people so special- ized that it can seem bizarre to outsiders. Technology companies respect and 913 914 Appendix A The Many Roles of a System Administrator reward those who play the role of pushing for new technology, whereas other companies often discourage too much change. A.1 Common Positive Roles Some roles within a company are more critical than others; some are good and some are bad. Here we list many common roles, the value they provide to the company, how those people derive satisfaction from the job, and what customers tend to expect from them. A.1.1 The Installer Some people view an SA as the person who installs “stuff.” This is the role that customers see most often, and so is most often associated with the career of system administration. The customer rarely sees the other, possibly more critical, positions, such as the people who design the infrastructure. The value to the company of Installers is their ability to follow through and see that the job gets done. They are often the final and most critical link in the deployment chain. When installation is being done on a large scale, the item that is being in- stalled is usually preconfigured at some central location. Installers are trained on the specific situations they are expected to see and have a second-tier re- source to call on if they come across an unexpected situation. In that case, the kind of person who makes a good Installer is one who enjoys meeting and helping the customers and gets satisfaction from doing the same task well many times over. On the other hand, in smaller deployments, the Installer is often expected to be a higher-skilled person because more unexpected situa- tions will be encountered. When you are the Installer, it is important to be friendly and polite. The Installer is the public face of the organization; people will assume that the entire organization acts the same way that you do. A.1.2 The Repair Person Things break. Some people view an SA as a Repair Person. Just as people call a dishwasher repair person when their dishwasher breaks, they call a com- puter Repair Person when their computer breaks. SAs also repair bigger and sometimes more nebulous things, such as “the Internet” and “the database.” Whether the real problem is simply a broken cable or a much larger problem is of little interest to the customer. A.1 Common Positive Roles 915 The value to the company of Repair People is their ability to bring the company back to life when technological problems stall a business. Repair People receive satisfaction from knowing they’ve helped one person or the entire company. They enjoy the challenge of a good puzzle or mystery. When you are the Repair Person, customers want to know that you are concerned about their problems. They want to feel as though their problems are the most important problems in the world. A.1.3 The Maintainer The Maintainer is the person who keeps previously built systems going. Main- tainers are very good at following the instructions presented to them, either in a written manual or through training. They do not seek to improve the system; they are willing to maintain it as it is. The value to the company of Maintainers is bringing stability to the environment. These people are not going to break things trying to improve them or replace them; nor do they spend all day reading magazines about new things to install. Once companies spend money to install something, they need it to be stable long enough to pay for itself before it is replaced with something newer. Maintainers receive satisfaction from knowing that their work is part of the big picture that keeps the organization working. They tend to be glad that they aren’t the people who have to figure out how to design and install the next generation of systems and may even have disdain for those who wish to replace their stable system with something new. When you are the Maintainer, customers want two opposing things: They want to know that you are maintaining the stability of their world, and they want you to be flexible when they seek customizations. A.1.4 The Problem Preventer A role that is invisible to most customers is the Problem Preventer, who looks for problems and fixes them before they become visible. Problem preventers do the behind-the-scenes planning and preventive maintenance that keeps problems from occurring at all. A good Problem Preventer collects metrics to find trends but also has an ear to the ground to know what future problems may arise. The value to the company of Problem Preventers averting problems, which is less costly than fixing problems when they happen. [...]... are the Customer Support SA, customers want their issues dealt with on their schedules If they perceive the situation to be an emergency, they expect you to stop everything and help them 932 Appendix A The Many Roles of a System Administrator A.1.32 The Policy Navigator The Policy Navigator understands the rules and regulations of the bureaucracy and can help others navigate through or around them The. .. by the process of the transition; and he did not do much, if any, testing The next day, when people returned to work, many of them found that they had no network connection: the CEO, the entire customer-support department, the Cowboy’s management chain, and many of the engineers The helpdesk and the rest of the SA group had no idea what had happened, because he had not told anyone, and he didn’t bother... have the same priorities that they do A.1.5 The Hero The SA can be the Hero who saves the day Like the firefighter who pulls people out of a burning building, the Hero receives adulation and praise The network was down, but now it is up The demo wasn’t going to be ready, but the SA worked all weekend to bring the network to that part of the building Heroes get satisfaction out of their jobs from the. .. from the old network to the new one The plan was agreed on, the equipment was ordered, a schedule was laid out and agreed on with the customers, and test plans were being built with the customers The day that the new equipment arrived, the Cowboy stayed late, ripped out the old network equipment, and installed the new gear He did not use the new architecture; he ignored all the components of the transition... reliable and easy for them to understand, and they want it now, because when you miss a deadline, it makes their projects late too A.1.8 The Policy Writer Policies are the backbone of IT They communicate the wishes of the top corporate officials and dictate how things should be done, tell when they should be done, and explain why they are done SAs are often asked to write policies on behalf of management... is happy with the current OS release, the current OS distribution, the current brand of computers, amount of bandwidth, and type of network topology The Staller doesn’t see the lack of technology refresh and the problems that it has created Ironically, this person used to be on the cutting edge but has now become stuck in a rut She may have been the person who eschewed the mainframes and adopted UNIX... emulators), and this mothering was exactly what they needed In her morning walks, she would answer hundreds of questions and resolve dozens of problems that would otherwise have been tickets submitted to the helpdesk The customers got very used to this level of service and soon came to rely on her morning visits as part of what kept them productive The value to the company of the Mother is her high degree of. .. it to the rest of the network via a routing-enabled workstation with two NICs These interconnections were unreliable because hosts route packets slowly, especially when they become overloaded The more overloaded the main networks became, the more dedicated subnets were created Eventually, much of the slowness of the network was caused by the slow interconnections between these private pools of bandwidth... that the customers use, so he can “feel their pain.” As this person becomes more sophisticated, he automates his monitoring but then watches the monitoring system s output and takes the time to fix things rather than simply clear the alarms The value to the company of the Monitor is that problems are noticed before customers start complaining This can give the perception of a troublefree network The. .. create a house of cards The value to the company of the Disaster Worrier is felt only in times of emergency Half the system is failing, but the other half keeps working because of controls put in place General system robustness can be the result of this person This person receives satisfaction from ensuring safety and stability When you are in this role, others around you may get tired of your constant . existing systems and improving them, scaling large systems into hu- mongous systems, and overhauling legacy systems and replacing them with newer systems. Infrastructure Builders are proud of their. are the backbone of IT. They communicate the wishes of the top corporate officials and dictate how things should be done, tell when they should be done, and explain why they are done. SAs are often. overloaded the main networks became, the more dedicated subnets were created. Eventually, much of the slowness of the network was caused by the slow interconnections between these private pools of bandwidth.