14.2 The Icing 381 information in step 3, which makes further steps difficult and may require contacting the customer unnecessarily. A written chart of who is responsible for what, as well as a list of standard information to be collected for each classification, will reduce these problems. 14.2.2 Holistic Improvement In addition to focusing on improving each step, you may also focus on im- proving the entire process. Transitioning to each new step should be fluid. If the customer sees an abrupt, staccato handoff between steps, the process can appear amateurish or disjointed. Every handoff is an opportunity for mistakes and miscommunication. The fewer handoffs, the fewer opportunities there are for mistakes. A site small enough to have a single SA has zero opportunities for this class of error. However, as systems and networks grow and become more complicated, it becomes impossible for a single person to understand, main- tain, and run the entire network. As a system grows, handoffs become a necessary evil. This explains a common perception that larger SA groups are not as effective as smaller ones. Therefore, when growing an SA group, you should focus on maintaining high-quality handoffs. Or, you might choose to develop a single point of contact or customer advocate for an issue. That results in the customers’ seeing a single face for the duration of a problem. 14.2.3 Increased Customer Familiarity If a customer talks to the same person whenever calling for support, the SA will likely become familiar with the customer’s particular needs and be able to provide better service. There are ways to improve the chance of this happening. For example, SA staff subteams may be assigned to particular groups of customers rather than to the technology they support. Or, if the answering phone-staff is extremely large, the group may be using a telephone call center system, whereby customers call a single number and the call center routes the call to an available operator. Modern call center systems can route calls based on caller ID, using this functionality, for example, to route the call to the same operator the caller spoke to last time, if that person is available. This means there will be a tendency for customers to be speaking to the same person each time. It can be very comforting to speak to someone who recognizes your voice. 382 Chapter 14 Customer Care 14.2.4 Special Announcements for Major Outages During a major network outage, many customers may be trying to report problems. If customers report problems through an automatic phone response system (“Press 1 for , press 2 for ”), such a system can usually be pro- grammed to announce the network outage before listing the options. “Please note the network connection to Denver is currently experiencing trouble. Our service provider expects it to be fixed by 3 PM. Press 1 for press 2 for ” 14.2.5 Trend Analysis Spend some time each month looking for trends, and take action based on them. This does not have to be a complicated analysis, as this case study describes. Case Study: Who Generates the Most Tickets? At one site, we simply looked at which customers opened the most tickets in the last year. We found that 3 of the 600 people opened 10 percent of all tickets. That’s a lot! It was easy to visit each person’s manager to discuss how we could provide better service; if the person was generating so many tickets, we obviously weren’t matching the person’s needs. One person opened so many tickets because he was pestering the SAs for workarounds to the bugs in the old version of the L A T E X typesetting package that he was using and refused to upgrade to the latest version, which fixed most of the prob- lems he was reporting. This person’s manager agreed that the best solution would be for him to require his employee to adopt the latest L A T E X and took responsibility for seeing to it that the change was made. The next manager felt that his employee was asking basic questions and decided to send the customer for training to make him more self-sufficient. The last manager felt that his employee was justified in making so many requests. However, the manager did appreciate knowing how much the employee relied on us to get his job done. The employee did become more self-sufficient in future months. Here are some other trends to look for: • Does a customer report the same issue over and over? Why is it re- curring? Does the customer need training, or is that system really that broken? • Are there many questions in a particular category? Is that system difficult to use? Could it be redesigned or replaced, or could the documentation be improved? 14.2 The Icing 383 • Are many customers reporting the same issue? Can they all be notified at once? Should such problems receive higher priority? • Can some categories of requests become self-service? Often, a customer request is that an SA is needed because something requires privileged access, such as superuser or administrator access. Look for ways to em- power customers to help themselves. Many of these requests can become self-service with a little bit of web programming. The U NIX world has the concept of set user ID (SUID) programs, which, when properly admin- istered, permit regular users to run a program that performs privileged tasks but then lose the privileged access once the program is finished ex- ecuting. Individual SUID programs can give users the ability to perform a particular function, and SUID wrapper programs can be constructed that gain the enhanced privilege level, run a third-party program, and then reduce the privileges back to normal levels. Writing SUID programs is very tricky, and mistakes can turn into security holes. Systems such as sudo (Snyder, Miller et al. 1986) let you manage SUID privilege on a per user and per command basis and have been analyzed by enough security experts to be considered a relatively safe way to provide SUID access to regular users. • Who are your most frequent customers? Calculate which department generates the most tickets or who has the highest average tickets per member. Calculate which customers make up your top 20 percent of requests. Do these ratios match your funding model, or are certain cus- tomer groups more “expensive” than others? • Is a particular time-consuming request one of your frequent requests? If customers often accidentally delete files and you waste a lot of time each week restoring files from tape, you can invest time in helping the user learn about rm -i or use other safe-delete programs. Or, maybe it would be appropriate to advocate for the purchase of a system that sup- ports snapshots or lets users do their own restores. If you can generate a report of the number and frequency of restore requests, management can make a more informed decision or decide to talk to certain users about being more careful. This chapter does not discuss metrics, but a system of metrics grounded in this model might be the best way to detect areas needing improvement. The nine-step process can be instrumented easily to collect metrics. Developing metrics that drive the right behaviors is difficult. For example, if SAs are rated 384 Chapter 14 Customer Care by how quickly they close tickets, one might accidentally encourage the closer behavior described earlier. As SAs proactively prevent problems, reported problems will become more serious and time consuming. If average time to completion grows, does that mean that minor problems were eliminated or that SAs are slower at fixing all problems? 3 14.2.6 Customers Who Know the Process A better-educated customer is a better customer. Customers who understand the nine steps that will be followed can be better prepared when reporting the problem. These customers can provide more complete information when they call, because they understand the importance of complete information in solv- ing the problem. In gathering this information, they will have narrowed the focus of the problem report. They might have specific suggestions on how to reproduce the problem. They may have narrowed the problem down to a spe- cific machine or situation. Their additional preparation may even lead them to solve the problem on their own! Training for customers should include explaining the nine-step process to facilitate interaction between customers and SAs. Preparing Customers at the Department of Motor Vehicles Tom noticed that the New Jersey Department of Motor Vehicles had recently changed its “on hold” message to include what four documents should be on hand if the person was calling to renew a vehicle registration. Now, rather than waiting to speak to a person only to find out that you didn’t have, say, your insurance ID number, there was a better chance that once connected, you had everything needed to complete the transaction. 14.2.7 Architectural Decisions That Match the Process Architectural decisions may impede or aid the classification process. The more complicated a system is, the more difficult it can be to identify and duplicate the problem. Sadly, some well-accepted software design concepts, such as delineating a system into layers, are at odds with the nine-step process. For example, a printing problem in a large U NIX network could be a problem 3. Strata advocates SA-generated tickets for proactive fixes and planned projects, to make the SA contributions clearer. 14.3 Conclusion 385 with DNS, the server software, the client software, misconfigured user envi- ronment, the network, DHCP, the printer’s configuration, or even the printing hardware itself. Typically, many of those layers are maintained by separate groups of people. To diagnose the problem accurately requires the SAs to be experts in all those technologies or that the layers do cross-checking. You should keep in mind how a product will be supported when you are designing a system. The electronics industry has the concept of “design for manufacture”; we should think in terms of “design for support.” 14.3 Conclusion This chapter is about communication. The process helps us think about how we communicate with customers, and it gives us a base of terminology to use when discussing our work. All professionals have a base of terminology to use to effectively communicate with one an another. This chapter presents a formal, structured model for handling requests from customers. The process has four phases: greeting, problem identifica- tion, planning and execution, and fix and verify. Each phase has distinct steps, summarized in Table 14.1. Following this model makes the process more structured and formalized. Once it is in place, it exposes areas for improvement within your organization. Table 14.1 Overview of Phases for Problem Solution Phase Step Role Phase A: “Hello!” 1. The greeting Greeter Phase B: “What’s wrong?” 2. Problem classification Classifier 3. Problem statement Recorder 4. Problem verification Reproducer Phase C: “Fix it” 5. Solution proposals Subject matter expert 6. Solution selection 7. Execution Craft worker Phase D: “Verify it” 8. Craft verification 9. Customer verification Customer 386 Chapter 14 Customer Care You can integrate the model into training plans for SAs, as well as educate customers about the model so they can be better advocates for themselves. The model can be applied for the gathering of metrics. It enables trend analysis, even if only in simple, ad hoc ways, which is better than nothing. We cannot stress enough the importance of using helpdesk issue-tracking software rather than trying to remember the requests in your head, using scraps of paper, or relying on email boxes. Automation reduces the tedium of managing incoming requests and collecting statistics. Software that tracks tickets for you saves time in real ways. Tom once measured that a group of three SAs was spending an hour a day per person to track issues. That is a loss of 2 staff days per week! The process described in this chapter brings clarity to the issue of cus- tomer support by defining what steps must be followed for a single successful call for help. We show why these steps are to be followed and how each step prepares you for future steps. Although knowledge of the model can improve an SA’s effectiveness by leveling the playing field, it is not a panacea; nor is it a replacement for cre- ativity, experience, or having the right resources. The model does not replace the right training, the right tools, and the right support from management, but it must be part of a well-constructed helpdesk. Many SAs are naturally good at customer care and react negatively to structured techniques like this one. We’re happy for those who have found their own structure and use it with consistently great results. We’re sure it has many of the rudiments discussed here. Do what works for you. To grow the number of SAs in the field, more direct instruction will be required. For the millions of SAs who have not found the perfect structure for themselves, consider this structure a good starting point. Exercises 1. Are there times when you should not use the nine-step model? 2. What are the tools used in your environment for processing customer requests, and how do they fit into the nine-step model? Are there ways they could fit better? 3. What are all the ways to greet customers in your environment? What ways could you use but don’t? Why? 4. In your environment, you greet customers by various methods. How do the methods compare by cost, speed (faster completion), and customers’ Exercises 387 preference? Is the most expensive method the one that customers prefer the most? 5. Some problem statements can be stated concisely, such as the routing problem example in step 3. Dig into your trouble tracking system to find five typically reported problems. What is the shortest problem statement that completely describes the issue? 6. Query your ticket-tracking software and determine who were your top ten ticket creators overall in the last 12 months; then sort by customer group or department. Then determine what customer groups have the highest per customer ticket count. Which customers make up your top 20 percent? Now that you have this knowledge, what will you do? Examine other queries from Section 14.2.5. 7. Which is the most important of the nine steps? Justify your answer. This page intentionally left blank Part III Change Processes This page intentionally left blank [...]... baffled The network- monitoring tools showed that the deletions were not coming from the customer’s PC or from a rogue machine or misprogrammed server The SAs had done their best to debug the problem using their knowledge of their part of the system, yet the problem remained unsolved Suddenly, one of the senior SAs with end-to-end knowledge of the system, including both Windows and UNIX and all the various... certification 15. 2.3 End-to-End Understanding of the System Finally, the ultimate debugging icing is to have at least one person who understands, end to end, how the system works On a small system, that’s easy As systems grow larger and more complex, however, people specialize and end up knowing only their part of the system Having someone who knows the entire system is invaluable when there is a major... to make you conscious of the finer points of the process and then discuss some ways of making it even smoother We encourage you to be systematic It’s better than randomly poking about 15. 1 The Basics This section offers advice for correctly defining the problem, introduces two models for finding the problem, and ends with some philosophy about the qualities of the best tools 1 And while we’re at it,... different office The uplink from the hub had become disconnected from the rest of the network Without knowing how the tool performed its tests, there was no way to determine why a tool would report such a claim, and further debugging would have been a wild goose chase Luckily, there was a hint that something was suspicious—it didn’t mention network C The process of questioning the conclusion drew them to the. .. sure that they can ping each other If they can’t, it’s a network problem, and traceroute can isolate the problem If ping succeeded, connectivity is good, and there must be a protocol problem From the client, the elements of the NFS protocol can be tested with rpcinfo.2 You can test the portmap traceroute function, then mountd, nfs, nlockmgr, and status If any of them fail, you can deduce that the appropriate... /.profile from scratch, he should copy one from another Solaris host Fix the problem once, Corollary B: Leverage what others have done; don’t reinvent the wheel By copying the generic /.profile that was on most other hosts in that lab, the SA was leveraging the effort put into the previous hosts He was also reversing the entropy of the system, taking one machine that was dissimilar from the others and. .. variables; then he would set the required variables and retype the failed command Finally, Tom politely suggested that if the SA had a /.profile that set those variables, he could focus more on the problem rather than on his environment Fix the problem once, Corollary A: Fix the problem permanently The SA agreed and started creating the host’s /.profile from scratch Tom stopped him and reminded him that rather... didn’t need to be rebooted so often in the first place 15. 1.3 Be Systematic It is important to be methodical, or systematic, about finding the cause and fixing it To be systematic, you must form hypotheses, test them, note the results, and make changes based on those results Anything else is simply making random changes until the problem goes away The process of elimination and successive refinement are... dissimilar from the others and making it the same again As the SA copied the /.profile from another machine, Tom questioned why they were doing this at all Shouldn’t Solaris JumpStart have already installed the fine /.profile that all the other machines have? In Chapter 3, we saw the benefits of automating the three big deployment issues: loading the OS, patching the OS, and network configuration This environment... in that time was 3 millilightseconds, or about 55 8 miles 15. 3 Conclusion Every SA debugs problems and typically develops a mental catalog of standard solutions to common problems However, debugging should be a systematic, or methodical, process that involves understanding what the customer is trying to do and fixing the root cause of the problem, rather than smoothing over the symptoms Some debugging . con- scious of the finer points of the process and then discuss some ways of making it even smoother. We encourage you to be systematic. It’s better than randomly poking about. 15. 1 The Basics This. this class of error. However, as systems and networks grow and become more complicated, it becomes impossible for a single person to understand, main- tain, and run the entire network. As a system. in the first place. 15. 1.3 Be Systematic It is important to be methodical, or systematic, about finding the cause and fixing it. To be systematic, you must form hypotheses, test them, note the results,