ptg5994185 34 CHAPTER 2ROLES FOR THE SCALABLE TECHNOLOGY ORGANIZATION 100% consistent with the design. For instance, if a design calls for a configurable number (max number undefined) of similarly configured read databases, to which all read transactions can be evenly distributed, and the software engineer implements a system capable of handling up to five read databases, he or she has implemented a system with a defined scale limit. Here, defined scale limit is the limitation the engi- neer put on how many databases can be implemented (five). A software engineer should understand the portion of the system that she sup- ports, maintains, or for which she creates code. He should also understand the end user and how the end user interacts with the software engineer’s portion of the sys- tem. The software engineer is a contributor to many of the scalability processes we define later in Part II. Operator The operator is responsible for handling the daily operations of the production sys- tem, whether that system is a Web 2.0 system or a back office IT system. She is responsible for monitoring the system against specific service levels, monitoring for out of bounds conditions, alerting individuals based on service level or boundary condition failures, and tracking incidents to closure. A form of operator, sometimes called an incident manager, is responsible for managing major problems to closure and issuing root cause and corrective action reports. Infrastructure Engineer Infrastructure engineer is a generic term used to identify database administrators, network engineers, and systems administration professionals. The infrastructure engineer is responsible for the selection, configuration, implementation, tuning, and proper functioning of the devices or systems under his purview. The infrastructure engineer, more than any other role, is responsible for the scal- ability of the systems that he supports. As such, a database analyst is responsible for identifying early when his database is going to fail based on capacity constraints and to identify potential opportunities for scaling. A systems analyst is expected to do the same for her systems and storage and a network engineer for the portions of the net- work that she supports. In addition to having a deep technical understanding of his specific discipline, a skilled infrastructure engineer should understand the product he helps to support, be conversant in the “sister” disciplines within the hardware and systems community (a great systems administrator for instance should have a basic understanding of net- works and a good understanding of how to troubleshoot basic database problems) in order to aid in troubleshooting, a good knowledge of competing technologies to those employed in his product or platform, and a good understanding of emerging technologies within his field. The infrastructure engineer should also understand the ptg5994185 AN ORGANIZATIONAL EXAMPLE 35 cost of operating his system and the opportunities to reduce that cost overtime. Finally, the best infrastructure engineers are agnostic to the technologies they employ, a point we will cover in Chapter 20, Designing for Any Technology. QA Engineer/Analyst The QA engineer or analyst is responsible for testing the application and the systems infrastructure to ensure that it meets the product specifications. A portion of her time should be dedicated to performance testing as it relates to scalability and as defined in Chapter 17, Performance and Stress Testing. Capacity Planner We’ve discussed the role and activity of the capacity planner in earlier sections of this chapter. Put simply, the capacity planner is responsible for matching the expected demand (typically generated by the business unit) to the current system as imple- mented to determine where additional changes need to be made in the system, plat- form, or product. The capacity planner is not responsible for defining what these changes are; rather, she outlines where changes need to occur. In the case where a change needs to be made to a system that scales horizontally, the capacity planner may have as part of her job description the responsibility to help kick off the purchase order process to bring in new equipment. More often than not, the capacity planner is also a critical part of the process of budgeting for new systems and new initiatives to meet the business forecasted demand. An Organizational Example The new CEO of AllScale analyzes her team over the first 90 days. The company has had a number of scalability related incidents with its flagship HRM product and Christine determines that the current CTO (in AllScale’s case, the CTO is the highest technology management position in the company) simply isn’t capable of handling the development of new functionality and the stabilization of the existing platform. Christine believes that one of the issues with the executive previously in charge of technology was that he really had no business acumen and could not properly explain the need for certain purchases or projects in business terms. The former CTO simply did not understand simple business concepts like returns on investment and discounted cash flow. Furthermore, he always expected the business folks to under- stand the need for any of what business peers believed were his pet projects and would simply say, “We either do this or we will die.” Although the technology team’s budget was nearly 20% of the company’s $200 million in revenue, systems still failed ptg5994185 36 CHAPTER 2ROLES FOR THE SCALABLE TECHNOLOGY ORGANIZATION and the old CTO would blame unfunded projects for outages and then blame the business people for not understanding technology. The CEO sits down with her new CTO, a person she picked from an array of can- didates with graduate degrees in both business and electrical engineering or computer science, and explains that while she will delegate the responsibility for technical deci- sions to the CTO and empower him to make decisions within his budget limitations, she will not and cannot delegate the accountability for his results. She explains that she wants to create a culture of scalability in the company along the lines of the old manufacturing mottos of “everyone is accountable for quality.” She will work to add a nod toward scalability in the corporate vision and add a corporate belief surround- ing the need to cost effectively scale to customer demands without quality of service or availability (a.k.a. downtime) problems. The new CTO, Johnny Fixer, asks for 30 days to review the organization, identify, and put in motion some quick win projects and report back with a plan to make the technology platform, organization, and processes highly scalable and highly avail- able. He promises to keep Christine informed and communicate the issues he finds and concerns he has. They agree to talk daily on the phone, exchange emails more often, and meet personally at least once a week. Johnny quickly identifies overlaps in jobs in certain areas and responsibilities that are completely missing from his team. For instance, no one is responsible for develop- ing a single cohesive capacity plan. Furthermore, teams do not work together to col- laborate on designs, architects are not engaged with the engineering teams and do not understand the current status of customer grief with the product, and quality defects are blamed on a QA team with no engineering ownership of bugs. Johnny works quickly to hire a capacity planner onto his team. As it is May and the company’s budgeting for the next year must be complete by October, he knows he must get good data about current system performance relative to peak theoretical capacity and start to get next year’s demand projections from the business to help the CFO create his next fiscal year budget. The newly hired capacity planner starts work- ing with the engineering team to install the appropriate monitoring systems to collect system data in order to identify capacity bottle necks and she works with finance to understand both the current budget and to help provide information to generate the next year’s budget. Although the CTO is worried about all of his technology problems, he knows that long term he is going to have to focus his teams on how they can work together and create shareholder value. He implements a tool for defining roles and responsibilities called RASCI for Responsible, Accountable, Supportive, Consulted, and Informed (this tool is defined further in the next section) and implements Joint Architecture Design and the Architecture Review Board (defined in Chapters 13 and 14) to help resolve the lack of cooperation between organizations. ptg5994185 A TOOL FOR DEFINING RESPONSIBILITIES 37 Johnny walks through the past 30 days of issues and identifies that the team is not keeping track of outages, incidents, and their associated impact to the business. He makes his head of technical operations responsible for all outage tracking and indi- cates that together they will review all issues daily and track them to closure. He fur- ther requires that all architects attend at least one of the daily operations meetings per month to help get them closer to the customer and to better understand the pains associated with the current system. While meeting with his engineering managers, Johnny indicates that all bugs will be considered engineering and QA failures rather than just QA failure and that the company will begin tracking defects (or bugs) per feature produced with a goal to reducing all failures. To help align his teams to the need for a more reliable and available site, Johnny implements a site uptime or availability metric and a goal to achieve greater than 99.99% availability by month within the next four months. With the CEO’s advice and permission, and with the help of his architects, engineers, and infrastructure engi- neers, he reprioritizes some projects to attack the site outage incidents that appear (given the small amount of data) to have caused the most grief to the company. Johnny then implements a governance council for all engineering projects consist- ing of the CEO, the CFO, and all of the business unit leaders. The council is respon- sible for prioritizing projects, including availability projects, and for additionally measuring their returns against the promised success and business metrics upon which they were based. After the first 30 days, Johnny covers his 30-, 60-, and 90-day forward plans with the CEO and they jointly agree on a vision and set of goals for the engineering team (see Chapter 4, Leadership 101). Christine then has an “all hands” meeting with the entire company explaining that scalability and availability of the platform are of the utmost priority and that it is “everyone’s job” to help ensure that the company and its services scale to meet customer demands. To help incent the company toward an appropriate culture that includes the notion of being “highly scalable,” she insists that all managers have as part of their bonus compensation a scalability related goal that represents no less than 5% of their bonus. She delegates the development of those goals to her subordinates and asks to review them in the next 30 days. A Tool for Defining Responsibilities Many of our clients use a simple tool to help them define role clarity for any given initiative. Often when we are brought in to help with scalability in a company, we employ this tool to define who should do what, and to help eliminate wasted work and ensure complete coverage of all scalability related needs. Although technically a process, as this is a chapter on roles and responsibility, we felt compelled to include this tool here. ptg5994185 38 CHAPTER 2ROLES FOR THE SCALABLE TECHNOLOGY ORGANIZATION The tool we most often use is called RASCI. It is a responsibility assignment chart and the acronym stands for Responsible, Accountable, Supportive, Consulted, and Informed. • R stands for Responsible. This is the person responsible for completing the project or initiative. • A stands for Accountable. This is the person to whom R is accountable and who must approve the work before it is okay to complete. The A is sometimes referred to as the approver of any initiative. • S stands for Supportive. These people provide resources to complete the project or initiative. • C stands for Consulted. These people have data or information that can be use- ful in completing the project. • I stands for Informed. These people should be notified, but do not need to be consulted or provide input to the project. RASCI can be used in a matrix, where each activity or initiative is spelled out along the y or vertical axis of the matrix and the individual contributors or organizations are spelled out on the x-axis of the matrix. The intersection of the activity (y-axis) and the organization (x-axis) contains one of the letters R, A, S, C, or I and may include nothing if that individual or organization is not part of the initiative. Ideally, in any case, there will be a single R and a single A for any given initiative. This helps eliminate the issue we identified earlier in this chapter of having multiple organizations or individuals feeling that they are responsible for any given initiative. By having a single person or organization responsible, you are abiding by the “one back to pat and one throat to choke” rule. A gentler way of saying this is that distrib- uted ownership is ownership by no one. This is not to say that others should not be allowed to provide input to the project or initiative. The RASCI model clearly allows and enforces the use of consultants or people within and outside your company who might add value to the initiative. An A should not sign off on an R’s approach until such time as the R has actually consulted with all of the appropriate people to develop the right course of action. And of course if the company has the right culture, not only is the R going to want to seek those people’s help, but the R is going to make them feel as if their input is valued and value added to the decision making process. You can add as many Cs, Ss, and Is as you would like and as add value or are needed to complete any given project. That said, protect against going overboard regarding who exactly you will inform. Remember our discussion in the previous chapter about people being bogged down with email and communication that does not concern them. It is common in young companies to allow everyone to feel that ptg5994185 A TOOL FOR DEFINING RESPONSIBILITIES 39 they should be involved in every decision or informed of every decision. This infor- mation distribution mechanism simply does not scale and results in people reading emails rather than doing what they should be doing to create shareholder value. A partially filled out example matrix is included in Table 2.1. Taking some of our discussion thus far regarding different roles, let’s see how we’ve begun to fill out this RASCI matrix. We earlier indicated that the CEO absolutely must be responsible for the culture of scalability, or the scale DNA of the company. Although it is theoretically possible for her to delegate this responsibility to someone else within the company from a practi- cal perspective, and as you will see in the chapter on leadership, she must live and walk the values associated with scaling the company and its platform. As such, even with delegation and as we are talking about how the company “acts” with respect to scale, the CEO absolutely must “own” this. Therefore, we have placed an R in the CEO’s column next to the Scalability Culture initiative row. The CEO is obviously responsible to the board of directors and, as the creation of scale culture has to do with overall culture creation, we have indicated that the board of directors is the A. Table 2.1 RASCI Matrix CEO Business Owner CTO CFO Arch Eng Ops Inf QA Board of Directors Scalability Culture RA Technical Scalability Vision A C RCSSSSS I Product Scale Design AR Software Scale Implementation ARS Hardware Scale Implementation ASR Database Scale Implementation ASR Scalability Implementation Validation AR ptg5994185 40 CHAPTER 2ROLES FOR THE SCALABLE TECHNOLOGY ORGANIZATION Who are the Ss of the Scalability Culture initiative? Who should be informed and who needs to be consulted? In developing your answer to this question, you are allowed to have people who are Ss of any situation also be Cs in the development of the solution. It is implied that Cs and Ss will be informed as a result of their jobs, so it is generally not necessary to include an I any place that you feel you need to com- municate a decision and a result. We’ve also completely filled out the row for Technical Scalability Vision. Here, as we’ve previously indicated, the CTO is responsible for developing the vision for scal- ability for the product/platform/system. The CTO’s boss is very likely the CEO, so she will be responsible for approving the decision or course. Note that it is not abso- lutely necessary that the R’s boss be the A in any given decision. It is entirely possible that the R will be performing actions on behalf of someone for whom he or she does not work. In this case, however, assuming that the CTO works for the CEO, there is very little chance that the CTO would actually have someone other than the CEO approve his or her scalability vision or scalability plan. Consultants to the scalability vision are the CTO’s peers—the people who rely on the CTO for either the availability of the product or the back office systems that run the company. These people need to be consulted because the systems that the CTO creates and runs are the lifeblood of the business units and the heart of the back office systems that the CFO needs to do his or her job. We have indicated that the CTO’s organizations (Architecture group, Engineering team, Operations team, Infrastructure team, and QA) are all supporters of the vision, but one or more of them could also be consultants to the solution. The less technical the CTO, the more he will need to rely upon his teams to develop the vision for scal- ability. Here, we have assumed that the CTO has the greatest technical experience on the team, which is obviously not always the case. The CTO may also want to bring in outside help in determining the scalability vision and/or plan. This outside help may be a retained advisory services firm or potentially the establishment of a technology advisory and governance board that provides for the technology team the same gov- ernance and oversight that a board of directors provides at a corporate level. Finally, we have indicated that the board of directors needs to be Informed of the scalability vision. This might be a footnote in a board meeting or a discussion around what is possible with the current platform and how the company will need to invest to meet the scalability objectives for the coming years. The remainder of the matrix has been partially filled out. Important points with respect to the matrix are that we have split up the tasks/initiatives to try to ensure that there aren’t any overlaps in the R category. For instance, the responsibility for infrastructure tasks has been split from the responsibility for software development or architecture and design tasks. This allows for clear responsibility in line with our “one back to pat and one throat to choke” philosophy. In so doing, however, the ptg5994185 CONCLUSION 41 organization might tend to move toward designing in a silo or vacuum, which is counter to what you would like to have long term. Should you structure your organi- zation in a similar fashion, it is very important that you implement processes that require teams to design together to create the best possible solution. Matrix orga- nized teams do not suffer from some of the silo mentality that exists within teams built in silos around functions or organizational responsibility, but they can still ben- efit from RASCI. You should still have a single responsible organization; but you want to ensure that collaboration happens. RASCI helps enforce that through the use of the C attribute. Please spend time working through the rest of the matrix in Table 2.1 to get com- fortable with the RASCI model. It is a very effective tool in clearly defining roles and responsibilities and can help eliminate duplicated work, unfortunate morale-deflating fights, and missed work assignments. Conclusion Providing role clarity is the responsibility of leaders and managers. Individuals as well as organizations need role clarity. We provided some examples of how roles might be clearly defined to help in the organization’s mission of attaining higher availability. We also argued that these are but one of many examples that might be created regarding individuals and organizations and their roles. The real answer for you may vary significantly as the roles should be developed consistent with company culture and need. In attempting to create role clarity, attempt to stay away from over- lapping responsibilities, as these can create wasted effort and value-destroying con- flicts. Also attempt to ensure that no areas are missing, as these will result in failures. We also introduced a tool called RASCI to help define roles and responsibilities within the organization. Feel free to use RASCI for your own organizational roles and for roles within initiatives. The use of RASCI can help eliminate duplicated work and make your organization more effective, efficient, and scalable. Key Points • Role clarity is critical for scale initiatives to be successful. • Overlapping responsibility creates wasted effort and value-destroying conflicts. • Areas missing responsibility create vacuums of activity and failed scale initiatives. • The CEO is the chief scalability officer of the company. • The CTO/CIO is the chief technical scale officer of the company. • Key scale related responsibilities for any organization include ptg5994185 42 CHAPTER 2ROLES FOR THE SCALABLE TECHNOLOGY ORGANIZATION q Creation of the scalability vision for the organization q Setting measurable scale related goals q Staffing the team with the appropriate skill sets necessary to meet the scalabil- ity objectives q Defining a scalable architecture q Implementing that architecture in systems and code q Testing the implementation against current and future user demand q Gathering data on current platform and product utilization to determine immediate needs for scale q Developing future demand projections and converting that demand projection into meaningful system demand q Analyzing the demand projections against the system to determine where changes are needed q Defining future changes based on the analysis q Developing processes to determine when and where systems will break and prioritizing fixes for those issues • RASCI is a tool that can help eliminate overlaps in responsibility and create clear role definition. RASCI is developed in a matrix in which q R stands for the person Responsible for deciding what to do and running tive. q A is Accountable or the Approver of the initiative and the results. q S stands for Supportive, referring to anyone providing services to accomplish the initiative. q C stands for those who should be Consulted before making a decision and regarding the results of the initiative. q I stands for those who should be Informed of both the decision and the results. ptg5994185 43 Chapter 3 Designing Organizations Management of many is the same as management of few. It is a matter of organization. —Sun Tzu In the past two chapters, we have discussed how important it is to establish the right roles within your team and to get the right people in those roles. This hopefully makes perfect sense and is in line with everything you believe: accomplishing great things starts with finding great people and getting them in the right roles. Now we have come to the organizational structure, and you are probably wondering why this has anything to do with the successful scaling of your application. The answer lies in what factors are affected by an organizational structure and in turn how important those factors are to the scalability of your application. This chapter will highlight the factors that an organizational structure can influ- ence and show how those are also key factors in an application’s or Web service’s scalability. There are two determinants of an organization: team size and team struc- ture. If you are given the size range of the teams and how the teams are related to each other, you have a clear description of the organization. These two descriptors, size and structure, will be covered in this chapter, providing insight into the various dimensions of each, large versus small team sizes and silo versus matrix structures. Organizational Influences That Affect Scalability The most important factors that the organizational structure can affect are communi- cation, efficiency, standards, quality, and ownership. Let’s take each factor and exam- ine how the organization can influence it as well as why that factor is also important to scalability. Thus, we can establish a causal relationship between the organization and scalability. As we will see in Part II, Building Processes for Scale, which deals with processes, communication is central to all processes. Failed organizational communication is a [...]... Mgr 2 Eng 2 Eng 2 QA Eng 2 QA Eng 2 PM 2 PM 2 Eng 3 QA Eng 3 Project Mgr 3 Figure 3 .2 Matrix Organization Chart PM 3 O RGANIZATIONAL S TRUCTURE for the assignment of tasks and the timeline In larger and more complex matrix organizations, many members of each team can belong to project teams Continuing with the project team responsible for implementing the new billing feature, we can start to realize the. .. organization This includes the separation of employees into departments, divisions, and teams as well as the management hierarchy that is used for command and control of the forces There are as many different structures as there are companies, but there are two basic structures from which everything else stems These structures are functional and matrix By understanding the pros and cons of each structure,... bosses Each of these two bosses may have different managerial responsibilities for instance, one (perhaps the team leader) handles administrative tasks and reviews, whereas the other (perhaps the project manager) handles the assignment of tasks and project status In Figure 3 .2, the traditional functional organization is augmented with a project management team on the side The right side of the organization... choose one or the other; or, perhaps a more likely scenario, create a hybrid of the two structures that best meets the needs of your company In the ensuing paragraphs, we will cover the basic definition of each structure, the benefits and drawbacks of each, and some ideas on when to use each one Recognize that the most important lesson is how to choose parts of one versus the other to create the organizational... concept in the context of the drawbacks of matrix organizations If we have solved or at least dramatically improved the drawbacks of the functional organizational structure through the implementation of the matrix, surely there is a cost for this improvement The truth is that while solving the problems of project ownership and communication, we introduce other problems involving multiple bosses and distraction... reliance and strengthening of your strengths and the mitigation of your weaknesses Few people fail in their objectives because of their strengths and few people win as a result of their weaknesses We must reduce the dilutive aspects of our leadership by mitigating weaknesses and increase our accretive aspects by multiplying and building upon our strengths Having discussed a model for leadership, and the. .. our teams when they were either too large or too small Some of these symptoms of course can be caused by other things besides team size but it is often the case that managers jump to conclusions with the most cursory of investigations into the real cause and often do not even consider the organizational structure as a possible cause Often, the most frequently blamed is the team leader and his inability... what happens and rarely are we able to solve a problem without consequences of another variety The next question, given these pros and cons of the matrix organization, is “when would we want to use such an organizational structure?” The appropriate times to use the matrix structure are when the fate of the project is extremely important either because of the timeline or other such factor In these cases,... peer departments, but the overwhelming majority of effort and tasks will be within the engineering discipline Decisions about how to split the database, how the application will handle the lookup, and all other similar decisions will fall to the engineering team The product management team may be requested to provide information about specific customer behavior or the cost to the business for changes... engineer and set up alerts through the source code repository that can alert each other if anything is changed in that particular file or class, therefore everyone can be aware of changes in their sections of the code The next item to consider is who will be the new manager This is an opportunity to hire someone new from the outside or promote someone internally into the position There are pros and cons of . also understand the end user and how the end user interacts with the software engineer’s portion of the sys- tem. The software engineer is a contributor to many of the scalability processes we define. out along the y or vertical axis of the matrix and the individual contributors or organizations are spelled out on the x-axis of the matrix. The intersection of the activity (y-axis) and the organization. that the CTO creates and runs are the lifeblood of the business units and the heart of the back office systems that the CFO needs to do his or her job. We have indicated that the CTO’s organizations