696 Chapter 29 Web Services and have not yet migrated to a database-driven system. As the SA, it is your job to encourage such sites to move to a database-driven model as early as possible. At QPS-second rates, such a server must be scaled like any database, with the usual database performance-tuning tools that are available. 29.1.4.4 Multimedia Servers A multimedia server is primarily a web server that has content that in- cludes media files, such as video or audio. Media files are often very large and sometimes are accessed through some type of special client or browser to comply with digital rights management. When serving media files, the underlying data storage and network bandwidth capabilities become more important. Media servers provide streaming-media support. Typically, streaming me- dia is simply using a web application on the server to deliver the media file using a protocol other than HTTP, so that it can be viewed in real time. The server delivers a data stream to a special-purpose application. For ex- ample, you might listen to an Internet radio station with a custom player or a proprietary audio client. Often, one purpose of a media server application is to enforce copy protection or rights management. Another purpose is to control the delivery rate for the connection so that the data is displayed at the right speed, if the web site does not allow the end user to simply down- load the media file. The application will usually buffer a few seconds of data so that it can compensate for delays. Streaming-media servers also provide fast-forward and rewind functions. When operating a media server that is transmitting many simultaneous streams, it is important to consider the playback speed of the type of me- dia you are serving when choosing the storage and network capabilities. In Chapter 25, we mention some characteristics of storage arrays that are op- timized for dealing with very large files that are seldom updated. Consider memory and network bandwidth in particular, since complete download of a file can take a great deal of memory and other system resources. Streaming-media servers go through great lengths to not overwork the disk. If multiple people are viewing the same stream but started at different times, the system could read the same data repeatedly to provide the service, but you would rather avoid this. Some streaming applications will read an entire media file into memory and track individual connections to it, choosing which bits to send to which open connections. If only one user is streaming a file, keeping it in memory is not efficient, but for multiple users, it is. 29.1 The Basics 697 With this method, performance is superior to reading it from disk but can require a lot of memory. Fortunately, there are alternatives. Other implementations read in a fixed amount of the media file for each connection, sending the appropriate bits out. This can be very efficient, as most operating systems are good at caching data in memory. An hour- long video clip may be several gigabytes in size. But the entire file does not need to be in system memory at once, only the several megabytes that the application is sending next to each open connection. Customers who con- nected within a short time of one another will see good response, as their segments will still be cached and won’t need to be read from disk. This ap- proach gives quick response owing to cache hits but allows more efficient resource use. For any kind of streaming-media server, CPU speed is also an issue. Some- times, an audio or video file is stored at high quality and is reencoded at a lower resolution on demand, depending on the needs of the user requesting it. Doing this in real time is a very expensive operation, requiring large amounts of CPU time. In many cases, special-purpose hardware cards are used to per- form the processing, leaving the CPU less loaded and better able to do the remaining work of moving the data off the disk, through the card, and onto the network. ❖ LAMP and Other Industry Terms Certain technology combinations, or platforms, are common enough that they have been named. These platforms usually include an OS, a web server, a database, and the pro- gramming language used for dynamic content. The most common com- bination is LAMP: Linux, Apache, MySQL, and Perl. LAMP can also stand for Linux, Apache, MySQL, and PHP; and for Linux, Apache, MySQL, and Python. The benefit of naming a particular platform is that confusion is reduced when everyone can use one word to mean the same thing. 29.1.4.5 Multiple Servers on One Host There are two primary options for offering a separate server without re- quiring a separate machine. In the first method, the web server can be lo- cated on the very same machine but installed in a separate directory and configured to answer on a port other than the usual port 80. If config- ured on port 8001, for instance, the address of the web server would be http://my.web.site:8001/. On some systems on which high-numbered ports 698 Chapter 29 Web Services are not restricted to privileged or administrator use, using an alternative port can allow a group to maintain the web server on its own without needing privileged access. This can be very useful for an administrator who wishes to minimize privileged access outside the systems staff. A problem with this approach is that many users will simply forget to include the port number and become confused when the web site they see is not what they expected. Another option for locating multiple web sites on the same machine without using alternative ports is to have multiple network interfaces, each with its own IP address. Since network services on a machine can be bound to individual IP addresses, the sites can be maintained separately. Without adding extra hardware, most operating systems permit one physical network interface to pose as multiple virtual interfaces (VIFs), each with its own IP address. Any network services on the machine can be specifically bound to an individual VIF address and thus share the network interface without conflicts. If one defines VIFs such that each internal customer group or department has its own IP address on the shared host, a separate web installation in its own directory can be created for each group. A side benefit of this approach is that, although it is slightly more work in the beginning, it scales very nicely. Since each group’s server is configured separately and runs on its own IP address, individual groups can, be migrated to other machines with very little work if the original host machine becomes overloaded. The IP address is simply disabled on the original machine and enabled on the new host and the web services moved in its entirety, along with any start-up scripts residing in the operating system. 29.1.5 Monitoring Monitoring your web services lets you find out how well you are scaling, areas for improvement, and whether you are meeting your SLA. Chapter 22 covers most of the material you will need for monitoring web services. You may wish to add a few web-specific elements to your monitoring. Web server errors are most often related to problems with the site’s content and are often valuable for the web development team. Certain errors or pat- terns of repeating error can be an indication of customer problems with the site’s scripts. Other errors may indicate an intrusion attempt. Such scenarios are worth investigating further. Typically, web servers allow logging of the browser client type and of the URL of the page containing the link followed to your site (the referring URL). 29.1 The Basics 699 Some web servers may have server-specific information that would be useful as well, such as data on active threads and per thread memory usage. We encourage you to become familiar with any special support for extended monitoring available on your web server platform. 29.1.6 Scaling for Web Services Mike O’Dell, founder of the first ISP (UUNET) once said, “Scaling is the only problem on the Internet. Everything else is a sub-problem.” If your web server is successful, it will get overloaded by requests. You may have heard the phrase “the slashdot effect” or “they’ve been slash- dotted.” The phrase refers to a popular Internet news site with so many readers that any site mentioned in its articles often gets overloaded and fails to keep up with the requests. There are several methods of scaling. A small organization with basic needs could improve a web server’s performance by simply upgrading the CPU, the disks, the memory, and the network connection—individually or in combination. When multiple machines are involved, the two main types of scaling are horizontal and vertical. They get their names from web architecture diagrams. When drawing a representation of the web service cluster, the machines added for horizontal scaling tended to be in the same row, or level; for vertical scaling, in groups arranged vertically, as they follow a request flowing through different subsystems. 29.1.6.1 Horizontal Scaling In horizontal scaling, a web server or web service resource is replicated and the load divided among the replicated resources. An example is two web servers with the same content, each getting approximately half the requests. Incoming requests must be directed to different servers. One way to do this is to use round-robin DNS name server records. DNS is configured so that a request for the IP address of a single name (www.example.com) returns multiple IP addresses in a random order. The client typically uses only the first IP address received; thus, the load is balanced among the various replicas. This method has drawbacks. Some operating systems, or the browsers running in them, cache IP addresses, which defeats the purpose of the round- robin name service. This approach can also be a problem when a server fails, as the name service can continue to provide the nonfunctioning server’s ad- dress to incoming requests. For planned upgrades and maintenance, the server 700 Chapter 29 Web Services address is usually temporarily removed from the name service. The name record takes time to expire, and that time is controlled in DNS. For planned maintenance, the expire time can be reduced in advance, so that the deletion takes effect quickly. However, careful use of DNS expire times for planned downtime does not help with unexpected machine outages. It is better to have a way of choosing which server to provide for any given request. Having a hardware device to be a load balancer is a better solution than using DNS. A load balancer sits between the web browser and the servers. The browser connects to the IP address of the load balancer, which forwards the request transparently to one of the replicated servers. The load balancer tracks which servers are down and stops directing traffic to a host until it returns to service. Other refinements, such as routing requests to the least- busy server, can be implemented as well. Load balancers are often general-purpose protocol and traffic shapers, routing not only HTTP but also other protocol requests, as required. This allows much more flexibility in creating a web services architecture. Almost anything can be load balanced, and it can be an excellent way to improve both performance and reliability. One of Strata’s early web service projects seemed to be going well, but the messaging system component of it was prone to mysterious failures during long system tests. The problem seemed to be related to load balancing the LDAP directory lookups; when direct connects to the LDAP servers were allowed, the problem did not appear. Some careful debugging by the systems staff revealed that the load balancers would time out an idle connection without performing a certain kind of TCP closure operation on the pruned connection. The messaging server application did not reopen a new connection after the old one timed out, because the operating system was not releasing the connection. Fortunately, one of the SAs on another part of the project was familiar with this behavior and knew of the only two vendors (at the time) whose load-balancing switches implemented a TCP FIN when closing down a connection that timed out. The TCP FIN packet directs the machine to close the connection rather than wait for it to time out. The SAs changed the hardware, and the architecture worked as designed. Since then, the operating system vendor has fixed its TCP stack to allow closing a connection when in FIN WAIT for a certain time. Similar types of problems will arise in the future as protocols are extended and hardware changes. 29.1.6.2 Vertical Scaling Another way to scale is to separate out the various kinds of subservices used in creating a web page rather than duplicating a whole machine. Such 29.1 The Basics 701 vertical scaling allows you to create an architecture with finer granularity, to put more resources at the most intensively used stages of page creation. It also keeps different types of requests from competing for resources on the same system. A good example of this might be a site containing a number of large video clips and an application to fill out a brief survey about a video clip. Reading large video files from the same disk while trying to write many small database updates is not an efficient way to use a system. Most operating systems have caching algorithms that are automatically tuned for one or the other but perform badly when both happen. In this case, all the video clips might be put on a separate web server, perhaps one with a storage array customized for retrieving large files. The rest of the web site would remain on the original server. Now that the large video clips are on a separate server, the original server can handle many more requests. As you might guess, horizontal and vertical scaling can be combined. The video survey web site might need to add another video clip server before it would need to scale the survey form application. 29.1.6.3 Choosing a Scaling Method Your site may need horizontal or vertical scaling or some combination of both. To know which you need, classify the various components that are used with your web server according to the resources they use most heavily. Then look at which components compete with one another or whether one component interferes with the function of other components. A site may include static files, CGI progams, and a database. Static files can range from comparatively small documents to large multimedia files. CGI programs can be memory-intensive or CPU-intensive and can produce large amounts of output. Databases usually require the lion’s share of system resources. Use system diagnostics and logs to see what kinds of resources are being used by these components. In some cases, such as the video survey site, you might choose to move part of the service to another server. Another example is an IS department web server that is also being used to create graphs of system logs. This can be a very CPU-intensive process, so the graphing scripts and the log data can be moved to another machine, leaving the other scripts and data in place. A nice thing about scaling is that it can be done one piece at a time. You can improve overall performance with each iteration and don’t necessarily have to figure out your exact resource profile the first time you attempt it. 702 Chapter 29 Web Services It is tempting to optimize many parts at once. We recommend the op- posite. Determine the most overloaded component, and separate it out or replicate it. Then, if there is still a problem, repeat the process for the next overloaded component. Doing this one component at a time has better re- sults and makes testing much easier. It can also be easier to obtain budget for incremental improvements than for one large upgrade. 29.1.6.4 Scaling Challenges Scaling subsystems that rely on a common resource can be a challenge. If the web site contains applications that maintain state, such as which pages of a registration form you have already filled out, that state must be either maintained by the client browser or somehow made available to any of the systems that might handle the next request. This was a common issue for early load-balancing systems, and Strata remembers implementing a number of cumbersome network topology ar- chitectures to work around the problem. Modern load balancers can track virtual sessions between a client and a web server and can route additional traffic from that specific client to the correct web server. The methods for do- ing so are still being refined further, as many organizations are now hidden behind network address translation (NAT) gateways , or firewalls that make all requests look as though they originate from a single IP address. CGI programs or scripts that manipulate information often use a local lock file to control access. If multiple servers will be hosting these programs, it is best to modify the CGI program to use a database to store information. Then the database-locking routines can substitute for the lock file. Scaling database usage can be a challenge. A common scaling method is to buy a faster server, but that works only up to a point, and the price tags get very steep. The best way to scale database-driven sites tends to be to separate the data into read-only views and read-write views. The read-only views can be replicated into additional databases for use in building pages. When frequent write access to a database is required, it is best to structure the database so that the writes occur in different tables. Then one may scale by hosting specific tables on different servers for writing. Another problem presented by scale is that pages may need to pull data from several sources and use it in unified views. Database-replication prod- ucts, such as Relational Junction, allow the SA to replicate tables from dif- ferent types of databases, such as MySQL, Postgres, or Oracle, and combine them into views. We predict increased use of these types of tools as the need for scaling database access increases. 29.1 The Basics 703 ❖ The Importance of Scaling Everyone thinks that scaling isn’t impor- tant to them, until it is too late. The Florida Election Board web site had very little information on it and therefore very little traffic. Dur- ing the 2000 U.S. national elections, the site was overloaded by people who thought that they might find something useful there. Since the web site was on the same network as the entire department, the entire de- partment was unable to access the Internet because the connection was overloaded by people trying to find updates. In summary, here is the general progression of scaling a typical web site that serves static content and dynamic content and includes a database. Initially, these three components are on the same machine. As the workload grows, we typically move each of these functions to a separate machine. As each of these components becomes overloaded, it can be scaled individually. The static content is easy to replicate. Often, many static content servers receive their content from a large, scalable network storage device: NFS server or SAN. The dynamic content servers can be specialized and/or replicated. For example, the dynamic pages related to credit card processing are moved to a dedicated machine; the dynamic pages related to a particular application, such as displaying pages of a catalog, are moved to another dedicated machine. These machines can then each be upgraded or replicated to handle greater loads. The database can be scaled in similar ways: individual databases for specific, related data, each replicated as required to handle the workload. 29.1.7 Web Service Security Implementing security measures is a vital part of providing web services. Security is a problem because people you don’t know are accessing your server. Some people feel that security is not an issue for them, since they do not have confidential documents or access to financial information or similar sensitive data. However, the use of the web server itself and the bandwidth it can access are in fact a valuable commodity to some people. Intruders often break into hosts to use them for entertainment or money- making purposes. Intruders might not even deface or alter a web site, since doing so would lead quickly to discovery. Instead, the intruders simply use the resources. Common uses of hijacked sites and bandwidth include distributing pirated software (“warez”), generating advertising email (“spam”), launching automated systems to try to compromise other systems, and even competing with other intruders to see who can run the largest farm of machines to launch 704 Chapter 29 Web Services all the preceding (“bot” farms). (Bot farms are used to perform fee-for-service attacks and are increasingly common.) Even internal web services should be secured. Although you may trust employees of your organization, there are still several reasons to practice good web security internally. • Many viruses transmit themselves from machine to machine via email and then compromise internal servers. • Intranet sites may contain privileged information that requires authen- tication to view, such as human resources or finance information. • Most organizations have visitors—temps, contractors, vendors, interviewees—who may be able to access your web site via conference room network ports or with a laptop while on site. • If your network is compromised, whether by malicious intent or ac- cidentally by a well-meaning person setting up a wireless access point reachable from outside the building, you need to minimize the potential damage that could occur. • Some web security patches or configuration fixes also protect against accidental denial-of-service attacks that could occur and will make your web server more reliable. In addition to problems that can be caused on your web server by intru- sion attempts, a number of web-based intrusion techniques can reach your customers via their desktop browsers. We talk about these separately after discussing web server security. New security exploits are frequently discovered and announced, so the most important part of security is staying up to date on new threats. We discuss sources for such information in Chapter 11. 29.1.7.1 Secure Connections and Certificates Usually, web sites are accessed using unencrypted, plaintext communication. The privacy and authenticity of the transmission can be protected by encrypt- ing the communication, using the HTTP over Secure Sockets Layer (SSL) to encrypt the web traffic. 1 We do this to prevent casual eavesdropping on our customers’ web sessions even if they are connecting via a wireless network in 1. SSL 4.0 is also known as Transport Layer Security (TLS) 1.0; earlier versions SSL 2.0 and 3.0 predate TLS. 29.1 The Basics 705 a public place, such as a coffeeshop. URLs using https:// instead of http:// are using SSL encryption. Implementing HTTPS on a web server is relatively simple, depending on the web server software being deployed. Properly managing the cryptographic certificates is not so easy. SSL depends on cryptographic certificates, which are strings of bits used in the encryption process. A certificate has two parts: the private half and the public half. The public half can be revealed to anyone. In fact, it is given out to anyone who tries to connect to the server. The private part, however, must be kept secret. If it is leaked to outsiders, they can use it to pretend to be your site. Therefore, one role of the web system administrator is to maintain a repository, or key escrow, of certificates for disaster-recovery purposes. Treat this data like other important secrets, such as root or administrator passwords. One technique is to maintain them on a USB key drive in a locked box or safe, with explicit procedures for storing new keys, recovering keys, and so on. One dangerous place to store the private half is on the web server that is going to be using it. Web servers are generally at a higher exposure risk than others are. Storing an important bit of information on a machine that is most likely to be broken into is a bad idea. However, the web server needs to read the private key to use it. How can this conflict be resolved? Usually, the private key is stored on the machine that needs it in encrypted form. A password is required to read the key. This means that any time a web server that supports SSL is restarted, a human must be present to enter a password. At high-security sites, one might find it reasonable to have a person avail- able at all hours to enter a password. However, most sites set up various alternatives. The most popular is to store the password obfuscated—encoded so that someone reading over your shoulder couldn’t memorize it, such as storing it in base64—in a hidden and unique directory, so an intruder can’t find it by guessing the directory name. To retrieve the password, a helper program is run that reads the file and communicates the password to the web server. The program itself is protected so that it cannot be read—to find what directory it refers to and can be executed only by the exact ID that needs to be able to run it. This is riskier than having someone enter the password manually every time, but it is better than nothing. A cryptographic certificate is created by the web system administrator using software that comes with the encryption package; OpenSSL is one popular system. The certificate is now “self-signed,” which means that it [...]... Consider adding the missing skills through training or consultants before hiring new people When deciding on the size of the SA team, the management of the organization should take into account several factors: the number and variety of people and machines in the company the complexity of the environment, the type of work that the company does, the service levels required by various groups, and how mission-critical... escape The solution was to create a list of people allowed to move data from which systems and the automation to make the functions self-service to take the SAs out of the loop Small programs were created to push data to each stage, and permissions were set using the UNIX sudo command so that only the appropriate people could execute the particular commands Soon, the SAs had extricated themselves from the. .. from the popular Craigslist site A mashup application has two parts and therefore two parts to scale The first part is the portion of the mashup that its author wrote to glue together the services used The second part is the services that are used Usually, the glue is lightweight, and scaling is a matter of the techniques listed previously However, one should note that if the mashup is truly novel and. .. Initially, the SAs were involved in pushing the contents, from one step to the next This put them in the middle of the political bickering Someone would tell the SAs to push the current QA contents to the live site, then a mistake would be found in the 714 Chapter 29 Web Services contents, and everyone would blame the SAs They would be asked to push a single file to the live site to fix a problem, and the. .. software company growing by about 30 percent annually was trying to control costs, so it restricted the growth of the budget for the SA team The management of the SA team knew that the team was suffering and that there would be more problems in the future, but it needed a way to quantify this and to express it to the upper management of the company The SA group performed a study to determine where the. .. successful, and the CGI script is told the username that properly authenticated via an environment variable To be more flexible, most CGI-based applications have 29.2 The Icing 721 some kind of native authentication system that handles its own username and password system However, well-designed systems can be configured to switch that off and simply use the preauthenticated username supplied from the Apache... out on their web sites One of Strata’s friends was stuck for a long time in the uncomfortable position of being the only person with the ability to update the web server’s content At least once a month, sometimes more often, someone from the marketing department cornered this person on the way out of work at the end of the day with an “urgent” update that had to go up on the server ASAP Since the systems... well as between the SAs and their customers.1 Both sets of communication are key to the SA team’s success The SAs at a site need to cooperate in order to build a solid, coherent computing infrastructure for the rest of the company to use However, they must do so in a way that meets the needs of the customers and provides them with solid support and good customer service Management and individuals should... upset that they hadn’t been consulted Management tried to implement a system whereby the SAs would get signoff on whether the QA contents could be copied to the live site, but everyone wanted signoff, and it was a disaster: the next time the site was to be pushed, not everyone was around to do the signoff, and marketing went ballistic, blaming the SA team for not doing the push fast enough The SA team... interact with them regularly in the role of webmaster, you are creating demand for more interaction The best way for them to meet that demand is to hire another person for the webmaster role Being clear about the depth of specialization required for a good webmaster will help make sure that they don’t offer to make you the full-time webmaster instead and hire another SA to do your job 29.1 .8. 2 Change . the processing, leaving the CPU less loaded and better able to do the remaining work of moving the data off the disk, through the card, and onto the network. ❖ LAMP and Other Industry Terms Certain. after the old one timed out, because the operating system was not releasing the connection. Fortunately, one of the SAs on another part of the project was familiar with this behavior and knew of the. sits between the web browser and the servers. The browser connects to the IP address of the load balancer, which forwards the request transparently to one of the replicated servers. The load balancer tracks