PART Three Application Performance Tools 16 433012 PP03.qxd 6/16/03 9:11 AM Page 239 16 433012 PP03.qxd 6/16/03 9:11 AM Page 240 241 Part III of this book approaches network performance from a different per- spective. Not only is it crucial to have the network configured for maximum performance, it also helps to have network applications that are configured to operate efficiently across the network. This part of the book describes some methods and tools that can be used to help determine how an application will behave within your network environment. By modeling the network, you can observe how network applications behave within the model environment, and extrapolate from the results an idea of how the application would behave in the production network environment. This chapter provides an overview of the methods used for modeling production networks and using those models to test network applications. Trying to determine how an application will perform on a network is often a task that falls to the network administrator. While programmers work to make network applications functional for their customers, they sometimes for- get to consider how the application will affect the existing production net- work, or conversely, how the existing production network will affect the application’s performance. Often, customers, programmers, and network administrators do not find out that a new network application is the cause of network performance issues (or again, that the network is responsible for net- work application performance issues) until it is too late. Measuring Application Performance CHAPTER 13 17 433012 Ch13.qxd 6/16/03 9:11 AM Page 241 This chapter begins by introducing the concept of network modeling, by showing the different methods used for testing network applications. All of the network models described are ways that programmers and network administrators have used to determine how a network application might per- form within the production network environment. Methods of Testing Network Applications The point of testing network applications is to observe how an application will perform when run on a network. In today’s world of “do everything on the network,” each network application is competing for network resources with lots of other applications, from bulk file transfers to Web browsing. The key to good network application performance is to determine how an individual application will behave in your particular network environment. There are four basic methods that can be used for testing network applications: ■■ Using a test network ■■ Using the production network ■■ Using a network emulator ■■ Using a network simulator Each of these testing environments has a unique set of benefits and challenges for programmers and network administrators. This section walks through each of the test environments, and describes the pros and cons of each. The Test Network Creating a test network is the most common method used in programming shops. It is usually the quickest and easiest solution for testing network appli- cations. Unfortunately, it can also produce the least useful results. It is not uncommon for a network application to perform flawlessly within the test net- work, but fail miserably when placed in the production network environment. Most test networks fail to take into consideration the problems associated with the actual production network. Usually, the test network is nothing more than a group of workstations con- nected together on a standalone network where no other applications are run- ning. This can create a false sense of security for the application developers, as real network problems are not addressed in the application. To create a useful test network, the testers should attempt to mimic the network problems of the production network (discussed later in the Modeling Network Problems section). 242 Chapter 13 17 433012 Ch13.qxd 6/16/03 9:11 AM Page 242 Production Network The most accurate method to use for testing network applications is to run the application in the production network environment. This ensures that all of the network factors present in the production network will affect the applica- tion, as in real life, which makes it the best method of determining if the net- work will affect application performance, or if the application will affect network performance. Of course, there are downsides to testing on the pro- duction network. Often it is not feasible for the application to be developed on the production network. Often, application developers are not located in the same area where the application will be run, making it impossible for them to use the produc- tion network for testing. This is the case for almost all commercial network applications used. However, even with in-house programming, the program- mers often do not have access to the application area where the program will be run. Even when the application developers do have access to the production net- work, it is not always a good idea to test the application on the network. There have been incidents of runaway applications that consumed the entire band- width of a production network, effectively crashing the network for other pro- duction data. In environments where high availability is a necessity, this could cause catastrophic results. Network Emulation The compromise between the test and production network is the network emu- lation. A network emulation is a canned environment, where all of the prob- lems and issues found in the production network are duplicated within a test network environment. Creating a standard test network, but including one or more devices that are capable of emulating the behavior and characteristics of the production network, will accomplish this. The key to network emulation is to have devices that can introduce network problems into the test network environment. There are two common methods that are used for this purpose: ■■ Have network devices that produce network traffic, emulating production traffic on the test network. ■■ Have network devices that can accept packets, and delay, drop, or misorder them, as in the production environment. The following sections describe how these two methods are used. Measuring Application Performance 243 17 433012 Ch13.qxd 6/16/03 9:11 AM Page 243 Network Traffic Generator The first method consists of network devices that can produce network traffic that mimics the traffic found in the production network. Just as the netperf net- work performance tool was capable of sending test data streams similar to application traffic, a network traffic generator can send traffic emulating any- thing from FTP data transfers and interactive Telnet sessions to database access and Web browsing. Many advanced network emulation devices also include options to combine different types and amounts of traffic generated, such as emulating 10 FTP users and 100 Web browser users simultaneously. By adding one or more network traffic generators to a simple test network, you can more accurately observe how the network application will perform given other network traffic. While this is still not a complete emulation of the production environment, it gets the tester closer to the desired results. Not only can network traffic generators be used to test network applica- tions, they are also used to test the performance of network devices. Many network administrators use network traffic generators to simulate normal net- work traffic for switches, routers, and WAN links. By simulating the actual traffic that could be present during normal production times, you can observe the behavior of the network devices before customers complain. Network Emulation Device Instead of generating network traffic for the test network, the second type of network emulation device provides a way to model an entire production net- work within one or more devices. As packets enter the network emulator, they are processed to simulate network problems that could be present on the pro- duction network. The network emulation device plugs in between the client and server devices testing the network application. For packets to pass from the client to the server, they must pass through the network emulator. The emulator is configured to process each packet in some manner, depending on the type of network emulated, before it is passed to the server. The idea is to inflict the same network problems on each packet sent as would be seen on the production network. To accomplish this, most network emulators create one or more pipelines between the device input and output. As packets are received on the input, they are fed through the pipelines on their way to the output. Within each pipeline, the packets are subjected to different delays, errors, and even drops before they are sent to the device output. This is demonstrated in Figure 13.1. 244 Chapter 13 17 433012 Ch13.qxd 6/16/03 9:11 AM Page 244 Figure 13.1 Network emulation within the pipeline. The network emulator sends packets between the different pipelines ran- domly, so each of the network effects is distributed among the incoming pack- ets. The pipeline can perform the network emulation using either hardware or software solutions. Often the pipelines are implemented internal network functions within the emulation device, such as interprocess communication (IPC) on Unix systems. Network Simulation Network simulators perform network-modeling functions completely within a software environment. Mathematical algorithms are used to model the behavior of each network link and device, and also represent the data pro- duced and consumed by the network application. The way the mathematical algorithms work on the input data is similar to the way the actual production network would affect the data, using delays, drops, and out-of-order packets. Each device within the network path is modeled, with the individual models connected together within the simulation. The network model connects the hubs, links, and router models together to produce a single network model. The output from one device model is fed into the input of another device model, representing the various aspects of the net- work connections. Using this technique, any type of network can be modeled in the network simulation environment. Modeling application data can be tricky. Just like the network traffic gener- ators, network simulators must simulate different types of network traffic. The only difference is that the simulators do it mathematically, without using real packets. By sending data streams that represent real traffic, the network simu- lation can produce results showing how the output of each network device would look. client server delay drop packet Error injector Network 1 Network 2 Network Emulator Measuring Application Performance 245 17 433012 Ch13.qxd 6/16/03 9:11 AM Page 245 A network simulator does not have to process information at the same speed as the modeled network (real-time processing). The purpose of the network simulator is not to simulate real-time data, but to produce results that indicate what the overall performance of the network application would be, given the modeled network. There are basically two types of network simulators: ■■ Discrete event ■■ Analytical Discrete Event Discrete event simulators perform calculations on individually simulated packets, just as a network would see each packet of a data stream and process it. Each network device and each data packet in the data stream is simulated. This produces the most realistic information, modeling the behavior of the net- work as data streams are handled. Unfortunately, this simulation method is computationally intensive, requir- ing lots of calculations for each packet within the data stream and each net- work device handling the packet along the network path. To simulate large quantities of network traffic within large networks, discrete event simulators often have to perform calculations for hours, and even days, before producing results. Analytical Analytical simulators attempt to decrease the calculation times by using math- ematical equations to estimate the behavior of different network devices and the way they would handle particular types of traffic. Each network device is simulated using a simple mathematical equation, and each data stream is rep- resented as information passes through the equations. Instead of having to process each packet through each device, the analytical simulator just needs to process a group of calculations on input data. This results in extremely fast simulation results, although they are not as accurate as the discrete event results. Modeling Network Problems No matter which network-testing method you use, to fully test a network application, you must be able to simulate the way the application behaves within the normal network environment. This means that the method must be robust enough to simulate all of the problems associated with busy networks. 246 Chapter 13 17 433012 Ch13.qxd 6/16/03 9:11 AM Page 246 Each of these problems must be duplicated within the network model used to predict the performance of the network. There are several problems that must be taken into consideration: ■■ Network bandwidth constraints ■■ Packet errors ■■ Lost packets ■■ Out-of-order packets ■■ Delayed packets This section explains each of these problems, how they can affect a network application running on a production network, and how the network model must take them into consideration when producing test results. Bandwidth Constraints A network application should never assume unrestrained communication between network devices. One of the primary factors in determining network application performance is how often the application sends data across the net- work. The network can make or break the network application’s performance. Network applications that send lots of packets between devices are depen- dent on the efficiency of the network. Any delays between packets introduced by busy networks can be catastrophic for the application. It should be a fun- damental design goal of all network application programmers to minimize the data traffic that must traverse the network. There are many applications that violate this principle. Database applica- tions are usually the biggest culprits. There are two basic ways to perform client/server database functions across the network. In the first method, the database engine is located on the client device, while the database data is located on the server. For a simple data query, the database engine must select each record within the database index to look for the query result. This requires each index record to be passed across the network to the client, causing lots of network traffic. In the second method, the database engine is located on the server device, along with the database data. The client contains simple code that sends the query to the database engine, which then does all of the database index lookups internally, and sends a single response with the result. This method produces minimal network traffic, and is more network-friendly, especially for slower networks. In both scenarios it is crucial to model how much network bandwidth is being consumed by the network application. The key is knowing the characteristics of the network application, and being able to quantify those characteristics. Net- work traffic generator applications can be used to simulate the network traffic Measuring Application Performance 247 17 433012 Ch13.qxd 6/16/03 9:11 AM Page 247 generated by each type of database application—either in real life, by sending dummy database packets across the network, or in simulation, by performing the calculations necessary to simulate the traffic. Packet Errors The network application should also be prepared to deal with faulty data received in packets. Most network applications that use the TCP or UDP pro- tocols do not have to worry about this, as it is taken care of at the network level. However, applications that use their own protocols must be prepared to handle errors. The most common method used in network applications to handle packet errors is the checksum method. Most standard protocols provide a system for calculating a checksum of the data contained in the packet, and including the value within the packet (the checksum part of the packet is set to zero for the calculation). The receiving device must extract the checksum value, perform its own checksum calculation, and compare the two values. If the values match, the packet is assumed to be error-free. NOTE There are lots of methods for calculating checksums. The most common method used for network packets uses 16-bit arithmetic to break the packet into 16-bit chunks, add them, and take the complement of the result. Network models simulate network errors by injecting errors within a cer- tain percentage of packets received in the model. Network emulation devices do this by purposely altering the packet before it is forwarded to the output. Network simulators perform this task mathematically. Lost Packets Besides error packets, the network application should also be prepared to deal with missing packets. While applications that use TCP don’t have to worry about this, it is a big concern for applications that use UDP for communications. For applications in which lost packets are devastating, a trustworthy proto- col, such as TCP, is recommended. Barring a total network failure, this will cause the underlying network devices to ensure that the remote host receives each packet sent. This often requires retransmission of packets that are not acknowledged as being received. While it is perfectly normal to have some retransmitted packets on the network, too many may be an indication of a net- work problem, such as an overloaded switch or router that is dropping packets. For some applications, however, lost packets are not a problem. Applica- tions that transmit information at regular intervals can just send another data 248 Chapter 13 17 433012 Ch13.qxd 6/16/03 9:11 AM Page 248 [...]... different sources Instead of allowing a single network source to monopolize the network resources, the router attempts to forward packets from multiple sources in a fair and equal manner This may result in packets being dropped from busy sources more frequently than from sources sending fewer packets It is assumed that the busy sources will recover from the loss of packets more quickly than the sources... production network environment Alternately, some network testing can be done on the actual production network This provides the best method for determining how a network application will perform for customers Unfortunately, it is often not possible to use the production network, and sometimes it is dangerous to test new applications on the production network To solve the testing dilemma, many network. .. administrators are turning to network emulation and simulation Network emulators provide a way to feed actual network application data through a device that emulates the actual production network, and watch the results The benefit to network emulation is that the actual network application can be used without having to use the actual production network Network simulators allow networks and applications... have two network cards Each network card is placed on a separate network segment, as shown in Figure 14.3 server dummynet NIC1 NIC2 Network 1 Network 2 client server Figure 14.3 Using dummynet as a network bridge or router dummynet All traffic from one network to the other must pass through the FreeBSD system, and dummynet The second method uses the FreeBSD bridging facilities, connecting two network. .. to network simulators is that they do not use the actual network application, but rather an estimation of the type of data it will produce on the network Network simulators can also require lots of processing time, for performing calculations to simulate each network device and link in the network path Both network emulators and simulators must be able to accurately model the devices found on the network. .. for the varying quality of wireless network behavior, most wireless network models focus on the effect of the slower network speed introduced by the wireless network In most LAN situations, the wireless network is the bottleneck within the network path, and should be considered the limiting factor within the network model Due to the significantly slower wireless network speed (usually less than 3 Mbps),... same subnet Network traffic that must traverse from one network segment to the other must pass through the dummynet system, which then injects the configured network problems into the network traffic This method provides an easy way to emulate network problems in an existing test network Since the FreeBSD system is configured as a bridge, no network configuration is required, and test network traffic... connected to the Internet, firewalls have become commonplace in most corporate networks (and even many home networks) Unfortunately, firewalls add another element to the network that can affect the performance of network applications While firewalls protect our networks from intruders, the downside is that protection comes at a price performance Each packet that traverses the firewall must be checked against... represents network overload), packets are dropped, representing network problems such as collisions and error packets on an overloaded network hub 251 252 Chapter 13 packets coming in from ports packets going out to ports Figure 13.3 Modeling a network hub The idea for modeling network hubs is to determine how much traffic causes the network hub to fill This can be the result of either a short burst of network. .. associated with networks Each network device has unique characteristics that must be accurately modeled, using either mathematical equations or a combination of hardware and software Each of the different problems found on the network, such as packet loss, delays, and errors, must also be factored into the emulation or simulation The next chapter begins the network application testing toolkit by describing . testing network applications: ■■ Using a test network ■■ Using the production network ■■ Using a network emulator ■■ Using a network simulator Each of these testing environments has a unique. have used to determine how a network application might per- form within the production network environment. Methods of Testing Network Applications The point of testing network applications is to. production network will affect the application’s performance. Often, customers, programmers, and network administrators do not find out that a new network application is the cause of network performance