764 Part IV • Check Point NG and Nokia IP Series Appliances |qfe2|192.168.12.131|Up | 0| 0| 0| |qfe3| 192.168.1.131|Down | 32000| 0| 0| Problem Notification table |Name |Status|Priority|Verified|Descr| |Synchronization|OK | 0| 1618| | |Filter |OK | 0| 1618| | |cphad |OK | 0| 0| | |fwd |OK | 0| 0| | fw1 # From this output, we can see that there is a problem with the local member (fw1 in this example) and that the status of interface qfe3 is down. How ClusterXL Works in Load-Sharing Mode ClusterXL in Load-Sharing mode works in a very similar way to ClusterXL in HA New mode but with the following unique distinctions: ■ The MAC address used for the VIP address is shared among cluster members for that subnet.This means that there is no MAC address change on failure of a member as far as the network equipment on the local subnet of the cluster is concerned. ■ The MAC address of the VIP address is a multicast MAC address. (in other words, its first octet is an odd number). ■ In a healthy load-sharing cluster, all members of the cluster should be active and routing a portion of the active traffic. Connections through the cluster are managed on a per-connection basis. For example, if a host on 192.168.1.200 initiates a connection through the cluster to 195.166.16.129 and member fw1 takes the connection, the connection will just go through member fw1 unless a failure of member fw1 occurs.The connection will continue through member fw1 until the session has completed. No asymmetric routing should occur on this particular connection. The member in the cluster a new connection will go through is based on a hash of specific parameters defined in the Advanced section for ClusterXL Load Sharing (see Figure 21.49). www.syngress.com 252_BDFW_21.qxd 9/18/03 5:48 PM Page 764 High Availability and Clustering • Chapter 21 765 Assuming a “normal” connection passing through the cluster, all the packets involved will have the same hash value. For an Internet firewall, this means that, for a particular connection, a packet arriving from the internal network and a packet arriving from the Internet will have the same hash value. However, if the cluster is performing NAT or if VPNs are involved, we have a potential problem.The IP addresses and ports will be different on the “inside” and “outside” of the firewall. FireWall-1’s stateful inspection helps us out here; because it understands what changes have been applied to connections, it can adjust the hashing accordingly. As with ClusterXL in HA New mode, the members of the cluster still have their real IP addresses bound to their interfaces.This is particularly useful when the SmartCenter server is communicating with the cluster members, because it need not be located on the secured net- work.This makes it easier for the SmartCenter server to manage other firewall modules as well as the cluster. Although all members are live and handling traffic, who should respond to ARP requests for the VIP address? The members in the ClusterXL cluster will agree on which member in the cluster will respond to the ARP request; however, that choice is not based on the member pri- ority in the cluster, and even if a member is designated as having a problem but the interfaces on the member are active, the problem member may still respond to the ARP request. Within the ARP reply packets is the multicast MAC address for the VIP. NOTE You need to make sure that adjacent hosts and routers will accept a multicast MAC reply for a nonmulticast IP address. For example, host 192.168.1.200 would ARP for 192.168.1.130—the VIP address of the cluster—in order to route packets through the cluster. The ARP response would contain a multicast MAC address. Different systems respond in different ways: Windows is generally fine, but Cisco routers on the same subnet will not accept the ARP reply and will not cache the multicast MAC address, so steps need to be taken to circumvent this problem for Cisco routers. These steps usually involve entering a static entry into the ARP table of the router for the multicast MAC address. ClusterXL Load-Sharing Mode Failover As with ClusterXL in HA New mode, the key to how failover works is in the CPHA protocol that the members send to all the other members in the cluster, using multicasts. www.syngress.com Figure 21.49 A Load-Sharing Algorithm Hash Can Be Based on These Parameters 252_BDFW_21.qxd 9/18/03 5:48 PM Page 765 766 Part IV • Check Point NG and Nokia IP Series Appliances There are many similarities between ClusterXL HA New mode and ClusterXL Load-Sharing mode CPHA protocol packets. In fact, they are identical in the way they work, apart from some details in the UDP data payload.The similarities include: ■ The source MAC address of the CPHA update packet is always 00:00:00:00:fe:<member number>, where member 1 would be 00, member 2 would be 01, and so on. ■ The destination MAC is always a multicast MAC address, ending with the VIP address in the last two octets of the MAC address. ■ The source IP of the CPHA update packet is always 0.0.0.0. ■ The destination IP address of the CPHA update packet is always the network IP address. ■ Layer 4 (the transport layer of the OSI model) is always UDP, source port 8116, destina- tion port 8116. ■ The first part of the CPHA payload within the UDP header packet is the same as ClusterXL in HA New mode, and the format of an FW_HA_MYSTATE payload is the same parameters but different data for these parameters. If we focus on the last point, we can see from Figure 21.50 how the data for the same param- eters differ. The main areas of note here are the HA mode, which states that the mode is mode 4 FWHA_Balance_mode—more than one member active. ClusterXL in HA New mode is referred to as mode 2 in this field. The other field of note is “Machine states.”This field communicates what the member origi- nating the CPHA packet thinks the status of all the other members is. As we can see in Figure 21.50, the sending member is aware that member 0 is active.This packet was originated from member 1, or fw2 from our example. www.syngress.com Figure 21.50 Packet Structure of a CPHA Packet When a Cluster Is in Load-Sharing Mode 252_BDFW_21.qxd 9/18/03 5:48 PM Page 766 High Availability and Clustering • Chapter 21 767 Under normal operation, these CPHA packets are multicast to all the other members in the cluster. Each member multicasts its perception of the state of the rest of the members in the cluster.This process occurs on each interface of a cluster member and is sent at regular intervals, several a second. Examining the CPHA protocol between cluster members, we see that if there is a problem on the member, the other members will show a “Machine states” value for that the member as down/dead.The member that is taking over a particular connection will then ARP for the MAC address of the local host it needs to push a packet to, and on response, it will continue the session. Note that the hosts on the local subnet do not notice any change in MAC address or IP address on failover.You could notice a small glitch in the data transfer while the failure occurs and failover to another member takes place, but the period of disruption should always be less than three seconds and is usually just over one second. Note that when a member in a load-sharing cluster takes over from another member, there is no gratuitous ARP broadcast, unlike HA mode.This is because it is unnecessary since there has been no MAC address change. Special Considerations for ClusterXL in Load-Sharing Mode We have covered the principles of how ClusterXL in Load-Sharing mode works. We now con- trast and compare how the special considerations for ClusterXL in Load-Sharing mode differ rel- ative to other cluster modes. Network Address Translation ClusterXL in Load-Sharing mode is actually quite forgiving with regard to NAT and how proxy ARP is performed, unlike HA mode. It will handle manual proxy ARP entries fine for NATed IP addresses, as long as you proxy ARP for the cluster multicast MAC address.You enter these static published ARP entries on all members in the cluster. Automatic ARP configuration can be selected in the Policy | Global Properties | Network Address Translation area of the SmartDashboard GUI.This works fine because the multicast MAC address is used for all the automatic ARPs that are required. Manual routes on the ISP router can also be used instead of using proxy ARPs. To summarize, as long as the multicast MAC address is used in any manual proxy ARPs, there should be no issues with Load-Sharing mode and NAT. User Authentication and One-Time Passcodes Like all HA and Load-Sharing clustering solutions, if you are using the Check Point security servers (for SMTP, HTTP, or FTP services) and a failover occurs, you will lose the connection and have to start again through the new member that the traffic is now going through.The secu- rity server and remote authentication issues discussed earlier in this chapter (comparing single gateway and clustering functionality) apply particularly to Load-Sharing mode, because sessions— with multiple connections—are always likely to be shared between all cluster members, unlike HA, when problems only occur on failover. www.syngress.com 252_BDFW_21.qxd 9/18/03 5:48 PM Page 767 768 Part IV • Check Point NG and Nokia IP Series Appliances Nokia IPSO Clustering ClusterXL is not available for the Nokia platform.This is because Nokia provides its own HA and load-sharing solutions. In this section, we look at the load-sharing cluster solution that Nokia provides on IPSO 3.6-FCS4, how to configure it, and how to configure FireWall-1 NG FP3 so that you have a complete Nokia load-sharing solution. We then talk about how you can test the cluster and go over any special considerations for this solution. Nokia Configuration To configure a Nokia load-sharing cluster, you need to take the following steps: 1. Configure the interfaces of a Nokia. 2. Configure FireWall-1. 3. Configure clustering in Voyager. We assume that you have installed IPSO 3.6 FCS-4 on your Nokia and that you have the Check Point FireWall-1 NG FP3 package installed and configured. As with setting up all clusters, it is recommended that you complete and test the physical connectivity first so that any problems that you encounter later aren’t due to a misconfigured switch or interface, because these could be difficult to spot later. In our example shown in Figure 21.51, you can see a sample Nokia cluster topology. www.syngress.com Figure 21.51 Our Example Nokia Clustering Topology Setup fw1 fw2 Hub Hub Hub Hub ISP Router cpmgr PDC 192.168.11.131 eth-s1p2c0 MAC=00:c0:95:e0:15:dd 192.168.11.132 eth-s2p2c0 MAC=00:c0:95:e2:b1:41 195.166.16.134 195.166.16.131 eth-s1p1c0 MAC=00:c0:95:e0:15:dc 195.166.16.132 eth-s2p1c0 MAC=00:c0:95:e2:b1:40 192.168.1.131 eth-s1p4c0 MAC=00:c0:95:e0:15:df 192.168.1.132 eth-s2p4c0 MAC=00:c0:95:e2:b1:43 195.166.12.131 eth-s1p3c0 MAC=00:c0:95:e0:15:de 192.168.12.132 eth-s2p3c0 MAC=00:c0:95:e2:b1:42 192.168.1.200 Default route = 192.168.1.130 Out to the Internet 195.166.16.129 State sync Network 192.168.11.0 /24 External Network 195.166.16.0/24 VIP = 195.166.16.130 VMAC=01:50:5a:a6:10:82 Cluster control Network 192.168.12.0/24 VIP = 192.168.12.130 VMAC=1:50:5a:a8:c:82 Internal Network 192.168.1.0/24 VIP = 192.168.1.130 VMAC=01:50:5a:a8:01:82 Domain = london.com 252_BDFW_21.qxd 9/18/03 5:48 PM Page 768 High Availability and Clustering • Chapter 21 769 The main difference in network topology between Nokia clustering and using Check Point ClusterXL is that you require a dedicated network for Nokia cluster control communications. This is in addition to the Check Point state sync network. As you can see from Figure 21.51, each network that has a VIP also has a virtual MAC address—a multicast MAC that is used for the VIP. From a network perspective of neighboring equipment on the same network as the cluster interfaces, it looks very similar to Check Point ClusterXL in Load-Sharing mode. You should ensure that you have configured all of the following using Voyager on each of the cluster members: ■ Make sure that interface speeds are consistent across host and switches on the subnet. Only use full-duplex where connected directly to full-duplex-enabled switches! ■ Make sure you have entries in each hosts file for the FireWall-1 management station and the other modules in the cluster. ■ Make sure you have the correct time and date and the correct default local for each member in the cluster and on the Check Point management station. ■ Make sure that the FireWall-1 NG FP3 package is installed. ■ Read the Nokia IPSO and Check Point NG FP3 release notes! A Few Points about Installing an Initial Configuration of NG FP3 on Nokia IPSO Installing software packages on the Nokia platform is very different compared to installing on other platforms. Packages are added to Nokia and “enabled” using the Voyager interface. However, that is not the end of the process of installing FireWall-1 NG FP3.You need to log out after the package install and run the cpconfig command on the Nokia console.The output you will see is fairly similar to the output you would see in the UNIX installation of a cluster, and the choices you would make are identical. One section during the install is specific to clustering: Would you like to install a Check Point clustering product (CPHA, CPLS or State Synchronization)? (y/n) [n] ? y Even though you will be using the Nokia clustering solution, make sure you answer y.This will make sure that you have the state synchronization available when you set up your cluster. That is essential for ensuring that connections continue through another member when failover occurs. Check Point FireWall-1 Configuration for a Nokia Cluster We will run through the most direct method of configuring FireWall-1 objects and rules for a Nokia cluster.This means that we will create the cluster member objects via the gateway cluster object directly and set up the SIC trusts between the management station and the cluster mem- bers within the cluster gateway object. Once the cluster gateway object is configured, we will www.syngress.com 252_BDFW_21.qxd 9/18/03 5:48 PM Page 769 770 Part IV • Check Point NG and Nokia IP Series Appliances install a basic policy to the cluster. If you have not already done so, it’s a good idea to look through the “Configuring FireWall-1 for ClusterXL in HA New Mode” configuration procedure described earlier in this chapter, because there are many similarities. Configuring the Gateway Cluster Object Within the NG FP3 SmartDashboard GUI, click Manage | Network Objects and click the New button. Select Check Point | Gateway Cluster.You will be presented with a pop-up window (see Figure 21.52). Particular areas of note here are that the IP address stated in the General tab is the external interface.The other important points to note are that the ClusterXL check box is unchecked. Now click the Cluster Members menu option on the left side of this screen. Run through the steps for creating a new member, as we did when configuring Cluster XL HA New mode. After the trust has been set up between the management module and enforcement module, click the Topology tab and click Get Topology. For each interface that is received, click it and set the topology for it. For example, the eth-s1p1c0 interface has IP address 195.166.16.131 assigned to it, and this is marked as our External interface in the topology. All the others are defined as This Network (see Figure 21.53). www.syngress.com Figure 21.52 Defining General Properties of the Nokia Cluster 252_BDFW_21.qxd 9/18/03 5:48 PM Page 770 High Availability and Clustering • Chapter 21 771 Click OK.You should now have the first member of the cluster defined and trusted. Repeat the procedure again in the Cluster Members menu to add the second cluster (and third, fourth, and so on). Use the New button each time. Once complete, you should have a list of cluster members defined. Click Availability Mode on the left side of the screen to select which mode the Nokia cluster will operate in. Make sure you select Load Sharing (see Figure 21.54). Note that in Nokia clustering, this setting has no functional effect, but it is useful to select the correct one so that when you look at it again, you know what mode you are operating your Nokia cluster in! It is also useful to avoid any confusion if you need to seek technical support. www.syngress.com Figure 21.53 Topology of a Cluster Member Figure 21.54 Availability Mode Configuration for a Nokia Cluster 252_BDFW_21.qxd 9/18/03 5:48 PM Page 771 772 Part IV • Check Point NG and Nokia IP Series Appliances Click the Synchronization menu option on the left side of the screen.You need to add the network you are going to use for synchronizing the FireWall-1 state tables. Note that this net- work should not be the same as your Nokia cluster control network. Click the Add button to add a synchronization network. In our example, the IP address is 192.168.11.0 , and the netmask is 255.255.255.0. (See step 8 and Figure 21.23 of “Configuring ClusterXL in HA New Mode” for an example.) Click OK. It is possible to add backup synchronization networks here; it would be acceptable to include the Nokia control network as a backup sync network, but if you do, the cluster should be moni- tored carefully so that a failover to this network is quickly identified and addressed. Configuring topology in the cluster object when using Nokia clustering is not mandatory— in fact, it is not recommended. Doing so will change the behavior of the cluster with regard to packets originating from a member in the cluster.The effect of configuring a topology is covered in “Special Considerations for Nokia Clusters” later in this chapter. Once this process is complete, you are ready to click OK and start defining your security policy. Configuring the Nokia Cluster Rule Base You have some choices as to the rule base you want to install.You can either see if the configura- tion of your cluster object is going to work and install an open policy, or you can create a strict policy now. Remember, there is still one more step to do, which is to configure the clustering on Nokia using Voyager.This being the case, you might want to install an open policy now and then tighten it later once you are happy that your clustering is working correctly. You need to allow IPSO cluster control protocols between each IP address of the Nokia cluster.This means you will have a rule, close to the top of your rule base, that will look some- thing like Figure 21.55. The group fwcluster-clusterips is made up of node host objects, one for each VIP address (195.166.16.130, 192.168.12.130, and 192.168.1.130 in our example). WARNING It is vital to add the fwcluster-clusterips group to security policy rules wherever you use the cluster object as a destination. This is because the cluster object does not include the VIP addresses, since we have not defined the cluster topology information. However, it is possible to connect to the VIP addresses. This is most important when defining the “stealth” rule (see Figure 21.57). www.syngress.com Figure 21.55 Rule Showing Communication between Cluster Members 252_BDFW_21.qxd 9/18/03 5:48 PM Page 772 High Availability and Clustering • Chapter 21 773 The ipso-cc group is made up of two services that we will call IPSO Cluster Control Protocol 1 and IPSO Cluster Control Protocol 2.You define these by clicking Manage | Services | New | TCP.The definition of these services is shown in Figures 6.56 and 6.57, respectively. Once defined, these services can be added to a service group (defined as ipso-cc in our example). In addition to the ipso-cc services, we have also accepted the Network Time Protocol (NTP). Running NTP is a good idea to make sure that the time between the Nokia cluster members does not drift. When you define the “stealth” rule in order to explicitly protect the cluster members, add the cluster object and the group of VIPs, fwcluster-clusterips, as shown in Figure 21.58. Once you have configured your policy, install it to the cluster object. Note that in Figure 21.59, we can see that the policy will fail to install if it does not install to all members of the cluster. One thing to be acutely aware of here is that if a member in the cluster is down or switched off and later comes online and becomes functional, it will first look at other members www.syngress.com Figure 21.56 Defining Service for IPSO Cluster Control Protocol 1 Figure 21.57 Defining Service for IPSO Cluster Control Protocol 2 Figure 21.58 A Stealth Rule on a Nokia Cluster Rule Base 252_BDFW_21.qxd 9/18/03 5:48 PM Page 773 [...]...252_BDFW_21.qxd 77 4 9/18/03 5:48 PM Page 77 4 Part IV • Check Point NG and Nokia IP Series Appliances of the cluster to compare the policy that it has against the policy that the other cluster members have If the other cluster members have a more recent policy, the cluster member that has just come up will download the policy from one of the other cluster members—before it attempts to download the policy from the. .. on the load-sharing algorithm .The member who will deal with the connection will pass the packet up through the IP stack to the Check Point FireWall- 1 NG FP3 kernel for the incoming interface .The TCP SYN packet will pass through the rule base of the firewall and, providing everything is fine, it will then send the packet out of its external interface, with the source IP address of 195.166.16.130 (the. .. alternative member in the cluster.This is done on the cluster control network Again, the key is the cluster control protocol that uses this network The Nokia cluster control protocol is used by the member that is the master .The master member sends out the status of the cluster to all other members in the cluster, using the cluster control protocol .The master member is usually the first member that is... If the master fails, another member will take over and become master There is only one master member in any cluster, but the member that is master can change depending on failures in the cluster www.syngress.com 252_BDFW_21.qxd 9/18/03 5:49 PM Page 78 7 High Availability and Clustering • Chapter 21 78 7 When the master member in the cluster communicates with the other members in the cluster, it uses the. .. protocol 0x90 (144 decimal) .The cluster control network is used exclusively (unlike the CPHA protocol used in ClusterXL) When the master communicates with the other members in the cluster, it is from the real source MAC address of the master on the control network, the source is the real IP address of the master, the destination MAC address is a multicast MAC address, and the IP address is a multicast... enabled in the Voyager VRRP page In the test we ran on our example network, a ping was initiated from the FireWall- 1 management station (195.166.16.134) to the VIP of the cluster (195.166.16.130) A packet trace was run at the same time on the management station to analyze the packet for the ping session If you look at the ARP cache of the local host initiating the ping, you should now have the VRRP MAC... MAC is 0:0:5e:0:1:4 (the default gateway MAC address) Only one member of the cluster will do anything with the packet the one with the VR in master state.This will pass the packet up through the IP stack to Check Point FireWall- 1 for the incoming interface .The TCP SYN packet will pass through the rule base of the firewall and, providing that everything is fine, it will then send the packet out of its... that a member in the cluster responded But can we tell which member? The answer is yes, we can, but only if we examine the packet trace we took when the ping session took place If we look at the reply packet, in the data link layer, we can see the real MAC address of the member that responded, as shown in the packet analysis in Figure 21. 67 Figure 21. 67 Analyzing the ICMP Echo Reply for the Source MAC... Page 77 9 High Availability and Clustering • Chapter 21 77 9 In our example, we can see that the source MAC is 00:0c:95:e2:b1:40, which corresponds with member fw2 in the cluster (see Figure 21.51) Note that even though the real MAC address of the fw2 member was used, the source IP address for the ICMP echo reply was the virtual IP of 195.166.16.130 Test 2: Determining the Status of Each Member in the. .. will be seen in the /var/adm/message file when the internal Ethernet interface cable is removed from one of the members of the cluster, and then restored Note that when this happens, you will also see the cluster members table show one fewer member in the cluster (see Figure 21 .71 ) Figure 21 .70 Sample of Nokia /var/log/messages After Internal Interface Was Removed, Then Restored Jan 27 07: 29:35 fw1 [LOG_NOTICE] . for the FireWall- 1 management station and the other modules in the cluster. ■ Make sure you have the correct time and date and the correct default local for each member in the cluster and on the. using the –c “com- mand” option. Once in the shell, you can use the command show clusters to determine the status of the members in the cluster (see Figure 21 .74 ). Figure 21 .74 Example of Use of the. with the VIP address in the last two octets of the MAC address. ■ The source IP of the CPHA update packet is always 0.0.0.0. ■ The destination IP address of the CPHA update packet is always the