518 Part IV: S RMAN in the Oracle Ecosystem ync and split technology is an example of an innovative (and challenging) solution for storage recovery that complements or duplicates many of the features RMAN can accomplish independently Over the past five years, sync and split has become a widely used technology to provide immediate and very fast system recovery at the storage hardware level In this chapter, we will provide an overview of what sync and split technology refers to We won’t be discussing any single implementation in particular, but rather discussing the implications for RMAN and database backups After the overview, we go into the specific steps required to integrate sync and split solutions into an RMAN backup strategy Sync and Split: Broken Mirror Backups In the beginning, doing sync and split backups involved nothing more complicated than extending the functionality of hardware mirroring The best way to explain this statement is through an example Suppose we have a disk controller that has two hard drives For redundancy, we set the RAID level to + so that we are mirroring everything on disk A to disk B This gives us immediate protection against any kind of hardware failure on either disk A or disk B The next step, then, is to try to leverage the hardware mirror to provide logical fault tolerance That is the goal of sync and split technology: to provide a fallback position in case of some failure that has occurred on both copies in the mirror For example, suppose that a user has deleted the entire oracle software tree or the oradata directory Such a deletion would immediately occur at both copies in our mirror, so having a mirrored copy would us no good So, what is the solution? The innovation is that any mirrored disk group may have two mirror groups, but may only ever have one mirror currently writing the identical bits as the primary disk group Let’s build an example with three logical volumes, A, B, and C, all dedicated to the same data Volumes A, B, and C are all mirrored copies of each other However, at P.M., volume A is split away from the mirror, leaving its bits “stuck” at the split time Volumes B and C continue to be bit-for-bit copies After four hours, at P.M., volume C is split from volume B so that it no longer gets writes of data At this point, there are three different copies of the data on the volume: a copy at P.M., a copy at P.M., and a current copy There is also no redundancy to protect against a disk failure Where Are We in RAID? Need a superfast, overly simplistic primer on RAID? We’re here for you There are hundreds of theories, from the radical to the traditional, that outline the best possible solution for disk failure protection Typically, the Oracle “technorati” have long taken the position that nothing beats RAID + 1, in which you have two disk groups, group and group 2, both of which have two disks The two disks on group are striped, so that data is evenly spread across both disks Group is an exact copy, bit for bit, of group This configuration gives us both performance, by striping across disks to avoid hot spots, and redundancy, by writing every bit twice Recently, we were reviewing the specs for a RAID + configuration, which is slightly different from + Instead of striping and then mirroring, a RAID + configuration mirrors and then stripes The difference is best represented visually, as shown in the following illustration Here, we mirror each disk separately so that we end up with four disk groups After mirroring each disk, we then stripe across the four mirrors Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Chapter 22: RMAN in Sync and Split Technology 519 It might seem like a small difference, but RAID + has greater fault tolerance, because the failure of any one disk does not take down the other mirrored disks In RAID + 1, if any disk in group fails, the whole group goes offline So, RAID + provides greater tolerance than RAID + for multiple disk failure, instead of single disk failure To get back to our RAID + configuration, disk volume A will be “resilvered” up to disk B, which runs at the current point in time This sync up is based on the fact that the volumes have a journaling mechanism in place that records all data changes This journaling is more I/O on top of the multiple writes to each volume Volume A will get access to the journals of changes on volume B and will apply all the changes until it is getting live writes at the same time as volume B At this point, then, you have volumes A and B in redundant mode, and volume C is your fallback position, at P.M Figure 22-1 illustrates this process Writes to disk A B Writes to disk C A 2:00 P.M 1:59 P.M FIGURE 22-1 B Writes to disk C 2:01 P.M 2:00 P.M A B C 2:00 P.M 6:01 P.M 6:00 P.M 6:00 P.M Sync and split technology in action Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 520 Part IV: RMAN in the Oracle Ecosystem This sync and split cycle goes on and on, ad infinitum Every four hours, a volume is synced up to the primary volume, and another volume is split away to provide a fallback position in case of a logical failure What happens at the time you actually encounter the logical failure? In our example, let’s assume that it is now P.M Volumes A and B are getting concurrent writes, and volume C is waiting idle at P.M At P.M., a DBA is doing some system maintenance and deletes the system datafile from the production database This is when the worrying begins Luckily, no unrecoverable data has been added to the database since the end of the day at P.M However, the nightly batch loads start in about 15 minutes The DBA has a small window to get the production database back up and running With the database running entirely on the mirrored disk volumes A and B, the sync and split architecture has given our DBA an immediate solution He immediately configures volume C, which was stuck at P.M., as the primary volume and starts up the database When the database looks for its datafiles, it finds all the files as they appear on volume C, at P.M., and no deletes have taken place By the time the DBA is finished, it is only 8:05 P.M The batch processes will kick off on time Figure 22-2 shows the process Oracle Databases on Sync and Split Volumes The Oracle software files can reside on a sync and split volume and thus can help protect against logical corruption that occurs in the binaries themselves No additional configuration is needed, from an Oracle perspective The files associated with an Oracle database, on the other hand, come with some very specific caveats and disclaimers when you start putting them on sync and split volumes These caveats and disclaimers relate to the fact that Oracle files are always open and always have active writes taking place (this being the primary importance of a good relational database) So, if you are actively writing to your database and it is mirrored on two drives, there will be consequences if you suddenly break the mirror, unbeknownst to the database Each vendor-specific solution is a bit different, but at some point, a volume that is getting active writes must turn off the writes to that volume while continuing to allow writes to another volume And regardless of how a salesperson might pitch it, the process of breaking a mirror is not instantaneous Breaking a mirror is more like peeling a banana—you start at the top and FIGURE 22-2 Sync and split in action Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Chapter 22: RMAN in Sync and Split Technology 521 separate the peel from the fruit until you get to the bottom Suppose your Oracle datafile is the fruit, and the mirrored copy of the datafile is the peel If you peel away the mirror copy, you are starting at the beginning of the datafile, and the break is complete when you reach the end of the datafile However, it is possible (likely) that Oracle will attempt to write to a block while the mirror is in the middle of peeling away So, on the primary volume, nothing is wrong—the file header knows that an SCN has been advanced in the file and knows which block it was—but on the split mirror, the datafile header knows nothing about the written block So, after the mirror break is complete, what we have on the split mirror volume? One fuzzy datafile that is unrecoverable Check out Figure 22-3 to see this Fear not, for there are ways to ensure that the split mirror is a healthy copy of the database It just takes a bit of work first How you configure Oracle database files in a sync and split environment depends on what type of files you are configuring: datafiles, control files, redo log files, or archive logs The following sections address each in turn Datafiles The previous section explained what happens to Oracle datafiles if a mirror split takes place without any preparation: the split volume copies of the files are left in a fuzzy, unusable state This is precisely the same predicament you run into if you simply take a copy of an online datafile without first putting it into hot backup mode So, before you break the mirror, you must put all datafiles into hot backup mode This is not an optional step, regardless of which vendor product you are using Because the split generally takes a very short time, the amount of time in hot backup mode is much shorter than it would be if you were doing a copy against the same datafiles And the I/O hit of running in backup mode (and producing more archive logs) will be relatively small, as well FIGURE 22-3 Unrecoverable fuzzy datafile Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 522 Part IV: RMAN in the Oracle Ecosystem To alleviate the headaches of hot backup mode for those implementing sync and split architectures, Oracle has added syntax that allows you to put an entire database into hot backup mode with a single command: alter database begin backup; Previously, you had to put each tablespace into hot backup mode If there is something preventing the file from going into backup mode, a warning is generated in the alert log, but the begin backup command proceeds anyway After the split is complete, you pull the database out of hot backup mode with the following command: alter database end backup; Control Files A split mirror copy of a control file is in an unusable state immediately after the split mirror operation completes The control file, in general, is up-to-date on the current state of all the datafiles However, based on the total duration of the split itself, and the overall activity on the database at the time of the split, the control file at the split volume may not reflect much accurate data about the state of the datafiles Putting the database into hot backup mode cures most of these ills With the database in hot backup mode, the control file is aware of a starting point at which recovery will be required, and from which it will be feasible However, the control file is still at odds with reality: it thinks of itself as a current control file of an active database This is hardly the case We’ve seen some implementations where a DBA insists on trying to keep the current control file available as such on the split volume, particularly if the split volume will be used for reporting purposes However, when the time comes to put this control file into service for the sake of recovery, you have to use the using backup controlfile command so that the control file understands that some of its checkpoint and SCN information may not reflect reality: recover database using backup controlfile until cancel; If you will be mounting the Oracle database on the split mirror volume for reporting purposes, you may want to use the using backup controlfile command, even if you will not be applying any archive logs, just so the control file is flagged as a backup We discuss this later in the section “Benefits of the Split Mirror Backup.” Redo Log Files Split mirror copies of the online redo logs are useless in every way, shape, and form If possible, don’t even bother putting them on the volume that is going through the sync and split There is no mechanism in the online redo logs to account for writes to the file during the split operation Archive Logs Archive logs are an excellent candidate to be put on a sync and split volume Doing so gives you a backup of existing archive logs on disk in a second location Of course, if you split the archive log volume at the same time as the datafile volume, you not get all the redo that you need to properly recover your database from the split volume We suggest that you keep your archive logs on a separate set of sync and split volumes from the set on which you keep your datafiles and Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Chapter 22: RMAN in Sync and Split Technology 523 control files That way, you can split the datafiles, take the database out of hot backup mode, force a log switch, and then split the archive log volumes Then the split mirror volume with the archive logs contains all of the redo required to start the split mirror copy of the database One last note on archive logs on split mirror volumes When the database begins to create an archive log on disk, the split operation may leave behind an unfinished archive log on the split mirror volume This archive log would be unusable during any recovery operation This poses a problem only for human-managed backup and recovery operations, where it is unknown if the archive log that is on-disk is complete or only half-written Here’s why it doesn’t pose a problem for RMAN: When an archive log is being generated, the control file is not updated with a record that such an archive log exists until the archive log is complete Therefore, in a split mirror scenario, if half of an archive log is generated on the split volume, the control file on the split volume has no record of that archive log During an RMAN operation, then, the control file would be consulted for archive log records, and the half-written file would not exist in the metadata To RMAN, the half-written file doesn’t really exist Benefits of the Split Mirror Backup We’ve discussed briefly the primary benefit of using the sync and split architecture: a nearly instantaneous fallback recovery point for all files on a particular set of disks This benefit expands beyond the scope of this book (the Oracle database) to include a fallback point for all files that exist on the volume There are also other primary benefits of the sync and split, which are discussed next Fast Point-In-Time Recovery From the database perspective, sync and split provides a point-in-time recovery option that can take minutes instead of hours You simply change the primary disk group to the split mirror, and the datafiles are ready Then, apply archive logs up to the point where the failure occurred, and you can open the database Speedy-Looking Backups Another benefit of the sync and split architecture is the relative speed of the backup operation itself Properly generating copies of the database files at the split mirror side takes only a few moments with the database in hot backup mode After that, a backup is ready to be pressed into service very quickly Of course, there’s no magic involved with sync and split I/O is I/O is I/O It might look like the backup is taking no time at all, but in reality the backup is being taken all the time at the hardware level, because prior to the split operation, the files are being written to simultaneously However, handing the backups over to the hardware architecture can prove to be extremely powerful in many organizations, where the hardware can be responsible for backing up more than just the database Mounting a Split Mirror Volume on Another Server Beyond the simplistic restore and recovery features, much of the true power of sync and split solutions currently in the marketplace comes from what you can with the split copy of the database Because the underlying hardware is likely to be a storage array with many computers connected to it, any volume on that storage array can theoretically be associated with any computer connected to it Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 524 Part IV: RMAN in the Oracle Ecosystem For example, let’s take a database, PROD PROD resides on disks in volume A, which is mirrored on volume B Both volume A and B are connected to server Dex Volumes A and B both exist on storage array Newton At P.M., volume B is split from volume A and disassociated from server Dex Immediately after this, volume B is mounted on a different server, Proto, which is also connected to storage array Newton After volume B is mounted on Proto, a copy of the database PROD that resided on Dex now resides on Proto, with almost real-time amounts of data The database copy that is on volume B, and mounted by server Proto, can be recovered and then opened for testing, development, or reporting Later, at P.M., when it is time to resilver volume B with volume A, Proto can dismount volume B, and then it can be remounted by Dex The sync operation takes place, overwriting any changes that occurred on volume B after the split at P.M Note that before you can open a split mirror copy of the database on a different node, a new backup control file should be taken and used When you resilver volume B with volume A, this new copy will be overwritten by the correct file on A Taking Backups from the Split Mirror Another benefit of sync and split backups, within the framework of this book about RMAN, is the ability to mount the split volume on a different server and, from there, back up the database to tape for long-term backup storage This allows you to offload the memory, CPU, and I/O operations of the RMAN backup to a completely different server and ensure that there is no impact to your production database RMAN and Sync and Split There are a few different contact points that RMAN has with a sync and split implementation: ■ If you use RMAN for recovery, you must make RMAN aware of the datafile copies that are created by the split operation ■ You can use RMAN to take backups from the split mirror volume instead of from the production database itself Registering Split Mirror Copies with RMAN If you are a dedicated RMAN user, then you probably understand the benefits that come from executing all recovery statements from within RMAN, instead of from SQL*Plus or elsewhere RMAN recovery provides access to the information in the control file so that you are not scrambling to uncover which backups exist where and trying to ensure that you are not missing any files The control file also aids in archive log management during recovery When a sync and split system is in place, RMAN doesn’t know about everything The act of splitting the mirror volumes effectively gives you a full datafile copy of every datafile in the database that can be used during a restore/ recovery operation, but RMAN has no idea these copies exist So, you have to make RMAN aware You this by registering the datafile copies with RMAN via the catalog command The catalog command can be used against a single datafile copy: catalog datafilecopy '/volumeA/oradata/system01.dbf'; Or, starting with 10g, you can catalog an entire directory by the directory name: catalog start with '/volumeA/oradata'; Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Chapter 22: RMAN in Sync and Split Technology 525 By using the catalog command, you take the split mirror copies and make them part of any future restore or recovery operation that might be required You might be asking yourself, “Why I need to make RMAN aware of the split mirror copies when I can just remount the entire volume as the primary volume and be up and running without RMAN’s help?” A valid question But what if it makes more sense to switch to only a single copy of the file? Perhaps doing a full database point-in-time recovery would be too expensive, but you still want to leverage the split mirror copy of a subset of files Beyond that, RMAN also greatly simplifies the recovery stage of any operation, so it makes sense to make RMAN aware of the copies of the archive logs, as well Taking RMAN Backups from the Split Mirror With increasing frequency, DBAs are realizing that with split mirror investments, an additional layer of protection is required, in the form of RMAN backups of the database The split mirror backup is by definition a short-lived copy—sooner or later, it will be lost when the volume is resilvered with the primary database volume But what about restoring from last night? Or last week? As you can see, a full-fledged media backup is still required With an idle copy of the database simmering on the back burner of the split mirror, a light bulb appears above the DBA’s head: “I should just mount the split mirror drive onto a different server, and take the RMAN backup from the split mirror directly to tape (or to a different disk volume that can be mounted on the primary).” Great idea! Sounds simple enough, right? Well, a few tricky points need to get worked out first; otherwise, you will have the case of the mysteriously disappearing backups Here’s the problem: RMAN accesses the control file to determine what to back up, and after the backup is complete, it updates the control file with the details of the backup If you are connected to a split mirror copy of the control file, that copy gets updated with the details about the backup So then, of course, when you go to resilver the split volume with the primary, the control file is overwritten with the data in the primary control file, and the backup data is lost forever The solution, you figure, is to use a recovery catalog when you back up at the split mirror That is a sound, logical decision: after the backup is complete, the split volume control file is updated with the backup records, which are then synchronized to the catalog Then, it’s simply a matter of syncing the catalog with the primary volume so that the backups can be used Too cool! So, suppose that you back up from the secondary volume, you sync the backup records to your recovery catalog, and then, you connect RMAN to the primary volume database and to the catalog You perform a resync This is where things get really, really weird Sometimes, when you try to perform an operation, you get this error: RMAN-20035: invalid high recid Other times, things work just fine, it seems, but the backups you took at the split mirror database have disappeared from the recovery catalog The problem, now, has become the internal mechanism of how RMAN handles record building in the control file and the recovery catalog Every record that is generated gets a record ID (RECID), which is generated at the control file When the backup occurs at the split mirror database, the control file gets its high RECID value updated, and this information gets passed to the catalog But the RECID at the primary database control file has not been updated, necessarily So, when you connect to the catalog and the primary database, if the catalog’s high RECID is higher than the one in the control file, you get the “invalid high recid” error If the RECID in the Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 526 Part IV: RMAN in the Oracle Ecosystem catalog is lower than the RECID of the primary database control file, RMAN initiates an update of the catalog that effectively eliminates all the records since the last sync operation with the primary control file Poof! Backup records from the split volume are gone The solution to this problem is to set the control file at the split mirror to become a backup control file If RMAN detects that it is backing up from a noncurrent control file (backup or standby), it does not increment the RECID in the catalog, so that the records are available after a resync with the current control file at the primary database You cannot use the control file autobackup feature if you will be taking backups from the split mirror volume Because the control file in use is a backup control file, autobackup is disallowed RMAN Workshop: Configure RMAN to Back Up from the Split Mirror Workshop Notes This workshop assumes that you put all the tablespaces into hot backup mode (a requirement) during the period of the split After the split, you connect the split volume to a new server that has 10g installed, and you now want to take an RMAN backup Because RMAN will give an error if files are in backup mode, you need to manually end backup for every file, as described in this workshop It’s best to write a script for this This workshop also assumes that you split the archive log destination and bring it across to the clone at the same time for archive log backup Step Mount the database on the clone server, and prepare the control file for RMAN backup: startup mount; alter database end backup; recover database using backup controlfile until cancel; cancel exit Step Connect RMAN to the clone instance (as the target) and the recovery catalog, and run the datafile backup: rman target / rman> connect catalog rman/password@rman cat db rman> backup database plus archivelog not backed up two times; Step Connect RMAN to the production database (as the target) and the catalog, perform a sync operation and archive log cleanup, and then back up the control file: rman target / rman> connect catalog rman/password@rman cat db rman> delete archivelog completed before sysdate -7; rman> backup controlfile; rman> resync catalog; Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Chapter 22: RMAN in Sync and Split Technology 527 Getting Sync and Split Functionality from Oracle Software There is considerable upside to having a hardware solution provide the architecture described in this chapter Typically, any operation that can be done purely at the hardware level will have performance increases over the same operation done by software By the same token, a hardware solution is always going to cost you more than a software solution Sync and split solutions are no different—the more work that is being done at the storage array, the faster it will go…and the more it will cost Starting with Oracle Database 10g Release 2, Oracle includes a full solution to provide sync and split functionality without paying for any third-party hardware or software solutions All you need is Oracle Database 10g Enterprise Edition, two servers (with the same OS), and a storage array Using a Standby Database, Flashback Database, and Incremental Apply for Sync and Split To implement a sync and split solution using only Oracle software, you need to employ a different feature set within the RDBMS: a standby database, Flashback Database, and RMAN incremental backup and incremental apply All of these features have already been discussed to some extent in previous chapters Here’s how it works First, you create a standby database of your production database (see the workshops in Chapter 20) Once you have the standby database fully operational as a disaster recovery solution, you need to implement Flashback Database on both production and standby databases: alter database flashback on; With Flashback Database enabled, you can set a restore point on the primary server: create restore point chapter 20; alter system switch logfile; Apply changes through the restore point to the standby database At this point, the standby database can be opened with reset logs for testing or reporting alter database activate standby database; To resilver your standby database with the primary database, you need to take an incremental backup by using the from scn keywords to specify the SCN of the restore point Once this backup is complete, move it to the standby database site backup database incremental from scn 120000; At the standby database, shut down and then remount the database again Perform a flashback database to the restore point specified before the standby database was opened: flashback database to restore point chapter 20; Once the flashback completes, apply the incremental backup from the production database to the standby database, bringing it up to the point of the backup: recover database until scn 1521321; Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark ... operation and archive log cleanup, and then back up the control file: rman target / rman> connect catalog rman/ password @rman cat db rman> delete archivelog completed before sysdate -7; rman> backup. .. end backup; recover database using backup controlfile until cancel; cancel exit Step Connect RMAN to the clone instance (as the target) and the recovery catalog, and run the datafile backup: rman. .. target / rman> connect catalog rman/ password @rman cat db rman> backup database plus archivelog not backed up two times; Step Connect RMAN to the production database (as the target) and the catalog,