Page iii Unix Backup and Recovery W. Curtis Preston Beijing • Cambridge • Farnham • Köln • Paris • Sebastopol • Taipei • Tokyo Page iv Disclaimer: This netLibrary eBook does not include data from the CD-ROM that was part of the original hard copy book. Unix Backup and Recovery by W. Curtis Preston Copyright (c) 1999 O'Reilly & Associates, Inc. All rights reserved. Printed in the United States of America. Published by O'Reilly & Associates, Inc., 101 Morris Street, Sebastopol, CA 95472. Editor: Gigi Estabrook Production Editor: Clairemarie Fisher O'Leary Printing History: November 1999: First Edition. Nutshell Handbook, the Nutshell Handbook logo, and the O'Reilly logo are registered trademarks of O'Reilly & Associates, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O'Reilly & Associates, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps. The association between the image of an Indian gavial and the topic of Unix backup and recovery is a trademark of O'Reilly & Associates, Inc. While every precaution has been taken in the preparation of this book, the publisher assumes no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. This book is printed on acid-free paper with 85% recycled content, 15% post-consumer waste. O'Reilly & Associates is committed to using paper with the highest recycled content available consistent with high quality. ISBN: 1-56592-642-0 Page v This book is dedicated to my lovely wife Celynn, my beautiful daughters Nina and Marissa, and to God, for continuing to bless my life with gifts such as these. -W. Curtis Preston Page vii TABLE OF CONTENTS Preface xiii I. Introduction 1 1. Preparing for the Worst 3 My Dad Was Right 3 Developing a Disaster Recovery Plan 4 Step 1: Define (Un)acceptable Loss 5 Step 2: Back Up Everything 7 Step 3: Organize Everything 10 Step 4: Protect Against Disasters 13 Step 5: Document What You Have Done 15 Step 6: Test, Test, Test 16 Put It All Together 17 2. Backing It All Up 18 Don't Skip This Chapter! 18 Why Should You Read This Book? 19 How Serious Is Your Company About Backups? 22 You Can Find a Balance 25 Deciding What to Back Up 30 Deciding When to Back Up 38 Deciding How to Back Up 43 Storing Your Backups 52 Testing Your Backups 56 Monitoring Your Backups 58 Page viii Following Proper Development Procedures 59 Unrelated Miscellanea 60 Good Luck 65 II. Freely Available Filesystem Backup & Recovery Utilities 67 3. Native Backup & Recovery Utilities 69 An Overview 69 Backing Up with the dump Utility 73 Restoring with the restore Utility 91 Limitations of dump and restore 101 Features to Check For 102 Backing Up and Restoring with the cpio Utility 103 Backing Up and Restoring with the tar Utility 114 Backing Up and Restoring with the dd Utility 122 Comparing tar, cpio, and dump 127 How Do I Read This Volume? 129 4. Free Backup Utilities 141 The hostdump.sh Utility 141 The infback.sh, oraback.sh, and syback.sh Utilities 142 A Really Fast tar Utility: star 142 Recording Configuration Data: The SysAudit Utility 143 Displaying Host Information: The SysInfo Utility 144 Performing Remote Detections: The queso Utility 144 Mapping Your Network: The nmap Utility 145 AMANDA 146 III. Commercial Filesystem Backup & Recovery Utilities 185 5. Commercial Backup Utilities 187 What to Look For 188 Full Support of Your Platforms 189 Backup of Raw Partitions 191 Backup of Very Large Filesystems and Files 192 Simultaneous Backup of Many Clients to One Drive 192 Simultaneous Backup of One Client to Many Drives 196 Page ix Data Requiring Special Treatment 202 Storage Management Features 205 Reduction in Network Traffic 208 Support of a Standard or Custom Backup Format 216 Ease of Administration 219 Security 222 Ease of Recovery 223 Protection of the Backup Index 225 Robustness 227 Automation 227 Volume Verification 228 Cost 229 Vendor 230 Conclusions 231 6. High Availability 232 What Is High Availability? 232 HA Building Blocks 238 Commercial HA Solutions 243 The Impact of an HA Solution 245 IV. Bare-Metal Backup & Recovery Methods 247 7. SunOS/Solaris 249 What About Fire? 250 Homegrown Bare-Metal Recovery 251 Recovering a SunOS/Solaris System 256 8. Linux 270 How It Works 270 A Sample Bare-Metal Recovery 275 9. Compaq True-64 Unix 282 Compaq's btcreate Utility 283 Homegrown Bare-Metal Recovery 284 10. HP-UX 290 HP's make_recovery Utility 291 The copyutil Utility 295 Using dump and restore 299 Page x 11. IRIX 306 SGI's Backup and Restore Utilities 307 System Recovery with Backup Tape 310 Homegrown Bare-Metal Recovery 315 12. AIX 323 IBM's mksysb Utility 324 IBM's Sysback/6000 Utility 330 System Cloning 337 V. Database Backup & Recovery 339 13. Backing Up Databases 341 Can It Be Done? 342 Confusion: The Mysteries of Database Architecture 343 The Muck Stops Here: Databases in Plain English 344 What's the Big Deal? 345 Database Structure 346 An Overview of a Page Change 360 What Can Happen to an RDBMS? 361 Backing Up an RDBMS 363 Restoring an RDBMS 370 Documentation and Testing 374 Unique Database Requirements 375 14. Informix Backup & Recovery 376 Informix Architecture 377 Automating Informix Startup: The dbstart.informix.sh Script 387 Protect the Physical Log, Logical Log, and sysmaster 392 Which Backup Utility Should I Use? 400 Physical Backups Without a Storage Manager: ontape 403 Physical Backups with a Storage Manager: onbar 424 Recovering Informix 428 Logical Backups 451 15. Oracle Backup & Recovery 455 Oracle Architecture 455 Physical Backups Without a Storage Manager 463 Physical Backups with a Storage Manager 476 Managing the Archived Redologs 480 Page xi Recovering Oracle 483 Logical Backups 526 A Broken Record 529 16. Sybase Backup & Recovery 531 Sybase Architecture 531 Physical Backups Without a Storage Manager 542 Physical Backups with a Storage Manager 554 Recovering Sybase 554 Logical Backups 583 An Ounce of Prevention 586 VI. Backup & Recovery Potpourri 589 17. ClearCase Backup & Recovery 591 ClearCase Architecture 592 VOB Backup and Recovery Procedures 598 View Backup and Recovery Procedures 608 Summary 615 18. Backup Hardware 616 Choosing on a Backup Drive 616 Using Backup Hardware 621 Tape Drives 625 Optical Drives 635 Automated Backup Hardware 641 Vendors 643 Hardware Comparison 645 19. Miscellanea 649 Volatile Filesystems 649 Demystifying dump 654 Gigabit Ethernet 663 Disk Recovery Companies 664 Yesterday 664 Trust Me About the Backups 665 Index 667 Page xiii PREFACE Like many people, I had to learn backups the hard way. I worked at a large company where I was responsible for backing up Unix SVr3/4, Ultrix, HP-UX 8-10, AIX 3, Solaris 2.3, Informix, Oracle, and Sybase. In those days I barely understood how Unix worked, and I really didn't understand how databases worked-yet it was my responsibility to back it all up. I did what any normal person would do. I went to the biggest bookstore I could find and looked for a book on the subject. There weren't any books on the shelf, so I went to the counter where they could search the Books in Print database. Searching on the word "backup" brought up one book on how to back up Macintoshes. Disillusioned, I did what many other people did: I read the backup chapters in several system and database administration books. Even the best books covered it on only a cursory level, and none of them told me how to automate the backups of 200 Unix machines that ran eight different flavors of Unix and three different database products. Another common problem with these chapters is that they would dedicate 90 percent or more to backup and less than 10 percent to recovery. So my company did what many others had done before us-we reinvented the wheel and wrote our own homegrown utilities and procedures. Then one day I realized that our backup/recovery needs had outgrown our homegrown utilities, which meant that we needed to look at purchasing a commercial utility. Again, there were no resources to help explain the differences between the various backup utilities that were available at that time, so we did what most people do-we talked to the vendors. Since most of the vendors just bashed one another, our job was to try to figure out who was telling the truth and who wasn't. We then wrote a Request For Information (RFI) and a Request For Proposal (RFP) and sent it to the vendors we were considering, whose quotes ranged from Page xiv $16,000 to $150,000. Believe it or not, the least expensive product also did the best on the RFI, and we bought and installed our first commercial backup utility. The day came for me to leave my first backup utility behind, as I was hired by a company that would one day become Collective Technologies. Finally, a chance to get out of backups and become a real system administrator! Interestingly enough, one of my first clients had been performing backups only sporadically, but I discovered that they had a valid license for the commercial product with which I was already familiar. (Imagine the luck.) While rolling out that product, they asked me also to look at how they were backing up their Oracle databases. The next thing I knew, I had ported my favorite Oracle backup script and published it. The response to that article was amazing. People around the world wrote me and thanked me for sharing it, and I caught the publishing bug. One of Collective Technologies' mottos is, ''If something is broken, fix it!" Normally, we're talking about problems within our own company, but I applied it to the backup and recovery industry and the dream of this book was born. I Wish I Had This Book My dream was to write a book that would make sure that no one ever had to start from scratch again, and I believe that my coauthors and I have done just that. It contains every backup tool that I wish I had had when I first entered the Unix business and every lesson and trick that I've learned along the way. It covers how to back up and recover everything from a basic Unix workstation to a complicated Informix, Oracle, or Sybase database. Whether your budget barely stretches to cover the cost of the backup media or allows you to buy a silo bigger than your house, this book has something for you. Whether your task is to figure out how to back up, with no commercial utilities, an environment such as the one I first encountered or to choose from among more than 50 commercial backup utilities, this book will tell you how to do it. With that in mind, let me mention a few things about this book that are unique. Only the Recovery Matters As a friend of mine used to tell me, "No one cares if you can back up-only if you can recover." Yet how many backup chapters have you read that dedicate less than 10 percent to recovery? You won't find that in this book. I have tried very hard to ensure that recovery is given treatment equal to that of backups. In fact, many times it is given greater treatment; the Oracle chapter has more than twice as much space dedicated to the recovery as it does to backups! Page xv Products Change Some people may be surprised that there are no product names mentioned in the commercial backup section. I did this for several reasons, the main one being that products change constantly. It would be impossible to keep this book up to date with the 50 different backup products that are available for Unix. In fact, the book would be out of date by the time it hit the shelves. Instead, this book explains the concepts of commercial backup and recovery software, allowing you to apply those concepts to the claims that the vendors are currently making. Up-to-date information about specific products has been placed on http://www.backupcentral.com. Backing Up Databases Is Not That Hard If you're a database administrator (DBA), you may not be familiar with the Unix backup commands necessary to back up your database. If you're a system administrator (SA), you may not be familiar with the architecture of your particular database platform. Both of these concepts are explained in detail in this book. I explain the backup utilities in plain language so that any DBA can understand them, and I explain database architecture in such a way that an SA, even one who has never before seen a database, can understand it. Bare-Metal Recovery Is Not That Hard One of these days you will lose the operating system disk for an important system, and you will need to recover it. This is called a "bare-metal recovery." The standard recovery method described in many backups products' documentation is to install a minimal operating system and restore on top of it. This is the worst possible method to do a bare-metal recovery of a Unix system; among other problems, you end up overwriting some of the system files while the system is running from the very disk to which you are trying to restore. The best ways to do bare-metal recoveries for six different versions of Unix are covered in detail in this book. The Scripts in This Book Actually Work Nothing bugs me more than to read a book in which the author talks about a really neat program, only to find out that the program is so full of bugs it won't work. Most of the programs in this book are already running at hundreds of sites around the world. With all the typical "unsupported" disclaimers in place, I do my best to ensure that they continue to work for the people who use them. If you're Page xvi interested in any of the programs in the book (and on the CD), make sure that you subscribe to the appropriate mailing list on http://www.backupcentral.com. I will provide updates as they become available. How This Book is Organized This book is divided into six parts: Part I, Introduction This part of this book contains just enough information to whet your backup and recovery appetite. Chapter 1, Preparing for the Worst, contains the six steps that you must go through to create and maintain a disaster recovery plan, one part of which will be a good backup and recovery system. Chapter 2, Backing It All Up, goes into detail about the essential elements of a good backup and recovery system. Part II, Freely Available Filesystem Backup & Recovery Utilities This section covers the freely available utilities that you can use to back up your systems if you can't afford a commercial backup package. Chapter 3, Native Backup & Recovery Utilities, covers Unix's native backup and recovery utilities in detail, including dump, tar, GNU tar, cpio, GNU cpio, and dd. Chapter 4, Free Backup Utilities, starts with some simple tools to assist you in your backups, and contains a complete overview of the popular AMANDA utility, which is used to back up many small to medium-sized Unix installations around the world. Part III, Commercial Filesystem Backup & Recovery Utilities If you have outgrown the capabilities of free utilities, or would just like to take advantage of new backup and recovery technologies, you'll need to look at a commercial product. Chapter 5, Commercial Backup Utilities, is your guide to the hundreds of features available in the over 50 commercial backup products available on the market today, allowing you to make an educated purchase decision. Page xvii Chapter 6, High Availability, details how, when backups just aren't fast enough, a high availability system is designed to keep you from ever needing to use your backups. Part IV, Bare-Metal Backup & Recovery Methods A bare-metal recovery is the fastest way to bring a dead system back to life, even if its root drive is completely destroyed. Chapter 7, SunOS/Solaris, contains an in-depth description of the "homegrown" bare-metal recovery procedure that can also be used to back up Linux, Compaq, HP- UX, and IRIX, as well as a detailed Solaris-based example of bare-metal recovery. Chapter 8, Linux, detail how you can perform a bare-metal recovery of a Linux system with a floppy, a backup device, pax, and lilo. Chapter 9, Compaq True-64 Unix, covers both Compaq True-64 Unix's bare-metal recovery tool and the Compaq version of the homegrown procedure covered in Chapter 7. Chapter 10, HP-UX, covers the make_recovery tool, which now comes with HP-UX to perform bare-metal recoveries, along with the HP version of the homegrown procedure. Chapter 11, IRIX, explains how the different versions of IRIX's Backup and Restore scripts work, as well as the IRIX version of the homegrown procedure. Chapter 12, AIX, discusses AIX, a procedure that does not support the homegrown procedure discussed in Chapter 7, but does use mksysb, probably one of the oldest and best-known bare-metal recovery tools. Part V, Database Backup & Recovery This section explains in plain language an area that presents some of the greatest backup and recovery challenges that a system administrator or database administrator will face-backing up and recovering databases. Chapter 13, Backing Up Databases, is a chapter that will be your friend if you're an SA who's afraid of databases or a DBA learning a new database. It explains database architecture in plain language, while relating each architectural element to the appropriate term in Informix, Oracle, and Sybase. Chapter 14, Informix Backup & Recovery, explains both the older ontape and the newer onbar, after which it provides a logically flowcharted recovery procedure that can be used with either utility. Page xviii Chapter 15, Oracle Backup & Recovery, explains how to perform Oracle hot backups whether you are using Oracle's native utilities, EBU, or RMAN, and then provides a detailed flowchart guiding you through even a difficult recovery. Chapter 16, Sybase Backup & Recovery, shows exactly how to use the Backup Server utility, including another flow chart to guide you through Sybase recoveries. Part VI, Backup & Recovery Potpourri The information contained in this part of the book is by no means unimportant; it simply wouldn't fit anywhere else! Chapter 17, ClearCase Backup & Recovery, explains in detail the unique backup and recovery challenges presented by ClearCase. Chapter 18, Backup Hardware, explains the many different types of backup hardware available today, as well as providing criteria that you may use to decide which type of backup drive is right for you. Chapter 19, Miscellanea, covers everything from the oft-debated "live filesystem dumps" question to a few jokes that I found about backup and recovery! Conventions The following typographical conventions are used in this book: Constant width Is used to indicate command-line computer output, computer-generated messages, and code examples. It is also used when referring to parameters in text. Constant width italic Is used to indicate variables in examples and text, and comments in examples. Constant width bold Is used to indicate user input in examples. Italic Is used to introduce new terms and to indicate URLs, variables or files and directories, commands, file extensions, filenames, and directory names. How to Contact Us We have tested and verified all the information in this book to the best of our ability, but you may find that features have changed (or even that we have made mistakes!). Please let us know about any errors you find, as well as your suggestions for future editions, by writing to: Page xix O'Reilly & Associates 101 Morris Street Sebastopol, CA 95472 1-800-998-9938 (in the U.S. or Canada) 1-707-829-0515 (international/local) 1-707-829-0104 (fax) You can also send messages electronically. To be put on our mailing list or to request a catalog, send email to: nuts@oreilly.com To ask technical questions or comment on the book, send email to: bookquestions@oreilly.com This Book Was a Team Effort I have never worked with a group of people like the ones I work with at Collective Technologies. Over the past three years, they have answered question after question about the various ways to back up and recover just about everything under the sun. Thanks to them, there is information in this book that would never have been otherwise. They sent me manpages and verified syntax for commands on versions of Unix that I've never even seen. They entered into technical debates about how to compare the architectures of Informix, Oracle, and Sybase. They tested the programs that are included in this book and even wrote a few of them. By far the greatest contribution that other people gave to this book is that several of the chapters were written by experts in a particular field. I realized about a year ago that I would never finish this book if I didn't ask some of my friends to help. The result was that more than 20 percent of the final book ended up being written by people other than me. Their expertise in a particular area made their chapters far better than anything I could have written on my own. Having said that, please allow me to formally thank all my of my coauthors: AIX bare-metal recovery Charles Gagnon and Brian Jensen of Collective Technologies AMANDA John R. Jackson and Alexandre Oliva from the AMANDA Core Development Team Clearcase backup and recovery Bob Fulwiler of Seattle, Washington Compaq/Digital Unix bare-metal recovery Matthew Huff of Collective Technologies Page xx Dump internals David Young of Collective Technologies High-availability systems Josh Newcomb and Gustavo Vegas of Collective Technologies HP-UX bare-metal recovery Steve Ferguson of Collective Technologies IRIX bare-metal recovery Blayne Puklich of Collective Technologies Sybase backup and recovery Bryn Smith of Collective Technologies Without these folks, either the book would never have been completed or it would contain substantially less data than the book you see today. Another group of people that I must thank is my technical reviewers. If every book's author had the team of technical reviewers I had, the world would contain far less misinformation. This book was actually reviewed on an ongoing basis by a number of Collective Technologies people. I set up an RCS system that allowed a team of about 30 reviewers to actually check out my chapters and edit them. They constantly kept me in check, identifying parts of the book that were inaccurate or that needed clarification. You can't imagine the benefit of having such a great team looking over your shoulder. This special ongoing technical review team consisted of: Scott Aschenbach Michael Clark Norman Hill Jason Perkins Rusty Atkins Nancy Cortez Todd Holloway Stephen Potter Ed Bailey Jim Donnelan Bill Huff Jason Stege David Bajot William Duffy Paul Iadonisi Vince Taluskie Mike Bush Steve Ferguson Brian Jensen Gustavo Vegas [...]... don't underestimate the value of an inventory printed on paper and stored off-site That paper may just get you out of a Page 33 catch-22 Given the single-point-of-failure factor, the recovery of your backup server should be the easiest and best-documented recovery that you have You may even want to investigate creating a special dump or tar backup of that data to make it even easier to recover during... both remote and local backups with this program, test both on a small scale • Try to simulate every potential error the program might encounter: - Eject a volume in the middle of the backup - Write-protect a volume - Reboot the system you are backing up while it is backing up - Drop the network connection and power down a disk drive - Know the program and the errors for which it is testing, and simulate... change you make might make your backup system that much more complex -and more difficult to explain to the new backup person One of the heads of support for a commercial backup product said that he sees the same thing over and over again One person gets to know the software really well, and writes various scripts to automate this and that Backups become a well-oiled machine-until they are turned over to... trust your backups, they will duplicate your backup efforts Employees will spend time and money backing up their systems locally Each person may decide to buy his own backup drive and backup software or even to come up with his own in-house script Their backups will be inefficient and costly at best and subject them to further data loss at worst When everybody takes matters into her own hands, you can... great backup system but aren't storing your media off-site, you'll be sorry when that tornado hits You may have the most well organized, well protected set of backup volumes,* but they won't be of any help if your backup and recovery system hasn't properly stored the data on those volumes Getting good backups may be an early step in your disaster recovery plan, but the rest of that plan-organizing and. .. but do you know what models they are? If you have all Brand-X 2.9gigabyte drives, then you have no problem, but many servers have a mixture of drives that were installed over time You may have a collection of 1-GB, 2-GB, 2.01GB, 2.1-GB, 2.9-GB, 4-GB and 9-GB drives, all on the same system Make sure that you are recording this in some way Most Unix systems record this already, by the way, usually in... down, and try to re-create it • Pretend that a particular volume is bad, and force yourself to use an alternate backup • Retrieve a few volumes from your off-site storage vendor • Pretend that your backup server is destroyed, and try to recover from that (This one's tough!) This test is extremely important if you are using a commercial backup utility Some products do not plan for this well, and you... it with dump, tar, cpio, and your standard database backup utilities, you couldn't do it The demand for midrange computers has grown astronomically in the last few year, and the need for bigger databases, larger filesystems, long filenames, and long pathnames grew proportionally As things typically go in the backup world, large filesystems and huge databases were designed and shipped long before the... fact, a previous version of my Oracle backup script (see Chapter 15, Oracle Backup & Recovery) did not back up the online redologs during a hot backup All my backup and recovery tests worked fine, until I attempted to restore the database to a different system We were able to restore all the database files, but the database needed the redologs in order to complete the recovery Since we had not backed up... matter what department you look at, if they do their work on a computer and you lose that data, you can lose considerable time, and money, in lost work What Will Downtime Cost You? When planning your backup and recovery program, you may have several options that will affect the speed of the recovery The faster the recovery, the more the backup system will cost you What you must ask yourself before deciding . perform a bare-metal recovery of a Linux system with a floppy, a backup device, pax, and lilo. Chapter 9, Compaq True-64 Unix, covers both Compaq True-64 Unix& apos;s bare-metal recovery tool and the. coauthors: AIX bare-metal recovery Charles Gagnon and Brian Jensen of Collective Technologies AMANDA John R. Jackson and Alexandre Oliva from the AMANDA Core Development Team Clearcase backup and recovery Bob. oldest and best-known bare-metal recovery tools. Part V, Database Backup & Recovery This section explains in plain language an area that presents some of the greatest backup and recovery