Page iii Unix Backup and Recovery W. Curtis Preston Beijing • Cambridge • Farnham • Köln • Paris • Sebastopol • Taipei • Tokyo Page iv Disclaimer: This netLibrary eBook does not include data from the CD-ROM that was part of the original hard copy book. Unix Backup and Recovery by W. Curtis Preston Copyright (c) 1999 O'Reilly & Associates, Inc. All rights reserved. Printed in the United States of America. Published by O'Reilly & Associates, Inc., 101 Morris Street, Sebastopol, CA 95472. Editor: Gigi Estabrook Production Editor: Clairemarie Fisher O'Leary Printing History: November 1999: First Edition. Nutshell Handbook, the Nutshell Handbook logo, and the O'Reilly logo are registered trademarks of O'Reilly & Associates, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O'Reilly & Associates, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps. The association between the image of an Indian gavial and the topic of Unix backup and recovery is a trademark of O'Reilly & Associates, Inc. While every precaution has been taken in the preparation of this book, the publisher assumes no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. This book is printed on acid-free paper with 85% recycled content, 15% post-consumer waste. O'Reilly & Associates is committed to using paper with the highest recycled content available consistent with high quality. ISBN: 1-56592-642-0 Page v This book is dedicated to my lovely wife Celynn, my beautiful daughters Nina and Marissa, and to God, for continuing to bless my life with gifts such as these. -W. Curtis Preston Page vii TABLE OF CONTENTS Preface xiii I. Introduction 1 1. Preparing for the Worst 3 My Dad Was Right 3 Developing a Disaster Recovery Plan 4 Step 1: Define (Un)acceptable Loss 5 Step 2: Back Up Everything 7 Step 3: Organize Everything 10 Step 4: Protect Against Disasters 13 Step 5: Document What You Have Done 15 Step 6: Test, Test, Test 16 Step 6: Test, Test, Test 16 Put It All Together 17 2. Backing It All Up 18 Don't Skip This Chapter! 18 Why Should You Read This Book? 19 How Serious Is Your Company About Backups? 22 You Can Find a Balance 25 Deciding What to Back Up 30 Deciding When to Back Up 38 Deciding How to Back Up 43 Storing Your Backups 52 Testing Your Backups 56 Monitoring Your Backups 58 Page viii Following Proper Development Procedures 59 Unrelated Miscellanea 60 Good Luck 65 II. Freely Available Filesystem Backup & Recovery Utilities 67 3. Native Backup & Recovery Utilities 69 An Overview 69 Backing Up with the dump Utility 73 Restoring with the r estore Utility 91 Restoring with the r estore Utility 91 Limitations of dump and restore 101 Features to Check For 102 Backing Up and Restoring with the cpio Utility 103 Backing Up and Restoring with the tar Utility 114 Backing Up and Restoring with the dd Utility 122 Comparing tar, cpio, and dump 127 How Do I Read This Volume? 129 4. Free Backup Utilities 141 The hostdump.sh Utility 141 The infback.sh, oraback.sh, and syback.sh Utilities 142 A Really Fast tar Utility: star 142 Recording Configuration Data: The SysAudit Utility 143 Displaying Host Information: The SysInfo Utility 144 Performing Remote Detections: The queso Utility 144 Mapping Your Network: The nmap Utility 145 AMANDA 146 III. Commercial Filesystem Backup & Recovery Utilities 185 5. Commercial Backup Utilities 187 What to Look For 188 Full Support of Your Platforms 189 Backup of Raw Partitions 191 Backup of Very Large Filesystems and Files 192 Simultaneous Backup of Many Clients to One Drive 192 Simultaneous Backup of Many Clients to One Drive 192 Simultaneous Backup of One Client to Many Drives 196 Page ix Data Requiring Special Treatment 202 Storage Management Features 205 Reduction in Network Traffic 208 Support of a Standard or Custom Backup Format 216 Ease of Administration 219 Security 222 Ease of Recovery 223 Protection of the Backup Index 225 Robustness 227 Automation 227 Volume Verification 228 Cost 229 Vendor 230 Conclusions 231 6. High Availability 232 What Is High Availability? 232 HA Building Blocks 238 Commercial HA Solutions 243 The Impact of an HA Solution 245 IV. Bare-Metal Backup & Recovery M ethods 247 IV. Bare-Metal Backup & Recovery M ethods 247 7. SunOS/Solaris 249 What About Fire? 250 Homegrown Bare-Metal Recovery 251 Recovering a SunOS/Solaris System 256 8. Linux 270 How It Works 270 A Sample Bare-Metal Recovery 275 9. Compaq True-64 Unix 282 Compaq's btcreate Utility 283 Homegrown Bare-Metal Recovery 284 10. HP-UX 290 HP's make_recovery Utility 291 The copyutil Utility 295 Using dump and restore 299 Page x 11. IRIX 306 SGI's Backup and Restore Utilities 307 System Recovery with Backup Tape 310 Homegrown Bare-Metal Recovery 315 12. AIX 323 IBM's mksysb Utility 324 IBM's Sysback/6000 Utility 330 IBM's Sysback/6000 Utility 330 System Cloning 337 V. Database Backup & Recovery 339 13. Backing Up Databases 341 Can It Be Done? 342 Confusion: The Mysteries of Database Architecture 343 The Muck Stops Here: Databases in Plain English 344 What's the Big Deal? 345 Database Structure 346 An Overview of a Page Change 360 What Can Happen to an RDBMS? 361 Backing Up an RDBMS 363 Restoring an RDBMS 370 Documentation and Testing 374 Unique Database Requirements 375 14. Informix Backup & Recovery 376 Informix Architecture 377 Automating Informix Startup: The dbstart.informix.sh Script 387 Protect the Physical Log, Logical Log, and sysmaster 392 Which Backup Utility Should I Use? 400 Physical Backups Without a Storage Manager: ontape 403 Physical Backups with a Storage Manager: onbar 424 Recovering Informix 428 Logical Backups 451 Logical Backups 451 15. Oracle Backup & Recovery 455 Oracle Architecture 455 Physical Backups Without a Storage Manager 463 Physical Backups with a Storage Manager 476 Managing the Archived Redologs 480 Page xi Recovering Oracle 483 Logical Backups 526 A Broken Record 529 16. Sybase Backup & Recovery 531 Sybase Architecture 531 Physical Backups Without a Storage Manager 542 Physical Backups with a Storage Manager 554 Recovering Sybase 554 Logical Backups 583 An Ounce of Prevention 586 VI. Backup & Recovery Potpourri 589 17. ClearCase Backup & Recovery 591 ClearCase Architecture 592 VOB Backup and Recovery Procedures 598 View Backup and Recovery Procedures 608 Summary 615 Summary 615 18. Backup Hardware 616 Choosing on a Backup Drive 616 Using Backup Hardware 621 Tape Drives 625 Optical Drives 635 Automated Backup Hardware 641 Vendors 643 Hardware Comparison 645 19. Miscellanea 649 Volatile Filesystems 649 Demystifying dump 654 Gigabit Ethernet 663 Disk Recovery Companies 664 Yesterday 664 Trust Me About the Backups 665 Index 667 Page xiii PREFACE Like many people, I had to learn backups the hard way. I worked at a large company where I was responsible for backing up Unix SVr3/4, Ultrix, HP-UX 8-10, AIX 3, Solaris 2.3, Informix, Oracle, and Sybase. In those days I barely understood how Unix worked, and I really didn't understand how databases worked-yet it was my responsibility to back it all up. I did what any normal person would do. I went to the biggest bookstore I could find and looked for a book on the subject. There weren't any books on the shelf, so I went to the counter where they could search the Books in Print database. Searching on the word "backup" brought up one book on how to back up Macintoshes. Disillusioned, I did what many other people did: I read the backup chapters in several system and database administration books. Even the best books covered it on only a cursory level, and none of them told me how to automate the backups of 200 Unix machines that ran eight different flavors of Unix and three different database products. Another common problem with these chapters is that they would dedicate 90 percent or more to backup and less than 10 percent to recovery. So my company did what many others had done before us-we reinvented the wheel and wrote our own homegrown utilities and procedures. Then one day I realized that our backup/recovery needs had outgrown our homegrown utilities, which meant that we needed to look at purchasing a commercial utility. Again, there were no resources to help explain the differences between the various backup utilities that were available at that time, so we did what most people do-we talked to the vendors. Since most of the vendors just bashed one another, our job was to try to figure out who was telling the truth and who wasn't. We then wrote a Request For Information (RFI) and a Request For Proposal (RFP) and sent it to the vendors we were considering, whose quotes ranged from Page xiv $16,000 to $150,000. Believe it or not, the least expensive product also did the best on the RFI, and we bought and installed our first commercial backup utility. The day came for me to leave my first backup utility behind, as I was hired by a company that would one day become Collective Technologies. Finally, a chance to get out of backups and become a real system administrator! Interestingly enough, one of my first clients had been performing backups only sporadically, but I discovered that they had a valid license for the commercial product with which I was already familiar. (Imagine the luck.) While rolling out that product, they asked me also to look at how they were backing up their Oracle databases. The next thing I knew, I had ported my favorite Oracle backup script and published it. The response to that article was amazing. People around the world wrote me and thanked me for sharing it, and I caught the publishing bug. One of Collective Technologies' mottos is, ''If something is broken, fix it!" Normally, we're talking about problems within our own company, but I applied it to the backup and recovery industry and the dream of this book was born. I Wish I Had This Book My dream was to write a book that would make sure that no one ever had to start from scratch again, and I believe that my coauthors and I have done just that. It contains every backup tool that I wish I had had when I first entered the Unix business and every lesson and trick that I've learned along the way. It covers how to back up and recover everything from a basic Unix workstation to a complicated Informix, Oracle, or Sybase database. Whether your budget barely stretches to cover the cost of the backup media or allows you to buy a silo bigger than your house, this book has something for you. Whether your task is to figure out how to back up, with no commercial utilities, an environment such as the one I first encountered or to choose from among more than 50 commercial backup utilities, this book will tell you how to do it. With that in mind, let me mention a few things about this book that are unique. [...]... Compaq, HP-UX, and IRIX, as well as a detailed Solaris-based example of bare-metal recovery Chapter 8, Linux, detail how you can perform a bare-metal recovery of a Linux system with a floppy, a backup device, pax, and lilo Chapter 9, Compaq True-64 Unix, covers both Compaq True-64 Unix' s bare-metal recovery tool and the Compaq version of the homegrown procedure covered in Chapter 7 Chapter 10, HP-UX, covers... element of a disaster recovery plan -the backup and recovery of your data Page 18 2 Backing It All Up In Chapter 1, Preparing for the Worst, we looked at disaster recovery as a whole The nuts and bolts of backup and recovery are but a small part of the overall disaster recovery picture Before we begin looking at the details of how to perform certain types of backups, let's look at backups in general Don't... Oracle backup script (see Chapter 15, Oracle Backup & Recovery) did not back up the online redologs during a hot backup All my backup and recovery tests worked fine, until I attempted to restore the database to a different system We were able to restore all the database files, but the database needed the redologs in order to complete the recovery Since we had not backed up the redologs, we did not have them... recorded for you if you use the SysAudit and SysInfo programs Step 3: Organize Everything Good organization is really the key to a good disaster recovery plan If you have hundreds or thousands of backup volumes but can't find them if you need them, what good are they? There is also the physical layout of the servers themselves If they are all laid out in a standard way, recovering from a disaster is a whole... understand it Bare-Metal Recovery Is Not That Hard One of these days you will lose the operating system disk for an important system, and you will need to recover it This is called a "bare-metal recovery. " The standard recovery method described in many backups products' documentation is to install a minimal operating system and restore on top of it This is the worst possible method to do a bare-metal recovery. .. decide to send them off-site only once a week On the other hand, the cost of re-creating that data must be taken into account, and you may not want to explain to a group of 200 developers why they have to re-create everything they did last week If that is the case, then you have defined that losing more than one day's worth of anyone's work is unacceptable Great! That's the purpose of this step There are... help if your backup and recovery system hasn't properly stored the data on those volumes Getting good backups may be an early step in your disaster recovery plan, but the rest of that plan-organizing and protecting those backups against a disaster-should follow soon after Although the task may seem daunting, it's not impossible Developing a Disaster Recovery Plan Devising a good disaster recovery plan... Unix' s native backup and recovery utilities in detail, including dump, tar, GNU tar, cpio, GNU cpio, and dd Chapter 4, Free Backup Utilities, starts with some simple tools to assist you in your backups, and contains a complete overview of the popular AMANDA utility, which is used to back up many small to medium-sized Unix installations around the world Part III, Commercial Filesystem Backup & Recovery Utilities... good backup and recovery system Chapter 2, Backing It All Up, goes into detail about the essential elements of a good backup and recovery system Part II, Freely Available Filesystem Backup & Recovery Utilities This section covers the freely available utilities that you can use to back up your systems if you can't afford a commercial backup package Chapter 3, Native Backup & Recovery Utilities, covers Unix' s... people like the ones I work with at Collective Technologies Over the past three years, they have answered question after question about the various ways to back up and recover just about everything under the sun Thanks to them, there is information in this book that would never have been otherwise They sent me manpages and verified syntax for commands on versions of Unix that I've never even seen They entered . a bare-metal recovery of a Linux system with a floppy, a backup device, pax, and lilo. Chapter 9, Compaq True-64 Unix, covers both Compaq True-64 Unix& apos;s bare-metal recovery tool and the Compaq. coauthors: AIX bare-metal recovery Charles Gagnon and Brian Jensen of Collective Technologies AMANDA John R. Jackson and Alexandre Oliva from the AMANDA Core Development Team Clearcase backup and recovery Bob. & Associates 101 Morris Street Sebastopol, CA 95472 1-8 0 0-9 9 8-9 938 (in the U.S. or Canada) 1-7 0 7-8 2 9-0 515 (international/local) 1-7 0 7-8 2 9-0 104 (fax) You can also send messages electronically.