Restoring data with ibbackup Restoring data requires applying the log described in the previous section. This oper- ation essentially points the MySQL instance to the backup copies of the database. It does not copy the files to the normal MySQL datadir location. If you want to do this, you must copy the files manually or use the innobackup script. The reason for this is that the ibbackup utility is designed to refuse to overwrite any data. To start a MySQL instance and use a backup of your data, execute a command like the following: mysqld -defaults-file=/home/cbell/backup.cnf The innobackup script The innobackup file is a Perl script designed to automate many of the operations of ibbackup. It can create backups; restore data; start a MySQL instance using the data from a backup; or copy data, index, and logfiles from a backup directory back to their original locations. The innobackup script is currently not available for use on Windows. Unlike the ibbackup utility, you do not need to specify the commands in a separate configuration file for innobackup. Rather, you specify the parameters for the backup or restore using command-line options. These options are described in Table 12-2. Table 12-2. innobackup options Option Function --help Displays a list of all options. --version Displays the version of the script. --apply-log Applies the backup log to the backup in preparation for starting a MySQL server with the backup files. --copy-back Copies data and index files from the backup location to their original locations. --use-memory=MB Passed to ibbackup, this option controls how much memory is used during restoration. --sleep=MS Passed to ibbackup, this option causes the utility to pause after every 1 Mb of data is copied. --compress=LEVEL Passed to ibbackup, this option provides the compression level to use. --include=REGEXP Passed to ibbackup, this option instructs the process to back up only those table files that match the regular expression. This is used for a selective backup. --uncompress Passed to ibbackup, this option uncompresses a compressed backup. --user=NAME Username to use for connecting to the server. --password=PWD Password for the user. --port=PORT Port for the server. --socket=SOCK Socket of the server. Backup Utilities and OS-Level Solutions | 427 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Performing a backup with innobackup To create a backup, you need only two options: the configuration file of the server and the location for your backup files: perl innobackup /etc/mysql/my.cnf /home/cbell/backup If you want a consistent backup, specify the --apply-log option as well: perl innobackup --apply-log /etc/mysql/my.cnf /home/cbell/backup Restoring data with innobackup To restore data, apply the log and use the --copy-back option to copy the files to the original location. We show a sample of these commands below. You must stop the server before the copy and then restart it afterward. perl innobackup --apply-log /etc/mysql/my.cnf /home/cbell/backup mysqladmin -uroot shutdown perl innobackup --copy-file /etc/mysql/my.cnf /home/cbell/backup /etc/init.d/mysql start Additional features InnoDB Hot Backup also supports PITR. See the InnoDB Hot Backup documenta tion for more details about this and other advanced features. You can download a trial version of InnoDB Hot Backup for evaluation purposes. This trial is active for 30 days, during which you can use InnoDB Hot Backup without limitations. To sign up for a free trial copy, visit http://www.innodb.com/products/hot-backup/order/order/. Physical File Copy The easiest and most basic form of backup for MySQL is a simple file copy. Unfortu- nately, this requires you to stop the server for best results. To perform a file copy, simply stop your server and copy the data directory and any setup files on the server. One common method for this is to use the Unix tar command to create an archive. You can then move this archive to another system and restore the data directory. The following is a typical sequence of tar commands that backs up the data from one database server and restores it on another system. Execute the following command on the server you want to back up, where backup_2009_09_09.tar.gz is the file you want to create and /usr/loca/mysql/data is the path for the data directory: tar -czf backup_2009_09_09.tar.gz /usr/loca/mysql/data/* The backup_2009_09_09.tar.gz file must reside on a directory shared by the two servers (or you must physically copy it to the new server). Now, on the server where you want 428 | Chapter 12: Protecting Your Investment Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. to restore the data, change to the root installation of your new installation of MySQL. Delete the existing data directory, if it exists, then execute the following: tar -xvf /backup_2009_09_09.tar.gz As mentioned earlier, it is always a good idea to use meaningful file- names for your backup images. As you can see from this example, it is very easy to back up your data at the operating system level. Not only can you get a single compressed archive file that you can move around easily, you also get the benefit of fast file copy. You can even do a selective backup by simply copying individual files or subdirectories from your data directory. Unfortunately, the tar command is available only on Linux and Unix platforms. If you have Cygwin installed on a Windows system and have included its version of the com- mand, you can also use tar there. Other than Cygwin or another Unix-on-Windows package, the closest equivalent to this command on Windows is the handy folder archive feature of the Explorer or an archive program such as WinZip. To do this, open Windows Explorer and navigate to your data directory. But instead of opening the directory, right-click the data directory and choose an option that compresses the data, which will have a label such as Send To→Compressed (zipped) Folder, and then provide a name for the .zip file. While a physical file copy is the quickest and easiest form of backup, it does require that you shut down the server. But that isn’t necessary if you are careful to ensure there are no updates occurring during the file copy. To do this, you must lock all tables and perform a flush tables command, then take your server offline before making the file copy. This is similar to the process for cloning a slave. See Chapter 2 for more details and an example of cloning a slave using file copy. Additionally, depending on the size of the data, your server must be offline not only for the time to copy the files, but also for any additional data loads like cache entries, the use of memory tables for fast lookups, etc. For this reason, physical copy backup may not be feasible for some installations. Fortunately, there is a Perl script, created by Tim Bunce, to automate this process. The name of the script is mysqlhotcopy.sh and it is located in the ./scripts folder of your MySQL installation. It allows you to make hot copies of databases. However, you can use it only to back up MyISAM or Archive storage engines and it works only on Unix and Netware operating systems. Backup Utilities and OS-Level Solutions | 429 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. The mysqlhotcopy.sh utility also includes customization features. You can find more information about it at http://dev.mysql.com/doc/refman/5.4/en/mysqlhotcopy.html. The mysqldump Utility The most popular alternative to the physical file copy feature is the mysqldump client application. It has been part of the MySQL installation for some time and was originally donated to MySQL by Igor Romanenko. mysqldump creates a set of SQL statements that re-create the databases when you rerun them. For example, when you run a backup, the output contains all of the CREATE statements needed to create the databases and the tables they contain, as well as all the INSERT statements needed to re-create the data in those tables. This can be very handy if you need to do a search-and-replace operation in the text of your data. Simply back up your database, edit the resulting file with a text editor, then restore the database to effect the changes. Many MySQL users use this technique to correct all sorts of errors caused by making batch edits to the data. You will find this much easier than writing, say, 1,000 UPDATE statements with complicated WHERE clauses. The drawbacks of using mysqldump are that it takes a lot more time than the binary copies made by file-level (physical) backups like InnoDB Hot Backup, LVM, or a simple offline file copy, and it requires a lot more storage space. This cost in time can be significant if you make frequent backups, want to restore a database quickly after a system failure, or need to transfer the backup file across a network. You can use mysqldump to back up all your databases, a specific subset of databases, or even particular tables within a given database. The following examples show each of these options: mysqldump -uroot -all-databases mysqldump -uroot db1, db2 mysqldump -uroot my_db t1 You can also use mysqldump to do a hot backup of InnoDB tables. The --single-trans action option issues a BEGIN statement at the start of the backup, which signals the InnoDB storage engine to read the tables as a consistent read. Thus, any changes you make are applied to the tables, but the data is frozen at the time of backup. However, no other connection should use data definition language (DDL) statements like ALTER TABLE, DROP TABLE, RENAME TABLE, TRUNCATE TABLE. This is because a consistent read is not isolated from DDL changes. The --single-transaction option and the --lock-tables option are mutually exclusive because LOCK TABLES issues an implicit commit. 430 | Chapter 12: Protecting Your Investment Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. The utility has several options that control the backup as well as what is included. Table 12-3 describes some of the more important options. See the online MySQL Ref- erence Manual at http://dev.mysql.com/doc/refman/5.4/en/mysqldump.html for a com- plete set of options. Table 12-3. mysqlbackup options Option Function --add-drop-database Includes a DROP DATABASE statement before each database. --add-drop-table Includes a DROP TABLE statement before each table. --add-locks Surrounds each included table with LOCK TABLES and UNLOCK TABLES. --all-databases Includes all databases. --create-options Includes all MySQL-specific table options in the CREATE TABLE statements. --databases Includes a list of databases only. --delete-master-logs On a master, deletes the binary logs after performing the backup. --events Backs up events from the included databases. --extended-insert Uses the alternative INSERT syntax that includes each row as a VALUES clause. --flush-logs Flushes the logfiles before starting the backup. --flush-privileges Includes a FLUSH PRIVILEGES statement after backing up the mysql database. --ignore-table=db.tbl Does not back up the specified table. --lock-all-tables Locks all tables across all databases during the dump. --lock-tables Locks all tables before including them. --log-error=filename Appends warnings and errors to the specified file. --master-data[=value] Includes the binlog filename and position in the output. --no-data Does not write any table row information (only CREATE statements). --password[=password] The password to use when connecting to the server. --port=port_num The TCP/IP port number to use for the connection. --result-file=filename Outputs to a specific file. --routines Includes stored routines (procedures and functions). --single-transaction Issues a BEGIN SQL statement before dumping data from the server. This allows for a consistent snapshot of the InnoDB tables. --tables Overrides the --databases option. --triggers Includes triggers. --where='condition' Includes only rows selected by the condition. --xml Produces XML output. Backup Utilities and OS-Level Solutions | 431 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. You can also include these options in a MySQL configuration file under the heading [mysqldump]. In most cases, you can specify the option sim- ply by removing the initial dashes. For example, to always produce XML output, include xml in your configuration file. One very handy feature of mysqldump is the ability to dump a database schema. You can normally do this using a set of the CREATE commands to re-create all of the objects without the INSERT statements that include the data. This usage can be very useful for keeping a historical record of the changes to your schema. If you use the --no-data option along with the options to include all of the objects (e.g., --routines, --triggers), you can use mysqldump to create a database schema. Notice the option --master-data. This option can be very helpful for performing PITR because it saves the binary log information like InnoDB Hot Backup does. There are many more options that allow you to control how the utility works. If creating a backup in the form of SQL statements sounds like the best option for you, feel free to explore the rest of the options for making mysqldump work for you. XtraBackup Percona, an independent open source provider and consulting firm specializing in all things MySQL (LAMP, actually), has created a storage engine called XtraDB, which is an open source storage engine based on the InnoDB storage engine. XtraDB has several improvements for better scaling on modern hardware and is backward compatible with InnoDB. In an effort to create a hot backup solution for XtraDB, Percona has created XtraBackup. This tool is optimized for InnoDB and XtraDB, but can also back up and restore MyISAM tables. It provides many of the features expected of backup solutions, in- cluding compression and incremental backup. You can download and build XtraBackup by getting the source code from Launchpad at https://launchpad.net/percona-xtrabackup. You can compile and execute XtraBackup on most platforms. It is compatible with MySQL versions 5.0 and 5.1. The online manual for XtraBackup is located at http://www.percona.com/docs/wiki/per cona-xtrabackup:xtrabackup_manual. Logical Volume Manager Snapshots Most Linux and some Unix systems provide another powerful method of backing up your MySQL database. It makes use of a technology called the logical volume man- ager (LVM). 432 | Chapter 12: Protecting Your Investment Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Microsoft Windows has a similar technology called Volume Shadow Copy. Unfortunately, there are no generic utilities to make a snapshot of a random partition or folder structure as there are for LVM. You can, however, make snapshots of an entire drive, which can be useful if your database directory is the only thing on that drive. See the Microsoft online documentation for more information. An LVM is a disk subsystem that gives you a lot of administrative power to create, remove, and resize volumes easily and quickly without using the older, often compli- cated and unforgiving disk tools. The added benefit for backup is the concept of taking a snapshot—that is, a copy of an active volume—without disrupting the applications that access the data on that vol- ume. The idea is to take a snapshot, which is a relatively fast operation, and then back up the snapshot instead of the original volume. Deep inside LVM, a snapshot is man- aged using a mechanism that keeps track of the changes since you took the snapshot, so that it stores only the disk segments that have changed. Thus, a snapshot takes up less space than a complete copy of the volume and when the backup is made, the LVM copies the files as they existed at the time of the snapshot. Snapshots effectively freeze the data. Another benefit of using LVM and snapshots for backing up database systems lies in how you use the volumes. The best practice is to use a separate volume for each of your MySQL installations so that all of the data is on the same volume, allowing you to create a backup quickly using a snapshot. Of course, it is also possible to use multiple logical volumes in some situations, such as using one logical volume for each tablespace or even different logical volumes for MyISAM and InnoDB tables. Getting started with LVM If your Linux installation does not have LVM installed, you can install it using your package manager. For example, on Ubuntu you can install LVM using the following command: sudo apt-get install lvm2 Although not all LVM systems are the same, the following procedure is based on a typical Debian distribution and works well on systems like Ubuntu. We don’t mean to write a complete tutorial on LVM but just to give you an idea of the complexity of using LVM for making database backups. Consult your operating system documentation for specifics about the type of LVM your system supports, or simply browse the many how- to documents available on the Web. Before we get started with the details, let’s take a moment to understand the basic concepts of LVM. There is a hierarchy of levels to the LVM implementation. At the lowest level is the disk itself. On top of that are partitions, which allow us to commu- nicate with the disk. On top of the partition we create a physical volume, which is the Backup Utilities and OS-Level Solutions | 433 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. control mechanism that the LVM provides. You can add a physical volume to a volume group (which can contain multiple physical volumes), and a volume group can contain one or more logical volumes. Figure 12-1 depicts the relationship among filesystems, volume groups, physical volumes, and block devices. Figure 12-1. Anatomy of LVM A logical volume can act as either a normal mounted filesystem or a snapshot. The creation of a snapshot logical volume is the key to using snapshots for backup. The following sections describe how you can get started experimenting with LVM and making backups of your data. There are several useful commands that you should become familiar with. The follow- ing list contains the most frequently used commands and their uses. Be sure to consult the documentation for more information about these commands: pvcreate Creates a physical volume pvscan Shows details about the physical volumes vgcreate Creates volume groups vgscan Shows details about volume groups lvcreate Creates a logical volume 434 | Chapter 12: Protecting Your Investment Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. lvscan Shows details about logical volumes lvremove Removes a logical volume mount Mounts a logical volume umount Unmounts a logical volume To use LVM, you need to have either a new disk or a disk device that you can logically unmount. The process is as follows (the output here was generated on a laptop running Ubuntu version 9.04): 1. Create a backup of an existing MySQL data directory. tar -czf ~/my_backups/backup.tar.gz /dev/mysql/datadir 2. Partition the drive. sudo parted select /dev/sdb mklabel msdos mkpart test quit 3. Create a physical volume for the drive. sudo pvcreate /dev/sdb 4. Create a volume group. sudo vgcreate /dev/sdb mysql 5. Create a logical volume for your data. Here we create a 20 GB volume. sudo lvcreate -L20G -ndatadir mysql 6. Create a filesystem on the logical volume. mke2fs /dev/mysql/datadir 7. Mount the logical volume. sudo mkdir /mnt sudo mount /dev/mysql/datadir /mnt 8. Copy the archive and restore your data to the logical volume. sudo cp ~/my_backups/backup.tar.gz sudo tar -xvf backup.tar.gz 9. Create an instance of a MySQL server and use --datadir to point to the folder on the logical volume. ./mysqld --console -uroot --datadir=/mnt Backup Utilities and OS-Level Solutions | 435 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. If you want to experiment with LVM, we recommend you use a disk whose data you can afford to lose. A good, cheap option is a small USB hard drive. That’s all you need to get started with a logical volume. Take some time to experiment with the LVM tools, until you are certain you can work with them effectively, before you start using them for your production systems. LVM in a backup and restore To do your backup, you need to flush and temporarily lock all of the tables, take the snapshot, and then unlock the tables. The lock is necessary to ensure all of your ongoing transactions are finished. The process is shown here along with the shell-level com- mands that perform the operations: 1. Issue a FLUSH TABLES WITH READ LOCK command in a MySQL client. 2. Create a snapshot of your logical volume (the -s option specifies a snapshot). sudo lvcreate -L20M -s -n backup /dev/mysql/datadir 3. Issue an UNLOCK TABLES command in a MySQL client (your server can now resume its operations). 4. Mount the snapshot. sudo mkdir /mnts sudo mount /dev/mysql/backup /mnts 5. Perform a backup of the snapshot. tar -[FIXTHIS]f snapshot.tar.gz /mnts Of course, the best use of the snapshot is to initiate a copy periodically so that you can do another backup. There are scripts available from volunteers on the Web to automate this process, but the tried and true mechanism is to remove the snapshot and re-create it using the following procedure: 1. Unmount the snapshot. sudo umount /mnts 2. Remove the snapshot (logical volume). sudo lvremove /dev/mysql/backup You can then re-create the snapshot and perform your backup. If you create your own script, we recommend adding the snapshot removal after you have verified the backup archive was created. This will ensure your script performs proper cleanup. If you need to restore the snapshot, simply restore the data. The real benefit of LVM is that all of the operations for creating the snapshot and the backup using the tar utility allow you to create a customized script that you can run periodically (such as a cron job), which can help you automate your backups. 436 | Chapter 12: Protecting Your Investment Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... command = ["mysqlbinlog", " start-position=%s" % (image.backup_position.pos), " stop-datetime=%s" % (datetime)] mysqlbinlog_proc = Popen(mysqlbinlog_command + files, stdout=PIPE) mysql_ command = [ "mysql" , " host=%s" % (server.host), " user=%s" % (server.sql_user.name), " password=%s" % (server.sql_user.password)] mysql_ proc = Popen (mysql_ command, stdin=mysqlbinlog_proc.stdout) output = mysql_ proc.communicate()[0]... storage engines), and you work on a Linux system, LVM is a good choice Backup and MySQL Replication There are two ways to use backup with MySQL replication In previous chapters, you learned about MySQL replication and its many uses for scale-out and high availability In this chapter, we examine two more common uses for MySQL replication involving backup These include using replication to create a backup... tools and shows you how they will save you time while keeping your MySQL servers running at the highest levels of performance and availability We also include an example of the monitoring tools running on a complex replication topology Getting Started with MySQL Enterprise MySQL Enterprise was launched in 2006 and comprises the Enterprise MySQL server release, a set of monitoring tools, and product support... customers who use MySQL for data management Early on, MySQL recognized the need organizations have for stability and reliability MySQL Enterprise was the answer to this need If you are not ready to purchase a MySQL Enterprise subscription or you want to try it out for a while before deciding, you can get a trial subscription that will last for 30 days Apply for a trial subscription at http:/ /mysql. com/trials/... Enterprise | 455 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark MySQL Enterprise Components MySQL Enterprise comprises the MySQL server software, the MEM toolset, and production support MySQL Enterprise Server Two versions of the MySQL server are included in the MySQL Enterprise subscription The Pro version is the production release with the most stable feature set The Advanced... available for monitoring MySQL is the MySQL Enterprise Monitor (MEM), which comes with the MySQL Enterprise subscription The MEM tools can greatly enhance monitoring and preventive maintenance and can dramatically reduce diagnostic time and downtime While it is a fee-based tool, the savings of having a well-maintained data center will more than cover the cost This chapter introduces the MySQL Enterprise suite... trial subscription at http:/ /mysql. com/trials/ 452 | Chapter 13: MySQL Enterprise Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark MySQL Enterprise adds the web-based MEM application, along with a separate MySQL server instance to be a repository for the metrics collected by other applications installed on your MySQL servers, called agents MEM combines the metrics into reports... MySQL server The platinum level is for organizations that need the highest level of support for their most critical data management needs With so many options to choose from, you can add more support as your business grows You can find specific details about pricing and what is included with each level at http:/ /mysql. com/products/enterprise/features.html Installation Overview When you purchase a MySQL. .. credentials to the MySQL Enterprise portal Connection to this portal is necessary to activate the MySQL Enterprise tools The portal is your one-stop location for downloading updates, checking for upgrades, news, information about your subscription, and access to the knowledge base The portal is called the MySQL Enterprise Customer Center and it is located at https: //enterprise .mysql. com/ For offline... install and configure the MySQL Enterprise suite Installation involves the following steps (at a minimum): 1 Install the service manager component, including the metrics repository (a separate installation of MySQL) , which is operating system–specific For example, you would install the file named mysqlmonitor-2.1.0.1096-osx-installer on a Mac OS X system 454 | Chapter 13: MySQL Enterprise Please purchase . --apply-log /etc /mysql/ my.cnf /home/cbell/backup mysqladmin -uroot shutdown perl innobackup --copy-file /etc /mysql/ my.cnf /home/cbell/backup /etc/init.d /mysql start. show each of these options: mysqldump -uroot -all-databases mysqldump -uroot db1, db2 mysqldump -uroot my_db t1 You can also use mysqldump to do a hot backup