Gilmore_862-8C37.fm Page 938 Monday, February 25, 2008 3:02 PM 939 ■ ■ ■ CHAPTER 38 Importing and Exporting Data Back in the Stone Age, cavemen never really had any issues with data incompatibility, as stones and one’s own memory were the only storage medium. Copying data involved pulling out the old chisel and getting busy on a new slab of granite. Now, of course, the situation is much different. Hundreds of data storage strategies exist, the most commonplace of which include spreadsheets and various types of databases. Working in a complex, often convoluted fashion, you often need to convert data from one storage type to another, say between a spreadsheet and a relational database, or between an Oracle database and MySQL. If this is done poorly, you could spend hours, and even days and weeks, massaging the converted data into a usable format. This chapter seeks to eliminate that conundrum, by introducing MySQL’s data import and export utilities, as well as various techniques and concepts central to lessening the pain involved in performing such tasks. By the conclusion of this chapter, you will be familiar with the following topics: • Common data-formatting standards recognized by most mainstream storage products •The SELECT INTO OUTFILE SQL statement •The LOAD DATA INFILE SQL statement •The mysqlimport utility • How to use PHP to mimic MySQL’s built-in import utilities Before delving into the core topics, take a moment to review the sample data used as the basis for examples presented in this chapter. Afterward, several basic concepts surrounding MySQL’s import and export strategies are introduced. Gilmore_862-8C38.fm Page 939 Tuesday, February 26, 2008 10:13 AM 940 CHAPTER 38 ■ IMPORTING AND EXPORTING DATA Sample Table If you would like to execute the examples as you proceed through the chapter, the following sales table will be the focus of several examples in this chapter: CREATE TABLE sales ( id SMALLINT UNSIGNED NOT NULL AUTO_INCREMENT, client_id SMALLINT UNSIGNED NOT NULL, order_time TIMESTAMP NOT NULL, sub_total DECIMAL(8,2) NOT NULL, shipping_cost DECIMAL(8,2) NOT NULL, total_cost DECIMAL(8,2) NOT NULL, PRIMARY KEY(id); This table is used to track basic sales information. Although it lacks many of the columns you might find in a real-world implementation, the additional detail is omitted in an attempt to keep the focus on the concepts introduced in this chapter. Using Data Delimitation Even if you’re a budding programmer, you’re probably already quite familiar with software’s exacting demands when it comes to data. All i’s must be dotted and all t’s must be crossed; a single out-of-place character is enough to produce unexpected results. Therefore, you can imagine the issues that might arise when attempting to convert data from one format to another. Thankfully, a particularly convenient format- ting strategy has become commonplace: delimitation. Information structures like database tables and spreadsheets share a similar conceptual organization. Each is broken down into rows and columns, each of which is further broken down into cells. Therefore, you can convert between formats as long as you institute a set of rules for determining how the columns, rows, and cells are recognized. An application utility will read in this data and, based on these rules, make the conversions necessary for adapting the data to its own formatting standards. Typically, a character or a character sequence is used as a delimiter, separating each cell within a row, and each row from the following row. For example, the sales table might be delimited in a format that separates each field by a comma and each row by a newline character: 12309,45633,2007-12-19 01:13:42,22.04,5.67,27.71 12310,942,2007-12-19 01:15:12,11.50,3.40,14.90 Gilmore_862-8C38.fm Page 940 Tuesday, February 26, 2008 10:13 AM CHAPTER 38 ■ IMPORTING AND EXPORTING DATA 941 12311,7879,2007-12-19 01:15:22,95.99,15.00,110.99 12312,55521,2007-12-19 01:30:45,10.75,3.00,13.75 Many data import and export utilities, including MySQL’s, revolve around the concept of data delimitation. Importing Data If you’ve invested the time to read this book, you likely are planning to make MySQL a significant part of your application infrastructure. Therefore, you’re probably already thinking about how you’re going to migrate existing data, possibly from several sources, into your new database environment. In this section, you’ll learn about the two built- in tools MySQL offers for importing delimited data sets into a table: LOAD DATA INFILE and mysqlimport. ■Tip You might consider using the mysqlimport client in lieu of LOAD DATA INFILE when you need to create batch imports executed from a cron job. Importing Data with LOAD DATA INFILE The LOAD DATA INFILE statement, a command that is executed much like a query is executed within the mysql client, is used to import delimited text files into a MySQL table. Its generalized syntax follows: LOAD DATA [LOW_PRIORITY | CONCURRENT] [LOCAL] INFILE 'file_name' [REPLACE | IGNORE] INTO TABLE table_name [CHARACTER SET charset_name] [FIELDS [TERMINATED BY 'character'] [[OPTIONALLY] ENCLOSED BY 'character'] [ESCAPED BY 'character'] ] [LINES [STARTING BY 'character'] [TERMINATED BY 'character'] ] [IGNORE number lines] [SET column_name = expression, )] Gilmore_862-8C38.fm Page 941 Tuesday, February 26, 2008 10:13 AM 942 CHAPTER 38 ■ IMPORTING AND EXPORTING DATA Certainly one of MySQL’s longer query commands you’ve seen thus far, isn’t it? Yet it’s this wide array of options that makes this feature so powerful. Each option is introduced next: • LOW PRIORITY: This option forces execution of the command to be delayed until no other clients are reading from the table. • CONCURRENT: Used in conjunction with a MyISAM table, this option allows other threads to retrieve data from the target table while the command is executing. • LOCAL: This option declares that the target infile must reside on the client side. If omitted, the target infile must reside on the same server hosting the MySQL database. When LOCAL is used, the path to the file can be either absolute or rela- tive according to the present location. When omitted, the path can be absolute, local, or, if not present, assumed to reside in MySQL’s designated database directory or in the presently chosen database directory. • REPLACE: This option results in the replacement of existing rows with new rows possessing identical primary or unique keys. • IGNORE: Including this option has the opposite effect of REPLACE. Read-in rows with primary or unique keys matching an existing table row will be ignored. • CHARACTER SET charset_name: MySQL will presume the input file contains char- acters matching the character set assigned to the system variable character_ set_database . If the characters do not match this setting, use this option to iden- tify the file’s character set. • FIELDS TERMINATED BY 'character': This option signals how fields will be termi- nated. Therefore, TERMINATED BY ',' means that each field will end with a comma, like so: 12312,55521,2007-12-19 01:30:45,10.75,3.00,13.75 The last field does not end in a comma because it isn’t necessary, as typically this option is used in conjunction with the LINES TERMINATED BY 'character' option. Encountering the character specified by this other option by default also delimits the last field in the file, as well as signals to the command that a new line (row) is about to begin. Gilmore_862-8C38.fm Page 942 Tuesday, February 26, 2008 10:13 AM CHAPTER 38 ■ IMPORTING AND EXPORTING DATA 943 • [OPTIONALLY] ENCLOSED BY 'character': This option signals that each field will be enclosed by a particular character. This does not eliminate the need for a terminating character. Revising the previous example, using the option FIELDS TERMINATED BY ',' ENCLOSED BY '"' implies that each field is enclosed by a pair of double quotes and delimited by a comma, like so: "12312","55521","2007-12-19 01:30:45","10.75","3.00","13.75" •The optional OPTIONALLY flag denotes that character strings only require enclo- sure by the specified character pattern. Fields containing only integers, floats, and so on need not be enclosed. • ESCAPED BY 'character': If the character denoted by the ENCLOSED BY option appears within any of the fields, it must be escaped to ensure that the field is not incorrectly read in. However, this escape character must be defined by ESCAPED BY so that it can be recognized by the command. For example, FIELDS TERMINATED BY ',' ENCLOSED BY ''' ESCAPED BY '\\' would allow the following fields to be properly parsed: 'jason@example.com','Excellent product! I\'ll return soon!','2007-12-20' • Note that because the backslash is treated by MySQL as a special character, you need to escape any instance of it by prefixing it with another backslash in the ESCAPED BY clause. • LINES: The following two options are pertinent to how lines are started and terminated, respectively: • STARTING BY 'character': This option defines the character intended to signal the beginning of a line, and thus a new table row. Use of this option is generally skipped in preference to the next option. • TERMINATED BY 'character': This option defines the character intended to signal the conclusion of a line, and thus the end of a table row. Although it could conceiv- ably be anything, this character is most often the newline (\n) character. In many Windows-based files, the newline character is often represented as \r\n. • IGNORE number LINES: This option tells the command to ignore the first x lines. This is useful when the target file contains header information. Gilmore_862-8C38.fm Page 943 Tuesday, February 26, 2008 10:13 AM 944 CHAPTER 38 ■ IMPORTING AND EXPORTING DATA • [(SET column_name = expression, )]: If the number of fields located in the target file does not match the number of fields in the target table, you need to specify exactly which columns are to be filled in by the file data. For example, if the target file containing sales information consists of only four fields, id, client_id, order_time, and total_cost, rather than the six fields used in prior examples ( id, client_id, order_time, sub_total, shipping_cost, and total_cost), yet in the target table all six fields remain, the command would have to be written like so: LOAD DATA LOCAL INFILE "sales.txt" INTO TABLE sales (id, client_id, order_time, total_cost); Keep in mind that such attempts could fail should one or several of the missing columns be designated as NOT NULL in the table schema. On such occasions, you need to either designate DEFAULT values for the missing columns or further manipulate the data file into an acceptable format. You can also set columns to variables such as the current timestamp. For example, presume the sales table was modified to include an additional column named added_to_table: LOAD DATA LOCAL INFILE "sales.txt" INTO TABLE sales (id, client_id, order_time, total_cost) SET added_to_table = CURRENT_TIMESTAMP; ■Tip If you would like the order of the fields located in the target file to be rearranged as they are read in for insertion into the table, you can do so by rearranging the order via the [(column_name, )] option. A Simple Data Import Example This example is based upon the ongoing sales theme. Suppose you want to import a file titled productreviews.txt, which contains the following information: '43','jason@example.com','I love the new Website!' '44','areader@example.com','Why don\'t you sell shoes?' '45','anotherreader@example.com','The search engine works great!' Gilmore_862-8C38.fm Page 944 Tuesday, February 26, 2008 10:13 AM CHAPTER 38 ■ IMPORTING AND EXPORTING DATA 945 The target table, aptly titled productreviews, consists of three fields, and they are in the same order ( comment_id, email, comment) as the information found in productreviews.txt: LOAD DATA LOCAL INFILE 'productreviews.txt' INTO TABLE productreviews FIELDS TERMINATED BY ',' ENCLOSED BY '\'' ESCAPED BY '\\' LINES TERMINATED BY '\n'; Once the import is completed, the productreviews table will look like this: + + + + | comment_id | email | comment | + + + + | 43 | jason@example.com | I love the new Website! | | 44 | areader@example.com | Why don't you sell shoes? | | 45 | anotherreader@example.com | The search engine works great! | + + + + Choosing the Target Database You might have noticed that the preceding example referenced the target table but did not clearly define the target database. The reason is that LOAD DATA INFILE assumes that the target table resides in the currently selected database. Alternatively, you can specify the target database by prefixing it with the database name, like so: LOAD LOCAL DATA INFILE 'productreviews.txt' into table corporate.productreviews; If you execute LOAD DATA INFILE before choosing a database, or without explicitly specifying the database in the query syntax, an error will occur. Security and LOAD DATA INFILE There are a few security issues that you should keep in mind regarding LOAD DATA INFILE: •If LOCAL is not used (meaning the file will be read from the server side), the executing user must possess the FILE privilege. • To disable LOAD DATA LOCAL INFILE, start the MySQL daemon with the local-infile=0 option. Gilmore_862-8C38.fm Page 945 Tuesday, February 26, 2008 10:13 AM 946 CHAPTER 38 ■ IMPORTING AND EXPORTING DATA Importing Data with mysqlimport The mysqlimport client is really just a command-line version of the LOAD DATA INFILE SQL query. Its general syntax follows: mysqlimport [options] database textfile1 [textfile2 textfileN] Useful Options Before reviewing any examples, take a moment to review many of the most commonly used mysqlimport options: • columns, -c: This option should be used when the number or ordering of the fields in the target file does not match that found in the table. For example, suppose you were inserting the following target file, which orders the fields as id, order_id, sub_total, shipping_cost, total_cost, and order_time: 45633,12309,22.04,5.67,27.71,2007-12-19 01:13:42 942,12310,11.50,3.40,14.90,2007-12-19 01:15:12 7879,12311,95.99,15.00,110.99,2007-12-19 01:15:22 •Yet the sales table presented at the beginning of this chapter lists the fields in this order: id, client_id, order_time, sub_total, shipping_cost, and total_cost. You can rearrange the input fields during the parsing process so that the data is inserted in the proper location, by including this option: columns=id,order_id,sub_total,shipping_cost,total_cost,and order_time • compress, -C: Including this option compresses the data flowing between the client and the server, assuming that both support compression. This option is most effective if you’re loading a target file that does not reside on the same server as the database. • debug, -#: This option is used to create trace files when debugging. • delete, -d: This option deletes the target table’s contents before importing the target file’s data. Gilmore_862-8C38.fm Page 946 Tuesday, February 26, 2008 10:13 AM [...]... DATA INFILE and mysqlimport functionality using a PHP script The following script uses PHP s file-handling functionality and a handy function known as fgetcsv() to open and parse the delimited sales data found at the beginning of this chapter: < ?php // Connect to the MySQL server and select the corporate database $mysqli = new mysqli("localhost","someuser","secret","corporate"); // Open and parse the... command with, 307 tracing connection request, 736 Bakken, Stig, 309 user table and, 738 bandwidth auto login example, session handling, 459–461 auto_append_file directive, PHP, 44 auto_detect_line_endings directive, PHP, 47 AUTO_INCREMENT datatype attribute, MySQL, 714 auto_prepend_file directive, PHP, 44 function libraries, 125 auto_start directive session handling, 449, 454 autocommit method, mysqli,... trigger naming, MySQL, 856 before triggers, MySQL, 850 after triggers compared, 852 naming conventions, 856 BEGIN command conforming to SQL-99 syntax, 930 BEGIN/END block, MySQL LEAVE command, 836 stored routines, 828, 829, 832 autoload function, 192 beginTransaction method, PDO, 817 autoloading objects, OOP, 191–192 BIGINT datatype, MySQL, 710 require_once statement, 191 auto-rehash option, mysql client,... HA PTER 38 ■ IMPORTING AND EXPORTING DA TA This command results in the compression and transmission of the data found in the local text file (C:\audit\inventory.txt) to the table inventory located in the company database Note that mysqlimport strips the extension from each text file and uses the resulting name as the table into which to import the text file’s contents Writing a mysqlimport Script Some... using PHP or Perl or another solution to do the job from the command line The next section switches directions of the data flow, explaining how to export data from MySQL into other formats Exporting Data As your computing environment grows increasingly complex, you’ll probably need to share your data among various disparate systems and applications Sometimes you won’t be able to cull this information from. .. 429 ldap_unbind function, 430 parameters, mysqli, 785 results, mysqli, 787 bindParam method, PDO, 807, 808, 809, 810 bindtextdomain function, 593, 595, 597 Bison installing Apache and PHP on Linux, 15 bitwise operators, 94 BLACKHOLE storage engine, MySQL, 704 BLOB datatype, MySQL, 713 blocks embedding multiple code blocks, 59 BOOL (BOOLEAN) datatype, MySQL, 710 Boole, George, 65 Boolean datatype, 65... responsible for dumping the data (in delimited format) from the mainframe and then pushing this data file via sftp to the Web server The second script, located on the Web server, was responsible for executing mysqlimport, loading this file to the MySQL database This script was quite trivial to create, and looked like this: #!/bin/sh /usr/local /mysql/ bin/mysqlimport delete silent \ fields-terminated-by='\t'... auto_start directive session handling, 449, 454 autocommit method, mysqli, 790 AUTOCOMMIT variable MySQL transactions tips, 933 autocommits PDO_ATTR_AUTOCOMMIT option, 800 auto-completion mysql client, 673 PEAR: HTML_QuickForm, 363 calculating network bandwidth, 423 testing user bandwidth, 422–424 base directory open_basedir directive, PHP, 36 base exception class see exception class baseclass class inheritance,... fgetcsv($fh, 100 0, ",")) { $id = $line[0]; $client_id = $line[1]; $order_time = $line[2]; $sub_total = $line[3]; $shipping_cost = $line[4]; $total_cost = $line[5]; // Insert the data into the sales table $query = "INSERT INTO sales SET id='$id', client_id='$client_id', order_time='$order_time', sub_total='$sub_total', shipping_cost='$shipping_cost', total_cost='$total_cost'"; $result = $mysqli->query($query);... the command line, a system user named productupdate was created, and a my.cnf file was placed in the user’s home directory, which looked like this: [client] host=localhost user=productupdate password=secret The permissions and ownership on this file were changed, setting the owner to mysql and allowing only the mysql user to read the file The final step involved adding the necessary information to the . LOAD DATA INFILE and mysqlimport functionality using a PHP script. The following script uses PHP s file-handling functionality and a handy function known as fgetcsv() to open and parse the delimited. ( C:auditinventory.txt) to the table inventory located in the company database. Note that mysqlimport strips the extension from each text file and uses the resulting name as the table into which to import. target MySQL server is running on a nonstandard port (MySQL s standard port is 3306), you need to specify that port value with this option. • replace, -r: This option causes mysqlimport to overwrite