www.it-ebooks.info For your convenience Apress has placed some of the front matter material after the index. Please use the Bookmarks and Contents at a Glance links to access them. www.it-ebooks.info v Contents at a Glance About the Author ��������������������������������������������������������������������������������������������������������������� xix About the Technical Reviewers ����������������������������������������������������������������������������������������� xxi Acknowledgments ����������������������������������������������������������������������������������������������������������� xxiii Introduction ���������������������������������������������������������������������������������������������������������������������� xxv Part 1: Getting Started with MySQL Development ■ ���������������������������������������� 1 Chapter 1: MySQL and The Open Source Revolution ■ ���������������������������������������������������������3 Chapter 2: The Anatomy of a Database System ■ ��������������������������������������������������������������23 Chapter 3: A Tour of the MySQL Source Code ■ ������������������������������������������������������������������57 Chapter 4: Test-Driven MySQL Development ■ �����������������������������������������������������������������117 Part 2: Extending MySQL ■ �������������������������������������������������������������������������� 151 Chapter 5: Debugging ■ ����������������������������������������������������������������������������������������������������153 Chapter 6: Embedded MySQL ■ ����������������������������������������������������������������������������������������195 Chapter 7: Adding Functions and Commands to MySQL ■ �����������������������������������������������251 Chapter 8: Extending MySQL High Availability ■ ��������������������������������������������������������������281 Chapter 9: Developing MySQL Plugins ■ ���������������������������������������������������������������������������339 Chapter 10: Building Your Own Storage Engine ■ �������������������������������������������������������������369 Part 3: Advanced Database Internals ■ ������������������������������������������������������� 453 Chapter 11: Database System Internals ■ ������������������������������������������������������������������������455 Chapter 12: Internal Query Representation ■ �������������������������������������������������������������������465 www.it-ebooks.info ■ Contents at a GlanCe vi Chapter 13: Query Optimization ■ ������������������������������������������������������������������������������������495 Chapter 14: Query Execution ■ �����������������������������������������������������������������������������������������543 Appendix ■ ������������������������������������������������������������������������������������������������������������������������587 Index ���������������������������������������������������������������������������������������������������������������������������������601 www.it-ebooks.info xxv Introduction MySQL has been identied as the world’s most popular open-source database and the fastest-growing database system in the industry. is book presents some of the topics of advanced database systems, examines the architecture of MySQL, and provides an expert’s workbook for examining, integrating, and modifying the MySQL source code for use in enterprise environments. e book provides insight into how to modify the MySQL system to meet the unique needs of system integrators and educators alike. Whom is Book Is For I have written this book with a wide variety of readers in mind. Whether you have been working in database systems for years, or maybe have taken an introductory database-theory class, or have even just read a good Apress book on MySQL, you will get a lot out of this book. Best of all, you can even get your hands into the source code. If you ever wanted to know what makes a database system like MySQL tick, this is your book! How is Book Is Structured e material presented is divided into three parts followed by an appendix. Each part is designed to present a set of topics ranging from introductory material on MySQL and the open-source revolution to extending and customizing the MySQL system. ere is even coverage on how to build an experimental query optimizer and execution engine as an alternative to the MySQL query engine. Part One e rst part of the book introduces concepts in developing and modifying open-source systems. Part One provides the tools and resources necessary to begin exploring the more advanced database concepts presented in the rest of the book. Chapters include: MySQL and the Open Source Revolution—is chapter is less technical and contains more 1� narration than the rest of the book. It guides you through the benets and responsibilities of an open-source system integrator. It highlights the rapid growth of MySQL and its importance in the open-source- and database-system markets. Additionally, it provides a clear perspective on the open-source revolution. e Anatomy of a Database System—is chapter covers the basics of what a database 2� system is and how it is constructed. e anatomy of the MySQL system is used to illustrate the key components of modern relational-database systems. www.it-ebooks.info ■ intRoduCtion xxvi A Tour of the MySQL Source Code—A complete introduction to the MySQL source 3� is presented in this chapter, along with how to obtain and build the system. You are introduced to the mechanics of the source code along with coding guidelines and best practices for how the code is maintained. Test-driven MySQL Development—is chapter introduces a key element in generating 4� high-quality extensions to the MySQL system. Software testing is presented along with the common practices of how to test large systems. Specic examples are used to illustrate the accepted practices of testing the MySQL system. Part Two is part of the book provides tools, using a hands-on approach to investigating the MySQL system. It introduces Debugging—is chapter provides you with debugging skills and techniques to help make 5� development easier and less prone to failure. Several debugging techniques are presented, along with the pros and cons of each. Embedded MySQL—is chapter presents you with a tutorial on how to embed the 6� MySQL system in enterprise applications. Example projects assist you in applying the skills presented to your own integration needs. Adding Functions and Command to MySQL—is chapter presents the most popular 7� modication to the MySQL code. You are shown how to modify the SQL commands and how to build custom SQL commands. It presents examples of how to modify SQL commands to add new parameters, functions, and new commands. Extending MySQL High Availability—is chapter provides an overview of the high-8� availability features of MySQL, including a tour of the replication source code and examples of how to extend the feature to meet your high-availability needs. Developing MySQL Plugins—is chapter presents an introduction to the pluggable 9� architecture in MySQL. You will discover how to build plugins and see a detailed example of an authentication plugin. Building Your Own Storage Engine—is chapter demonstrates the pluggable-storage-10� engine feature. e architecture is explored using examples and projects that permit you to build a sample storage engine. Part Three is part of the book takes a deeper look into the MySQL system and provides you with an insider’s look at what makes the system work. e section begins with an introduction into the more advanced database technologies. eory and practices are presented in a no-nonsense manner to enable you to apply the knowledge gained to tackle the more complex topics of database systems. is section also presents examples of how to implement an internal- query representation, an alternative query optimizer, and an alternative query-execution mechanism. Examples and projects are discussed in detail. Chapters 12 through 14 provide the skills and techniques needed to alter the internal structure of the MySQL system to execute using alternative mechanisms. ese chapters provide you with a unique insight into how large systems can be built and modied. www.it-ebooks.info ■ intRoduCtion xxvii Database-systems Internals—is chapter is designed to present the more advanced 11� database techniques by examining the MySQL architecture. Topics include query execution, multiuser concerns, and programmatic considerations. Internal-query Representation—is chapter presents the MySQL internal-query 12� representation. You are provided with an example alternative query representation. A discussion is included of how to alter the MySQL source code to implement an alternative query representation. Query Optimization—is chapter presents the MySQL internal-query optimizer. You 13� are provided with an example alternative query optimizer that uses the alternative query representation from the previous chapter. It discusses how to alter the MySQL source code to implement the alternative query optimizer. Query Execution—is chapter combines the techniques from the previous chapters 14� to provide you with instructions on how to modify the MySQL system to implement alternative query-processing-engine techniques. Appendix is section of the book provides a list of resources on MySQL, database systems, and open-source software. Using the Book for Teaching Database-Systems Internals Many excellent database texts oer coverage of relational theory and practice. Few, however, oer material suitable for a classroom or lab environment. ere are even fewer resources available for students to explore the inner workings of database systems. is book oers an opportunity for instructors to augment their database classes with hands-on laboratories. is text can be used in a classroom setting in three ways: 1� e text can be used to add depth to an introductory undergraduate or graduate database course. Parts 1 and 2 can be used to provide in-depth coverage of special topics in database systems. Suggested topics for lectures include those presented in Chapters 2, 3, 4, and 6. ese can be used in addition to more traditional database-theory or systems texts. Hands-on exercises or class projects can be drawn from Chapters 6 and 7. 2� An advanced database course for undergraduate or graduate students can be based on Parts 1 and 2; each chapter can be presented over the course of 8 to 12 weeks. e remainder of the lectures can discuss the implementation of physical storage layers and the notion of storage engines. Semester projects can be based on Chapter 10, with students building their own storage engines. 3� A special-topics course on database-systems internals for the senior undergraduate or graduate students can be based on the entire text, with lectures based on the rst eleven chapters. Semester projects can be derived from Part 3 of the text, with students implementing the remaining features of the database experimental platform. ese features include applications of language theory, query optimizers, and query-execution algorithms. www.it-ebooks.info ■ intRoduCtion xxviii Conventions roughout the book, I’ve kept a consistent style for presenting SQL and results. Where a piece of code, a SQL reserved word, or a fragment of SQL is presented in the text, it is presented in xed-width Courier font, such as: select * from dual; Where I discuss the syntax and options of SQL commands, I use a conversational style so you can quickly reach an understanding of the command or technique. is means that I haven’t duplicated large syntax diagrams that better suit a reference manual. Downloading the Code www.apress.com. A link can be drcharlesbell@gmail.com. www.it-ebooks.info PART 1 Getting Started with MySQL Development This section introduces you to concepts in developing and modifying open source systems. Chapter 1 guides you through the benefits and responsibilities of an open source system integrator. It highlights the rapid growth of MySQL and its importance in the open-source- and database-system markets. Chapter 2 covers the basics of what a database system is and how it is constructed. Chapter 3 provides a complete introduction to the MySQL source presented in this chapter along with how to obtain and build the system. Chapter 4 introduces a key element in generating high-quality extensions to the MySQL system. You’ll learn about software testing as well as common practices for testing large systems. www.it-ebooks.info 3 ChAPTeR 1 MySQL and The Open Source Revolution Open source systems are rapidly changing the software landscape. Information technology professionals everywhere are taking note of the high quality, and in many cases world-class, development and support offered by open-source- software vendors. Corporations are paying attention, because for the first time they have an alternative to commercial proprietary software vendors. Small businesses are paying attention because open source software can significantly lower the cost of their information systems. Individuals are paying attention because they have more choices with more options than ever before. The majority of the underpinnings that make the Internet what it is today are based on open source software systems, such as Linux, Apache HTTP server, BIND, Sendmail, OpenSSL, MySQL, and many others. The most common reason businesses use open source software is cost. Open source software, by its very nature, reduces the total cost of ownership (TCO) and provides a viable business model on which businesses can build or improve their markets. This is especially true of open-source database systems, as the cost of commercial proprietary systems can easily go into tens or hundreds of thousands of dollars. For small businesses just starting, this outlay of funds could impact its growth. For example, if a startup has to spend a significant potion of its reserves, it may be unable to get its products to market and therefore may not be able to gain a foothold in a highly competitive market. Open source provides startups with the opportunity to defer their software purchases until they can afford the investment. That doesn’t mean, however, that they are building an infrastructure out of inferior components. Open source software once was considered by many to be limited to the hobbyist or hacker bent on subverting the market of large commercial software companies. Although some developers may feel that they are playing the role of David to Microsoft’s Goliath, the open source community is not about that at all. It does not profess to be a replacement for commercial proprietary software, but rather, it proposes the open source philosophy as an alternative. As you will see in this chapter, not only is open source a viable alternative to commercial software, but it is also fueling a worldwide revolution in how software is developed, evolved, and marketed. Note ■ In this book, the term “hacker” refers to Richard Stallman’s definition: “someone who loves to program and enjoys being clever about,” 1 and not the common perception of a nefarious villain bent on stealing credit cards and damaging computer systems. 1 http://www.gnu.org/gnu/thegnuproject.html www.it-ebooks.info [...]... uses MySQL client libraries, check out the MySQL FOSS exception for complete details http://www .mysql. com/about/legal/licensing/foss-exception/ Oracle offers several major MySQL editions, or versions, of the server Most are commercial offerings that may not have a corresponding GPL release For example, while you may download a GPL release of MySQL Cluster, you cannot download a commercial release of MySQL. .. Commercial Call MySQL sales for pricing MySQL Community Edition freely downloadable version of MySQL GPL Free $10,000.00** **Costs shown are representative as of the publication of this work Contact Oracle for accurate and up-to-date pricing 17 www.it-ebooks.info Chapter 1 ■ MySQL and The Open Source Revolution ■■Tip  Learn more about Oracle’s pricing and purchasing options at http:/ /mysql. com/buy -mysql/ So,... challenge you are likely to face with modifying MySQL is the question, “Why MySQL? ” Experts will be quick to point out that there are several open-source-database systems to choose from The most popular are MySQL, Firebird, PostgreSQL, and Berkeley DB Reasons to choose MySQL for your development projects over some of the other database systems include: • MySQL is a relational database-management system... nearrelease-candidate state A DMR therefore is a key method of tracking and preparing for adopting major advances in MySQL development You can find DMRs at http://dev .mysql. com/downloads /mysql/ #downloads You can read more about general MySQL development, labs releases, and DMRs at http://dev .mysql. com/doc /mysql- development-cycle/en/index.html The parallel development strategy permits Oracle to maintain its current... highdemand web, cloud, and communications services MySQL Enterprise Edition advanced enterprise features including management tools and technical support to achieve the highest levels of MySQL scalability, security, reliability, and uptime It is targeted for developing, deploying, and managing business-critical MySQL applications Commercial $5,000.00** MySQL Standard Edition includes the InnoDB storage engine... researching MySQL, you most likely have encountered at least one database system that claims to be a derivative of MySQL While several vendors offer a variant (sometimes called a fork or port) of MySQL some being backed by small startup companies and one by the original developers of MySQL none of these will match the level of expertise and development prowess of Oracle’s fostering of MySQL You may... you cannot download a commercial release of MySQL Cluster Carrier Grade Edition Table 1-1 summarizes the various server editions currently available (when this chapter was written) from Oracle and their base licensing cost Table 1-1.  MySQL Server Products and Pricing Product Description License Cost MySQL Cluster Carrier Grade Edition high-performance, in-memory clustered database Commercial solution... alternative to MySQL and recently tossed its hat into the ring as a replacement for the “M” in the LAMP stack (see “What Is the LAMP Stack?”) Proponents for Apache Derby cite licensing issues with MySQL and feature limitations Neither has deterred the MySQL install base, nor have these “issues” limited MySQL s increasing popularity WHAT IS THE LAMP STACK? LAMP stands for Linux, Apache, MySQL, and PHP/Perl/Python... addition, MySQL Replication allows you to deliver high-performance and scalable applications It is targeted for scalable Online Transaction Processing (OLTP) applications Commercial $2,000.00** MySQL Classic embedded database for ISVs, OEMs and VARs developing read-intensive applications using the MyISAM storage engine Commercial Call MySQL sales for pricing MySQL Embedded (OEM/ISV) any of the above editions... owner In the case of MySQL, the company, Oracle, originated the project and therefore it owns the MySQL system According to the GPL that MySQL adheres to, Oracle owns all the source code and any modifications made under the GPL The GPL gives you the right to modify MySQL, but it does not give you the right to claim the source code as your property 10 www.it-ebooks.info Chapter 1 ■ MySQL and The Open . major advances in MySQL development. You can find DMRs at http://dev .mysql. com/downloads /mysql/ #downloads. You can read more about general MySQL development,. Oracle, owns the source code for its MySQL product. (For a complete description of Oracle’s MySQL open source license, see www .mysql. com/company/legal/licensing/opensource-license.htm.) Developers