SHELVE IN: COMPUTERS/PROGRAMMING $39.95 ($41.95 CDN) W E B B O T S, S P I D E R S, A N D S C R EEN S C R A P E R S W E B B O T S , S P I D E R S , A N D S C R EEN S C R A P E R S S C H R E N K 2 N D E D I T ION AND W E BBO T S, SPI DE R S, AND SCR EEN SC R A PE RS W E BBO T S, SPI DE R S, SCR EEN SC R A PE RS A G U I D E T O D E V E L O P I N G I N T E R N E T A G E N T S W I T H P H P / CUR L M I C H A E L S C H R E N K 2 N D E D ITI O N “ I LI E FL AT .” This book uses RepKover —a durable bi nding that won’t snap shut. www.nostarch.com TH E F INE ST I N G EEK ENTE RTA IN ME N T ™ There’s a wealth of data online, but sorting and gathering it by hand can be tedious and time consuming. Rather than click through page after endless page, why not let bots do the work for you? Webbots, Spiders, and Screen Scrapers will show you how to create simple programs with PHP/CURL to mine, parse, and archive online data to help you make informed decisions. Michael Schrenk, a highly regarded tolerant designs, how best to launch and schedule the webbot developer, teaches you how to develop fault- work of your bots, and how to create Internet agents that: Sample projects for automating tasks like price monitoring and news aggregation will show you how to put the concepts you learn into practice. information quickly • Send email or SMS notifications to alert you to new • Search different data sources and combine the results on one page, making the data easier to interpret and analyze activities to save time • Automate purchases, auction bids, and other online Valley to Moscow, for clients like the BBC, foreign A B O U T T H E A U T H O R Michael Schrenk has developed webbots for over 15 years, working just about everywhere from Silicon governments, and many Fortune 500 companies. He’s a frequent Defcon speaker and lives in Las Vegas, Nevada. SCRAPE, SCRAPE, AUTOMATE, AUTOMATE, AND CONTROL AND CONTROL THE INTERNET THE INTERNET To download the scripts and code libraries used in the book, visit http:// WebbotsSpidersScreenScrapers.com webbots that mimic human search behavior, and using discover the possibilities of web scraping, you’ll see how webbots can save you precious time and give you much greater control over the data available on the Web. This second edition of Webbots, Spiders, and Screen Scrapers includes tricks for dealing with sites that are resistant to crawling and scraping, writing stealthy regular expressions to harvest specific data. As you TECHNICAL REVIEW BY DANIEL STENBERG, CREATOR OF CURL AND LIBCURL TECHNICAL REVIEW BY DANIEL STENBERG, CREATOR OF CURL AND LIBCURL WEBBOTS, SPIDERS, AND SCREEN SCRAPERS, 2ND EDITION webbots2e.book Page i Thursday, February 16, 2012 11:59 AM webbots2e.book Page ii Thursday, February 16, 2012 11:59 AM WEBBOTS, SPIDERS, AND SCREEN SCRAPERS 2ND EDITION A Guide to Developing Internet Agents with PHP/CURL by Michael Schrenk San Francisco webbots2e.book Page iii Thursday, February 16, 2012 11:59 AM WEBBOTS, SPIDERS, AND SCREEN SCRAPERS, 2ND EDITION. Copyright © 2012 by Michael Schrenk. All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher. 16 15 14 13 12 1 2 3 4 5 6 7 8 9 ISBN-10: 1-59327-397-5 ISBN-13: 978-1-59327-397-2 Publisher: William Pollock Production Editor: Serena Yang Cover and Interior Design: Octopod Studios Developmental Editor: Tyler Ortman Technical Reviewer: Daniel Stenberg Copyeditor: Paula L. Fleming Compositor: Serena Yang Proofreader: Alison Law For information on book distributors or translations, please contact No Starch Press, Inc. directly: No Starch Press, Inc. 38 Ringold Street, San Francisco, CA 94103 phone: 415.863.9900; fax: 415.863.9950; info@nostarch.com; www.nostarch.com The Library of Congress has catalogued the first edition as follows: Schrenk, Michael. Webbots, spiders, and screen scrapers : a guide to developing internet agents with PHP/CURL / Michael Schrenk. p. cm. Includes index. ISBN-13: 978-1-59327-120-6 ISBN-10: 1-59327-120-4 1. Web search engines. 2. Internet programming. 3. Internet searching. 4. Intelligent agents (Computer software) I. Title. TK5105.884.S37 2007 025.04 dc22 2006026680 No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc. Other product and company names mentioned herein may be the trademarks of their respective owners. Rather than use a trademark symbol with every occurrence of a trademarked name, we are using the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The information in this book is distributed on an “As Is” basis, without warranty. While every precaution has been taken in the preparation of this work, neither the author nor No Starch Press, Inc. shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in it. webbots2e.book Page iv Thursday, February 16, 2012 11:59 AM In loving memory Charlotte Schrenk 1897–1982 webbots2e.book Page v Thursday, February 16, 2012 11:59 AM webbots2e.book Page vi Thursday, February 16, 2012 11:59 AM BRIEF CONTENTS About the Author xxiii About the Technical Reviewer xxiii Acknowledgments xxv Introduction 1 PART I: FUNDAMENTAL CONCEPTS AND TECHNIQUES 7 Chapter 1: What’s in It for You? 9 Chapter 2: Ideas for Webbot Projects 15 Chapter 3: Downloading Web Pages 23 Chapter 4: Basic Parsing Techniques 37 Chapter 5: Advanced Parsing with Regular Expressions 49 Chapter 6: Automating Form Submission 63 Chapter 7: Managing Large Amounts of Data 77 PART II: PROJECTS 91 Chapter 8: Price-Monitoring Webbots 93 Chapter 9: Image-Capturing Webbots 101 webbots2e.book Page vii Thursday, February 16, 2012 11:59 AM viii Brief Contents Chapter 10: Link-Verification Webbots 109 Chapter 11: Search-Ranking Webbots 117 Chapter 12: Aggregation Webbots 129 Chapter 13: FTP Webbots 139 Chapter 14: Webbots That Read Email 145 Chapter 15: Webbots That Send Email 153 Chapter 16: Converting a Website into a Function 163 PART III: ADVANCED TECHNICAL CONSIDERATIONS 171 Chapter 17: Spiders 173 Chapter 18: Procurement Webbots and Snipers 185 Chapter 19: Webbots and Cryptography 193 Chapter 20: Authentication 197 Chapter 21: Advanced Cookie Management 209 Chapter 22: Scheduling Webbots and Spiders 215 Chapter 23: Scraping Difficult Websites with Browser Macros 227 Chapter 24: Hacking iMacros 239 Chapter 25: Deployment and Scaling 249 PART IV: LARGER CONSIDERATIONS 263 Chapter 26: Designing Stealthy Webbots and Spiders 265 Chapter 27: Proxies 273 Chapter 28: Writing Fault-Tolerant Webbots 285 webbots2e.book Page viii Thursday, February 16, 2012 11:59 AM [...]... data you don’t need This chapter discloses the basics for scraping web pages Chapter 5: Advanced Parsing with Regular Expressions Once you know the basics of parsing, it’s time to explore the advanced features available with regular expressions and to know when, or when not, to use them Chapter 6: Automating Form Submission To truly automate web agents, your application needs the ability to automatically... manuscript Finally, a special tip of the hat goes to the great (and by great, I mean patient) folks at No Starch Press, specifically: Tyler, Serena, Alison, Travis, and, of course, Bill You guys never cease to amaze me with your in-depth knowledge of publishing and your ability to make me readable I also want to thank you for expanding my appreciation for bourbon at last year’s Defcon xxvi A c k n owl... automatically upload data to online forms This chapter teaches you how to write webbots that fill out forms Chapter 7: Managing Large Amounts of Data Spiders in particular can generate huge amounts of data That’s why it’s important for you to know how to effectively store and reduce the size of web pages, text files, and images After reading this chapter, you’ll know how to compress, thumbnail, store, and. .. 11:59 AM  Is it possible to write stealthy webbots that run without detection?  What is the trick to writing robust, fault-tolerant webbots that won’t break as Internet content changes? Learn from My Mistakes I’ve written webbots, spiders, and screen scrapers for over 15 years, and in the process I’ve made most of the mistakes someone can make Because webbots are capable of making unconventional demands... webbots2e.book Page 7 Thursday, February 16, 2012 11:59 AM PART I FUNDAMENTAL CONCEPTS AND TECHNIQUES Whereas most web development books explain how to create websites, this book teaches developers how to combine, adapt, and automate existing websites to fit their specific needs You may have experience from other areas of computer science that you can apply to developing webbots, spiders, and screen scrapers. .. (especially hard drives) if they are allowed to download too many files Software In an effort to be as relevant as possible, the software examples in this book use PHP,1 cURL,2 and MySQL.3 All of these software technologies are available as free downloads from their respective websites In addition to being free, these software packages are wonderfully portable and function well on a variety of computers and. .. Vegas, Nevada ABOUT THE TECHNICAL REVIEWER Daniel Stenberg is the author and maintainer of cURL and libcurl He is a computer consultant, an internet protocol geek, and a hacker He’s been programming for fun and profit since 1985 Read more about Daniel, his company, and his open source projects at http://daniel haxx.se/ webbots2e.book Page xxiv Thursday, February 16, 2012 11:59 AM webbots2e.book Page xxv... system administrators can confuse webbots’ requests with attempts to hack into their systems Thankfully, none of my mistakes has ever led to a courtroom, but they have resulted in intimidating phone calls, scary emails, and very awkward moments Happily, I can say that I’ve learned from these situations, and it’s been a very long time since I’ve been across the desk from an angry system administrator You... not teach you how to program or how TCP/IP, the protocol of the Internet, works Hardware You don’t need elaborate hardware to start writing webbots If you have a secondhand computer, you probably have the minimum requirement to play with all the examples in this book Any of the following hardware is appropriate for using the examples and information in this book:  A personal computer that uses a Windows... Emulate Browsers 75 Avoid Form Errors 75 7 M AN A GI N G L A R G E A M O U N T S O F D A TA 77 Organizing Data 77 Naming Conventions 78 Storing Data in Structured Files 79 Storing Text in a Database 80 Storing Images in a Database 83 Database or File? 85 Making Data Smaller 85 Storing References to Image Files . Data 77 Naming Conventions 78 Storing Data in Structured Files 79 Storing Text in a Database 80 Storing Images in a Database 83 Database or File? 85 Making Data Smaller 85 Storing References to. EDITION webbots2e.book Page i Thursday, February 16, 2012 11:59 AM webbots2e.book Page ii Thursday, February 16, 2012 11:59 AM WEBBOTS, SPIDERS, AND SCREEN SCRAPERS 2ND EDITION A Guide to Developing Internet Agents. specific data. As you TECHNICAL REVIEW BY DANIEL STENBERG, CREATOR OF CURL AND LIBCURL TECHNICAL REVIEW BY DANIEL STENBERG, CREATOR OF CURL AND LIBCURL WEBBOTS, SPIDERS, AND SCREEN SCRAPERS, 2ND EDITION webbots2e.book