Data Science on the Google Cloud Platform Implementing End-to-End Real-Time Data Pipelines: From Ingest to Machine Learning Valliappa Lakshmanan Data Science on the Google Cloud Platform by Valliappa Lakshmanan Copyright © 2018 Google Inc All rights reserved Printed in the United States of America Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472 O’Reilly books may be purchased for educational, business, or sales promotional use Online editions are also available for most titles (http://oreilly.com/safari) For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com Editor: Tim McGovern Production Editor: Kristen Brown Copyeditor: Octal Publishing, Inc Proofreader: Rachel Monaghan Indexer: Judith McConville Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Rebecca Demarest January 2018: First Edition Revision History for the First Edition 2017-12-12: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781491974568 for release details The O’Reilly logo is a registered trademark of O’Reilly Media, Inc Data Science on the Google Cloud Platform, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work Use of the information and instructions contained in this work is at your own risk If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights 978-1-491-97456-8 [LSI] Preface In my current role at Google, I get to work alongside data scientists and data engineers in a variety of industries as they move their data processing and analysis methods to the public cloud Some try to the same things they on-premises, the same way they them, just on rented computing resources The visionary users, though, rethink their systems, transform how they work with data, and thereby are able to innovate faster As early as 2011, an article in Harvard Business Review recognized that some of cloud computing’s greatest successes come from allowing groups and communities to work together in ways that were not previously possible This is now much more widely recognized—an MIT survey in 2017 found that more respondents (45%) cited increased agility rather than cost savings (34%) as the reason to move to the public cloud In this book, we walk through an example of this new transformative, more collaborative way of doing data science You will learn how to implement an end-to-end data pipeline—we will begin with ingesting the data in a serverless way and work our way through data exploration, dashboards, relational databases, and streaming data all the way to training and making operational a machine learning model I cover all these aspects of data-based services because data engineers will be involved in designing the services, developing the statistical and machine learning models and implementing them in large-scale production and in real time Who This Book Is For If you use computers to work with data, this book is for you You might go by the title of data analyst, database administrator, data engineer, data scientist, or systems programmer today Although your role might be narrower today (perhaps you only data analysis, or only model building, or only DevOps), you want to stretch your wings a bit—you want to learn how to create data science models as well as how to implement them at scale in production systems Google Cloud Platform is designed to make you forget about infrastructure The marquee data services—Google BigQuery, Cloud Dataflow, Cloud Pub/Sub, and Cloud ML Engine—are all serverless and autoscaling When you submit a query to BigQuery, it is run on thousands of nodes, and you get your result back; you don’t spin up a cluster or install any software Similarly, in Cloud Dataflow, when you submit a data pipeline, and in Cloud Machine Learning Engine, when you submit a machine learning job, you can process data at scale and train models at scale without worrying about cluster management or failure recovery Cloud Pub/Sub is a global messaging service that autoscales to the throughput and number of subscribers and publishers without any work on your part Even when you’re running open source software like Apache Spark that’s designed to operate on a cluster, Google Cloud Platform makes it easy Leave your data on Google Cloud Storage, not in HDFS, and spin up a job-specific cluster to run the Spark job After the job completes, you can safely delete the cluster Because of this job-specific infrastructure, there’s no need to fear overprovisioning hardware or running out of capacity to run a job when you need it Plus, data is encrypted, both at rest and in transit, and kept secure As a data scientist, not having to manage infrastructure is incredibly liberating The reason that you can afford to forget about virtual machines and clusters when running on Google Cloud Platform comes down to networking The network bisection bandwidth within a Google Cloud Platform datacenter is PBps, and so sustained reads off Cloud Storage are extremely fast What this means is that you don’t need to shard your data as you would with traditional MapReduce jobs Instead, Google Cloud Platform can autoscale your compute jobs by shuffling the data onto new compute nodes as needed Hence, you’re liberated from cluster management when doing data science on Google Cloud Platform These autoscaled, fully managed services make it easier to implement data science models at scale— which is why data scientists no longer need to hand off their models to data engineers Instead, they can write a data science workload, submit it to the cloud, and have that workload executed automatically in an autoscaled manner At the same time, data science packages are becoming simpler and simpler So, it has become extremely easy for an engineer to slurp in data and use a canned model to get an initial (and often very good) model up and running With well-designed packages and easyto-consume APIs, you don’t need to know the esoteric details of data science algorithms—only what each algorithm does, and how to link algorithms together to solve realistic problems This convergence between data science and data engineering is why you can stretch your wings beyond your current role Rather than simply read this book cover-to-cover, I strongly encourage you to follow along with me by also trying out the code The full source code for the end-to-end pipeline I build in this book is on GitHub Create a Google Cloud Platform project and after reading each chapter, try to repeat what I did by referring to the code and to the README.md1 file in each folder of the GitHub repository Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords Constant width bold Shows commands or other text that should be typed literally by the user Constant width italic Shows text that should be replaced with user-supplied values or by values determined by context TIP This element signifies a tip or suggestion NOTE This element signifies a general note WARNING This element indicates a warning or caution Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/GoogleCloudPlatform/data-science-on-gcp This book is here to help you get your job done In general, if example code is offered with this book, you may use it in your programs and documentation You not need to contact us for permission unless you’re reproducing a significant portion of the code For example, writing a program that uses several chunks of code from this book does not require permission Selling or distributing a CDROM of examples from O’Reilly books does require permission Answering a question by citing this book and quoting example code does not require permission Incorporating a significant amount of example code from this book into your product’s documentation does require permission We appreciate, but not require, attribution An attribution usually includes the title, author, publisher, and ISBN For example: “Data Science on the Google Cloud Platform by Valliappa Lakshmanan (O’Reilly) Copyright 2018 Google Inc., 978-1-491-97456-8.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com O’Reilly Safari Safari (formerly Safari Books Online) is a membership-based training and reference platform for enterprise, government, educators, and individuals Members have access to thousands of books, training videos, Learning Paths, interactive tutorials, and curated playlists from over 250 publishers, including O’Reilly Media, Harvard Business Review, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Adobe, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, and Course Technology, among others For more information, please visit http://oreilly.com/safari How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information You can access this page at http://bit.ly/datasci_GCP To comment or ask technical questions about this book, send email to bookquestions@oreilly.com For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia Acknowledgments When I took the job at Google about a year ago, I had used the public cloud simply as a way to rent infrastructure—so I was spinning up virtual machines, installing the software I needed on those machines, and then running my data processing jobs using my usual workflow Fortunately, I realized that Google’s big data stack was different, and so I set out to learn how to take full advantage of all the data and machine learning tools on Google Cloud Platform The way I learn best is to write code, and so that’s what I did When a Python meetup group asked me to talk about Google Cloud Platform, I did a show-and-tell of the code that I had written It turned out that a walk-through of the code to build an end-to-end system while contrasting different approaches to a data science problem was quite educational for the attendees I wrote up the essence of my talk as a book proposal and sent it to O’Reilly Media A book, of course, needs to have a lot more depth than a 60-minute code walk-through Imagine that you come to work one day to find an email from a new employee at your company, someone who’s been at the company less than six months Somehow, he’s decided he’s going to write a book on the pretty sophisticated platform that you’ve had a hand in building and is asking for your help He is not part of your team, helping him is not part of your job, and he is not even located in the same office as you What is your response? Would you volunteer? What makes Google such a great place to work is the people who work here It is a testament to the company’s culture that so many people—engineers, technical leads, product managers, solutions architects, data scientists, legal counsel, directors—across so many different teams happily gave of their expertise to someone they had never met (in fact, I still haven’t met many of these people in person) This book, thus, is immeasurably better because of (in alphabetical order) William Brockman, Mike Dahlin, Tony Diloreto, Bob Evans, Roland Hess, Brett Hesterberg, Dennis Huo, Chad Jennings, Puneith Kaul, Dinesh Kulkarni, Manish Kurse, Reuven Lax, Jonathan Liu, James Malone, Dave Oleson, Mosha Pasumansky, Kevin Peterson, Olivia Puerta, Reza Rokni, Karn Seth, Sergei Sokolenko, and Amy Unruh In particular, thanks to Mike Dahlin, Manish Kurse, and Olivia Puerta for reviewing every single chapter When the book was in early access, I received valuable error reports from Anthonios Partheniou and David Schwantner Needless to say, I am responsible for any errors that remain A few times during the writing of the book, I found myself completely stuck Sometimes, the problems were technical Thanks to (in alphabetical order) Ahmet Altay, Eli Bixby, Ben Chambers, Slava Chernyak, Marian Dvorsky, Robbie Haertel, Felipe Hoffa, Amir Hormati, Qi-ming (Bradley) Jiang, Kenneth Knowles, Nikhil Kothari, and Chris Meyers for showing me the way forward At other times, the problems were related to figuring out company policy or getting access to the right team, document, or statistic This book would have been a lot poorer had these colleagues not unblocked me at critical points (again in alphabetical order): Louise Byrne, Apurva Desai, Rochana Golani, Fausto Ibarra, Jason Martin, Neal Mueller, Philippe Poutonnet, Brad Svee, Jordan Tigani, William Vampenebe, and Miles Ward Thank you all for your help and encouragement Thanks also to the O’Reilly team—Marie Beaugureau, Kristen Brown, Ben Lorica, Tim McGovern, Rachel Roumeliotis, and Heather Scherer for believing in me and making the process of moving from draft to published book painless Finally, and most important, thanks to Abirami, Sidharth, and Sarada for your understanding and patience even as I became engrossed in writing and coding You make it all worthwhile For example, see https://github.com/GoogleCloudPlatform/data-science-ongcp/blob/master/06_dataproc/README.md project creation, Getting Started with the Code-Getting Started with the Code reference architecture, Real-Time Stream Processing REST API actions, Interacting with Google Cloud Platform speed of networking on, Data in Situ with Colossus and Jupiter support for data engineers, The Cloud Makes Data Engineers Possible-The Cloud Makes Data Engineers Possible Google Cloud SQL creating instances, Create a Google Cloud SQL Instance loading data onto, Loading Data into Google Cloud SQL streaming pipelines and, Possible Streaming Sinks Google Cloud Storage copying input text files to, Run on Cloud file handling in, Uploading Data to Google Cloud Storage job speeds on, Data in Situ with Colossus and Jupiter removing duplicates, Removing Duplicates-Removing Duplicates staging BigQuery data on, Staging on Cloud Storage uploading data to, Uploading Data to Google Cloud Storage-Uploading Data to Google Cloud Storage Google File System (GFS), Data in Situ with Colossus and Jupiter, Apache Hadoop Graphic Detail blog, Accuracy, Honesty, and Good Design graphics, types of, Accuracy, Honesty, and Good Design (see also dashboards) H Hadoop Distributed File System (HDFS), Scaling Out, Data in Situ with Colossus and Jupiter, Apache Hadoop Hadoop ecosystem, Need for Higher-Level Tools head-of-line blocking, Data in Situ with Colossus and Jupiter help command, Interacting with Google Cloud Platform hexbin plot, Independence Check Using BigQuery histogram equalization, Histogram Equalization-Histogram Equalization hyperparameter tuning, Hyperparameter Tuning-Changing learning rate, Summary I independence check, Independence Check Using BigQuery-Independence Check Using BigQuery independent samples, Evaluating the Model ingesting data into the cloud, Airline On-Time Performance Data-Dataset Attributes initialization actions, Initialization Actions integrated development environment (IDE), Setting Up Development Environment interactive data exploration benefits of, Exploratory Data Analysis development of, Interactive Data Exploration-Exploratory Data Analysis goals of, Exploratory Data Analysis loading flight data into BigQuery, Loading Flights Data into BigQuery-Reading a query explanation model development, Arrival Delay Conditioned on Departure Delay-The Answer Is model evaluation, Evaluating the Model-Training and Testing, Evaluating the Bayesian Model, Evaluating a Model quality control, Quality Control-Filtering Data on Occurrence Frequency using Cloud Datalab, Exploratory Data Analysis in Cloud Datalab-Exploring arrival delays J Java benefits of, Dataflow in Java development environment setup, Setting Up Development Environment filtering data with Beam, Filtering with Beam-Filtering with Beam parsing CSV files into objects, Parsing into Objects-Parsing into Objects representing JSON request and response as, Java Classes for Request and Response running pipelines in the cloud, Run on Cloud specifying input and output files, Pipeline Options and Text I/O Java Software Development Kit (SDK), Setting Up Development Environment Java Database Connectivity (JDBC), Need for Higher-Level Tools Jupiter networks, Data in Situ with Colossus and Jupiter-Data in Situ with Colossus and Jupiter Jupyter Notebooks (see also Cloud Datalab) overview of, Jupyter Notebooks support for multiple interpreters, Jupyter Magic for Google Cloud Platform-Exploring arrival delays K kernel density plots, Exploring arrival delays knowability, Knowability L L-BFGS algorithm, Spark Logistic Regression latency, Transactions, Throughput, and Latency-Querying from Cloud Bigtable learning rate, changing, Changing learning rate line graphics, Accuracy, Honesty, and Good Design LinearClassifier class (TensorFlow), Linear Classifier-Deep Neural Network Model local aggregation, Need for Higher-Level Tools logistic loss function, Spark Logistic Regression logistic regression, Logistic Regression-Evaluating a Model, Toward More Complex Models as a classification method, Logistic Regression data verification, Dealing with Corner Cases expression as a neural network, Toward More Complex Models making predictions, Predicting by Using a Model methodology, Logistic Regression model development, Spark ML Library model evaluation, Evaluating a Model Spark ML library for, Spark ML Library Spark ML regression choices, Spark Logistic Regression Spark ML setup, Getting Started with Spark Machine Learning training dataset creation, Creating a Training Dataset training example creation, Creating Training Examples training the model, Training M machine learning classifier using TensorFlow, Machine Learning Classifier Using TensorFlow-Summary creating more complex models, Toward More Complex Models-Toward More Complex Models feature engineering, Feature Engineering-Scalable, Repeatable, Real Time heart of, Machine Learning logistic loss function, Spark Logistic Regression logistic regression on Spark, Logistic Regression-Evaluating a Model minimizing cross-entropy, Spark Logistic Regression real-time, Real-Time Machine Learning-Summary supervised learning problems, Spark ML Library time-windowed aggregate features, Time-Windowed Aggregate Features-Summary making better decisions based on data, Making Better Decisions Based on Data-Summary map operations, Scaling Out MapReduce, Scaling Out, MapReduce and the Hadoop Ecosystem-How MapReduce Works marginal distributions, Marginal Distributions Maven, Windowing a pipeline, Setting Up Development Environment, Run on Cloud Millwheel, Apache Beam/Cloud Dataflow ML Engine (see Cloud ML Engine) monitoring, troubleshooting, and performance tuning, Monitoring, Troubleshooting, and Performance Tuning-Removing Duplicates MySQL benefits of, Loading Data into Google Cloud SQL connecting to databases, Controlling Access to MySQL controlling access to, Controlling Access to MySQL creating tables in, Create Tables determining host IP address, Create Tables importing files into, Create Tables managed service on Google Cloud SQL, Create a Google Cloud SQL Instance passwords, Controlling Access to MySQL populating tables in, Populating Tables verifying import success, Create Tables N Naive Bayes, Quantization Using Spark SQL narrative graphics, Accuracy, Honesty, and Good Design normal distribution, Normal distribution O one-hot encoding, Categorical Variables, Linear Classifier, Embeddings out-of-order records, Late and Out-of-Order Records overfitting, Toward More Complex Models P pairs technique, Need for Higher-Level Tools Pandas dataframe, Exploring arrival delays, Arrival Delay Conditioned on Departure Delay, Marginal Distributions parallel collection (PCollection), Parallel Do with Side Input, Redesigning the Pipeline, Flattening PCollections parsimony, principle of, Feature Engineering passwords, Controlling Access to MySQL performance tuning, Monitoring, Troubleshooting, and Performance Tuning-Removing Duplicates persistent drives, Scaling Up pie charts, Showing Proportions with a Pie Chart-Showing Proportions with a Pie Chart Pig Latin, Need for Higher-Level Tools (see also Apache Pig) polling, Scheduling Monthly Downloads prediction service, invoking, Invoking Prediction Service-Client of Prediction Service predictions, optimizing, Spark ML Library preemptible instances, Jobs, Not Clusters, Dynamically Resizing Clusters principle of parsimony, Feature Engineering probabilistic decision making, A Probabilistic Decision-A Probabilistic Decision probabilistic decision-making, Applying Probabilistic Decision Threshold, Quantization Using Spark SQL probability density function (PDF), A Probabilistic Decision, Exploring arrival delays Project Object Model (POM), Setting Up Development Environment Pub/Sub (see Cloud Pub/Sub) Python, ingesting in, Ingesting in Python-Ingesting in Python Q quality control (see also troubleshooting) filtering data on occurrence frequency, Filtering Data on Occurrence Frequency filtering data on occurrence frequency, Filtering Data on Occurrence Frequency recognizing outliers, Quality Control removing outliers, Outlier Removal: Big Data Is Different quantization thresholds, Quantization Using Spark SQL-Dynamically Resizing Clusters Bayes classification for, Quantization Using Spark SQL calculating conditional probability, Quantization Using Spark SQL determining using Spark SQL, Quantization Using Spark SQL dynamically resizing clusters, Dynamically Resizing Clusters histogram equalization and, Histogram Equalization independence check using BigQuery, Independence Check Using BigQuery using Datalab on Dataproc, Google Cloud Datalab on Cloud Dataproc using Spark on Datalab, Spark SQL in Google Cloud Datalab questions and comments, How to Contact Us R R programming language, Loading Flights Data into BigQuery random shuffling, Random Shuffling real-time machine learning adding predictions to flight information, Adding Predictions to Flight Information-Batching Requests evaluating model performance, Evaluating Model Performance-Identifying Behavioral Change invoking prediction service, Invoking Prediction Service-Client of Prediction Service selecting output sinks, Transactions, Throughput, and Latency-Querying from Cloud Bigtable streaming pipeline, Streaming Pipeline-Watermarks and Triggers real-time stream processing analyzing streaming data, Analyzing Streaming Data in BigQuery-Analyzing Streaming Data in BigQuery co-join by key, Co-join by key executing stream processing, Executing the Stream Processing Java dataflow, Streaming in Java Dataflow real-time dashboard, Real-Time Dashboard-Real-Time Dashboard reference architecture for, Real-Time Stream Processing streaming aggregation, Streaming aggregation-Streaming aggregation windowing pipelines, Windowing a pipeline Rectified Linear Units (ReLUs), Toward More Complex Models reduce operations, Scaling Out reduce stragglers, Need for Higher-Level Tools reference architecture, Real-Time Stream Processing relational databases, Loading Data into Google Cloud SQL, Advantages of a Serverless Columnar Database relational graphics, Accuracy, Honesty, and Good Design requests, batching, Batching Requests resource stranding, Data in Situ with Colossus and Jupiter REST API calls, Interacting with Google Cloud Platform reverse-engineering web forms, Reverse Engineering a Web Form-Reverse Engineering a Web Form Root Mean Squared Error (RMSE), Experimental Framework, Performing a Training Run, Evaluating Performance row keys (Cloud Bigtable), Designing the Row Key S scaling out, Scaling Out-Scaling Out scaling up, Scaling Up-Scaling Up scatter plots, Accuracy, Honesty, and Good Design, Creating Charts Secure Shell (SSH), Google Cloud Dataproc securing URLs, Securing the URL security issues (see also sensitive data) bucket names, Uploading Data to Google Cloud Storage controlling access to BigQuery, Access Control-Access Control controlling access to MySQL, Controlling Access to MySQL frequency analysis attacks, Handling Sensitive Information privacy, Many Similar Decisions securing URLs, Securing the URL sensitive data governance policy for, Establishing a Governance Policy handling of, Handling Sensitive Information identifying, Identifying Sensitive Data-Sensitive data in unstructured content protecting, Protecting Sensitive Data-Coarsening Sensitive Data security implications of, Considerations for Sensitive Data within Machine Learning Datasets service accounts, Access Control serving, Spark ML Library sharding, Who This Book Is For, Scaling Out side-inputs, Parallel Do with Side Input, Side Input Limitations-Side Input Limitations single-region buckets, Uploading Data to Google Cloud Storage, Uploading Data to Google Cloud Storage single-user projects, Access Control Software-Defined Networking (SDN), Data in Situ with Colossus and Jupiter Spark ML algorithms included in, Spark ML Library data verification, Dealing with Corner Cases logistic loss function and, Spark Logistic Regression logistic regression choices, Spark Logistic Regression making predictions, Predicting by Using a Model model evaluation, Evaluating a Model model training, Training setup, Getting Started with Spark Machine Learning SparkContext variable sc, Getting Started with Spark Machine Learning SparkSession variable spark, Getting Started with Spark Machine Learning training dataset creation, Creating a Training Dataset training example creation, Creating Training Examples Spark SQL on Cloud Datalab, Spark SQL in Google Cloud Datalab-Spark SQL in Google Cloud Datalab overview of, Need for Higher-Level Tools quantization using, Quantization Using Spark SQL-Dynamically Resizing Clusters speed-up factor, Publishing an Event Stream to Cloud Pub/Sub Stackdriver, Marginal Distributions streaming aggregation, Streaming aggregation streaming data, publication and ingest, Streaming Data: Publication and Ingest-Summary (see also data streaming) streaming pipelines, Streaming Pipeline-Watermarks and Triggers stripes technique, Need for Higher-Level Tools strong consistency, Uploading Data to Google Cloud Storage supervised learning problems, Spark ML Library T tables contingency table, Contingency Table, Explaining a Contingency Table-Explaining a Contingency Table creating, Create Tables-Create Tables designing in Cloud Bigtable, Designing Tables populating, Populating Tables-Populating Tables tablets (Cloud Bigtable), Cloud Bigtable TensorFlow benefits of, Machine Learning Classifier Using TensorFlow, Summary creating more complex models, Toward More Complex Models-Toward More Complex Models Experiment class, Setting Up an Experiment-Distributed Training in the Cloud, Summary LinearClassifier class, Linear Classifier-Deep Neural Network Model model deployment, Deploying the Model-Explaining the Model model improvement, Improving the ML Model-Changing learning rate reading data into, Reading Data into TensorFlow-Reading Data into TensorFlow setting up experiments, Setting Up an Experiment-Distributed Training in the Cloud TensorFlow CSV reader, Reading Data into TensorFlow TFRecord files, Reading Data into TensorFlow TFRecord files, Reading Data into TensorFlow threshold optimization, Threshold Optimization throughput, Transactions, Throughput, and Latency-Querying from Cloud Bigtable time averages computing, Computing Time Averages-Running in the Cloud need for, The Need for Time Averages-The Need for Time Averages time, converting to UTC, Time Correction, Converting Times to UTC-Converting Times to UTC time-windowed aggregate features computing time averages, Computing Time Averages-Running in the Cloud Dataflow in Java, Dataflow in Java-Parsing into Objects monitoring, troubleshooting, and tuning, Monitoring, Troubleshooting, and Performance Tuning-Removing Duplicates need for time averages, The Need for Time Averages-The Need for Time Averages time-windowed analytics, Windowing a pipeline time–series graphics, Accuracy, Honesty, and Good Design tools and data, Data and Tools-Getting Started with the Code TPUs (Tensor Processing Units), Toward More Complex Models training examples, Creating Training Examples training-serving skew, Training–Serving Skew training–serving skew, Spark ML Library transactions, throughput, and latency, Transactions, Throughput, and Latency-Querying from Cloud Bigtable triggers, Watermarks and Triggers troubleshooting (see also quality control) batching requests, Batching Requests checking model behavior, Checking Model Behavior-Identifying Behavioral Change execution graphs, Monitoring, Troubleshooting, and Performance Tuning finding pipeline bottlenecks, Troubleshooting Pipeline identifying pipeline inefficiencies, Identifying Inefficiency pipeline redesign, Redesigning the Pipeline removing duplicates, Removing Duplicates-Removing Duplicates side-inputs, Side Input Limitations-Side Input Limitations typographical conventions, Conventions Used in This Book U uploading data, Uploading Data to Google Cloud Storage-Uploading Data to Google Cloud Storage URLs, securing, Securing the URL US Bureau of Transportation Statistics (BTS), Ingesting Data into the Cloud V View (Cloud Dataflow), Parallel Do with Side Input violin plots, Exploratory Data Analysis, Exploring arrival delays virtual machines (VMs), Who This Book Is For visualization, Creating Compelling Dashboards (see also dashboards) W watermarks (Apache Beam), Watermarks and Triggers web forms reverse engineering, Reverse Engineering a Web Form-Reverse Engineering a Web Form scripting, Ingesting Data wide-and-deep models, Wide-and-Deep Model-Wide-and-Deep Model Y YARN, Apache Hadoop About the Author Valliappa (Lak) Lakshmanan is currently a Tech Lead for Data and Machine Learning Professional Services for Google Cloud His mission is to democratize machine learning so that it can be done by anyone anywhere using Google’s amazing infrastructure, without deep knowledge of statistics or programming or ownership of a lot of hardware Before Google, he led a team of data scientists at the Climate Corporation and was a Research Scientist at NOAA National Severe Storms Laboratory, working on machine learning applications for severe weather diagnosis and prediction Colophon The animal on the cover of Data Science on the Google Cloud Platform is a buff-breasted sandpiper (Calidris subruficollis) While most sandpipers are considered shorebirds, this species is uncommon near the coast It breeds in tundra habitat in Canada and Alaska, and migrates thousands of miles to South America during winter, flying over the Midwest region of the United States Some small flocks can be found in Great Britain and Ireland Buff-breasted sandpipers are small birds, about 7–9 inches in length and with an average wingspan of 18 inches They have brown plumage on their backs, as well as light tan chests that give them their common name During mating season, the birds gather on display grounds (called “leks”) where the males point their bills upward, raise their wings to show the white undersides, and shake their body They may mate with multiple females if successful Female sandpipers have separate nesting grounds, where they lay eggs on the ground in a shallow hole lined with moss, leaves, and other plant matter Insects are the primary food source of the sandpiper; it hunts by sight by standing still and making a short run forward to catch the prey in its short thin bill Outside of breeding season, buff-breasted sandpipers prefer habitat with short grass—airfields, plowed fields, and golf courses are common resting places for the birds as they pass through or winter in developed areas They are currently classified as Near Threatened due to pesticide use, as well as habitat loss in their Arctic breeding grounds The cover image is from British Birds III The cover fonts are URW Typewriter and Guardian Sans The text font is Adobe Minion Pro; the heading font is Adobe Myriad Condensed; and the code font is Dalton Maag’s Ubuntu Mono .. .Data Science on the Google Cloud Platform Implementing End -to- End Real-Time Data Pipelines: From Ingest to Machine Learning Valliappa Lakshmanan Data Science on the Google Cloud Platform... solutions to your thorniest problems You don’t need to be a polymath to be a data engineer—you simply need to learn how to data science on the cloud Saying that the cloud is what makes data engineers... everything on the public cloud, we found that we could store all of the radar data on cloud storage, and as long as we were accessing it from virtual machines (VMs) in the same region, data transfer