Dataproc Cookbook: Running Spark and Hadoop Workloads in Google Cloud - Softcover

Sadineni, Narasimha; Venkataraman, Anuyogam

Softcover

ISBN 10: 1098157702 ISBN 13: 9781098157708

Publisher: O'Reilly Media, 2025

Synopsis

Get up to speed with Dataproc, the fully managed and highly scalable service for running open source big data tools and frameworks, including Hadoop, Spark, Flink, and Presto. This cookbook shows data engineers, data scientists, data analysts, and cloud architects how to use Dataproc, integrated with Google Cloud, for data lake modernization, ETL, and secure data science at a fraction of the cost.

Narasimha Sadineni from Google and former Googler Anu Venkataraman show you how to set up and run Hadoop and Spark jobs on Dataproc. You'll learn how to create Dataproc clusters and run data engineering and data science workloads in long-running, ephemeral, and serverless ways. In the process, you'll gain an understanding of Dataproc, orchestration, logging and monitoring, Spark History Server, and migration patterns.

This cookbook includes hands-on examples for configuring, logging, securing clusters, and migrating from on-prem to Dataproc. You'll learn how to:

• Create Dataproc clusters on Compute Engine and Kubernetes Engine
• Run data science workloads on Dataproc
• Execute Spark jobs on Dataproc Serverless
• Optimize Dataproc clusters to be cost effective and performant
• Monitor Spark jobs in various ways
• Orchestrate various workloads and activities
• Use different methods for migrating data and workloads from existing Hadoop clusters to Dataproc

About this title

About the Authors

Narasimha Sadineni is a data engineer at Google who has 12 years of experience in Data & Analytics. While working as a professional services team member at Google and Cloudera, he helped 50+ organizations in solving BigData problems using tools like Hadoop and Google Cloud technologies. He has several years of teaching experience in Hadoop.

Anu Venkataraman is a Senior Program Manager. She previously served as a Data Lake Engineer at Google, accumulating extensive experience in data technologies. Anu assists customers in migrating large-scale distributed systems to the cloud. She finds joy in speaking at universities and contributing technical blogs and videos to the Data community, aiming to expedite customers' journeys to the cloud. Anu played a key role as one of the leads for the Professional Services Tech Talk playlist on the Google Cloud Tech YouTube channel. She holds a Master's degree in Electrical and Computer Engineering from Ryerson University, specializing in Medical Image Processing and Machine Learning.

Edition details

• Publisher: O'Reilly Media
• Publication date: 2025
• Language: English
• ISBN 10: 1098157702
• ISBN 13: 9781098157708
• Binding: Paperback
• Edition number: 1
• Number of pages: 436