Data engineering has gained even more relevance than before, and data engineering patterns are key to the successful implementation of data engineering projects. This book enables a data engineer to not only become familiar with data engineering patterns but also understand their application in real world use cases.

This book presents a comprehensive collection of data engineering patterns, each illustrated with relevant enterprise use cases to highlight their value and simplicity. It showcases both open-source and cloud technologies, guiding readers in building data systems for on-premise and cloud environments. The book covers patterns for data ingestion, transformation, storage, and serving, while also offering insights into performance engineering for data pipelines. Once we understand fundamental data engineering patterns, we then shift focus to patterns that help us build high-performance low latency data systems. We cover data caching, partitioning, replication, and how to select the technology stack for building out the patterns in this book.

By the end of the book, readers will have a deep understanding of various data engineering use cases and will be able to map the appropriate patterns to address them. They will also be equipped to choose the right technical stack for implementing these patterns, enabling them to create robust and efficient data systems in a secure and a cost-effective manner.

What you will learn

● Key data engineering patterns.

● Data ingestion and processing patterns.

● Modern architectures like Lambda.

● Explore time-tested data patterns of ETL and ELT.

● Modern data systems like data lake and medallion architectures.

● Domain-specific patterns and also on data orchestration, observability, and security.

● Overcoming performance challenges in building complex data systems.

Who this book is for

This book is designed for data engineers with beginner to intermediate experience in building enterprise-grade data systems. ETL developers transitioning into data engineering roles will also find this book valuable for understanding essential data engineering patterns. The code snippets provided throughout the book are written in Python or Scala, so a basic understanding of either language will help readers more easily grasp the concepts presented.

Table of Contents

1. Understanding Data Engineering

2. Data Engineering Patterns, Terminologies, and Technical Stack

3. Batch Ingestion and Processing

4. Real-time Ingestion and Processing

5. Micro-batching

6. Lambda Architecture

7. ETL and ELT

8. Data Fundamentals

9. Databases and Transactional Data

10. Data Warehouse and Data Analytics

11. Data Lake and Medallion Architecture

12. Data Replication and Partitioning

13. Hot Versus Cold Data Storage

14. Data Caching and Low Latency Serving

15. Data Search Patterns

16. Domain Specific Patterns

17. Data Security Patterns

18. Data Observability and Monitoring Patterns

19. Idempotency and Deduplication Patterns

20. Data Orchestration Patterns

21. Common Performance Pitfalls

22. Technology and Infrastructure Selection

23. Recap and Next Steps

About the Author

Amit Kulkarni has 14+ years of experience working in distributed systems, databases, and cloud storage systems. As a senior manager working in Couchbase India, he has gained expertise in building and managing large-scale, performant, and fault-tolerant systems.

"About this title" may belong to another edition of this title.

Store Description

Visit Seller's Storefront

Seller's business information

RAREWAVES.COM LIMITED
Elsley Court, 20-22 Great Titchfield Street, London, W1W 8BE, United Kingdom

Sale & Shipping Terms

Terms of Sale

Right of cancellation

If you are a consumer you can cancel the contract in accordance with the following. Consumer means any natural person who is acting for purposes which are outside his trade, business, craft or profession.

INFORMATION REGARDING THE RIGHT OF CANCELLATION

Statutory Right to cancel

You have the right to cancel this contract within 14 days for any reason.

The cancellation period will expire after 14 days from the day on which you acquire, or a third party other than the carrier and indicated by you acquires, physical possession of the the last good or the last lot or piece.

To exercise the right to cancel, you must inform us, Rarewaves, Unit 144 The Lightbox, 111 Power Road, W4 5PY, London, London, United Kingdom, of your decision to cancel this contract by a clear statement (e.g. a letter sent by post, fax or e-mail). You may use the attached model cancellation form, but it is not obligatory. You can also electronically fill in and submit a clear statement on our website, under "My Purchases" in "My Account". If you use this option, we will communicate to you an acknowledgement of receipt of such a cancellation on a durable medium (e.g. by e-mail) without delay.

To meet the cancellation deadline, it is sufficient for you to send your communication concerning your exercise of the right to cancel before the cancellation period has expired.

Effects of cancellation

If you cancel this contract, we will reimburse to you all payments received from you, including the costs of delivery (except for the supplementary costs arising if you chose a type of delivery other than the least expensive type of standard delivery offered by us).

We may make a deduction from the reimbursement for loss in value of any goods supplied, if the loss is the result of unnecessary handling by you.

We will make the reimbursement without undue delay, and not later than 14 days after the day on which we are informed about your decision to cancel with contract.

We will make the reimbursement using the same means of payment as you used for the initial transaction, unless you have expressly agreed otherwise; in any event, you will not incur any fees as a result of such reimbursement.

We may withhold reimbursement until we have received the goods back or you have supplied evidence of having sent back the goods, whichever is the earliest.

You shall send back the goods or hand them over to us or Rarewaves, Unit 144 The Lightbox, 111 Power Road, W4 5PY, London, London, United Kingdom, without undue delay and in any event not later than 14 days from the day on which you communicate your cancellation from this contract to us. The deadline is met if you send back the goods before the period of 14 days has expired. You will have to bear the direct cost of returning the goods. You are only liable for any diminished value of the goods resulting from the handling other than what is necessary to establish the nature, characteristics and functioning of the goods.

Exceptions to the right of cancellation

The right of cancellation does not apply to:

the delivery of newspapers, journals or magazines with the exception of subscription contracts; and
the supply of digital content which is not supplied on a tangible medium (e.g. on a CD or DVD) if you accepted when you placed your order that we could start to deliver it, and that you could not cancel it once delivery had started.

Model withdrawal form

(complete and return this form only if you wish to withdraw from the contract)

To: (Rarewaves, Unit 144 The Lightbox, 111 Power Road, W4 5PY, London, London, United Kingdom)

I/We (*) hereby give notice that I/We (*) withdraw from my/our (*) contract of sale of the following goods (*)/for the provision of the following goods (*)/for the provision of the following service (*),

Ordered on (*)/received on (*)

Name of consumer(s)

Address of consumer(s)

Signature of consumer(s) (only if this form is notified on paper)

Date

* Delete as appropriate.

Shipping Terms

Please note that we do not offer Priority shipping to any country.

We currently do not ship to the below countries:

Russia
Belarus
Ukraine
Israel

Please do not attempt to place orders with any of these countries as a ship to address - they will be cancelled.

Shipping rates from United Kingdom to U.S.A.

Shipping rates from United Kingdom to U.S.A.
Order quantity	7 to 12 business days	7 to 12 business days
First item	� 0.00	� 0.00

Delivery times are set by sellers and vary by carrier and location. Orders passing through Customs may face delays and buyers are responsible for any associated duties or fees. Sellers may contact you regarding additional charges to cover any increased costs to ship your items.