There are lots of books on Spark, but not a lot that aimed at the data engineer. Data engineers use Spark to ingest and transform data, which is different from what data scientists use it for.<\/p>\n
On the Roaring Elephant podcast I heard an interview with Jean-Georges Perrin<\/a>, author of Spark in Action, 2nd Edition<\/a>, and it was clear that this would be a very data engineering centered Spark book. So I decided to buy the ebook (also because, as a Patreon of the Roaring Elephant podcast, I have a discount key at Manning Publishing).<\/p>\n Spark in Action, 2nd Edition, is not yet finished. It’s a so called MEAP (Manning Early Access Program), which means the author is still writing parts. But he already wrote chapters 1 to 15 and many appendices, so he seems pretty far advanced. I’ve read all the regular chapters and I can honestly say that I did a little proofreading.<\/p>\n <\/p>\n The one disadvantage of the book for me personally is that it’s mainly coded in Java. I know very little Java, though I can read it a bit. And learning Java is not high on my already long list of things to learn at this moment.\u00a0 I knew beforehand that this would be a slight challenge. Reading the book however, I think I could write a lot of the code in Python. I might do this later as a learning exercise.<\/p>\n Nevertheless I was able to follow the idea behind the book quite well. The main advantage of the book is that it is based on real life examples of a data engineer. It describes all kinds of ways that you might need to ingest data: different kinds of files (csv, JSON, XML, ORC), databases (relational and ElasticSearch), API’s and there’s a chapter on streaming.\u00a0 There also are a number of chapters on transformation and aggregation. If you know your SQL, a few exercises will look not that unfamiliar for you.<\/p>\n All in all this book was a very worthwhile read, despite my lack of Java knowledge. There’s a lot in here that data engineers starting with Spark will find useful.<\/p>\n","protected":false},"excerpt":{"rendered":" There are lots of books on Spark, but not a lot that aimed at the data engineer. Data engineers use Spark to ingest and transform data, which is different from what data scientists use it for. On the Roaring Elephant podcast I heard an interview with Jean-Georges Perrin, author of […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[191,192],"tags":[110,193,30,84],"class_list":["post-826","post","type-post","status-publish","format-standard","hentry","category-data-engineering","category-spark","tag-apache-spark","tag-jean-georges-perrin","tag-roaring-elephant-podcast","tag-spark"],"_links":{"self":[{"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/posts\/826","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/comments?post=826"}],"version-history":[{"count":4,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/posts\/826\/revisions"}],"predecessor-version":[{"id":857,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/posts\/826\/revisions\/857"}],"wp:attachment":[{"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/media?parent=826"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/categories?post=826"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/tags?post=826"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Java<\/h3>\n
File formats, databases and streaming<\/h3>\n