Tag Archives: Apache Spark

I built a working Hadoop-Spark-Hive cluster on Docker. Here is how.

Posted on October 25, 2020 by Marcel-Jan Krijgsman

TL;DR: I made a Docker compose that runs Hadoop, Spark and Hive in a multi-container environment. You can find the necessary files for it here: https://github.com/Marcel-Jan/docker-hadoop-spark [Update 2021-11-09: Since Docker Desktop turned “Expose daemon on tcp://localhost:2375 without TLS” off by … Continue reading →

Posted in Howto, Spark | Tagged , , , , , , | 23 Comments

Book review: Spark in Action, 2nd edition

Posted on September 1, 2019 by Marcel-Jan Krijgsman

There are lots of books on Spark, but not a lot that aimed at the data engineer. Data engineers use Spark to ingest and transform data, which is different from what data scientists use it for. On the Roaring Elephant … Continue reading →

Posted in Data engineering, Spark | Tagged , , , | 2 Comments

Dataworks Summit Berlin 2018, day one

Posted on April 19, 2018 by Marcel-Jan Krijgsman

I’m back at Dataworks Summit this year. This time I didn’t win any ticket, but my new employer, Port of Rotterdam, has arranged that I could go. Pretty cool, because I did not want to miss it. This time it’s … Continue reading →

Posted in Conferences, Events | Tagged , , , , , , , | 2 Comments