Category Archives: Howto

I built a working Hadoop-Spark-Hive cluster on Docker. Here is how.

Posted on October 25, 2020 by Marcel-Jan Krijgsman

TL;DR: I made a Docker compose that runs Hadoop, Spark and Hive in a multi-container environment. You can find the necessary files for it here: https://github.com/Marcel-Jan/docker-hadoop-spark   How it started We at DIKW are working on a Certified Data Engineering … Continue reading →

Posted in Howto, Learning Big Data, Spark | Tagged , , , , , , | 8 Comments

A humidity sensor network on a Raspberry Pi with Zigbee2MQTT

Posted on November 3, 2019 by Marcel-Jan Krijgsman

I was looking for a way to detect leakage in my appartement with some kind of IoT solution. Someone on the Dutch technology forum Tweakers.net told me Xiaomi Humidity sensors, combined with a Zigbee2MQTT might be a good fit. The … Continue reading →

Posted in Howto | Tagged , , , , , , | Leave a comment

Neo4J: Loading rocket data in a graph database

Posted on June 8, 2019 by Marcel-Jan Krijgsman

When I first learned about graph databases, like Neo4J, I didn’t get it. That’s how I always start with new technology: not getting at all why people getting so enthusiastic about them. Then I read “Seven Databases in Seven Weeks, … Continue reading →

Posted in Active Learning, Howto, NoSQL | Tagged , , , , | Leave a comment

Showing a complex Excel sheet who’s boss with Python and pandas

Posted on March 8, 2019 by Marcel-Jan Krijgsman

Data engineering isn’t always creating serverless APIs and ingressing terrabyte a minute streams with do-hickeys on Kubernetes. Sometimes people just want their Excel sheet in the data lake. Is that big data? Not even close. It’s very small. But for … Continue reading →

Posted in Howto, Python | Tagged , , , , , , , | 5 Comments

Building HDP 2.6 on AWS, Part 3: the worker nodes

Posted on April 10, 2018 by Marcel-Jan Krijgsman

This is part 3 in a series on how to build a Hortonworks Data Platform 2.6 cluster on AWS. By now we have an edge node to run Ambari Server, three master nodes for Hadoop name nodes and such. Now … Continue reading →

Posted in Howto, Learning Big Data | Tagged , , , , , , , | Leave a comment

Recovering your HDP 2.6.1 Sandbox on VirtualBox after a restart

Posted on November 17, 2017 by Marcel-Jan Krijgsman

If you’ve worked with the Hortonworks Data Platform 2.x sandbox of later versions in VirtualBox and made it shutdown rather vigorously, you might have noticed that you won’t get past this startup screen when you try to start it up … Continue reading →

Posted in Apache Products for Outsiders, Howto, Learning Big Data | Tagged , , , , , , , , , | 2 Comments

Fun with Data: Python and space rocks!

Posted on August 7, 2017 by Marcel-Jan Krijgsman

Last week I had a little fun with playing with Python, the pandas and matplotlib library and a JSON file with asteroid data. Here is what I did.

Posted in Howto, Python | Tagged , , , , , | Leave a comment

Building HDP 2.6 on AWS, Part 2: the master nodes

Posted on May 26, 2017 by Marcel-Jan Krijgsman

This is part 2 in a series on how to build a Hortonworks Data Platform 2.6 cluster on AWS. In part 1 we created an edge node where we will later install Ambari Server. The next step is creating the … Continue reading →

Posted in Howto, Learning Big Data | Tagged , , , , , , , | 5 Comments

Building HDP 2.6 on AWS, Part 1: the edge node

Posted on May 15, 2017 by Marcel-Jan Krijgsman

Installing Hortonworks Data Platform 2.6 on Amazon Web Services (Amazon’s cloud platform), how hard could it be? It’s click, click, next, next, confirm, right? Well-lll, not quite. Especially if HDP or AWS is new to you. There are many steps … Continue reading →

Posted in Howto, Learning Big Data | Tagged , , , , , , | 2 Comments