-
Recent Posts
Recent Comments
- Marcel-Jan Krijgsman on I built a working Hadoop-Spark-Hive cluster on Docker. Here is how.
- Chris on I built a working Hadoop-Spark-Hive cluster on Docker. Here is how.
- admin_r0g1nuq9 on I built a working Hadoop-Spark-Hive cluster on Docker. Here is how.
- LJ on I built a working Hadoop-Spark-Hive cluster on Docker. Here is how.
- admin_r0g1nuq9 on I built a working Hadoop-Spark-Hive cluster on Docker. Here is how.
Archives
Categories
Meta
-
Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy
Category Archives: Learning Big Data
I built a working Hadoop-Spark-Hive cluster on Docker. Here is how.
TL;DR: I made a Docker compose that runs Hadoop, Spark and Hive in a multi-container environment. You can find the necessary files for it here: https://github.com/Marcel-Jan/docker-hadoop-spark How it started We at DIKW are working on a Certified Data Engineering … Continue reading
Posted in Howto, Learning Big Data, Spark
Tagged Apache Spark, Big Data Europe, DIKW, Docker, docker-compose, Hadoop, Hive
8 Comments
R Studio: Doing data science on my health data – Part 1
Up to seven years ago my doctor would nag me every half year that I should lose some weight. Nagging didn’t work on me that much. What did work however was competition. I wanted to become faster in a bike … Continue reading
Posted in Learning Big Data, Weird experiments
Tagged Big Data, ggplot2, health data, R Studio, Udemy
Leave a comment
Check your /tmp on HDFS
If you have sensitive data on your Hadoop cluster, you might want to check /tmp on HDFS once a while to see what ends up there. /tmp is used by several components. Hive for example stores its “scratch data” there. … Continue reading
Making a Hertzsprung-Russell diagram from Gaia DR2 data with Elasticsearch
Elasticsearch was one of the open source products on my list to try out, ever since I got rejected for a couple of assignments as a consultant last year. Apparently it’s a popular product. But why do you need a … Continue reading
Posted in Learning Big Data, NoSQL
Tagged 2001 A Space Odyssey, ElasticSearch, Frank Kane, Gaia, Hertzsprung-Russell diagram, Kibana, Logstash, Udemy, Vega
Leave a comment
Building HDP 2.6 on AWS, Part 3: the worker nodes
This is part 3 in a series on how to build a Hortonworks Data Platform 2.6 cluster on AWS. By now we have an edge node to run Ambari Server, three master nodes for Hadoop name nodes and such. Now … Continue reading
Posted in Howto, Learning Big Data
Tagged Amazon Web Services, AWS, cloning nodes, Hadoop, HDP, Hortonworks Data Platform, Ubuntu Server, worker nodes
Leave a comment
Hadoop High Availability In A Hurry – Part 2: YARN
If you don’t know a lot about YARN and why it’s called a data operating system, you’re in luck. I found it necessary to explain how YARN works before I could explain the solutions for high availability. At first YARN … Continue reading
Posted in Apache Products for Outsiders, Learning Big Data
Tagged Application Master, Container, Hadoop, Node Manager, Resource Manager, YARN, ZooKeeper
1 Comment
Hadoop High Availability In A Hurry – Part 1: HDFS
I’ve been studying for a couple of hours how Hadoop high availability works, for the HDPCA exam. And now I’ve condensed that knowledge to a video on HDFS HA in just under 9 minutes. Enjoy!
Posted in Apache Products for Outsiders, Learning Big Data
Tagged DataNode, edits file, Fencing, fsimage, Hadoop, HDFS, High availability, JournalNode, NameNode, Split brain, ZKFC, ZooKeeper
1 Comment