[Update 2021-11-09<\/strong>: Since Docker Desktop turned “Expose daemon on tcp:\/\/localhost:2375 without TLS” off by default there have been all kinds of connection problems running the complete docker-compose. Turning this option on again (Settings > General > Expose daemon on tcp:\/\/localhost:2375 without TLS) makes it all work. I’m still looking for a more secure solution to this]<\/p>\n <\/p>\n We at DIKW<\/a> are working on a Certified Data Engineering Professional course<\/a>. It is a course where you learn all aspects we could think of of being a data engineer: the cool big data stuff, but also how data warehousing works and how it all can work together.<\/p>\n One of the topics is Hadoop. Now our course has an important practical aspect. We’re not just going to bombard you with theory. You have to try the products\/methods yourself. On your own laptop. So for the Hadoop module I suggested using the Cloudera sandbox<\/a> on Docker, because our practice environments work on Docker and the Cloudera sandbox has it all.<\/p>\n And at one moment my colleague Hugo Koopmans told me we had a problem: building the Cloudera sandbox on his laptop took way too long and required way too much memory. Could we use an simpler (and much older) Hadoop implementation instead?<\/p>\n My thoughts were: Simpler? Yes! Old version? No way! We’re not going to start a new course with a 5 year old Hadoop version. And off I went on a quest for lightweight Hadoop cluster on Docker. Ideally with Spark and maybe Hive. Because I like databases.<\/p>\n <\/p>\n After searching and finding all kinds of Hadoop on Docker images, I found most of them where old. But it turned out that Big Data Europe has a Docker environment with Hadoop 3.2.1<\/a> and it’s only 9 months old. Much better. Their Spark version<\/a> is also pretty much up to date.<\/p>\n But how to get the Spark nodes to connect to the Hadoop nodes? I could not get the docker-composed Hadoop nodes and the docker-composed Spark nodes to speak to each other.<\/p>\n (There was a reason for that and I just found out why. I thought I used Big Data Europe’s Spark setup, but it looks like I got a different one. One that had a spark-net network defined. And I can’t remember where I got it from. It looks like sdesliva26’s version<\/a> but it’s not that one either.)<\/p>\n Anyhow, I learned gradually I needed to combine the docker-compose.yml files somehow.<\/p>\n <\/p>\n Now when you’ve worked with docker-compose for a while, you might think “how hard could it be?”. But I had no idea what the principles of this thing were.<\/p>\n Docker-compose is a way to quickly create a multi-container environment. Perfect to create clusters, like a Hadoop cluster with a namenode and datanode. And it is defined in a docker-compose.yml file. But from the docker-compose.yml file there can be references to shell scripts to run or files with environmental settings. So you need these files too.<\/p>\n I’ve spent countless hours combining docker-compose services and trying to get it to work and not understanding why it would not. But after removing these Spark networks it worked much better. Turns out that when you don’t define any network in docker-compose, the services are all part of the same network that Docker creates automatically.<\/p>\n <\/p>\n You can skip this section if you just want to run the Docker Hadoop environment and don’t really care how. (Go ahead. I won’t be judgemental. That’s how I started myself.)<\/em><\/p>\n So here is a simplified example of one service I took from the Hadoop docker-compose.yml:<\/p>\n You can see it starts with a version. That’s the version of the docker-compose file and 3 is the latest.<\/p>\n The namenode service is based on an image prepared by Big Data Europe. Docker images are like blueprints for Docker containers. I sometimes think of the Docker image as an installation file and the container is the actual application running. I hope you get the idea. This service defintion refers to where the image can be found on Docker Hub. Docker Hub is like an app store for Docker images.<\/p>\n We can also see ports defined. Hadoop namenodes has some stuff running on these ports and we want to address these from outside the container. For example, when you have started the containers, you find namenode information on http:\/\/localhost:9870. Should you want a different port on your laptop, because multiple containers run on port 80 or something, an important thing to remember is that the first port is the port on the outside, the second is the one in the container.<\/p>\n The namenode also needs some permanent place to store data. For this there is the volume definition. But remember you have to add the list of volumes after defining the services also.<\/p>\n And there is a .env file apparently and it contains all kinds of environmental variables necessary for running Hadoop.<\/p>\n <\/p>\n Now you can download Big Data Europe’s docker-hadoop repository or from my docker-hadoop-spark repository and from the directory where you placed it all it takes is this command to get the multi-container environment running:<\/p>\n The -d means it runs in de background.<\/p>\n BTW the config file can have another name than docker-compose.yml. But then you need the -f option to point docker-compose to the correct file:<\/p>\n And you can break it all down again by going to that same directory and running this:<\/p>\n All the containers will then be stopped and removed. But: the images and volumes stay! So don’t be surprised that the csv file you uploaded to HDFS will still be there.<\/p>\n <\/p>\n It turns out you can copy paste services from the Spark docker-compose.yml to the Hadoop docker-compose.yml, provided that I added the directories provided in the docker-spark Github repository. And I learned that I needed to remove the spark-network network (wherever it came from).<\/p>\n And I thought I needed to break the environment down and build it up again every time I changed docker-compose.yml. Because that’s how stuff usually works. But not with docker-compose. You can edit the docker-compose.yml file and run “docker-compose up” again. I learned that after a whole lot of building up and breaking down docker environments BTW.<\/p>\n <\/p>\n So in the end it was a question of adding services from one docker-compose.yml to the other and all the necessary files. But it took me a while to understand how to use it and from where. But I got that all figured out now and I’ve written the quick starts for HDFS, Spark and Hive.<\/p>\n <\/p>\n Find the Container ID of the namenode.<\/p>\n Copy breweries.csv to the namenode.<\/p>\n Go to the bash shell on the namenode with that same Container ID of the namenode.<\/p>\n Create a HDFS directory \/data\/\/openbeer\/breweries.<\/p>\n Copy breweries.csv to HDFS:<\/p>\n <\/p>\nHow it started<\/h2>\n
The quest for a lightweight and up to date Hadoop cluster<\/h2>\n
Quick! Learn docker-compose<\/h2>\n
Disecting a docker-compose.yml file.<\/h2>\n
version: \"3\"\r\n\r\nservices:\r\n namenode:\r\n image: bde2020\/hadoop-namenode:2.0.0-hadoop3.2.1-java8\r\n container_name: namenode\r\n restart: always\r\n ports:\r\n - 9870:9870\r\n - 9000:9000\r\n volumes:\r\n - hadoop_namenode:\/hadoop\/dfs\/name\r\n environment:\r\n - CLUSTER_NAME=test\r\n - CORE_CONF_fs_defaultFS=hdfs:\/\/namenode:8020\r\n env_file:\r\n - .\/hadoop.env\r\n\r\nvolumes:\r\n hadoop_namenode:<\/pre>\n
Let’s get this thing started<\/h2>\n
docker-compose up -d<\/pre>\n
docker-compose up -f mymulticontainers.yml -d<\/pre>\n
docker-compose down<\/pre>\n
Combining docker-compose files<\/h2>\n
How the Hadoop-Spark-Hive docker-compose was built<\/h2>\n
Quick starts<\/h2>\n
Quick start HDFS<\/h3>\n
docker ps |grep namenode\r\n\r\n1df7a57164de bde2020\/hadoop-namenode:2.0.0-hadoop3.2.1-java8 \"\/entrypoint.sh \/run\u2026\" 27 hours ago Up 12 hours (healthy) 0.0.0.0:9000->9000\/tcp, 0.0.0.0:9870->9870\/tcp namenode\r\n\r\n<\/code><\/pre>\n docker cp breweries.csv 1df7a57164de:breweries.csv\r\n<\/code><\/pre>\n docker exec -it 1df7a57164de bash\r\n<\/code><\/pre>\nhdfs dfs -mkdir \/data\r\nhdfs dfs -mkdir \/data\/openbeer\r\nhdfs dfs -mkdir \/data\/openbeer\/breweries\r\n<\/code><\/pre>\n hdfs dfs -put breweries.csv \/data\/openbeer\/breweries\/breweries.csv\r\n<\/code><\/pre>\nQuick start Spark<\/h3>\n
![\"Screenshot](\"https:\/\/marcel-jan.eu\/datablog\/wp-content\/uploads\/2020\/10\/docker-hadoop-spark-hive-1024x650.png\")
<\/p>\n