{"id":236,"date":"2018-04-10T14:54:41","date_gmt":"2018-04-10T14:54:41","guid":{"rendered":"http:\/\/marcel-jan.eu\/datablog\/?p=236"},"modified":"2018-04-11T12:23:52","modified_gmt":"2018-04-11T12:23:52","slug":"building-hdp-2-6-on-aws-part-3-the-worker-nodes","status":"publish","type":"post","link":"https:\/\/marcel-jan.eu\/datablog\/2018\/04\/10\/building-hdp-2-6-on-aws-part-3-the-worker-nodes\/","title":{"rendered":"Building HDP 2.6 on AWS, Part 3: the worker nodes"},"content":{"rendered":"

This is part 3 in a series on how to build a Hortonworks Data Platform 2.6 cluster on AWS. By now we have an edge node<\/a> to run Ambari Server, three master nodes<\/a> for Hadoop name nodes and such. Now we need worker nodes for processing the data.<\/p>\n

Creating the worker nodes is not that much different from creating the master nodes. But the workers need more powerful nodes.<\/p>\n

Creating the first worker node<\/h1>\n

Log in at Amazon Web Services again, in the same AWS district as the edge and master nodes. We start with one worker node and clone 2 more later on. Go to the EC2 dashboard in the AWS interface and click \u201cLaunch instance\u201d. Then choose Ubuntu Server 16.04 from the Amazon Machine Images.<\/p>\n

\"\"<\/p>\n

For the workers we need machines with a little more oomph. Select a general purpose instance with type m4.2xlarge.<\/p>\n

\"\"<\/p>\n

Networking<\/h2>\n

The worker nodes will be in the same subnet as the masters.<\/p>\n

Storage<\/h2>\n

This is possibly more storage than necessary. 100GB for root, 25 GB of type EBS for \/dev\/sdc and 100 GB of type EBS for \/dev\/sdb.<\/p>\n

\"\"<\/p>\n

Tags<\/h2>\n

The tags of course are different.<\/p>\n

\"\"<\/p>\n

Review<\/h2>\n

\"\"<\/p>\n

Extra preparation to install Ambari and HDP<\/h2>\n

You need unzip:<\/p>\n

sudo apt-get update\r\n\r\nsudo apt install unzip<\/pre>\n
Elastic IP<\/h2>\n
Create a new Elastic IP and associate the worker node with it.<\/p>\n
 <\/p>\n
Clone the worker node<\/h1>\n
Now we clone the first worker node to two new workers. This works in the same way as we cloned the master nodes, except this time m4.2xlarge nodes were chosen.<\/p>\n
Change the tags in the instance list:<\/p>\n
\"\"<\/p>\n
The nodes have the same software as the first worker node, but passworless access is something you have to configure on all of them. You need to put the worker nodes in the\u00a0de OCS-POC edge \u2013 sg security group and associate them, one by one, with the Elastic IP, so you can log in directly.<\/p>\n
\"\"<\/p>\n
Test the connection<\/h2>\n
When the master nodes are started, you should have access as root from the edge nodes.<\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
This is part 3 in a series on how to build a Hortonworks Data Platform 2.6 cluster on AWS. By now we have an edge node to run Ambari Server, three master nodes for Hadoop name nodes and such. Now we need worker nodes for processing the data. Creating the […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55],"tags":[50,49,54,4,48,47,51,108],"class_list":["post-236","post","type-post","status-publish","format-standard","hentry","category-howto","tag-amazon-web-services","tag-aws","tag-cloning-nodes","tag-hadoop","tag-hdp","tag-hortonworks-data-platform","tag-ubuntu-server","tag-worker-nodes"],"_links":{"self":[{"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/posts\/236","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/comments?post=236"}],"version-history":[{"count":8,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/posts\/236\/revisions"}],"predecessor-version":[{"id":477,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/posts\/236\/revisions\/477"}],"wp:attachment":[{"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/media?parent=236"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/categories?post=236"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/tags?post=236"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}