{"id":123,"date":"2017-05-11T07:32:02","date_gmt":"2017-05-11T07:32:02","guid":{"rendered":"http:\/\/marcel-jan.eu\/datablog\/?p=123"},"modified":"2017-05-11T07:32:02","modified_gmt":"2017-05-11T07:32:02","slug":"quickly-start-of-the-nifi-crash-course","status":"publish","type":"post","link":"https:\/\/marcel-jan.eu\/datablog\/2017\/05\/11\/quickly-start-of-the-nifi-crash-course\/","title":{"rendered":"Quickly start of the Nifi crash course"},"content":{"rendered":"

As I said last in my last blogpost<\/a>, I have followed the Apache NiFi crash course that Hortonworks provides. Now the tutorial describes several different scenarios and options and you have to read through that to find which you want. And you don’t have time for that. You’re probably doing this in your spare time and you have a whole Netflix backlog.<\/p>\n

So in this guide we cut right to the chase. It took me about 10 hours to follow Tutorial 0, 1, 2 and 3. But perhaps this guide can make you do it in about 4 hours.<\/p>\n

1. Preparing the VM<\/h2>\n

First download the Hortonworks Sandbox. There’s a VirtualBox (used in this example), VMWare and Docker image that come preinstalled with many products, but NiFi isn’t installed just yet (this guide is based on the HDP 2.6 sandbox).<\/p>\n

<\/p>\n

https:\/\/hortonworks.com\/products\/sandbox\/<\/p>\n

When you start it (here in VirtualBox), you can go to the prompt with Alt-F5 (or Mac: Fn-Alt-F5), but in this window it’s not really very friendly. You don’t have any control (that I could find) over the command line window, like number of lines and such.<\/p>\n

\"\"

The prompt in VirtualBox.<\/p><\/div>\n

So instead let’s use Putty<\/a>. Connect to 127.0.0.1 on port 2222.<\/p>\n

\"\"<\/p>\n

You can log in with root and password: hadoop. The last thing you have to change. While you’re at it, better change the ambari admin password right away also. Because we don’t have time for anything less than full privileges.<\/p>\n

# ambari-admin-password-reset\r\nPlease set the password for admin:<\/pre>\n
What you also can do, is put sandbox.hortonworks.com in your \/etc\/hosts or c:\\Windows\\System32\\Drivers\\etc\\hosts file. It’s not very necessary. But you will see that in Ambari the link to the NiFi UI doesn’t work, because your browser doesn’t go to sandbox.hortonworks.com. Instead you can also change it manually to 127.0.0.1. No biggie.<\/p>\n
127.0.0.1\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 localhost sandbox.hortonworks.com<\/pre>\n
2. Install NiFi<\/h2>\n
Open this address in your browser: http:\/\/127.0.0.1:8080<\/a>. Welcome to Ambari. The tutorial logs in with user: maria_dev, password: maria_dev. I overlooked that and used admin. Anyway, you will land at the dashboard, which will look something like this:<\/p>\n
\"\"
The Ambari dashboard<\/p><\/div>\n
Now unlike many other products, NiFi isn’t up and running yet. We’ll have to do a little bit of installation. In the Ambari dashboard, scroll down until you see a button Actions (on your left hand under the list of services). This is a drop down menu and there you’ll find the option Add Service.<\/p>\n
\"\"<\/p>\n
Now you can make a choice out of a whole lot of services, most of which are already installed.<\/p>\n
\"\"<\/p>\n
Scroll, scroll, scroll and you’ll find NiFi. Check that mark.<\/p>\n
\"\"<\/p>\n
Then it’s a lot of button pushing. These to be precise:<\/p>\n
\"\"<\/p>\n
Bit of waiting and..<\/p>\n
\"\"<\/p>\n
Click Next (I think it was) and you’ll get the summary. Click Complete.<\/p>\n
\"\"<\/p>\n
As you could see (before pushing Complete) some services have to be restarted. Don’t restart them one by one. Let Ambari sort it out. So in the Ambari dashboard, scroll down, click Actions and Restart All Required.<\/p>\n
\"\"<\/p>\n
After this, I’ve noticed serveral times, not every service is starting up all that well. For example Oozie has a big red warning mark next to it. For your NiFi tutorial this won’t matter, though.<\/p>\n
Still in the dashboard, go to the NiFi service (way down below in the services list). What you get now, is the NiFi dashboard. There’s a drop down menu called Quick Links and it has one option: NiFi UI. Now if you didn’t set up your (\/etc\/) hosts file, this is where you’ll end up nowhere. Instead go to: http:\/\/127.0.0.1:9090\/nifi\/<\/a>.<\/p>\n
\"\"<\/p>\n
And welcome to NiFi.<\/p>\n
\"\"
An empty NiFi canvas<\/p><\/div>\n
Now would probably the time to read about how NiFi works<\/a> (to, you know, understand what you’re doing). Or you can start the tutorial right at where the action is and still catch up on House of Cards.<\/p>\n
 <\/p>\n
3. Couple of last things<\/h2>\n
For the tutorials you can download templates, which I assume should install all necessary (data) processors right away. Now:<\/p>\n
    \n
  1. I haven’t seen it working for Lab 1 and haven’t tried any other templates.<\/li>\n
  2. We want this tutorial to go fast, but I assume you want to learn at least something about NiFi.<\/li>\n<\/ol>\n
    For Tutorial 1 you’ll need to upload a zip file with traffic patterns (go to tutorial 1<\/a>, under Approach 1). You can use WinSCP<\/a> to do that.<\/p>\n
    \"\"
    WinSCP connection to the sandbox.<\/p><\/div>\n
    Also, you’re going to need a couple of directories with 777 permissions to run the tutorial. Best to create these (as root), before you forget them.<\/p>\n
    cd \/tmp\r\nmkdir nifi\r\nchmod 777 nifi\r\nmkdir nifi\/input\r\nchmod 777 nifi\/input\r\nmkdir \/tmp\/nifi\/output\/filtered_transitLoc_data\r\nchmod 777 \/tmp\/nifi\/output\/filtered_transitLoc_data\r\nmkdir \/tmp\/nifi\/output\/nearby_neighborhoods_search\r\nchmod 777 \/tmp\/nifi\/output\/nearby_neighborhoods_search\r\nmkdir \/tmp\/nifi\/output\/nearby_neighborhoods_liveStream\r\nchmod 777 \/tmp\/nifi\/output\/nearby_neighborhoods_liveStream<\/pre>\n
    For tutorial 2<\/a> you will need a Google account, because you need to create an Application key. The tutorial will explain how. Now it is possible that in tutorial 2 the processors in line after InvokeHTTP after a while don’t work right. I had that problem and after a while, I checked the URLs NiFi was sending (which you can find by looking in the details of the Data provenance) in my browser\u00a0 and Google’s response was:<\/p>\n
    \"\"<\/p>\n
    Now according to Google this is your daily<\/em> request quota (if you use the Google’s Places API for free without further registration). So if this works how I think it works, you’ll have to call your NiFi experiment a day. To make this less likely to happen, better to adjust the ControlRate processor.<\/p>\n
    The way this works is: the rate is “Maximum Rate” per “Time Duration”. So either make Maximum Rate lower (but it is already 1) or Time Duration higher, like I did here.<\/p>\n
    \"\"<\/p>\n
     <\/p>\n
    You can now start at tutorial 1<\/a>, Approach 2 (step 2.1).<\/p>\n
    https:\/\/hortonworks.com\/hadoop-tutorial\/learning-ropes-apache-nifi\/#section_4<\/p>\n
    So have fun with Apache NiFi. I think you’ll find it’s a great product. In no time you can start your own NSA and direct all those data sources to your own Hadoop cluster. (Did I say that out loud?)<\/p>\n
    And with that, you can go back to Frank Underwood.<\/p>\n","protected":false},"excerpt":{"rendered":"
    As I said last in my last blogpost, I have followed the Apache NiFi crash course that Hortonworks provides. Now the tutorial describes several different scenarios and options and you have to read through that to find which you want. And you don’t have time for that. You’re probably doing […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[35],"tags":[12,44,42,23,43],"class_list":["post-123","post","type-post","status-publish","format-standard","hentry","category-apache-products-for-outsiders","tag-apache-nifi","tag-google-places-api","tag-hdp-sandbox","tag-hortonworks","tag-tutorial"],"_links":{"self":[{"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/posts\/123","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/comments?post=123"}],"version-history":[{"count":9,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/posts\/123\/revisions"}],"predecessor-version":[{"id":163,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/posts\/123\/revisions\/163"}],"wp:attachment":[{"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/media?parent=123"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/categories?post=123"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/tags?post=123"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}