Marcel-Jan Krijgsman

Quickly start of the Nifi crash course

As I said last in my last blogpost, I have followed the Apache NiFi crash course that Hortonworks provides. Now the tutorial describes several different scenarios and options and you have to read through that to find which you want. And you don’t have time for that. You’re probably doing this in your spare time and you have a whole Netflix backlog.

So in this guide we cut right to the chase. It took me about 10 hours to follow Tutorial 0, 1, 2 and 3. But perhaps this guide can make you do it in about 4 hours.

1. Preparing the VM

First download the Hortonworks Sandbox. There’s a VirtualBox (used in this example), VMWare and Docker image that come preinstalled with many products, but NiFi isn’t installed just yet (this guide is based on the HDP 2.6 sandbox).

(more…)

By Marcel-Jan Krijgsman, ago

The new product anxiety cycle

Am I the only one who has this? Let me know.

Phase 1: Discovery of New Product

Suddenly everybody talks about New Product. It’s said it changes everything. Articles about New Product appear on Hacker News for weeks. Then colleagues on LinkedIn even mention New Product (Warning! People you know, know New Product!). (Or they’re just linking to articles about New Product, so they look cool. Either way: they must know New Product!)

(more…)

By Marcel-Jan Krijgsman, ago

My first experiences with Apache NiFi

There are a lot of data-related Apache products out there and it’s hard to keep up with all of them. There are several products to stream or flow data (what’s the difference?). Like Kafka, Storm, Flink and NiFi. Yes, all products have documentation, but for an outsider their description sounds like “enterprise scalable streaming solutions”. What does that tell you?

I followed a Crash Course on Apache Nifi at the DataWorks Summit in München last month and was quite impressed. At heart I’m a command line kind of guy, but this graphical interface is really slick and it’s amazing what you can do to find out where your data goes to with NiFi. I decided to organize a workshop for my colleagues at Open Circle Solutions. (more…)

By Marcel-Jan Krijgsman, ago

How to learn Big Data

“How do you got in Big Data?”, is a question that people asked me a couple of times now. So let me give that answer in a blogpost as well.

I’ve used eight sources of Big Data related knowledge and skills:

  • Massive Open Online Courses (MOOCs)
  • Books
  • Meetups and summits
  • Podcasts
  • Videos
  • Online documentation
  • Hands-on experience
  • Learning sites/”universities” of vendors

(more…)

By Marcel-Jan Krijgsman, ago

Dataworks Summit München 2017 – day one

Just left the beergarten party at Dataworks Summit 2017 in München. Okay, let’s see how well I blog after three of these large beers. Luckily I took notes before. Tell me when I start to become incoherent.

So actually for me the summit started yesterday at the Partner day, but today the breakout sessions started. I’ve attended a cool hands-on session on Apache Ranger and Atlas yesterday. These are the new security and governance tools for Hadoop. And I think we can say that the days that you could say Hadoop is largely insecure are about to be over.

The first session today I attended was the keynote of course. I was a little bit late, but it turned out that the second row seats were largely available, so I had a good view of the speakers. A couple of them tried to convince us that (big) data was going to change everything. I already knew that. (more…)

By Marcel-Jan Krijgsman, ago