data lineage

Don’t do data management just because you have to

Lately more and more organizations are doing data management. Suddenly there are data owners, data stewards and metadata repositories (in whatever form) everywhere. We all seem to do this mainly because we have to. Because of the GDPR or the California Consumer Privacy Act (CCPA). Or because other institutions demand we can explain where our data comes from.

But in my oppinion there is one important reason that mostly is overlooked. One that nevertheless has an important positive impact on business results, but also doesn’t seem to end up in the KPI’s. And that is how much time it takes to find the right data when building data products. (more…)

By Marcel-Jan Krijgsman, ago

My first experiences with Apache NiFi

There are a lot of data-related Apache products out there and it’s hard to keep up with all of them. There are several products to stream or flow data (what’s the difference?). Like Kafka, Storm, Flink and NiFi. Yes, all products have documentation, but for an outsider their description sounds like “enterprise scalable streaming solutions”. What does that tell you?

I followed a Crash Course on Apache Nifi at the DataWorks Summit in München last month and was quite impressed. At heart I’m a command line kind of guy, but this graphical interface is really slick and it’s amazing what you can do to find out where your data goes to with NiFi. I decided to organize a workshop for my colleagues at Open Circle Solutions. (more…)

By Marcel-Jan Krijgsman, ago