Category Archives: Data engineering

What I think data engineering is (revisited)

Posted on May 19, 2021 by Marcel-Jan Krijgsman

Four years now I’ve been working as a data engineer. And when I started writing about how to enter this field (because people sometimes ask me), I found out it’s beter to start writing about what data engineering actually is. … Continue reading →

Posted in Data engineering | Tagged , | Leave a comment

Don’t do data management just because you have to

Posted on February 3, 2021 by Marcel-Jan Krijgsman

Lately more and more organizations are doing data management. Suddenly there are data owners, data stewards and metadata repositories (in whatever form) everywhere. We all seem to do this mainly because we have to. Because of the GDPR or the … Continue reading →

Posted in Data engineering | Tagged , , , , , | Leave a comment

Tech dossier: pandas

Posted on September 26, 2019 by Marcel-Jan Krijgsman

I’m keeping tech dossiers in Evernote on open source products I want to keep track of.  And I decided to put them on my blog. My previous ones were on Kubernetes and Elasticsearch. This one is on the Python data … Continue reading →

Posted in Data engineering, Python, Tech dossier | Tagged , , , , , | Leave a comment

Things I’ve learned about metadata for a data lake

Posted on September 7, 2019 by Marcel-Jan Krijgsman

I’ve been thinking of writing a blogpost about Apache Atlas. For one and a half years I’ve gained a unique experience with this product that I would like to share with the world. But first we need to talk about … Continue reading →

Posted in Data engineering | Tagged , , , , , , , | Leave a comment

Book review: Spark in Action, 2nd edition

Posted on September 1, 2019 by Marcel-Jan Krijgsman

There are lots of books on Spark, but not a lot that aimed at the data engineer. Data engineers use Spark to ingest and transform data, which is different from what data scientists use it for. On the Roaring Elephant … Continue reading →

Posted in Data engineering, Spark | Tagged , , , | 2 Comments