Marcel-Jan Krijgsman

A screenshot of Obsidian, showing the graph mode

Migrating from Evernote to Obsidian

Fourteen years ago I discovered Evernote, the note taking app. It was one of the first apps on my first iPad. Back then it was the very cool note taking app everyone was using.

In 2016 they started asking to pay a subscription. As someone who leaned more and more on this app that wasn’t unreasonable. I’m okay if that supports the development teams making the app better. In 2022 Evernote was bought by Bending Spoons, a company sometimes called the place where apps come to die. Development continued, but the keyword was “AI”. Sometimes useful, but I like to write my own notes, thank you.

Evernote became central for a lot of things in my life: all my notes for my astronomy and space podcast were stored in Evernote notebooks. My todo lists, links to interesting articles on data engineering and AI and upcoming purchases.

(more…)
By Marcel-Jan Krijgsman, ago

Data modelling – the detective work you need to do to understand your source data.

It is almost 9 years ago that I became a data engineer and I can honestly say my job is vastly different than what I thought it would be at the start. I still love what I do, but the reality is that data-product building is far more than just handling big data systems. Nine years ago though, I thought I would be kind of a Big Data DBA. I hadn’t given a lot of thought about the data modelling involved.

Gradually I learned that building data products takes quite a bit of forethought. And when you build data products you need to know your source data quite well.

(more…)
By Marcel-Jan Krijgsman, ago

Data engineering in the European cloud – Part 2: Scaleway

This is Part 2 in a series where I try to create a data engineering environment in the European cloud. In Part 1 I described my plan for creating a data lakehouse in the European cloud. Now it’s time to get our hands dirty. We’re going to do this in the Scaleway cloud.

The architecture

To get this data lakehouse running we will create a Kubernetes cluster and object storage for our data storage. In Kubernetes we can run containerised applications that will run our data lakehouse. I’ve consulted ChatGPT for this architecture. It had a better and more modern solution than I originally had in mind.

We’re going to use the Apache Iceberg open table format. This will allow us to create database like tables based on Parquet formatted files. Nessie will be the Iceberg data catalog (Hive Metastore was another option). It allows our data solutions to find the Iceberg tables and underlying Parquet files.

Trino will be the query engine. That will be the fastest way to get our first queries going.

(more…)
By Marcel-Jan Krijgsman, ago

Data engineering in the European cloud – Part 1: the plan

We all know how dependent we have become in Europe on US cloud providers. We know about the risks of this in the current political climate. And yet we keep using more and more US cloud services. Read Bert Hubert’s writings about the European cloud situation.

And to be honest, when customers ask for advice on starting a new data engineering ecosystem, Azure Fabric and Databricks are on the top of my list.

But while it might be hard to switch from Office 365 to open source solutions (especially moving all your users to these unknown platforms), in the data engineering landscape there are so many widely adopted open source solutions. Solutions that end users rarely need to deal with directly. Couldn’t we run these products somewhere else? So I went on an investigation.

(more…)
By Marcel-Jan Krijgsman, ago
Detail of the Inky Impression e-ink display with a star map depicting on it.

inkystarmap – an always up to date starmap on the wall

Last year I did a talk at Pycon Ireland 2024 about e-ink displays, in which I gave several examples of ways you can program e-ink displays on a Raspberry Pi with Python. For this talk I developed one extra application: displaying a star map on an e-ink display. But the e-ink displays I had available back then were a bit small for this purpose.

Enter Pimoroni’s new Inky Impression 13.3 inch e-ink display. As soon as it arrived, I worked on the star map again. It turned out that 9 months later, some things had changed. But after 2 evenings experimenting, I got a new working version. Now utilising the gradient display of the Python package starplot. On the new 13.3 inch display with brighter colours, it worked perfectly.

(more…)
By Marcel-Jan Krijgsman, ago

Masterclass Machine Learning in Cycling

Last Tuesday Paul van Herpt and I traveled to Lille for a special Machine Learning and Cycling Masterclass. As data partner of Soudal Quick-Step Pro Cycling Team, these are exactly the applications that touch where we as Transfer Solutions can make the difference. Hence Paul and I followed this special course from the IDLab (UGent – UAntwerpen – imec).

The author (left) and Paul van Herpt at the Masterclass Machine Learning in Lille.

Machine learning is already used a lot in sports. In soccer, for example, a huge amount of statistics is at hand: who has how long ball contact, who usually shoots to whom, makes the most runs, who is the most dangerous? That kind of data is already very easily traceable. And in tennis, it is easy to track the ball, calculate speed, etc..

(more…)
By Marcel-Jan Krijgsman, ago

How to use data to find the best spot for a sponsor event

As you might know I’m currently doing sponsor events for Tour for Life, to collect funds for the Daniel den Hoed Foundation, for cancer research.

Aniel, me and Transfer Solutions CTO Albert Leenders at a sponsor event last Saturday in Ede.

Aniel and me have been doing this for the 3rd year now. And we noticed quite big differences in proceeds per location. You’d think large crowds (like on Dam Square in Amsterdam) would guarantee large amounts of donations. Not so. A more humble place like my home town Gouda outdid them by a factor of 9 in the same year!

(more…)
By Marcel-Jan Krijgsman, ago

Visiting PyGrunn 2025

Conferences are a great way to learn diverse topics in your field. That’s why I like to go to events like Pycons and last Friday, PyGrunn. PyGrunn is a Python event in Groningen, the Netherlands. I submitted two talks for the event myself. One of them was selected.

Here is a recap of the talks I attended and the stuff I learned, so you maybe get inspired to attend Python conferences and even speak at these events.

Keeping your Python in check – Mark Boer

Python was originally developed to make coding more accessible. Where at other programming languages you had to tell what type of data type your variables are, Python deduced this automatically. Good for beginning coders, maybe not so good for advanced data solutions.

Mark Boer has experience in strong typing in his data science solutions. He shared how you can ensure typing in different ways: in data classes, using Pydantic and named tuples. The talk assumed that the attendees already had experience with typing. I had not, so it was a lot to take in. But if I can review the video in a few weeks, I hope to catch on.

(more…)
By Marcel-Jan Krijgsman, ago