Last Tuesday Paul van Herpt and I traveled to Lille for a special Machine Learning and Cycling Masterclass. As data partner of Soudal Quick-Step Pro Cycling Team, these are exactly the applications that touch where we as Transfer Solutions can make the difference. Hence Paul and I followed this special course from the IDLab (UGent – UAntwerpen – imec).
The author (left) and Paul van Herpt at the Masterclass Machine Learning in Lille.
Machine learning is already used a lot in sports. In soccer, for example, a huge amount of statistics is at hand: who has how long ball contact, who usually shoots to whom, makes the most runs, who is the most dangerous? That kind of data is already very easily traceable. And in tennis, it is easy to track the ball, calculate speed, etc..
Conferences are a great way to learn diverse topics in your field. That’s why I like to go to events like Pycons and last Friday, PyGrunn. PyGrunn is a Python event in Groningen, the Netherlands. I submitted two talks for the event myself. One of them was selected.
Here is a recap of the talks I attended and the stuff I learned, so you maybe get inspired to attend Python conferences and even speak at these events.
Keeping your Python in check – Mark Boer
Python was originally developed to make coding more accessible. Where at other programming languages you had to tell what type of data type your variables are, Python deduced this automatically. Good for beginning coders, maybe not so good for advanced data solutions.
Mark Boer has experience in strong typing in his data science solutions. He shared how you can ensure typing in different ways: in data classes, using Pydantic and named tuples. The talk assumed that the attendees already had experience with typing. I had not, so it was a lot to take in. But if I can review the video in a few weeks, I hope to catch on.
Originally I wanted to write a blogpost about what data engineers are going to do with AI writing their code. But before I can write that, I need to share my experiences so far. Because from this you’ll get an idea where they work and where they lack.
This is not meant as a treatise of AI coding assistants and agentic AI tools. But here are some of the tools I’ve tried:
I’ve worked with VSCode and Copilot now for at least a year.
I regularly use ChatGPT and Phind.com for advice on programming tasks.
I’ve used VSCode with Cline / Roo Code extensions and LLM models.
And I’ve used Claude Code (which is not free, but there seems to be a trial amount of free tokens). Claude Code works from the command line.
The agentic AI solutions are interesting. They are quite capable to create whole Python projects based on your requests. But it doesn’t mean these projects will work right out of the box. Usually there needs to be some tweaking and restarting and checking results.
When you got a dataset to explore, there are several ways to do that in PySpark. You can do a describe or a summary. But if you want something a little more advanced, and if you want to get a bit of a view of what is in there, you might want to go data profiling.
Older documentation might point you to Pandas profiling, but this functionality is now part of the Python package ydata-profiling (which is imported as ydata_profiling).
I’ve been following this blog on starting with ydata-profiling:
Getting ydata-profiling to work is not exactly a walk in the park. You’d think you can just feed it your messy dataset and it will show you what the data is like. But I encountered some problems:
I got errors about missing Python packages in some situations.
ydata doesn’t seem to like dataframes with only string columns.
Currently I’m helping colleagues to read open data in Azure Data Fabric. Here are some of my experiences with it. I don’t want to do an extensive description of what Data Fabric is. In short, if you have an organisational Azure account, you can enable Data Fabric. You can then Read more
I think it was last year when I announced that I wanted to go back to conferences again. Preferably as a speaker. But what conference is the best for data engineers? I couldn’t quite figure it out. Then the call for papers for PyCon Ireland 2024 came by on my Read more
How it started For several years I kept track of my weight and fat with a Soehnle Body Balance, which I bought in 2018. That worked quite well until I saw more and more these weird deviations. Take a look at the red line (fat percentage) in the graph below: Read more
These Python and Raspberry Pi projects. They are fun aren’t they? And often they look deceptively simple. But you don’t see all the projects that failed and usually not where they struggled. This project got stuck (and almost failed) at:
Not being able to scrape dynamic website content.
When I found out how to do that, I couldn’t run my working Python code on the Raspberry Pi.
That turned out to be because the scraping packages use a chromium browser, but not for the ARM processor that the Raspberry Pi has.
And to top it all off, the Python package for the Inky Impression e-ink display had some kind of problem running numpy.
I have a hobby project I’m working on. It’s an astronomy news feed reader. Long story short: I currently gather links to interesting articles about astronomy by hand. And I want to automate this, so that I have more time to actually read the news. What I want is that Read more
This is part 3 of a series of blogposts on how I created a Strava dashboard on a Inky Impression e-ink display with a Raspberry Pi.
OAuth2
This was the part that I expected to be the hard part: getting my data from Strava. Or, to be more precise: getting the connection right so the Strava API would allow me to get that data. Because it requires authentication via the OAuth2 protocol and I’ve tried a similar thing a few years back with a Google API and I just didn’t get it. But now I do.
