Today I am studying for the Microsoft Certified: Azure Data Engineer Associate exam. And currently I’m going through some dry stuff on database security. Today I also read this article on Krebs on Security about the LAPSUS$ hackers who stole lots of source code. Private chat messages of this hacker collective got out in the open recently and it is telling how they look at their targets. (more…)
Recently I got the request to add specific data quality metadata with csv datasets that my client delivers to customers. It was very simple. Just counts, min, max and -in case of integers – sums per attribute. Not a difficult task. After a short talk with architects we decided to build this with Python and pandas. Because my efforts were required in another project at that time, my fellow DIKW entrepeneur Wyas build most of it. It worked out well. It ran through a couple of GBs in minutes.
I never imagined myself as a maintainer of a data engineering related open source thing. Yet. But when I was working on our data engineering course, I needed some kind of data lake software. At first I used the Cloudera sandbox, but some of my colleagues tried it and they complained it took way too much time to start and way to many resources of their laptop. It would be a good bet that our students would get that problem too.
Long story short: I found that Big Data Europe already had a simple Dockerized Hadoop. They actually did all the hard work. But I wanted it to have Hive and Spark too. I went playing with docker-compose yml files and learned a lot from that BTW. And after some initial frustrations it finally worked. (more…)
Five years ago I made the switch from Oracle database administration to data engineering. It has been quite a ride. I made a video about this to celebrate.
Last week my OAK-D Lite from Luxonis arrived. I can imagine you’ve never heard from it. Basically it is a camera that can do all kinds of AI tasks on the device itself. I got mine via Kickstarter. And where I say camera, I actually mean it has multiple cameras. That’s how it can see depth for example.
It can do much more. Load an algorithm, point it at the street next to your house and it starts detecting cars, cyclists and pedestrians. Load the human posture algorithm and it starts showing your posture. Or gestures, sign language, face recognition or COVID-19 mask detection.
So at the end of 2021 I found myself in the waiting room of an emergency dentist. An infection above my front teeth became unbearable. Fortunately antibiotics makes my live much better now. Let that event not colour my view on 2021. For me 2021 was a great year, despite Read more
It’s weekend and it’s raining. Time to play some computer games. Bikrash In meatspace I try to avoid crashing with my bike. But in this free game they are rampant. And you can actually win by causing them, by kicking other racers. Also watch out for the road spanning, Read more
I have to say, I’m a fan of long summer nights where I can ride my bike for hours after work in short sleeves. This is not that time. The sun goes under at 16:45 now. So if I do decide I want to ride my bike after work, I Read more
For a few years I’ve been gathering data on my workouts. In Excel. It’s not exactly state of the art data architecture, but it was fine for a while. But data alone doesn’t do much. I wanted some questions answered. Lately I’ve been hearing a lot about Apache Superset. (Well, Read more
Last weekend I was too busy either cycling or celebrating that I rode 10,000 kms this year. Nevertheless, here are this weekend’s links. Rickrolling your high school by hacking the IPTV system This student managed to gain access to his high school’s IPTV system. And he carefully prepared a Read more