So far I’ve found geolocations in XML metadata that my actioncam stores on disk as seperate .XML files and I’ve found them in JPG files. When I showed the cool maps I made to my father, he asked if I could create maps from his holiday videos. So that he can show cool maps in his video compilations.
Where do locations get stored in video files?
My father has a Sony PJ650VE video camera that makes videos in AVCHD format. Even the camera itself can show you a map of a video location. So I knew it should store geolocations somewhere. But looking on disk I saw no handy metadata files for me to read. So where did the locations go?
I learned that video formats like MP4, Quicktime (.mov) and AVCHD have EXIF metadata stored in them, just like JPG files. Luckily I had all the videos my father had made of our trip to the east coast of the USA in 2013. So I had lots of examples of AVCHD files to work with.
When I was finished last week with creating my video location map in Python, I thought “shame I can’t plot photo locations”. That’s because my Fuji X-T30 camera doesn’t store GPS info. When I bought the camera I assumed every modern camera had GPS tagging, so I didn’t even checked that feature. Too bad. But I also made some photo’s during my vacations with my humble iPhone 8. And it does have GPS tags. So let’s plot some photo locations.
Huh? I already got a working map now, didn’t I? Well creating a map with matplotlib is a bit of a hassle. You’ve got to download a base map from Openstreetmap.org. And if it’s too big, like the map of all my rides in the Vercors and Drôme (France) last year, you might not get it.
Folium
A quick look at blogs about Folium tell me you don’t need to download a map. It even does zoomable maps. Okay. Wasn’t exactly looking for that. But sounds great.
And I do like the markers you can create. Even with popup texts. Yes please!
So I already have my dataframe with geo data. Don’t need to change that. But I’m going to change all the plotting stuff.
For Matplotlib we needed to define a boundary box. For Folium we only have to have the center point of the map. And you don’t need to load map images or anything here. So that is very nice.
# Find center of folium map
latitude_mean = geodf['latitude'].mean()
longitude_mean = geodf['longitude'].mean()
And for each point in my Pandas dataframe I want a marker. Folium markers allow you to add popup text, which you can make nice with HTML tags. I decided I wanted to have my filename and creation date in here.
for index, georow in geodf.iterrows():
folium.Marker([georow['latitude'], georow['longitude']], popup=f"filename: {georow['xmlfilename']}</br>creationdate: {georow['creationdate']}").add_to(my_map)
Showing the map
But why didn’t PyCharm show my map? Well, it turns out that Folium is more notebook (Jupyter) oriented. Not to worry. You can save the html:
my_map.save('videogps_folium.html')
And when you open this file, there you have it: a wonderful, zoomable map with markers for all my video locations.
And it turns out that now the map is zoomable, this is actually very useful. Remember last blogpost that I would probably create my maps per day? Not anymore. I will just zoom in to that particular ride now.
As mentioned, you can find my new version of this code in my Github repo:
Two years ago I bought a Sony FDR-X3000 actioncam to record video on my bike rides. And I’m really happy about it. It’s just great reliving my rides in 4K, going downhill for kilometers from some col I climbed. I also make compilation videos for fellow cyclists. Like these:
Recently I got the request to add specific data quality metadata with csv datasets that my client delivers to customers. It was very simple. Just counts, min, max and -in case of integers – sums per attribute. Not a difficult task. After a short talk with architects we decided to build this with Python and pandas. Because my efforts were required in another project at that time, my fellow DIKW entrepeneur Wyas build most of it. It worked out well. It ran through a couple of GBs in minutes.
I never imagined myself as a maintainer of a data engineering related open source thing. Yet. But when I was working on our data engineering course, I needed some kind of data lake software. At first I used the Cloudera sandbox, but some of my colleagues tried it and they complained it took way too much time to start and way to many resources of their laptop. It would be a good bet that our students would get that problem too.
Long story short: I found that Big Data Europe already had a simple Dockerized Hadoop. They actually did all the hard work. But I wanted it to have Hive and Spark too. I went playing with docker-compose yml files and learned a lot from that BTW. And after some initial frustrations it finally worked. Continue reading →
Five years ago I made the switch from Oracle database administration to data engineering. It has been quite a ride. I made a video about this to celebrate.
Last week my OAK-D Lite from Luxonis arrived. I can imagine you’ve never heard from it. Basically it is a camera that can do all kinds of AI tasks on the device itself. I got mine via Kickstarter. And where I say camera, I actually mean it has multiple cameras. That’s how it can see depth for example.
It can do much more. Load an algorithm, point it at the street next to your house and it starts detecting cars, cyclists and pedestrians. Load the human posture algorithm and it starts showing your posture. Or gestures, sign language, face recognition or COVID-19 mask detection.
So at the end of 2021 I found myself in the waiting room of an emergency dentist. An infection above my front teeth became unbearable. Fortunately antibiotics makes my live much better now. Let that event not colour my view on 2021.
For me 2021 was a great year, despite the pandemic, lockdowns and those damned curfews. Luckily 2021 eventually also had vaccinations.
And what a difference a year makes. Last year I was frantically working on the last modules of the Certified Data Engineering Professional course. This year I have time for whimsical things like watching a movie (on tv) or playing a computer game.
Work
I’ve learned so much by writing and teaching modules for this course. You want to learn stuff really, really fast? Teaching it is probably the best way to go. Also creating a course you can repeat is a great investment in the future. For every day I teach I have to spend time to prepare of course, but the greatest investment is happily behind me.
Me (right) and my colleague Jeroen Odijk during the first Certified Data Engineering Professional course. That one was held online. The second one fortunately was on site.
I also managed to get an assignment at a Dutch financial company as a data engineer working with Hadoop, NiFi and other stuff. This is the first kind of data engineering assignment as I envisioned it back when I started changing from Oracle database expert to data engineering. The latest development is that I’m getting more and more into the practical aspects of data management.
Also got my Azure Fundamentals certification, but that will be just the start of my Azure certifications.
Github
I never saw myself as a contributor of open source software. I wanted it to be, but I thought most of these are written by people who fluently write Java or Python. But then I created a docker-compose for a Hadoop environment for our data engineering course. And I decided to put it on GitHub. And to my surprise this repo has 38 stars now. Turns out there are other people who would like to run Hadoop with Spark and Hive on their own laptop.
So now I’m suddenly a maintainer of a repo. Halfway through the year Docker decided to harden their software (which is good). And whoops: suddenly some of the containers couldn’t communicate propery with each other. It took me a while before I found out what happened. What changed was that Docker Desktop no longer exposes the daemon without TLS. When you do expose it, everything works again. I need to find a better solution for that BTW.
Vacations
In August I went cycling in France in the Vercors and Drome and that was very nice. This area of France has some beautiful rides, hewn out of rock, like the Col de la Machine. The climbs are usually less steep than the onces I’ve encountered in other years in the Alpes. Except the Col de Noyer BTW.
Then in October I decided to go to Paris for a week. I’ve been in Paris before, about 30 years ago. On sort of a 24 hour race through the city to see all the sights. As you do when you’re a student. I remember running through the Louvre with my brother just to see the Mona Lisa and then run out to see the next thing (didn’t matter, because the ticket was free).
This time I took my time to see all the sights. I went in all the famous museums: Musee d’ Orsay, the Louvre, Centre Pompidou and really took time to soak it all in. I went through the Louvre bottom to top. Amazing. I didn’t know the Venus de Milo is also there. The Nike of Samothrace was also stunning. A different museum I visited was the Musee des Arts et Metiers, which has a particle collider from the 1930s and a steampunk diving suit from the 1880s.
A 19th century diving suit. Also seen in Bioshock games.
The modern art in the Centre Pompidou is just fantastic. I’m not exactly a modern art connaisseur. I didn’t expect the joy of seeing this collection. I learned to love artists I never knew, like Frantisek Kupka and Suzanne Valadon. (Surprised there are hardly any books on Kupka, except in French).
A painting by Frantisek Kupka. Sorry for the fisheye lens photo. I was tired of switching lenses.
I had the most fun in years. The food was great. And I got to see the Arc de Triomphe, Wrapped by Cristo (only for 3 weeks). It was a stunning sight.
Arc de Triomphe, Wrapped
If I have any resolutions for 2022, it’s to do another city trip like this. Preferably by train. Maybe Barcelona.
Cycling
I set myself a goal to ride 10,000 kms this year. I knew I could do it. In 2020 I reached 8754 kms. And already halfway in October I managed to pass 10,000 kms. And just on the last day of the year I reached the stretch goal of 12,000 km.
View from my bike somewhere north of Nieuwkoop.
BTW I have no plans for even larger goals in 2022. There’s no need to overdo it. Cycling takes time. Riding 12,000 kms took 436 hours. I need to leave time to do other things.
But I do keep looking out for new rides. This year my favorite discoveries were the Kromme Mijdrecht, a river south of Amsterdam. You can follow it along a road next to the river in both ways.
I also found my ride along the coast between Noordwijk and Katwijk is wonderful. This is a ride with a view on the North Sea for several kms. I need to ride about 110 kms in total to experience it, but in summer, when the days are long this is quite achievable. Also a ride I did with my brother in Noord-Brabant was a good one.
This year I rode the Gran Fondo Rosa and the Vael Ouwe. Both beautiful but long rides (although the Gran Fondo Rosa was very, very wet), both about 160 kms long. Which was a new record for me for kms on one day. Actually the Gran Fondo Rosa ended up being 170 kms because I couldn’t find the car park.
(Oh and almost forgot: in Februari it froze long enough that I was able to go ice skating. Which was the first time in many years. Even went round on one of the lakes near Reeuwijk. Great experience.)
Astronomy and space
This is my hobby that gets less and less time. Observing already suffered for several years. It’s especially hard to combine with the amount of cycling I did. After several hours of training your natural tendency is to go to bed early. And not stay up late to watch the skies.
But I was happy to observe the noctilucent clouds in June. And the passage of ISS and the new Nauka module (and its 3rd rocket stage).
I used to do a lot more writing about the solar system (in Dutch), but the work on the course and the hours of cycling pushed that to the side. I did enjoy the landing of the Perseverance rover and the recent launch of the James Webb Space Telescope. Not to mention the bellyflops of the several Starship tests. Those were amazing. We’ll have to wait a couple of months for the first images of JWST, but that’s going to be game changing.
Writing
I’ve not done as much writing as I would like to, but my blogpost about critical thinking is one that was kind of influential in my personal life. It is about how I got advise from a homeopath to stop taking one of my prescribed medicine with bad results. I never wrote about something so personal, but in the end it worked out well.
Stage acting
This is the only thing that never got of the ground after the earlier lockdowns. I got together with my stage acting group only once in the pub. We made plans and then the new peak of COVID-19 happened.
Cooking
I never order meals at home. I just keep cooking myself. And the freezer is usually well stocked with extra portions for other days. One of my favorite new recipes was bibimbap, a Korean dish.
Okonomiyaki with “cycling” beer
Recently I invited two friends of mine who I hadn’t seen in a while, for a meal at my house. But again COVID-19 infections peaked, so we saw another lockdown. And my friends understandably were rather safe than sorry. So I changed the plan. They live nearby, so I decided to cook Thai curry (there’s a recipe I’ve grown to like) and delivered it to their home with rice, beer and krupuk (sort of Indonesian deep fried crackers you get at a lot of Indonesian and Chinese meals in the Netherlands). We ate it both at our home and then discussed via Zoom. Despite the limitations these Zoom calls gives us, it all it turned out to be a very fun evening. I hope to do this more often.
2022
I don’t do new years resolutions. I have some ideas. They might pan out or not. I kept my Strava year goal on 10,000 km cycling. That is more than enough to stay healthy. Azure certifications will be a thing and I’m thinking of short new courses for DIKW. We’ll see.
Best of whishes for 2022. Let’s make it a good one.
Oh, and of course you will be able to buy the NFT of this blogpost as soon as I know how to do that.
