Adding the track of my bike ride on a Folium map

Having markers of videos and photos taken during my bike ride is cool and all, but how about having a track of the bike ride itself? All my bike rides are registered on Strava, the cycling and running app. Strava has an API for developers, but it requires connecting via OAuth 2.0 and knowledge of the API. I decided to go an easier route: because I’m Strava Premium member, I can download the GPX track of any ride, including my own.

These .gpx track files are of the same XML structure as we saw embedded in video files in my last blogpost. I can just open the file and use almost the same Python code to read the locations.

Continue reading

Posted in Howto, Python | Tagged , , , , , , , | Leave a comment

Photo locations, marker icons and displaying photos on my map

When I was finished last week with creating my video location map in Python, I thought “shame I can’t plot photo locations”. That’s because my Fuji X-T30 camera doesn’t store GPS info. When I bought the camera I assumed every modern camera had GPS tagging, so I didn’t even checked that feature. Too bad. But I also made some photo’s during my vacations with my humble iPhone 8. And it does have GPS tags. So let’s plot some photo locations.

Continue reading

Posted in Data engineering, Howto, Python | Tagged , , , , , , , | Leave a comment

Making my video location map even better with Folium

Yesterday I shared how I plotted locations of videos shot with my Sony FDR-X3000 camera on a map. I was already pretty happy. Then I got a tip from Twitter user Bob Haffner (@bobhaffner): why not use Folium to create my maps?

Huh? I already got a working map now, didn’t I? Well creating a map with matplotlib is a bit of a hassle. You’ve got to download a base map from Openstreetmap.org. And if it’s too big, like the map of all my rides in the Vercors and Drôme (France) last year, you might not get it.

 

Folium

A quick look at blogs about Folium tell me you don’t need to download a map. It even does zoomable maps. Okay. Wasn’t exactly looking for that. But sounds great.

And I do like the markers you can create. Even with popup texts. Yes please!

 

Changing the code

Of course, the new version of this code is on Github in my repo: https://github.com/Marcel-Jan/media_gpsplot.

First of all we need to import Folium:

import folium

So I already have my dataframe with geo data. Don’t need to change that. But I’m going to change all the plotting stuff.

For Matplotlib we needed to define a boundary box. For Folium we only have to have the center point of the map. And you don’t need to load map images or anything here. So that is very nice.

# Find center of folium map
latitude_mean = geodf['latitude'].mean()
longitude_mean = geodf['longitude'].mean()

Now I define a map:

my_map = folium.Map(location=[latitude_mean, longitude_mean], zoom_start=12)

And for each point in my Pandas dataframe I want a marker. Folium markers allow you to add popup text, which you can make nice with HTML tags. I decided I wanted to have my filename and creation date in here.

for index, georow in geodf.iterrows():
    folium.Marker([georow['latitude'], georow['longitude']], popup=f"filename: {georow['xmlfilename']}</br>creationdate: {georow['creationdate']}").add_to(my_map)

 

Showing the map

But why didn’t PyCharm show my map? Well, it turns out that Folium is more notebook (Jupyter) oriented. Not to worry. You can save the html:

my_map.save('videogps_folium.html')

And when you open this file, there you have it: a wonderful, zoomable map with markers for all my video locations.

And it turns out that now the map is zoomable, this is actually very useful. Remember last blogpost that I would probably create my maps per day? Not anymore. I will just zoom in to that particular ride now.

 

As mentioned, you can find my new version of this code in my Github repo:

https://github.com/Marcel-Jan/media_gpsplot

You can also find an example output file:

https://github.com/Marcel-Jan/media_gpsplot/blob/main/videogps_folium.html

 

Other blogposts I wrote about geo data in Python:

Adding the track of my bike ride in Folium (Antpaths and Polylines)

Digging into video files for geolocations (Exif data in video files, running OS commands from Python, processing XML)

Photo location markers and displaying photos on a map (Accessing exif data in JPGs with Python, projecting photos on a Folium map).

 

Posted in Data engineering, Howto, Python | Tagged , , , | Leave a comment

Plotting video locations from my Sony camera in Python

Two years ago I bought a Sony FDR-X3000 actioncam to record video on my bike rides. And I’m really happy about it. It’s just great reliving my rides in 4K, going downhill for kilometers from some col I climbed. I also make compilation videos for fellow cyclists. Like these:

Continue reading

Posted in Data engineering, Howto, Python | Tagged , , , , , , | Leave a comment

Actually, you do security to stop these guys

Today I am studying for the Microsoft Certified: Azure Data Engineer Associate exam. And currently I’m going through some dry stuff on database security. Today I also read this article on Krebs on Security about the LAPSUS$ hackers who stole lots of source code. Private chat messages of this hacker collective got out in the open recently and it is telling how they look at their targets. Continue reading

Posted in Active Learning | Tagged , , , , | Leave a comment

Handling far future dates in pandas

Recently I got the request to add specific data quality metadata with csv datasets that my client delivers to customers. It was very simple. Just counts, min, max and -in case of integers – sums per attribute. Not a difficult task. After a short talk with architects we decided to build this with Python and pandas. Because my efforts were required in another project at that time, my fellow DIKW entrepeneur Wyas build most of it. It worked out well. It ran through a couple of GBs in minutes.

Continue reading

Posted in Python | Tagged , , , , , | Leave a comment

My Github repo got 50 stars

I never imagined myself as a maintainer of a data engineering related open source thing. Yet. But when I was working on our data engineering course, I needed some kind of data lake software. At first I used the Cloudera sandbox, but some of my colleagues tried it and they complained it took way too much time to start and way to many resources of their laptop. It would be a good bet that our students would get that problem too.

Long story short: I found that Big Data Europe already had a simple Dockerized Hadoop. They actually did all the hard work. But I wanted it to have Hive and Spark too. I went playing with docker-compose yml files and learned a lot from that BTW. And after some initial frustrations it finally worked. Continue reading

Posted in Apache Products for Outsiders, Data engineering | Tagged , , , , | Leave a comment

Five years of data engineering

Five years ago I made the switch from Oracle database administration to data engineering. It has been quite a ride. I made a video about this to celebrate.

Posted in Active Learning, Data engineering | Tagged | Leave a comment

First experiments with the OAK-D Lite

Last week my OAK-D Lite from Luxonis arrived. I can imagine you’ve never heard from it. Basically it is a camera that can do all kinds of AI tasks on the device itself. I got mine via Kickstarter. And where I say camera, I actually mean it has multiple cameras. That’s how it can see depth for example.

It can do much more. Load an algorithm, point it at the street next to your house and it starts detecting cars, cyclists and pedestrians. Load the human posture algorithm and it starts showing your posture. Or gestures, sign language, face recognition or COVID-19 mask detection.

Continue reading

Posted in Active Learning, Weird experiments | Tagged , , , , | Leave a comment