Let’s face it: my purchase of the Pimoroni Inky Impression 5.7 inch display was a solution looking for a problem. I saw a video about it and I was sold on the idea of having an e-ink display on one of my Raspberry Pi’s.
The Pimoroni Inky Impression on a Raspberry Pi 3B
While having a 7-colour e-ink display is cool and all, I had to come up with a good plan to utilize one. So it wouldn’t end up in a drawer after a short experiment.
You can use it to display images, but it is 7-colour. So you have to “dither” full colour images to have it display well. Actually comic book style images are displayed much better than the average dithered photo. The resolution is quite low (600×448) and the refresh rate is quite slow (10-20 seconds). But for some applications this is just fine.
Having markers of videos and photos taken during my bike ride is cool and all, but how about having a track of the bike ride itself? All my bike rides are registered on Strava, the cycling and running app. Strava has an API for developers, but it requires connecting via OAuth 2.0 and knowledge of the API. I decided to go an easier route: because I’m Strava Premium member, I can download the GPX track of any ride, including my own.
These .gpx track files are of the same XML structure as we saw embedded in video files in my last blogpost. I can just open the file and use almost the same Python code to read the locations.
So far I’ve found geolocations in XML metadata that my actioncam stores on disk as seperate .XML files and I’ve found them in JPG files. When I showed the cool maps I made to my father, he asked if I could create maps from his holiday videos. So that he can show cool maps in his video compilations.
Where do locations get stored in video files?
My father has a Sony PJ650VE video camera that makes videos in AVCHD format. Even the camera itself can show you a map of a video location. So I knew it should store geolocations somewhere. But looking on disk I saw no handy metadata files for me to read. So where did the locations go?
I learned that video formats like MP4, Quicktime (.mov) and AVCHD have EXIF metadata stored in them, just like JPG files. Luckily I had all the videos my father had made of our trip to the east coast of the USA in 2013. So I had lots of examples of AVCHD files to work with.
When I was finished last week with creating my video location map in Python, I thought “shame I can’t plot photo locations”. That’s because my Fuji X-T30 camera doesn’t store GPS info. When I bought the camera I assumed every modern camera had GPS tagging, so I didn’t even checked that feature. Too bad. But I also made some photo’s during my vacations with my humble iPhone 8. And it does have GPS tags. So let’s plot some photo locations.
Yesterday I shared how I plotted locations of videos shot with my Sony FDR-X3000 camera on a map. I was already pretty happy. Then I got a tip from Twitter user Bob Haffner (@bobhaffner): why not use Folium to create my maps? Huh? I already got a working map now, Read more
Two years ago I bought a Sony FDR-X3000 actioncam to record video on my bike rides. And I’m really happy about it. It’s just great reliving my rides in 4K, going downhill for kilometers from some col I climbed. I also make compilation videos for fellow cyclists. Like these:
Recently I got the request to add specific data quality metadata with csv datasets that my client delivers to customers. It was very simple. Just counts, min, max and -in case of integers – sums per attribute. Not a difficult task. After a short talk with architects we decided to build this with Python and pandas. Because my efforts were required in another project at that time, my fellow DIKW entrepeneur Wyas build most of it. It worked out well. It ran through a couple of GBs in minutes.
I’m keeping tech dossiers in Evernote on open source products I want to keep track of. And I decided to put them on my blog. My previous ones were on Kubernetes and Elasticsearch. This one is on the Python data management library pandas. A short description – in English Read more
Data engineering isn’t always creating serverless APIs and ingressing terrabyte a minute streams with do-hickeys on Kubernetes. Sometimes people just want their Excel sheet in the data lake. Is that big data? Not even close. It’s very small. But for some people it’s a first step in a data driven world.
But does Hadoop read Excel? Not to my knowledge. But NiFi, that wonderful open source data flow software has an Excel processor. It can even help you to work the data a little. But some Excel sheets simply need too much reworking. And that’s simply too big a job for NiFi. I’ve used Python and the pandas library to create a csv file that Hadoop can handle.
