It’s weekend and it’s raining. Time to play some computer games.
Bikrash
In meatspace I try to avoid crashing with my bike. But in this free game they are rampant. And you can actually win by causing them, by kicking other racers. Also watch out for the road spanning, enormous potholes.
This is fun to watch. The Stackrabbit algorithm plays Tetris so fast, at a certain point it arrives in territory where the game doesn’t have programmed colour combinations anymore. And eventually it breaks.
Professionally I’m reading Data Pipelines with Apache Airflow by Bas Harenslak and Julian Rutger de Ruiter. After a bit slow intro, it really helps you along building your first pipelines in Airflow.
Recreationally I’m rereading the Foundation trilogy by Isaac Asimov. Because I’m also watching the series on Apple+. I think it’s at least 25 years ago that I read it the first time and I didn’t remember any of it. Except that it was about Hari Seldon and there was a Second Foundation.
I have to say, I’m a fan of long summer nights where I can ride my bike for hours after work in short sleeves. This is not that time. The sun goes under at 16:45 now. So if I do decide I want to ride my bike after work, I need some good bike lights to see where I am. But hey, at least it’s not as cold as in Yakutia. (Please summer, come back quickly)
You think it’s cold where you live?
Wait till you hear from Yakutia, in Northern Siberia. Living in an area where the temperatures drop to -71°C has its challenges. But on the other hand: you don’t need to buy a freezer.
I never miss an episode of the Weekly Space Hangout. Hosted by Fraser Cain from Universe Today you get to hear the latest news in space and astronomy. It’s informative and funny.
But when I heard they had a guest called the Space Juggler, I could not keep one of my eyebrows from rising. It turns out Dr. Adam Dipert is doing incredibly interesting and diverse research. On the one hand he tries to find out why there was more matter than antimatter in the Universe after the Big Bang. On the other hand he experiments with juggling on zero-G flights. You will find that that research is equally mindblowing.
Grow your own sodium chloride crystals
Wait don’t I already have salt crystals in my cupboard? Not big transparent crystals like these, my friend. This looks like an awesome experiment you can do at home. Maybe with the kids?
For a few years I’ve been gathering data on my workouts. In Excel. It’s not exactly state of the art data architecture, but it was fine for a while. But data alone doesn’t do much. I wanted some questions answered.
Lately I’ve been hearing a lot about Apache Superset. (Well, I’ve been hearing lately about lots of products actually. It’s hard to choose one product to spend a lot of time on.) Apache Superset is open source data visualization software. I decided to give it a try for this particular problem.
The video
Apart from the installation I demo most things in this video:
Starting on your laptop
If there’s anything I’ve learned recently working with Docker Desktop, it’s that it is often very easy to get a working environment of most open source data products in one or more Docker containers. Usually they have an image or docker-compose on their site somewhere to get you started. Same for Superset.
All you need to get started is Git and Docker Desktop (available for Windows, Mac and Linux). I use Git for Windows. To build your Dockerized Superset environment, just follow the instructions on the Superset documentation site:
This will start 6 Docker containers. One of them (named superset_db) runs a PostgreSQL 10 database, which contains the sample data, but also can be used to upload your CSV data. Another, superset_init, will only be used for installation and won’t run anymore after that. That is fine.
You can log in here with username admin, password admin. Now the Superset welcome screen will show. You can have a look at the example charts and dashboards.
Uploading my own dataset
Superset allows you to upload a CSV file. It will do this in the example (PostgreSQL) database. But first you need to edit the settings of the example database to allow uploads. Go to Data, Databases and click on edit:
Go to Advanced, Security and check Allow data upload. All this is explained in above video.
How to get a proper datetime column
I also explain also all the settings you can do to upload the CSV file. But there was a problem. PostgreSQL didn’t recognise the date and time format. Despite the “Infer Datetime Format” setting I had enabled.
An example of my date and time data was this: 2021-10-21 17:52:00
Because PostgreSQL didn’t recognise this as a datetime, Superset didn’t allow more advanced time-related features. For example, when I used a Time-series Bar Chart v2 or Time-series Area Chart v2, and I chose a Time Grain: week or year, it would come up with an “Unexpected error”:
Error: function date_trunc(unknown, text) does not exist LINE 1: SELECT DATE_TRUNC('year', "Datum") AS __timestamp, ^ HINT: No function matches the given name and argument types. You might need to add explicit type casts.
To see the problem, go to your dataset, More dataset options and Edit dataset:
In my case my Datum column was of the type “TEXT”, not “DATETIME”. It doesn’t matter that I chose this to be temporal data. You will run into this Unexpected error.
This is more of a PostgreSQL problem. To solve it, let’s go into the superset_db container and change this datatype.
You need to solve this on the command prompt. In Windows I’ve used Powershell for this.
Log in in the superset_db container with this command:
docker exec -it superset_db bash
Connect to the PostgreSQL database:
psql -h 127.0.0.1 -p 5432 -U superset
Check if you can access your table from here. In my case the table was called Workouts2. Because the name is case sensitive, I had to use double quotes around the name.
select * from "Workouts2";
This is how you change the data type of your date column. In my case it was called “Datum”. Here I create a temporary column called temp_date with the DATETIME data type. I then load the data from the “Datum” column:
ALTER TABLE "Workouts2" ADD COLUMN temp_date TIMESTAMP without time zone NULL;
UPDATE "Workouts2" SET temp_date = "Datum"::TIMESTAMP;
ALTER TABLE "Workouts2" ALTER COLUMN "Datum" TYPE TIMESTAMP without time zone USING temp_date;
ALTER TABLE "Workouts2" DROP COLUMN temp_date;
After this small operation, you will have the same old “Datum” column, but now with the DATETIME data type. And Superset will notice this too:
Now you can really start using the fun time related features in Superset, which I demoed in the video.
My Superset experience
Once I had my datetime column set up, things really took off. Superset became really fun to use. I really was able to gain insights quickly. Superset is indeed very powerful. If possible I certainly will use it in the future.
Furure work
My Superset dashboard looks awesome, but for every update to my Excel sheet, I need to upload it in Superset. That is not something I’m looking forward doing in the future.
In fact, I don’t like to enter my health data in Excel at all. Here is how I would like my data architecture to look like:
I want an app on my iPad or iPhone to enter my weight data. (Now I use Evernote and copy that to Excel). The app uploads this data directly in a central database in my home. Possibly on a Raspberry Pi.
I have a pipeline that retrieves new workout data from Strava and Polar and enters this data in my central database.
On this central database I run Superset, where my dashboard is always up to date.
This will require quite some work and time. I have no idea how to create an app that works on my iPad or iPhone. I hope it can be done in Python, but even creating an app in Python is new ground for me.
If I can create this, I will share it on this blog. But don’t hold your breath just now.
Let’s face it: if we’re dependent on all polical leaders to do their part to reduce greenhouse gasses… well, just looking to my country’s (the Netherlands) leadership, I don’t expect them to even work very hard on it.
Credit: Tang et al.
So let’s capture all that carbon dioxide. That could work, right? Sure, but up to now you needed a lot of energy to pull it off. How do we generate that energy? Please don’t say “fossil fuels”.
But this new method by using the metal gallium can convert carbon dioxide to solid products and oxygen, and you don’t seem to need massive amounts of energy. Having solid carbonaceous products is a plus when you want to store it. And getting your oxygen back is positive too.
Last Wednesday William Shatner flew on the 17th flight of Blue Origin’s New Shepard rocket. And when he came back, he was literally moved to tears after seeing the Earth from 100 km altitude.
It’s moving to see him trying to describe his experience. And to think Jeff Bezos almost ruined it, waving his champagne bottle around. If we must do space tourism, please send more people like William Shatner. And not just because he was captain James T. Kirk in Star Trek.
I was listening to Lex Fridman’s podcast where he interviews Daniel Kahneman. You might have heard about Kahneman: he wrote the influential book “Thinking Fast and Slow”. It is about the two modes of thinking our brain: System 1 (fast, instinctive and emotional) and System 2 (slower, more deliberative, and more logical).
At one point in the interview Fridman and Kahneman discuss happiness. Kahneman tells he gave up on happiness research. And he explains this, hopefully hypothetical, scenario:
Suppose you go on a vacation. But at the end of the vacation you’ll get an amnesic drug and you won’t remember anything. And all your pictures will be destroyed. Would you choose the same vacation?
That is kind of interesting. I look fondly back on my recent vacation in France: the beautiful routes I’ve ridden, the wonderful meals I had, the cols I’ve climbed. And I’m still busy making video compilations of each day. And I’m looking proudly back on Strava on the rides I’ve done. Suppose all that was deleted? What would I change my holiday? That is such an interesting question.
I think I would do another cycling holiday, just because I feel great afterwards. Part of the fun though is reviewing the videos and photos I shot. Would I do an intensive cycling holiday in a less beautiful area, just because I would not remember anything about it anyway?
An asteroid hit Jupiter
This is something almost only amateur astronomers find: impacts of asteroids on Jupiter. Like Jose Luis Pereira from Brasil, who made this find, while imaging the largest planet in our solar system.
(You’d think spacecraft would pick that up first, but then you overestimate how many active spacecraft there are around our planets. Currently there’s only NASA’s Juno mission in orbit around Jupiter. Juno is in a very elongated orbit to stay out of Jupiter’s harmful radiation belt for most of the time. It was that, or make spacecraft with expensive fully radiation hardened electronics. Also Juno does have a camera, but that was more or less added to the spacecraft for public outreach. It can’t view Jupiter any better than amateur astronomers can from the point in it’s orbit it is currently in. It will take until October 16th for Juno to get close again and by that time probably there will be no trace left of the impact.)
A new impact flash has been discovered on Jupiter by Brazilian observer Jose Luis Pereira – the flash occurred at Sep 13th, 22:39:30 UTC. It looks quite a bright one. Follow up images of this region will be useful to see if any dark scar was left following the event. pic.twitter.com/LL9rDGvodm
When I was young I remember about this possible map, drawn by Vikings, which would have Northern America on it. Well, it was analyzed by scientists at Yale University. The map is old, but if you do X-ray spectroscopy on it, it will show that the ink used to depict America has titanium in it. And those types of ink started being used only since the 1920s.
This is Eastern Europe, shot at night from the ISS. In front you see the Soyuz spaceship (used to bring cosmonauts to ISS and back home) and the new Nauka module
I’m back for a wonderful cycling holiday in the Vercors and Drôme regions of France. And this is what it looked like:
But enough of that. Let’s have some weekend links.
One little RNA change: Boom! 50% more potato for you
Scientists found that by changing one methyl group in the structure of RNA of potato plants causes it to yield 50% bigger potatoes. And it’s not just a more watery potato. There were no changes to starch and protein. And it doesn’t stop at potatoes. In rice plants it causes more rice (not bigger). The reason of this seems to be because the roots of the plants grow deeper and the photosynthesis is quite more effective.
So the world rejoyces, right? Bigger potatoes and more rice for everyone. No more world hunger! Probably more tests need to be done. And likely the stigma of genetically modified organisms (GMO) will prevent much change happen here in Europe. But hopefully in other countries it will indeed cause less hunger some day?
The GPT-3 is a language model that can generate very impressive texts. But it has its limits. You can use it to create the text of an exciting court case with plot twists and everything. But that doesn’t mean it understands how courts work.
I won’t end this week without a few weekendlinks. This time: SciBabe, an ugly rock from Mars and Have I Been Powned Domain Search.
SciBabe’s Moment of Science
Yvette d’Entremont, also known as SciBabe, writes daily about either adorable creatures, strange deceases, monstrous chemicals, nuclear f%$k-ups and ways Australian flora and fauna can harm or kill you.
In this blogpost I will reveal something about myself, which I never shared anywhere in my long life on the Internet. It’s rather personal and I don’t like to share a lot on that online. The more because of what the great data aggregators nowadays do with that. But to tell this story, I need to explain what happened to me and how that changed things. Also, I’m fine, really. I do 100+ km bike rides now and this week I rode 60 kms with an average speed of 33.5 km/hr, so that hopefully proves I am currently pretty healthy.
Never grow up!
Stories like this always start by saying that when I grew up I was in every aspect a normal kid. But I’m actually not sure :). Anyway at age 11 my parents noticed that I was not growing very much. This fact was underscored by the fact that my 4 year younger brother was getting taller than me: I was about 120 cm (4 feet) at the time. My mother decided this needed to be checked out by a doctor. And before I knew it, I was dragged to a local medical facility where a nurse drew some blood. I can honestly say I hated people poking needles in my arm very much. Continue reading →
The weather is looking good for this weekend. Time to do some cycling after a creative, but also hectic week. Talking about cycling…
The Alt Tour
You think my 160 km bike ride was crazy? How about a guy who did every kilometer in the Tour de France plus the distances between race starts and finishes (for which the riders take trains, busses and airplanes). A distance of 5500 km and double the altitude gain! That is the Alt Tour that Lachlan Morton rode.
All this with no support team or teammembers to keep him out of the wind. In fact, without hotels even. He has to gather his own food. And his nights of sleep were also less than to be desired. But nevertheless he made it in half the time of the real Tour de France.
In the previous weekend I rode the Vael Ouwe bike ride. We had lots of luck with the weather. My video, made with my Sony actioncam, turned out really nice. The ride through the heather with wildflowers was beautiful and at the halfway point we rode past Radio Kootwijk, a large art deco style building that has been used for radio transmission. Near the end we climbed the Posbank twice. You can find details of the route here.
