TL;DR:

These Python and Raspberry Pi projects. They are fun aren’t they? And often they look deceptively simple. But you don’t see all the projects that failed and usually not where they struggled. This project got stuck (and almost failed) at:

• Not being able to scrape dynamic website content.
• When I found out how to do that, I couldn’t run my working Python code on the Raspberry Pi.
• That turned out to be because the scraping packages use a chromium browser, but not for the ARM processor that the Raspberry Pi has.
• And to top it all off, the Python package for the Inky Impression e-ink display had some kind of problem running numpy.

Origin story

Last week I was riding my racing bike on a trainer, on the Tiendewegbrug in Gouda to collect money for cancer research. Like last year, I’m going to ride the Tour for Life. If I manage to get the 2500 euro starting fee that is. My DIKW Intelligence colleague Aniel Narain did the talking to the passersby, and I did the riding.

It was a very successful event. But sometimes we encountered people who said they were going to donate with our QR code, but actually they stopped somewhere before they accepted the payment. So Aniel said that it would be great if we had a live display showing the amount gifted. And then we could see live that the money increases. Couldn’t I do that with my Raspberry Pi’s and e-ink displays?

The plan

How hard could it be? I’ve got my fair share of experience with the Inky Impression display. I’ve done a bit of scraping before, with BeautifulSoup4.

This particular exercise would require the following things:

• Find out where to get the gift amount on the website.
• Display the gift amount on the e-ink display and refresh it whenever needed. (Refresh could also be with the push of a button.)
• Find a way to power the Raspberry Pi “in the field”.
• Find a way to get an Internet connection without known WiFi available.
• (Make a version (Python package? Github?) that can be shared with others.)

Scraping

On my MacOS I created a new Python project. This time with “poetry”, a tool I’m trying out. It worked very well. From my PythonProjecfts directory I ran this command:

poetry new --src pidonationtracker

And out comes a nice Python src structure. Next I run:

poetry install

And I can start adding Python packages I needed: requests and BeautifulSoup4.

poetry add BeautifulSoup4

poetry add requests

And we’re off to the races. Now I need to go to my team’s donor page: https://www.tourforlife.nl/de-rollende-rijders. And look up what element has the gift amount. For this I fire up my browser’s dev tools (F12 usually).

See, there’s a class called gift-amount. That’s the class we’re going to need. Next I use BeautifulSoup (Next time use “from bs4 import BeautifulSoup” instead of “import BeautifulSoup, to make things easier) and we get the data from this class.

gift_amount = soup.find("div", {"class": "gift-amount"}).text

And… nothing. Absolutely nothing. A bit of searching through the HTML learns that gift-amount is not in it. It turns out what we’re dealing with here, is dynamic content. A bit of Duckduckgoing (instead of Googling) learns I need the requests-html package for that.

poetry add requests-html

I copy some example code from Stack Overflow and try to get the data from gift-amount. The first error I get is:

ImportError: lxml.html.clean module is now a separate project lxml_html_clean.

More Duckduckgoing learns I need to install this package:

poetry add lxml_html_clean

Problem is, I have no idea how requests-html is supposed to work. And I don’t have time to read the entire documentation. Luckily there’s Youtube. I ran into this video and after watching it, things became a lot clearer.

After altering my code, I manage to get the gifted amount in a variable. And using Pillow, I get it in a jpg with a nice background.

Oh look at the time. That’s 2 hours gone. A lot of that went into trying BeautifulSoup4, and cursing about not getting any data. The neighbours might have heard.

On to the Raspberry Pi

Let’s bring this Python code to the Raspberry Pi. I just want to quickly see if it works, so I’m going to quickly create a virtual environment, pip install the required packages and let’s run this sucker.

python -m venv .venv

source .venv/bin/activate

pip install BeautifulSoup4

pip install requests

pip install requests-html

pip install lxml_html_clean

pip install Pillow

pip install inky[rpi,example-depends]

So let’s run my Python script:

$ python3 pidonationtracker.py

And..

[INFO] Starting Chromium download.
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 183M/183M [00:17<00:00, 10.4Mb/s]
[INFO] Beginning extraction
[INFO] Chromium extracted to: /home/inky3/.local/share/pyppeteer/local-chromium/1181205
Traceback (most recent call last):
  File "/home/inky3/PythonProjects/pidonationtracker/src/pidonationtracker/pidonationtracker.py", line 59, in <module>
    donation_amount = get_donation_amount()
[..]

  File "/usr/lib/python3.9/subprocess.py", line 1823, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/home/inky3/.local/share/pyppeteer/local-chromium/1181205/chrome-linux/chrome'

I did my fair share of looking up the error and finding all kinds of articles on Stack Overflow. Many were years old BTW. I tried all kinds of commands: installing chromium browser, downgrading chromium browser, quick! maybe selenium works instead? Nope, nope, nope.

And time (my weekend) was running out. I started this Saturday evening. Now it was Sunday evening. In my mind I saw a countdown clock quickly going 5.. 4.. 3.. 2.. 1.. Anddd put you keyboard down! The weekend is over! Who knows when you will have time to work on this again?

And that’s how projects die. Sometimes it’s really hard to pick up the pieces weeks later on.

I asked questions on Mastodon (thanks @philbetts@mastodon.social and @karlwelzel@toot.community for trying to help me). And got quick replies. But the solution eluded me. I was close to giving up. But on the other hand: this should be possible, should it not? A few days later I wrote a question on Stack Overflow in all detail: https://stackoverflow.com/questions/78448382/scraping-with-requests-html-on-raspberry-pi-4-fails-with-oserror-errno-8-exec

But there came no answer.

Maybe AI can help?

Then I decided to try AI. I didn’t get my hopes up. LLM’s can do interesting stuff, but could it also solve this kind of specialised stuff? I tried phind.com and to my surprise it actually came up with some good hints:

https://www.phind.com/search?cache=qascp9zgk79vwh5xmd52vwcx

The Chromium browser and the Raspberry Pi

It turned out I was actually pretty close in my approach. I was following advice on Github (https://github.com/miyakogi/pyppeteer/issues/250) to install an older version of the Chromium browser codecs.

So here’s what actually was the problem: the requests-html package uses a thing called puppeteer. And puppeteer runs a Chromium browser in the background to pick up the dynamic content that’s not in the regular HTML. But the Chromium browser that it starts, is a version for Intel processors. And Raspberry Pi’s have ARM processors.

And there does exist an ARM version of the Chromium browser and ARM compatible codecs, but these haven’t been updated since 2018. So all you have to do, is uninstall them and install the ARM versions from 2018.

So I downloaded this one: http://launchpadlibrarian.net/361669486/chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb and this one: http://launchpadlibrarian.net/361669485/chromium-browser_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb:

wget http://launchpadlibrarian.net/361669486/chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb
wget http://launchpadlibrarian.net/361669485/chromium-browser_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb

When I tried to install the first one, I got errors about dependencies:

$ sudo dpkg -i chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb

Selecting previously unselected package chromium-codecs-ffmpeg.
dpkg: considering removing chromium-codecs-ffmpeg-extra in favour of chromium-codecs-ffmpeg ...
dpkg: no, cannot proceed with removal of chromium-codecs-ffmpeg-extra (--auto-deconfigure will help):
 rpi-chromium-mods depends on chromium-codecs-ffmpeg-extra
  chromium-codecs-ffmpeg-extra is to be removed.

dpkg: regarding chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb containing chromium-codecs-ffmpeg:
 chromium-codecs-ffmpeg conflicts with chromium-codecs-ffmpeg-extra
  chromium-codecs-ffmpeg-extra (version 120.0.6099.102-rpt1) is present and installed.

dpkg: error processing archive chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb (--install):
 conflicting packages - not installing chromium-codecs-ffmpeg
Errors were encountered while processing:
 chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb

I tried uninstalling these dependencies. But the list got longer and longer. And at one point among the dependencies to be uninstalled were Firefox and the Konqueror browsers. Surely I don’t need to do that?

What I really should have done, was run the same command but with the –force-all option.

sudo dpkg --force-all -i chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb
sudo dpkg --force-all -i chromium-browser_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb

And this worked much better. But it wasn’t the entire solution. I also needed to link the chromium browser to the installed version. That OSError: [Errno 8] Exec format error shows the place where you need to be.

First check the location of the Chromium browser:

$ whereis chromium-browser

chromium-browser: /usr/bin/chromium-browser /usr/lib/chromium-browser /etc/chromium-browser /usr/share/chromium-browser /usr/share/man/man1/chromium-browser.1.gz

So it’s in /usr/bin/chromium-browser.

Now go to the puppeteer directory indicated by the error:

cd /home/inky3/.local/share/pyppeteer/local-chromium/1181205/chrome-linux/

And link to /usr/bin/chromium-browser

ln -s /usr/bin/chromium-browser chrome

And finally the scraping code worked. And that was 6 days after I started the project. Time well spent.

Projecting the gift amount on the Inky display

With the gift amount number ready and the Pillow image also ready, it was one small step to getting it on the display. But instead I got an error that numpy wasn’t there.

  File "/home/inky3/PythonProjects/pidonationtracker/.venv/lib/python3.9/site-packages/inky/inky.py", line 10, in <module>

    raise ImportError('This library requires the numpy module\nInstall with: sudo apt install python-numpy')

ImportError: This library requires the numpy module

Install with: sudo apt install python-numpy

But when I tried to install it, I got the message it was already there.

So more searching, more Stack Overflow questions and answers. More messing up my Raspberry Pi install. Maybe install it with apt instead of pip? Maybe install an older version (great! more old stuff)?

sudo python3 -m pip install 'numpy>1.0, <1.15' --force-reinstall

Running this took very, very long and eventually it gave an error stack of about 20 pages. The answer must be somewhere in here?

Eventually I found the real problem in this part of the error stack:

Original error was: libopenblas.so.0: cannot open shared object file: No such file or directory

Which could be solved with this install command:

sudo apt-get install libopenblas-dev

Note that in this solution I’ve now used three different methods to install things: pip, apt and dpkg. But who’s complaining, when the result finally is projected on the e-ink display?

Halleluja! It has finally happened. And I was only on day 6 of the project.

Now I still have to find out how to use my iPhone’s hotspot on the Raspberry Pi 4, but I guess that’s easy. Right? Right?

Lessons learned

Let’s not kid ourselves. This stuff can sometimes be really hard. Yes, sure it is easy when you know what you’re doing. But who really knows the ins and outs of Python, scraping, installing stuff on a Raspberry Pi (with different methods), browsers, and e-ink displays?

Yes, sometimes you get away with creating such a project in a matter of 2 hours. Recently I got Google Calendar data and projected it on this same Raspberry Pi with Inky Impressions display and I was done in one evening. But that’s not how it always goes.

The beautiful Raspberry Pi projects you see on the Internet are great, but they’re also the result of survivorship bias.

So I guess what I’m trying to say is: don’t feel bad when your Python project fails. It’s not because you’re stupid, or “not cut out for this”. If anything, blame Google for not creating a ARM based Chromium browser ffmpeg codec since 2018.

Also, take notes

What did work BTW, was rigorous note taking along the way. I’ve created an Evernote document for this project and added all the commands I’ve tried and the links where I found useful advice. Well, almost all. Sometimes I keep trying and trying stuff and I forget. But I shouldn’t. Because these notes really do help finding out what has changed, and how I can do the same for future projects. (Like that Inky related numpy problem. It was not the first time.)

But most importantly: I can more easily find where I left off. When I need to stop working on this project because “I’m out of weekend”, it could be days before I can pick this project up again. This is where note taking really helps, raising the chances of it becoming successful.

BTW, if you liked this (way too long) post, would you like to donate here?:

https://www.tourforlife.nl/de-rollende-rijders