{"id":1968,"date":"2026-01-07T16:25:30","date_gmt":"2026-01-07T16:25:30","guid":{"rendered":"https:\/\/marcel-jan.eu\/datablog\/?p=1968"},"modified":"2026-03-30T09:57:05","modified_gmt":"2026-03-30T09:57:05","slug":"data-engineering-in-the-european-cloud-part-2-scaleway","status":"publish","type":"post","link":"https:\/\/marcel-jan.eu\/datablog\/2026\/01\/07\/data-engineering-in-the-european-cloud-part-2-scaleway\/","title":{"rendered":"Data engineering in the European cloud – Part 2: Scaleway"},"content":{"rendered":"\n

This is Part 2 in a series where I try to create a data engineering environment in the European cloud. In Part 1<\/a> I described my plan for creating a data lakehouse in the European cloud. Now it’s time to get our hands dirty. We’re going to do this in the Scaleway cloud<\/a>.<\/p>\n\n\n\n

<\/p>\n\n\n\n

The architecture<\/h2>\n\n\n\n

To get this data lakehouse running we will create a Kubernetes<\/a> cluster and object storage<\/a> for our data storage. In Kubernetes we can run containerised applications that will run our data lakehouse. I’ve consulted ChatGPT for this architecture. It had a better and more modern solution than I originally had in mind.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

We’re going to use the Apache Iceberg<\/a> open table format. This will allow us to create database like tables based on Parquet<\/a> formatted files. Nessie<\/a> will be the Iceberg data catalog (Hive Metastore was another option). It allows our data solutions to find the Iceberg tables and underlying Parquet files.<\/p>\n\n\n\n

Trino<\/a> will be the query engine. That will be the fastest way to get our first queries going.<\/p>\n\n\n\n\n\n\n\n

Starting at Scaleway<\/h2>\n\n\n\n

First of all we need to create an account at Scaleway. You get 52 days of free usage (as I recall). So you have time to try things out. One important thing when creating your account: you will be asked to verify your account. You get limited quota if you don’t and these will not be enough to create the Kubernetes cluster. So I guess you have to. During the verification you will be asked to hold an ID in front of the camera and make a photo of your face.<\/p>\n\n\n\n

After creating your account you will be asked to define an organisation. Note: I named my organisation here after my employer, but this is project really is a private one of my own.<\/p>\n\n\n\n

Next I made a project in which I misspelled sovereign. And you can’t rename it afterwards. So I have to live with that for now.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

Creating the Kubernetes cluster<\/h2>\n\n\n\n

After logging in at Scaleway you enter the Scaleway Console. On the left hand menu you find the products you can build. Here, go to Containers, and then Kubernetes. Choose the region where you want to create the Kubernetes cluster. I’m starting completely greenfield here, but if you have other parts of your architecture in a particular region, you probably want this new cluster in the same region.<\/p>\n\n\n\n

I’m going for the cheapest options here. So I’m going with the free control pane.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

You are also going to need a Virtual Private Network, which you can create during this process. And this is free. <\/p>\n\n\n\n

Next you need to configure a pool and this is where the costs enter the picture. Notice that you can choose a Development node type, which has the cheapest node type: DEV-M1. It is \u20ac57.53 per month. But an important point: unlike Azure Kubernetes Service<\/a>, you can’t stop the cluster to cut down the costs. Once it runs, it runs. (As far as I can remember I was able to stop AKS when I wasn’t using it).<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Once the cluster is running, you can go to the cluster page and then to the Kubernetes Dashboard to see the state of the cluster, or generate a kubeconfig.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

I’m using kubectl and helm on the command line to get things installed. For that, the kubeconfig is very handy.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Object storage<\/h2>\n\n\n\n

You can find this in the Scaleway Console on the left hand menu under Storage, Object storage. I have created a bucket with the name “lakehouse” in the same region as the Kubernetes cluster. You can choose between Private and Public visibility, but don’t choose Public<\/strong>. It is not secure and really Private works fine in this project.<\/p>\n\n\n\n

You can choose the use case for the bucket. “Big data” seemed like a good choice.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

And good news: object storage comes with a free 90 day trial up to 750 GB. That is more than enough for now.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

It also has an outgoing transfer setting for which I chose 10 GB. <\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

Access to the object storage<\/h2>\n\n\n\n

I am going to give you the benefit of not making the same mistakes here as I did. So that you don’t need to wonder why you can’t seem to reach the block storage.<\/p>\n\n\n\n

In the Scaleway Console, under Security & Identity (in the left hand menu), you’ll find IAM, or Identity and Access Management. Go to Applications and create an application. I have named it Nessie.<\/p>\n\n\n\n

Under this Application we can add an API key, which we’ll be going to need when we’re installing Nessie. And yes, this API key will be used for Object Storage.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Copy the Access Key ID and Secret Key ID. It will only be shown once.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Your API key still has zero access. For this we need to add a Policy to the Application. <\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Next we need to create rules. The scope of these will be our Project.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Next we choose the permission sets. Under Storage, choose ObjectStorageFullAccess. I made the mistake to pick ObjectStorageBucketPolicyFullAccess instead and that does not work.<\/p>\n\n\n\n

With the policy created, your application should look something like this:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

[Update 2026-03-30]<\/strong> Now you also need to add a Kubernetes access policy. So create an extra Permission set with KubernetesFullAccess.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Checking the Kubernetes cluster<\/h2>\n\n\n\n

The next steps will be done on the command line. For this you have to install kubectl<\/a> and helm<\/a> on your machine. I have done this on a MacBook. According to the kubectl and helm pages there are Windows versions as well.<\/p>\n\n\n\n

To connect to your Kubernetes cluster with kubectl use the kubeconfig from your Kubernetes cluster overview and download the file to a directory of your choosing.<\/p>\n\n\n\n

On Mac\/Linux you create variable called KUBECONFIG. Now kubectl and helm will find their way to the Kubernetes cluster.<\/p>\n\n\n\n

export KUBECONFIG=kubeconfig-k8s-ams-amazing-merkle.yaml<\/code><\/pre>\n\n\n\n
Let’s see if this has worked:<\/p>\n\n\n\n
kubectl cluster-info<\/code><\/pre>\n\n\n\n
This will give output like this:<\/p>\n\n\n\n
Kubernetes control plane is running at https:\/\/d5752251-e566-4dea-af21-f718b1f9466c.api.k8s.nl-ams.scw.cloud:6443\nCoreDNS is running at https:\/\/d5752251-e566-4dea-af21-f718b1f9466c.api.k8s.nl-ams.scw.cloud:6443\/api\/v1\/namespaces\/kube-system\/services\/coredns:dns\/proxy\n\nTo further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.<\/code><\/pre>\n\n\n\n
That vague message tells us that Kubernetes is up and running.<\/p>\n\n\n\n
[Update 2026-03-30]<\/strong> If you get the message “Unhandled Error” err=”couldn’t get current server API group list: the server has asked for the client to provide credentials”, you have to add a Permission set to your application with KubernetesFullAccess.<\/p>\n\n\n\n
Also create a namespace for this project:<\/p>\n\n\n\n
kubectl create namespace lakehouse<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
Installing Nessie<\/h2>\n\n\n\n
Time to install Nessie. We’re going to use Helm for this. Helm is the package manager for Kubernetes. It allows us to install all necessary components for an application.<\/p>\n\n\n\n
First we add the Nessie repository to our Helm repos:<\/p>\n\n\n\n
helm repo add nessie https:\/\/charts.projectnessie.org\nhelm repo update<\/code><\/pre>\n\n\n\n
Now we need to tell Nessie where to find our object storage. For this I created a nessie-values.yaml<\/a> file. You can find my example<\/a> in my European cloud Github repo. <\/p>\n\n\n\n
Note: you need to change the following parameters:<\/p>\n\n\n\n