Last Friday I almost felt I had to explain a colleague that I don’t always win raffles and lotteries. Because yep, I won another ticket. Again via the Roaring Elephant podcast. It’s pretty worthwhile listening to them, is all I’m saying.
This was a ticket for CodeMotion Amsterdam 2018. CodeMotion is a conference for developers with topics like the blockchain, Big Data, Internet of Things, DevOps, software architectures, but also front-end development, game development and AR/VR.
I was a little late for the opening, but not too late for the keynote. Ton Roosendaal from the 3D computer graphics software Blender told about open source software. When he started Blender the alternatives were expensive software like 3D Studio Max. Nowadays Blender is used in movies (Guardians of the Galaxy II), tv-series (The Man in the High Castle) and games.
They are currently working on a movie based on the Dutch comic Agent 327 and it was already looking very interesting.
Microservices in action at the Dutch National Police
Bert Jan Schrijver of the Dutch National Police shared with us how they handle microservices. His departement has 5 DevOps teams with 30 people. He discussed not only the technical side of things, but also the organisational aspects.
They maintain their own private cloud. It is built with OpenStack and contains 500+ servers. Bert Jan didn’t explain what the applications were, but as he said, you could make a guess from the data they were ingesting, like Twitter data.
Kafka brings that data to Spark and Kafka Streams and the results of that are stored in Elasticsearch, Cassandra and MongoDB.
Regular expressions already are not a topic for beginners, but Juliette Reinders Folmer immediately set expectations of the entry level for her talk about the subject. Probably because of the loads of information she had to share in a short time. I passed the entry level barely.
To begin with: there is not one single regular expression engines and therefor there are different “dialects” that handle some stuff differently. It is another reason why regular expressions must be your last resort. According to Juliette: “if you can do it differently, do that”.
She says: “Regular expressions are a form of blacklisting”, you need to know that to handle it properly. She had many advice. One was that (long) regular expressions can be looking forwards and backwards in data. If you have a lot of it, this costs performance.
Apart from using regular expressions to find the beginning (^) and the end ($) of a line, there are also boundaries between words (\b). And if you are using regular expressions on text in different languages, you should know that it is locale dependent. This requires good testing. A tip to escape meta (or escape) characters was to put them in between square brackets, like [|]. regex101.com is a good place to start testing and test you must.
Building a scalable learning platform with Terraform, Kubernetes and Google Cloud
Erik Veld told how his company Instruqt created a learning platform. With it you can, for example, learn Git or Kubernetes (which sounds useful to me).
When you use this platform, it actually spins up the Google Kubernetes Engine to provide you with a Docker image. Instruqt leans heavily on Google Cloud and services to run this. Terraform was used to create the game environment.
You can learn at Instruqt, but you can also create new courses, also private courses for your organization. As I’m quite the active learner these days I certainly will try this platform out.
Scaling the Internet of Things: 1 to 1 Billion
Johan Stokking from The Things Network is one of the people who started the LoRaWAN standard for Internet of Things. I had hoped this would be more a session about how to scale to handle data from many IoT devices, but this was more about the technology behind LoRa. Interesting detail: LoRa devices don’t have IP addresses. IP requires too much energy.
Though they started with a sensor to detect if your boat is making water (when you’re not in it), the most popular product is a IoT enabled mouse trap.
How to Leverage the Apache Kafka Ecosystem to Productionize Machine Learning
Kafka nowadays is more than a distributed commit log. Waehner Kai from Confluent showed how Kafka nowadays can be the key product between ingest sources, model building and the production environment.
Kafka is capable of sending large amounts of data through your architecture. LinkedIn handles 4-5 trillion messages with Kafka per day. NetFlix uses Kafka to handle 6 PB per day. Or as Waehner says: “you don’t need a PoC”.
Kafka can send through data, but it is also a storage system. The newest developments are Kafka Streams with KSQL. With Kafka Streams you can process data while it’s in motion. For example with Java or Python. With KSQL you can query data in motion, for example like this:
SELECT event_id, anomaly(SENSORINPUT) from health_sensor
The term anomaly() could be an analytical model you apply under water. I might see more of this in the near future.
Open-Source and Exponential Business
Melanie Rieback is founder of Radically Open Security. This a not-for-profit organization that does security for customers (other organizations). Strange right? She is the only employee. And yet 30 people work for Radically Open Security, doing pentests and security trainings. And Radically Open Security produces open source pentesting software. Still, the company is doing allright.
However, this was not a session on her company or about security, it’s why you should embrace open source. Not only as a user, but by producing it. She hardly had to convince me. Some of the arguments mentioned made me think “really, people still say that nowadays?” But it can be necessary to convince others sometimes.
So what are the arguments not to do open source?
“We need to pay our bills”. Like RedHat, Hortonworks and many,many, many others have a problem with that. “Services aren’t scalable”. Ask Uber and AirBnB how that worked out for them. Actually, says Melanie, “”Services are MORE scalable than products now”.
“It’s harder to get funding”. Here Melanie’s surprise reply was “Why do you need funding anyways?”. According to her with venture capital it’s harder to maintain integrity anyways. I think we’ve seen enough examples of that.
“Competitors will steal our IP”. Don’t worry, says Melanie, there’s always the “not invented here” syndrome. (For sure: does anyone need any new data streaming product these days? People seem to invent them every week). And stealing someone elses code is actually quite hard, because you have to understand someone elses code.
“Customers won’t take us seriously”. How does Radically Open Security get new customers? Well, “win the heart of a few geeks and they will FIGHT to get you into their company”. True. She says: “Open-source is the best sales and marketing you can’t buy”.
Sally Goble from Deliveroo explained why we need to say goodbye to perfection. Back when shipping software meant putting it on disks and putting it in boxes to be sold at stores, it had to be perfect. But not anymore.
She says Deliveroo has adopted the “Ship it and see” culture. That brought speed, but initially not always a lot of quality. The told: one developer whispered: “I’m ashamed of my code”.
But if things don’t go right, Deliveroo has a toolbox to set things right fast. New code is rolled out with feature flags. It’s not turned on initially. That will happen later, incrementally. When anything goes wrong, the feature can be turned off.
After the keynote there was quiche (real men don’t eat it, but I did) and drinks. And that was the end of a sunny code day in Amsterdam.