Then it hit me - we’re running Docker and Kubernetes, and the containers in Docker don’t have a process management system in place. This means that if a process in a container spawns child processes, they exist without waiting for termination. Thus, child processes become zombie processes, and they pile up and cause instability. Without an init system, there is no process to clean up zombie processes.

Since Docker 1.13.0, they have added the –init flag when you run Docker via the CLI. This is acceptable for our local development purposes, however, in the production environment, we are utilizing GKE and required a solution to address the issue. There are a couple of ways to run it in k8s:

1. Use tini/dumb-init or any other init system as ENTRYPOINT in your Dockerfile, with CMD as your program:

FROM debian:stable-slim
...
RUN apt install tini
...
ENTRYPOINT ["tini" "--"]
CMD ["/app/screenshoter"

2. Use init container that starts before your main app in Kubernetes deployment:

apiVersion: v1
kind: Pod
metadata:
  name: screenshoter
spec:
  initContainers:
  - name: init-tini
  # sample image with tini
    image: krallin/ubuntu-tini:latest
    command: ["tini", "--", "true"]
  containers:
  - name: screenshoter
    image: screenshoter:latest
    command: ["/app/screenshoter"]

After init container completes, any zombie process will be reaped by tini.

3. (theoretically) set shareProcessNamespace: true in securityContext in Kubernetes. I haven’t tried this and I do not plan to, as I only touch securityContext when there is an absolute need for it.

I chose to go with option number 1 as it was the simplest and I like to have same setup locally and on all other environments, meaning same Docker image with init system both locally with Docker (actually, Orbstack) and in dev/prod in GKE.

If you read more carefully than me, you will find out that even NodeJS doesn’t recommend running NodeJS processes as PID 1 as they would not respond to SIGINT. This isn’t only applicable for NodeJS, but for any process like this, as we can see that it also happens on Golang app.

Not knowing something isn’t stupid - not wanting to learn new things is.

On Nov 10th Apple confirmed the rumors that they are cutting ties with Intel CPUs and starting using their own - Apple Silicon ARM-based CPUs. While that came as a surprise to some, Apple has been working for years to make this transition. They don’t want to rely on Intel more, and Intel has arguably slowed down the evolution of CPUs in the last few years.

Long story short, Apple has announced two M1 powered MacBooks as a start - MacBook Air with passive cooling and with all physical keys and MacBook Pro with active cooling and Touch Bar.

I wanted to wait for the second iteration of Apple Silicon chips and MacBooks as I thought that the transition process won’t be that smooth. I remember the transition from PowerPC to Intel - it wasn’t nice and I expected general problems: slowness, not-so-fast adoption, and just regular bugs. After the first reviews popped in they were surprisingly positive - even from hard-core Apple haters. Again, I opted-in to wait. I had my MacBook Pro 2018 15.4" with i7 and 16GB RAM, why would I need a new computer? Well, again, I was wrong, as the IT professionals, devs, SREs, started showing numbers from new MacBooks in their regular work and the general feeling is that we have something magical here.

I was hooked. Pulled the trigger and ordered MacBook Air M1 with 16GB RAM.

Software

Big Sur came out with a shiny new design which I don’t like completely. There is much-unused space, rounded corners of the Windows, etc. But, this is not about Apple’s software, more of third party software that I daily use. I will just list the ones that have native versions. You can always use https://isapplesiliconready.com/ to check if some software not listed here has a native ARM version. Native:

• Brave
• Slack
• VSCode (Insiders version)
• Firefox
• iTerm
• Telegram
• TablePlus
• PyCharm
• Goland
• Fantastical
• Paw
• Bear
• Things
• iAWriter
• Parallels Preview
• nodejs (npm)
• python
• vim
• postgres
• brew

Non-native (for now):

• Dropbox
• Stay (support incoming in 1.4)
• Spotify
• Signal
• Golang and all tools related to it (kubectl, helm, etc)

Golang 1.16 is coming out this or the next week and then all tools related to it will support aarch64.

Rosetta 2

When the app isn’t complied natively, it goes through a transition layer called Rosetta 2. Rosetta has to “translate” the app for Silicon to “understand” and the first start of the app lasts 5-20s. Each next start of the app is like a native app, no matter if you restart your computer or not.

Brew - to ARM or not to ARM

With 3.0.0 version, Brew officially supports installations in /opt/homebrew. For now, the preferred way is to have it running both, one in the (good) ol’ /usr/local directory, and the other in /opt/homebrew. After that you should put both in the path, but /opt/homebrew first.

Virtualization

When Apple announced the M1 the biggest problem with it is how to run virtualization, aka Windows and Linux in virtual machines. Unfortunately, I have to use Windows for a few things - managing my company’s bank account and some government access. Windows 10 has insider preview for ARM64, and virtualization software Parallels has Technical Preview with support for arm64 of W10/Ubuntu.

My experience with it has been silky smooth - it passes through my USB token, Java e-banking software installs correctly and everything works. Kudos Parallels devs!

Docker technical preview

Docker is a go-to environment for almost any dev these days. Its performance on Mac has historically been a major pain, mostly because of the way how it goes around macOS limitation to use the filesystem. Read no further, it is still awful, but this time a bit better because everything works faster because of the new SoC.

Other than that - you will need arm64 images of your apps to achieve better performance. You can still run the amd64 image without any problems via QEMU emulation (Docker does it automatically if it doesn’t find arm64 version of the container), but it will be slower.

I have set up our images at The Remote Company to be built on GitHub Actions for both amd64 and arm64 images and they work pretty much great. Other than that, there are still performance issues vs the Linux version of Docker.

Why Air instead of Pro for power users?

This machine is cold. Colder than the heart of my mother-in-law. Hopefully she won’t read this, but my wife will. I regret nothing! Please take care of my cat. It has no fans so it is silent. Of course, I generally run things that require a short burst of performance, not constant performance like rendering a video, then I would suggest you go with MacBook Pro as its performance is a little bit better as it doesn’t throttle under high load for a long time.

Other than that, I dislike touch bar from the bottom of my heart. Air still has physical keys and that is what nudged me to it. Apple has been stubborn for years with the touch bar that 99% of power users are annoyed with, but the reports say that with incoming 14" and 16" MacBook Pros they will finally go back to physical keys.

Good business model - introducing old things as new that users will like.

Keyboard

Let’s face it, all Apple keyboards after MacBook Pro 2015 have been abysmal. The butterfly keyboard is a failure that broke in all 3 of its iteration. Last year Apple announced that they are going back to the old-style keyboard on MacBook Pro 2020, and they have continued with that practice on M1 MacBooks. The new Magic keyboard is amazing - it feels even better than the old favorite MacBook Pro 2015.

I guess anything is better than the piece of plywood called MacBook Pro 2016-2020 keyboard. May it rest in peace or for someone, literally in pieces.

Battery life

I almost forgot one big improvement. Battery life on M1 is crazy. I regularly get 12-16 hours, depending on the usage, and I don’t even use Safari. Knowing that you can go a full day without charging the battery or even two working days is a great thing to have.

NPM performance benchmark

Now, a few benchmarks. I compared nodejs compilation time on both of my machines - MacBook Pro 2018 (i7 6c, 12t, 16GB RAM) and MacBook Air M1. Both have nodejs v15.8.0 natively installed and both have the tests running the same commands in docker image node:v15.8.0-buster.

Native runs:

Docker runs:

You can conclude yourself, but in most of the tests, Silicon has been more than 50% faster.

Problems

The only problem that I had is that my USB-c screen stopped showing the correct color range as Macbook now recognizes it as a TV, so it has a limited color output. I use Caldigit TS3+ and LG38WK95-c and my workaround was to buy a USB type-c to HDMI adapter and connect it to Caldigit and colors are now fine. A little bit frustrating, but a software problem that I hope some future Big Sur version will fix.

Conclusion

MBA is the best value for money in the market currently. ~1600€ for a 16GB/512SSD version that outperforms even desktop CPUs in a small and portable package as Macbook 13" is truly astonishing and I’m excited to see how will new Pro machines perform with improved M1s and more RAM as M1 is limited to 16GB.

Bravo, Apple!

Around 2 years ago I got the offer to join MailerLite as a remote contractor. I was over-thinking whether I should start working remotely, what would it mean for me, my habits, social life, and my career progress. There were too many unknowns at that point as I have only done some freelance gigs remotely, but have been involved with distributed teams all over the world in previous companies. I took the red pill and my remote journey began.

All my doubts were cleared right away.

The first one - traffic. I hate the traffic in Belgrade. The city isn’t building its infrastructure (roads, bridges) as fast as it is growing and it is showing. On a good day, it took around 30 minutes of driving to work, which is not a problem. But, in the majority of days going to work and back home took on average 2 hours. I say 2 hours because you have to take into account that you have to find a parking space, probably 10 minutes from the office, and then walk to the office which all adds to 2 hours of lost time each day. If you calculate - it means that you are losing 22 days per year in the traffic.

What could you do with that extra time that you have? I remember one analogy when I talked to one of my friends as he said: “It is like when people are telling that smokers could put all money they spend on cigarettes aside and can buy a new car after some time of saving. Then you see people who stopped smoking and after some time you don’t see them with a new car”. That wasn’t the case for me. I rediscovered hobbies that I have stopped practicing because of lack of time - started playing guitar more, recording new music, and most importantly, I had time to read more non-business related books. You can become dull and get into a work-home-sleep routine because of the lack of time.

Micromanagement was the other thing that I thought could happen in remote work. Fortunately, it didn’t. It is just a matter if you trust your employees or not. If you don’t trust your employees, you either didn’t hire the right people for your company or you are suffering from trust issues and adapting to the situation. Usually, it is the second thing. People who slack off because there aren’t any measures of control because they are working remotely show their faces and bad habits really fast and you can easily filter them.

When working remotely, you are your own manager. You must plan your time and goals, and how much you give yourself to work. Not too little and not too much, balance, as always, is the key.

There are so many tools to make communication while working remotely easy. We utilize Slack, have regular calls on Zoom, write our monthly feedback on 15five, and much more. If we need to brainstorm, we get on a video call or write in Slack.

There is one potential downside of working remotely for one target group - junior engineers. I will explain why I don’t think it is a rule but purely depends on the person. While junior engineers can always sit next to their mentors in a regular office job, here it is a bit different. Your company has to have really good documentation and processes so they don’t get lost. But, on the other hand, at MailerLite we are always looking for persons that are “self-starters” and can pick-up things themselves without or with minimal interference from other colleagues. In my experience, juniors can struggle at the beginning for the first (half of) year, but after that, they are usually stronger with knowledge and work ethics than juniors in regular offices. They are more independent in work and can easily learn new things and implement them.

The key thing in remote work is overcommunication. I can’t stress enough how much it is important. As a pitfall of remote work, you must write and overshare because there is no such thing as over-communication. Why do we do that? We are a remote-first company, but we have an office in Vilnius if someone wants to work from it. A couple of people are working from the office, but as I have been there a couple of times it is the most silent office I have ever been in. The communication is done via Slack or conference calls with everyone, so no one is left out, as this can be a problem when you have a combination of office and remote work in your company.

Productivity, personally, is sky-rocketing once I switched to the remote. I get at least 50% more stuff done as there are no distractions, unnecessary small-talks, listening to office gossips, braggings, and other things that can happen in the office.

Freedom is also an important thing. Sometimes I have to do something personal in the morning, or the middle of the day. Sometimes, I can go to the gym during the day and go back refreshed and start working. I can go to my hometown when I want to and work from there. I can travel wherever I want and work from there. I have become more social as I have more time for my friends and family and am less stressed. No one is looking at how much time you are spending working - but if you did the task or not which is the only important thing.

The biggest indicator that remote work is a benefit for both an employee and the company is that Microsoft and Google are now permanently remote if you want to. Pandemic nudged them in that direction, but they have seen the benefits and adapted.

Times are changing. Gosh, what a cliche sentence. Times are always changing, it is up to us if we want to adapt. Remote work has been around for a long time, but most of the companies resisted the change. I’m happy that companies are now forced to change because it requires a change of mindset, but there is a quote from my favourite TV series of all time, Twin Peaks, “There’s some fear in letting go”.

To maximize the throughput of email servers you can do a few things, optimise the resources, optimise MTA, but there is one more thing that you can optimise and that is obvious but overlooked - DNS tweaking. I won’t go into the whole flow of sending and receiving the message, but the DNS part. When you want to send the email, MTA has to do at least two DNS queries - to get the MX record of the server you are trying to send the email to, and to get an A record of that server and there are even more — it could be getting PTR records. You can imagine how time costly is this operation when you are sending 30 million emails per day. To get a clearer picture — it is 21000 emails per minute. If you multiply it with the number of DNS queries that each email sending has to do, you come to crazy numbers.

To mitigate that overhead of running those DNS queries to an external DNS server, we came up with the idea of using a local recursive, validating, and caching DNS server. That is where Unbound comes into the picture.

Recursive part of Unbound gets the request from the client to traverse the path of DNS to get the answer to the question.

We will assume that the question in this example is “What is the A record of mailerlite.com”:

• Client sends a query “what is the A record of mailerlite.com” to the server configured in resolv.conf
• DNS server (Unbound) checks if it has that record in its cache
• If the server wasn’t asked before for that record it won’t be found in the cache so it has to look it up on the root servers
• Unbound asks the root servers for the record
• Root server replies to TLD servers for .com
• Unbound asks another query — this time to the .com TLD servers
• This time - TLD server replies with referral to the authoritative name server for mailerlite.com at CloudFlare (we host all our domains at CF)
• Once more, Unbound sends another query for A record of mailerlite.com to CF
• CF replies with the A record of mailerlite.com which contains the IP address for it
• Unbound answers with the IP address of mailerlite.com to the client

Sounds like a longer process than it should be, right? Imagine doing this each time we want to send an email. Yikes, it would kill our performance.

That is where caching part of Unbound comes into place. If the query has been asked before, Unbound will respond from its cache, if not — it will recursively resolve the query if it hasn’t seen it before. This operation saves a lot of time as cached results are returned in a couple of microseconds while usual DNS queries (recursive) can take hundreds of miliseconds or more.

The following is the speed comparison when the query from cache and from publicly available DNS servers:

Grafana graphs:

This one shows the cache hit/miss ratio. For the last 24 hours, we had 432 million hits and only 9.28 million misses. That means that only 2.14% of the queries went to external DNS servers to check the records and that the other 97.86% was answered from the cache in matter of microseconds.

Unbound is a fantastical piece of software, and as you have seen it provides quite a boost and also increases our email throughput.

Creating them manually via command line is a bad practice and makes problem managing, updating, and adding passwords. There are quite a few tools that address this problem, the most popular of them is Hashicorp Vault. While I very much like what Hashicorp is doing and what Vault provides, it generates a big overhead in managing it and most of the systems/companies don’t need it so I wanted do use something simple, manageable and secure for all of our workloads at The Remote Company.

A little of the background - we use Helm charts for each of our apps. They are deployed to GKE with Github Actions, utilizing Skaffold as a build/deploy tool, so we wanted something compatible with it. Queue the Helm Secrets plugin from Zendesk - wonderful implementation of an even more wonderful Mozilla SOPS. SOPS is an editor from Mozilla able to encrypt the data with AWS KMS, GCP KMS, Azure Key Vault, and PGP. Originally, it was written in Python, but Mozilla rewrote it in Golang making it easier to build and redistribute as Golang compiles everything into a single binary. Helm secrets is a helm plugin that uses SOPS in the background to encrypt/decrypt the data, but also automatically applies the secrets to Kubernetes cluster and attaches it to the deployment on demand (we will cover it a bit later).

The instructions that follow are how you would manage it from macOS.

The tools that we will be using are:

• Helm
• Helm secrets
• Skaffold
• GCloud SDK
• Mozilla SOPS
• Kubectl

Dependency setup

Install dependencies via brew.

brew cask install google-cloud-sdk
brew install kubectl helm skaffold sops
helm plugin install https://github.com/zendesk/helm-secrets

Login to our Google Cloud project. Follow the instructions on setting up your project and compute region and zone.

gcloud auth login
gcloud auth application-default login

Creating KMS key

Create Google KMS key used for encryption and decryption of the content

gcloud kms keyrings create sops --location global
gcloud kms keys create sops-key --location global --keyring sops --purpose encryption
gcloud kms keys list --location global --keyring sops

# You should see similar output to

NAME                                                                  PURPOSE          PRIMARY_STATE
projects/my-project/locations/global/keyRings/sops/cryptoKeys/sops-key ENCRYPT_DECRYPT  ENABLED

Setup secrets

You should have a helmchart in your project. In your helmchart folder, create .sops.yaml file.

creation_rules:
  - gcp_kms: "projects/my-project/locations/global/keyRings/sops/cryptoKeys/sops-key"

In order to easily diff in git, you would need .gitattributes in the same folder with sopsdiffer.

secrets.yaml diff=sopsdiffer
secrets.*.yaml diff=sopsdiffer

In our helmchart, we will need secrets file. When you start, you can add secrets as plain text in that file, later on, we will encrypt them with SOPS. In this example, we will create secrets.dev.yaml in our helmchart folder.

secrets:
  first-password: very-secure
  second-password: much-locked
  third-password: so-encrypted

Now we can encrypt the secrets file. The file will be encrypted with KMS key.

helm secrets enc secrets.dev.yaml

It should look something like this:

secrets:
  first-password: ENC[AES256_GCM,data:0DKw4JAoAv8kg139TfEatrrhdVEhBKFtFYpuNBVlxDqd4qc6u6SaK1MbsGUqHLuuMhyCuUNl2MepmMwm2QsgVqx7euBIeqKSe6ZUYdiAKuwFrGpLzQJ==,type:str]
  second-password: ENC[AES256_GCM,data:Mv1Frwz6CT2Jnq5EmBgZJl6sn7cYzXFceLSlOiu5djguJi6cRJAW90cpIkwpcuvYc3ulyZagoB7vfO2nL17IOa3hZ4GZRhzXZXqo0wDAgXFjucR1L4B==,type:str]
  third-password: ENC[AES256_GCM,data:B9FxMPPSOisPrlnsDfwilwtagxVfWbqCwWJw7XsfqElRqitvCpkB3anvj1uW0hpruJ96LwAJDwplqu8iaZmz14OyeKMLREuCpwRx5b4OhEUbRGTtESA==,type:str]
sops:
  kms: []
  gcp_kms:
    - resource_id: projects/my-project/locations/global/keyRings/sops/cryptoKeys/sops-key
      created_at: "2020-09-04T11:21:03Z"
      enc: somevalue=
  azure_kv: []
  hc_vault: []
  lastmodified: "2020-09-30T15:12:35Z"
  mac: ENC[AES256_GCM,data:somevalue==,type:str]
  pgp: []
  unencrypted_suffix: _unencrypted
  version: 3.6.1

You can see that our secrets are now encrypted and they can be referenced in helm template.

Referencing secrets in helm values

First, we need to create a template for helm so it can iterate over your secrets and add them automatically. Here’s the example of the template:

apiVersion: v1
kind: Secret
metadata:
  name: {{ include "app.fullname" . }}
  labels: {{- include "app.labels" . | nindent 4 }}
type: Opaque
data:
  {{- range $key, $val := .Values.secrets }}
  {{ $key }}: {{ $val | b64enc | quote }}
  {{- end }}

This template goes over secrets file, looks for the secrets key and then adds all the keys referenced with their values to a Secret object in k8s.

Usually, your secrets are in env and you would have a template in helm that can iterate over your env variables and attach them to the deployment. Here is one example of a template for k8s deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "app.fullname" . }}
  labels:
    {{- include "app.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "app.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "app.selectorLabels" . | nindent 8 }}
    spec:
    {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
    {{- end }}
      securityContext:
        {{- toYaml .Values.podSecurityContext | nindent 8 }}
      containers:
        - name: {{ .Chart.Name }}
          securityContext:
            {{- toYaml .Values.securityContext | nindent 12 }}
          image: "{{ .Values.image }}"
          imagePullPolicy: {{ .Values.imagepullPolicy }}
          {{- if .Values.volumeMounts }}
          volumeMounts:
            {{- range .Values.volumeMounts }}
            - name: {{ .name }}
              mountPath: {{ .mountPath }}
              {{- if .subPath }}
              subPath: {{ .subPath }}
              {{- end }}
              readOnly: {{ .readOnly }}
            {{- end }}
          {{- end }}
          env:
            {{- range .Values.env }}
            - name: "{{ .name }}"
              value: "{{ .value }}"
            {{- end }}
            {{- range .Values.envExtra }}
            - name: "{{ .name }}"
              value: "{{ .value }}"
            {{- end }}
            {{- range .Values.secretEnv }}
            - name: "{{ .name }}"
              valueFrom:
                secretKeyRef:
                  name: "{{ .secretKeyRefName }}"
                  key: "{{ .secretKeyRefKey }}"
            {{- end }}
          ports:
            - name: http
              containerPort: {{ .Values.service.port }}
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /probe
              port: http
          readinessProbe:
            httpGet:
              path: /probe
              port: http
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
      {{- if .Values.volumeMounts }}
      volumes:
        {{- range .Values.volumeMounts }}
        - name: {{ .name }}
          secret:
            secretName: {{ .name }}
        {{- end }}
      {{- end }}
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
    {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
    {{- end }}
    {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
    {{- end }}

You can see env and secretEnv values, which we will use to define env and secret env variables in our values file.

env:
  - name: SOME_GREAT_ENV
    value: "notasecretvariable"
secretEnv:
  - name: COOL_ENV
    secretKeyRefName: app
    secretKeyRefKey: first-password
  - name: SLIGHTLY_LESS_COOL_ENV
    secretKeyRefName: app
    secretKeyRefKey: second-password
  - name: CHILL_ENV
    secretKeyRefName: app
    secretKeyRefKey: third-password

As you can see, we added a simple env which is not a secret, and then used secretEnv to add a couple of variables that we have in our secret. secretKeyRefName in this case will be the name of the helm release, as our secrets template will create an object under that name. Next, we will just use the key that we entered into our secrets.dev.yaml to let Kubernetes know which env should be referenced to which secret. After that, we are ready to go for deployment.

Deploying with Skaffold

I won’t get into the whole flow with Github Actions, Helm charts, and Skaffold, but will show you how you can simply tell Skaffold to utilize Helm secrets automatically.

apiVersion: skaffold/v1
kind: Config
profiles:
  - name: dev
    build:
      artifacts:
        - image: our-repo/our-image
          docker:
            dockerfile: docker/Dockerfile
      tagPolicy:
        gitCommit: {}
    deploy:
      helm:
        releases:
          - name: app
            chartPath: helmchart
            namespace: app
            values:
              image: our-repo/our-image
            valuesFiles:
              - helmchart/values.dev.yaml
              - helmchart/secrets.dev.yaml
            useHelmSecrets: true
            wait: true
        flags:
          global: []
          upgrade: ["--debug", "--install", "--timeout", "6000s"]

A couple of important things here:

• You need to add useHelmSecrets: true so Skaffold knows that you want to utilize Helm secrets plugin and that it can apply the secrets automatically
• valuesFiles needs to contain both a path to values and secrets file in order for a build to succeed.
• Your Google IAM user must have KMS Encrypter role (if you want to encrypt the data), and KMS Decrypter role (used for CIs as they don’t need to encrypt the data, just decrypt).
• You would need to login to your GKE cluster before running Skaffold.

Afterward you would just call skaffold run -p dev and watch the tools do its work. If you want to edit, or add some value, you would simply run helm secrets edit helmchart/secrets.dev.yaml and it would open the decrypted file in your $EDITOR. You can add data, change passwords, etc, and save the file and it would automatically get encrypted so you can commit it to git and push it to your CI.

This sounds like a lot of steps to manage “such a simple thing” as secrets, but you only need to do this when you start a project and forget about it. If you have any questions - let me know in the comments.

One thing that differentiates Caddy from other webservers is its ask endpoint. Basically - it asks the endpoint if it is allowed to issue the certificate for the particular domain and only then issues the certificate for it. This is useful for us at MailerLite and our landing pages feature which allows users to easily create landing pages for them via drag and drop editor and they can use their custom domains for it. As we are also aiming for a more secure web, we are giving out the free certificates for each custom domain users set.

The company of our size has a lot of custom domains used in our landing pages, so we wanted to use some other storage rather than files for certificates. Fortunately, there is a plugin that utilizes Redis for that operation. It is really easy to add plugins to your Caddy installation with Caddy builder. You can find our implementation here. This Dockerfile also includes Cloudflare DNS plugin so you can use dns-01 verification instead of http-01 (useful when running certificates on your local network).

Back to the implementation, we utilize the following caddy configuration:

{
	email team@mailerlite.com
	on_demand_tls {
		ask {$CADDY_ASK_HOST}:{$CADDY_ASK_PORT}
	}

	storage redis {
		host          {$CADDY_CLUSTERING_REDIS_HOST}
		port          {$CADDY_CLUSTERING_REDIS_PORT}
		aes_key       {$CADDY_CLUSTERING_REDIS_AESKEY}
	}
}

:8080 {
    respond /healthz "200 ok"
}

:443 {
	tls {
		on_demand
	}
	reverse_proxy {$CADDY_LANDINGS_HOST}:{$CADDY_LANDINGS_PORT} {
		transport http {
			keepalive 30s
		}
	}
}

As you can see, we are utilizing env variables to set our ask endpoint, as well the other config, redis host, and finally, our microservice that servers landing pages. Our stack for landing services contains a couple of microservices - Caddy, Redis, NodeJS landings microservice, and Golang ask endpoint microservice. I won’t get into depth on how we deploy the apps to Kubernetes in the post, but in a nutshell, we are using Helm, Helm Secrets, Skaffold, and GitHub Actions.

To get a better overview of how the flow is, you can check it out in the following image:

Overall, we are really satisfied with how easy and performant Caddy is, and in version 2.2.0 you can even have native Prometheus integration so you could increase your observability in Grafana, for example.

Let me know in the comments if you have any questions about the implementation.

Posle puno pitanja i pojedinačnog objašnjavanja na Slacku Digitalne zajednice, rešio sam da napišem kratko uputstvo kako sam namestio da aplikacija ePorezi radi na macOSu sa sertifikatima MUPa.

Napomena: Sve ovo koristite na sopstvenu odgovornost. Modul je dobijen od PKS CA i nije prepravljan, već je u izvornom obliku u kome sam ga i dobio. Ovo bi sve trebalo da radi i za sertifikate koje je izdala PKS, ali nisam imao mogućnosti da proverim.

Uslovi da bi vam ovo proradilo:

• Lična karta izdata posle 18.8.2014. godine
• macOS
• modifikovana ePorezi aplikacija koju možete skinuti ovde
• Instalirana Java
• Library za middleware koji MUP koristi koji možete skinuti ovde

Instalirajte Javu, iskopirajte middleware u /usr/local/lib i prebacite modifikovanu verziju ePorezi za macOS u Applications folder. Na Catalini je potrebno da držite ctrl i pritisnete desni klik i open prilikom prvog otvaranja aplikacije zbog novih sigurnosnih pravila na ovoj iteraciji OSa.

To bi trebalo da bude to. Ubacite ličnu karticu i zakačite smart card čitač i trebalo bi sve da radi.

Moguće greške i problemi:

• Ako piše da čitač ili lična karta nisu povezani, proverite da li je čitač prepoznat od strane macOSa. Isto tako, proverite kada Vam je izdata lična karta jer datum mora biti posle 18.8.2014.
• Probajte da odete u /usr/local/lib i da otvorite libnstpkcs11.dylib sa desnim klikom dok držite ctrl i da pritisnite open.