I’ve continued to play around with the K3S Beta cluster from Civo Cloud (you can request to join the beta), the aim was to also try out the new Traefik V2 which was released some time ago.
In this post we will:
- Update the default traefik install on k3s to v2.0.1
- Secure the k3s install
- Create and secure the ingress to the traefik dashboard
- Enable Let's Encrypt certs for traefik
- Some thoughts after the journey
Let’s start.
Update Traefik to V2
By default the k3s cluster comes with Traefik v1.7. It’s a sane default, nothing wrong with it. As I am using a k3s cluster on a beta service, seemed like the perfect opportunity to play around with Traefik v2, it gives me the opportunity of exploring the new configuration, getting used to serving k8s services, as it matures, I’ll be ready to switch my production cluster to it.
The documentation around v2 is missing a lot of information, I’ve had to jump around quite a lot to understand how to get it going. (The Traefik documentation is open source, I could definetely contribute to it and it has been on my mind.) Through the post I’ll link to the relevant pages.
Delete the existing traefik deployment and service
The existing Traefik deployment and service are under the name traefik, we just need to delete them:
kubectl delete deployment traefik -n kube-system
kubectl delete service traefik -n kube-system
Deploy Traefik v2
There is an official user guide that helps you setup Traefik with Let’s Encrypt enabled on minikube, 75% of it is totally usable in our setup as well. We won’t enable Let’s Encrypt from the beginning, we will need to take some extra steps to do so.
Let’s create the CRD definitions, I’m using the kube-system namespace, not the default one as it is in the guide:
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressroutes.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRoute
plural: ingressroutes
singular: ingressroute
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressroutetcps.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRouteTCP
plural: ingressroutetcps
singular: ingressroutetcp
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: middlewares.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: Middleware
plural: middlewares
singular: middleware
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: tlsoptions.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: TLSOption
plural: tlsoptions
singular: tlsoption
scope: Namespaced
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- traefik.containo.us
resources:
- middlewares
verbs:
- get
- list
- watch
- apiGroups:
- traefik.containo.us
resources:
- ingressroutes
verbs:
- get
- list
- watch
- apiGroups:
- traefik.containo.us
resources:
- ingressroutetcps
verbs:
- get
- list
- watch
- apiGroups:
- traefik.containo.us
resources:
- tlsoptions
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
name: traefik-ingress-controller
namespace: kube-system
Now we’re ready to create the deployment file together with the ServiceAccount used by Traefik:
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: kube-system
name: traefik-ingress-controller
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
namespace: kube-system
name: traefik
labels:
app: traefik
spec:
replicas: 1
selector:
matchLabels:
app: traefik
template:
metadata:
labels:
app: traefik
spec:
serviceAccountName: traefik-ingress-controller
containers:
- name: traefik
image: traefik:v2.0
args:
- --log.level=INFO
- --api.dashboard=true
- --api.insecure=true
- --accesslog
- --global.sendanonymoususage
- --entrypoints.web.Address=:80
- --entrypoints.websecure.Address=:443
- --providers.kubernetescrd
ports:
- name: web
containerPort: 80
- name: websecure
containerPort: 443
- name: admin
containerPort: 8080
Explanation on some of the options (args) passed, the full list of options can be find on the CLI Reference Page:
--log.level=INFO - By tailing the logs, we will be able to see more information--api.dashboard - Enable the Traefik Dashboard--api.insecure - Enable access to the Dashboard, we will secure this with basic auth later on--acceslog - Enable access logs--global.sendanonymoususage - It’s a test setup, the guys from Traefik maybe can find some anonymous data useful. Details of the collected data can be found here.
Last piece of the puzzle is the service:
apiVersion: v1
kind: Service
metadata:
name: traefik
namespace: kube-system
spec:
ports:
- protocol: TCP
name: http
port: 80
nodePort:
- protocol: TCP
name: admin
port: 8080
- protocol: TCP
name: https
port: 443
nodePort:
selector:
app: traefik
type: LoadBalancer
Let’s deploy what we have right now:
kubectl apply -f traefik-crd.yaml
kubectl apply -f traefik-deployment.yaml
kubectl apply -f traefik-svc.yaml
In a couple of seconds the pod should start and the service should be able to serve content. You can check all of these with some of the following commands:
kubectl get pods -n kube-system
kubectl logs traefik-<random-hash> -n kube-system
kubectl get svc -n kube-system
Secure the Civo Cloud k3s install
Civo Cloud has almost all ports open by default, this has been discussed in the #kube100 Slack channel, it will be looked at and secured fully, until then we just have to do another manual approach. Each k3s cluster deployed has a firewall created for it, open your Civo Cloud dashboard, click on Kubernetes, click on the name of your cluster and you’ll see Firewall: Kubernetes cluster: <name of your cluster>, click on it and you’ll be presented with the page where you can modify the rules.
Keep in mind this is a test cluster, you shouldn’t have any valuable information stored inside of it, if you do, it’s totally up to you to keep it secured.
These are the ports I’ve kept open:
ICMP N/A 0.0.0.0/0 Ping/Traceroute - Just so I’m able to ping the k3s nodes
TCP 443 0.0.0.0/0 HTTPS - Access to the https endpoint for Traefik
TCP 80 0.0.0.0/0 HTTP - Access to the http endpoint for Traefik, Let’s encrypt needs access to this so it can do the HTTP challenge
TCP 6443 0.0.0.0/0 K3S API - Access to the secure k3s API
If you want to secure the cluster even more, feel free to use your own public IP instead of 0.0.0.0/0, if the IP is dynamic, don’t forget to update it.
Create and secure the access to the Traefik Dashboard
The k3s nodes all have public IP addresses, Traefik runs the dashboard on port 8080, with the service created, by default it means anyone who opens one of the public IPs on port 8080 can have unauthenticated access to the dashboard. We mitigated this risk by not allowing port 8080 to be open to these public IPs. In order to go around it we will create an IngressRule on a specific host that will route all of our requests to the Traefik Dashboard.
Point domain to Public IP
The first step is to make sure you point a domain to one of the public IP addresses. I’ve chosen the same public IP where the kubectl context is configured, you can see this with the following command:
civo kubernetes show <name of your cluster> | grep API
Simply create an A record that points to the public IP Address. We also need this step for the Let’s Encrypt step.
Create the IngressRule
We will set up an IngressRule that will route any requests coming in on our domain name to the Traefik dashboard, for security we will add basic auth.
The basic auth documentation page explain how it works in the new Traefik v2, the step I’ve missed was on how to craete the basic auth. With a bit of exploring, it turns out it’s the same as in the old version, you create the htpassd file, create a secret from it and you present it to Traefik:
htpasswd -c ./users <your username>
kubectl create secret generic mysecret --from-file users --namespace=kube-system
Time to create the IngressRoute and Middleware:
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: test-auth
namespace: kube-system
spec:
basicAuth:
secret: mysecret
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: traefik-dashboard
namespace: kube-system
spec:
entryPoints:
- web
routes:
- match: Host(`<your-domain.fqdn>`)
kind: Rule
services:
- name: traefik
port: 8080
middlewares:
- name: test-auth
Let’s apply it:
kubectl apply -f traefik-ingress-dashboard.yaml
There we go, if you visit http://your-domain.fqdn you should be presented with basic auth, use your username and password and voila, the new great Traefik Dashboard will appear! Feel free to get used to the new layout and all the new information presented (I do miss the statistics, if I’m honest).
Enable Let’s Encrypt
Traefik’s one big and great feature is that it can handle SSL Certs for you automagically, let’s use it.
Configuration to Traefik is passed via the args field in the configuration file, we modify the previously created file and we’ll make sure it looks like this:
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: kube-system
name: traefik-ingress-controller
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
namespace: kube-system
name: traefik
labels:
app: traefik
spec:
replicas: 1
selector:
matchLabels:
app: traefik
template:
metadata:
labels:
app: traefik
spec:
serviceAccountName: traefik-ingress-controller
containers:
- name: traefik
image: traefik:v2.0
args:
- --log.level=INFO
- --api.dashboard=true
- --api.insecure=true
- --accesslog
- --global.sendanonymoususage
- --entrypoints.web.Address=:80
- --entrypoints.websecure.Address=:443
- --providers.kubernetescrd
- --certificatesresolvers.default.acme.tlschallenge
- --certificatesresolvers.default.acme.email=<your email address@fqdn.com>
ports:
- name: web
containerPort: 80
- name: websecure
containerPort: 443
- name: admin
containerPort: 8080
The two new fields:
--certificatesresolvers.default.acme.tlschallenge - Enable TLS Challenge
--certificatesresolvers.default.acme.email=<your email [email protected] - The email address used to register the SSL Certs, used only for communication
Save the file and apply it:
kubectl apply -f traefik-deployment.yaml
Traefik will restart the pods, it’s fast but until then we can simply update our IngressRoute to make use of the SSL enabled. Open the previously saved traefik-ingress-dashboard.yaml and update it:
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: test-auth
namespace: kube-system
spec:
basicAuth:
secret: mysecret
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: traefik-dashboard
namespace: kube-system
spec:
entryPoints:
- web
- websecure
routes:
- match: Host(`<your-domain.fqdn>`)
kind: Rule
services:
- name: traefik
port: 8080
middlewares:
- name: test-auth
The difference between the two IngressRoutes are:
entryPoints - Traefik has 3 entrypoints, http, https and admin. For https, port 443, we used the name websecure, you can see this in the deployment definition.tls.certResolver - You can have multiple certResolvers, which is great, imagine using some protected, company owned cert provider but you also want to have public endpoints. The name of the certResolver is default, as defined in the --certificatesresolvers.default.acme.tlschallenge option in the deployments.
Let’s apply:
kubectl apply -f traefik-ingress-deployment.yaml
Once you go to https://your-domain.fqdn you should be presented with the basic auth, once you log in, the dashboard should present its self behind SSL in all of its glory! Congratulations, we have achieved something great today!
Some thoughts after the journey
Civo Cloud
The beta k3s cluster presented by Civo Cloud had no issues, everything worked correctly. The configuration around it is not optimal at this point, but it’s a beta feature, we’re helping by testing the cluster its self. Network security around it will be addressed, at least that’s what I’ve seen from the discussions in slack. I’d like to see an option where I can simply disable the public IPs and just have a load balancer in front of it. Locked down security groups.
I want to work on some sort of automation on configuring the firewalls once they get created, I’ve seen some write up about using terraform with Civo Cloud, which is great news!
Civo Cloud does have the option for Load Balancers, I just didn’t look in to it long enough on how can I use one with the k3s cluster.
Traefik
I am very excited about Traefik v2 for the simple fact that I’ve been using their product for two years now, I’ve seen it mature, it works in production, it’s just getting better. Especilly with the new v2 feature of Observability, which I do want to look in to more, maybe that will be another post down the line. Then there is Maesh, the simple service mesh!
Documentation for v2 is scarce, though as mentioned, I could be contributing, so it’s on me as well. I’m very glad that I got v2 working on a k3s install, from here on it’s just me playing around with all the new options and I’m really curious to see what metrics are exposed for Prometheus, which is the next big thing I want to look at, especially that there are no stats in the new Traefik v2 Dashboard.
