Gateway API for Teleport K8S

If you've been reading some of my mutterings for a bit you know I run Teleport and love it. Over the past couple of years I've been running Teleport within my OpenStack cloud and while that's been great, I've wanted to push Teleport into my Kubernetes cluster, however, I've been reluctant to do it because VM was working so well. That said, with the development of Genestack we've been using the Gateway API and I found a desire to tinker, a need to learn. So I did what any self-respecting nerd would, and decided to break an otherwise stable and perfectly functioning service so that I can redeploy in a new and different way.

Background

In Kubernetes the Gateway API has been released and introduces some much needed functionality that carries it beyond where the basic Ingress provider could. Read more about the Gateway API here. Within Genestack we're using the NGINX Gateway Fabric (by default) to provide access to our API services. This system scalable and able to be used in a lot different ways across the Kubernetes cluster environment.

Teleport provides a Kubernetes deployment model which makes several assumptions about how and where systems are deployed. That said, the core functionality of Teleport is able to be serviced on any Kubernetes environment so long as the basic access features are able to satisfied. This is where the Gateway API comes in, for the purpose of my environment, Teleport will be deployed in the teleport-cluster namespace via helm and the Gateway API will service as our "load balancer."

In the following examples you must change your.domain.tld to the domain you wish to use within the teleport cluster.

System Overview

At the time of this writing here's the overview of my environment

Kubernetes v1.28.6
NGINX Gateway Fabric 1.4.0
- The NGINX Gateway Fabric deployment was done with experimental features enabled
- Service deployment is using apiVersion v1alpha2
Teleport 16.4.x

System Setup

I'm not going into the actual deployment of the gateway services or Teleport itself, both of those processes are well documented upstream.

That said, I will share my teleports helm values file because getting this all working is Kind of a nightmare and I wouldn't want anyone to have to go through this discovery adventure...

clusterName: teleport.your.domain.tld
chartMode: standalone
proxyListenerMode: multiplex
#proxyProtocol: "off"
acme: false
publicAddr:
  - teleport.your.domain.tld:443
authentication:
  type: github
  localAuth: true
proxy:
  highAvailability:
    replicaCount: 1
highAvailability:
  replicaCount: 1
  podDisruptionBudget:
    enabled: true
  certManager:
    enabled: true
    issuerName: "letsencrypt-production"
    addCommonName: true
    addPublicAddrs: true
ingress:
  enabled: false
log:
  level: INFO

High level this deployment config is making some assumptions that cert-manger is deployed within the Kubernetes cluster and will be using letsencrypt to issue certificates. This is also setup to use github authentication to users. Change these values to suit your needs.

Gateway Setup

This post is all about using the Gateway API for our services in conjunction with Genestack; effectively combining our Kubernetes and OpenStack clusters to serve a greater purpose. So this post is mostly covering the deployment of the routes and listeners I use within those systems to support the workload, Teleport.

Routing

The first setup is to setup our routes. The routes allow Teleport to terminate TLS within the pods on port 443 and ensure that any connection going to the proxy service are redirected to a TLS connection. To do this achieve the goals, create a simple manifest named teleport-tlsroute.yaml and apply it to the environment.

💡

Notice: the apiVersion is set to v1alpha2. Teleport is running in the teleport-cluster namespace and our gateway services named flex-gateway are running in the nginx-gateway namespace.

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: custom-teleport-cluster-http-gateway-route
  namespace: teleport-cluster
spec:
  parentRefs:
  - name: flex-gateway
    sectionName: teleport-cluster-http
    namespace: nginx-gateway
  hostnames:
  - "teleport.your.domain.tld"
  rules:
    - filters:
      - type: RequestRedirect
        requestRedirect:
          scheme: https
          statusCode: 301
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
  name: custom-teleport-cluster-gateway-route
  namespace: teleport-cluster
spec:
  parentRefs:
  - name: flex-gateway
    sectionName: teleport-cluster-https
    namespace: nginx-gateway
  hostnames:
  - "teleport.your.domain.tld"
  rules:
    - backendRefs:
      - name: teleport-cluster
        port: 443
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
  name: custom-teleport-cluster-gateway-wild-route
  namespace: teleport-cluster
spec:
  parentRefs:
  - name: flex-gateway
    sectionName: teleport-cluster-wild-https
    namespace: nginx-gateway
  hostnames:
  - "*.teleport.your.domain.tld"
  rules:
    - backendRefs:
      - name: teleport-cluster
        port: 443

Example manifest

Certificate Issuer

Here's the certificate issuer I'm using, this is made so that the Gateway API can be used to validate http and dns certificate requests. This file is saved at teleport-issuer-gateway.yaml.

---
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: teleport-cluster
type: Opaque
stringData:
  api-token: SUPER_SECRETE_TOKEN_FROM_CLOUDFLARE

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-production
  namespace: teleport-cluster
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: your-name@your.domain.tld
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-production
    # Enable the HTTP-01 challenge provider
    solvers:
      - http01:
          gatewayHTTPRoute:
            parentRefs:
              - name: flex-gateway
                namespace: nginx-gateway
                kind: Gateway
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token-secret
              key: api-token

Example teleport-issuer-gateway.yaml file

Take note that this is using the CloudFlare API to validate DNS01 challenges from LetsEncrypt. If you do not use LetsEncrypt or do not use CloudFlare, be sure to adjust this file according to your environment.

To apply the manifest use a simple kubectl command like so.

kubectl apply -f teleport-tlsroute.yaml teleport-issuer-gateway.yaml

Listener Patches

Next we patch the listeners to include the new TLS protocol and route. To run this patch create a new patch file named teleport-tls-listener.json with the following content.

[
    {
        "op": "add",
        "path": "/spec/listeners/-",
        "value": {
            "name": "teleport-cluster-http",
            "port": 80,
            "protocol": "HTTP",
            "hostname": "teleport.your.domain.tld",
            "allowedRoutes": {
                "namespaces": {
                    "from": "All"
                }
            }
        }
    },
    {
        "op": "add",
        "path": "/spec/listeners/-",
        "value": {
            "name": "teleport-cluster-https",
            "port": 443,
            "protocol": "TLS",
            "hostname": "teleport.your.domain.tld",
            "allowedRoutes": {
                "namespaces": {
                    "from": "All"
                }
            },
            "tls": {
                "mode": "Passthrough"
            }
        }
    },
    {
        "op": "add",
        "path": "/spec/listeners/-",
        "value": {
            "name": "teleport-cluster-wild-http",
            "port": 80,
            "protocol": "HTTP",
            "hostname": "*.teleport.your.domain.tld",
            "allowedRoutes": {
                "namespaces": {
                    "from": "All"
                }
            }
        }
    },
    {
        "op": "add",
        "path": "/spec/listeners/-",
        "value": {
            "name": "teleport-cluster-wild-https",
            "port": 443,
            "protocol": "TLS",
            "hostname": "*.teleport.your.domain.tld",
            "allowedRoutes": {
                "namespaces": {
                    "from": "All"
                }
            },
            "tls": {
                "mode": "Passthrough"
            }
        }
    }
]

Example Patch file teleport-tls-listener.json

Apply the patch to the running gateway instance

kubectl patch -n nginx-gateway gateway flex-gateway --type=json --patch-file=teleport-tls-listener.json

Validate Functionality

Once the patch is applied run a describe on the gateway service to validate functionality.

kubectl -n nginx-gateway describe gateways.gateway.networking.k8s.io flex-gateway

The output will show you all of the listeners running, the output your looking for will be something like so.

    Attached Routes:  1
    Conditions:
      Last Transition Time:  2024-08-24T16:43:53Z
      Message:               Listener is accepted
      Observed Generation:   18
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2024-08-24T16:43:53Z
      Message:               Listener is programmed
      Observed Generation:   18
      Reason:                Programmed
      Status:                True
      Type:                  Programmed
      Last Transition Time:  2024-08-24T16:43:53Z
      Message:               All references are resolved
      Observed Generation:   18
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
      Last Transition Time:  2024-08-24T16:43:53Z
      Message:               No conflicts
      Observed Generation:   18
      Reason:                NoConflicts
      Status:                False
      Type:                  Conflicted
    Name:                    teleport-cluster-https
    Supported Kinds:
      Group:          gateway.networking.k8s.io
      Kind:           TLSRoute

Example Describe for a functioning TLSRoute

Accessing Teleport

Assuming everything is working normally, you should now be able to access Teleport on the domain defined, in the example that would be your.domain.tld.

There's absolutely no need for an load balancer (AWS or otherwise) now that we have the power of the Gateway API.

Example web access

From here, simply access Teleport normally. If you're on the setup process, proceed to Step 2/2 to generate some local users.

Because Teleport is attempting to use a Proxy, you will need to set the environment variable TELEPORT_TLS_ROUTING_CONN_UPGRADE with a value of true. This environment variable can be defined on the command line, or in your terminal environment RC file: .bashrc, .zshrc, etc... This export can also be used on the command line at auth runtime, like so:

💡

I can likely get rid of this environment just as soon as I can figure out how to ensure that the PROXY headers are passed through the TLSRoute. Until then, I'm setting the option accordingly.

# Running the command
TELEPORT_TLS_ROUTING_CONN_UPGRADE=true tsh login --proxy teleport.your.domain.tld

# Command Output
If browser window does not open automatically, open it by clicking on the link:
 http://127.0.0.1:49398/2147e66f-5ee3-4e2b-8e2c-ddea29967373
> Profile URL:        https://teleport.your.domain.tld:443
  Logged in as:       YOUR_SPECIAL_USER
  Cluster:            teleport.your.domain.tld
  Roles:              access
  Logins:             YOUR_SPECIAL_USER
  Kubernetes:         enabled
  Valid until:        2024-08-26 05:00:04 -0500 CDT [valid for 12h0m0s]
  Extensions:         login-ip, permit-agent-forwarding, permit-port-forwarding, permit-pty, private-key-policy

Example environment login settings

That's is, That's all

Simply blog post on how I use the Kubernetes Gateway API to run my Teleport environment, now running on the same cluster as my OpenStack Cloud, all powered by Genestack.