Why though?#

In my previous post, I worked to set up Pangolin as a self-hosted alternative to something like Cloudflare tunnels. I got it working well enough, but something was nagging at me. We can see the old quote:

After a little digging (not a ton tbqh, but enough to start throwing yaml at the wall and seeing what sticks)

Shoulda done a little more digging, I would’ve realized that I couldn’t just fully declare the config. You do a bit of configuration with Pangolins blueprints feature, but that doesn’t get quite as deep as I’d like. There’s a lot of stuff which still needs to be manually configured on the VPS.

What next?#

I found the Awesome Tunneling repo, which is basically a ton of different solutions to the exact solution I’m trying to solve. So I went through a number of the solutions on that repo and decided on frp, which is stateless, can be configured from just the files I push with Ansible. So lets get started working on setting frp up.

Edit: We also don’t have to do things per domain other than DNS, which makes things a bit easier.

Redoing the Ansible side#

First things, first - I sshed to the VPS and manually stopped the Docker compose stack for Pangolin, and manually made a /opt/frp directory, just to make sure that it already existed.

git rm -r docker-compose/config/ will get rid of the old Pangolin files we don’t need anymore.

frp config#

The new Docker compose that we’ll push for frp is pretty simple:

docker-compose/docker-compose.yml

name: frp
services:
  frps:
    image: docker.io/snowdreamtech/frps:latest
    container_name: frps
    restart: unless-stopped
    environment:
      # frp tunnel token, injected by Ansible from a Gitea secret in the same
      # way we did for the Pangolin solution. We do this so that the secret is
      # never committed.
      FRP_TOKEN: "${FRP_TOKEN:?FRP_TOKEN must be set}"
    volumes:
      - ./frps.toml:/etc/frp/frps.toml:ro
    ports:
      - "7000:7000"  # frp control channel
      - "443:443"    # public HTTPS
      - "222:222"    # SSH for git SSH

For the VPS side of frp we’ll push the following:

docker-compose/frps.toml

bindPort = 7000  # control channel

auth.method = "token"
auth.token = "{{ .Envs.FRP_TOKEN }}"

allowPorts = [
  { single = 443 },  # https   > traefik.traefik.svc:443
  { single = 222 },  # git-ssh > gitea-ssh.gitea.svc:22
]

Gitea runner & Ansible#

We have to tweak our Gitea action to run the Ansible playbook, though the biggest change is really just swapping out Pangolin environment variables for the FRP variable:

.gitea/workflows/ansible.yaml

name: Run Ansible Playbook

on:
  push:
    branches:
      - main
  pull_request:
  workflow_dispatch:

jobs:
  ansible:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v7

      - name: Install packages
        run: |
          apt-get update
          apt-get install -y python3-pip python3-venv
          python3 -m pip install ansible
          ansible-galaxy collection install community.docker      # Docker tools
          ansible-galaxy role install -r ansible/requirements.yml # Docker installation

      - name: Configure SSH key
        env:
          SSH_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
          KNOWN_HOSTS_DATA: ${{ secrets.SSH_KNOWN_HOST }}
        run: |
          mkdir -p ~/.ssh
          echo "$SSH_KEY" > ~/.ssh/vps_deploy
          chmod 600 ~/.ssh/vps_deploy
          echo "$KNOWN_HOSTS_DATA" > ~/.ssh/known_hosts
          chmod 644 ~/.ssh/known_hosts

      - name: Update the inventory IP address
        run: |
          sed -i 's/VPS_HOST/${{ secrets.VPS_HOST }}/' ansible/inventory.ini

      - name: Run the playbook
        env:
          FRP_TOKEN: ${{ secrets.FRP_TOKEN }}
        run: |
          ansible-playbook -i ansible/inventory.ini ansible/playbook.yaml \
            --private-key ~/.ssh/vps_deploy \
            -e frp_token="$FRP_TOKEN"

And we’ll have to update the Ansible playbook as well:

ansible/playbook.yaml

- name: Deploy frps
  hosts: myhost
  become: true

  roles:
    - role: geerlingguy.docker

  vars:
    project_path: "/opt/frp"

    docker_edition: 'ce'
    docker_service_state: started
    docker_service_enabled: true
    docker_install_compose_plugin: true
    docker_compose_package: docker-compose-plugin
    docker_users:
      - "{{ ansible_user_id }}"


  tasks:
    - name: Ping my hosts
      ansible.builtin.ping:

    - name: Ensure target directory exists
      ansible.builtin.file:
        path: "{{ project_path }}"
        state: directory
        mode: '0755'

    - name: Copy compose stack to server
      ansible.builtin.copy:
        src: ../docker-compose/
        dest: "{{ project_path }}/"

    - name: Deploy or update the frps stack
      community.docker.docker_compose_v2:
        project_src: "{{ project_path }}"
        state: present
        pull: always
      environment:
        FRP_TOKEN: "{{ frp_token }}"

I then generated FRP_TOKEN with openssl rand -base64 32 and saved it as a secret in my repo, so that the action can reference it.

I committed & pushed it, the actions completed and I confirmed that frps is now running on the VPS.

Cluster-side#

Okay, so we also have to set up the actual cluster-side things. Here’s the deployment itself:

frp/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frpc
  namespace: frp
  labels:
    app: frpc
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frpc
  template:
    metadata:
      labels:
        app: frpc
    spec:
      containers:
        - name: frpc
          image: docker.io/snowdreamtech/frpc:latest
          env:
            - name: FRP_TOKEN
              valueFrom:
                secretKeyRef:
                  name: frp-secrets
                  key: FRP_TOKEN
            - name: VPS_IP
              valueFrom:
                secretKeyRef:
                  name: frp-secrets
                  key: VPS_IP
          volumeMounts:
            - name: config
              mountPath: /etc/frp
      volumes:
        - name: config
          configMap:
            name: frpc-config

The configmap for configuring frp:

frp/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: frpc-config
  namespace: frp
data:
  frpc.toml: |
    serverAddr = "{{ .Envs.VPS_IP }}"
    serverPort = 7000

    auth.method = "token"
    auth.token = "{{ .Envs.FRP_TOKEN }}"

    [[proxies]]
    name = "https"
    type = "tcp"
    localIP = "traefik.traefik.svc.cluster.local"
    localPort = 443
    remotePort = 443
    transport.proxyProtocolVersion = "v2"

    [[proxies]]
    name = "git-ssh"
    type = "tcp"
    localIP = "gitea-ssh.gitea.svc.cluster.local"
    localPort = 222
    remotePort = 222

frp/secret-store.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eso-auth
  namespace: frp
---
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: openbao-backend
  namespace: frp
spec:
  provider:
    vault:
      server: "http://192.168.1.23:8200"
      version: "v2"
      path: "secret"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "eso-role"
          serviceAccountRef:
            name: "eso-auth"
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: frp-token-sync
  namespace: frp
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: openbao-backend
    kind: SecretStore
  target:
    name: frp-secrets
    creationPolicy: Owner
  data:
    - secretKey: FRP_TOKEN
      remoteRef:
        key: frp
        property: FRP_TOKEN

    - secretKey: VPS_IP
      remoteRef:
        key: frp
        property: VPS_IP

We also have to trust the connections being proxied from frp but editing the Traefik config:

infra/traefik/values.yaml

[...]
ports:
  websecure:
    proxyProtocol:
      trustedIPs:
        - "10.244.0.0/16"
[...]

Lastly, get ArgoCD managing it from the cluster side:

argocd/apps/frp.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: frp
  namespace: argocd
spec:
  project: default
  source:
    repoURL: ssh://<my-repo>.git
    targetRevision: main
    path: frp
  destination:
    server: https://kubernetes.default.svc
    namespace: frp
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Okay, I think we got everything.

404?#

Well, I think it’s working in general (at least it all deploys), but we’re getting a 404 when trying to get to the site. But turns out the Pangolin solution had us get rid of TLS cert generation and only use an internal ingress. I just applied my old regular ingress and it seems to work as hoped.

For the future, I also need to forward port 80 for the Let’s Encrypt HTTP-01 Challenge.

Future stuff (again)#

Time to leave Anubis in my thoughts, we’ll see.

I checked off the “make sure all of the configs that are VPS side get saved” part of my previous post, so we’re good on that.