Plex on Kubernetes

Plex on Kubernetes

By in Kubernetes, Linux

Giving Plex on Kubernetes a Real LAN IP with br0, Multus, and GitOps

I moved Plex from Docker Compose to Kubernetes for the same reason I moved a lot of my home-lab services: I wanted the running system to match the system I had in Git. Plex already worked in Compose. The migration was not about escaping a broken setup. It was about making Plex part of the same GitOps-managed platform as the rest of the lab without taking away the LAN identity that Plex clients already knew.

That second part was the constraint that shaped the whole design. Plex is not only a web app. It is a media server that local clients discover, remember, and connect to directly. If I had hidden it behind only a normal Kubernetes Service or HTTP ingress, Kubernetes would have looked tidy, but the user experience on the LAN could have become weird.

So I gave the Plex pod a real LAN address. Cilium still owns normal Kubernetes pod networking. Multus adds a second interface to the Plex pod. The bridge CNI attaches that interface to br0. Plex advertises the address on that interface, and ingress can still route to it for browser access.

The high-level shape is:

Kubernetes pod network
        |
   Cilium interface
        |
Plex pod
        |
   lan0 from Multus
        |
host bridge br0
        |
home LAN

For the example in this repo, the pod gets 192.168.1.50/24 on lan0, and the shared internal ingress VIP is a separate address. That split is important: one address belongs to Plex itself, and the other belongs to the ingress path.

Why I Did Not Use Only a Service

Most Kubernetes workloads are happy behind a ClusterIP, Ingress, or LoadBalancer Service. Plex is awkward because it behaves like both a web application and a LAN appliance. Local clients expect to find it, connect to a stable address, and sometimes use traffic patterns that do not fit cleanly into an HTTP-only route.

I considered the usual Kubernetes options:

  • hostNetwork: true, which would make the pod share the node’s network
    namespace.
  • A Cilium or MetalLB LoadBalancer VIP, which would advertise a Service
    address on the LAN.
  • A secondary pod interface with Multus, where Plex gets its own LAN address.

The secondary interface was the cleanest fit. Plex keeps an actual LAN IP inside its own pod network namespace, while Cilium continues to own the normal Kubernetes interface and default route.

That distinction matters:

  • A Service VIP is a virtual load-balancer address.
  • A Multus LAN IP is assigned to the workload itself.
  • Plex discovery and client behavior are easier to reason about when Plex
    advertises the address that is actually on its own interface.
  • Ordinary web apps should usually stay behind ClusterIP, Gateway, Ingress, or
    a LoadBalancer VIP instead of getting direct LAN presence.

What I Needed Before Starting

Before I touched the live Plex process, I wanted the prerequisites to be boring. This is the part of the migration where I would rather be slow than clever, because a LAN IP conflict or a broken host bridge turns a clean app migration into a network incident.

The checklist was:

  • A Linux bridge on the LAN. In this example, the bridge is br0.
  • A reserved, unused LAN address for Plex. In this example, that address is
    192.168.1.50.
  • A Kubernetes node selector or scheduling constraint that pins Plex to the node
    with br0, the media paths, and the device mounts.
  • Cilium configured so Multus can coexist with it.
  • Multus installed on every node that might run Plex.
  • The bridge CNI plugin installed on the node.
  • The static Plex IP excluded from DHCP or reserved in the network source of
    truth.
  • A rollback path for the old Compose instance.

That last point matters because the first migration reused the same Plex config directory. I did not want Kubernetes Plex and Compose Plex writing to the same state. The rollback command was intentionally simple:

docker compose -f /opt/plex/docker-compose.yml up -d

But that command only belongs after the Kubernetes pod has been scaled down and the LAN address has been released.

The Network Model

The deployment uses both Cilium and Multus, but they have different jobs.

Cilium remains the primary CNI. The Plex pod still gets its normal Kubernetes interface, DNS behavior, service routing, and cluster reachability from Cilium. Multus is installed as the shim that lets a pod request extra interfaces. The bridge CNI plugin does the actual attachment to br0.

The Plex-specific NetworkAttachmentDefinition is intentionally small:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: plex-br0
  labels:
    app.kubernetes.io/name: plex
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "type": "bridge",
      "bridge": "br0",
      "ipam": {
        "type": "static",
        "addresses": [
          {
            "address": "192.168.1.50/24"
          }
        ]
      }
    }

There is no gateway in that attachment. I wanted the secondary interface to provide LAN identity, not steal the pod’s default route from Cilium.

The Deployment asks for the attachment with the Multus pod annotation:

template:
  metadata:
    annotations:
      k8s.v1.cni.cncf.io/networks: '[{"name":"plex-br0","interface":"lan0"}]'

Naming the interface lan0 is mostly for sanity. When I validate the pod, I can look for lan0 instead of guessing whether the extra interface became net1, eth1, or something else.

The same Deployment pins Plex to the node with the bridge, media paths, and device mounts:

spec:
  nodeSelector:
    kubernetes.io/hostname: k8s-node-1

That pin is part of the design. This version of the migration reuses local host paths and a node-local LAN bridge, so it is not pretending to be a portable multi-node Plex deployment.

What br0 Has to Provide

The Kubernetes manifests assume the node already has a bridge named br0 that is connected to the physical LAN. In my lab that bridge already existed because other host services and containers used it. If I were starting from a plain Linux server, I would create the bridge first and move the host’s LAN address onto the bridge instead of leaving it on the physical NIC.

The target shape is:

physical NIC, for example eno1
        |
      br0
        |
host IP, for example 192.168.1.10/24

A minimal Netplan-style example looks like this:

network:
  version: 2
  ethernets:
    eno1:
      dhcp4: false
      dhcp6: false
  bridges:
    br0:
      interfaces:
        - eno1
      addresses:
        - 192.168.1.10/24
      routes:
        - to: default
          via: 192.168.1.1
      nameservers:
        addresses:
          - 192.168.1.1
      parameters:
        stp: false
        forward-delay: 0

For a DHCP-based server, the bridge can use dhcp4: true. The important point is that the host’s LAN presence belongs to br0, because the Plex pod’s secondary interface will be attached there.

I would only apply bridge changes from an attended console or an out-of-band management path. A bad bridge config can disconnect SSH. After changing it, I check the basics before scheduling any LAN-present workload:

ip -br link show br0
ip -br addr show br0
bridge link
ip route

Then I confirm the node can still reach the LAN gateway, DNS, and the Kubernetes API.

Installing the Bridge CNI Plugin

Multus does not attach the interface to br0 by itself. It reads the NetworkAttachmentDefinition and dispatches the request to the CNI plugin named there. In this design, that plugin is bridge, so the node needs the bridge CNI binary installed, normally under /opt/cni/bin/bridge.

Before cutover, I check the selected node directly:

ssh k8s-node-1 'test -x /opt/cni/bin/bridge && ls -l /opt/cni/bin/bridge'
ssh k8s-node-1 'ls -1 /opt/cni/bin | sed -n "1,40p"'

If the binary is missing, I install the standard CNI plugins through the node bootstrap path. I do not work around a missing bridge plugin by switching Plex to hostNetwork. That would change the security model and lose the separate pod-owned LAN identity that this design is trying to preserve.

Making Cilium and Multus Coexist

Because Cilium is still the primary network, it needs to leave room for Multus. The key Cilium Helm value is:

values:
  cni:
    exclusive: false

That prevents Cilium from taking exclusive ownership of the host CNI config in a way that would break the Multus shim.

In this cluster, Cilium is also pinned to the LAN-facing bridge:

values:
  devices: br0
  nodePort:
    directRoutingDevice: br0

That is a single-cluster assumption. If the nodes in a different cluster do not all expose the LAN through the same bridge name, I would not copy that blindly. I would either standardize the bridge name or use a device selection policy that matches the actual node fleet.

The Multus platform package installs the NetworkAttachmentDefinition CRD, RBAC, a multus ServiceAccount, and the kube-multus-ds DaemonSet in kube-system. The DaemonSet writes a shim config that delegates the primary pod network to Cilium:

{
  "cniVersion": "0.3.1",
  "name": "multus-cni-network",
  "type": "multus-shim",
  "logLevel": "verbose",
  "logToStderr": true,
  "clusterNetwork": "/host/etc/cni/net.d/05-cilium.conflist"
}

Before reconciling the live platform, I render the packages:

kubectl kustomize config/kubernetes/platform/multus
kubectl kustomize config/kubernetes/platform/live

After GitOps applies them, I check that Multus is actually ready:

kubectl -n kube-system get ds kube-multus-ds
kubectl -n kube-system get pods -l app.kubernetes.io/name=multus -o wide
kubectl get crd network-attachment-definitions.k8s.cni.cncf.io

Preserving Plex Behavior

The migration reused the behavior that mattered from the Docker Compose setup. Plex still sees its existing config and media paths, and it still advertises the LAN address clients should use:

env:
  - name: ADVERTISE_IP
    value: http://192.168.1.50:32400/

The Deployment exposes the known Plex ports in the container spec:

  • TCP 32400
  • TCP 8324
  • TCP 32469
  • UDP 1900
  • UDP 5353
  • UDP 32410 through 32414

The image tracks the LinuxServer Plex image tag but pins the resolved digest in Git:

image: lscr.io/linuxserver/plex:latest@sha256:b785bdd60e781662f16e0526a6b54c07856739df95ab558a674a3c084dbde423

The pod also mounts the same host data layout the Compose container used:

  • /opt/plex/config mounted at /config
  • media libraries under /srv/media
  • photo libraries under /srv/photos
  • /dev/dvb
  • /dev/shm backed by a memory emptyDir

That is not a portable, multi-node Plex design. It pins Plex to the node with the media, devices, and br0 bridge. I accepted that tradeoff because the goal was a low-risk migration, not a storage redesign.

GitOps Layout

The manifests are split between platform and app packages.

The platform side includes Cilium and Multus:

  • config/kubernetes/platform/cilium/helmrelease.yaml
  • config/kubernetes/platform/multus/multus.yaml
  • config/kubernetes/platform/multus/kustomization.yaml
  • config/kubernetes/platform/live/kustomization.yaml

The app side includes Plex:

  • config/kubernetes/apps/plex/network-attachment-definition.yaml
  • config/kubernetes/apps/plex/deployment.yaml
  • config/kubernetes/apps/plex/service.yaml
  • config/kubernetes/apps/plex/endpointslice.yaml
  • config/kubernetes/apps/plex/dns-records.yaml
  • config/kubernetes/apps/plex/ingress.yaml
  • config/kubernetes/apps/plex/ingress-public-cloudflare.yaml
  • config/kubernetes/apps/plex/kustomization.yaml

The live app kustomization includes the Plex package, and Flux reconciles the desired state from Git. That matters during rollout and rollback because the manifests are not a one-off kubectl apply artifact. They are the source of truth.

Connecting Ingress to the LAN IP

Once Plex has a LAN address on the pod itself, ingress can target that address through a selectorless Service and a manual EndpointSlice.

The Service is just a stable Kubernetes object for ingress to reference:

apiVersion: v1
kind: Service
metadata:
  name: plex-lan
spec:
  type: ClusterIP
  ports:
    - name: https
      protocol: TCP
      port: 32400
      targetPort: 32400

The EndpointSlice points at the pod’s LAN address:

apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: plex-lan
  labels:
    kubernetes.io/service-name: plex-lan
addressType: IPv4
ports:
  - name: https
    protocol: TCP
    port: 32400
endpoints:
  - addresses:
      - 192.168.1.50
    conditions:
      ready: true

That gives the ingress controllers a normal Kubernetes backend while Plex remains a LAN-resident endpoint.

In this repo, the internal ingress uses the internal nginx class and proxies to plex-lan on port 32400. The public Cloudflare Tunnel ingress points to the same service. I kept public exposure separate from LAN presence: Plex does not need a direct router port forward just because it has a LAN IP.

DNS

I split DNS the same way I split networking.

The LAN name points directly at the Plex pod’s LAN IP:

spec:
  name: plex.lan.example.com
  recordType: A
  address: 192.168.1.50

The normal internal app name points at the shared internal ingress VIP:

spec:
  name: plex.example.com
  recordType: A
  address: 192.168.1.20

That gives me two intentional paths:

  • plex.lan.example.com is direct-to-Plex on the LAN.
  • plex.example.com is the ingress path.

Public DNS is owned by the Cloudflare Tunnel ingress controller, not by the internal DNS record resources. During cutover, the public route moved from the old tunnel target to the Kubernetes-managed tunnel target, while the internal records kept the split-horizon LAN behavior.

Cutover

I treated cutover as a single-writer problem. The old Compose container and the new Kubernetes pod must not run at the same time because they would share the same config path and the same LAN address.

The sequence I used is deliberately conservative.

First, render the platform and app packages:

kubectl kustomize config/kubernetes/platform/multus
kubectl kustomize config/kubernetes/platform/live
kubectl kustomize config/kubernetes/apps/plex
kubectl kustomize config/kubernetes/apps/live

Then reconcile platform GitOps and confirm Multus is ready:

flux -n flux-system reconcile source git your-gitops-repo
flux -n flux-system reconcile kustomization platform
kubectl -n kube-system rollout status ds/kube-multus-ds
kubectl get network-attachment-definitions.k8s.cni.cncf.io -A

Then stop the old Plex process:

docker compose -f /opt/plex/docker-compose.yml down

Then reconcile the app:

flux -n flux-system reconcile kustomization apps
kubectl -n plex rollout status deploy/plex

Only after Plex was healthy on the LAN did I treat the browser-facing routes as done.

Validation

The first thing I validate is the rendered intent. The app package should contain the network attachment and the Deployment annotation:

kubectl kustomize config/kubernetes/apps/plex | less

Then I check that the pod has both interfaces:

kubectl -n plex exec deploy/plex -- ip -br addr

Expected shape:

lo               UNKNOWN        127.0.0.1/8
eth0             UP             <pod-cidr-address>/...
lan0             UP             192.168.1.50/24

Then I check that Plex answers directly on the LAN address:

curl -fsSI http://192.168.1.50:32400/web

The Kubernetes ingress path depends on the selectorless Service and EndpointSlice, so I check those next:

kubectl -n plex get svc plex-lan
kubectl -n plex get endpointslice plex-lan -o yaml
kubectl -n plex get ingress plex plex-public-cloudflare
curl -I https://plex.example.com

During the migration, the public route returned HTTP/2 401 with x-plex-protocol: 1.0, which proved the Cloudflare path reached Plex. A 401 was the expected unauthenticated response, not a failure.

DNS should show the split paths:

dig +short plex.lan.example.com
dig +short plex.example.com

Expected internal answers:

192.168.1.50
192.168.1.20

I also check that the pod sees the data and devices it needs:

kubectl -n plex exec deploy/plex -- ls -la /config
kubectl -n plex exec deploy/plex -- ls -la /srv/media/Videos
kubectl -n plex exec deploy/plex -- ls -la /dev/dvb

If GPU transcoding is part of the deployment, I validate the node resource and then confirm hardware transcoding through Plex during real playback:

kubectl describe node k8s-node-1 | rg -n "nvidia.com/gpu|Capacity|Allocatable"
kubectl -n plex exec deploy/plex -- nvidia-smi

Some Plex images do not include nvidia-smi, so I treat that as a convenience check, not the only proof. The final proof is playback behavior in Plex.

Failure Modes I Watch For

When the pod gets stuck in ContainerCreating, I start with events:

kubectl -n plex describe pod -l app.kubernetes.io/name=plex

The usual causes are straightforward:

  • Multus is not installed or is not ready on the node.
  • The NetworkAttachmentDefinition name is wrong or in the wrong namespace.
  • The bridge CNI binary is missing from /opt/cni/bin.
  • The host bridge br0 does not exist on the selected node.

If the pod starts but lan0 is missing, I check the annotation:

kubectl -n plex get deploy plex -o jsonpath='{.spec.template.metadata.annotations.k8s\.v1\.cni\.cncf\.io/networks}{"\n"}'

Then I check the network attachment and Multus logs:

kubectl -n plex get network-attachment-definition plex-br0 -o yaml
kubectl -n kube-system logs -l app.kubernetes.io/name=multus --tail=200

If the LAN IP conflicts with another device, the symptoms look like ordinary network confusion: ARP flapping, intermittent connectivity, clients disagreeing about reachability, or the router showing the wrong MAC. Before cutover, I want the address reserved and quiet:

ping -c 1 -W 1 192.168.1.50
arp -n 192.168.1.50

If a conflict appears after cutover, I scale the Kubernetes Deployment to zero first. I do not start Compose until the duplicate address is gone.

If the pod cannot reach the internet, I check routes:

kubectl -n plex exec deploy/plex -- ip route

The secondary LAN interface should not become the default route. If it does, the bridge CNI IPAM config probably grew a gateway or route that does not belong there.

If LAN clients cannot discover Plex, I check that Plex is listening and advertising the LAN address:

kubectl -n plex exec deploy/plex -- printenv ADVERTISE_IP
curl -fsSI http://192.168.1.50:32400/web

Then I check whether the client discovery path depends on multicast traffic that local firewall policy has to allow. The manifest documents UDP 1900, 5353, and 32410-32414, but Kubernetes container ports are metadata. Actual firewall behavior still depends on the host, CNI, and network policy stack.

For this migration, the final proof was not only curl. I verified a local browser, a remote browser path, and a local Plex app connecting directly to the LAN address.

If direct LAN access works but ingress fails, I look at the selectorless Service, EndpointSlice, and ingress controller logs:

kubectl -n plex get svc plex-lan -o yaml
kubectl -n plex get endpointslice plex-lan -o yaml

In this setup, the internal ingress talks HTTPS to the backend with certificate verification disabled:

nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/proxy-ssl-verify: "off"

If Plex serves plain HTTP in a future revision, that annotation has to change.

Security Notes

Putting a pod directly on the LAN changes the security model. The workload is no longer reachable only through Kubernetes Services and ingress controllers. It is a LAN endpoint.

I treat the Plex pod like a host on the home network:

  • Reserve the IP in the network source of truth before assigning it in the
    NetworkAttachmentDefinition.
  • Pin Plex to the node where the bridge, host paths, and devices are expected.
  • Do not reuse the same LAN IP in Compose, LXC, DHCP, or another Kubernetes
    workload.
  • Use this pattern only for workloads that need real LAN presence.
  • Keep public exposure separate from LAN exposure.
  • Be careful with host paths because the pod can read and write mounted media
    and config directories.
  • Be careful with device mounts such as /dev/dvb and GPU access.
  • Keep account tokens and provider credentials out of Git.

NetworkPolicy may still be useful for the primary Cilium interface, but I do not assume it automatically describes all traffic entering through the bridge-attached LAN interface. I validate actual enforcement behavior before relying on it for LAN-side isolation.

Rollback

Rollback is intentionally simple because the migration kept the old data layout.

First, scale Kubernetes Plex to zero:

kubectl -n plex scale deploy/plex --replicas=0

Then confirm the LAN address is no longer assigned to the pod:

kubectl -n plex get pods -o wide
ping -c 1 -W 1 192.168.1.50

Then start the preserved Compose service:

docker compose -f /opt/plex/docker-compose.yml up -d

Then validate Plex directly:

curl -fsSI http://192.168.1.50:32400/web

The main rule is that Compose Plex and Kubernetes Plex must not both run against /opt/plex/config and 192.168.1.50 at the same time.

I also avoid deleting evidence during an incident. If the manifests themselves are destabilizing the platform, I would suspend the relevant Flux Kustomization instead of removing the app package from Git while I am still debugging.

What I Would Reuse

I would reuse this pattern for another LAN-appliance workload, but not for every web app.

For another workload, I would change:

  • Namespace and labels.
  • The NetworkAttachmentDefinition name.
  • The static LAN IP.
  • The pod annotation.
  • The node selector.
  • Any advertised URL or callback URL.
  • The Service and EndpointSlice if ingress needs to route to the LAN IP.
  • DNS records and firewall policy.

I would not copy the Plex host paths, ports, or GPU settings unless the new workload actually needed them.

The general rule is the same one I used for Plex: keep Cilium as the Kubernetes network, use Multus only for the extra LAN interface, and make the LAN IP assignment explicit in Git.

References

Comments are closed.