Use nginx proxy in VKS

bài NLB(layer4) -> nginx-ingress(layer7) trong k8s -> pod backend là bài phổ biến note cho 1 docs về config realip xuyên suốt 3 lớp này

Pre-requirement

Already install VNGCloud Controller Manager

Update to dev version of VNGCloud Controller Manager

kubectl patch deployment -n kube-system vngcloud-controller-manager -p '{"spec": {"template": {"spec": {"containers": [{"name":"vngcloud-controller-manager","image":"vcr.vngcloud.vn/60108-annd2-ingress/vngcloud-controller-manager:v0.2.0"}]}}}}'
kubectl patch deployment -n kube-system vngcloud-controller-manager -p '{"spec": {"template": {"spec": {"containers": [{"name":"vngcloud-controller-manager","image":"vcr.vngcloud.vn/81-vks-public/vngcloud-controller-manager:v0.2.0"}]}}}}'


kubectl patch statefulsets -n kube-system vngcloud-ingress-controller -p '{"spec": {"template": {"spec": {"containers": [{"name":"vngcloud-ingress-controller","image":"vcr.vngcloud.vn/60108-annd2-ingress/vngcloud-ingress-controller:v0.2.0"}]}}}}'
kubectl patch statefulsets -n kube-system vngcloud-ingress-controller -p '{"spec": {"template": {"spec": {"containers": [{"name":"vngcloud-ingress-controller","image":"vcr.vngcloud.vn/81-vks-public/vngcloud-ingress-controller:v0.2.0"}]}}}}'

Install sample app to debug via Helm

helm repo add anngdinh https://anngdinh.github.io/helm-charts
helm install goapp anngdinh/goapp-debug

Install nginx-ingress-controller via Helm

helm install nginx-ingress-controller oci://ghcr.io/nginxinc/charts/nginx-ingress --namespace kube-system

Update configmap for nginx-ingress-controller

kubectl edit cm -n kube-system nginx-ingress-controller

data:
  proxy-protocol: "True"
  real-ip-header: proxy_protocol
  real-ip-recursive: "True"
  set-real-ip-from: 0.0.0.0/0

Update service nginx-ingress-controller

kubectl annotate service -n kube-system nginx-ingress-controller-controller vks.vngcloud.vn/enable-proxy-protocol="http,https"

Apply sample ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: kkk.example.com
    http:
      paths:
      - backend:
          service:
            name: prometheus-node-exporter
            port:
              number: 9100
        path: /metrics
        pathType: Exact

Check in log

kubectl logs $(kubectl get pods -n kube-system -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep nginx-ingress-controller-controller) -n kube-system -f

Curl sample:

curl -H 'Host: kkk.example.com' http://_____IP_____/metrics

Update CCM to new version

Get release name of vngcloud-controller-manager

$ helm list -A | grep vngcloud-controller-manager

vngcloud-controller-manager-1716448250          kube-system     10              2024-06-10 17:00:17.866548653 +0700 +07 deployed        vngcloud-controller-manager-0.2.3       v0.2.0

Then update release to latest version. You can specific version by using flag --version

helm upgrade vngcloud-controller-manager-1716448250 oci://vcr.vngcloud.vn/81-vks-public/vks-helm-charts/vngcloud-controller-manager \
  --namespace kube-system

vContainer Helm Infra Documentation