IT 끄적이는 물고기

2023.05.29

★ kube-apiserver에서 사용하고 있는 인증서 파일 위치 확인

1. kube-apiserver.yaml 파일 확인

-> --tls-cert-file=/etc/kubernetes/pki/apiserver.crt

controlplane ~ ➜ cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep tls
​​​​- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
​​​​- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

controlplane ~ ➜

2. kube-apiserver pod 확인

-> --tls-cert-file=/etc/kubernetes/pki/apiserver.crt

controlplane ~ ➜  kubectl -n kube-system describe pods kube-apiserver-controlplane | grep tls
​​​​​​--tls-cert-file=/etc/kubernetes/pki/apiserver.crt
​​​​​​--tls-private-key-file=/etc/kubernetes/pki/apiserver.key

controlplane ~ ➜

★ kube-apiserver에서 사용하고 있는 etcd-client 인증서 파일 위치 확인

-> etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt

controlplane ~ ➜  cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd    
​​​​- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
​​​​- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
​​​​- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
​​​​- --etcd-servers=https://127.0.0.1:2379

controlplane ~ ➜

★ kube-apiserver에서 사용하고 있는 kubelet 키 파일 위치 확인

-> --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key

controlplane ~ ➜  cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep kubelet
​​​​- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
​​​​- --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
​​​​- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname

controlplane ~ ➜

★ etcd server 에서 사용하고 있는 인증서 파일 위치 확인

-> --cert-file=/etc/kubernetes/pki/etcd/server.crt

controlplane ~ ➜  cat /etc/kubernetes/manifests/etcd.yaml | grep cert
​​​​- --cert-file=/etc/kubernetes/pki/etcd/server.crt
​​​​- --client-cert-auth=true
​​​​- --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
​​​​- --peer-client-cert-auth=true
​​​​​​name: etcd-certs
​​​​name: etcd-certs

controlplane ~ ➜

★ etcd server 에서 사용하고 있는 자체 ca 인증서 파일 위치 확인

-> --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

controlplane ~ ➜  cat /etc/kubernetes/manifests/etcd.yaml | grep ca
​​​​- --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
​​​​- --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
​​priorityClassName: system-node-critical

controlplane ~ ➜

★ Kube-apiserver 인증서에 구성된 CN 이름은 무엇인가요?

-> Subject: CN = kube-apiserver

controlplane ~ ➜  openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text
Certificate:
​​​​Data:
​​​​​​​​Version: 3 (0x2)
​​​​​​​​Serial Number: 8483000120492181273 (0x75b9abaa2b884719)
​​​​​​​​Signature Algorithm: sha256WithRSAEncryption
​​​​​​​​Issuer: CN = kubernetes
​​​​​​​​Validity
​​​​​​​​​​​​Not Before: May 29 11:43:33 2023 GMT
​​​​​​​​​​​​Not After : May 28 11:43:33 2024 GMT
​​​​​​​​Subject: CN = kube-apiserver
​​​​​​​​Subject Public Key Info:
​​​​​​​​​​​​Public Key Algorithm: rsaEncryption
​​​​​​​​​​​​​​​​RSA Public-Key: (2048 bit)
...

★ Kube-apiserver 인증서를 발급한 CA의 이름은 무엇인가요?

-> Issuer: CN = kubernetes

controlplane ~ ➜  openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text
Certificate:
​​​​Data:
​​​​​​​​Version: 3 (0x2)
​​​​​​​​Serial Number: 8483000120492181273 (0x75b9abaa2b884719)
​​​​​​​​Signature Algorithm: sha256WithRSAEncryption
​​​​​​​​Issuer: CN = kubernetes
​​​​​​​​Validity
​​​​​​​​​​​​Not Before: May 29 11:43:33 2023 GMT
​​​​​​​​​​​​Not After : May 28 11:43:33 2024 GMT
​​​​​​​​Subject: CN = kube-apiserver
​​​​​​​​Subject Public Key Info:
​​​​​​​​​​​​Public Key Algorithm: rsaEncryption
​​​​​​​​​​​​​​​​RSA Public-Key: (2048 bit)
...

★ Kube-apiserver 인증서에 구성된 대체 이름은 무엇인가요?

-> DNS:controlplane, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:192.26.249.9

controlplane ~ ➜  openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text
Certificate:
​​​​Data:
​​​​​​​​Version: 3 (0x2)
​​​​​​​​Serial Number: 8483000120492181273 (0x75b9abaa2b884719)
​​​​​​​​Signature Algorithm: sha256WithRSAEncryption
​​​​​​​​Issuer: CN = kubernetes
​​​​​​​​Validity
​​​​​​​​​​​​Not Before: May 29 11:43:33 2023 GMT
​​​​​​​​​​​​Not After : May 28 11:43:33 2024 GMT
​​​​​​​​Subject: CN = kube-apiserver
​​​​​​​​Subject Public Key Info:
​​​​​​​​​​​​Public Key Algorithm: rsaEncryption
​​​​​​​​​​​​​​​​RSA Public-Key: (2048 bit)
​​​​​​​​​​​​​​​​Modulus:
​​​​​​​​​​​​​​​​​​​​00:a3:2f:ff:75:ed:b2:38:74:01:f9:b1:41:51:aa:
​​​​​​​​​​​​​​​​​​​​f5:bb:9a:39:02:46:c2:5b:05:b1:0e:8f:75:9b:46:
​​​​​​​​​​​​​​​​​​​​18:a5:35:52:2f:2d:22:3b:fe:37:e3:ea:98:32:c5:
​​​​​​​​​​​​​​​​​​​​79:b4:2d:1b:f2:67:cd:f6:7d:4e:fa:e8:a0:69:b4:
​​​​​​​​​​​​​​​​​​​​4b:c8:25:46:20:4b:ad:69:dd:fa:63:56:b4:5c:4f:
​​​​​​​​​​​​​​​​​​​​ce:b7:28:bb:43:de:59:5f:c6:e7:c7:16:08:11:cf:
​​​​​​​​​​​​​​​​​​​​28:b2:4a:7f:20:74:3d:f4:53:6a:b6:33:37:25:98:
​​​​​​​​​​​​​​​​​​​​3e:a7:02:56:da:1b:75:7a:39:bd:0a:31:d5:26:cb:
​​​​​​​​​​​​​​​​​​​​30:8b:3d:bf:a5:58:48:8c:a8:5d:b4:eb:51:0d:72:
​​​​​​​​​​​​​​​​​​​​52:32:85:60:0d:56:2f:46:3c:65:90:4a:9b:a3:01:
​​​​​​​​​​​​​​​​​​​​b3:d9:01:b2:d9:ea:70:68:38:49:d5:1a:29:9f:52:
​​​​​​​​​​​​​​​​​​​​b8:54:72:71:0c:4a:88:4b:73:63:6f:05:a0:b6:23:
​​​​​​​​​​​​​​​​​​​​03:31:12:be:c3:cf:6c:b7:2b:e6:4e:50:a1:1b:7f:
​​​​​​​​​​​​​​​​​​​​ab:2a:ba:5f:92:16:3d:4c:ac:d8:02:11:78:8b:bf:
​​​​​​​​​​​​​​​​​​​​4e:43:3b:e5:0c:57:fb:6f:8a:81:ef:51:7e:a3:92:
​​​​​​​​​​​​​​​​​​​​2a:de:2b:96:ae:95:2e:dc:e3:97:ce:c7:af:8d:42:
​​​​​​​​​​​​​​​​​​​​67:2c:6a:3a:fa:fa:67:79:d2:14:52:47:eb:65:ca:
​​​​​​​​​​​​​​​​​​​​53:af
​​​​​​​​​​​​​​​​Exponent: 65537 (0x10001)
​​​​​​​​X509v3 extensions:
​​​​​​​​​​​​X509v3 Key Usage: critical
​​​​​​​​​​​​​​​​Digital Signature, Key Encipherment
​​​​​​​​​​​​X509v3 Extended Key Usage: 
​​​​​​​​​​​​​​​​TLS Web Server Authentication
​​​​​​​​​​​​X509v3 Basic Constraints: critical
​​​​​​​​​​​​​​​​CA:FALSE
​​​​​​​​​​​​X509v3 Authority Key Identifier: 
​​​​​​​​​​​​​​​​keyid:AB:7D:E2:A1:2C:F0:E0:27:53:52:72:D8:C9:46:76:09:F8:77:0D:63

​​​​​​​​​​​​X509v3 Subject Alternative Name: 
​​​​​​​​​​​​​​​​DNS:controlplane, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:192.26.249.9

★ etcd server 인증서에 구성된 CN은 무엇인가요?

-> Subject: CN = controlplane

controlplane ~ ➜  openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -text
Certificate:
​​​​Data:
​​​​​​​​Version: 3 (0x2)
​​​​​​​​Serial Number: 253720388574558331 (0x38565596142b87b)
​​​​​​​​Signature Algorithm: sha256WithRSAEncryption
​​​​​​​​Issuer: CN = etcd-ca
​​​​​​​​Validity
​​​​​​​​​​​​Not Before: May 29 11:43:34 2023 GMT
​​​​​​​​​​​​Not After : May 28 11:43:34 2024 GMT
​​​​​​​​Subject: CN = controlplane
​​​​​​​​Subject Public Key Info:
​​​​​​​​​​​​Public Key Algorithm: rsaEncryption
​​​​​​​​​​​​​​​​RSA Public-Key: (2048 bit)
...

★ etcd server 인증서는 발급일로부터 얼마 동안 유효하나요?

-> 1 years 

controlplane ~ ➜  openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -text
Certificate:
​​​​Data:
​​​​​​​​Version: 3 (0x2)
​​​​​​​​Serial Number: 253720388574558331 (0x38565596142b87b)
​​​​​​​​Signature Algorithm: sha256WithRSAEncryption
​​​​​​​​Issuer: CN = etcd-ca
​​​​​​​​Validity
​​​​​​​​​​​​Not Before: May 29 11:43:34 2023 GMT
​​​​​​​​​​​​Not After : May 28 11:43:34 2024 GMT
​​​​​​​​Subject: CN = controlplane
​​​​​​​​Subject Public Key Info:
​​​​​​​​​​​​Public Key Algorithm: rsaEncryption
​​​​​​​​​​​​​​​​RSA Public-Key: (2048 bit)
...

★ etcd server ca인증서는 발급일로부터 얼마 동안 유효하나요?

->  10 years

controlplane ~ ➜  openssl x509 -in /etc/kubernetes/pki/ca.crt -text
Certificate:
​​​​Data:
​​​​​​​​Version: 3 (0x2)
​​​​​​​​Serial Number: 0 (0x0)
​​​​​​​​Signature Algorithm: sha256WithRSAEncryption
​​​​​​​​Issuer: CN = kubernetes
​​​​​​​​Validity
​​​​​​​​​​​​Not Before: May 29 11:43:33 2023 GMT
​​​​​​​​​​​​Not After : May 26 11:43:33 2033 GMT
​​​​​​​​Subject: CN = kubernetes
​​​​​​​​Subject Public Key Info:
​​​​​​​​​​​​Public Key Algorithm: rsaEncryption
​​​​​​​​​​​​​​​​RSA Public-Key: (2048 bit)
...

★ kubectl 명령어가 동작하지 않습니다. etcd server를 확인해보고 틀린부분을 수정하시오.

-> etcd 서버 yaml 파일을 확인해보면 cert-file 경로에 server-certificate.crt 파일이라고 저장되어있다.

controlplane ~ ➜  cat /etc/kubernetes/manifests/etcd.yaml | grep cert-file
​​​​- --cert-file=/etc/kubernetes/pki/etcd/server-certificate.crt
​​​​- --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt

controlplane ~ ➜

->  해당 경로에 crt 파일을 확인해보면 파일명이 다른걸 확인할 수 있다.

controlplane ~ ➜  ls /etc/kubernetes/pki/etcd/server* | grep .crt
/etc/kubernetes/pki/etcd/server.crt

controlplane ~ ➜

->  경로명을 수정해주고 기다리면 api-server가 정상적으로 작동한다.

controlplane ~ ➜  vi /etc/kubernetes/manifests/etcd.yaml 

controlplane ~ ➜  cat /etc/kubernetes/manifests/etcd.yaml | grep cert-file
​​​​- --cert-file=/etc/kubernetes/pki/etcd/server.crt
​​​​- --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt

controlplane ~ ➜

★ kubectl 명령어가 동작하지 않습니다. apiserver 서버의 logs를 확인해서 문제를 해결하시오.

-> crictl ps -a | grep kube-apiserver 명령어로 pod 상태 확인 

controlplane ~ ➜  crictl ps -a | grep kube-apiserver
9bd19c0102ca6       a31e1d84401e6       36 seconds ago      Exited              kube-apiserver            5                   5413bff6f15be       kube-apiserver-controlplane

controlplane ~ ➜

-> logs를 확인해보니 etcd ca 인증서에 문제가 있는 것을 확인했습니다.

controlplane ~ ➜  crictl logs --tail=5 9bd19c0102ca6
​​"BalancerAttributes": null,
​​"Type": 0,
​​"Metadata": null
}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"
E0529 13:03:01.408411       1 run.go:74] "command failed" err="context deadline exceeded"

controlplane ~ ➜

-> kube-apiserver.yaml 파일의 etcd cafile 위치가 잘못되어있습니다. 수정해주세요.

controlplane ~ ➜  cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd
​​​​- --etcd-cafile=/etc/kubernetes/pki/ca.crt
​​​​- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
​​​​- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
​​​​- --etcd-servers=https://127.0.0.1:2379

controlplane ~ ➜  cd /etc/kubernetes/pki/etcd/

controlplane kubernetes/pki/etcd ➜  ls -al
total 40
drwxr-xr-x 2 root root 4096 May 29 08:44 .
drwxr-xr-x 3 root root 4096 May 29 08:44 ..
-rw-r--r-- 1 root root 1086 May 29 08:44 ca.crt
-rw------- 1 root root 1675 May 29 08:44 ca.key
-rw-r--r-- 1 root root 1159 May 29 08:44 healthcheck-client.crt
-rw------- 1 root root 1679 May 29 08:44 healthcheck-client.key
-rw-r--r-- 1 root root 1208 May 29 08:44 peer.crt
-rw------- 1 root root 1679 May 29 08:44 peer.key
-rw-r--r-- 1 root root 1208 May 29 08:44 server.crt
-rw------- 1 root root 1675 May 29 08:44 server.key

controlplane kubernetes/pki/etcd ➜

-> --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt

controlplane ~ ➜  cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd
​​​​- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
​​​​- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
​​​​- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
​​​​- --etcd-servers=https://127.0.0.1:2379

controlplane ~ ➜