K8s服务器由于机房维护，关机后启动问题-开发者社区

一、背景

服务器由于升级原因，临时断电，断电前手动停止程序并手动关机（k8s环境没有理会）。恢复供电后，问题处理.
环境: centos7 docker k8s
[root@wdy01 ~]# kubectl get node
E0727 13:54:51.638488 160568 memcache.go:265] couldn’t get current server API group list: Get “https://10.1.1.1:6443/api?timeout=32s”: dial tcp 10.1.1.1:6443: connect: connection refused
E0727 13:54:51.638859 160568 memcache.go:265] couldn’t get current server API group list: Get “https://10.1.1.1:6443/api?timeout=32s”: dial tcp 10.1.1.1:6443: connect: connection refused
E0727 13:54:51.640220 160568 memcache.go:265] couldn’t get current server API group list: Get “https://10.1.1.1:6443/api?timeout=32s”: dial tcp 10.1.1.1:6443: connect: connection refused
E0727 13:54:51.641583 160568 memcache.go:265] couldn’t get current server API group list: Get “https://10.1.1.1:6443/api?timeout=32s”: dial tcp 10.1.1.1:6443: connect: connection refused
E0727 13:54:51.642889 160568 memcache.go:265] couldn’t get current server API group list: Get “https://10.1.1.1:6443/api?timeout=32s”: dial tcp 10.1.1.1:6443: connect: connection refused
The connection to the server 10.1.1.1:6443 was refused - did you specify the right host or port?
[root@wdy01 ~]#

二、解决过程

2.1 先检查集群有效期，发现正常

[root@wdy01 ~]# kubeadm certs check-expiration[check-expiration]Reading configurationfromthe cluster...[check-expiration]FYI: You can look at this config file with'kubectl -n kube-system get cm kubeadm-config -o yaml'[check-expiration]Error reading configurationfromthe Cluster.Falling back to default configuration CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Jul 07,2027 00:21 UTC 344d ca no apiserver Jul 07,2027 00:21 UTC 344d ca no apiserver-etcd-client Jul 07,2027 00:21 UTC 344d etcd-ca no apiserver-kubelet-client Jul 07,2027 00:21 UTC 344d ca no controller-manager.conf Jul 07,2027 00:21 UTC 344d ca no etcd-healthcheck-client Jul 07,2027 00:21 UTC 344d etcd-ca no etcd-peer Jul 07,2027 00:21 UTC 344d etcd-ca no etcd-server Jul 07,2027 00:21 UTC 344d etcd-ca no front-proxy-client Jul 07,2027 00:21 UTC 344d front-proxy-ca no scheduler.conf Jul 07,2027 00:21 UTC 344d ca no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Jul 02,2035 01:45 UTC 8y no etcd-ca Jul 02,2035 01:45 UTC 8y no front-proxy-ca Jul 02,2035 01:45 UTC 8y no

2.2 查看kubelet日志

Jul2714:22:56 wdy01 systemd[1]: kubelet.service holdofftimeover, scheduling restart. Jul2714:22:56 wdy01 systemd[1]: Stopped kubelet: The Kubernetes Node Agent. Jul2714:22:56 wdy01 systemd[1]: Started kubelet: The Kubernetes Node Agent. Jul2714:22:56 wdy01 kubelet[237447]: Flag --container-runtime-endpoint has been deprecated, This parameter should besetvia the configfilespecified by the Kubelet's--configflag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/formoreinformation. Jul2714:22:56 wdy01 kubelet[237447]: Flag --pod-infra-container-image has been deprecated, will be removedina future release. Image garbage collector will get sandbox image information from CRI. Jul2714:22:56 wdy01 kubelet[237447]: I072714:22:56.600080237447server.go:203]"--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"Jul2714:22:56 wdy01 kubelet[237447]: I072714:22:56.605137237447server.go:467]"Kubelet version"kubeletVersion="v1.2x.2"Jul2714:22:56 wdy01 kubelet[237447]: I072714:22:56.605175237447server.go:469]"Golang settings"GOGC=""GOMAXPROCS=""GOTRACEBACK=""Jul2714:22:56 wdy01 kubelet[237447]: I072714:22:56.606111237447server.go:895]"Client rotation is on, will bootstrap in background"Jul2714:22:56 wdy01 kubelet[237447]: I072714:22:56.607831237447certificate_store.go:130]Loading cert/key pair from"/var/lib/kubelet/pki/kubelet-client-current.pem".Jul2714:22:56 wdy01 kubelet[237447]: I072714:22:56.608791237447dynamic_cafile_content.go:157]"Starting controller"name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"Jul2714:22:56 wdy01 kubelet[237447]: W072714:22:56.609247237447logging.go:59][core][Channel#1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Jul2714:22:56 wdy01 kubelet[237447]:"Addr":"/var/run/cri-dockerd.sock", Jul2714:22:56 wdy01 kubelet[237447]:"ServerName":"/var/run/cri-dockerd.sock", Jul2714:22:56 wdy01 kubelet[237447]:"Attributes":null, Jul2714:22:56 wdy01 kubelet[237447]:"BalancerAttributes":null, Jul2714:22:56 wdy01 kubelet[237447]:"Type":0, Jul2714:22:56 wdy01 kubelet[237447]:"Metadata":null Jul2714:22:56 wdy01 kubelet[237447]:}. Err: connection error: desc="transport: Error while dialing: dial unix /var/run/cri-dockerd.sock: connect: connection refused"Jul2714:22:56 wdy01 kubelet[237447]: E072714:22:56.610057237447run.go:74]"command failed"err="failed to run Kubelet: validate service connection: validate CRI v1 runtime API for endpoint\"unix:///var/run/cri-dockerd.sock\": rpc error: code = Unavailable desc = connection error: desc =\"transport: Error while dialing: dial unix /var/run/cri-dockerd.sock: connect: connection refused\""Jul2714:22:56 wdy01 systemd[1]: kubelet.service: main process exited,code=exited,status=1/FAILURE Jul2714:22:56 wdy01 systemd[1]: Unit kubelet.service entered failed state. Jul2714:22:56 wdy01 systemd[1]: kubelet.service failed.

如上日志：发现kubelet 启动失败核心原因：
dial unix /var/run/cri-dockerd.sock: connect: connection refused
cri-dockerd 服务没启动 / 未安装，kubelet 无法对接 Docker 的 CRI 接口，直接崩溃退出

2.3 解决

1）查看服务状态

systemctl status cri-dockerd

2）查看套接字文件是否存在

ls -l /var/run/cri-dockerd.sock

3）发现断电重启后，cri-dockerd服务丢失了

我下载过 cri-dockerd-0.3.14-3.el7.x86_64.rpm，执行rpm -ivh cri-dockerd-0.3.14-3.el7.x86_64.rpm安装。

4）重启并确认

systemctl daemon-reload# 开机自启并启动systemctlenable--nowcri-docker# 查看状态确认runningsystemctl status cri-docker# 检查socket文件是否生成ls/var/run/cri-dockerd.sock[root@wdy02 ~]# ls /var/run/cri-dockerd.sock/var/run/cri-dockerd.sock

[root@wdy02 ~]# systemctl status cri-docker● cri-docker.service - CRI InterfaceforDocker Application Container Engine Loaded: loaded(/usr/lib/systemd/system/cri-docker.service;enabled;vendor preset: disabled)Active: active(running)since Thu2026-08-2711:14:06 CST;1h 26min ago Docs: https://docs.mirantis.com Main PID:231334(cri-dockerd)Tasks:38Memory:46.6M CGroup: /system.slice/cri-docker.service └─231334 /usr/bin/cri-dockerd --container-runtime-endpoint fd:// --pod-infra-container-image=registry.aliyuncs.com/google_containers/... Aug2711:31:06 wdy02 cri-dockerd[231334]: delegateAdd: netconf sent to delegate plugin: Aug2711:32:02 wdy02 cri-dockerd[231334]:{"cniVersion":"0.3.1","hairpinMode":true,"ipMasq":false,"ipam":{"ranges":[[{"subnet":"10.244... msg="Wi Aug2711:32:03 wdy02 cri-dockerd[231334]: map[string]interface{}{"cniVersion":"0.3.1","hairpinMode":true,"ipMasq":false,"ipam":map...IP{0xa, Aug2711:32:03 wdy02 cri-dockerd[231334]: delegateAdd: netconf sent to delegate plugin: Aug2712:17:16 wdy02 cri-dockerd[231334]:{"cniVersion":"0.3.1","hairpinMode":true,"ipMasq":false,"ipam":{"ranges":[[{"subnet":"10.244... msg="Wi Aug2712:17:16 wdy02 cri-dockerd[231334]: map[string]interface{}{"cniVersion":"0.3.1","hairpinMode":true,"ipMasq":false,"ipam":map...IP{0xa, Aug2712:17:16 wdy02 cri-dockerd[231334]: delegateAdd: netconf sent to delegate plugin: Aug2712:17:17 wdy02 cri-dockerd[231334]:{"cniVersion":"0.3.1","hairpinMode":true,"ipMasq":false,"ipam":{"ranges":[[{"subnet":"10.244... msg="Wi Aug2712:17:17 wdy02 cri-dockerd[231334]: map[string]interface{}{"cniVersion":"0.3.1","hairpinMode":true,"ipMasq":false,"ipam":map...IP{0xa, Aug2712:17:17 wdy02 cri-dockerd[231334]: delegateAdd: netconf sent to delegate plugin: Hint: Some lines were ellipsized, use-lto showinfull.

5）重启并确认

修复 kubelet 配置，消除 cri-dockerd.sock 连接拒绝报错

vi/var/lib/kubelet/config.yaml

追加一行（没有就加，有就确认一致）：

containerRuntimeEndpoint: unix:///var/run/cri-dockerd.sock

6) 重启kubelet

重启kubelet

systemctl daemon-reload systemctl restart kubelet# 实时观察日志journalctl-ukubelet-f

如有必要，重启kube-apiserver、kube-controller-manage、kube-scheduler

# 如果是docker作为容器的话，可执行如下命令。其余容器方法类似dockerps|grepkube-apiserver|grep-vpause|awk'{print $1}'|xargs-idockerrestart{}dockerps|grepkube-controller-manage|grep-vpause|awk'{print $1}'|xargs-idockerrestart{}dockerps|grepkube-scheduler|grep-vpause|awk'{print $1}'|xargs-idockerrestart{}

三、补充

补充1：根据实际情况，docker必须保障正常。

systemctl restart docker
systemctl enable docker

补充2：如启动后发现，worker节点也没有正常ready，则worker节点也需要排查并安装cri-dockerd。

四、结果

如下图，执行kubectl get nodes查看集群状态已正常。

五、新问题

后端程序链接nacos报错：java.net.NoRouteToHostException: No route to host (Host unreachable)
测试发现，宿主机worker是可以连通01部署的nacos的。但程序pod内不通。关闭防火墙也不行。
最后发现：02 节点上 docker/flannel 残留 iptables nat 规则拦截了 Pod 访问宿主机网段的回包，清空 iptables 后立刻恢复连通。
最后是在02 清空所有 iptables 规则（nat+filter）解决的。

# 清空过滤链iptables-F# 清空nat转发链（关键）iptables-tnat-F# 设置默认转发允许iptables-PFORWARD ACCEPT

## 补充1：过了一段时间，不知什么原因又出现NoRouteToHostException了

最后是这样解决的：
首先是docker配置文件添加编辑/etc/docker/daemon.json，增加"ip-forward":false

## 修改 docker 配置，防止重启又生成 DOCKER 链不让docker乱改 filter 表 FORWARD 链、不注入 DOCKER 隔离防火墙规则；内核 IP 转发仍然开启，calico/kube‑proxy 全权接管集群网络。 ```bash{"exec-opts":["native.cgroupdriver=systemd"],"data-root":"/UData/docker","insecure-registries":["ip:81"],"registry-mirrors":["https://docker.1panel.live"],"log-driver":"json-file","log-opts":{"max-size":"100m","max-file":"5"},"ip-forward":false}

重启 docker 与 cri‑dockerd

systemctl daemon-reload systemctl restartdockercri-docker

删除旧的规则
1)先删除 FORWARD 链中跳转的引用（关键，解决 Too many links）

# 删除FORWARD里跳转到DOCKER‑USERiptables-DFORWARD-jDOCKER-USER# 删除FORWARD里跳转到DOCKER‑ISOLATION‑STAGE‑1iptables-DFORWARD-jDOCKER-ISOLATION-STAGE-1# 删除FORWARD里跳转到FLANNEL‑FWDiptables-DFORWARD-jFLANNEL-FWD# 删除firewalld残留跳转iptables-DFORWARD-jFORWARD_direct iptables-DFORWARD-jFORWARD_IN_ZONES_SOURCE iptables-DFORWARD-jFORWARD_IN_ZONES iptables-DFORWARD-jFORWARD_OUT_ZONES_SOURCE iptables-DFORWARD-jFORWARD_OUT_ZONES

2)现在就可以清空并删除这些链

# docker链iptables-FDOCKER-USERiptables-XDOCKER-USERiptables-FDOCKER-ISOLATION-STAGE-1 iptables-XDOCKER-ISOLATION-STAGE-1 iptables-FDOCKER iptables-XDOCKER# flanneliptables-FFLANNEL-FWD iptables-XFLANNEL-FWD# firewalld残留iptables-FFORWARD_direct iptables-XFORWARD_direct iptables-FFORWARD_IN_ZONES iptables-XFORWARD_IN_ZONES iptables-FFORWARD_IN_ZONES_SOURCE iptables-XFORWARD_IN_ZONES_SOURCE iptables-FFORWARD_OUT_ZONES iptables-XFORWARD_OUT_ZONES iptables-FFORWARD_OUT_ZONES_SOURCE iptables-XFORWARD_OUT_ZONES_SOURCE

3)删除那两条致命 REJECT 规则（不用‑m reject，按行序号删除）
先查看行号：

iptables-LFORWARD-n--line-numbers

输出会显示每一行的 num 编号，找到两条 REJECT 行的行号，例如假设行号是 20、24：

# 替换为你实际看到的行号，先删大序号，再删小序号iptables-DFORWARD24iptables-DFORWARD20

注意：删除行号的时候，先删数字大的行，因为删掉前面行，后面行号会变化。

4）清理完核对结果

[root@woker02 ~]# iptables -L FORWARD -n --line-numbers Chain FORWARD (policy ACCEPT) num target prot opt source destination 1 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 2 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 3 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 4 KUBE-PROXY-FIREWALL all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes load balancer firewall */ 5 KUBE-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */ 6 KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */ 7 KUBE-EXTERNAL-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes externally-visible service portals */ 8 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 9 ACCEPT all -- 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED 10 ACCEPT all -- 192.168.122.0/24 0.0.0.0/0 11 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 12 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 13 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 14 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 15 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 16 FLANNEL-FWD all -- 0.0.0.0/0 0.0.0.0/0 /* flanneld forward */