k8s pod服务异常重启terminated

现象

项目运维人员反馈k8s服务发生重启，需找到重启原因。

排查步骤

查看k8s重启原因

kubectl -n bms-prod describe pods bms-job-autobill-557d8dc577-nx2qv

Last State:     Terminated
      Reason:       Error
      Exit Code:    143

根据错误代码，初步判断是外部异常导致

查看服务重启之前的日志是否异常

kubectl -n bms-prod logs --previous bms-job-autobill-557d8dc577-nx2qv

查看日志并无异常

查看事件

kubectl -n bms-prod get events

并无异常事件

获取具体事件

kubectl -n bms-prod get events -o json > 1.json

查看服务器系统日志

vim /var/logs/message
dmesg -T

发现了连接nfs错误

[Fri Sep 10 01:38:03 2021] nfs: server 10.0.15.1 not responding, still trying
[Fri Sep 10 01:38:50 2021] nfs: server 10.0.15.1 not responding, still trying
[Fri Sep 10 01:39:02 2021] nfs: server 10.0.15.1 not responding, still trying
[Fri Sep 10 01:39:21 2021] nfs: server 10.0.15.1 not responding, timed out
[Fri Sep 10 01:41:42 2021] nfs: server 10.0.15.1 OK
[Fri Sep 10 01:41:42 2021] nfs: server 10.0.15.1 OK
[Fri Sep 10 01:41:42 2021] nfs: server 10.0.15.1 OK
[Fri Sep 10 02:53:27 2021] nfs: server 10.0.15.1 not responding, still trying
[Fri Sep 10 02:53:27 2021] nfs: server 10.0.15.1 not responding, still trying
[Fri Sep 10 02:53:55 2021] nfs: server 10.0.15.1 not responding, timed out
[Fri Sep 10 02:54:51 2021] nfs: server 10.0.15.1 not responding, still trying
[Fri Sep 10 02:55:15 2021] nfs: server 10.0.15.1 OK
[Fri Sep 10 02:55:15 2021] nfs: server 10.0.15.1 OK
[Fri Sep 10 02:56:47 2021] nfs: server 10.0.15.1 OK
[Fri Sep 10 02:58:15 2021] nfs: server 10.0.15.1 not responding, still trying
[Fri Sep 10 02:58:15 2021] nfs: server 10.0.15.1 not responding, still trying
[Fri Sep 10 02:58:46 2021] nfs: server 10.0.15.1 OK
[Fri Sep 10 02:58:46 2021] nfs: server 10.0.15.1 OK
[Fri Sep 10 05:55:53 2021] nfs: server 10.0.15.1 not responding, still trying
[Fri Sep 10 05:56:17 2021] nfs: server 10.0.15.1 not responding, still trying
[Fri Sep 10 05:56:18 2021] nfs: server 10.0.15.1 OK
[Fri Sep 10 05:56:18 2021] nfs: server 10.0.15.1 OK

因此初步判断挂载windows nfs server导致异常重启。