文档中所有helm服务的部署都Kubernetes-1.24.6集群中进行的,低版本或者高版本部分api可能不支持,请自行研究。

Helm的常用命令

查看helm的版本信息

1
$ helm version 

repo仓库相关

1
2
3
4
$ helm repo list
$ helm repo add jumpserver https://jumpserver.github.io/helm-charts
$ helm repo update
$ helm repo remove jumpserver

search搜索相关

1
$ helm search repo gitlab

show查看相关

1
2
3
4
5
6
7
8
9
10
11
# 显示chart的全部信息
$ helm show all

# 显示chart的定义信息
$ helm show chart gitlab/gitlab

# 显示chart的readme信息
$ helm show readme

# 显示chart的values信息
$ helm show values gitlab/gitlab

install安装相关

1
2
3
4
5
6
7
8
# 【常用】通过外部的values.yaml文件,直接通过仓库安装指定版本的应用到集群中
$ helm install ingress-nginx --version 4.4.3 -f ./ingress-nginx-values.yaml ingress-nginx/ingress-nginx

# 通过--set的方式设置values中的值,直接通过仓库安装指定版本的应用到集群中
$ helm install ingress-nginx --version 4.4.3 --set nfs.server=192.168.1.43 ingress-nginx/ingress-nginx

# 【比较少用】下载helm服务的chart包解压到本地,然后修改values.yaml后直接指定本地目录安装
$ helm install ingress-nginx ingress-nginx/

安装nfs-client-provisioner

安装nfs文件共享服务

1
2
3
4
5
6
7
8
$ yum install -y nfs-utils
$ mkdir /data/volumes/nfs
$ vim /etc/exports
/data/volumes/nfs 192.168.1.0/24(rw,no_root_squash)
$ systemctl restart nfs
$ showmount -e 192.168.1.43
Export list for 192.168.1.43:
/data/volumes/nfs 192.168.1.0/24

k8s集群中的所有Node安装nfs-utils工具。

1
$ yum install -y nfs-utils

通过helm安装nfs-client-provisioner

1
2
3
4
5
6
7
8
9
$ helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
$ helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner --version 4.0.17 \
--set nfs.server=192.168.1.43 \
--set nfs.path=/data/volumes/nfs \
--set image.repository=putianhui/nfs-subdir-external-provisioner \
--set image.tag=v4.0.2 \
--set storageClass.create=true \
--set storageClass.name=nfs-sc \
--set storageClass.reclaimPolicy=Retain
  • nfs.server:你的nfs服务器ip地址
  • nfs.path:nfs服务器共享的目录
  • image.repository:由于默认的仓库地址是k8s官方的谷歌仓库需要科学上网,我这里把镜像pull下来上传到了我自己的仓库下。
  • image.tag:镜像的版本
  • storageClass.create:自动创建一个存储类
  • storageClass.name:自动创建的存储类名称
  • storageClass.reclaimPolicy:自动创建存储类的回收规则,默认是delete

查看helm安装后nfs-provisioner的pod和sc是否正常启动并创建

1
2
3
4
5
6
7
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
nfs-subdir-external-provisioner-cbc4c9bcb-5mrbc 1/1 Running 0 3m22s

$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
nfs-sc cluster.local/nfs-subdir-external-provisioner Retain Immediate true 3m30s

创建deploy通过存储类自动创建pvc并挂载验证是否可用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
$ vim test-sc-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-nfs-mount-pvc
spec:
storageClassName: "nfs-sc"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-nfs-mount-pvc-deploy
spec:
selector:
matchLabels:
app: test-nfs-mount
strategy:
type: Recreate
template:
metadata:
labels:
app: test-nfs-mount
spec:
containers:
- image: centos:7
name: centos
volumeMounts:
- name: test-nfs-pvc
mountPath: /data/
command: ["/bin/bash","-c","sleep 9999999"]
volumes:
- name: test-nfs-pvc
persistentVolumeClaim:
claimName: test-nfs-mount-pvc

应用测试用的nfs-deploy文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 应用清单
$ kubectl apply -f test-sc-pvc.yaml
persistentvolumeclaim/test-nfs-mount-pvc created
deployment.apps/test-nfs-mount-pvc-deploy created

# 查看pvc是否自动创建并绑定
$ kubectl get pvc
test-nfs-mount-pvc Bound pvc-5e717506-9ddf-4d69-8932-515844789c9c 20Gi RWO nfs-sc 80s

# 查看pod是否启动
$ kubectl get pod
test-nfs-mount-pvc-deploy-7565c5586b-lf7tn 1/1 Running 0 109s

# 进去pod的pvc挂载目录生成一个文件,然后把pod删除重建看数据还在就ok了。

Helm3安装ingress-nginx

安装并验证ingress

通过helm安装ingress-nginx

1
2
$ helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
$ helm repo update

新建一个values.yml文件,修改默认values.yml文件中的部分配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$ vim ingress-nginx-values.yaml
controller:
image:
## Keep false as default for now!
chroot: false
registry: docker.io
image: putianhui/controller
tag: 'v1.6.1'
digest: ''
digestChroot: ''
pullPolicy: IfNotPresent
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
nodeSelector:
kubernetes.io/os: linux
ingress: 'true'
kind: DaemonSet
ingressClassResource:
# -- Is this ingressClass enabled or not
enabled: true
# -- Is this the default ingressClass for the cluster
default: true
admissionWebhooks:
patch:
enabled: true
image:
registry: docker.io
image: putianhui/kube-webhook-certgen
tag: v20220916-gd32f8c343
digest: ""

为节点打上ingress可调度的标签

1
$ kubectl label nodes lingang-k8s03-43 ingress=true

通过helm应用安装ingress-nginx到集群中

1
2
3
4
# helm 安装
$ helm install ingress-nginx-controller --version 4.4.3 -f ./ingress-nginx-values.yaml ingress-nginx/ingress-nginx

# 安装成功后会看到节点上有ingress=true标签的节点会创建一个ingress-nginx-controller的pod。

创建ingress规则及deployment资源,测试ingress是否生效

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
$ vim test-ingress.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ingress-svc-demo
spec:
replicas: 2
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- image: putianhui/myapp:v1
name: ingress-svc-demo
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: ingress-svc-demo
spec:
selector:
app: myapp
ports:
- targetPort: 80
port: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demo-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
rules:
- host: "demo.mrpu.cn"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ingress-svc-demo
port:
number: 80

应用清单并访问ingress的域名验证

1
2
3
4
5
6
7
8
9
# 应用清单
$ kubectl apply -f test-ingress.yaml

# 访问ingress域名验证
$ curl demo.mrpu.cn/hostname.html
ingress-svc-demo-58d45b676b-t6d9k

$ curl demo.mrpu.cn/hostname.html
ingress-svc-demo-58d45b676b-xcgkg

配置https访问ingress

首先将对应域名的https证书公钥和私钥base64编码一下,拿到编码后的内容

1
2
3
4
5
# base64编码 tls.crt公钥内容
$ echo -n "tls.crt公钥内容" |base64

# base64编码 tls.key私钥内容
$ echo -n "tls.key私钥内容" |base64

新建一个secret存储https证书编码后的公私钥。

1
2
3
4
5
6
7
8
9
10
$ vim test-ingress-tls-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: demo-mrpu-cn-tls
namespace: default
data:
tls.crt: # 前面base64编码后的公钥内容
tls.key: # 前面base64编码后的私钥内容
type: kubernetes.io/tls

应用并创建证书secret

1
$ kubectl apply -f test-ingress-tls-secret.yaml

修改验证ingress中的部署清单,添加https证书的配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ vim test-ingress.yaml
# 只修改ingress部分,spec配置字段中添加tls配置段,secretName为你刚刚创建的证书secret
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demo-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
tls:
- hosts:
- demo.mrpu.cn
secretName: demo-mrpu-cn-tls
rules:
- host: "demo.mrpu.cn"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ingress-svc-demo
port:
number: 80

重新应用修改后的ingress配置清单

1
$ kubectl apply -f test-ingress.yaml

应用成功之后重新访问ingress域名即可成功访问https站点。

配置ingress访问日志分析

效果图

创建ingress-nginx日志持久化的pvc

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# pvc创建清单,nfs-sc为存储类名称
$ vim ingress-nginx-log-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ingress-nginx-accesslog
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 50Gi
storageClassName: nfs-sc

# 应用清单并创建pvc
$ kubectl apply -f ingress-nginx-log-pvc.yaml

修改ingress-nginx的helm部署配置清单ingress-nginx-values.yaml,添加ingress-nginx的访问日志文件持久化、访问日志的json格式化。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
controller:
... 上面内容和部署配置一致省略....

# 以下内容为添加ingress-nginx控制器访问日志格式化的配置
# 配置初始化容器,为了解决挂载的日志目录权限问题
extraInitContainers:
# 修改挂载后nginx日志的目录权限
- command:
- chown
- -R
- 101:101
- /var/log/nginx
image: alpine:3
name: nginx-data-dir-ownership
volumeMounts:
- mountPath: /var/log/nginx
name: ingress-nginx-accesslog-pvc
securityContext:
runAsUser: 0
# 挂载Nginx访问日志的pvc和挂载路径配置,这里的ingress-nginx-accesslog为前面创建的pvc名称
extraVolumeMounts:
- name: ingress-nginx-accesslog-pvc
mountPath: '/var/log/nginx'
extraVolumes:
- name: ingress-nginx-accesslog-pvc
persistentVolumeClaim:
claimName: ingress-nginx-accesslog
config:
# 日志格式化参考文档https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#access-log-params
http-access-log-path: '/var/log/nginx/httpAccess.log'
# 日志格式参考文档https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/log-format/
log-format-upstream: '{"@timestamp":"$time_iso8601",
"@source":"$server_addr",
"hostname":"$hostname",
"ip":"$remote_addr",
"client":"$remote_addr",
"request_method":"$request_method",
"scheme":"$scheme",
"domain":"$server_name",
"referer":"$http_referer",
"request":"$request_uri",
"args":"$args",
"size":$body_bytes_sent,
"status": $status,
"responsetime":$request_time,
"upstreamtime":"$upstream_response_time",
"upstreamaddr":"$upstream_addr",
"http_user_agent":"$http_user_agent",
"ns":"$namespace",
"ingress_name":"$ingress_name",
"service_name":"$service_name",
"service_port":"$service_port",
"server_port": "$server_port",
"https":"$https"}'

如果你是手动下载ingress-nginx部署清单部署的控制器,请使用下面的方法手动修改configMap配置来设置访问日志的格式化

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# 修改ingress-nginx-controller的configMap
# 在cm 中的 data 配置字段中添加一个 log-format-upstream 的配置字段格式化ingress的访问日志
$ kubectl get cm ingress-nginx-controller-controller -oyaml
apiVersion: v1
data:
log-format-upstream: '{"@timestamp":"$time_iso8601",
"@source":"$server_addr",
"hostname":"$hostname",
"ip":"$remote_addr",
"client":"$remote_addr",
"request_method":"$request_method",
"scheme":"$scheme",
"domain":"$server_name",
"referer":"$http_referer",
"request":"$request_uri",
"args":"$args",
"size":$body_bytes_sent,
"status": $status,
"responsetime":$request_time,
"upstreamtime":"$upstream_response_time",
"upstreamaddr":"$upstream_addr",
"http_user_agent":"$http_user_agent",
"ns":"$namespace",
"ingress_name":"$ingress_name",
"service_name":"$service_name",
"service_port":"$service_port",
"server_port": "$server_port",
"https":"$https"}'
...... 后面配置省略.......

# 修改cm后重建ingress控制器pod后访问日志就使用自定义的日志格式了,注意访问日志的持久化参考上面文档自行解决。

部署临时测试用的es,你已经有存在的es可以忽略此步

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
$ vim es681-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: elasticsearch
spec:
selector:
matchLabels:
app: elasticsearch
replicas: 1
template:
metadata:
labels:
app: elasticsearch
spec:
containers:
- name: elasticsearch
image: elasticsearch:6.8.1
ports:
- containerPort: 9200
name: http
- containerPort: 9300
name: transport
volumeMounts:
- name: data
mountPath: /usr/share/elasticsearch/data
volumes:
- name: data
persistentVolumeClaim:
claimName: elasticsearch-data
---
apiVersion: v1
kind: Service
metadata:
name: elasticsearch
spec:
selector:
app: elasticsearch
ports:
- name: http
port: 9200
targetPort: 9200
nodePort: 31920
- name: transport
port: 9300
targetPort: 9300
nodePort: 31930
type: NodePort
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: nfs-sc


# 应用部署清单到Kubernetes集群,部署成功后可以通过Nodeport的31920来访问到es
$ kubectl apply -f es681-deploy.yaml

部署logstash挂载ingress-nginx持久化日志的pvc,然后收集访问日志格式化后存储到es中

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
$ vim ingress-nginx-log-logstash.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: logstash
spec:
replicas: 1
selector:
matchLabels:
app: logstash
template:
metadata:
labels:
app: logstash
spec:
containers:
- name: logstash
image: logstash:6.8.1
# 这里使用指定logstash配置文件启动,配置文件后面会创建configMap挂载过来
command: ["/bin/sh", "-c", "logstash -f /usr/share/logstash/pipeline/logstash.conf"]
# 这里挂载ingress-nginx访问日志pvc和logstash的配置文件configMap
volumeMounts:
- name: ingress-nginx-accesslog
mountPath: /var/log/nginx/
- name: logstash-config-volume
mountPath: /usr/share/logstash/pipeline/
# 这里定义ingress-nginx的pvc和logstash的configMap获取内容
volumes:
- name: ingress-nginx-accesslog
persistentVolumeClaim:
claimName: ingress-nginx-accesslog
- name: logstash-config-volume
configMap:
name: logstash-config
items:
- key: logstash.conf
path: logstash.conf
---
# 下面的hosts为你的es地址,192.168.1.41:31920为前面临时部署es的nodeport地址
apiVersion: v1
kind: ConfigMap
metadata:
name: logstash-config
data:
logstash.conf: |
input {
file {
path => [ "/var/log/nginx/*.log" ]
ignore_older => 0
codec => json
}
}

filter {
mutate {
convert => [ "status","integer" ]
convert => [ "size","integer" ]
convert => [ "upstreatime","float" ]
remove_field => "message"
}
geoip {
source => "ip"
}
if [user_ua] != "-" {
useragent {
target => "agent"
source => "http_user_agent"
}
}
}

output {
elasticsearch {
hosts => "192.168.1.41:31920"
index => "logstash-nginx-access-%{+YYYY.MM.dd}"
}
}

临时部署grafana测试,你已有grafana请忽略此步骤

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
$ vim grafana-deploy.yaml
apiVersion: v1
kind: Service
metadata:
name: grafana-nginx-log
labels:
k8s-app: grafana
spec:
type: NodePort
ports:
- name: http
port: 3000
targetPort: 3000
nodePort: 30088
selector:
k8s-app: grafana
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana
labels:
k8s-app: grafana
spec:
selector:
matchLabels:
k8s-app: grafana
template:
metadata:
labels:
k8s-app: grafana
spec:
initContainers: #初始化容器,用于修改挂载的存储的文件夹归属组与归属用户
- name: init-file
image: busybox:1.28
imagePullPolicy: IfNotPresent
command: ['chown', '-R', '472:0', '/var/lib/grafana']
volumeMounts:
- name: data
mountPath: /var/lib/grafana
subPath: grafana
containers:
- name: grafana
image: grafana/grafana:8.5.24
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 3000
env: #配置环境变量,设置Grafana 的默认管理员用户名/密码
- name: GF_SECURITY_ADMIN_USER
value: 'admin'
- name: GF_SECURITY_ADMIN_PASSWORD
value: 'admin'
readinessProbe: #就绪探针
failureThreshold: 10
httpGet:
path: /api/health
port: 3000
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 30
livenessProbe: #存活探针
failureThreshold: 10
httpGet:
path: /api/health
port: 3000
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
volumeMounts: #容器挂载配置
- name: data
mountPath: /var/lib/grafana
subPath: grafana
volumes: #共享存储挂载配置
- name: data
persistentVolumeClaim:
claimName: grafana-pvc #指定使用的 PVC
---
# 这里使用了nfs-sc这个存储类来创建grafana使用的pvc
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: grafana-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: nfs-sc

# grafana部署成功后就可以使用nodeport的30088来访问到grafana了,默认账号密码为admin

登录grafana添加Elasticsearch数据源,注意数据源的名称为nginx-es,如果不是这个可能导入仪表盘无法找到数据源(你可以改仪表盘json来搞)

导入grafana仪表盘ID2292打开仪表盘即可出图,也可以使用下面我修改后的仪表盘json导入。

下载我修改好的仪表盘json,导入到grafana即可出图

1
https://www.putianhui.cn/package/script/grafana/ingress-nginx-logs-grafana.json

Helm3安装jumpserver

安装前依赖mysql5.7redis6.0存储类ingress控制器对应域名证书的secret在执行安装前需要提前安装好,文档中不提及安装事宜,直接使用。

通过helm安装Jumpserver

1
2
$ helm repo add jumpserver https://jumpserver.github.io/helm-charts
$ helm repo update

新建jumpserver的values清单

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
$ vim jms-values.yaml
# 这里设置全局的存储类名称,前面文档安装的nfs存储
global:
storageClass: 'nfs-sc'

# 外部的mysql配置信息
externalDatabase:
engine: mysql
host: 192.168.1.43
port: 3306
user: root
password: 'putianhui'
database: jumpserver

# 外部的redis配置信息
externalRedis:
host: 192.168.1.43
port: 6379
password: 'putianhui'

# ingress配置
ingress:
enabled: true
annotations:
# kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/proxy-body-size: '4096m'
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header Upgrade "websocket";
proxy_set_header Connection "Upgrade";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# ingress访问的域名
hosts:
- 'jms.mrpu.cn'
# 域名https配置
tls:
# ingress域名的https证书secret,注意提前创建
- secretName: demo-mrpu-cn-tls
hosts:
- jms.mrpu.cn

core:
enabled: true
config:
secretKey: 'GDPzrwvPD2uc0VKvTkfsruQgMNonhyHBH0JAqadlocgt3ooaQC'
bootstrapToken: 'TUZQD80hlrcpjE8fGcW9B788'
persistence:
# 数据持久化存储类名称
storageClassName: nfs-sc
accessModes:
- ReadWriteMany
size: 200Gi

koko:
enabled: true
persistence:
# 数据持久化存储类名称
storageClassName: nfs-sc
accessModes:
- ReadWriteMany
size: 50Gi

lion:
enabled: true
persistence:
# 数据持久化存储类名称
storageClassName: nfs-sc
accessModes:
- ReadWriteMany
size: 50Gi

magnus:
enabled: true
persistence:
# 数据持久化存储类名称
storageClassName: nfs-sc
accessModes:
- ReadWriteMany
size: 30Gi

omnidb:
persistence:
# 数据持久化存储类名称
storageClassName: nfs-sc
accessModes:
- ReadWriteMany
size: 10Gi

razor:
persistence:
# 数据持久化存储类名称
storageClassName: nfs-sc
accessModes:
- ReadWriteMany
size: 50Gi

web:
enabled: true
persistence:
# 数据持久化存储类名称
storageClassName: nfs-sc
accessModes:
- ReadWriteMany
size: 10Gi

helm通过jms自定义values清单创建服务

1
$ helm  install jumpserver --version 2.28.6 -f ./jms-values.yaml jumpserver/jumpserver

部署成功后浏览器访问前面的ingress域名,默认账号密码是admin/admin

Helm3安装Harbor

安装前依赖``存储类ingress控制器对应域名证书的secret`在执行安装前需要提前安装好,文档中不提及安装事宜,直接使用。

安装Harbor

通过helm安装Harbor仓库

1
2
$ helm repo add harbor https://helm.goharbor.io
$ helm repo update

新建一个values.yml文件,修改默认values.yml文件中的部分配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
expose:
# 服务暴露的类型使用ingress域名
type: ingress
# tls的配置
tls:
# 是否开启tls
enabled: true
# secret的类型为secret,该类型为自己手动创建证书的secret
certSource: secret
secret:
# 这里指定手动创建https的secret名称,因为我https证书是通配证书,所有这里创建一个secret,其他域名都用这一个secret就行了
secretName: 'mrpu-cn-tls'
notarySecretName: 'mrpu-cn-tls'
# ingress配置
ingress:
hosts:
# core使用的域名,这个域名未来也是前端访问的入口
core: harbor.mrpu.cn
# notary使用的域名
notary: notary-harbor.mrpu.cn
# default即可
controller: default
# 你的ingress控制器名称,也可以为空,使用集群的默认ingress控制器
className: 'nginx'
annotations:
ingress.kubernetes.io/ssl-redirect: 'true'
ingress.kubernetes.io/proxy-body-size: '0'
nginx.ingress.kubernetes.io/ssl-redirect: 'true'
nginx.ingress.kubernetes.io/proxy-body-size: '0'

# 这里的域名和ingress中的core域名对应
externalURL: https://harbor.mrpu.cn

ipFamily:
ipv6:
enabled: false
# 数据持久化配置
persistence:
enabled: true
resourcePolicy: 'keep'
# pvc配置
persistentVolumeClaim:
registry:
# 存储类名称
storageClass: 'nfs-sc'
accessMode: ReadWriteOnce
size: 100Gi
chartmuseum:
# 存储类名称
storageClass: 'nfs-sc'
accessMode: ReadWriteOnce
size: 50Gi
jobservice:
jobLog:
# 存储类名称
storageClass: 'nfs-sc'
accessMode: ReadWriteOnce
size: 10Gi
scanDataExports:
# 存储类名称
storageClass: 'nfs-sc'
accessMode: ReadWriteOnce
size: 10Gi
database:
# 存储类名称
storageClass: 'nfs-sc'
accessMode: ReadWriteOnce
size: 100Gi
redis:
# 存储类名称
storageClass: 'nfs-sc'
accessMode: ReadWriteOnce
size: 50Gi
trivy:
# 存储类名称
storageClass: 'nfs-sc'
accessMode: ReadWriteOnce
size: 50Gi
imageChartStorage:
disableredirect: false
type: filesystem
filesystem:
rootdirectory: /storage

# Harbor的密码
harborAdminPassword: 'Harbor12345'

# 开启监控指标
metrics:
enabled: true

helm通过Harbor自定义values清单创建服务到集群

1
$ helm  install harbor --version 1.11.0 -f ./harbor-values.yaml harbor/harbor

清单应用到集群等pod都启动成功后,就可以通过externalURL域名访问到harbor了(前提是需要先将域名解析到ingress的控制器节点ip),账号密码:admin/清单中定义的harborAdminPassword

验证推拉容器镜像

使用docker尝试登录刚刚创建的harbor仓库,然后推送一个镜像尝试。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 登录到harbor
$ docker login https://harbor.mrpu.cn
Username: admin
Password: harbor的密码
Login Succeeded

# 我这里拿一个 putianhui/myapp:v1 的镜像,然后重新打tag提交到harbor。
$ docker pull putianhui/myapp:v1

# 将拉取的镜像重新打tag为私有镜像仓库的地址以及名称tag
$ docker tag putianhui/myapp:v1 harbor.mrpu.cn/library/myapp:v1

# 将打为私有镜像仓库地址tag的镜像推送到私有Harbor中,当推送成功之后就可以在harbor中看到刚刚推送的镜像了
$ docker push harbor.mrpu.cn/library/myapp:v1

将本地的镜像删除,然后重新去harbor服务器去拉取镜像尝试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 查看镜像列表
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
harbor.mrpu.cn/library/myapp v1 d4a5e0eaa84f 4 years ago 15.5MB

# 删除镜像
$ docker rmi harbor.mrpu.cn/library/myapp:v1

# 重新查看镜像列表为空的
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE

# 尝试拉取harbor服务器上的镜像(记得提前登录私有harbor仓库哦)
$ docker pull harbor.mrpu.cn/library/myapp:v1

验证推拉helm的chart包

先本地安装helm的push插件

1
2
3
# 在线安装
$ yum install -y git
$ helm plugin install https://github.com/chartmuseum/helm-push

Helm添加Harbor默认的chart仓库

1
2
3
4
5
6
7
# 添加仓库
$ helm repo add myrepo --username=admin --password=Harbor12345 https://harbor.mrpu.cn/chartrepo

# 查看仓库列表
$ helm repo list
NAME URL
myrepo https://harbor.mrpu.cn/chartrepo

helm本地创建demo的chart项目并推送到Harbor的chart仓库

1
2
3
4
5
6
7
8
9
10
11
# 创建一个helm的demo
$ helm create mydemo

# 把刚刚创建的mydeo项目推送到Harbor的myrepo仓库中
$ helm cm-push --username=admin --password=Harbor12345 mydemo myrepo

# 上传成功后更新一下repo信息,然后就能搜到刚刚上传的项目了
$ helm repo update
$ helm search repo demo
NAME CHART VERSION APP VERSION DESCRIPTION
myrepo/library/mydemo 0.1.0 1.16.0 A Helm chart for Kubernetes

也可以在Harbor平台里面看到刚刚推送的项目,项目libraryHelm Chartsmydemo

Helm3安装Jenkins

安装Jenkins

添加Jenkins的helm仓库并更新,官方chart文档

1
2
$ helm repo add jenkinsci https://charts.jenkins.io
$ helm repo update

新建Jenkins的values.yaml清单

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
$ vim jenkins-values.yaml
controller:
# 管理员的账号和密码配置
adminUser: 'admin'
adminPassword: 'putianhui'

# 不开启ingress,因为这个chart版本的ingress创建会有点问题,后面会手动去创建Jenkins用的ingress
ingress:
enabled: false

# 默认安装的Jenkins插件
installPlugins:
- kubernetes:3845.va_9823979a_744
- workflow-aggregator:590.v6a_d052e5a_a_b_5
- git:4.13.0
- git-parameter:0.9.18
- maven-plugin:3.20
- config-file-provider:3.11
- extended-choice-parameter:359.v35dcfdd0c20d
- description-setter:1.10
- build-name-setter:2.2.0
- configuration-as-code:1569.vb_72405b_80249
- localization-zh-cn:1.0.24
- gitlab-plugin:1.7.4

# Jenkins-slave的配置
agent:
enabled: true
jenkinsUrl: http://jenkins.default.svc.cluster.local:8080
jenkinsTunnel: jenkins-agent.default.svc.cluster.local:50000
namespace: default

# 数据持久化配置
persistence:
enabled: true
storageClass: 'nfs-sc'
accessMode: 'ReadWriteOnce'
size: '100Gi'

应用并安装

1
$ helm install jenkins --version 4.3.0 -f ./jenkins-values.yaml jenkinsci/jenkins

上面把ingress关闭了,因为这个版本的chart包在v1.24.6集群上面创建ingress会出错,所以不让helm创建,我们服务跑起来之后手动创建ingress即可

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# 创建Jenkins-ingress清单
$ vim jenkins-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: jenkins-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
# ingress类名,我这里用nginx
ingressClassName: nginx
rules:
# ingress的域名
- host: jenkins.mrpu.cn
http:
paths:
- backend:
service:
# svc的名称及端口号
name: jenkins
port:
number: 8080
path: /
pathType: Prefix
# https配置,注意提前准备好证书的secret
tls:
- hosts:
# https的域名
- jenkins.mrpu.cn
# 域名使用的https证书,需要提前创建secret
secretName: mrpu-cn-tls

应用ingress清单并创建ingress

1
$ kubectl apply -f ./jenkins-ingress.yaml

当helm安装的pod都成功启动之后,就可以用上面的ingress域名jenkins.mrpu.cn去访问到Jenkins了,账号密码为jenkins-values.yaml文件中设置的admin/putianhui

安装成功后需要做一下配置:

1、系统管理全局安全管理Host Key Verification Strategy—改为No Verification

2、系统管理全局安全管理代理指定端口50000

配置Kubernetes-cloud

登录Kubernetes集群中的一台节点,通过~/.kube/config文件来获取集群的ca证书以及客户端证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
cd ~/.kube
cad=`cat ./config |grep certificate-authority-data|awk -F : '{print $2}'`
ccd=`cat ./config |grep client-certificate-data|awk -F : '{print $2}'`
ckd=`cat ./config |grep client-key-data|awk -F : '{print $2}'`

# 其中 certificate-authority-data 对应的值转换成 ca.crt文件
echo $cad | base64 -d > ./ca.crt

# client-certificate-data 对应的值我们转换成 client.crt 文件
echo $ccd | base64 -d > ./client.crt

# client-key-data 对应的值转换成./client.key文件
echo $ckd | base64 -d > ./client.key

# 获取Kubernetes 服务证书Key的ca内容,这里的内容后面要填在Jenkins配置中
cat ./ca.crt
-----BEGIN CERTIFICATE-----
...此处省略一万字...
6no=
-----END CERTIFICATE-----

将上面获取的客户端证书转换成Jenkins链接k8s使用的p12证书文件

1
2
3
$ openssl pkcs12 -export -out ./client.pfx -inkey ./client.key -in ./client.crt -certfile ./ca.crt
$ ls
cache ca.crt client.crt client.key client.pfx config

将前面一步生成的client.pfxp12格式证书添加到Jenkins的凭据中,系统管理Manage CredentialsSystem全局凭据Add Credentials

添加Kubernetes集群,系统管理节点管理Configure CloudsKubernetes Cloud Details

注意:如果在连接测试时提示如下错误信息,请升级Kubernetes的插件版本到最新。

No httpclient implementations found on the context classloader, please ensure your classpath includes an implementation jar

添加Jenkins-agent的配置信息

验证Jenkins-agent

新建一个测试用的Pipeline任务,验证Jenkins-slave是否ok,新建任务名称随意流水线确定

添加如何Pipeline脚本,指定使用从节点去运行,这里的从节点为Kubernetes集群中的pod,也就是前面配置的标签jenkins-jnlp

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
pipeline {
agent {
kubernetes {
// cloud中pod template配置的agent标签
label "jenkins-jnlp"
}
}

stages {
stage('2.Maven代码编译') {
steps {
sh '''
echo "测试Jenkins-agent是否ok"
'''
}
}
}
}

点击立即构建,让任务跑一下看看是否运行成功,运行成功后看到以下日志即ok

常用的Pipeline模板

讲Pod template参考的文档

讲Jenkins-模板使用继承实现多容器多环境的构建文档

常用的Jenkins-slave的流水线Pipeline模板文档

声明式模板

方式一:将Pod Template模板定义在Jenkins的Pipeline中

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
pipeline {
agent {
kubernetes {
cloud 'kubernetes'
namespace 'default'
workingDir '/home/jenkins/agent'
// 继承模板
inheritFrom 'jenkins-slave'
defaultContainer 'jnlp'
// yamlFile 'KubernetesPod.yaml'
yaml """\
apiVersion:
kind: Pod
metadata:
labels:
jenkins-jnlp: "true"
jenkins/label: "jenkins-slave"
spec:
containers:
- image: "jenkins/inbound-agent:4.11.2-4"
imagePullPolicy: "IfNotPresent"
name: "jnlp"
securityContext:
privileged: true
tty: true
volumeMounts:
- mountPath: "/tmp"
name: "volume-0"
readOnly: false
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
- image: "putianhui/myapp:v1"
imagePullPolicy: "IfNotPresent"
name: "myapp"
resources: {}
securityContext:
privileged: true
tty: true
volumeMounts:
- mountPath: "/tmp"
name: "volume-0"
readOnly: false
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
restartPolicy: "Never"
volumes:
- name: "volume-0"
persistentVolumeClaim:
claimName: "jumpserver-jms-web-logs"
readOnly: false
- emptyDir:
medium: ""
name: "workspace-volume"
""".stripIndent()
}
}
stages {
stage ('git-checkout') {
steps {
echo "declarative Pipeline - kubernetes"
container('jnlp') {
sh """
echo "git checkout"
touch git.txt
df -hT
"""
}
}
}

stage ('mvn编译') {
steps {
container('myapp') {
sh """
ls -l
"""
}
}
}

stage ('restart-Myapp') {
steps {
container('myapp') {
sh """
echo "myapp"
df -hT
"""
}
}
}
}
}

方式二:通过系统管理–配置集群–Pod Template中配置pod模板jenkins-podTemplate-1,里面有两个容器,一个为Jenkins-agent容器用于连接Jenkins-master,另外一个为测试用的容器。

注意:jnlp容器运行的命令默认貌似是sleep,记得为空,不然会覆盖容器的执行命令,导致jnlp容器无法连接到Jenkins-master,从而导致无法执行任务。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
pipeline {
agent {
// 匹配管理工具中创建的Pod Template名称对应的标签列表值
label 'jenkins-podTemplate-1-jnlp'
}
stages {

stage ('git获取代码') {
steps {
// 此步骤在Pod Template中哪个容器内执行,因为pod Template中定义了两个容器,此步骤在容器名称为jnlp这个容器中执行。
container('jnlp') {
sh """
echo "git checkout"
touch git.txt
df -hT
"""
}
}
}

stage ('mvn编译') {
steps {
// 此步骤在pod Template中名为myapp的容器中执行
container('myapp') {
sh """
ls -l
"""
}
}
}

stage ('restart-Myapp') {
steps {
container('myapp') {
sh """
echo "myapp"
df -hT
"""
}
}
}
}
}

脚本式模板

方式一:将Pod Template模板定义在Jenkins的Pipeline中

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
// 定义pod Template
podTemplate(name: 'jenkins-slave', cloud: 'kubernetes',
namespace: 'default', label: 'jenkins-slave',serviceAccount: 'default',
containers: [
containerTemplate(
name: 'jnlp',
image: 'jenkins/inbound-agent:4.11.2-4',
args: '${computer.jnlpmac} ${computer.name}',
ttyEnabled: true,
privileged: true,
alwaysPullImage:false,
),
containerTemplate(
name: 'myapp',
image: 'putianhui/myapp:v1',
ttyEnabled: true,
privileged: true,
alwaysPullImage:false,
)
],
volumes: [
persistentVolumeClaim(mountPath: '/tmp/', claimName: 'jumpserver-jms-web-logs')
]
)

// 定义Jenkins的Pipeline
{
// 这里匹配前面定义的Pod Template标签
node('jenkins-slave') {

stage('git-checkout') {
// 在Pod Template中的jnlp容器中执行以下步骤
container('jnlp') {
sh """
echo "git checkout"
echo "123456" > ~/git.txt
"""
}
}

stage('mvn-package') {
container('jnlp') {
sh """
echo "mvn --- $HOSTNAME"
"""
}
}

stage('restart-Myapp') {
// 在Pod Template中的myapp容器中执行以下步骤
container('myapp') {
sh """
echo "myapp --- $HOSTNAME"
"""
}
}
}
}

方式二:直接使用系统管理中创建的Pod Template模板,根据创建时指定的模板标签去匹配。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
// 定义Jenkins的Pipeline
podTemplate{
// 这里匹配前面系统管理中配置的Pod Template标签
node('jenkins-podTemplate-1-jnlp') {

stage('git-checkout') {
// 在Pod Template中的jnlp容器中执行以下步骤
container('jnlp') {
sh """
echo "git checkout"
echo "123456" > ~/git.txt
"""
}
}

stage('mvn-package') {
container('jnlp') {
sh """
echo "mvn --- $HOSTNAME"
"""
}
}

stage('restart-Myapp') {
// 在Pod Template中的myapp容器中执行以下步骤
container('myapp') {
sh """
echo "myapp --- $HOSTNAME"
cat ~/git.txt
"""
}
}
}
}

制作podTemplate中用的工具镜像

制作tools镜像

安装的工具:docker、crictl、kubectl、helm、git、net-tools

创建docker镜像目录并下载常用的命令、生成客户端的配置文件到当前目录下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ mkdir tools-image && cd tools-image/

#下载helm
$ wget https://get.helm.sh/helm-v3.11.0-linux-amd64.tar.gz && tar xzvf helm-v3.11.0-linux-amd64.tar.gz && mv linux-amd64/helm ./

# 下载kubectl
$ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && chmod +x kubectl

# 下载crictl
$ wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.26.0/crictl-v1.26.0-linux-amd64.tar.gz && tar xzvf crictl-v1.26.0-linux-amd64.tar.gz

# 复制当前机器的docker命令二进制到本地目录一份
$ cp /usr/bin/docker .

# 准备crictl的客户端配置文件
$ cat > ./crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 2
debug: false
pull-image-on-create: true
EOF

新建Dockerfile文件内容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
$ cat >Dockerfile<<EOF
# 基本镜像,ubuntu:20.04
FROM ubuntu:20.04

# 跳过时区选择(必须执行,否则后续构建镜像时会卡死到时区选择)
ENV DEBIAN_FRONTEND=noninteractive

# 编码格式
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

# 指定用户
USER root

# 复制本地下载的工具二进制到容器内
COPY ./crictl ./kubectl ./helm ./docker /usr/bin/

# 生成crictl的配置文件
COPY ./crictl.yaml /etc/crictl.yaml

# 切换到国内镜像源(镜像、容器内部的)、安装常用的软件、安装docker-cli
RUN set -x; apt-get update \
&& apt-get install -y sudo curl vim git net-tools tzdata \
&& ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
&& rm -rf /var/lib/apt/lists/*

CMD ["sleep", "9999999"]
EOF

构建工具镜像

1
2
# 我这里构建成功后推送到我自己的dockerHub仓库中,你们也可以推送到自己的仓库中,或者后面使用时用我仓库中的镜像也可以
$ docker build -t putianhui/jenkins-tools:v1 .

镜像在使用时需要注意一下几点:

1、在使用helm、kubectl工具时,需要将Kubernetes的客户端配置文件挂载到容器的/root/.kube/config文件

2、在使用crictl工具时,需要将containerd的sock文件挂载到容器的/run/containerd/containerd.sock文件

3、在使用docker工具时,需要将docker的sock文件挂载到容器的/var/run/docker.sock

Pod模板多容器构建Pipeline测试

  1. 每次构建任务会在Kubernetes集群中启动一个Jenkins-agent的pod来运行这个构建任务。
  2. 在系统管理–节点管理中配置一个通用的pod模板,里面包含两个容器,一个是Jenkins-jnlp的容器,一个是Jenkins-tools的容器。
  3. 在对应项目的Jenkinsfile中,指定对应语言构建的容器去构建,然后把通用的pod模板通过继承的方式加载进来,这样的话在pipeline中代码编译步骤中就使用对应语言的容器去构建制品,docker镜像的打包或者helm部署时就用工具容器去执行操作。

第一步:我们先在系统管理–节点管理中添加一个pod-template,这个模板中共有两个容器,一个是Jenkins-jnlp的容器,一个是Jenkins-tools容器(刚刚构建的镜像)。

由于工具容器中使用了docker、crictl我们需要把这俩服务的sock文件挂载到容器的对应位置,还使用了helm、kubectl这俩工具我们需要把kubernetes的客户端配置文件~/.kube/config挂载到容器中。

第二步:创建一个job,使用下面的jenkinsFile内容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
pipeline {
agent {
kubernetes {
cloud 'kubernetes'
namespace 'default'
workingDir '/home/jenkins/agent'
// 继承的通用模板,这里写刚刚系统管理中创建的通用模板名称
inheritFrom 'jenkins-podTemplate-1'
// 默认的容器
defaultContainer 'jnlp'
// 可以指定yaml文件名称也可以直接yaml内容
// yamlFile 'KubernetesPod.yaml'
// yaml中我指定了一个是golang镜像一个是maven镜像,这里可以根据当前任务构建的语言选择一个语言镜像即可
yaml """\
apiVersion: v1
kind: Pod
metadata:
labels:
jenkins-jnlp: "true"
jenkins/label: "jenkins-slave"
spec:
containers:
- image: "maven:alpine"
imagePullPolicy: "IfNotPresent"
name: "maven"
command:
- cat
tty: true
- image: "golang:1.16.5"
imagePullPolicy: "IfNotPresent"
name: "golang"
command:
- sleep
args:
- 99d
""".stripIndent()
}
}
stages {
stage ('查看 jnlp的java版本') {
steps {
container('jnlp') {
sh """
id;java -version
pwd
touch "jnlp-version.txt"
"""
}
}
}

stage ('查看golang版本') {
steps {
container('golang') {
sh """
go version
pwd
ls -l
touch "golang-version.txt"
"""
}
}
}

stage ('查看maven版本') {
steps {
container('maven') {
sh """
id;mvn -version
pwd
ls -l
touch "maven-version.txt"
"""
}
}
}

stage ('查看tools工具信息') {
steps {
container('tools') {
sh """
kubectl get pod
crictl images
#docker images
helm list
pwd
touch "tools-version.txt"
ls -l
"""
}
}
}
}
}

上面Jenkinsfile的任务运行时对应的Jenkins-slave的pod会有四个容器,包含通用pod模板的jnlp和tools容器,还有yaml指定的golang和maven构建的语言容器,通过继承的方式可以很方便的组合。

假如我要构建java的应用,那么就可以yaml中继承通用的pod模板,然后yaml中只需要指定maven的构建语言容器即可,构建node应用是同理,继承通用pod模板,然后yaml中只需要去指定nodejs的容器即可。

各个流水线阶段需要构建代码时就指定构建语言的容器、需要打包docker镜像时就用tools容器、需要发布时也还是指定tools容器通过helm发布即可。

实践验证

1、创建一个ssh公私钥,将公钥上传至gitlab个人中的ssh秘钥中。

2、将ssh私钥的内容拿出来在Jenkins上创建凭据,系统管理Manage CredentialsSystem全局凭据Add Credentials

3、通过下面的Pipeline脚本构建流水线测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
// gitlab的ssh在Jenkins的凭据id,前面创建的,注意替换成自己的
def git_auth = "c30ba02f-f182-476d-87fe-983bc56969b8"

// 当前项目的git仓库地址,注意替换成自己的
def git_url = "ssh://git@git.mrpu.cn:32022/ops/devops-test.git"

// 定义部署的环境
env.Deployment="dev"
// 定义构建后jar包前缀变量名称
env.jar_name="digital-assetd"
// Harbor的仓库地址
env.harbor_host="harbor.mrpu.cn/library"


pipeline {
// agent配置
agent {
kubernetes {
cloud 'kubernetes'
namespace 'default'
workingDir '/home/jenkins/agent'
// 继承的通用模板,这里写刚刚系统管理中创建的通用模板名称
inheritFrom 'jenkins-podTemplate-1'
// 默认的容器
defaultContainer 'jnlp'
// 可以指定yaml文件名称也可以直接yaml内容
// yamlFile 'KubernetesPod.yaml'
yaml """\
apiVersion: v1
kind: Pod
metadata:
labels:
jenkins-jnlp: "true"
jenkins/label: "jenkins-slave"
spec:
containers:
- image: "maven:alpine"
imagePullPolicy: "IfNotPresent"
name: "maven"
command:
- cat
tty: true
""".stripIndent()
}
}

parameters {
gitParameter name: 'git_branch',
type: 'PT_BRANCH_TAG',
branchFilter: 'origin/(.*)',
defaultValue: 'main',
selectedValue: 'DEFAULT',
sortMode: 'DESCENDING_SMART',
description: '选择一个发布代码的分支'
}

stages {

stage ('1、拉取代码') {
steps {
container('tools') {
script {
checkout([$class: 'GitSCM', branches: [[name: "${git_branch}"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_url}"]]])
}
}
}
}

stage ('2、maven构建制品') {
steps {
container('maven') {
sh """
id;mvn -version
pwd
ls -l
source /etc/profile &> /dev/null
#rm -rf /root/.m2/repository/*
# 构建mvn制品
#mvn clean package -Dmaven.test.skip=true -P ${Deployment}
"""
}
}
}

stage('3.构建镜像&推送仓库') {
steps {
sh '''
docker_img_name="${harbor_host}/${jar_name}:${BUILD_NUMBER}"
echo \$docker_img_name

#docker build -t \$docker_img_name .
#docker push \$docker_img_name
'''
}
}

stage ('4、Helm部署') {
steps {
container('tools') {
sh """
kubectl get pod
crictl images
#docker images
helm list
pwd
touch "tools-version.txt"
ls -l
git version
"""
}
}
}

}
}

Helm3安装Gitlab

安装Gitlab添加Gitlab的helm仓库并更新

1
2
$ helm repo add  gitlab    https://charts.gitlab.io 
$ helm repo update

新建Gitlab的values.yaml清单,下面的内容是我修改后的。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
$ vim gitlab-values.yaml

# 全局配置 https://docs.gitlab.com/charts/charts/globals.html
global:
# Gitlab的版本
edition: ce

# 不创建application
application:
create: false
allowClusterRoles: true

# 这里的shell端口控制在项目的git克隆地址中反应的端口号
shell:
port: 32022

# 访问域名的一些配置
hosts:
# 顶级域名
domain: mrpu.cn
# 是否开启https,为true时注意提前创建一个通配的证书secret
https: true
ssh: git.mrpu.cn
gitlab:
name: git.mrpu.cn
https: true
minio:
name: minio-git.mrpu.cn
registry:
name: registry-git.mrpu.cn
smartcard:
name: smartcard-git.mrpu.cn
kas:
name: kas-git.mrpu.cn
pages:
name: pages-git.mrpu.cn

# ingress配置
ingress:
# 是否开启ingress的创建
enabled: true
# ingress控制器的版本
apiVersion: 'networking.k8s.io/v1'
# 关闭自动创建证书,使用我自己创建的通配证书secret
configureCertmanager: false
# ingress的类
provider: nginx
class: nginx
# https配置
tls:
# 是否开启https
enabled: true
# 通配的https证书secret
secretName: 'mrpu-cn-tls'
path: /
pathType: Prefix

# 配置时区
time_zone: Asia/Shanghai

# 更新检查的配置,这里关闭
upgradeCheck:
enabled: false

# 自动证书管理的配置,这里关闭,自己手动创建secret
certmanager:
installCRDs: false
install: false

# 自动安装nginx-ingress控制器,集群中我已经安装过了,这里关闭。
nginx-ingress:
enabled: false

# Prometheus的配置
prometheus:
install: false

# Gitlab下的子chart资源配置 https://docs.gitlab.com/charts/charts/gitlab/
gitlab:
# gitlab-shell 子chart配置
gitlab-shell:
enabled: true
# 控制Gitlab-shell的svc暴露方式,我这里选择nodeport
service:
type: 'NodePort'
nodePort: 32022

# gitaly子chart配置
gitaly:
persistence:
enable: true
accessMode: 'ReadWriteOnce'
size: '100Gi'
storageClass: 'nfs-sc'

# 需要数据持久化的几个服务配置
# minio配置:https://docs.gitlab.com/charts/charts/minio/index.html
minio:
persistence:
enable: true
accessMode: 'ReadWriteOnce'
size: '100Gi'
storageClass: 'nfs-sc'

# redis数据持久化配置
redis:
install: true
cluster:
enabled: false
master:
persistence:
enable: true
size: '100Gi'
storageClass: 'nfs-sc'

# postgresql数据持久化配置
postgresql:
persistence:
enable: true
size: '100Gi'
storageClass: 'nfs-sc'

通过helm安装Gitlab

1
$ helm install gitlab --version 6.8.1 -f ./gitlab-values.yaml gitlab/gitlab

获取root的登录密码

1
2
# helm安装成功后,所有pod启动就绪就可以通过ingress的git域名访问到页面,默认账号是:root,密码通过以下命令获取
$ kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode

Helm3安装PrometheusAlert

由于需要使用mysql存储数据,首先需要安装数据库然后创建数据库,服务的pod启动后再把对应sql导入进来

1
2
3
# 创建数据库
$ mysql -uroot -p
mysql> CREATE DATABASE prometheusalert CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

下载官方helm的安装包

1
2
3
$ wget https://github.com/feiyu563/PrometheusAlert/releases/download/v4.8.2/helm.zip
$ unzip helm.zip
$ cd helm/prometheusalert

修改helm包中的app.conf配置文件内容,以下展示内容为修改后的内容,其他默认即可,当app.conf配置变更时需要重新修改helm目录下的app.conf配置文件,然后再helm upgrade更新到集群中。更新配置后可能pod需要重建才能识别到。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# 切换到配置目录并修改配置文件
$ cd config
$ vim app.conf

appname = PrometheusAlert
#登录用户名
login_user=admin
#登录密码
login_password=putianhui
# mysql数据库配置
db_driver=mysql
db_host=192.168.1.43
db_port=3306
db_user=root
db_password=putianhui
db_name=prometheusalert
#是否开启告警记录 0为关闭,1为开启
AlertRecord=1
#是否开启告警记录定时删除 0为关闭,1为开启
RecordLive=1

# 告警渠道的开关
open-dingding=0
open-weixin=0
open-feishu=1
# 默认使用的飞书机器人url,在自定义模板不指定飞书机器人url时就会用这个默认的去发送告警。
fsurl=https://open.feishu.cn/open-apis/bot/hook/xxxxxxxxx

修改helm的values.yaml文件,以下展示的为修改后的内容,其他不展示为默认

1
2
3
4
5
6
7
$ cd ../ && vim values.yaml
# 这里先禁用ingress,后面再手动创建ingress
ingress:
enabled: false
image:
repository: feiyu563/prometheus-alert:v4.8.2
pullPolicy: IfNotPresent

通过helm执行安装到Kubernetes集群中

1
$ helm  install prometheus-alert ./

创建ingress清单并应用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
$ vim prometheusalert-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prometheusalert-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
# ingress类名,我这里用nginx
ingressClassName: nginx
rules:
- host: prometheusalert.mrpu.cn
http:
paths:
- backend:
service:
# svc的名称及端口号
name: prometheus-alert-prometheusalert
port:
number: 8080
path: /
pathType: Prefix
# https配置,注意提前准备好证书的secret
tls:
- hosts:
# https的域名
- prometheusalert.mrpu.cn
# 域名使用的https证书,需要提前创建secret
secretName: mrpu-cn-tls

# 应用ingress的创建清单
$ kubectl apply -f ./prometheusalert-ingress.yaml

将模板的sql模板文件导入到prometheusalert的mysql数据库中

1
2
3
$ mysql -uroot -p
mysql> use prometheusalert;
mysql> source /tmp/init.sql

做到此步骤即可通过手动创建的ingress域名去访问到平台了https://prometheusalert.mrpu.cn,账号密码为app.conf配置文件中的admin/putianhui

为了后面我们Kube-Promethues的Alertmanager对接PrometheusAlert平台作为告警中心,我们需要新建往飞书发送Prometheus告警信息的自定义模板。

自定义模板内容如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
{{ $var := .externalURL}}{{ range $k,$v:=.alerts }}
{{if eq $v.status "resolved"}}**[Prometheus恢复信息]({{$v.generatorURL}})**
**======== 💚 告警恢复 💚 ========**
🙋 告警名称: [{{$v.labels.alertname}}]({{$v.generatorURL}})
🛎️ 告警级别:{{$v.labels.severity}}
⏱️ 开始时间:{{TimeFormat $v.startsAt "2006-01-02 15:04:05"}}
⏱️ 结束时间:{{TimeFormat $v.endsAt "2006-01-02 15:04:05"}}
📋 NameSpace: {{$v.labels.namespace}}
🏷️ Job名称: {{$v.labels.job}}
👀 故障实例:{{$v.labels.instance}}
🚀 Pod名称: {{$v.labels.pod}}
🌕 Svc名称: {{$v.labels.service}}

{{else}}**[Prometheus告警信息]({{$v.generatorURL}})**
**======== 💔 故障告警 💔 ========**
🙋 告警名称: [{{$v.labels.alertname}}]({{$v.generatorURL}})
🛎️ 告警级别:{{$v.labels.severity}}
⏱️ 开始时间:{{TimeFormat $v.startsAt "2006-01-02 15:04:05"}}
{{if eq $v.endsAt "0001-01-01T00:00:00Z"}}⏱️ 结束时间: {{ printf "告警仍然存在!"}}
{{else}}⏱️ 结束时间:{{TimeFormat $v.endsAt "2006-01-02 15:04:05"}}{{end}}📋 NameSpace: {{$v.labels.namespace}}
🏷️ Job名称: {{$v.labels.job}}
👀 故障实例:{{$v.labels.instance}}
🚀 Pod名称: {{$v.labels.pod}}
🌕 Svc名称: {{$v.labels.service}}
{{ $urimsg:=""}}{{ range $key,$value:=$v.labels}}{{$urimsg = print $urimsg $key "=\"" $value "\"," }}{{end}}{{$data:=urlquery $urimsg }}[【点我屏蔽该告警】]({{$var}}/#/silences/new?filter=%7B{{SplitString $data 0 -3}}%7D)

📝 **Alert Detail**:
**{{$v.annotations.description}}**
{{end}}{{ end }}

测试模版是否正常,使用下面的测试Json数据填写到消息协议JSON内容中,点击模板测试验证飞书是否收到格式化的告警消息。

1
{"receiver":"default-alert","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"etcdDatabaseHighFragmentationRatio","endpoint":"http-metrics","instance":"192.168.1.41:2381","job":"kube-etcd","namespace":"kube-system","pod":"etcd-192.168.1.41","prometheus":"default/kube-prometheus-kube-prome-prometheus","service":"kube-prometheus-kube-prome-kube-etcd","severity":"warning"},"annotations":{"description":"etcd cluster \"kube-etcd\": database size in use on instance 192.168.1.41:2381 is 48.65% of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"startsAt":"2023-02-10T02:31:19.067Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":"http://prometheus.mrpu.cn/graph?g0.expr=%28last_over_time%28etcd_mvcc_db_total_size_in_use_in_bytes%5B5m%5D%29+%2F+last_over_time%28etcd_mvcc_db_total_size_in_bytes%5B5m%5D%29%29+%3C+0.5\u0026g0.tab=1","fingerprint":"ef8e3b0c3c4071e2"}],"groupLabels":{"alertname":"etcdDatabaseHighFragmentationRatio"},"commonLabels":{"alertname":"etcdDatabaseHighFragmentationRatio","endpoint":"http-metrics","instance":"192.168.1.41:2381","job":"kube-etcd","namespace":"kube-system","pod":"etcd-192.168.1.41","prometheus":"default/kube-prometheus-kube-prome-prometheus","service":"kube-prometheus-kube-prome-kube-etcd","severity":"warning"},"commonAnnotations":{"description":"etcd cluster \"kube-etcd\": database size in use on instance 192.168.1.41:2381 is 48.65% of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space.","runbook_url":"https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation","summary":"etcd database size in use is less than 50% of the actual allocated storage."},"externalURL":"http://alertmanager.mrpu.cn","version":"4","groupKey":"{}:{alertname=\"etcdDatabaseHighFragmentationRatio\"}","truncatedAlerts":0}

找到我们刚刚添加的自定义模板webhook通知地址,记录下来等下在alertmanager里面添加webhook_configs时要用到。

Helm3安装kube-Promethues

安装kube-Promethues

通过Helm去安装prometheus-operator,各个组件的兼容性在这里查看,详细的helm安装指南在官方这里可以找到。

添加helm仓库并更新

1
2
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update

新建kube-promethues-values.yamlvalues.yaml清单

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
$ vim kube-promethues-values.yaml

alertmanager:
enabled: true
config:
global:
resolve_timeout: 5m
route:
# 这里改成按告警名称去分组告警
group_by: ['alertname']
group_wait: 30s
group_interval: 1m
repeat_interval: 5m
# 默认的告警渠道,和下面新建的Prometheusalert飞书告警渠道名称对应
receiver: 'default-alert'
routes:
- receiver: 'null'
matchers:
- alertname =~ "InfoInhibitor|Watchdog"
receivers:
- name: 'null'
# 新增的Prometheusalert飞书渠道告警
- name: 'default-alert'
# 调用Prometheusalert告警平台告警,使用我们的自定义模板:prometheus-fs
webhook_configs:
- url: 'http://prometheus-alert-prometheusalert.default:8080/prometheusalert?type=fs&tpl=prometheus-fs'
templates:
- '/etc/alertmanager/config/*.tmpl'

ingress:
# 关闭ingress默认的ingress创建会有点问题,等下手动创建
enabled: false

## ref: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#alertmanagerspec
alertmanagerSpec:
image:
registry: quay.io
repository: prometheus/alertmanager
tag: v0.25.0
sha: ''
# 定义告警发出时外部可以访问到Alertmanager的地址,如果外部通过ingress访问,这里就填Alertmanager的域名
externalUrl: 'http://alertmanager.mrpu.cn'
## ref: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/storage.md
storage:
volumeClaimTemplate:
spec:
storageClassName: nfs-sc
accessModes: ['ReadWriteOnce']
resources:
requests:
storage: 100Gi

prometheus:
enabled: true

ingress:
# 关闭ingress默认的ingress创建会有点问题,等下手动创建
enabled: false

prometheusSpec:
image:
registry: quay.io
repository: prometheus/prometheus
tag: v2.41.0
sha: ''
# 定义告警发出时外部可以访问到Prometheus的地址,如果外部通过ingress访问,这里就填Prometheus的域名
externalUrl: 'http://prometheus.mrpu.cn'

storageSpec:
volumeClaimTemplate:
spec:
storageClassName: nfs-sc
accessModes: ['ReadWriteOnce']
resources:
requests:
storage: 100Gi

## Using default values from https://github.com/grafana/helm-charts/blob/main/charts/grafana/values.yaml
grafana:
enabled: true
defaultDashboardsTimezone: 'Asia/Shanghai'
adminPassword: 'putianhui'

ingress:
enabled: true
ingressClassName: nginx
hosts:
- grafana.mrpu.cn
tls:
- secretName: mrpu-cn-tls
hosts:
- grafana.mrpu.cn

persistence:
type: pvc
enabled: true
storageClassName: nfs-sc
accessModes:
- ReadWriteOnce
size: 100Gi

# 修改operator的镜像地址,默认从谷歌拉取,国内无法获取
prometheusOperator:
enabled: true
admissionWebhooks:
enabled: true
patch:
enabled: true
image:
registry: docker.io
repository: putianhui/ingress-nginx-kube-webhook-certgen
tag: v1.3.0

通过helm安装

1
$ helm install kube-prometheus --version 44.4.1 -f ./kube-prometheus-values.yaml prometheus-community/kube-prometheus-stack

创建alertmanager和Prometheus的ingress,因为前面清单中我们只开启了Grafana的ingress自动创建,另外两个需要我们手动创建。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
$ vim alertmanager-prometheus-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: alertmanager-prometheus-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
# ingress类名,我这里用nginx
ingressClassName: nginx
rules:
# ingress的域名
- host: alertmanager.mrpu.cn
http:
paths:
- backend:
service:
# svc的名称及端口号
name: kube-prometheus-kube-prome-alertmanager
port:
number: 9093
path: /
pathType: Prefix
- host: prometheus.mrpu.cn
http:
paths:
- backend:
service:
# svc的名称及端口号
name: kube-prometheus-kube-prome-prometheus
port:
number: 9090
path: /
pathType: Prefix
# https配置,注意提前准备好证书的secret
tls:
- hosts:
# https的域名
- alertmanager.mrpu.cn
- prometheus.mrpu.cn
# 域名使用的https证书,需要提前创建secret
secretName: mrpu-cn-tls

创建并应用ingress规则

1
$ kubectl apply -f ./alertmanager-prometheus-ingress.yaml

安装成功后即可通过ingress域名访问到PrometheusGrafanaAlertmanager服务,Grafana的密码为部署清单中设置的adminPassword

修复Prometheus中部分无法没上线的问题

修复etcdcontroller-managerscheduler这三个组件无法Prometheus无法上线问题,在master节点执行以下操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# 切换到master节点下的Kubernetes托管pod目录下
$ cd /etc/kubernetes/manifests

# 修改etcd的清单,command配置下的添加或者修改成以下的配置,原配置--listen-metrics-urls=http://127.0.0.1:2381
$ vim etcd.yaml
--listen-metrics-urls=http://0.0.0.0:2381

# 修改controller-manager,command配置下的添加或者修改成以下的配置,原配置--bind-address=127.0.0.1
$ vim kube-controller-manager.yaml
--bind-address=0.0.0.0

# 修改scheduler,command配置下的添加或者修改成以下的配置,原配置--bind-address=127.0.0.1
$ vim kube-scheduler.yaml
--bind-address=0.0.0.0

# 以上修改成功之后,重启下当前节点的kubelet服务即可这三个服务在Prometheus中被发现了。
$ systemctl restart kubelet

修改kube-proxy组件服务在Prometheus无法上线问题,由于该组件是以daemonset资源部署的,所有直接修改集群中的cm配置后重启下pod即可。

1
2
3
4
5
6
7
8
9
10
11
12
# 修改kube-proxy的configmap
$ kubectl edit cm/kube-proxy -n kube-system
apiVersion: v1
data:
config.conf: |-
apiVersion: kubeproxy.config.k8s.io/v1alpha1
# metricsBindAddress: ""
metricsBindAddress: "0.0.0.0"
······省略

# 删除kube-proxy的Pod重建后,即可在Prometheus中看到服务被发现了
$ kubectl delete pod -l k8s-app=kube-proxy -n kube-system

对接PrometheusAlert发送飞书告警

PrometheusAlert告警平台我们前面已经部署好了,现在修改Kube-Prometheus的helm下自定义values.yaml文件,在Alertmanager配置项下添加PrometheusAlert的告警webhook_configs配置。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# 修改前面helm安装时的自定义values.yaml清单,配置对接Prometheusalert的告警配置。
$ vim kube-promethues-values.yaml

# 这里只修改Alertmanager的配置
.... 省略 ....
alertmanager:
enabled: true
config:
global:
resolve_timeout: 5m
route:
# 这里改成按告警名称去分组告警
group_by: ['alertname']
group_wait: 30s
group_interval: 1m
repeat_interval: 5m
# 默认的告警渠道,和下面新建的Prometheusalert飞书告警渠道名称对应
receiver: 'default-alert'
routes:
- receiver: 'null'
matchers:
- alertname =~ "InfoInhibitor|Watchdog"
receivers:
- name: 'null'
# 新增的Prometheusalert飞书渠道告警
- name: 'default-alert'
# 调用Prometheusalert告警平台告警,使用我们的自定义模板:prometheus-fs
webhook_configs:
- url: 'http://prometheus-alert-prometheusalert.default:8080/prometheusalert?type=fs&tpl=prometheus-fs'
.... 省略 ....

通过更新后的vaules.yaml文件更新服务

1
$ helm upgrade kube-prometheus --version 44.4.1 -f ./kube-prometheus-values.yaml prometheus-community/kube-prometheus-stack

Prometheus触发一条告警即可在飞书查看到内容。