k8s 业务容器化实战案例 1 业务容器化优势:
提高资源利用率、节约部署IT成本。
提高部署效率，基于kubernetes实现微服务的快速部署与交付、 容器的批量调度与秒级启动。
实现横向扩容、灰度部署、回滚、链路追踪、服务治理等。
可根据业务负载进行自动弹性伸缩。
容器将环境和代码打包在镜像内，保证了测试与生产运行环境
紧跟云原生社区技术发展的步伐，不给公司遗留技术债，为后
为个人储备前沿技术，提高个人level。
2 业务规划及镜像分层构建
通过基础镜像构建，如：ubuntu: 22.04
在基础镜像上安装运行必要的环境，如JDK11
以安装JDK11的镜像为基础，构建tomcat, dubbo, spring 镜像
通过tomcat/dubbo/spring镜像打包业务代码，构建具体业务镜像
2.1 构建基础镜像 [root@manager centos]# cat build-command.sh # 构建脚本# !/bin/bash # 自定义Centos 基础镜像
执行sh build-command.sh , 看到如下输出
说明镜像构建并上传完成，查看harbor可以看到已上传镜像：
2.2 通过基础镜像构建jdk镜像 [root@manager jdk-1.8.212]# cat build-command.sh# !/bin/bash # JDK Base Image
构建镜像：
[root@manager jdk-1.8.212]# sh build-command.sh ---> 72b13222d098 ---> Running in e6d098a60b47 ---> a2ae6dace6d2 ---> 82bda0383e23 ---> Running in 026500d482c9 ---> ff0d3d834344 ---> 673880b7c07f ---> Running in dc89db687bcb ---> 44cbb3fb9715 ---> Running in f14546ac8229 ---> 72d797a0fcaf ---> Running in 05c9258eb922 ---> 2a2f8f6bb43a ---> Running in c5fe236e089b ---> 5f53217d7526
harbor中查看：
2.3 通过基础镜像构建nginx镜像 [root@manager nginx-base]# cat build-command.sh# !/bin/bash # Nginx Base Image
构建镜像
[root@manager nginx-base]# sh build-command.sh ---> 72b13222d098 ---> Using cache ---> a2ae6dace6d2 ---> Running in f655d2b5c708 ---> ffc80b4becf4 ---> c19e1687d011 ---> Running in c9627a4fd ---> 5b1c620f0538
在harbor中查看
2.4 通过JDK镜像构建tomcat镜像 [root@manager tomcat-base-8.5.43]# ls# !/bin/bash # Tomcat 8.5.43基础镜像
2.5 通过tomcat镜像构建app镜像 [root@manager tomcat-app1]# cat Dockerfile# tomcat web1 # 自行准备对应配置文件 # !/bin/bash
构建
[root@manager tomcat-app1]# ./build-command.sh tomcat-app1-20220731 ---> b92d19e3edb3 ---> 67e7db5c915f ---> 37e3d4ea6059 ---> 94226df8dad7 ---> 583b3b1830c3 ---> Running in 6c707556ca45 ---> cf73cc40f72d ---> Running in d69b80d89ec1 ---> 0f4df0fda324 ---> Running in 7d8d9fa07eea ---> 3ea8b69863be
下面使用刚才制作好的镜像实现nginx + tomcat + nfs 动静分离
实战案例一： 自定义镜像运行nginx及tomcat服务并基于NAS实现动静分离
基于tomcat-base配置tomcat业务镜像 构建镜像
[root@manager tomcat-app1]# cat Dockerfile # tomcat web1 # !/bin/bash ---> ee5755a0eeee ---> 088261a93ba2 ---> 65e5a76a394c ---> 89ad99b8f780 ---> f7e79a435dd4 ---> Running in 76a6a9bc8698 ---> bff60f96c4f9 ---> Running in 0796e1bd118a ---> 36a9fad9a841 ---> Running in 9dd900d7bf37 ---> 4b20463708f4
编写yaml文件：
kind: Deployment apiVersion: apps/v1 metadata: labels: app: magedu-tomcat-app1-deployment-label name: magedu-tomcat-app1-deployment namespace: myserver spec: replicas: 2 selector: matchLabels: app: magedu-tomcat-app1-selector template: metadata: labels: app: magedu-tomcat-app1-selector spec: containers: - name: magedu-tomcat-app1-container image: harbor.qintianjun.local/k8s/tomcat-app1:tomcat-app1-20220731 imagePullPolicy: IfNotPresent ports: - containerPort: 8080 protocol: TCP name: http env: - name: "password" value: "123456" - name: "age" value: "18" resources: limits: cpu: 1 memory: "512Mi" requests: cpu: 500m memory: "512Mi" --- kind: Service apiVersion: v1 metadata: labels: app: magedu-tomcat-app1-service-label name: magedu-tomcat-app1-service namespace: myserver spec: type: NodePort ports: - name: http port: 80 protocol: TCP targetPort: 8080 nodePort: 30092 selector: app: magedu-tomcat-app1-selector
使用kubectl apply -f tomcat-app1.yaml 命令使之生效，查看状态：
[root@master lab]# kubectl get pod,svc,ep -n myserver -o wide
此时访问宿主机30092端口可以看到之前在tomcat镜像中打好的测试页：
说明tomcat正常工作。
基于nginx-base配置nginx的业务镜像 编写nginx.conf
user tomcat tomcat; worker_processes auto;daemon off ;events {worker_connections 1024 ;http {include mime.types;default_type application/octet-stream;sendfile on ;keepalive_timeout 65 ;upstream tomcat_webserver { server magedu-tomcat-app1-service.myserver.svc.magedu.local:80 ;server {listen 80 ;server_name localhost;location / {root html;index index.html index.htm;location /webapp { root html;index index.html index.htm;location /myapp { proxy_pass http://tomcat_webserver;proxy_set_header Host $host ;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for ;proxy_set_header X-Real-IP $remote_addr ;error_page 500 502 503 504 /50x.html;location = /50x.html {root html;
编写nginx-web1对应的Dockerfile:
[root@manager nginx]# cat Dockerfile# Nginx 1.20.2 # 静态资源挂载路径 # !/bin/bash
编辑nginx的yaml:
[root@master lab ]kind: Deployment apiVersion: apps/v1 metadata: labels: app: magedu-nginx-deployment-label name: magedu-nginx-deployment namespace: myserver spec: replicas: 1 selector: matchLabels: app: magedu-nginx-selector template: metadata: labels: app: magedu-nginx-selector spec: containers: - name: magedu-nginx-container image: harbor.qintianjun.local/k8s/nginx-web1:20220801-test1 imagePullPolicy: IfNotPresent ports: - containerPort: 80 protocol: TCP name: http - containerPort: 443 protocol: TCP name: https env: - name: "password" value: "123456" - name: "age" value: "20" resources: limits: cpu: 500m memory: 500Mi requests: cpu: 500m memory: 500Mi --- kind: Service apiVersion: v1 metadata: labels: app: magedu-nginx-service-label name: magedu-nginx-service namespace: myserver spec: type: NodePort ports: - name: http port: 80 protocol: TCP targetPort: 80 nodePort: 30090 selector: app: magedu-nginx-selector
使用kubectl apply命令生效，访问宿主机30090端口：
访问/webapp,返回静态资源页：
访问/myapp，转发给后端tomcat：
此时nginx转发后端生效。
验证 分别进入两个tomcat容器，修改/data/tomcat/webapps/myapp/index.html为v1,v2:
[root@master lab]# kubectl exec -it pod/magedu-tomcat-app1-deployment-67b5fdcfff-f5497 -n myserver -- bash
此时反复刷新页面，可以看到页面交替显示v1,v2
说明后端tomcat服务是交替轮询的
也可以使用NAS(NFS)存放静态资源文件，参考：
[root@master lab ]kind: Deployment apiVersion: apps/v1 metadata: labels: app: magedu-nginx-deployment-label name: magedu-nginx-deployment namespace: myserver spec: replicas: 1 selector: matchLabels: app: magedu-nginx-selector template: metadata: labels: app: magedu-nginx-selector spec: containers: - name: magedu-nginx-container image: harbor.qintianjun.local/k8s/nginx-web1:20220801-test1 imagePullPolicy: IfNotPresent ports: - containerPort: 80 protocol: TCP name: http - containerPort: 443 protocol: TCP name: https env: - name: "password" value: "123456" - name: "age" value: "20" resources: limits: cpu: 500m memory: 500Mi requests: cpu: 500m memory: 500Mi volumeMounts: - name: magedu-images mountPath: /usr/local/nginx/html/webapp/images readOnly: false - name: magedu-static mountPath: /usr/local/nginx/html/webapp/static readOnly: false volumes: - name: magedu-images nfs: server: 172.31 .7 .109 path: /data/k8sdata/magedu/images - name: magedu-static nfs: server: 172.31 .7 .109 path: /data/k8sdata/magedu/static --- kind: Service apiVersion: v1 metadata: labels: app: magedu-nginx-service-label name: magedu-nginx-service namespace: myserver spec: type: NodePort ports: - name: http port: 80 protocol: TCP targetPort: 80 nodePort: 30090 - name: https port: 443 protocol: TCP targetPort: 443 nodePort: 30091 selector: app: magedu-nginx-selector
至此tomcat+nginx动静分离环境构建完成
实战案例二：pv/pvc + zookeeper 通过jdk 镜像构建zookeeper镜像
准备Dockerfile和build脚本
[root@manager zookeeper]# cat Dockerfile# FROM harbor-linux38.local.com/linux38/slim_java:8 # Download Zookeeper # !/bin/bash
准备zookeeper对应配置文件：
[root@manager zookeeper]# cat conf/log4j.properties# snapCount=100000
准备启动脚本
[root@manager zookeeper]# cat bin/zkReady.sh# !/bin/bash
集群服务的配置文件提供方式：
提前在镜像中定义
在部署集群的时候在yaml中传递变量
在entrypoint脚本中定义
准备pv及pvc对应yaml：
[root@master1 pv ]--- apiVersion: v1 kind: PersistentVolume metadata: name: zookeeper-datadir-pv-1 spec: capacity: storage: 20Gi accessModes: - ReadWriteOnce nfs: server: 192.168 .88 .139 path: /data/k8sdata/magedu/zookeeper-datadir-1 --- apiVersion: v1 kind: PersistentVolume metadata: name: zookeeper-datadir-pv-2 spec: capacity: storage: 20Gi accessModes: - ReadWriteOnce nfs: server: 192.168 .88 .139 path: /data/k8sdata/magedu/zookeeper-datadir-2 --- apiVersion: v1 kind: PersistentVolume metadata: name: zookeeper-datadir-pv-3 spec: capacity: storage: 20Gi accessModes: - ReadWriteOnce nfs: server: 192.168 .88 .139 path: /data/k8sdata/magedu/zookeeper-datadir-3 root@master1 pv ]--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: zookeeper-datadir-pvc-1 namespace: myserver spec: accessModes: - ReadWriteOnce volumeName: zookeeper-datadir-pv-1 resources: requests: storage: 10Gi --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: zookeeper-datadir-pvc-2 namespace: myserver spec: accessModes: - ReadWriteOnce volumeName: zookeeper-datadir-pv-2 resources: requests: storage: 10Gi --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: zookeeper-datadir-pvc-3 namespace: myserver spec: accessModes: - ReadWriteOnce volumeName: zookeeper-datadir-pv-3 resources: requests: storage: 10Gi
确认pv状态为Available：
确认pvc状态：
编写zookeeper.yaml并应用:
[root@master1 zookeeper ]apiVersion: v1 kind: Service metadata: name: zookeeper namespace: magedu spec: ports: - name: client port: 2181 selector: app: zookeeper --- apiVersion: v1 kind: Service metadata: name: zookeeper1 namespace: magedu spec: type: NodePort ports: - name: client port: 2181 nodePort: 32181 - name: followers port: 2888 - name: election port: 3888 selector: app: zookeeper server-id: "1" --- apiVersion: v1 kind: Service metadata: name: zookeeper2 namespace: magedu spec: type: NodePort ports: - name: client port: 2181 nodePort: 32182 - name: followers port: 2888 - name: election port: 3888 selector: app: zookeeper server-id: "2" --- apiVersion: v1 kind: Service metadata: name: zookeeper3 namespace: magedu spec: type: NodePort ports: - name: client port: 2181 nodePort: 32183 - name: followers port: 2888 - name: election port: 3888 selector: app: zookeeper server-id: "3" --- kind: Deployment apiVersion: apps/v1 metadata: name: zookeeper1 namespace: magedu spec: replicas: 1 selector: matchLabels: app: zookeeper template: metadata: labels: app: zookeeper server-id: "1" spec: volumes: - name: data emptyDir: {}- name: wal emptyDir: medium: Memory containers: - name: server image: 192.168 .88 .138 /k8s/zookeeper:2022080701 imagePullPolicy: IfNotPresent env: - name: MYID value: "1" - name: SERVERS value: "zookeeper1,zookeeper2,zookeeper3" - name: JVMFLAGS value: "-Xmx1G" ports: - containerPort: 2181 - containerPort: 2888 - containerPort: 3888 volumeMounts: - mountPath: "/zookeeper/data" name: zookeeper-datadir-pvc-1 volumes: - name: zookeeper-datadir-pvc-1 persistentVolumeClaim: claimName: zookeeper-datadir-pvc-1 --- kind: Deployment apiVersion: apps/v1 metadata: name: zookeeper2 namespace: magedu spec: replicas: 1 selector: matchLabels: app: zookeeper template: metadata: labels: app: zookeeper server-id: "2" spec: volumes: - name: data emptyDir: {}- name: wal emptyDir: medium: Memory containers: - name: server image: 192.168 .88 .138 /k8s/zookeeper:2022080701 imagePullPolicy: IfNotPresent env: - name: MYID value: "2" - name: SERVERS value: "zookeeper1,zookeeper2,zookeeper3" - name: JVMFLAGS value: "-Xmx1G" ports: - containerPort: 2181 - containerPort: 2888 - containerPort: 3888 volumeMounts: - mountPath: "/zookeeper/data" name: zookeeper-datadir-pvc-2 volumes: - name: zookeeper-datadir-pvc-2 persistentVolumeClaim: claimName: zookeeper-datadir-pvc-2 --- kind: Deployment apiVersion: apps/v1 metadata: name: zookeeper3 namespace: magedu spec: replicas: 1 selector: matchLabels: app: zookeeper template: metadata: labels: app: zookeeper server-id: "3" spec: volumes: - name: data emptyDir: {}- name: wal emptyDir: medium: Memory containers: - name: server image: 192.168 .88 .138 /k8s/zookeeper:2022080701 imagePullPolicy: IfNotPresent env: - name: MYID value: "3" - name: SERVERS value: "zookeeper1,zookeeper2,zookeeper3" - name: JVMFLAGS value: "-Xmx1G" ports: - containerPort: 2181 - containerPort: 2888 - containerPort: 3888 volumeMounts: - mountPath: "/zookeeper/data" name: zookeeper-datadir-pvc-3 volumes: - name: zookeeper-datadir-pvc-3 persistentVolumeClaim: claimName: zookeeper-datadir-pvc-3
查看pod状态：
后面测试过程暂时省略，主要思路就是修改yaml中镜像地址为错误地址，导致其中某个pod创建失败，观察其他pod的follower及leader角色变化。
实战案例三： PV/PVC及Redis单机 构建redis镜像 [root@manager redis]# cat Dockerfile # Redis Image # !/bin/bash
创建pv与pvc [root@master1 pv ]--- apiVersion: v1 kind: PersistentVolume metadata: name: redis-datadir-pv-1 namespace: magedu spec: capacity: storage: 10Gi accessModes: - ReadWriteOnce nfs: path: /data/k8sdata/magedu/redis-datadir-1 server: 192.168 .88 .139 root@master1 pv ]--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: redis-datadir-pvc-1 namespace: magedu spec: volumeName: redis-datadir-pv-1 accessModes: - ReadWriteOnce resources: requests: storage: 10Gi
运行redis服务 [root@master1 redis ]kind: Deployment apiVersion: apps/v1 metadata: labels: app: devops-redis name: deploy-devops-redis namespace: magedu spec: replicas: 1 selector: matchLabels: app: devops-redis template: metadata: labels: app: devops-redis spec: containers: - name: redis-container image: 192.168 .88 .138 /k8s/redis:2022080701 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: "/data/redis-data/" name: redis-datadir volumes: - name: redis-datadir persistentVolumeClaim: claimName: redis-datadir-pvc-1 --- kind: Service apiVersion: v1 metadata: labels: app: devops-redis name: srv-devops-redis namespace: magedu spec: type: NodePort ports: - name: tcp port: 6379 targetPort: 6379 nodePort: 30379 selector: app: devops-redis sessionAffinity: ClientIP sessionAffinityConfig: clientIP: timeoutSeconds: 10800
验证redis数据读写 访问对应node节点30379端口
[root@node2 ~]# telnet 192.168.88.136 30379$ 6
此时进入nfs服务挂载点所在目录
可以看到文件被写入对应pvc, 查看aof文件可以看到刚才执行的命令：
