K8s Cluster uses ingress to realize the practice of static and dynamic separation of website entrance

In March this year, in the company's internal k8s training session, and research colleagues discussed in detail the application deployment of containerized deployment of several issues, the problem is as follows:

1. containerized Deployment of Java applications
First, the full-scale war package is compiled with the automated Deployment tool, the war package is compiled directly into the Docker image, pushed to the private repository and versioned, and the application needs to be containerized by updating the deployment Yaml file for deployment and subsequent rolling updates.

The difficulty and effort here lies in container mirroring and versioning, and then preparing to use Harbor as an enterprise private repository.

2, Dashborad Panel control of permissions
Solution Reference: http://blog.51cto.com/ylw6006/2113542

3, how to collect the application log
Solution Reference: http://blog.51cto.com/ylw6006/2107307

4, the site static and dynamic separation
First, all dynamic applications respond by deploying the Tomcat pod, and the static resources are uniformly deployed in a nginx pod mode;
Secondly, the rule configuration of static and dynamic separation is given to Traefik ingress.
Finally, the files of the static resources are stored uniformly on the PV, and updating the static resources does not need to compile the Docker image.

First, static and dynamic separation examples illustrate

For example, the virtual host site configuration (underlying service) for the Intranet test environment 2:
Domain name test2.oprman.com (PS: Research and development grandpa like their yy domain name) static resources configuration as follows
1,/--> corresponding/usr/local/6.0_files/oprman_test2 directory
2,/mfs--> corresponding to the/mnt/mfs directory (this is actually a distributed file system mount point)

The dynamic resource configuration of domain name test2.oprman.com is as follows
1./web--> back-end tomcat for the corresponding
2./api--> back-end tomcat for the corresponding

Back-end Tomcat configuration information

Example of a virtual host site configuration (platform service) for the Intranet test environment 2:
The static resource configuration of domain name resourcesharing.test2.59iedu.com is as follows
1,/--> corresponding/data/static_files/test2/zygxpt/portal directory
2,/mfs--> corresponding to the/mnt/mfs directory (this is actually a distributed file system mount point)
3,/admin--> corresponding/data/static_files/test2/zygxpt/admin directory
4,/login--> corresponding/data/static_files/test2/zygxpt/login directory
5,/play--> corresponding/data/static_files/test2/zygxpt/play directory

The dynamic resource configuration of domain name test2.oprman.com is as follows
1./web--> back-end tomcat for the corresponding

Back-end Tomcat configuration information

Here to add that there are three sets of environments in the intranet
1, development environment: Mainly used for developer development, debugging
2. Test environment 1: For testing personnel verification
3. Test Environment 2: for self-test and validation of developers

From the type of the site can be divided into the underlying services and platform services two types, most of the underlying services are internal calls through Dobbo, after the container as long as the network between the pod can be interconnected to call each other, a small number of low-level services need to expose the HTTP port. Platform services need to expose HTTP ports externally.

The nginx rule of static and dynamic separation is mainly the above two categories, in which the number of platform services is mostly, and the bottom service has less static and dynamic separation rules.

Ii. deployment of the Traefik portal

Traefik the specific deployment method can refer to the previous article, the previous portal: http://blog.51cto.com/ylw6006/2073718

# kubectl get svc,pod,ingress -n kube-system

Iii. creating applications that deal with dynamics

1, build dynamic part of the image and push to the private warehouse

# cat dockerfile                   FROM registry.59iedu.com/tomcat_base:v1.0MAINTAINER yangliangwei "[email protected]"COPY ROOT-20180509.tgz /home/RUN tar zxf /home/ROOT-20180509.tgz -C /home/tomcat/webapps/ && rm -rf /home/ROOT-20180509.tgz# docker build -t oprman-test2:v1 .

# docker tag oprman-test2:v1 registry.59iedu.com/oprman-test2:v1# docker push registry.59iedu.com/oprman-test2:v1

2. Create a dynamic app with Yaml files

# cat configmap.yaml apiVersion: v1kind: ConfigMapmetadata:  name: filebeat-oprman-test2-configdata:  filebeat.yml: |    filebeat.prospectors:    - input_type: log      paths:        - "/log/*"    output.elasticsearch:      hosts: ["192.168.1.19:9600"]      

# Cat Oprman-test2-tomcat-dp.yaml Apiversion:extensions/v1beta1kind:deploymentmetadata:name:oprman-test2-tomcat NA Mespace:defaultspec:replicas:1 Template:metadata:labels:name:oprman-test2-tomcat Spec:con Tainers:-image:registry.59iedu.com/filebeat:v5.4.0 imagepullpolicy:always Name:filebeat Vol Umemounts:-name:app-logs mountpath:/log-name:filebeat-oprman-test2-config mountpath :/etc/filebeat/-Image:registry.59iedu.com/oprman-test2:v1 Name:oprman-test2-tomcat ImagePullPoli  Cy:always env:-name:java_opts value: "-xmx2048m-xms512m" Ports:-Containerport: 8080 volumemounts:-name:app-logs Mountpath:/home/tomcat/logs volumes:-Name:app-lo GS Emptydir: {}-Name:filebeat-oprman-test2-config Configmap:name:filebeat-oprman-test2-co nfig# Kubectl create-f Oprman-test2-tomcat-dp.yaml   

# cat oprman-test2-tomcat-svc.yaml apiVersion: v1kind: Servicemetadata:  name: oprman-test2-tomcat  labels:    name: oprman-test2-tomcatspec:  ports:  - port: 8080    protocol: TCP    targetPort: 8080    name: http  

Demonstrates that only the backend tomcat that deploys the underlying service handles dynamic requests, like the platform's back-end Tomcat deployment approach, which is not covered for the sake of article length.

Iii. creating applications to handle static parts
1, we use CONFIGMAP way to handle the static part of the request route, the dynamic part of the request routing will be given to
Ingress to achieve. Nginx configuration file for the platform part of the configuration because the path is the same, so the domain name using $host for the wildcard, the bottom of the need to be configured separately. (Fortunately, the underlying service needs to expose not much HTTP)

# cat nginx.conf user nginx;worker_processes auto;error_log/usr/share/nginx/html/nginx-error.log;pid/run/nginx.pid; # Load dynamic modules.    see/usr/share/nginx/readme.dynamic.include/usr/share/nginx/modules/*.conf;events {worker_connections 102400; Use Epoll;} HTTP {log_format main ' $remote _addr-$remote _user [$time _local] "$request" ' $status $body _by    Tes_sent "$http _referer" "$http _user_agent" "$http _x_forwarded_for";    Server_tokens off;    Access_log/usr/share/nginx/html/nginx-default-access.log main;    Sendfile on;    Tcp_nopush on;    Tcp_nodelay on;    Keepalive_timeout 65;    Types_hash_max_size 2048;    Include/etc/nginx/mime.types;    Default_type Application/octet-stream;    include/etc/nginx/conf.d/*.conf;        Include/etc/nginx/conf/extra/*.conf;server {Listen default_server;        Index index.html index.htm; access_log/uSr/share/nginx/html/logs/test2-static-access.log main;        Location/{root/usr/share/nginx/html/$host/portal;        Index index.html index.htm;        } location/admin{root/usr/share/nginx/html/$host;        Index index.html index.htm;        } location/login {root/usr/share/nginx/html/$host;        Index index.html index.htm;        } location/play {root/usr/share/nginx/html/$host;        Index index.html index.htm;        } LOCATION/MFS {Root/mnt/mfs;        }}server {Listen 80;        server_name test2.oprman.com;        Index index.html index.htm;        Access_log/usr/share/nginx/html/logs/test2-static-access.log main;        Location/{root/usr/share/nginx/html/test2.oprman.com;        Index index.html index.htm;        } LOCATION/MFS {Root/mnt/mfs;  }}}# kubectl Create Configmap test2-static-etc--from-file nginx.conf

2, the creation of PV, PVC

# cat test2-static-data-pv-pvc.yaml apiVersion: v1kind: PersistentVolumemetadata:  name: test2-static-dataspec:  capacity:    storage: 100Gi   accessModes:  - ReadWriteMany   nfs:     path: /home/test2-static-data    server: 192.168.115.5  persistentVolumeReclaimPolicy: Recycle ---kind: PersistentVolumeClaimapiVersion: v1metadata:  name: test2-static-dataspec:  accessModes:    - ReadWriteMany  resources:    requests:      

3. Create Deployment

# Cat Test2-static-deployment.yaml Apiversion:extensions/v1beta1kind:deploymentmetadata:name: Test2-staticapiversion:extensions/v1beta1kind:deploymentmetadata:name:test2-static Labels:name:test2-staticspec : Replicas:2 template:metadata:labels:name:test2-static spec:containers:-Name:test2            -static image:registry.59iedu.com/nginx:latest volumemounts:-Mountpath:/usr/share/nginx/html Name:test2-static-data-mountpath:/etc/nginx/nginx.conf subPath:nginx.conf N           AME:TEST2-STATIC-ETC Ports:-containerport:80 Volumes:-Name:test2-static-data             PERSISTENTVOLUMECLAIM:CLAIMNAME:TEST2-STATIC-DATA-NAME:TEST2-STATIC-ETC Configmap: NAME:TEST2-STATIC-ETC items:-key:nginx.conf path:nginx.conf # Kubectl Create-f Test2-static-deployment.yaml

Iv. creation of a unified portal ingress

# cat test2-ingress.yaml apiVersion: extensions/v1beta1kind: Ingressmetadata:  name: traefik-oprman-test2  namespace: default  annotations:    nginx.ingress.kubernetes.io/rewrite-target: /spec:  rules:  - host: resourcesharing.test2.59iedu.com    http:      paths:      - path: /        backend:          serviceName: test2-static          servicePort: 80  - host: test2.oprman.com    http:      paths:      - path: /        backend:          serviceName: test2-static          servicePort: 80      - path: /web        backend:          serviceName: oprman-test2-tomcat          servicePort: 8080      - path: /api        backend:          serviceName: oprman-test2-tomcat          servicePort: 8080# kubectl create -f test2-ingress.yaml

V. Access testing
1. Modify host parsing

2. Bottom-level service backend Tomcat access test

3. Platform front-end access test

4, the platform background entrance access test (the previous Nginx configuration/admin section, the actual static content, enter the user name and password before the request to the back-end tomcat)

Here to highlight, the distributed file system attached to the Nginx host and released to the user access, in fact, is a very undesirable way, when Nginx and MFS master Host network connectivity problems, will cause Nginx request is blocked, 80, 443-port connection is not smooth, greatly affect the user experience, this follow-up need to improve together!

K8s Cluster uses ingress to realize the practice of static and dynamic separation of website entrance