ingress
Service虽然解决了服务发现和负载均衡的问题，但它在使用上还是有一些限制：
- 只支持4层负载均衡，没有7层功能
- 对外访问时，NodePort类型需要在外部搭建额外的负载均衡，而LoadBalancer要求kubernetes必须跑在支持的cloud provider上
一、ingress引入历程
我们已经了解到Kubernetes暴露服务的方式目前只有三种：LoadBalancer Service、ExternalName、NodePort Service；那还有没有更好的方式呢？我们先来了解一下以下几个问题：
1、Pod漂移问题
我们知道Pod是有生命周期的，Pod宕机之后，其IP必然会改变，那我们如何把这个动态的Pod IP暴露出去呢？总所周知，我们可以借助Kubernetes的Service机制，在Service中以标签的形式选定一组带有指定标签的Pod，并监控和自动负载到它们的Pod IP，那么我们向外只需要暴露Service IP就行了，这就是我们常使用的NodePort模式：即在每个节点上开启一个端口，然后转发到内部的Pod IP上。此时的访问方式为：http: //nodeip:nodeport/
2、端口管理问题
采用NodePort方式暴露服务面临的问题是：服务一旦躲起来，NodePort在每个节点上开启的端口会极其庞大，而且难以维护。想一想，鉴于nginx的能力，我们能够使用一个nginx直接对内进行转发呢？由于Pod与Pod之间是可以互相通信的，而Pod是可以共享宿主机的网络名称空间的，此时Pod上所监听的就是Node上的端口。那这又该如何实现呢？简单的实现就是使用DaemonSet在每个Node上监听80端口，然后写好规则，因为nginx外面绑定了宿主机80端口(就像NodePort)，本身又在集群内，那么向后直接转发到相应Service IP就行了
3. 域名分配及动态更新问题
用上面的方法，我们采用nginx Pod似乎已经解决了问题，但是当每次有新服务加入又该如何修改nginx配置呢？nginx可以通过虚拟主机域名区分不同的服务，而每个服务通过upstream进行定义不同的负载均衡池，再加上location进行负载均衡的反向代理，在日常使用中只需要修改nginx.conf即可实现，但是在k8s中又该如何实现这种方式调度呢？
假设后端的服务开始只有ecshop，后面增加了bbs和member服务，那么又该如何将这2个服务加入到nginx Pod进行调度呢？总不能每次手动改后者Rolling update前端nginx pod吧！所以k8s引入了ingress，ingress就是为了解决这些问题而生的，ingress包含两大组件：ingress controller和ingress。
- ingress
简单理解就是你原来需要修改nginx配置，然后配置各种域名对应哪个Service，现在把这个动作抽象出来，变成一个Ingress对象，你可以用yaml创建，每次不要去修改nginx了，直接改yaml然后创建/更新就行了，那么问题又来了：nginx该如何处理呢？
- ingress controller
这个东西就是解决nginx如何处理这个问题的，ingress controller通过与kubernetes API交互，动态的去感知集群中Ingress规则变化，然后读取它，按照它自己的模板生成一段nginx配置，再写到nginx Pod中，最后reload以下，工作流程如下图
图表说明：
首先我们有一个外部的负载均衡器externalLB把请求调度到一个nodePort类型的Service(ingress-nginx)上，然后nodePort类型的Service(ingress-nginx)又把它调度到内部的叫做ingressController的Pod上，ingressCtroller根据ingress中的定义(虚拟主机还是URL)，每一组主机名或者URL都对应后端的Pod资源，并且用Service分组(图中的<service>site1/<service>site2只是用来做分组用的)
二、Ingress Nginx部署
使用Ingress功能步骤：
-
安装部署ingressCtroller Pod
-
部署ingress-nginx service
-
部署后端服务
-
部署ingress
从前面的描述我们知道，Ingress可以使用yaml的方式进行创建，从而得知Ingress也是标准的K8S资源，其定义的方式也可以使用
kubectl explain进行查看：[root@master manifests]# kubectl explain ingress KIND: Ingress VERSION: extensions/v1beta1 DESCRIPTION: Ingress is a collection of rules that allow inbound connections to reach the endpoints defined by a backend. An Ingress can be configured to give services externally-reachable urls, load balance traffic, terminate SSL, offer name based virtual hosting etc. fieldS: apiVersion <string> APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources kind <string> Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds metadata <Object> Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata spec <Object> Spec is the desired state of the Ingress. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status status <Object> Status is the current state of the Ingress. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status
2.1. 部署ingress controller
ingress-nginx在github上的地址
2.1.1. 下载ingress相关yaml文件
[root@master ~]# mkdir ingress-nginx
[root@master ~]# cd ingress-nginx/
[root@master ingress-nginx]# for filr in namespace.yaml configmap.yaml rbac.yaml tcp-services-configmap.yaml with-rbac.yaml udp-services-configmap.yaml default-backend.yaml;do wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/$file;done
[root@master ingress-nginx]# ls -l | awk '{print $9}'
configmap.yaml #用于为nginx从外部注入配置的
default-backend.yaml #配置默认后端服务
namespace.yaml #创建独立的名称空间
rbac.yaml #用于集群角色授权
tcp-services-configmap.yaml
udp-services-configmap.yaml
with-rbac.yaml
2.1.2. 创建ingress-nginx名称空间
[root@k8s-master ingress-nginx]# cat namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
---
[root@k8s-master ingress-nginx]# kubectl APPly -f namespace.yaml
namespace/ingress-nginx created
2.1.3. 创建ingress controller的Pod
[root@master ingress-nginx]# kubectl apply -f .
configmap/nginx-configuration created
deployment.extensions/default-http-backend created
service/default-http-backend created
namespace/ingress-nginx configured
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole configured
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding configured
configmap/tcp-services created
configmap/udp-services created
deployment.extensions/nginx-ingress-controller created
此处遇到一个问题，新版本的Kubernetes在安装部署中，需要从k8s.grc.io仓库拉去所需镜像文件，但是由于国内防火墙问题导致无法正常拉去。
辛亏docker.io仓库对Google的容器做了mirror，可以通过下列命令拉取相关镜像：
[root@node01 ~]# docker pull mirrorgooglecontainers/defaultbackend-AMD64:1.5
1.5: Pulling from mirrorgooglecontainers/defaultbackend-amd64
9ecb1e82bb4a: Pull complete
Digest: sha256:d08e129315e2dd093abfc16283cee19eabc18ae6b7cb8c2e26cc26888c6fc56a
Status: Downloaded newer image for mirrorgooglecontainers/defaultbackend-amd64:1.5
[root@node01 ~]# docker tag mirrorgooglecontainers/defaultbackend-amd64:1.5 k8s.gcr.io/defaultbackend-amd64:1.5
[root@node01 ~]# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
mirrorgooglecontainers/defaultbackend-amd64 1.5 b5af743e5984 34 hours ago 5.13MB
k8s.gcr.io/defaultbackend-amd64 1.5 b5af743e5984 34 hours ago 5.13MB
2.2. 部署ingress-nginx service
通过ingress-controller对外提供服务，现在还需要手动给ingress-controller建立一个servcie，接收集群外部流量。
2.2.1 下载ingress-controller service的yaml文件
[root@master ingress-nginx]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml
[root@master ingress-nginx]# cat service-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
- name: https
port: 443
targetPort: 443
protocol: TCP
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
2.2.2. 创建ingress-controller的service
[root@master ingress-nginx]# kubectl apply -f service-nodeport.yaml
service/ingress-nginx created
[root@master ingress-nginx]# kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default-http-backend ClusterIP 10.107.114.219 <none> 80/TCP 27m
ingress-nginx NodePort 10.108.12.235 <none> 80:31875/TCP,443:32020/TCP 38s
2.3. 部署ingress
2.3.1. 查看ingress的配置清单选项
[root@master ingress-nginx]# kubectl explain ingress.spec
KIND: Ingress
VERSION: extensions/v1beta1
RESOURCE: spec <Object>
DESCRIPTION:
Spec is the desired state of the Ingress. More info:
https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status
IngressSpec describes the Ingress the user wishes to exist.
FIELDS:
backend <Object> #定义后端有哪几个主机
A default backend capable of servicing requests that don't match any rule.
At least one of 'backend' or 'rules' must be specified. This field is
optional to allow the loadbalancer controller or defaulting logic to
specify a global default.
rules <[]Object> #定义规则
A list of host rules used to configure the Ingress. If unspecified, or no
rule matches, all traffic is sent to the default backend.
tls <[]Object>
TLS configuration. Currently the Ingress only supports a single TLS port,
443. If multiple members of this list specify different hosts, they will be
multiplexed on the same port according to the hostname specified through
the SNI TLS extension, if the ingress controller fulfilling the ingress
supports SNI.
2.3.2. 编写ingress的配置清单并创建ingress
[root@master ingress-nginx]# vim ingress-default-backend.yaml
apiVersion: extensions/v1beta1 #api版本
kind: Ingress #清单类型
metadata: #元数据
name: ingress-default-backend #ingress名称
namespace: ingress-nginx #所属命名空间
annotations: #注解信息
kubernetes.io/ingress.class: "nginx"
spec: #规格
rules: #定义后端转发规则
- host: default.backend.com #通过域名进行转发
http:
paths:
- path: #配置访问路径
backend: #配置后端服务
serviceName: default-http-backend
servicePort: 80
[root@master ingress-nginx]# kubectl apply -f ingress-default-backend.yaml
ingress.extensions/ingress-default-backend created
查看ingress-default-backend的详细信息：
[root@master ingress-nginx]# kubectl describe ingress -n ingress-nginx ingress-default-backend
Name: ingress-default-backend
Namespace: ingress-nginx
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
default.backend.com
default-http-backend:80 (<none>)
Annotations:
kubernetes.io/ingress.class: nginx
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"ingress-default-backend","namespace":"ingress-nginx"},"spec":{"rules":[{"host":"default.backend.com","http":{"paths":[{"backend":{"serviceName":"default-http-backend","servicePort":80},"path":null}]}}]}}
Events:
Type Reason Age From message
---- ------ ---- ---- -------
Normal CREATE 9m nginx-ingress-controller Ingress ingress-nginx/ingress-default-backend
进入nginx-ingress-controller进行查看是否注入了nginx配置：
[root@master ingress-nginx]# kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
default-http-backend-7db7c45b69-znpxp 1/1 Running 0 1h
nginx-ingress-controller-6bd7c597cb-fgdz8 1/1 Running 0 1h
[root@master ingress-nginx]# kubectl exec -n ingress-nginx -ti nginx-ingress-controller-6bd7c597cb-fgdz8 -- /bin/sh
$ cat nginx.conf
....
....
## start server default.backend.com
server {
server_name default.backend.com ;
listen 80;
set $proxy_upstream_name "-";
location / {
set $namespace "ingress-nginx";
set $ingress_name "ingress-default-backend";
set $service_name "default-http-backend";
set $service_port "80";
set $location_path "/";
rewrite_by_lua_block {
balancer.rewrite()
}
log_by_lua_block {
balancer.log()
monitor.call()
}
port_in_redirect off;
set $proxy_upstream_name "ingress-nginx-default-http-backend-80";
client_max_body_size "1m";
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Request-ID $req_id;
proxy_set_header X-Real-IP $the_real_ip;
proxy_set_header X-Forwarded-For $the_real_ip;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme $pass_access_scheme;
# Pass the original X-Forwarded-For
proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
proxy_connect_timeout 5s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_buffering "off";
proxy_buffer_size "4k";
proxy_buffers 4 "4k";
proxy_request_buffering "on";
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of ERRORs try the next upstream server before returning an error
proxy_next_upstream error timeout;
proxy_next_upstream_tries 3;
proxy_pass http://upstream_balancer;
proxy_redirect off;
}
}
## end server default.backend.com
....
....
可以发现我们刚刚部署的ingress的的确确已经注入到nginx-ingress-controller的nginx配置中了
2.3.3. 通过ingress访问服务
修改本地host文件：
192.168.116.128 default.backend.com
192.168.116.129 default.backend.com
三、构建TLS站点
3.1. 添加tomcat服务
[root@master ingress-nginx]# vim tomcat-demo.yaml
apiVersion: v1
kind: Service
metadata:
name: tomcat
namespace: default
spec:
selector:
app: tomcat
release: stable
ports:
- name: http
port: 8080
targetPort: 8080
- name: ajp
port: 8009
targetPort: 8009
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-demo
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: tomcat
release: stable
template:
metadata:
labels:
app: tomcat
release: stable
spec:
containers:
- name: tomcat
image: tomcat:8.5.34-jre8-alpine
ports:
- name: http
containerPort: 8080
- name: ajp
containerPort: 8009
[root@master ingress-nginx]# kubectl create -f tomcat-demo.yaml
service/tomcat created
deployment.apps/tomcat-demo created
[root@master ingress-nginx]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 12d
tomcat ClusterIP 10.108.64.162 <none> 8080/TCP,8009/TCP 8s
[root@master ingress-nginx]# kubectl get deployments
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
tomcat-demo 3 3 3 0 16s
[root@master ingress-nginx]# kubectl get pods
NAME READY STATUS RESTARTS AGE
tomcat-demo-798cdf69b9-qk9pd 1/1 Running 0 7m
tomcat-demo-798cdf69b9-t642t 1/1 Running 0 7m
tomcat-demo-798cdf69b9-tlfq6 1/1 Running 0 7m
3.2. 准备证书并生成secret
准备证书：
[root@master ingress-nginx]# openssl genrsa -out tls.key 2048
Generating RSA private key, 2048 bit long modulus
............................+++
..................+++
e is 65537 (0x10001)
[root@master ingress-nginx]# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=tomcat.zwx.com
生成secret：
[root@master ingress-nginx]# kubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.key
secret/tomcat-ingress-secret created
[root@master ingress-nginx]# kubectl get secret
NAME TYPE DATA AGE
default-token-l9jws kubernetes.io/service-account-token 3 12d
tomcat-ingress-secret kubernetes.io/tls 2 33s
[root@master ingress-nginx]# kubectl describe secret tomcat-ingress-secret
Name: tomcat-ingress-secret
Namespace: default
Labels: <none>
Annotations: <none>
Type: kubernetes.io/tls
Data
====
tls.crt: 1285 bytes
tls.key: 1679 bytes
3.3. 创建ingress
[root@master ingress-nginx]# vim ingress-tomcat-tls.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-tomcat-tls
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
tls:
- hosts:
- tomcat.zwx.com
secretName: tomcat-ingress-secret
rules:
- host: tomcat.zwx.com
http:
paths:
- backend:
serviceName: tomcat
servicePort: 8080
[root@master ingress-nginx]# kubectl apply -f ingress-tomcat-tls.yaml
ingress.extensions/ingress-tomcat-tls created
3.4. 访问测试
查看ingress-nginx 服务
[root@master ingress-nginx]# kubectl get svc -n ingress-nginx -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default-http-backend ClusterIP 10.107.114.219 <none> 80/TCP 4h app.kubernetes.io/name=default-http-backend,app.kubernetes.io/part-of=ingress-nginx
ingress-nginx NodePort 10.108.12.235 <none> 80:31875/TCP,443:32020/TCP 3h app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx
所以访问https://tomcat.zwx.com:32020
相关阅读
kubernetes1.7新特性：PodDisruptionBudget控制器变化
背景概念在Kubernetes中，为了保证业务不中断或业务SLA不降级，需要将应用进行集群化部署。通过PodDisruptionBudget控制器可以设置应
Docker-Compose Docker Swarm和Kubernetes的区别 Docker-Compose Docker-Compose 是用来管理你的容器的，有点像一个容器的管家，想
Kubernetes系列之监控Metres-server实战篇
image本次系列使用的所需部署包版本都使用的目前最新的或最新稳定版，安装包地址请到公众号内回复【K8s实战】获取 介绍
其实这不是一个软件，而是一个安装包。 公司的产品用到docker/kubernetes这些技术，每次安装都得从网上下载一堆软件包，加上有一些软件
云原生存储编排利器Rook详解 1. Rook介绍 1.1 Rook简介 Rook是一个运行在Kubernetes集群中的开源云原生存储服务编排工具，为各种存