必威体育Betway必威体育官网
当前位置:首页 > IT技术

Kubernetes之服务发现ingress & ingress controller

时间:2019-10-13 20:44:33来源:IT技术作者:seo实验室小编阅读:50次「手机版」
 

ingress

Service虽然解决了服务发现和负载均衡的问题,但它在使用上还是有一些限制:

  • 只支持4层负载均衡,没有7层功能
  • 对外访问时,NodePort类型需要在外部搭建额外的负载均衡,而LoadBalancer要求kubernetes必须跑在支持的cloud provider上

一、ingress引入历程

我们已经了解到Kubernetes暴露服务的方式目前只有三种:LoadBalancer Service、ExternalName、NodePort Service;那还有没有更好的方式呢?我们先来了解一下以下几个问题:

1、Pod漂移问题

我们知道Pod是有生命周期的,Pod宕机之后,其IP必然会改变,那我们如何把这个动态的Pod IP暴露出去呢?总所周知,我们可以借助Kubernetes的Service机制,在Service中以标签的形式选定一组带有指定标签的Pod,并监控和自动负载到它们的Pod IP,那么我们向外只需要暴露Service IP就行了,这就是我们常使用的NodePort模式:即在每个节点上开启一个端口,然后转发到内部的Pod IP上。此时的访问方式为:http: //nodeip:nodeport/

在这里插入图片描述

2、端口管理问题

采用NodePort方式暴露服务面临的问题是:服务一旦躲起来,NodePort在每个节点上开启的端口会极其庞大,而且难以维护。想一想,鉴于nginx的能力,我们能够使用一个nginx直接对内进行转发呢?由于Pod与Pod之间是可以互相通信的,而Pod是可以共享宿主机网络名称空间的,此时Pod上所监听的就是Node上的端口。那这又该如何实现呢?简单的实现就是使用DaemonSet在每个Node上监听80端口,然后写好规则,因为nginx外面绑定了宿主机80端口(就像NodePort),本身又在集群内,那么向后直接转发到相应Service IP就行了

在这里插入图片描述

在这里插入图片描述

3. 域名分配及动态更新问题

用上面的方法,我们采用nginx Pod似乎已经解决了问题,但是当每次有新服务加入又该如何修改nginx配置呢?nginx可以通过虚拟主机域名区分不同的服务,而每个服务通过upstream进行定义不同的负载均衡池,再加上location进行负载均衡的反向代理,在日常使用中只需要修改nginx.conf即可实现,但是在k8s中又该如何实现这种方式调度呢?

假设后端的服务开始只有ecshop,后面增加了bbs和member服务,那么又该如何将这2个服务加入到nginx Pod进行调度呢?总不能每次手动改后者Rolling update前端nginx pod吧!所以k8s引入了ingress,ingress就是为了解决这些问题而生的,ingress包含两大组件:ingress controller和ingress。

  • ingress

    简单理解就是你原来需要修改nginx配置,然后配置各种域名对应哪个Service,现在把这个动作抽象出来,变成一个Ingress对象,你可以用yaml创建,每次不要去修改nginx了,直接改yaml然后创建/更新就行了,那么问题又来了:nginx该如何处理呢?

  • ingress controller

    这个东西就是解决nginx如何处理这个问题的,ingress controller通过与kubernetes API交互,动态的去感知集群中Ingress规则变化,然后读取它,按照它自己的模板生成一段nginx配置,再写到nginx Pod中,最后reload以下,工作流程如下图

    在这里插入图片描述

    图表说明:

    首先我们有一个外部的负载均衡器externalLB把请求调度到一个nodePort类型的Service(ingress-nginx)上,然后nodePort类型的Service(ingress-nginx)又把它调度到内部的叫做ingressController的Pod上,ingressCtroller根据ingress中的定义(虚拟主机还是URL),每一组主机名或者URL都对应后端的Pod资源,并且用Service分组(图中的<service>site1/<service>site2只是用来做分组用的)

二、Ingress Nginx部署

使用Ingress功能步骤:

  1. 安装部署ingressCtroller Pod

  2. 部署ingress-nginx service

  3. 部署后端服务

  4. 部署ingress

    从前面的描述我们知道,Ingress可以使用yaml的方式进行创建,从而得知Ingress也是标准的K8S资源,其定义的方式也可以使用kubectl explain进行查看:

     [root@master manifests]# kubectl explain ingress
     KIND:     Ingress
     VERSION:  extensions/v1beta1
    
     DESCRIPTION:
     Ingress is a collection of rules that allow inbound connections to reach
     the endpoints defined by a backend. An Ingress can be configured to give
     services externally-reachable urls, load balance traffic, terminate SSL,
     offer name based virtual hosting etc.
    
     fieldS:
     apiVersion	<string>
      APIVersion defines the versioned schema of this representation of an
      object. Servers should convert recognized schemas to the latest internal
      value, and may reject unrecognized values. More info:
      https://git.k8s.io/community/contributors/devel/api-conventions.md#resources
    
     kind	<string>
      Kind is a string value representing the REST resource this object
      represents. Servers may infer this from the endpoint the client submits
      requests to. cannot be updated. In CamelCase. More info:
      https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds
    
     metadata	<Object>
      Standard object's metadata. More info:
      https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata
    
     spec	<Object>
      Spec is the desired state of the Ingress. More info:
      https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status
    
     status	<Object>
      Status is the current state of the Ingress. More info:
      https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status
    

2.1. 部署ingress controller

ingress-nginx在github上的地址

2.1.1. 下载ingress相关yaml文件

[root@master ~]# mkdir ingress-nginx
[root@master ~]# cd ingress-nginx/
[root@master ingress-nginx]# for filr in namespace.yaml configmap.yaml rbac.yaml tcp-services-configmap.yaml with-rbac.yaml udp-services-configmap.yaml default-backend.yaml;do wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/$file;done
[root@master ingress-nginx]# ls -l | awk '{print $9}'

configmap.yaml						#用于为nginx从外部注入配置的
default-backend.yaml				#配置默认后端服务
namespace.yaml						#创建独立的名称空间
rbac.yaml							#用于集群角色授权
tcp-services-configmap.yaml
udp-services-configmap.yaml
with-rbac.yaml

2.1.2. 创建ingress-nginx名称空间

[root@k8s-master ingress-nginx]# cat namespace.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx

---
[root@k8s-master ingress-nginx]# kubectl APPly -f namespace.yaml 
namespace/ingress-nginx created

2.1.3. 创建ingress controller的Pod

[root@master ingress-nginx]# kubectl apply -f .
configmap/nginx-configuration created
deployment.extensions/default-http-backend created
service/default-http-backend created
namespace/ingress-nginx configured
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole configured
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding configured
configmap/tcp-services created
configmap/udp-services created
deployment.extensions/nginx-ingress-controller created

此处遇到一个问题,新版本的Kubernetes在安装部署中,需要从k8s.grc.io仓库拉去所需镜像文件,但是由于国内防火墙问题导致无法正常拉去。

辛亏docker.io仓库对Google的容器做了mirror,可以通过下列命令拉取相关镜像:

[root@node01 ~]# docker pull mirrorgooglecontainers/defaultbackend-AMD64:1.5
1.5: Pulling from mirrorgooglecontainers/defaultbackend-amd64
9ecb1e82bb4a: Pull complete 
Digest: sha256:d08e129315e2dd093abfc16283cee19eabc18ae6b7cb8c2e26cc26888c6fc56a
Status: Downloaded newer image for mirrorgooglecontainers/defaultbackend-amd64:1.5

[root@node01 ~]# docker tag mirrorgooglecontainers/defaultbackend-amd64:1.5 k8s.gcr.io/defaultbackend-amd64:1.5
[root@node01 ~]# docker image ls
REPOSITORY                                    TAG                 IMAGE ID            CREATED             SIZE
mirrorgooglecontainers/defaultbackend-amd64   1.5                 b5af743e5984        34 hours ago        5.13MB
k8s.gcr.io/defaultbackend-amd64               1.5                 b5af743e5984        34 hours ago        5.13MB   

2.2. 部署ingress-nginx service

通过ingress-controller对外提供服务,现在还需要手动给ingress-controller建立一个servcie,接收集群外部流量

2.2.1 下载ingress-controller service的yaml文件

[root@master ingress-nginx]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml
[root@master ingress-nginx]# cat service-nodeport.yaml 
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---

2.2.2. 创建ingress-controller的service

[root@master ingress-nginx]# kubectl apply -f service-nodeport.yaml 
service/ingress-nginx created
[root@master ingress-nginx]# kubectl get svc -n ingress-nginx
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
default-http-backend   ClusterIP   10.107.114.219   <none>        80/TCP                       27m
ingress-nginx          NodePort    10.108.12.235    <none>        80:31875/TCP,443:32020/TCP   38s

2.3. 部署ingress

2.3.1. 查看ingress的配置清单选项

[root@master ingress-nginx]# kubectl explain ingress.spec
KIND:     Ingress
VERSION:  extensions/v1beta1

RESOURCE: spec <Object>

DESCRIPTION:
     Spec is the desired state of the Ingress. More info:
     https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status

     IngressSpec describes the Ingress the user wishes to exist.

FIELDS:
   backend  <Object>     #定义后端有哪几个主机
     A default backend capable of servicing requests that don't match any rule.
     At least one of 'backend' or 'rules' must be specified. This field is
     optional to allow the loadbalancer controller or defaulting logic to
     specify a global default.

   rules    <[]Object>    #定义规则
     A list of host rules used to configure the Ingress. If unspecified, or no
     rule matches, all traffic is sent to the default backend.

   tls  <[]Object>
     TLS configuration. Currently the Ingress only supports a single TLS port,
     443. If multiple members of this list specify different hosts, they will be
     multiplexed on the same port according to the hostname specified through
     the SNI TLS extension, if the ingress controller fulfilling the ingress
     supports SNI.

2.3.2. 编写ingress的配置清单并创建ingress

[root@master ingress-nginx]# vim ingress-default-backend.yaml
apiVersion: extensions/v1beta1				#api版本
kind: Ingress								#清单类型
metadata:									#元数据
  name: ingress-default-backend				#ingress名称
  namespace: ingress-nginx					#所属命名空间
  annotations:								#注解信息
    kubernetes.io/ingress.class: "nginx"
spec:										#规格
  rules:									#定义后端转发规则
  - host: default.backend.com				#通过域名进行转发
    http:
      paths:
        - path:								#配置访问路径
          backend:							#配置后端服务
            serviceName: default-http-backend
            servicePort: 80 
[root@master ingress-nginx]# kubectl apply -f ingress-default-backend.yaml 
ingress.extensions/ingress-default-backend created

查看ingress-default-backend的详细信息:

[root@master ingress-nginx]# kubectl describe ingress -n ingress-nginx ingress-default-backend
Name:             ingress-default-backend
Namespace:        ingress-nginx
Address:          
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host                 Path  Backends
  ----                 ----  --------
  default.backend.com  
                          default-http-backend:80 (<none>)
Annotations:
  kubernetes.io/ingress.class:                       nginx
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"ingress-default-backend","namespace":"ingress-nginx"},"spec":{"rules":[{"host":"default.backend.com","http":{"paths":[{"backend":{"serviceName":"default-http-backend","servicePort":80},"path":null}]}}]}}

Events:
  Type    Reason  Age   From                      message
  ----    ------  ----  ----                      -------
  Normal  CREATE  9m    nginx-ingress-controller  Ingress ingress-nginx/ingress-default-backend

进入nginx-ingress-controller进行查看是否注入了nginx配置:

[root@master ingress-nginx]# kubectl get pods -n ingress-nginx
NAME                                        READY     STATUS    RESTARTS   AGE
default-http-backend-7db7c45b69-znpxp       1/1       Running   0          1h
nginx-ingress-controller-6bd7c597cb-fgdz8   1/1       Running   0          1h
[root@master ingress-nginx]# kubectl exec -n ingress-nginx -ti nginx-ingress-controller-6bd7c597cb-fgdz8 -- /bin/sh
$ cat nginx.conf
....
....
## start server default.backend.com
server {
	server_name default.backend.com ;
	
	listen 80;
	
	set $proxy_upstream_name "-";
	
	location / {
		
		set $namespace      "ingress-nginx";
		set $ingress_name   "ingress-default-backend";
		set $service_name   "default-http-backend";
		set $service_port   "80";
		set $location_path  "/";
		
		rewrite_by_lua_block {
			
			balancer.rewrite()
			
		}
		
		log_by_lua_block {
			
			balancer.log()
			
			monitor.call()
		}
		
		port_in_redirect off;
		
		set $proxy_upstream_name "ingress-nginx-default-http-backend-80";
		
		client_max_body_size                    "1m";
		
		proxy_set_header Host                   $best_http_host;
		
		# Pass the extracted client certificate to the backend
		
		# Allow websocket connections
		proxy_set_header                        Upgrade           $http_upgrade;
		
		proxy_set_header                        Connection        $connection_upgrade;
		
		proxy_set_header X-Request-ID           $req_id;
		proxy_set_header X-Real-IP              $the_real_ip;
		
		proxy_set_header X-Forwarded-For        $the_real_ip;
		
		proxy_set_header X-Forwarded-Host       $best_http_host;
		proxy_set_header X-Forwarded-Port       $pass_port;
		proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
		
		proxy_set_header X-Original-URI         $request_uri;
		
		proxy_set_header X-Scheme               $pass_access_scheme;
		
		# Pass the original X-Forwarded-For
		proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
		
		# mitigate HTTPoxy Vulnerability
		# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
		proxy_set_header Proxy                  "";
		
		# Custom headers to proxied server
		
		proxy_connect_timeout                   5s;
		proxy_send_timeout                      60s;
		proxy_read_timeout                      60s;
		
		proxy_buffering                         "off";
		proxy_buffer_size                       "4k";
		proxy_buffers                           4 "4k";
		proxy_request_buffering                 "on";
		
		proxy_http_version                      1.1;
		
		proxy_cookie_domain                     off;
		proxy_cookie_path                       off;
		
		# In case of ERRORs try the next upstream server before returning an error
		proxy_next_upstream                     error timeout;
		proxy_next_upstream_tries               3;
		
		proxy_pass http://upstream_balancer;
		
		proxy_redirect                          off;
		
	}
	
}
## end server default.backend.com
....
....

可以发现我们刚刚部署的ingress的的确确已经注入到nginx-ingress-controller的nginx配置中了

2.3.3. 通过ingress访问服务

修改本地host文件

192.168.116.128 default.backend.com
192.168.116.129 default.backend.com

在这里插入图片描述

三、构建TLS站点

3.1. 添加tomcat服务

[root@master ingress-nginx]# vim tomcat-demo.yaml
apiVersion: v1
kind: Service
metadata:
  name: tomcat
  namespace: default
spec:
  selector:
    app: tomcat
    release: stable
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  - name: ajp
    port: 8009
    targetPort: 8009
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-demo
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: tomcat
      release: stable
  template:
    metadata:
      labels:
        app: tomcat
        release: stable
    spec:
      containers:
      - name: tomcat
        image: tomcat:8.5.34-jre8-alpine
        ports:
        - name: http
       	  containerPort: 8080
        - name: ajp
      	  containerPort: 8009
[root@master ingress-nginx]# kubectl create -f tomcat-demo.yaml 
service/tomcat created
deployment.apps/tomcat-demo created
[root@master ingress-nginx]# kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP             12d
tomcat       ClusterIP   10.108.64.162   <none>        8080/TCP,8009/TCP   8s
[root@master ingress-nginx]# kubectl get deployments
NAME          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
tomcat-demo   3         3         3            0           16s
[root@master ingress-nginx]# kubectl get pods
NAME                           READY     STATUS    RESTARTS   AGE
tomcat-demo-798cdf69b9-qk9pd   1/1       Running   0          7m
tomcat-demo-798cdf69b9-t642t   1/1       Running   0          7m
tomcat-demo-798cdf69b9-tlfq6   1/1       Running   0          7m

3.2. 准备证书并生成secret

准备证书:

[root@master ingress-nginx]# openssl genrsa -out tls.key 2048
Generating RSA private key, 2048 bit long modulus
............................+++
..................+++
e is 65537 (0x10001)
[root@master ingress-nginx]# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=tomcat.zwx.com

生成secret:

[root@master ingress-nginx]# kubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.key
secret/tomcat-ingress-secret created
[root@master ingress-nginx]# kubectl get secret
NAME                    TYPE                                  DATA      AGE
default-token-l9jws     kubernetes.io/service-account-token   3         12d
tomcat-ingress-secret   kubernetes.io/tls                     2         33s
[root@master ingress-nginx]# kubectl describe secret tomcat-ingress-secret
Name:         tomcat-ingress-secret
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.crt:  1285 bytes
tls.key:  1679 bytes

3.3. 创建ingress

[root@master ingress-nginx]# vim ingress-tomcat-tls.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-tomcat-tls
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - tomcat.zwx.com
    secretName: tomcat-ingress-secret
  rules:
  - host: tomcat.zwx.com
    http:
      paths:
      - backend:
          serviceName: tomcat
          servicePort: 8080
[root@master ingress-nginx]# kubectl apply -f ingress-tomcat-tls.yaml 
ingress.extensions/ingress-tomcat-tls created

3.4. 访问测试

查看ingress-nginx 服务

[root@master ingress-nginx]# kubectl get svc -n ingress-nginx -o wide
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE       SELECTOR
default-http-backend   ClusterIP   10.107.114.219   <none>        80/TCP                       4h        app.kubernetes.io/name=default-http-backend,app.kubernetes.io/part-of=ingress-nginx
ingress-nginx          NodePort    10.108.12.235    <none>        80:31875/TCP,443:32020/TCP   3h        app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx

所以访问https://tomcat.zwx.com:32020

在这里插入图片描述

相关阅读

kubernetes1.7新特性:PodDisruptionBudget控制器变化

背景概念在Kubernetes中,为了保证业务不中断或业务SLA不降级,需要将应用进行集群化部署。通过PodDisruptionBudget控制器可以设置应

Kubernetes基础知识详解

Docker-Compose Docker Swarm和Kubernetes的区别 Docker-Compose Docker-Compose 是用来管理你的容器的,有点像一个容器的管家,想

Kubernetes系列之监控Metres-server实战篇

image本次系列使用的所需部署包版本都使用的目前最新的或最新稳定版,安装包地址请到公众号内回复【K8s实战】获取 介绍

我写过的软件之Kubernetes All In One

其实这不是一个软件,而是一个安装包。 公司的产品用到docker/kubernetes这些技术,每次安装都得从网上下载一堆软件包,加上有一些软件

Kubernetes平台云原生存储编排利器Rook详解

云原生存储编排利器Rook详解 1. Rook介绍 1.1 Rook简介 Rook是一个运行在Kubernetes集群中的开源云原生存储服务编排工具,为各种存

分享到:

栏目导航

推荐阅读

热门阅读