k8s环境部署及使用方式-白红宇

安装k8s+docker集群环境（基于centos7系统）

集群机器：

centos-master = 192.168.121.9

centos-minion-1 = 192.168.121.65

centos-minion-2 = 192.168.121.66

centos-minion-3 = 192.168.121.67

1.配置yum源

[centos-master]:cat /etc/yum.repos.d/virt7-docker-common-release.repo

[virt7-docker-common-release] name=virt7-docker-common-release baseurl=http://cbs.centos.org/repos/virt7-docker-common-release/x86_64/os/ gpgcheck=0

加载安装包：

[centos-master]:yum repolist

virt7-docker-common-release virt7-docker-common-release 15

2.安装集群必要软件—-etcd/flannel/kubernetes

Etcd服务在k8s集群中用于配置共享和服务发现。

Flannel是针对k8s设计一个网络规划服务，让集群中的不同节点主机创建的Docker容器都具有全集群唯一的虚拟IP地址。

[centos-master]:yum -y install –enablerepo=virt7-docker-common-release kubernetes etcd flannel

3.如果集群中没有使用DNS解析，那么需要在master节点的/etc/hosts中添加node的主机名信息，比如：

echo "192.168.121.9 centos-master192.168.121.65 centos-minion-1192.168.121.66 centos-minion-2192.168.121.67 centos-minion-3" >> /etc/hosts

4.修改配置master节点的kubernetes配置文件/etc/kubernetes/config

#表示错误日志记录到文件还是输出到stderrKUBE_LOGTOSTDERR="--logtostderr=true" #日志等级KUBE_LOG_LEVEL="--v=0" #允许运行特权容器KUBE_ALLOW_PRIV="--allow-privileged=false" #apiserver的服务地址，controller-manager、scheduler及kubelet都会用到这个配置，这里配置为192.168.121.9KUBE_MASTER="--master=http://192.168.121.9:8080"

5.k8s集群中涉及的端口比较多，所以centos中的防火墙需要设置对应规则，并需关闭selinux。为确保k8s集群的正常运行，我们可以直接关闭iptables与seliinux服务。

setenforce 0systemctl stop firewalld.servicesystemctl stop firewalld.service

6.修改配置master节点的etcd配置文件/etc/etcd/etcd.conf

etcd服务的可调参数比较多，根据需求开启对应功能，此处我们大概调整如下几个功能：

# [member] #etcd名称ETCD_NAME=default #etcd数据存储位置ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #监听的端口ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379" #[cluster] #集群监听的端口ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"

7.修改配置master节点的apiserver配置文件/etc/kubernetes/apiserver

#监听的接口，如果配置为127.0.0.1则只监听localhost，配置为0.0.0.0会监听所有接口，这里配置为0.0.0.0KUBE_API_ADDRESS="--address=0.0.0.0" #apiserver的监听端口，默认8080KUBE_API_PORT="--port=8080" #minion上kubelet监听的端口，默认10250KUBELET_PORT="--kubelet-port=10250" #etcd服务地址，前面已经启动了etcd服务,端口为2379KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379" #kubernetes可以分配的ip的范围，kubernetes启动的每一个pod以及serveice都会分配一个ip地址，将从这个范围分配KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16" #需要额外添加的配置项，简单地启用一个集群无需配置KUBE_API_ARGS=""

8.启动并且配置etcd的网段，此网段一定是未被使用的

systemctl start etcdetcdctl mkdir /kube-centos/networketcdctl mk /kube-centos/network/config "{
     \"Network\": \"172.30.0.0/16\", \"SubnetLen\": 24, \"Backend\": {
     \"Type\": \"vxlan\" } }"

9.修改配置master节点的flanneld配置文件/etc/sysconfig/flanneld

#etcd的访问地址及端口FLANNEL_ETCD_ENDPOINTS="http://192.168.121.9:2379" #服务范围FLANNEL_ETCD_PREFIX="/kube-centos/network" #其他FLANNEL_OPTIONS=""

10.启动k8s集群

for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler flanneld; do systemctl restart $SERVICES systemctl enable $SERVICES systemctl status $SERVICESdone

以上就是k8s，master节点的安装及配置

接下来，配置nodes节点

11.修改配置nodes节点kubelet的配置文件/etc/kubernetes/kubelet

#minion监听的地址，每个minion根据实际的ip配置,这里配置为0.0.0.0KUBELET_ADDRESS="--address=0.0.0.0" #监听的端口KUBELET_PORT="--port=10250" #apiserver的访问地址及端口KUBELET_API_SERVER="--api-servers=http://192.168.121.9:8080" #额外增加的参数KUBELET_ARGS="--logtostderr=false --v=0 --log-dir=/data/logs/kubernetes"

12.修改配置nodes节点flanneld的配置文件/etc/sysconfig/flanneld

#etcd的访问地址及端口FLANNEL_ETCD="http://192.168.121.9:2379" #etcd服务范围FLANNEL_ETCD_KEY="/kube-centos/network"

13.启动k8s集群服务

for SERVICES in kube-proxy kubelet flanneld docker; do systemctl restart $SERVICES systemctl enable $SERVICES systemctl status $SERVICES done

14.设置kubectl的配置文件

kubectl config set-cluster default-cluster --server=http://192.168.121.9:8080kubectl config set-context default-context --cluster=default-cluster --user=default-adminkubectl config use-context default-context

15.检查集群状态

[centos-master]:kubectl get nodes

NAME STATUS AGE VERSIONcentos-minion-1 Ready 3d v1.5.0+fff5156centos-minion-2 Ready 3d v1.5.0+fff5156centos-minion-3 Ready 3d v1.5.0+fff5156

至此，集群构建完毕

搭建私有库

私有库用于系统内部存储成品镜像，能够快速进行下载及被k8s调度。

1.下载并启动私有库

[centos-master]:docker run --name registry -v /etc/localtime:/etc/localtime -v /opt/registry:/var/lib/registry -p 5000:5000 -itd docker.io/registry#--name 表示启动的容器后名称，此处为registry #-v 表示挂载路径 格式为宿主机路径:容器内路径 #-p 表示映射端口 格式为宿主机端口:容器内端口 #-itd docker的内部参数，此处声明后台运行容器并分配一个伪终端并绑定到容器的标准输入上，后跟镜像名称此处为docker.io/registry

2.创建一个secret服务，用于k8s调度私有库容器时的“令牌”。简单来说，secret服务就是一个存储密码的服务

[centos-master]:kubectl create secret docker-registry registrykey --docker-server=registry.evehicle.cn --docker-username=docker --docker-password=docker --docker-email=lienhua@zhongchuangsanyou.com [centos-master]:kubectl get secretNAME TYPE DATA AGEregistrykey kubernetes.io/dockercfg 1 6s

此时登录时会提示认证错误

[centos-master]:docker login -u docker -p docker -e lienhua@zhongchuangsanyou.com registry.evehicle.cnFlag --email has been deprecated, will be removed in 1.13.Error response from daemon: login attempt to https://registry.evehicle.cn/v2/ failed with status: 401 Unauthorized

这是因为Docker官方是推荐采用Secure Registry的工作模式的，即transport采用tls。这样我们就需要为Registry配置tls所需的key和crt文件了

3.配置nginx反向代理

[centos-master]: cat registry.evehicle.cn.conf

# For versions of nginx > 1.3.9 that include chunked transfer encoding support # Replace with appropriate values where necessary upstream docker-registry { server 192.168.121.9:5000; #server 10.44.170.95:5000;}# uncomment if you want a 301 redirect for users attempting to connect # on port 80 # NOTE: docker client will still fail. This is just for convenience # server {
     # listen *:80; # server_name my.docker.registry.com; # return 301 https://$server_name$request_uri; # } server { listen 443; server_name registry.evehicle.cn; ssl on; ssl_certificate ssl/registry.evehicle.cn.crt; ssl_certificate_key ssl/registry.evehicle.cn.key; client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image uploads # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486) chunked_transfer_encoding on; location / { auth_basic "Restricted"; auth_basic_user_file passwd; add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always; proxy_pass http://docker-registry; proxy_set_header Host $http_host; # required for docker client's sake proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 900; } location /_ping { auth_basic off; include docker-registry.conf; } location /v1/_ping { auth_basic off; include docker-registry.conf; } location /v2/_ping { auth_basic off; include docker-registry.conf; }}

将key及crt证书文件放到../ssl目录下。使用htpasswd生成密码放于./上一级目录

htpasswd -bcm passwd docker docker #-c：创建一个加密文件 #-m：md5加密，默认可不填写 #-b：表示用户名密码在命令行中一并输入，不用分别填写

4.再次登录

[centos-master]:docker login -u docker -p docker -e lienhua@zhongchuangsanyou.com registry.evehicle.cn Login Succeeded表示成功，此时再pull\push既在私有库中进行

构建服务

docker的本意是将代码包含在容器内制作成镜像形成“产品”。但出于公司的（频繁修改代码及服务器资源受限）的特殊性，我们将代码以“外挂”的形式运行在宿主机上。下面以部署官网（apache）服务为例：

1.从docker的公有库里下载centos7的原生镜像

[centos-master]:docker pull centosUsing default tag: latestTrying to pull repository docker.io/library/centos ...latest: Pulling from docker.io/library/centosd9aaf4d82f24: Downloading [> ] 540 kB/73.39 MBd9aaf4d82f24: Pulling fs layerDigest: sha256:eba772bac22c86d7d6e72421b4700c3f894ab6e35475a34014ff8de74c10872eStatus: Downloaded newer image for centos:latest

2.编写Dockerfile制造apache基础镜像

######httpd####FROM centosMAINTAINER lienhua lienhua@zhongchuangsanyou.comRUN yum -y install epel-releaseRUN yum -y install httpd php php-mysql php-memcache* php-mbstringADD httpd.conf /etc/httpd/conf/httpd.confEXPOSE 80CMD ["/usr/sbin/apachectl", "-D", "FOREGROUND"]

其中httpd.conf文件需要在当前目录下真实存在，此处其内容为

ServerRoot "/etc/httpd" Listen 80Listen 8080Include conf.modules.d/*.confInclude zcsy/*.confUser apacheGroup apacheServerAdmin root@localhost
     AllowOverride none Require all denied DocumentRoot "/var/www/html" 
    
      AllowOverride None Require all granted
     
    
      Options Indexes FollowSymLinks AllowOverride None Require all granted
     
    
      DirectoryIndex index.html
     
    
      Require all denied
     ErrorLog "logs/error_log" LogLevel warn
    
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined LogFormat "%h %l %u %t \"%r\" %>s %b" common 
     
       LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio 
      CustomLog "logs/access_log" combined
     
    
      ScriptAlias /cgi-bin/ "/var/www/cgi-bin/" 
     
    
      AllowOverride None Options None Require all granted
     
    
      TypesConfig /etc/mime.types AddType application/x-compress .Z AddType application/x-gzip .gz .tgz AddType application/x-httpd-php .php AddType text/html .shtml AddOutputFilter INCLUDES .shtml
     AddDefaultCharset UTF-8
    
      MIMEMagicFile conf/magic
     EnableSendfile off EnableMMAP off IncludeOptional conf.d/*.conf

执行[centos-master]:docker build -t registry.evehicle.cn/httpd . 命令制作名为”registry.evehicle.cn/httpd”的镜像（注意此处的点必须要有，并且其意义代表当前目录下的Dockerfile文件）

3.将制作好的镜像上传到私有库

docker push registry.evehicle.cn/httpd

4.编写启动apache服务的yaml文件

[centos-master]:cat 13-rc-httpd.yaml

apiVersion: v1kind: ReplicationControllermetadata: name: 13-rc-httpd labels: name: 13-rc-httpdspec: replicas: 2 selector: name: 13-rc-httpd template: metadata: labels: name: 13-rc-httpd spec: containers: - name: 13-rc-httpd image: registry.evehicle.cn/httpd env: - name: LANG  value: en_US.UTF-8 ports: - containerPort: 80  hostPort: 80 volumeMounts: - name: time mountPath: /etc/localtime - name: zcsy mountPath: /etc/httpd/zcsy - name: deploy mountPath: /docker/httpd/deploy - name: log mountPath: /var/log/httpd volumes: - name: time hostPath: path: /etc/localtime - name: zcsy hostPath: path: /docker/httpd/zcsy - name: deploy hostPath: path: /docker/httpd/deploy - name: log hostPath: path: /docker/httpd/log nodeSelector: slave: "13" imagePullSecrets: - name: registrykey

5.给其中一个node加上标签为“13”

kubectl label nodes centos-minion-1 slave=13

6.此时拥有标签“13”的nodes应具备的条件

/docker/httpd/zcsy下需要有官网的配置文件


    
      ServerName www.evehicle.cn DocumentRoot /var/deploy/wordpress/ RewriteEngine on RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-d RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-f RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} !^.*\.(ico|pdf|flv|jpe?g|js|gif|png|html|shtml|zip|xml|gz|rar|swf|txt|apk|bmp|css|m4a|ogg|mp3|ipa|plist)$ RewriteCond %{REQUEST_URI} !^/server-status$ RewriteRule . /index.php [QSA,PT,L] 
     
     Options FollowSymLinks AllowOverride All Require all granted

以及/docker/httpd/deploy下需要有官网的代码

7.运行yaml文件启动容器

[centos-master]: kuberctl create -f 13-rc-httpd.yaml

8.查看服务

[centos-master]: kuberctl get rc

NAME DESIRED CURRENT AGE13-rc-httpd 2 2 168d

9.程序中涉及的mysql\redis\memcache等服务也需使用容器运行起来

[centos-master]: docker pull redis

[centos-master]: docker tag registry.evehicle.cn/redis redis

[centos-master]: docker push registry.evehicle.cn/redis

[centos-master]: kubectl create -f rc-redis.yaml

[centos-master]: cat rc-redis.yaml

apiVersion: v1kind: ReplicationControllermetadata: name: redis labels: name: redisspec: replicas: 2 selector: name: redis template: metadata: labels: name: redis spec: containers: - name: redis image: registry.evehicle.cn/redis ports: - containerPort: 6379  hostPort: 6379 volumeMounts: - name: data mountPath: /data - name: time mountPath: /etc/localtime volumes: - name: data hostPath: path: /docker/redis/6379 - name: time hostPath: path: /etc/localtime nodeSelector: slave: "13" imagePullSecrets: - name: registrykey

启动memcache

[centos-master]: docker pull memcache

[centos-master]: docker tag registry.evehicle.cn/memcached memcache

[centos-master]: docker push registry.evehicle.cn/memcached

[centos-master]: kubectl create -f rc-memcached.yaml

[centos-master]: cat rc-memcached.yaml

apiVersion: v1kind: ReplicationControllermetadata: name: memcached labels: name: memcachedspec: replicas: 3 selector: name: memcached template: metadata: labels: name: memcached spec: containers: - name: memcached image: registry.evehicle.cn/memcached ports: - containerPort: 11211 hostPort: 11211 #nodeSelector: # slave: "13" imagePullSecrets: - name: registrykey

制造mysql镜像

[centos-master]: cat Dockerfile

FROM alpineCOPY startup.sh /startup.shRUN addgroup mysql && \ adduser -H -D -s /bin/false -G mysql mysql && \ apk add --update mysql mysql-client && rm -f /var/cache/apk/* && \ mkdir /data && \ chown -R mysql:mysql /data /etc/mysql && \ chmod 755 /startup.sh \ ;WORKDIR /data VOLUME /data VOLUME /etc/mysqlEXPOSE 3306 CMD ["/startup.sh"]

启动mysql（建议mysql在宿主机启动）

[centos-master]: docker build -t registry.evehicle.cn/mysql

[centos-master]: docker push registry.evehicle.cn/mysql

[centos-master]: kubectl create -f rc-mysql.yaml

[centos-master]: cat rc-mysql.yaml

apiVersion: v1kind: ReplicationControllermetadata: name: 13-rc-mysql labels: name: 13-rc-mysqlspec: replicas: 2 selector: name: 13-rc-mysql template: metadata: labels: name: 13-rc-mysql spec: containers: - name: 13-rc-mysql image: registry.evehicle.cn/mysql env: - name: MYSQL_DATABASE  value: admin - name: MYSQL_USER  value: tony - name: MYSQL_PASSWORD  value: 456 - name: MYSQL_ROOT_PASSWORD  value: 123 ports: - containerPort: 3306  hostPort: 3306 volumeMounts: - name: time mountPath: /etc/localtime - name: data mountPath: /data - name: etc mountPath: /etc/mysql - name: run mountPath: /run/mysqld volumes: - name: time hostPath: path: /etc/localtime - name: data hostPath: path: /docker/mysql/data - name: etc hostPath: path: /docker/mysql/etc - name: run hostPath: path: /docker/mysql/run nodeSelector: slave: "13" imagePullSecrets: - name: registrykey

为方便代码编写及统一管理，应提前做好内部DNS解析。将所负责的应用规整到对应的机器上。

本文转自CSDN-