hyman-海曼工具

一些简单的记录

0%

基于K3S构建高可用Rancher集群

发表于 更新于 分类于

K3S简述

K3s (轻量级 Kubernetes)： 和 RKE 类似，也是经过认证的 Kubernetes 发行版。它比 RKE 更新，更易用且更轻量化，全部组件都在一个小于 100 MB 的二进制文件中。从 Rancher v2.4 开始，Rancher 可以安装在 K3s 集群上。

详情见：https://rancher2.docs.rancher.cn/docs/installation/_index

Rancher简述

Rancher 是为使用容器的公司打造的容器管理平台。Rancher 简化了使用 Kubernetes 的流程，开发者可以随处运行 Kubernetes（Run Kubernetes Everywhere），满足 IT 需求规范，赋能 DevOps 团队。

详情见：https://rancher2.docs.rancher.cn/docs/overview/_index

部署前系统环境准备

关闭防火墙和SeLinux

  • centos ：
    1
    2
    3
    4
    systemctl stop firewalld
    systemctl disable firewalld
    setenforce 0
    sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
  • Ubuntu：
    1
    sudo ufw disable
  • 节点及Docker功能调优
    1
    https://rancher2.docs.rancher.cn/docs/best-practices/optimize/os/_index

配置host文件

  • 设置永久主机名称
1
2
hostnamectl set-hostname node1
hostnamectl set-hostname node2
  • 配置host文件，并确保每台机器上都可以通过主机名互通
1
2
10.0.0.10 node1
10.0.0.11 node2

部署

Kubectl

1
2
3
4
5
6
7
8
9
10
11
12
13
14
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

yum clean all
yum makecache fast

yum install -y kubectl

Helm

  • 安装参考Helm官网，Helm是Kubernetes的包管理器，Helm的版本需要高于v3
1
2
3
4
5
6
7
8
# 下载安装包
wget https://get.helm.sh/helm-v3.2.1-linux-amd64.tar.gz
# 解压
tar zxvf helm-v3.2.1-linux-amd64.tar.gz
# 将二进制文件移动至/usr/local/bin/
sudo mv linux-amd64/helm /usr/local/bin/helm
# 验证安装
helm help

docker-ce

  • 安装repo源

    1
    yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

  • 移除旧版本Docker

1
2
3
4
5
6
7
8
9
10
11
yum remove -y docker \
              docker-client \
              docker-client-latest \
              docker-common \
              docker-latest \
              docker-latest-logrotate \
              docker-logrotate \
              docker-selinux \
              docker-engine-selinux \
              docker-engine \
              container*
  • 安装docker-ce
1
2
3
export docker_version=17.03.2
version=$(yum list docker-ce.x86_64 --showduplicates | sort -r|grep ${docker_version}|awk '{print $2}')
yum -y install --setopt=obsoletes=0 docker-ce-${version} docker-ce-selinux-${version}
  • 启动docker
1
2
3
systemctl enable docker
systemctl start docker
systemctl status docker

MySQL 5.7

1
2
3
4
5
6
docker run -p 3306:3306 --name mysql \
-v /usr/local/docker/mysql/conf:/etc/mysql \
-v /usr/local/docker/mysql/logs:/var/log/mysql \
-v /usr/local/docker/mysql/data:/var/lib/mysql \
-e MYSQL_ROOT_PASSWORD=123456 \
-d mysql:5.7

k3s

  • 所有k3s节点上都要运行此命令
1
2
3
4
5
6
7
8
9
# 启动 k3s Server
curl -sfL https://docs.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh -s - server \
--datastore-endpoint="mysql://root:123456@tcp(10.0.0.12:3306)/k3s"
# 验证
sudo k3s kubectl get nodes
# 在每个 Rancher Server 节点上安装 K3s 时，会在节点上/etc/rancher/k3s/k3s.yaml位置创建一个kubeconfig文件。该文件包含用于完全访问集群的凭据。# 复制 k3s.yaml 到 ~/.kube/config
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
# 验证 kubectl
sudo kubectl get pods --all-namespaces

Rancher

添加 Helm Chart 仓库

1
helm repo add rancher-stable http://rancher-mirror.oss-cn-beijing.aliyuncs.com/server-charts/stable

为 Rancher 创建 Namespace

1
sudo kubectl create namespace cattle-system

生成证书(使用自签发证书，其他参考官网)

  • 一键生成ssl自签名证书脚本
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
    135
    136
    137
    138
    139
    140
    141
    142
    143
    144
    145
    146
    147
    148
    149
    150
    151
    152
    153
    154
    155
    156
    157
    158
    159
    160
    161
    162
    163
    164
    165
    166
    167
    168
    #!/bin/bash -e

    # * 为必改项
    # * 更换为你自己的域名
    CN='' # 例如: demo.rancher.com

    # 扩展信任IP或域名
    ## 一般ssl证书只信任域名的访问请求，有时候需要使用ip去访问server，那么需要给ssl证书添加扩展IP，
    ## 多个IP用逗号隔开。如果想多个域名访问，则添加扩展域名（SSL_DNS）,多个SSL_DNS用逗号隔开
    SSL_IP='' # 例如: 1.2.3.4
    SSL_DNS='' # 例如: demo.rancher.com

    # 国家名(2个字母的代号)
    C=CN

    # 证书加密位数
    SSL_SIZE=2048

    # 证书有效期
    DATE=${DATE:-3650}

    # 配置文件
    SSL_CONFIG='openssl.cnf'

    if [[ -z $SILENT ]]; then
    echo "----------------------------"
    echo "| SSL Cert Generator |"
    echo "----------------------------"
    echo
    fi

    export CA_KEY=${CA_KEY-"cakey.pem"}
    export CA_CERT=${CA_CERT-"cacerts.pem"}
    export CA_SUBJECT=ca-$CN
    export CA_EXPIRE=${DATE}

    export SSL_CONFIG=${SSL_CONFIG}
    export SSL_KEY=$CN.key
    export SSL_CSR=$CN.csr
    export SSL_CERT=$CN.crt
    export SSL_EXPIRE=${DATE}

    export SSL_SUBJECT=${CN}
    export SSL_DNS=${SSL_DNS}
    export SSL_IP=${SSL_IP}

    export K8S_SECRET_COMBINE_CA=${K8S_SECRET_COMBINE_CA:-'true'}

    [[ -z $SILENT ]] && echo "--> Certificate Authority"

    if [[ -e ./${CA_KEY} ]]; then
        [[ -z $SILENT ]] && echo "====> Using existing CA Key ${CA_KEY}"
    else
        [[ -z $SILENT ]] && echo "====> Generating new CA key ${CA_KEY}"
        openssl genrsa -out ${CA_KEY} ${SSL_SIZE} > /dev/null
    fi

    if [[ -e ./${CA_CERT} ]]; then
        [[ -z $SILENT ]] && echo "====> Using existing CA Certificate ${CA_CERT}"
    else
        [[ -z $SILENT ]] && echo "====> Generating new CA Certificate ${CA_CERT}"
        openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} \
        -days ${CA_EXPIRE} -out ${CA_CERT} -subj "/CN=${CA_SUBJECT}" > /dev/null || exit 1
    fi

    echo "====> Generating new config file ${SSL_CONFIG}"
    cat > ${SSL_CONFIG} <<EOM
    [req]
    req_extensions = v3_req
    distinguished_name = req_distinguished_name
    [req_distinguished_name]
    [ v3_req ]
    basicConstraints = CA:FALSE
    keyUsage = nonRepudiation, digitalSignature, keyEncipherment
    extendedKeyUsage = clientAuth, serverAuth
    EOM

    if [[ -n ${SSL_DNS} || -n ${SSL_IP} ]]; then
        cat >> ${SSL_CONFIG} <<EOM
    subjectAltName = @alt_names
    [alt_names]
    EOM
        IFS=","
        dns=(${SSL_DNS})
        dns+=(${SSL_SUBJECT})
        for i in "${!dns[@]}"; do
          echo DNS.$((i+1)) = ${dns[$i]} >> ${SSL_CONFIG}
        done

        if [[ -n ${SSL_IP} ]]; then
            ip=(${SSL_IP})
            for i in "${!ip[@]}"; do
              echo IP.$((i+1)) = ${ip[$i]} >> ${SSL_CONFIG}
            done
        fi
    fi

    [[ -z $SILENT ]] && echo "====> Generating new SSL KEY ${SSL_KEY}"
    openssl genrsa -out ${SSL_KEY} ${SSL_SIZE} > /dev/null || exit 1

    [[ -z $SILENT ]] && echo "====> Generating new SSL CSR ${SSL_CSR}"
    openssl req -sha256 -new -key ${SSL_KEY} -out ${SSL_CSR} \
    -subj "/CN=${SSL_SUBJECT}" -config ${SSL_CONFIG} > /dev/null || exit 1

    [[ -z $SILENT ]] && echo "====> Generating new SSL CERT ${SSL_CERT}"
    openssl x509 -sha256 -req -in ${SSL_CSR} -CA ${CA_CERT} \
        -CAkey ${CA_KEY} -CAcreateserial -out ${SSL_CERT} \
        -days ${SSL_EXPIRE} -extensions v3_req \
        -extfile ${SSL_CONFIG} > /dev/null || exit 1

    if [[ -z $SILENT ]]; then
    echo "====> Complete"
    echo "keys can be found in volume mapped to $(pwd)"
    echo
    echo "====> Output results as YAML"
    echo "---"
    echo "ca_key: |"
    cat $CA_KEY | sed 's/^/  /'
    echo
    echo "ca_cert: |"
    cat $CA_CERT | sed 's/^/  /'
    echo
    echo "ssl_key: |"
    cat $SSL_KEY | sed 's/^/  /'
    echo
    echo "ssl_csr: |"
    cat $SSL_CSR | sed 's/^/  /'
    echo
    echo "ssl_cert: |"
    cat $SSL_CERT | sed 's/^/  /'
    echo
    fi

    if [[ -n $K8S_SECRET_NAME ]]; then

      if [[ -n $K8S_SECRET_COMBINE_CA ]]; then
        [[ -z $SILENT ]] && echo "====> Adding CA to Cert file"
        cat ${CA_CERT} >> ${SSL_CERT}
      fi

      [[ -z $SILENT ]] && echo "====> Creating Kubernetes secret: $K8S_SECRET_NAME"
      kubectl delete secret $K8S_SECRET_NAME --ignore-not-found

      if [[ -n $K8S_SECRET_SEPARATE_CA ]]; then
        kubectl create secret generic \
        $K8S_SECRET_NAME \
        --from-file="tls.crt=${SSL_CERT}" \
        --from-file="tls.key=${SSL_KEY}" \
        --from-file="ca.crt=${CA_CERT}"
      else
        kubectl create secret tls \
        $K8S_SECRET_NAME \
        --cert=${SSL_CERT} \
        --key=${SSL_KEY}
      fi

      if [[ -n $K8S_SECRET_LABELS ]]; then
        [[ -z $SILENT ]] && echo "====> Labeling Kubernetes secret"
        IFS=$' \n\t' # We have to reset IFS or label secret will misbehave on some systems
        kubectl label secret \
          $K8S_SECRET_NAME \
          $K8S_SECRET_LABELS
      fi
    fi

    echo "4. 重命名服务证书"
    mv ${CN}.key tls.key
    mv ${CN}.crt tls.crt

ca证书secret

1
2
3
4
5
kubectl -n cattle-system create secret tls tls-rancher-ingress \
--cert=./tls.crt --key=./tls.key

kubectl -n cattle-system create secret generic tls-ca \
--from-file=./cacerts.pem

部署 Rancher 集群

1
2
3
4
5
sudo helm install rancher rancher-stable/rancher \
 --namespace cattle-system \
 --set hostname=rancher.local.com \
 --set ingress.tls.source=secret \
 --set privateCA=true

执行helm，如提示

1
Error: Kubernetes cluster unreachable

因为使用的是k3s的集群，只需执行如下命令手动指定KUBECONFIG的位置即可。

1
执行命令：export KUBECONFIG=/etc/rancher/k3s/k3s.yaml