知乎专栏 | 多维度架构 |
https://gitlab.com/gitlab-examples
Gitlab(仓库) -> Gitlab Runner(持续集成/部署) -> Remote host(远程部署主机)
为远程服务器创建 www 用户,我们将使用该用户远程部署,远程启动程序。
[root@netkiller ~]# groupadd -g 80 www [root@netkiller ~]# adduser -o --uid 80 --gid 80 -G wheel -c "Web Application" www [root@netkiller ~]# id www uid=80(www) gid=80(www) groups=80(www),10(wheel) [root@netkiller ~]# PASSWORD=$(cat /dev/urandom | tr -dc [:alnum:] | head -c 32) [root@netkiller ~]# echo www:${PASSWORD} | chpasswd [root@netkiller ~]# echo "www password: ${PASSWORD}" www password: 0Uz1heY9v9KJyRKbvTi0VlAzfEoFW9GH
mkdir -p /opt/netkiller.cn/www.netkiller.cn chown www:www -R /opt/netkiller.cn
进入项目设置界面,点击 Settings,再点击 CI / CD
点击 Expand 按钮 展开 Runners
这时可以看到 Set up a specific Runner manually, 后面会用到 http://192.168.1.96/ 和 zASzWwffenos6Jbbfsgu
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash dnf install gitlab-runner cp /etc/gitlab-runner/config.toml{,.original} systemctl enable gitlab-runner
使用 SSH 登录 Gitlab runner 服务器,运行 gitlab-runner register
[root@localhost ~]# gitlab-runner register Runtime platform arch=amd64 os=linux pid=92925 revision=ac2a293c version=11.11.2 Running in system-mode. Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/): http://192.168.1.96/ Please enter the gitlab-ci token for this runner: zASzWwffenos6Jbbfsgu Please enter the gitlab-ci description for this runner: [localhost.localdomain]: Please enter the gitlab-ci tags for this runner (comma separated): Registering runner... succeeded runner=zASzWwff Please enter the executor: docker, docker-ssh, shell, ssh, docker-ssh+machine, parallels, virtualbox, docker+machine, kubernetes: shell Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
返回 gitlab 查看注册状态
[root@gitlab ~]# gitlab-runner register Runtime platform arch=amd64 os=linux pid=1020084 revision=c1edb478 version=14.0.1 Running in system-mode. Enter the GitLab instance URL (for example, https://gitlab.com/): http://git.netkiller.cn/ Enter the registration token: DyKdKyaJaq5KN-irgNGz Enter a description for the runner: [gitlab]: Enter tags for the runner (comma-separated): Registering runner... succeeded runner=DyKdKyaJ Enter an executor: parallels, virtualbox, docker+machine, custom, docker, docker-ssh, shell, ssh, docker-ssh+machine, kubernetes: shell Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
/etc/gitlab-runner/config.toml 配置文件
[root@gitlab ~]# cat /etc/gitlab-runner/config.toml concurrent = 1 check_interval = 0 [session_server] session_timeout = 1800 [[runners]] name = "gitlab" url = "http://git.netkiller.cn/" token = "kVkzjDM74xZUN-aKbdPp" executor = "shell" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure]
持续集成和部署运行在 gitlab-runner 用户下,切换到 gitlab-runner 用户
[root@gitlab ~]# su - gitlab-runner Last login: Mon Jul 19 19:01:37 CST 2021
生成 SSH 证书
[gitlab-runner@gitlab ~]$ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/home/gitlab-runner/.ssh/id_rsa): Created directory '/home/gitlab-runner/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/gitlab-runner/.ssh/id_rsa. Your public key has been saved in /home/gitlab-runner/.ssh/id_rsa.pub. The key fingerprint is: SHA256:l90LYBeSF9l9JHXJUHeO+IyvscCziz4C8vFNpJoKEjo gitlab-runner@gitlab The key's randomart image is: +---[RSA 3072]----+ | ..o===B| | ..oo.**| | o.o . o| | .. = = | |. oS o + + | |... o . .o o . | |E o * o + . o | |.o + o o. + + | | .. oo.o.o | +----[SHA256]-----+ [gitlab-runner@gitlab ~]$
正常情况下,当我们链接一个 SSH 主机,会让我们输入 yes 确认继续链接。
[gitlab-runner@gitlab ~]$ ssh www@192.168.40.10 The authenticity of host '192.168.40.10 (192.168.40.10)' can't be established. ECDSA key fingerprint is SHA256:xmFF266MPdXhnlAljS+QWhQsw6jOw1sOwQXRr/PHi2w. Are you sure you want to continue connecting (yes/no/[fingerprint])?
配置 SSH
[gitlab-runner@gitlab ~]$ cat > ~/.ssh/config <<'EOF' Host * ServerAliveInterval=30 StrictHostKeyChecking no UserKnownHostsFile=/dev/null EOF chmod 600 -R ~/.ssh/config
授权远程执行 Shell
[gitlab-runner@gitlab ~]$ ssh-copy-id www@www.netkiller.cn
在构建过程中,我们需要备份数据库/同步数据库,下面安装了一些所需的工具
[root@localhost ~]# dnf install -y mysql
设置数据库备份账号和密码,这里偷懒使用了 root 账号,生产环境请创建专用的备份账号。
[root@localhost ~]# su - gitlab-runner Last login: Wed Sep 1 19:17:48 CST 2021 [gitlab-runner@localhost ~]$ vim ~/.my.cnf [gitlab-runner@localhost ~]$ cat ~/.my.cnf [mysql] user=root password=test [mysqldump] user=root password=test
测试数据库是否畅通
[gitlab-runner@localhost ~]$ mysql -h mysql.netkiller.cn Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 37602 Server version: 8.0.21 Source distribution Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql>
JRE:java-11-openjdk
JDK:java-11-openjdk-devel
[root@gitlab ~]# dnf install -y java-11-openjdk java-11-openjdk-devel [root@gitlab ~]# dnf install -y maven
修改 Maven 镜像路
[root@gitlab ~]# vim /etc/maven/settings.xml <mirrors> <mirror> <id>aliyun</id> <name>aliyun maven</name> <url>http://maven.aliyun.com/nexus/content/groups/public/</url> <mirrorOf>central</mirrorOf> </mirror> </mirrors>
如果需要安装最新版本 maven 使用下面脚本。
#!/bin/bash cd /usr/local/src/ wget https://mirrors.bfsu.edu.cn/apache/maven/maven-3/3.8.2/binaries/apache-maven-3.8.2-bin.tar.gz tar zxf apache-maven-3.8.2-bin.tar.gz mv apache-maven-3.8.2 /srv/ rm -f /srv/apache-maven ln -s /srv/apache-maven-3.8.2 /srv/apache-maven alternatives --install /usr/local/bin/mvn apache-maven-3.8.2 /srv/apache-maven-3.8.2/bin/mvn 0
[root@localhost src]# mvn -v Apache Maven 3.8.2 (ea98e05a04480131370aa0c110b8c54cf726c06f) Maven home: /srv/apache-maven-3.8.2 Java version: 17-ea, vendor: Red Hat, Inc., runtime: /usr/lib/jvm/java-17-openjdk-17.0.0.0.26-0.2.ea.el8.x86_64 Default locale: en_US, platform encoding: ANSI_X3.4-1968 OS name: "linux", version: "4.18.0-338.el8.x86_64", arch: "amd64", family: "unix"
apache-maven-3.8.2 配置
[root@localhost ~]# vim /srv/apache-maven/conf/settings.xml <mirrors> <!-- mirror | Specifies a repository mirror site to use instead of a given repository. The repository that | this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used | for inheritance and direct lookup purposes, and must be unique across the set of mirrors. | <mirror> <id>mirrorId</id> <mirrorOf>repositoryId</mirrorOf> <name>Human Readable Name for this Mirror.</name> <url>http://my.repository.com/repo/path</url> </mirror> --> <mirror> <id>maven-default-http-blocker</id> <mirrorOf>external:http:*</mirrorOf> <name>Pseudo repository to mirror external repositories initially using HTTP.</name> <url>http://0.0.0.0/</url> <blocked>true</blocked> </mirror> </mirrors>
apache-maven-3.8.2 默认会阻止其他镜像,需要会去掉 maven-default-http-blocker 配置
切换到 gitlab-runner 用户,随便运行一下 mvn 命令,这样就会产生 ~/.m2 文件夹
[root@gitlab ~]# su - gitlab-runner [gitlab-runner@gitlab ~]$ mvn -v
mvnd 是一个实验产品,用于替代 maven 编译速度比较快
cd /usr/local/src wget https://github.com/apache/maven-mvnd/releases/download/0.7.1/mvnd-0.7.1-linux-amd64.zip unzip mvnd-0.7.1-linux-amd64.zip mv mvnd-0.7.1-linux-amd64 /srv/mvnd-0.7.1 ln -s /srv/mvnd-0.7.1 /srv/mvnd alternatives --remove mvnd /usr/local/bin/mvnd alternatives --install /usr/local/bin/mvnd mvnd-0.7.1 /srv/mvnd-0.7.1/bin/mvnd 0
修改配置文件 mvnd.properties 制定 JAVA_HOME
[root@localhost cloud.netkiller.cn]# grep java.home /srv/mvnd/conf/mvnd.properties java.home=/usr/lib/jvm/java
[root@netkiller ~]# dnf install -y nodejs
安装 cnpm
[root@netkiller ~]# npm config set registry https://registry.npm.taobao.org [root@netkiller ~]# npm config get registry https://registry.npm.taobao.org/ [root@netkiller ~]# npm install -g cnpm
yarn
[root@netkiller ~]# curl -sL https://dl.yarnpkg.com/rpm/yarn.repo -o /etc/yum.repos.d/yarn.repo [root@netkiller ~]# dnf install -y yarn
yarn config set registry https://registry.npm.taobao.org
pm2 进程管理
[root@netkiller ~]# npm install -g pm2
设置 pm2 启动开启
[root@netkiller ~]# pm2 startup [root@netkiller ~]# pm2 save --force [root@netkiller ~]# systemctl enable pm2-root [root@netkiller ~]# systemctl start pm2-root [root@netkiller ~]# systemctl status pm2-root
[gitlab-runner@gitlab api.netkiller.cn]$ ssh www@192.168.40.10 "sudo ls" Warning: Permanently added '192.168.40.10' (ECDSA) to the list of known hosts. sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
解决方案一
ssh -t www@www.netkiller.cn "echo <yourpassword> |sudo -S <yourcommand>"
解决方案二
cat > /etc/sudoers.d/www <<-EOF www ALL=(ALL) NOPASSWD: ALL EOF
tags 是给 Gitlab Runner 打个标签,我的用法是多次注册,例如 shell 执行器的标签是 shell, Docker 执行器的标签是 docker,这样便可以在.gitlab-ci.yml文件中来选择使用那个执行器来触发操作。
下面是 Shell 执行器
[root@localhost ~]# gitlab-runner register Runtime platform arch=amd64 os=linux pid=268363 revision=58ba2b95 version=14.2.0 Running in system-mode. Enter the GitLab instance URL (for example, https://gitlab.com/): http://git.netkiller.cn/ Enter the registration token: k_SsvMQV397gAMaP_q1v Enter a description for the runner: [localhost.localdomain]: development Enter tags for the runner (comma-separated): shell Registering runner... succeeded runner=k_SsvMQV Enter an executor: docker, docker-ssh, virtualbox, docker-ssh+machine, kubernetes, custom, parallels, shell, ssh, docker+machine: shell Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
下面是 Docker 执行器
[root@localhost ~]# gitlab-runner register Runtime platform arch=amd64 os=linux pid=268397 revision=58ba2b95 version=14.2.0 Running in system-mode. Enter the GitLab instance URL (for example, https://gitlab.com/): http://git.netkiller.cn/ Enter the registration token: k_SsvMQV397gAMaP_q1v Enter a description for the runner: [localhost.localdomain]: development Enter tags for the runner (comma-separated): docker Registering runner... succeeded runner=k_SsvMQV Enter an executor: custom, docker-ssh, parallels, shell, ssh, docker-ssh+machine, docker, virtualbox, docker+machine, kubernetes: docker Enter the default Docker image (for example, ruby:2.6): maven:latest Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
注册后的效果
[root@localhost ~]# cat /etc/gitlab-runner/config.toml concurrent = 1 check_interval = 0 [session_server] session_timeout = 1800 [[runners]] name = "development" url = "http://git.netkiller.cn/" token = "EztTBypKRW5ibtC5rs2h" executor = "shell" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure] [[runners]] name = "development" url = "http://git.netkiller.cn/" token = "51948sQbQsXGV-RxFMty" executor = "docker" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure] [runners.docker] tls_verify = false image = "maven:latest" privileged = false disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = false volumes = ["/cache"] shm_size = 0
gitlab-runner 用户需要 访问 /var/run/docker.sock 所以需要将 gitlab-runner 用户加入到 docker 组中。
[root@gitlab ~]# ll /var/run/docker.sock srw-rw---- 1 root docker 0 Nov 25 17:04 /var/run/docker.sock [root@gitlab ~]# id gitlab-runner uid=989(gitlab-runner) gid=984(gitlab-runner) groups=984(gitlab-runner) [root@gitlab ~]# usermod -aG docker gitlab-runner [root@gitlab ~]# id gitlab-runner uid=989(gitlab-runner) gid=984(gitlab-runner) groups=984(gitlab-runner),991(docker)
注册 Docker 执行器
[root@localhost ~]# gitlab-runner register Runtime platform arch=amd64 os=linux pid=268397 revision=58ba2b95 version=14.2.0 Running in system-mode. Enter the GitLab instance URL (for example, https://gitlab.com/): http://git.netkiller.cn/ Enter the registration token: k_SsvMQV397gAMaP_q1v Enter a description for the runner: [localhost.localdomain]: development Enter tags for the runner (comma-separated): docker Registering runner... succeeded runner=k_SsvMQV Enter an executor: custom, docker-ssh, parallels, shell, ssh, docker-ssh+machine, docker, virtualbox, docker+machine, kubernetes: docker Enter the default Docker image (for example, ruby:2.6): maven:latest Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
配置缓存
[root@localhost ~]# cat /etc/gitlab-runner/config.toml concurrent = 1 check_interval = 0 [session_server] session_timeout = 1800 [[runners]] name = "development" url = "http://192.168.30.5/" token = "EztTBypKRW5ibtC5rs2h" executor = "shell" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure] [[runners]] name = "development" url = "http://192.168.30.5/" token = "GP-ozvd6uw2nDxyRohZ-" executor = "docker" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure] [runners.docker] tls_verify = false image = "maven:latest" privileged = false disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = false volumes = ["/cache","/root/.m2"] pull_policy = ["never"] shm_size = 0
volumes = ["/cache","/root/.m2"] 将 Maven 仓库缓存
.gitlab-ci.yaml 编排脚本
cache: untracked: true stages: - build - test - deploy build-job: image: maven:3.8.2-openjdk-17 stage: build tags: - docker script: - mvn clean package -Dmaven.test.skip=true - ls target/*.jar artifacts: name: "$CI_PROJECT_NAME" paths: - target/*.jar test-job: image: maven:3.8.2-openjdk-17 stage: test variables: GIT_STRATEGY: none tags: - docker script: - mvn test deploy-job: stage: deploy variables: GIT_STRATEGY: none HOST: 192.168.30.14 DOCKER_HOST: unix:///var/run/docker.sock mvn clean install docker:build environment: name: development url: https://api.netkiller.cn only: - development tags: - shell before_script: - mvn docker:build -DpushImage # - mvn docker:push - rm -rf *.sql.gz - mysqldump -hmysql.netkiller.cn test | gzip > test.$(date -u +%Y-%m-%d.%H:%M:%S).sql.gz artifacts: name: "$CI_PROJECT_NAME" paths: - ./*.sql.gz script: - scp src/main/docker/docker-compose.yaml www@$HOST:/opt/netkiller.cn/api.netkiller.cn/ - ssh www@$HOST "sudo docker-compose -f /opt/netkiller.cn/api.netkiller.cn/docker-compose.yaml up" - ssh www@$HOST "sudo docker-compose -f /opt/netkiller.cn/api.netkiller.cn/docker-compose.yaml restart"
[root@agent-5 ~]# gitlab-runner register Runtime platform arch=amd64 os=linux pid=1259091 revision=43b2dc3d version=15.4.0 Running in system-mode. Enter the GitLab instance URL (for example, https://gitlab.com/): https://gitlab.netkiller.cn/ Enter the registration token: GR1348941WLrZvRebkiCocQgdGFwC Enter a description for the runner: [agent-5]: Kubernetes executor Enter tags for the runner (comma-separated): kubernetes Enter optional maintenance note for the runner: Registering runner... succeeded runner=GR1348941WLrZvReb Enter an executor: docker-ssh, shell, docker+machine, docker-ssh+machine, kubernetes, custom, docker, parallels, ssh, virtualbox: kubernetes Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! Configuration (with the authentication token) was saved in "/etc/gitlab-runner/config.toml"
/etc/gitlab-runner/config.toml
[root@agent-5 ~]# cat /etc/gitlab-runner/config.toml concurrent = 1 check_interval = 0 [session_server] session_timeout = 1800 [[runners]] name = "Kubernetes executor" url = "https://gitlab.netkiller.cn/" id = 3 token = "5J6amB15rYWie_zGscFC" token_obtained_at = 2022-10-19T07:16:07Z token_expires_at = 0001-01-01T00:00:00Z executor = "kubernetes" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure] [runners.kubernetes] host = "" bearer_token_overwrite_allowed = false image = "" namespace = "" namespace_overwrite_allowed = "" pod_labels_overwrite_allowed = "" service_account_overwrite_allowed = "" pod_annotations_overwrite_allowed = "" [runners.kubernetes.affinity] [runners.kubernetes.pod_security_context] [runners.kubernetes.init_permissions_container_security_context] [runners.kubernetes.init_permissions_container_security_context.capabilities] [runners.kubernetes.build_container_security_context] [runners.kubernetes.build_container_security_context.capabilities] [runners.kubernetes.helper_container_security_context] [runners.kubernetes.helper_container_security_context.capabilities] [runners.kubernetes.service_container_security_context] [runners.kubernetes.service_container_security_context.capabilities] [runners.kubernetes.volumes] [runners.kubernetes.dns_config] [runners.kubernetes.container_lifecycle]
将CA证书复制给 gitlab-runner
mkdir /etc/ssl/kubernetes [root@master ~]# scp /var/lib/rancher/k3s/server/tls/server-ca.crt root@agent-5:/etc/ssl/kubernetes/ca.crt
或者用下面命令查看 CA 证书,然后保存到 /etc/ssl/kubernetes/ca.crt 文件
[gitlab-runner@agent-5 ~]$ kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 -d -----BEGIN CERTIFICATE----- MIIBeDCCAR2gAwIBAgIBADAKBggqhkjOPQQDAjAjMSEwHwYDVQQDDBhrM3Mtc2Vy dmVyLWNhQDE2NjI2MTYwOTMwHhcNMjIwOTA4MDU0ODEzWhcNMzIwOTA1MDU0ODEz WjAjMSEwHwYDVQQDDBhrM3Mtc2VydmVyLWNhQDE2NjI2MTYwOTMwWTATBgcqhkjO PQIBBggqhkjOPQMBBwNCAARGT8u7K3jFNKGid7qMWSYUMuv+kYQzvk5RQHFyEXA6 zNnGd0PBpDvsKpZGjkIwJnla0v98nFzsK6hp9eEDIVw3o0IwQDAOBgNVHQ8BAf8E BAMCAqQwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQU7IRDLJoNv7bF8wkkJ4yQ FA6gTBowCgYIKoZIzj0EAwIDSQAwRgIhAKMiz13pxq+IIXZfZT5R+Lh+pDoX2H1u AskoLxoAutCPAiEA4ubxiK1DqjatxGb1/ovMLd4pfcPeAvg1AIokwhFhueU= -----END CERTIFICATE-----
[root@master ~]# kubectl create serviceaccount secrets serviceaccount/secrets created [root@master ~]# kubectl create token secrets [root@master ~]# cat role.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: secrets namespace: default rules: - apiGroups: ["*"] resources: ["pods"] verbs: ["list", "get", "watch", "create", "delete"] - apiGroups: ["*"] resources: ["pods/exec"] verbs: ["create"] - apiGroups: ["*"] resources: ["pods/log"] verbs: ["get"] - apiGroups: ["*"] resources: ["pods/attach"] verbs: ["list", "get", "create", "delete", "update"] - apiGroups: ["*"] resources: ["secrets"] verbs: ["list", "get", "create", "delete", "update"] - apiGroups: ["*"] resources: ["configmaps"] verbs: ["list", "get", "create", "delete", "update"] [root@master ~]# kubectl create rolebinding gitlab-runner-binding --role=secrets --serviceaccount=default:secrets
[[runners]] name = "Kubernetes executor" url = "https://gitlab.netkiller.cn/" id = 3 token = "5J6amB15rYWie_zGscFC" token_obtained_at = 2022-10-19T07:16:07Z token_expires_at = 0001-01-01T00:00:00Z executor = "kubernetes" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure] [runners.kubernetes] host = "https://k8s.netkiller.cn:6443" ca_file = "/etc/ssl/kubernetes/ca.crt" tls_verify = true bearer_token_overwrite_allowed = true image = "" namespace = "" namespace_overwrite_allowed = "" pod_labels_overwrite_allowed = "" service_account_overwrite_allowed = "" pod_annotations_overwrite_allowed = "" [runners.kubernetes.affinity] [runners.kubernetes.pod_security_context] [runners.kubernetes.init_permissions_container_security_context] [runners.kubernetes.init_permissions_container_security_context.capabilities] [runners.kubernetes.build_container_security_context] [runners.kubernetes.build_container_security_context.capabilities] [runners.kubernetes.helper_container_security_context] [runners.kubernetes.helper_container_security_context.capabilities] [runners.kubernetes.service_container_security_context] [runners.kubernetes.service_container_security_context.capabilities] [runners.kubernetes.volumes] [runners.kubernetes.dns_config] [runners.kubernetes.container_lifecycle]
.gitlab-ci.yml
cache: # untracked: true paths: - target/ #variables: # KUBERNETES_SERVICE_ACCOUNT: secrets # KUBERNETES_BEARER_TOKEN: eyJhbGciOiJSUzI1NiIsImtpZCI6IktCOHRvYlZOLXFPRmEyb1JWdlQxSzBvN0tvZF9HNFBGRnlraDR5UU1jakkifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJrM3MiXSwiZXhwIjoxNjY2MTc0Mzk1LCJpYXQiOjE2NjYxNzA3OTUsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZWZhdWx0Iiwic2VydmljZWFjY291bnQiOnsibmFtZSI6InNlY3JldHMiLCJ1aWQiOiIxMTM5NjVlMy1iZGVkLTQ5NGEtOGMyNS0zYjU3OTFmMTIzZjEifX0sIm5iZiI6MTY2NjE3MDc5NSwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6c2VjcmV0cyJ9.KvTpp7vplWIBWZFKYK-zPFk0NJhoiMHHiE-0Bj3qmFvHaxF2D3A8YrsEfQxSIJ1p8J5IZSPYiAX_BqoNi4-fziFsJIIbza0bcPj-RWLoBY2Nz0gyzw93r2to71WS90OSxryWRMMmqr8rCFO0ewtdOPzC8-PQ1uSIblJ3gM2EbN7b9VfHJssrog7UMWcT0GuiNM27AzwxYmEvkioeTmQaCzLNUGpyFnu1wg0e7mHzHMxPwSwMiOUFEE7SKKFpyZT8ZLc8ZgEfZKMXK2FwCTpoktBr_h7u-2zpNK4x9Dwl2aqkMBNZL-QVNpaXXnA0K20PAVjK5-x7IkiELFFq_CSezg stages: - build # - deploy build-job: stage: build image: maven:latest tags: # - docker - kubernetes # before_script: # after_script: script: - mvn -T 1C -Dmaven.test.skip=true package artifacts: name: "$CI_PROJECT_NAME" paths: - target/*.jar
[root@master ~]# kubectl create namespace gitlab namespace/gitlab-runner created [root@master ~]# kubectl create serviceaccount gitlab-runner -n gitlab serviceaccount/secrets created [root@master ~]# kubectl create token gitlab-runner -n gitlab [root@master ~]# cat role.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: gitlab-runner namespace: gitlab rules: - apiGroups: ["*"] resources: ["pods"] verbs: ["list", "get", "watch", "create", "delete"] - apiGroups: ["*"] resources: ["pods/exec"] verbs: ["create"] - apiGroups: ["*"] resources: ["pods/log"] verbs: ["get"] - apiGroups: ["*"] resources: ["pods/attach"] verbs: ["list", "get", "create", "delete", "update"] - apiGroups: ["*"] resources: ["secrets"] verbs: ["list", "get", "create", "delete", "update"] - apiGroups: ["*"] resources: ["configmaps"] verbs: ["list", "get", "create", "delete", "update"] [root@master ~]# kubectl apply -f role.yaml role.rbac.authorization.k8s.io/gitlab created [root@master ~]# kubectl create rolebinding gitlab-runner --namespace=gitlab --role=gitlab-runner --serviceaccount=gitlab:gitlab-runner
kubectl create -n gitlab -f - <<EOF apiVersion: v1 kind: Secret metadata: name: gitlab-runner-token annotations: kubernetes.io/service-account.name: gitlab-runner type: kubernetes.io/service-account-token EOF kubectl get secret gitlab-runner-token -o jsonpath='{.data.token}' -n gitlab | base64 -d
账号和Token创建完毕之后,使用下面命令检查
kubectl get namespace gitlab kubectl get serviceaccounts gitlab-runner -n gitlab kubectl get role gitlab-runner -n gitlab kubectl get rolebinding gitlab-runner -n gitlab kubectl get secrets gitlab-runner-token -n gitlab kubectl get secret gitlab-runner-token -o jsonpath='{.data.token}' -n gitlab | base64 -d
创建 PVC
apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: driver.longhorn.io volume.kubernetes.io/storage-provisioner: driver.longhorn.io creationTimestamp: "2022-10-19T12:46:33Z" finalizers: - kubernetes.io/pvc-protection managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:spec: f:accessModes: {} f:resources: f:requests: .: {} f:storage: {} f:storageClassName: {} f:volumeMode: {} manager: kube-explorer operation: Update time: "2022-10-19T12:46:33Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:pv.kubernetes.io/bind-completed: {} f:pv.kubernetes.io/bound-by-controller: {} f:volume.beta.kubernetes.io/storage-provisioner: {} f:volume.kubernetes.io/storage-provisioner: {} f:spec: f:volumeName: {} manager: k3s operation: Update time: "2022-10-19T12:46:35Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:status: f:accessModes: {} f:capacity: .: {} f:storage: {} f:phase: {} manager: k3s operation: Update subresource: status time: "2022-10-19T12:46:35Z" name: maven namespace: gitlab resourceVersion: "4846862" uid: ab0dc609-ff07-438e-b537-7e6182cec008 spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: longhorn volumeMode: Filesystem volumeName: pvc-ab0dc609-ff07-438e-b537-7e6182cec008 status: accessModes: - ReadWriteOnce capacity: storage: 10Gi phase: Bound apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: driver.longhorn.io volume.kubernetes.io/storage-provisioner: driver.longhorn.io creationTimestamp: "2022-10-19T12:48:39Z" finalizers: - kubernetes.io/pvc-protection managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:spec: f:accessModes: {} f:resources: f:requests: .: {} f:storage: {} f:storageClassName: {} f:volumeMode: {} manager: kube-explorer operation: Update time: "2022-10-19T12:48:39Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:pv.kubernetes.io/bind-completed: {} f:pv.kubernetes.io/bound-by-controller: {} f:volume.beta.kubernetes.io/storage-provisioner: {} f:volume.kubernetes.io/storage-provisioner: {} f:spec: f:volumeName: {} manager: k3s operation: Update time: "2022-10-19T12:48:42Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:status: f:accessModes: {} f:capacity: .: {} f:storage: {} f:phase: {} manager: k3s operation: Update subresource: status time: "2022-10-19T12:48:42Z" name: builds namespace: gitlab resourceVersion: "4847301" uid: 09f9a7b0-558a-4142-b9b3-e1318aff223a spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: longhorn-storage volumeMode: Filesystem volumeName: pvc-09f9a7b0-558a-4142-b9b3-e1318aff223a
配置 Gitlab Runner
[[runners]] name = "Kubernetes executor" url = "https://gitlab.netkiller.cn/" id = 3 token = "5J6amB15rYWie_zGscFC" token_obtained_at = 2022-10-19T07:16:07Z token_expires_at = 0001-01-01T00:00:00Z executor = "kubernetes" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure] [runners.kubernetes] host = "https://172.18.200.5:6443" #cert_file = "/etc/ssl/kubernetes/api.crt" #key_file = "/etc/ssl/kubernetes/api.key" ca_file = "/etc/ssl/kubernetes/ca.crt" tls_verify = true bearer_token = "eyJhbGciOiJSUzI1NiIsImtpZCI6IktCOHRvYlZOLXFPRmEyb1JWdlQxSzBvN0tvZF9HNFBGRnlraDR5UU1jakkifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJrM3MiXSwiZXhwIjoxNjY2MTg1NDc5LCJpYXQiOjE2NjYxODE4NzksImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJnaXRsYWIiLCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoiZ2l0bGFiLXJ1bm5lciIsInVpZCI6IjdjYjhmNjM3LTc5YjQtNDc5Yi1hZjAzLTRhOGQ2ZDliM2YzOSJ9fSwibmJmIjoxNjY2MTgxODc5LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6Z2l0bGFiOmdpdGxhYi1ydW5uZXIifQ.ExhowiFt9k-vFwaxohs_KGF_JmzPIMnUyoOzENN9g250taWu3jwtRasixIckYhO5JavpzZRU1XTdhRmbEQ9hehlY5q6kXtGCZ4AImh-q8gj7lglBLdWvZLMcmaqu3o4pIjb7Cw0qP25y-F79IeDtUPZeLU2b_ocTIM5GPd5g6MxilTzw84GJ_hVevWVtdpTrRg48b64S6crAWpkUgMSVwyqwyjsTkGuEcn_V32fzO9ro7EjumuWxVmOp5TA0mOXl4L8W1C6ZYVT_qGKsMJJ_AS3uZkKqcxT6FmSS0Ta4vjgQsVN1l6OjQNmrIWFBjRzNCiXPhlk5JzR0ZID2J20UQQ" bearer_token_overwrite_allowed = true privileged = true image = "" namespace = "gitlab" namespace_overwrite_allowed = "" pod_labels_overwrite_allowed = "" service_account = "gitlab-runner" service_account_overwrite_allowed = "" pod_annotations_overwrite_allowed = "" [runners.kubernetes.affinity] [runners.kubernetes.pod_security_context] [runners.kubernetes.init_permissions_container_security_context] [runners.kubernetes.init_permissions_container_security_context.capabilities] [runners.kubernetes.build_container_security_context] [runners.kubernetes.build_container_security_context.capabilities] [runners.kubernetes.helper_container_security_context] [runners.kubernetes.helper_container_security_context.capabilities] [runners.kubernetes.service_container_security_context] [runners.kubernetes.service_container_security_context.capabilities] [runners.kubernetes.volumes] [[runners.kubernetes.volumes.host_path]] name = "cache" mount_path = "/cache" host_path = "/tmp/cache" [[runners.kubernetes.volumes.pvc]] name = "maven" mount_path = "/root/.m2" [[runners.kubernetes.volumes.pvc]] name = "builds" mount_path = "/builds" [runners.kubernetes.dns_config] [runners.kubernetes.container_lifecycle]
/etc/gitlab-runner/config.toml
bearer_token_overwrite_allowed = false 改为 bearer_token_overwrite_allowed = true
.gitlab-ci.yml
variables: KUBERNETES_SERVICE_ACCOUNT: gitlab KUBERNETES_BEARER_TOKEN: eyJhbGciOiJSUzI1NiIsImtpZCI6IktCOHRvYlZOLXFPRmEyb1JWdlQxSzBvN0tvZF9HNFBGRnlraDR5UU1jakkifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJrM3MiXSwiZXhwIjoxNjY2MTg1MTE3LCJpYXQiOjE2NjYxODE1MTcsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJnaXRsYWItcnVubmVyIiwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImdpdGxhYiIsInVpZCI6Ijc3MGE4ODk1LTcwNjAtNGUwYi04MTc0LTVkNmMzNDUzYTQxOSJ9fSwibmJmIjoxNjY2MTgxNTE3LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6Z2l0bGFiLXJ1bm5lcjpnaXRsYWIifQ.PO96hsJSG7-h2EJhDoUc6xWCPzDp9AwoZ1-CrsM-GUN73ft7ge9prdarq0of-rzgAP-wn-07r6vIxdcT2G0ZHl-GsElyxaAm5Poz1WX2bQLuLvcpoFVQyGFH1Mkkno5MDCfi2CPfNYUuEBl5vQ_jDooE5dUnildLmbEv7ooxLYeWAPnEd5HmOyXKjC2FiBqKQ88oxkDbmq3Hwbxm-XmTma7T3NqXxST_m7qE6tb2n0RsG3o1lJGEDftdf9bF1eEfAykaa077tzHNNZtQvK87LK69XZVUbOVun_G98-rLL7afSridgP6mhia6CPeful7xvedJ8l4g8V-Ku8qixKb5Yg
[root@agent-5 ~]# cat /etc/gitlab-runner/config.toml concurrent = 5 check_interval = 0 [session_server] session_timeout = 1800 [[runners]] name = "Kubernetes local cluster" url = "https://gitlab.netkiller.cn/" id = 41 token = "y1QnvNhSwMYVX2-z3x4E" token_obtained_at = 2022-10-20T00:19:08Z token_expires_at = 0001-01-01T00:00:00Z executor = "kubernetes" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure] [runners.kubernetes] host = "https://172.18.200.5:6443" ca_file = "/etc/ssl/kubernetes/ca.crt" bearer_token_overwrite_allowed = true bearer_token = "eyJhbGciOiJSUzI1NiIsImtpZCI6IktCOHRvYlZOLXFPRmEyb1JWdlQxSzBvN0tvZF9HNFBGRnlraDR5UU1jakkifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJrM3MiXSwiZXhwIjoxNjY2MjMwNjc5LCJpYXQiOjE2NjYyMjcwNzksImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJnaXRsYWIiLCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoiZ2l0bGFiLXJ1bm5lciIsInVpZCI6IjdjYjhmNjM3LTc5YjQtNDc5Yi1hZjAzLTRhOGQ2ZDliM2YzOSJ9fSwibmJmIjoxNjY2MjI3MDc5LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6Z2l0bGFiOmdpdGxhYi1ydW5uZXIifQ.e6RvBIUCUlZ4ciVKRQ-i2kEBBTXaPq876L-EzA5NGNWxXmfkuyx_IpGrrzmTMPV9prR3XLQfmyT1OoEgkO8riMLtJL1yPZNiaTo0zhAfuWvvXP3gOnJsFRdcx2PsvA1dxuuhtXNp1Y2BAboU8jV10-OK6WU40i3CKb9OTQpN-BRAlwXylQ-55ZIRT7J3ghpCLfM6BplwZOYfCP6XbHJfgAKD2wy5sl8Ni1XIpsLUOp20TlBaG22k2admcWiRavxyjp68EHMwq3izI5_4qU9hFcYBOsUY-PBC27nGw9ICH0sIRuq4Xs0GQ8oXOzS5Dbja5uKKVq6-bWV3onV13AjcxQ" image = "" namespace = "gitlab" namespace_overwrite_allowed = "" pull_policy = ["if-not-present"] image_pull_secrets = ["registry"] pod_labels_overwrite_allowed = "" service_account = "gitlab-runner" service_account_overwrite_allowed = "" pod_annotations_overwrite_allowed = "" [runners.kubernetes.affinity] [runners.kubernetes.pod_security_context] [runners.kubernetes.init_permissions_container_security_context] [runners.kubernetes.init_permissions_container_security_context.capabilities] [runners.kubernetes.build_container_security_context] [runners.kubernetes.build_container_security_context.capabilities] [runners.kubernetes.helper_container_security_context] [runners.kubernetes.helper_container_security_context.capabilities] [runners.kubernetes.service_container_security_context] [runners.kubernetes.service_container_security_context.capabilities] [runners.kubernetes.volumes] [[runners.kubernetes.volumes.host_path]] name = "docker" mount_path = "/var/run/docker.sock" host_path = "/var/run/docker.sock" [[runners.kubernetes.volumes.pvc]] name = "maven" mount_path = "/root/.m2" [[runners.kubernetes.volumes.pvc]] name = "builds" mount_path = "/builds" [[runners.kubernetes.volumes.pvc]] name = "cache" mount_path = "/cache" [runners.kubernetes.dns_config] [runners.kubernetes.container_lifecycle] [[runners]] name = "Docker" url = "https://gitlab.netkiller.cn/" id = 42 token = "Y7Df44aY8YrRwASUXWE5" token_obtained_at = 2022-10-20T02:40:25Z token_expires_at = 0001-01-01T00:00:00Z executor = "docker" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure] [runners.docker] tls_verify = false image = "docker:latest" privileged = false disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = false volumes = ["/cache"] shm_size = 0 [[runners]] name = "Shell" url = "https://gitlab.netkiller.cn/" id = 43 token = "s2tuoKTrj1s1iv_mfYh5" token_obtained_at = 2022-10-20T03:18:08Z token_expires_at = 0001-01-01T00:00:00Z executor = "shell" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure]
.gitlab-ci.yml
stages: - build - docker - deploy variables: DOCKER_REGISTRY: registry.netkiller.cn IMAGE: $DOCKER_REGISTRY/$CI_COMMIT_BRANCH/$CI_PROJECT_NAME:$CI_COMMIT_SHORT_SHA-$CI_PIPELINE_ID KUBERNETES_BEARER_TOKEN: eyJhbGciOiJSUzI1NiIsImtpZCI6IktCOHRvYlZOLXFPRmEyb1JWdlQxSzBvN0tvZF9HNFBGRnlraDR5UU1jakkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJnaXRsYWIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiZ2l0bGFiLXJ1bm5lci10b2tlbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJnaXRsYWItcnVubmVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiN2NiOGY2MzctNzliNC00NzliLWFmMDMtNGE4ZDZkOWIzZjM5Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmdpdGxhYjpnaXRsYWItcnVubmVyIn0.pU4-8D4szeL8iud1SvesdN7nV7L3GLaNsa2UbsxkGQ4SDGN85zKTXJl6MtqDsuJB9HBUlOTMnyEa0gCbgHOJlR3fd2HcegitrRLeybvUuotniiLpCPO7vAO-oS5Fej7oUFBXqZJYIx-xMbFoyt3rnGs273c_yE8avI8EGdEPNhOWRgF_GZBYstvwiEjO2IUDWbutzCTtGloPvJ5Ur0s7drLJkCQvT2nod5tSSnY5R0lpNyD2FodkFR28KU1EgFoHUnH_ERtUAS5qObIETWSwm5SmCnd2Ogjh70DDxmIHSU-saFU0zSqPpZ1oX9hgO9YMkcJXPHOEnqIVEagZ5CSf2w cache: key: ${CI_COMMIT_REF_SLUG} paths: - target/ build-job: stage: build image: registry.netkiller.cn/common/maven:latest script: - mvn clean package -Dautoconfig.skip=true -Dmaven.test.skip=true -Dmaven.test.failure.ignore=true after_script: - md5sum target/*.jar only: - dev - test tags: - kubernetes artifacts: name: "$CI_PROJECT_NAME" paths: - target/*.jar build-docker: stage: docker image: docker:latest before_script: - echo "$CI_REGISTRY_PASSWORD" | docker login $DOCKER_REGISTRY --username $CI_REGISTRY_USER --password-stdin after_script: - docker images | grep $CI_PROJECT_NAME script: - docker build -t $IMAGE -f Dockerfile . - docker push $IMAGE only: - dev - test tags: - kubernetes deploy-job: stage: deploy variables: GIT_STRATEGY: none before_script: - kubectl -n test get pod | grep $CI_PROJECT_NAME script: - kubectl set image deployment/${CI_PROJECT_NAME} ${CI_PROJECT_NAME}=${IMAGE} -n ${CI_COMMIT_BRANCH} after_script: - kubectl -n test get pod | grep $CI_PROJECT_NAME only: - dev - test tags: - shell environment: name: $CI_COMMIT_BRANCH url: $CI_COMMIT_BRANCH.netkiller.cn/api/monitor/health
自建 Maven 仓库,需要配置 settings.xml,这时就需要制作一个 Maven 镜像,同事有 COPY 命令把 settings.xml 文件复制到 /usr/share/maven/conf/settings.xml 目录
[root@netkiller jdk11]# ls Dockerfile build.sh settings.xml [root@netkiller jdk11]# cat Dockerfile FROM maven:3.8.6-openjdk-11 COPY settings.xml /root/.m2/settings.xml COPY settings.xml /usr/share/maven/conf/settings.xml [root@netkiller jdk11]# cat build.sh docker build -t "registry.netkiller.cn/share/maven:3.8.6-openjdk-11" . docker push registry.netkiller.cn/share/maven:3.8.6-openjdk-11
JaCoCo Java Code Coverage Library https://www.jacoco.org/jacoco/index.html
pom.xml 中必须有单元测试依赖
<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <scope>test</scope> </dependency>
不能跳过单元测试
<plugin> <artifactId>maven-surefire-plugin</artifactId> <configuration> <skip>false</skip> </configuration> </plugin>
添加 JaCoCo 插件
<plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <executions> <execution> <goals> <goal>prepare-agent</goal> </goals> </execution> <execution> <id>report</id> <phase>test</phase> <goals> <goal>report</goal> </goals> </execution> </executions> </plugin>
最后运行 mvn test 调试一下,输入类似下面
[INFO] ------------------------< cn.netkiller:config >------------------------- [INFO] Building config 0.0.1-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- jacoco-maven-plugin:0.8.7:prepare-agent (default) @ config --- [INFO] argLine set to -javaagent:/Users/neo/.m2/repository/org/jacoco/org.jacoco.agent/0.8.7/org.jacoco.agent-0.8.7-runtime.jar=destfile=/Users/neo/workspace/microservice/config/target/jacoco.exec [INFO] [INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ config --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Using 'UTF-8' encoding to copy filtered properties files. [INFO] Copying 1 resource [INFO] Copying 6 resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ config --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ config --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Using 'UTF-8' encoding to copy filtered properties files. [INFO] Copying 1 resource [INFO] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ config --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ config --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] [INFO] Results: [INFO] [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] [INFO] --- jacoco-maven-plugin:0.8.7:report (report) @ config --- [INFO] Loading execution data file /Users/neo/workspace/microservice/config/target/jacoco.exec [INFO] Analyzed bundle 'config' with 1 classes [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 3.335 s [INFO] Finished at: 2021-10-22T15:52:36+08:00 [INFO] ------------------------------------------------------------------------
配置持续集成流水线 .gitlab-ci.yml 文件
cache: untracked: true stages: - build - test - deploy test-job: stage: test variables: GIT_STRATEGY: none only: - tags - development - testing script: - mvn test after_script: - lrsync 'zito-admin/target/site/*' www@report.netkiller.cn:/opt/netkiller.cn/report.netkiller.cn - wechat -t 1 代码覆盖率报告 http://report.netkiller.cn/jacoco/index.html
Maven 项目 parent 文件
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>cn.netkiller</groupId> <artifactId>test</artifactId> <version>0.0.1${project.branch}${project.phase}</version> <packaging>pom</packaging> <name>test</name> <url>http://maven.apache.org</url> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <maven.compiler.source>18</maven.compiler.source> <maven.compiler.target>${maven.compiler.source}</maven.compiler.target> <selenium.version>4.8.1</selenium.version> <project.phase>-SNAPSHOT</project.phase> <project.branch></project.branch> </properties> <distributionManagement> <repository> <id>repository</id> <name>Release repository</name> <url>https://maven.netkiller.cn/repository/repository/</url> </repository> <snapshotRepository> <id>snapshots</id> <name>Snapshots repository</name> <url>https://maven.netkiller.cn/repository/snapshots/</url> </snapshotRepository> </distributionManagement> <modules> <module>common</module> <module>selenium</module> </modules> </project>
模块
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <packaging>jar</packaging> <parent> <groupId>cn.netkiller</groupId> <artifactId>test</artifactId> <version>0.0.1-SNAPSHOT</version> </parent> <artifactId>common</artifactId> <!-- <version>${project.parent.version}${project.branch}</version>--> <properties> <maven.compiler.source>18</maven.compiler.source> <maven.compiler.target>18</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> </properties> </project>
构建快照版本 common-0.0.1-SNAPSHOT.jar
neo@macbook-pro-neo ~/w/test> mvn clean package deploy [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] test [pom] [INFO] common [jar] [INFO] selenium [jar] [INFO] [INFO] -------------------------< cn.netkiller:test >-------------------------- [INFO] Building test 0.0.1-SNAPSHOT [1/3] [INFO] --------------------------------[ pom ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ test --- [INFO] [INFO] ------------------------< cn.netkiller:common >------------------------- [INFO] Building common 0.0.1-SNAPSHOT [2/3] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ common --- [INFO] Deleting /Users/neo/workspace/test/common/target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ common --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 0 resource [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ common --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /Users/neo/workspace/test/common/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ common --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/neo/workspace/test/common/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ common --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ common --- [INFO] No tests to run. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ common --- [INFO] Building jar: /Users/neo/workspace/test/common/target/common-0.0.1-SNAPSHOT.jar [INFO] [INFO] -----------------------< cn.netkiller:selenium >------------------------ [INFO] Building selenium 0.0.1-SNAPSHOT [3/3] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ selenium --- [INFO] Deleting /Users/neo/workspace/test/selenium/target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ selenium --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/neo/workspace/test/selenium/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ selenium --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 8 source files to /Users/neo/workspace/test/selenium/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ selenium --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/neo/workspace/test/selenium/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ selenium --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 2 source files to /Users/neo/workspace/test/selenium/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ selenium --- [INFO] [INFO] --- maven-jar-plugin:3.1.1:jar (default-jar) @ selenium --- [INFO] Building jar: /Users/neo/workspace/test/selenium/target/selenium-0.0.1-SNAPSHOT.jar [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for test 0.0.1-SNAPSHOT: [INFO] [INFO] test ............................................... SUCCESS [ 0.125 s] [INFO] common ............................................. SUCCESS [ 1.422 s] [INFO] selenium ........................................... SUCCESS [ 1.457 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 3.158 s [INFO] Finished at: 2023-04-05T18:32:45+08:00 [INFO] ------------------------------------------------------------------------
构建开发环境快照版本 common-0.0.1-dev-SNAPSHOT.jar
neo@macbook-pro-neo ~/w/t/common> mvn clean package -Dproject.branch=-dev [INFO] Scanning for projects... [INFO] [INFO] ------------------------< cn.netkiller:common >------------------------- [INFO] Building common 0.0.1-dev-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ common --- [INFO] Deleting /Users/neo/workspace/test/common/target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ common --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 0 resource [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ common --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /Users/neo/workspace/test/common/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ common --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/neo/workspace/test/common/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ common --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ common --- [INFO] No tests to run. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ common --- [INFO] Building jar: /Users/neo/workspace/test/common/target/common-0.0.1-dev-SNAPSHOT.jar [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1.384 s [INFO] Finished at: 2023-04-05T18:34:50+08:00 [INFO] ------------------------------------------------------------------------
构建RELEASE版本 common-0.0.1.jar
neo@macbook-pro-neo ~/w/t/common> mvn clean package -Dproject.phase= [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for test 0.0.1: [INFO] [INFO] common ............................................. SUCCESS [ 6.828 s] [INFO] selenium ........................................... SUCCESS [ 6.351 s] [INFO] test ............................................... SUCCESS [ 3.274 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 16.631 s [INFO] Finished at: 2023-04-05T16:16:28+08:00 [INFO] ------------------------------------------------------------------------
构建带有 “-RELEASE” 后缀的版本 common-0.0.1-RELEASE.jar
neo@macbook-pro-neo ~/w/t/common> mvn clean package -Dproject.phase=-RELEASE [INFO] Scanning for projects... [INFO] [INFO] ------------------------< cn.netkiller:common >------------------------- [INFO] Building common 0.0.1-RELEASE [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ common --- [INFO] Deleting /Users/neo/workspace/test/common/target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ common --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 0 resource [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ common --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /Users/neo/workspace/test/common/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ common --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/neo/workspace/test/common/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ common --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ common --- [INFO] No tests to run. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ common --- [INFO] Building jar: /Users/neo/workspace/test/common/target/common-0.0.1-RELEASE.jar [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1.656 s [INFO] Finished at: 2023-04-05T18:36:30+08:00 [INFO] ------------------------------------------------------------------------
首先说说什么是数据库结构,什么事版本控制。
数据库结构是指数据库表结构,数据库定义语言导出的DDL语句。主要由CREATE TABLE, DROP TABLE等等构成。
再来说说什么事版本控制,如果你从事开发工作应该会很容易理解,版本控制就是记录每一次变化,可以随时查看历史记录,并可回撤到指定版本。
软件开发过程中需要常常对数据库结构作调整,这是无法避免的,甚至很多想起启动后,需求还不明确,开发人员只能按照所理解需求创建表。需求往往会发生变化,一旦变化,代码需要修改,表结构也避免不了。 我们常常刚改好数据库结构,需求部门有发来通知,不用修改了,维持原有设计。甚至是过了几周再次回撤。
所以我们要将数据库结构的变化进行版本控制,通常的做法是DBA人工管理,但我觉完全可以自动化的工作,没有必要浪费人力资源,且自动化不会犯错更稳定,仅仅需要人工定期查看工作状态即可。
首先下载脚本 https://github.com/oscm/shell/blob/master/backup/backup.mysql.struct.sh
wget https://raw.githubusercontent.com/oscm/shell/master/backup/backup.mysql.struct.sh mv backup.mysql.struct.sh /usr/local/bin chmod +x /usr/local/bin/backup.mysql.struct
创建备份用户
CREATE USER 'backup'@'localhost' IDENTIFIED BY 'chen'; GRANT SELECT, LOCK TABLES ON *.* TO 'backup'@'localhost'; FLUSH PRIVILEGES; SHOW GRANTS FOR 'backup'@'localhost';
配置脚本
BACKUP_HOST="localhost" 数据库主机 BACKUP_USER="backup" 备份用户 BACKUP_PASS="chen" 备份密码 BACKUP_DBNAME="neo netkiller" 版本控制那些数据库,多个数据库使用空格分隔 BACKUP_DIR=~/backup 数据库结构放在那里 GIT=git@gitlab.netkiller.cn:netkiller.cn/db.netkiller.cn.git
初始化仓库
# /usr/local/bin/backup.mysql.struct init Initialized empty Git repository in /www/database/struct/.git/
# /usr/local/bin/backup.mysql.struct Usage: /usr/local/bin/backup.mysql.struct {init|start|stop|status|restart}
开始脚本
# /usr/local/bin/backup.mysql.struct start
查看状态
# /usr/local/bin/backup.mysql.struct status 9644 pts/1 S 0:00 /bin/bash /usr/local/bin/backup.mysql.struct start
停止脚本
# /usr/local/bin/backup.mysql.struct status
通过 git log 命令查看历史版本
# cd /www/database/struct/ # git status # On branch master nothing to commit (working directory clean) # git log commit d38fc624c21cad0e2f55f0228bff0c1be981827c Author: root <root@slave.example.com> Date: Wed Dec 17 12:33:55 2014 +0800 2014-12-17.04:33:55
这里仅仅将数据库结构版本控制,关于版本控制软件更多细节,延伸阅读 《Netkiller Version 手札》
背景,微服务开发中,常常会用到注册中心和配置中心,目前国内比较流行使用 Nacos,Nacos 的配置是保存在数据库中的,不方便维护。代码的变更与配置的版本是没有强关联的,尤其是并行开发中,我们需要多套开发和测试环境时,配置管理的工作会带来挑战,如果配置中心管理不善,就会出现各种问题,例如相互覆盖,版本不一致等等。
虽然 Nacos 也有历史记录,需要为每个人创建一个帐号,才能实现在版本管理中看到谁在什么时间修改了配置。但是即使这样,配置的版本与代码是没有关联的,我们不清楚不同的代码版本需要那些必要的配置项。
我更趋向让配置与代码版本强关联,实现配置的版本与代码的版本一致,让配置管理与持续集成和部署融合,配置变更由持续部署流水线自行完成,而不是让管理员去 Nacos 后台手工处理。于是变产生了下面的工具。
安装 netkiller-devops 工具
# pip 命令安装: root@netkiller ~# pip install netkiller-devops # 如果此前已经安装,可以使用下面命令更新: root@netkiller ~# pip install netkiller-devops --upgrade # Docker 方式安装: root@netkiller ~# docker pull netkiller/netkiller-devops:latest root@netkiller ~# docker run --rm -it --name=netkiller --entrypoint=sh netkiller-devops:latest /srv # nacos Usage: nacos [options] message Options: -h, --help show this help message and exit -s localhost:8848, --server-addr=localhost:8848 localhost:8848 -u USERNAME, --username=USERNAME -p PASSWORD, --password=PASSWORD -n public, --namespace=public -d DATAID, --dataId=DATAID -g DEFAULT_GROUP, --group=DEFAULT_GROUP Config: --push --show --save -f FILE, --file=FILE .yaml file -t yaml, --type=yaml yaml|text|json|xml|Properties --delete Homepage: https://www.netkiller.cn Author: Neo <netkiller@msn.com> Help: https://github.com/netkiller/devops/blob/master/doc/
帮助信息
root@netkiller ~# nacos Usage: nacos [options] Options: -h, --help show this help message and exit -s http://localhost:8848, --server-addr=http://localhost:8848 Nacos 服务器地址 -u USERNAME, --username=USERNAME 用户名 -p PASSWORD, --password=PASSWORD 密码 -n public, --namespace=public 命名空间 -d DATAID, --dataId=DATAID 配置ID -g DEFAULT_GROUP, --group=DEFAULT_GROUP 分组 配置管理: --push 发布配置 --show 查看配置 --save 保存配置 -f FILE, --file=FILE .yaml 文件 -t yaml, --type=yaml yaml|text|json|xml|Properties --delete 删除配置 Homepage: https://www.netkiller.cn Author: Neo <netkiller@msn.com> Help: https://github.com/netkiller/devops/blob/master/doc/
发布 Nacos 配置
准备配置文件 root@netkiller ~# cat test.yaml server: servlet: context-path: /netkiller spring: servlet: multipart: max-file-size: 10MB max-request-size: 10MB 发布 test.yaml 配置文件到 Nacos 配置中心 root@netkiller ~# nacos -s http://nacos.netkiller.cn -u nacos -p nacos -n test -d test --push -f test.yaml
查看 Nacos 配置
root@netkiller ~# nacos -s http://nacos.netkiller.cn -u nacos -p nacos -n test -d test --show server: servlet: context-path: /netkiller spring: servlet: multipart: max-file-size: 10MB max-request-size: 10MB
保存 Nacos 配置
root@netkiller ~# nacos -s http://nacos.netkiller.cn -u nacos -p nacos -n test -d test --save -f save.yaml root@netkiller ~# cat save.yaml server: servlet: context-path: /netkiller spring: servlet: multipart: max-file-size: 10MB max-request-size: 10MB
删除 Nacos 配置
root@netkiller ~# nacos -s http://nacos.netkiller.cn -u nacos -p nacos -n test -d test —delete 返回 None 表示配置不存在 root@netkiller ~# nacos -s http://nacos.netkiller.cn -u nacos -p nacos -n test -d test --show None
方案一,使用 Shell 执行器,在 gitlab runner 节点上安装 netkiller-devops 包
stages: - deploy deploy-job: stage: deploy variables: NACOS: http://nacos.netkiller.cn before_script: - nacos -s $NACOS -u nacos -p nacos -n $CI_COMMIT_BRANCH -d $CI_PROJECT_NAME --show script: - nacos -s $NACOS -u nacos -p nacos -n $CI_COMMIT_BRANCH -d $CI_PROJECT_NAME --push -f nacos/$CI_COMMIT_BRANCH.yaml after_script: - nacos -s $NACOS -u nacos -p nacos -n $CI_COMMIT_BRANCH -d $CI_PROJECT_NAME --show only: - dev - test - master tags: - shell
方案二、使用 Kubernetes 或者 Docker 执行器
deploy-job-kubernetes: stage: deploy image: netkiller/netkiller-devops:latest variables: NACOS: http://nacos.netkiller.cn:8848 before_script: - cat nacos/$CI_COMMIT_BRANCH.yaml script: - nacos -s $NACOS -u nacos -p nacos -n $CI_COMMIT_BRANCH -d $CI_PROJECT_NAME --push -f nacos/$CI_COMMIT_BRANCH.yaml after_script: - nacos -s $NACOS -u nacos -p nacos -n $CI_COMMIT_BRANCH -d $CI_PROJECT_NAME --show only: - dev - test - master tags: - kubernetes