目录

Harbor对接Ceph S3安装及使用手册

Harbor对接Ceph S3安装及使用手册


1 前言

容器镜像仓库是一种存储库(或存储库集合),用于存储Kubernetes、DevOps和基于容器的应用开发的容器镜像。

容器化部署时候,容器平台需要从一个位置使用容器镜像时,保存和访问创建的镜像,这正是容器镜像仓库的作用。

镜像仓库主要用于存储容器镜像,可通过上传(推送)和下载(提取)过程共享。

容器镜像仓库有两种类型包括公有镜像仓库和私有镜像仓库。其中公共镜像仓库部署方便,使用简单,适用于希望尽快启动并运行镜像仓库的个人或小团队。

小型的初创企业可利用标准和开源镜像从头开始,逐渐发展。然而,在发展过程中,可能会出现补丁、隐私和访问控制等安全问题。

通过私有镜像仓库,您可将安全与隐私整合到远程托管或内部托管的企业容器镜像存储中。

公司可选择创建并部署自己的容器镜像仓库,也可选择商业支持的私有镜像仓库服务。目前成熟的私有镜像仓库包括VMware团队开发的开源Harbor产品和Redhat的Quay产品。

这些私有镜像仓库通常具有高级安全功能和技术支持。例如harbor目前支持Aqua Trivy镜像扫描,以及chartmuseum私有helm仓库等多项功能。

之前在阐述安装Openshift集群过程中,都需要临时部署镜像仓库例如Docker Registry,以便在点火安装时候能够像本文开头那样说明的,从一个位置使用容器镜像。但临时部署的Docker Registry,功能简单,无法使用企业内部生产环境。

本文就VMware团队开发的开源Harbor产品的安装、部署及实际使用的一些经验进行说明,最后会实战将Openshift4.8.0相关镜像推送到harbor镜像仓库上。

进一步的操作可参考官网【2.3版本】

https://goharbor.io/docs/2.3.0/install-config/

1.1 系统环境

操作系统版本:Red Hat Enterprise Linux release 8.4 (Ootpa)

Harbor 版本:2.3.0

CPU:2 CPU

内存:8GB

虚拟化平台:VMware VSphere 7.0.2 + vSAN 7.0.2

Ceph:Pacific版本

harbor主机名(访问harbor的最终域名):registry.cj.io,注意相关的DNS解析需要提前做好,方便后续客户端进行访问。

1.2 安装总体步骤

Harbor的部署本身都是基于容器化部署。一种是在操作系统层上采用Docker及Docker Compose直接运行,本文按照此种方式进行阐述。

这种方式的问题是在高可用方面不具备能力。因此如果寻求高可用方式,可以采用Helm方式在类似K8S的容器化平台上进行部署。

具体安装步骤如下:

1、下载介质,一般都选择离线方式介质

2、编辑harbor.yml配置文件

3、利用prepare脚本进行安装前的环境检查以及最终安装前的配置准备。prepare文件本身就是一个shell脚本。

4、利用install脚本进行harbor最终的安装,包括harbor本身镜像及所依赖数据库(redis、postgreSQL)镜像的启动。

最后本文将针对性的讲解一些运维操作。

1.3 其他注意事项

1、需要关闭系统SELINUX功能

2、建议打开https访问

3、打开防火墙的80、443端口

4、Harbor支持外部数据库,但只支持PostgreSQL

5、Harbor的镜像保存方式支持swift、S3以及本地存储方式,本文采用Ceph S3的对象存储方式对接harbor

2 Harbor安装

2.1 准备前工作

安装前需要检查硬件条件

官方建议的硬件配置如下.

资源 最小 建议
CPU 2 CPU 4 CPU
Mem 4 GB 8 GB
Disk 40 GB 160 GB

另外

安装Harbor需要

1、Docker,需要17.06-ce或者更高版本。

2、Docker Compose需要1.18或者更高版本。

2.2 下载介质

访问github,下载离线安装介质【如果需要科学上网请自行安排】

https://github.com/goharbor/harbor/releases

https://typorabyethancheung911.oss-cn-shanghai.aliyuncs.com/typora/image-20210627154647222.png

建立单独的文件夹,开始下载压缩包。

1
2
3
4
5
6
$ mkdir harbor
$ cd harbor/
$ wget https://github.com/goharbor/harbor/releases/download/v2.3.0/harbor-offline-installer-v2.3.0.tgz
--2021-06-27 16:11:48--  https://github.com/goharbor/harbor/releases/download/v2.3.0/harbor-offline-installer-v2.3.0.tgz
Resolving github.com (github.com)... 13.250.177.223
Connecting to github.com (github.com)|13.250.177.223|:443... connected.

解压缩

1
$ tar xzvf harbor-offline-installer-v2.3.0.tgz --strip-components=1

验证

注意刚解压缩后,是没有common这个目录的。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
$ tree
.
├── common.sh
├── harbor-offline-installer-v2.3.0.tgz
├── harbor.v2.3.0.tar.gz
├── harbor.yml.tmpl
├── input
├── install.sh
├── LICENSE
└── prepare

3 配置

我不打算直接说配置文件的每个字段说明,这个官方有对应参考,建议看原版英文。

这里从空配置文件逐个执行,通过报错方式来理解配置文件的用意。

3.1 配置文件模板

安装文件解压缩后,进入目录,在没有配置harbor.yml配置文件下,我们运行prepare脚本看看会如何?

1
2
3
$ ./prepare
prepare base dir is set to /root/harbor
no config file: /root/harbor/harbor.yml

很明显,脚本会提示当前目录下找不到配置文件。

事实上,默认harbor已经准备了一份模板,不过名字叫做harbor.yml.tmpl

我们直接从该模板文件拷贝一份,重命名为harbor.yml,但是我们不对里面内容进行编辑。

3.2 编辑访问域名

我们再次执行prepare脚本

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
$ ./prepare
prepare base dir is set to /root/harbor
Unable to find image 'goharbor/prepare:v2.3.0' locally
v2.3.0: Pulling from goharbor/prepare
7ec05e6a2d8c: Pull complete
c1449fd9a967: Pull complete
ddf99bdd9530: Pull complete
1c616d9ac013: Pull complete
ccbb90dbb863: Pull complete
05d9a118336e: Pull complete
58e7bf715f0e: Pull complete
da5d6b00cd46: Pull complete
Digest: sha256:50c54dddd73cb670bbd2382eb4eb94c914143a4f0f1907677ffe7297aee737cd
Status: Downloaded newer image for goharbor/prepare:v2.3.0
Error happened in config validation...
ERROR:root:Please specify hostname

系统再次提示,因为配置文件中的hostname未编辑报错。

我们按照之前规划【前言1.1 阐述】,我们将harbor.yml中hostname字段从默认的reg.mydomain.com设置为registry.cj.io

3.3 配置服务器证书

我们再次执行prepare脚本

1
2
3
4
$ ./prepare
prepare base dir is set to /root/harbor
Error happened in config validation...
ERROR:root:Error: The protocol is https but attribute ssl_cert is not set

系统再次提示,未发现证书文件。

我们这里开始配置服务器证书,我们采用自签名的服务器证书

具体的证书制作教程可以参考我写的另外一篇文章《CFSSL使用方法》

1、个人博客地址:https://www.ethanzhang.xyz/cfssl%E4%BD%BF%E7%94%A8%E6%96%B9%E6%B3%95/

2、公众号地址:https://mp.weixin.qq.com/s/tV2onhmI5PlQfvPimqQmMA

注意服务器证书的SAN属性为harbor服务器的域名访问地址registry.cj.io和实际IP地址两项。

制作完服务器证书和服务器私钥后,将上述两个文件拷贝至harbor当前工作目录,即/root/harbor/

编辑harbor.yml配置文件,将其中certificateprivate_key字段进行编辑。

1
2
certificate: /root/harbor/harbor.pem
private_key: /root/harbor/harbor-key.pem

3.4 配置S3后端存储

我们先配置如何与外部Ceph S3的对象存储进行对接

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
storage_service:
#   # ca_bundle is the path to the custom root ca certificate, which will be injected into the truststore
#   # of registry's and chart repository's containers.  This is usually needed when the user hosts a internal storage with self signed certificate.
#   ca_bundle:
  s3:
    region: default
    bucket: harbor-ceph
    accesskey: D6BUKT15CQ4U7UR84UMM
    secretkey: j0Xvu7xVLpqn8h5tgOSaO8L1UCkBQhLFdbblLyJs
    regionendpoint: http://172.18.3.11
    secure: false
    multipartcopythresholdsize: "5368709120"

注意这里的multipartcopythresholdsize能够将镜像文件最大值设置为5G,解决docker push较大块镜像时不断重试的问题。

3.5 正确运行prepare脚本

我们再次执行prepare脚本

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
$ ./prepare
prepare base dir is set to /root/harbor
Clearing the configuration file: /config/portal/nginx.conf
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /data/secret/keys/secretkey
Successfully called func: create_root_cert
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir

由此可以,prepare脚本除了检查环境外,另一个很重要的作用是生成后续安装部署的实际配置文件。

/root/harbor/common/config的文件目录架构如下,其中common目录下只有config一个目录。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
$ tree
.
├── core
│   ├── app.conf
│   ├── certificates
│   └── env
├── db
│   └── env
├── jobservice
│   ├── config.yml
│   └── env
├── log
│   ├── logrotate.conf
│   └── rsyslog_docker.conf
├── nginx
│   ├── conf.d
│   └── nginx.conf
├── portal
│   └── nginx.conf
├── registry
│   ├── config.yml
│   ├── passwd
│   └── root.crt
├── registryctl
│   ├── config.yml
│   └── env
└── shared
    └── trust-certificates

至此配置脚本prepare正确地执行完毕。

3.6 配置文件重点内容解析

hostname是域名访问的地址.

https默认为443端口,启用后,http自动重定向到443端口.

certificate及private_key分别是服务器证书及私钥。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
hostname: registry.cj.io

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80

# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /root/harbor/harbor.pem
  private_key: /root/harbor/harbor-key.pem

prometheus监控相关配置

1
2
3
4
metric:
  enabled: false
  port: 9090
  path: /metrics

4 正式安装

4.1 默认安装

配置脚本prepare正确地执行完毕后,开始执行安装脚本install.sh

不带任何参数的install.sh脚本,是不会安装扫描器、Helm仓库及notary功能

整个安装步骤为四个环节

其中第一个环节为检查docker运行时

第二个环节为检查docker-compose运行时

第三个环节为加载harbor本体和依赖镜像文件

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
./install.sh

[Step 0]: checking if docker is installed ...

Note: docker version: 20.10.7

[Step 1]: checking docker-compose is installed ...

Note: docker-compose version: 1.29.2

[Step 2]: loading Harbor images ...
b8cb4f2fe042: Loading layer [==================================================>]  8.112MB/8.112MB
Loaded image: goharbor/nginx-photon:v2.3.0
2b6cbbd060e5: Loading layer [==================================================>]  6.186MB/6.186MB
69fc2a5d7057: Loading layer [==================================================>]  4.096kB/4.096kB
421b6d0db9f6: Loading layer [==================================================>]  3.072kB/3.072kB
bc5645ad7d34: Loading layer [==================================================>]  19.02MB/19.02MB
962c38e6d231: Loading layer [==================================================>]  19.81MB/19.81MB
Loaded image: goharbor/registry-photon:v2.3.0
4d82179a9400: Loading layer [==================================================>]  9.914MB/9.914MB
675e2c32eb5d: Loading layer [==================================================>]  3.584kB/3.584kB
8818aef7131e: Loading layer [==================================================>]   2.56kB/2.56kB
9d5d2ed48330: Loading layer [==================================================>]  55.83MB/55.83MB
475c7500b29b: Loading layer [==================================================>]  5.632kB/5.632kB
10e61cefb27a: Loading layer [==================================================>]   93.7kB/93.7kB
f2e373a19887: Loading layer [==================================================>]  11.78kB/11.78kB
f622f2e58e5a: Loading layer [==================================================>]  56.73MB/56.73MB
d220e8c2ccdb: Loading layer [==================================================>]   2.56kB/2.56kB
Loaded image: goharbor/harbor-core:v2.3.0
fa7dc2e6a798: Loading layer [==================================================>]  6.186MB/6.186MB
0f612ded4ff1: Loading layer [==================================================>]  4.096kB/4.096kB
fbb3f0d75fe7: Loading layer [==================================================>]  19.02MB/19.02MB
11c455c1ad44: Loading layer [==================================================>]  3.072kB/3.072kB
5b5dc65c296c: Loading layer [==================================================>]   25.4MB/25.4MB
073cf43e62b3: Loading layer [==================================================>]   45.2MB/45.2MB
Loaded image: goharbor/harbor-registryctl:v2.3.0
b106b5752942: Loading layer [==================================================>]  1.096MB/1.096MB
a10890f8f147: Loading layer [==================================================>]  5.888MB/5.888MB
b7d39d927c0a: Loading layer [==================================================>]  209.2MB/209.2MB
c7b75a8ac758: Loading layer [==================================================>]  15.05MB/15.05MB
685aea1b7e0e: Loading layer [==================================================>]  4.096kB/4.096kB
29eea9e3830a: Loading layer [==================================================>]  6.144kB/6.144kB
6c89dc4abd54: Loading layer [==================================================>]  3.072kB/3.072kB
8f6ddd91b278: Loading layer [==================================================>]  2.048kB/2.048kB
44885e6b8efc: Loading layer [==================================================>]   2.56kB/2.56kB
d0650c47bd17: Loading layer [==================================================>]   2.56kB/2.56kB
baa9d588c87c: Loading layer [==================================================>]   2.56kB/2.56kB
c0948a512263: Loading layer [==================================================>]  8.704kB/8.704kB
Loaded image: goharbor/harbor-db:v2.3.0
230bb4d21843: Loading layer [==================================================>]  9.914MB/9.914MB
3b267db69816: Loading layer [==================================================>]  17.67MB/17.67MB
48f062b756ef: Loading layer [==================================================>]  4.608kB/4.608kB
83cea239dd18: Loading layer [==================================================>]  18.46MB/18.46MB
Loaded image: goharbor/harbor-exporter:v2.3.0
7ce03d8b76bc: Loading layer [==================================================>]  156.8MB/156.8MB
146ee77daba1: Loading layer [==================================================>]  3.072kB/3.072kB
7980207d1d35: Loading layer [==================================================>]   59.9kB/59.9kB
4599e620911a: Loading layer [==================================================>]  61.95kB/61.95kB
Loaded image: goharbor/redis-photon:v2.3.0
e1e5285ecc15: Loading layer [==================================================>]  6.181MB/6.181MB
69fa33ea8a76: Loading layer [==================================================>]  6.207MB/6.207MB
90ecbcead336: Loading layer [==================================================>]  14.89MB/14.89MB
05a6e541d31d: Loading layer [==================================================>]  27.38MB/27.38MB
14d6723ce8f3: Loading layer [==================================================>]  22.02kB/22.02kB
cfe4608e735e: Loading layer [==================================================>]  14.89MB/14.89MB
Loaded image: goharbor/notary-server-photon:v2.3.0
65081183b9d5: Loading layer [==================================================>]  8.112MB/8.112MB
4f1c9ff2daf9: Loading layer [==================================================>]  11.64MB/11.64MB
15070dd6843f: Loading layer [==================================================>]  1.688MB/1.688MB
Loaded image: goharbor/harbor-portal:v2.3.0
3ff2cc3e192a: Loading layer [==================================================>]    161MB/161MB
b39b810a2c31: Loading layer [==================================================>]  3.584kB/3.584kB
bc5176327384: Loading layer [==================================================>]  3.072kB/3.072kB
d178cb37812e: Loading layer [==================================================>]   2.56kB/2.56kB
a0eb93f025fe: Loading layer [==================================================>]  3.072kB/3.072kB
26e033b26702: Loading layer [==================================================>]  3.584kB/3.584kB
0532e71b3bd4: Loading layer [==================================================>]  19.97kB/19.97kB
Loaded image: goharbor/harbor-log:v2.3.0
0cae3dac3e77: Loading layer [==================================================>]  9.914MB/9.914MB
90bebc66effb: Loading layer [==================================================>]  3.584kB/3.584kB
595eefd6fb57: Loading layer [==================================================>]   2.56kB/2.56kB
11c19159aa0f: Loading layer [==================================================>]  62.49MB/62.49MB
5d8a4f259631: Loading layer [==================================================>]  63.28MB/63.28MB
Loaded image: goharbor/harbor-jobservice:v2.3.0
7f36930441df: Loading layer [==================================================>]  6.186MB/6.186MB
f3d478212fb5: Loading layer [==================================================>]  67.47MB/67.47MB
028a02a35dde: Loading layer [==================================================>]  3.072kB/3.072kB
4dd4f408cedc: Loading layer [==================================================>]  4.096kB/4.096kB
6bf984b97419: Loading layer [==================================================>]  68.26MB/68.26MB
Loaded image: goharbor/chartmuseum-photon:v2.3.0
3b9eb50911fc: Loading layer [==================================================>]  41.95MB/41.95MB
a817012987ff: Loading layer [==================================================>]  4.096kB/4.096kB
31d1db570868: Loading layer [==================================================>]  3.072kB/3.072kB
e6eb84749dcb: Loading layer [==================================================>]  31.52MB/31.52MB
6217368c82fa: Loading layer [==================================================>]  11.39MB/11.39MB
91c725a368fd: Loading layer [==================================================>]   43.7MB/43.7MB
Loaded image: goharbor/trivy-adapter-photon:v2.3.0
Loaded image: goharbor/prepare:v2.3.0
fe023c4074c2: Loading layer [==================================================>]  6.181MB/6.181MB
64f739adb147: Loading layer [==================================================>]  6.207MB/6.207MB
ae8dd74e5ae3: Loading layer [==================================================>]  13.35MB/13.35MB
f35b97e2a785: Loading layer [==================================================>]  27.38MB/27.38MB
0a1f6bad7db8: Loading layer [==================================================>]  22.02kB/22.02kB
7a5ef06c750b: Loading layer [==================================================>]  13.35MB/13.35MB
Loaded image: goharbor/notary-signer-photon:v2.3.0

接下来第四个环节是检查启动环节

第五个环节是准备harbor的配置文件,这步是情况common目录下的config目录下的很多配置文件,其中/config/registry/config.yml很关键。

第六个环节是通过docker-composer启动harbor【在此之前,会生成/compose_location/docker-compose.yml文件】

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
[Step 3]: preparing environment ...

[Step 4]: preparing harbor configs ...
prepare base dir is set to /root/harbor
Clearing the configuration file: /config/portal/nginx.conf
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/core/env
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/registry/passwd
Clearing the configuration file: /config/registry/config.yml.bak
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/db/env
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/jobservice/config.yml
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /data/secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir



[Step 5]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating redis         ... done
Creating harbor-portal ... done
Creating harbor-db     ... done
Creating registryctl   ... done
Creating registry      ... done
Creating harbor-core   ... done
Creating harbor-jobservice ... done
Creating nginx             ... done
✔ ----Harbor has been installed and started successfully.----

至此默认安装结束,harbor可以正常访问。

https://typorabyethancheung911.oss-cn-shanghai.aliyuncs.com/typora/image-20210713132337412.png

4.2 增加扫描器、Helm仓库及notary功能

1
$ ./install.sh --with-notary --with-trivy --with-chartmuseum

要使用 Notary 进行安装,您必须将 Harbor 配置为使用 HTTPS。

Harbor v2.1 及之前的版本内置了 Clair 镜像扫描器,在 v2.2 中,Harbor使用了 Aqua Trivy 作为缺省扫描器。

1
✖ Clair is deprecated please remove it from installation arguments !!!

安装完成后,查看功能

https://typorabyethancheung911.oss-cn-shanghai.aliyuncs.com/typora/image-20210713124827037.png

Helm

https://typorabyethancheung911.oss-cn-shanghai.aliyuncs.com/typora/image-20210713124845410.png

验证

1
cat ca.pem >> /etc/pki/tls/certs/ca-bundle.crt

5 故障处理

在整个安装过程中,不免会遇到一些问题,甚至遇到之前配置与所希望不符合,需要重新安装等场景。

这章节注重针对故障处理和运维进行说明。

5.1 查看当前运行状态

正常情况下,State列是显示Up (healthy)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
$ docker-compose ps
      Name                     Command                  State                                          Ports
------------------------------------------------------------------------------------------------------------------------------------------------
harbor-core         /harbor/entrypoint.sh            Up (healthy)
harbor-db           /docker-entrypoint.sh 96 13      Up (healthy)
harbor-jobservice   /harbor/entrypoint.sh            Up (healthy)
harbor-log          /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp
harbor-portal       nginx -g daemon off;             Up (healthy)
nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:80->8080/tcp,:::80->8080/tcp, 0.0.0.0:443->8443/tcp,:::443->8443/tcp
redis               redis-server /etc/redis.conf     Up (healthy)
registry            /home/harbor/entrypoint.sh       Up (healthy)
registryctl         /home/harbor/start.sh            Up (healthy)

往往有问题情况下,registryregistryctl会显示Restarting等情况。

5.2 停止harbor服务

在某些场景下,需要停止harbor服务或者重启harbor服务

特别注意的是:

在运行docker-compose等系列命令时候,如果系统提示如下

1
2
3
4
docker-compose down -v
ERROR:
Can't find a suitable configuration file in this directory or anyparent. Are you in the right directory?
Supported filenames: docker-compose.yml, docker-compose.yaml, compose.yml, compose.yaml

则说明,当前的工作目录不对,需要在包含docker-compose.yml, docker-compose.yaml, compose.yml, compose.yaml等文件下目录执行。

我们通过执行docker-compose down -v可以停止服务

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
$ docker-compose down -v
Stopping nginx             ... done
Stopping harbor-jobservice ... done
Stopping notary-server     ... done
Stopping harbor-core       ... done
Stopping notary-signer     ... done
Stopping trivy-adapter     ... done
Stopping redis             ... done
Stopping chartmuseum       ... done
Stopping harbor-portal     ... done
Stopping harbor-db         ... done
Stopping registry          ... done
Stopping registryctl       ... done
Stopping harbor-log        ... done
Removing nginx             ... done
Removing harbor-jobservice ... done
Removing notary-server     ... done
Removing harbor-core       ... done
Removing notary-signer     ... done
Removing trivy-adapter     ... done
Removing redis             ... done
Removing chartmuseum       ... done
Removing harbor-portal     ... done
Removing harbor-db         ... done
Removing registry          ... done
Removing registryctl       ... done
Removing harbor-log        ... done
Removing network harbor_harbor
Removing network harbor_harbor-notary
Removing network harbor_harbor-chartmuseum
Removing network harbor_notary-sig

【请注意最后移除的几个网桥内容】

再次恢复服务时候可以执行以下命令。

1
$ docker-compose up -d

5.3 全新重新安装

如果我们要重新安装,需要移除Harbor的数据库和镜像数据(进行干净的重新安装)

1、通过docker-compose down -v停止harbor服务。

2、删除/data目录下内容,这个目录具体值详见harbor.ymldata_volume字段,默认值是/data

1
$ rm -rf /data/

3、删除/common目录下内容,实际是删除common目录下唯一的目录config

1
$ rm -rf common/

4、通过install.sh重启安装服务服务。

5.4 重新修改配置文件

如果需要对全局的配置参数进行修改,请修改harbor.yml文件调整相关字段。

1、通过docker-compose down -v停止harbor服务。

2、务必执行prepare脚本。

3、通过docker-compose up -d再次启动服务。

切勿执行install脚本,切勿删除/data

5.5 持久化数据

默认情况下,镜像数据保存在主机的/data目录中。这个目录具体值详见harbor.ymldata_volume字段,默认值是/data。即使在删除和/或重新创建Harbor的容器时,该数据仍然保持不变。

本文的镜像数据是保存在外部Ceph S3对象存储中。

可以在harbor.yml中编辑data_volume字段来更改此目录。

Harbor使用 rsyslog收集每个容器的日志. 默认情况下,这些日志文件存储在目录中 /var/log/harbor/。

PostgreSQL数据库目录如下

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$ tree database/pg13 -L 1
database/pg13
├── base
├── global
├── pg_commit_ts
├── pg_dynshmem
├── pg_hba.conf
├── pg_ident.conf
├── pg_logical
├── pg_multixact
├── pg_notify
├── pg_replslot
├── pg_serial
├── pg_snapshots
├── pg_stat
├── pg_stat_tmp
├── pg_subtrans
├── pg_tblspc
├── pg_twophase
├── PG_VERSION
├── pg_wal
├── pg_xact
├── postgresql.auto.conf
├── postgresql.conf
├── postmaster.opts
└── postmaster.pid

整个/data目录结构如下,开启了trivy扫描器功能。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
$ tree -L 1
.
├── ca_download
├── chart_storage
├── database
├── job_logs
├── redis
├── registry
├── secret
└── trivy-adapter

5.6 网络冲突解决

harbor安装完之后docker0和br网桥可能会占用172.17或者18、19之类的网段,如果和公司内部地址有冲突就需要自定义进行修改了。

可以先行停止harbor服务,然后修改docker的默认地址。

5.7 监控

2.2 版本开始支持Prometheus集成

首先在安装前,将相关注释取消,

另外切记编辑enabled: true,默认是false

1
2
3
4
metric:
  enabled: true
  port: 9090
  path: /metrics

验证

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
$ ss -tulnp | grep 9090
tcp   LISTEN 0      128          0.0.0.0:9090      0.0.0.0:*    users:(("docker-proxy",pid=2216907,fd=4))
tcp   LISTEN 0      128             [::]:9090         [::]:*    users:(("docker-proxy",pid=2216913,fd=4))


$ curl http://127.0.0.1:9090/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0.001800412
go_gc_duration_seconds{quantile="0.25"} 0.001800412
go_gc_duration_seconds{quantile="0.5"} 0.001800412
go_gc_duration_seconds{quantile="0.75"} 0.001800412
go_gc_duration_seconds{quantile="1"} 0.001800412
go_gc_duration_seconds_sum 0.001800412
go_gc_duration_seconds_count 1
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 10
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.15.12"} 1

6 客户端使用

6.1 登录

我们首先在harbor上建立一个单独的账号ocp

https://typorabyethancheung911.oss-cn-shanghai.aliyuncs.com/typora/image-20210713152237453.png

我们登录到

1
2
3
4
$ podman login registry.cj.io
Username: ocp
Password:
Error: error authenticating creds for "registry.cj.io": error pinging docker registry registry.cj.io: Get "https://registry.cj.io/v2/": x509: certificate signed by unknown authority

你会发现系统会报 x509: certificate signed by unknown authority错误。

处理方式是将CA证书添加到操作系统内

对于Ubuntu系列执行下面命令

1
2
$ cp ca.pem /usr/local/share/ca-certificates/ca.pem 
update-ca-certificates

对于红帽(CentOS等)执行下面命令

1
2
$ cp ca.pem /etc/pki/ca-trust/source/anchors/ca.pem
$ update-ca-trust

我们再次登录到registry.cj.io

1
2
3
4
$ podman login registry.cj.io
Username: ocp
Password:
Login Succeeded!

实际是登录到443端口,可通过以下命令验证。

1
2
3
4
$ podman login registry.cj.io:443
Username: ocp
Password:
Login Succeeded!

至此我们可以正常登录

【利用授权文件直接登录】

通过--authfile参数可以一次性生成登录免密文件,后续直接通过这个文件可以直接免密码登录。

1
2
3
4
$ podman login --authfile registry-secret registry.cj.io
Username: ocp
Password:
Login Succeeded!

验证

1
2
3
4
5
6
7
8
$ cat registry-secret
{
	"auths": {
		"registry.cj.io": {
			"auth": "b2NwOkx1Y2lmZXIxMjN+IUA="
		}
	}
}

6.2 推送

我们登录到registry后,开始向镜像仓库推送。

再推送镜像之前,我们登录到ceph集群上,查看对象存储情况。

1
2
$ rados ls -p default.rgw.buckets.data
# 显示空

为了验证真实生产场景下镜像数据量,我们直接利用最新的Openshift4.8.0的镜像推送

【有关离线安装Openshift的教程可以查看之前文档,这里oc命令不详细展开】

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
$ oc adm release mirror -a redhat.json  --from=quay.io/openshift-release-dev/ocp-release:4.8.0-x86_64 --to-dir=ocp-imagee-4.8.0
$ oc image mirror -a registry-secret --dir=ocp-image-4.8.0 file://openshift/release:4.8.0* registry.cj.io/ocp4/openshift4
  
  
  stats: shared=0 unique=279 size=8.633GiB ratio=1.00

phase 0:
  registry.cj.io ocp4/openshift4 blobs=279 mounts=0 manifests=136 shared=0

info: Planning completed in 220ms
info: Mirroring completed in 3m53.42s (39.71MB/s)

我们最终登录到harbor上查看,镜像全部推送完毕。

https://typorabyethancheung911.oss-cn-shanghai.aliyuncs.com/typora/image-20210713223826929.png

推送完毕后我们再次登录ceph集群查看

1
2
$ rados ls -p default.rgw.buckets.data | wc -l
3859

一共有3850个镜像文件。

https://typorabyethancheung911.oss-cn-shanghai.aliyuncs.com/typora/image-20210713224308085.png

查看整体状况

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
$ rados df
POOL_NAME                      USED  OBJECTS  CLONES  COPIES  MISSING_ON_PRIMARY  UNFOUND  DEGRADED   RD_OPS       RD   WR_OPS       WR  USED COMPR  UNDER COMPR
.rgw.root                    72 KiB        6       0      18     0        0         0      104  104 KiB        6    6 KiB         0 B    0 B
default.rgw.buckets.data     26 GiB     3859       0   11577     0        0         0    16595  552 MiB    49104  8.6 GiB         0 B    0 B
default.rgw.buckets.index   730 KiB       11       0      33     0        0         0    98377   96 MiB    17543  9.4 MiB         0 B    0 B
default.rgw.buckets.non-ec   85 KiB        0       0       0     0        0         0     7605  4.9 MiB     4110  1.2 MiB         0 B    0 B
default.rgw.control             0 B        8       0      24     0        0         0        0      0 B        0      0 B        0 B     0 B
default.rgw.log             4.0 MiB      209       0     627     0        0         0  2373225  2.3 GiB  1577553  1.6 MiB         0 B    0 B
default.rgw.meta             48 KiB        5       0      15     0        0         0      713  559 KiB       31   14 KiB         0 B    0 B
device_health_metrics           0 B        0       0       0     0        0         0        0      0 B        0      0 B         0 B    0 B

total_objects    4098
total_used       31 GiB
total_avail      569 GiB
total_space      600 GiB

后记,简单用Trivy扫描了下,ocp4.8镜像漏洞不少啊。

https://typorabyethancheung911.oss-cn-shanghai.aliyuncs.com/typora/image-20210713224835552.png