环境概述

  1. 本地环境为由Beijing、shanghai、guangzhou三台Linux主机构成的数据备份源；
  2. 远端备份目的为本地自建S3存储和紫光云存储；
  3. 使用rclone将本地多主机文件备份到远端存储；
  4. 具体工具有：rclone、minio和sftp。

主机地址

主机名 角色 IP地址
Rclone 同步工具 192.168.174.150
Minio 开源S3服务器 192.168.174.160
UNicloud 公网S3服务器 *
Beijing 备份站点 192.168.174.141
Shanghai 备份站点 192.168.174.142
Guangzhou 备份站点 192.168.174.143

Minio的建立

Mnino是一个轻巧、快速的开源S3服务器，其官方主页为：minio https://min.io/

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
# 下载二进制执行文件
wget https://dl.minio.io/server/minio/release/linux-amd64/minio
# 添加可执行权限
chmod +x minio
# 复制到指定路径
cp minio /usr/local/bin/
# 建立程序运行账户
useradd -r minio -s /sbin/nologin
# 对程序属主进行配置
chown minio:minio /usr/local/bin/minio
# 配置程序启动参数
touch /etc/default/minio
cat>/etc/default/minio<<EOF
MINIO_VOLUMES="/usr/local/share/minio/"
MINIO_OPTS="-C /etc/minio --address 192.168.174.160:9000"
MINIO_ACCESS_KEY="minio"
MINIO_SECRET_KEY="miniostorage"
EOF
# 配置systemctl管理程序
touch minio.service
cat >> minio.service <<EOF
[Unit]
Description=MinIO
Documentation=https://docs.min.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio

[Service]
WorkingDirectory=/usr/local/

User=minio
Group=minio

EnvironmentFile=/etc/default/minio
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi"

ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES

# Let systemd restart this service always
Restart=always

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no

[Install]
WantedBy=multi-user.target
# Built for ${project.name}-${project.version} (${project.name})
EOF

# 复制配置文件到指定位置
cp minio.service /etc/systemd/system
systemctl daemon-reload
systemctl enable minio.service --now

# 配置防火墙
firewall-cmd --permanent --add-service={ssh,http,https}
firewall-cmd --permanent --add-port=9000/tcp
firewall-cmd --reload

建立桶存储并上传文件

1
2
3
4
[root@minio ~]# ls /usr/local/share/minio/
test
[root@minio ~]# ls /usr/local/share/minio/test/
This is MinIO.mp4

接下来，我们可以使用官方客户端mc来访问我们建立的桶：

1
2
3
4
5
6
7
wget http://dl.minio.org.cn/client/mc/release/linux-amd64/mc
chmod +x mc
./mc config host add minio http://192.168.174.160:9000 minio miniostroage
[root@minio ~]# ./mc ls minio
[2020-06-01 23:50:56 CST]      0B test/
[root@minio ~]# ./mc ls minio/test/
[2020-06-01 23:51:01 CST]  8.7MiB This is MinIO.mp4

Rclone部署

rclone的部署很简单，从官方网站下载安装rclone包即可。

1
2
wget https://downloads.rclone.org/v1.52.0/rclone-v1.52.0-linux-amd64.rpm
yum localinstall -y rclone-v1.52.0-linux-amd64.rpm

然后，使用“rclone config”进行配置即可。配置完成的配置文件如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[minio]
type = s3
provider = Minio
env_auth = false
access_key_id = minio
secret_access_key = miniostorage
region = us-east-1
endpoint = http://192.168.174.160:9000

[unicloud]
type = s3
provider = Other
env_auth = false
access_key_id = __
secret_access_key = ______
endpoint = ______.unicloudsrv.com
acl = private

接下来，我们进行测试：

1
2
3
4
5
6
7
8
[root@rclone ~]# rclone lsd minio:
          -1 2020-06-01 23:50:56        -1 test
[root@rclone ~]# rclone ls minio:test
  9117336 This is MinIO.mp4
[root@rclone ~]# rclone copy rclone-v1.52.0-linux-amd64.rpm minio:test
[root@rclone ~]# rclone ls minio:test
  9117336 This is MinIO.mp4
 13825921 rclone-v1.52.0-linux-amd64.rpm

备份其他上传

  1. 各个服务器需要开启时间同步服务并配置rclone主机同各个备份主机的ssh免密登录；
  2. 使用rclone config 建立sftp类型的配置文件；
  3. 测试
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    [root@beijing opt]# touch beijing/192.168.174.141
    [root@beijing opt]# hostname >> ./beijing/192.168.174.141 

    [root@rclone ~]# rclone copy beijing:/opt/beijing minio:/test
    [root@rclone ~]# rclone ls minio:/test/
            9 192.168.174.141
      9117336 This is MinIO.mp4
     13825921 rclone-v1.52.0-linux-amd64.rpm
    # 在公有云新建桶
    [root@rclone ~]# rclone mkdir unicloud:sujx
    [root@rclone ~]# rclone lsd unicloud:
              -1 2020-05-27 14:16:27        -1 backup-ucd-public-prod
              -1 2020-05-27 14:16:22        -1 backup-ucd-public-test
              -1 2020-05-20 09:20:30        -1 crmbak
              -1 2020-05-25 16:36:46        -1 fs-pcf
              -1 2020-05-25 16:36:35        -1 fs-pcf-temp
              -1 2020-06-01 19:28:14        -1 sujx
    # 在本地S3和远端S3 同步
    [root@rclone ~]# rclone sync minio:test unicloud:sujx
    [root@rclone ~]# rclone ls unicloud:sujx
            9 192.168.174.141
      9117336 This is MinIO.mp4
     13825921 rclone-v1.52.0-linux-amd64.rpm