Docker部署分布式MinIO存储集群

准备3个节点，每个节点创建4个挂载点，由于集群模式下不能使用根磁盘，这里使用docker卷作为挂载点。
3节点的集群，故障一个节点时不影响对集群的读写操作。

开始安装

所有节点安装docker和docker-compose

1
curl https://www.putianhui.cn/package/script/install_docker.sh

创建docker-compose和data挂载目录

1
mkdir -p /data/minio/{minio-dockerfile,data01,data02,data03,data04}

所有节点配置主机名解析

1
2
3
4
5
cat >> /etc/hosts << EOF
192.168.99.8 minio-1
192.168.99.9 minio-2
192.168.99.10 minio-3
EOF

配置时间同步，关闭防火墙和selinux。

1
ntpdate ntp1.aliyun.com

部署minio集群，3个节点每个节点挂载4个目录

minio-1节点执行

1
2
3
4
5
6
7
8
9
10
11
12
13
docker run -d --name minio01 \
  --restart=always  --net=host \
  -e "MINIO_ROOT_USER=admin" \
  -e "MINIO_ROOT_PASSWORD=admin123" \
  -v /etc/localtime:/etc/localtime \
  -v /data/minio/data01:/data1 \
  -v /data/minio/data02:/data2 \
  -v /data/minio/data03:/data3 \
  -v /data/minio/data04:/data4 \
  minio/minio:RELEASE.2021-11-24T23-19-33Z.fips server \
  --address "192.168.99.8:49000" \
  --console-address ":49001" \
  http://minio-{1...3}/data{1...4}

minio-2节点执行

1
2
3
4
5
6
7
8
9
10
11
12
13
docker run -d --name minio02 \
  --restart=always  --net=host \
  -e MINIO_ROOT_USER=admin \
  -e MINIO_ROOT_PASSWORD=admin123 \
  -v /etc/localtime:/etc/localtime \
  -v /data/minio/data01:/data1 \
  -v /data/minio/data02:/data2 \
  -v /data/minio/data03:/data3 \
  -v /data/minio/data04:/data4 \
  minio/minio:RELEASE.2021-11-24T23-19-33Z.fips server \
  --address "192.168.99.9:49000" \
  --console-address ":49001" \
  http://minio-{1...3}/data{1...4}

minio-3节点执行

1
2
3
4
5
6
7
8
9
10
11
12
13
docker run -d --name minio03 \
  --restart=always  --net=host \
  -e MINIO_ROOT_USER=admin \
  -e MINIO_ROOT_PASSWORD=admin123 \
  -v /etc/localtime:/etc/localtime \
  -v /data/minio/data01:/data1 \
  -v /data/minio/data02:/data2 \
  -v /data/minio/data03:/data3 \
  -v /data/minio/data04:/data4 \
  minio/minio:RELEASE.2021-11-24T23-19-33Z.fips server \
  --address "192.168.99.10:49000" \
  --console-address ":49001" \
  http://minio-{1...3}/data{1...4}

说明：docker部署集群模式时必须指定–net=host参数，使用主机网络，使用端口映射无法创建集群。

查看容器运行状态

1
2
[root@survey-cdh01 /]# docker ps |grep minio
d89139771f1a   minio/minio:RELEASE.2021-11-24T23-19-33Z.fips   "/usr/bin/docker-ent…"   4 minutes ago   Up 4 minutes                            minio01

客户端查看minio状态，3个节点及每个节点上的Drivers全部在线：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ mc config host add minio http://192.168.99.8:49000 admin admin123
$ mc admin info minio
●  minio-1:49000
   Uptime: 14 minutes
   Version: 2021-11-24T23:19:33Z
   Network: 3/3 OK
   Drives: 4/4 OK

●  minio-2:49000
   Uptime: 12 minutes
   Version: 2021-11-24T23:19:33Z
   Network: 3/3 OK
   Drives: 4/4 OK

●  minio-3:49000
   Uptime: 8 minutes
   Version: 2021-11-24T23:19:33Z
   Network: 3/3 OK
   Drives: 4/4 OK

53 MiB Used, 1 Bucket, 4 Objects
12 drives online, 0 drives offline

使用nginx配置minio和minio-console负载均衡器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
upstream minio {
    server minio-1:49000;
    server minio-2:49000;
    server minio-3:49000;
}

upstream console {
    ip_hash;
    server minio-1:49001;
    server minio-2:49001;
    server minio-3:49001;
}

server {
    listen       80;
    server_name  minio.putianhui.cn;

    # To allow special characters in headers
    ignore_invalid_headers off;
    # Allow any size file to be uploaded.
    # Set to a value such as 1000m; to restrict file size to a specific value
    client_max_body_size 0;
    # To disable buffering
    proxy_buffering off;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 300;
        # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;

        proxy_pass http://minio;
    }
}

server {
    listen       80;
    server_name  minio-console.putianhui.cn;

    # To allow special characters in headers
    ignore_invalid_headers off;
    # Allow any size file to be uploaded.
    # Set to a value such as 1000m; to restrict file size to a specific value
    client_max_body_size 0;
    # To disable buffering
    proxy_buffering off;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-NginX-Proxy true;

        # This is necessary to pass the correct IP to be hashed
        real_ip_header X-Real-IP;

        proxy_connect_timeout 300;
            
        # To support websocket
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
            
        chunked_transfer_encoding off;

        proxy_pass http://console;
    }
}

浏览器访问http://minio-console.putianhui.cn到console页面，账号密码为运行docker时指定的admin/admin123