搜索到
3
篇与
的结果
-
使用RGW(S3 API) 客户端安装与配置一、创建RGW用户#用于s3cmd客户端连接RGW 保存好user1用户的access_key、secret_key信息。 root@ubuntu01:~# radosgw-admin user create --uid="user1" --display-name="user1" { "user_id": "user1", "display_name": "user1", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [], "keys": [ { "user": "user1", "access_key": "OD3YSHLB5J11CGNIJLLO", "secret_key": "vIRD7vv8RXuxiJme41JJmmQn1r3hoS7BKXqrHiJQ" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "default_storage_class": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] } #保存好user1用户的access_key、secret_key信息。二、安装s3cmd客户端root@ceph-client:~# s3cmd --configure --no-check-certificate # 进入配置模式并禁用证书验证 Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options. Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables. Access Key: KKX8OPSFKL3RR3JQRO5D # 用户access key Secret Key: MDjM5KkaKEgWIdrvLr6fSsFmQsZlfuPNOFyBa3IB # 用户Secret Key Default Region [US]: CN # 地区，可选 Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3. S3 Endpoint [s3.amazonaws.com]: ceph-rgw.local.com # RGW域名 Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used if the target S3 system supports dns based buckets. DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: ceph-rgw.local.com/%(bucket) # bucket域名格式 Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: 123.com # 密码 Path to GPG program [/usr/bin/gpg]: # gpg命令路径，用于认证管理，直接回车 When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP, and can only be proxied with Python 2.7 or newer Use HTTPS protocol [Yes]: Yes # 是否使用Https加密 On some networks all internet access must go through a HTTP proxy. Try setting it here if you can t connect to S3 directly HTTP Proxy server name: # 使用代理 # 生成配置内容 New settings: Access Key: KKX8OPSFKL3RR3JQRO5D Secret Key: MDjM5KkaKEgWIdrvLr6fSsFmQsZlfuPNOFyBa3IB Default Region: CN S3 Endpoint: ceph-rgw.local.com DNS-style bucket+hostname:port template for accessing a bucket: ceph-rgw.local.com/%(bucket) Encryption password: 123.com Path to GPG program: /usr/bin/gpg Use HTTPS protocol: True HTTP Proxy server name: HTTP Proxy server port: 0 # 测试连接 Test access with supplied credentials? [Y/n] y Please wait, attempting to list all buckets... Success. Your access key and secret key worked fine :-) Now verifying that encryption works... Success. Encryption and decryption worked fine :-) # 配置文件保存 Save settings? [y/N] y Configuration saved to '/root/.s3cfg's3cmd是一个通过命令行访问ceph RGW实现创建存储桶、上传、下载以及管理数据到对象存储的命令行工具。 root@k8s01:~/helm/traefik/traefik-helm-chart-35.4.0/traefik# s3cmd --configure --no-check-certificate Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options. Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables. Access Key: ^C^C Configuration aborted. Changes were NOT saved. root@k8s01:~/helm/traefik/traefik-helm-chart-35.4.0/traefik# ^C root@k8s01:~/helm/traefik/traefik-helm-chart-35.4.0/traefik# s3cmd --configure --no-check-certificate Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options. Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables. Access Key: OD3YSHLB5J11CGNIJLLO Secret Key: vIRD7vv8RXuxiJme41JJmmQn1r3hoS7BKXqrHiJQ Default Region [US]: CN Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3. S3 Endpoint [s3.amazonaws.com]: ceph-rgw.local.com Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used if the target S3 system supports dns based buckets. DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: ceph-rgw.local.com/%(bucket) Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: 123.com Path to GPG program [/usr/bin/gpg]: When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP, and can only be proxied with Python 2.7 or newer Use HTTPS protocol [Yes]: yes On some networks all internet access must go through a HTTP proxy. Try setting it here if you can't connect to S3 directly HTTP Proxy server name: New settings: Access Key: OD3YSHLB5J11CGNIJLLO Secret Key: vIRD7vv8RXuxiJme41JJmmQn1r3hoS7BKXqrHiJQ Default Region: CN S3 Endpoint: ceph-rgw.local.com DNS-style bucket+hostname:port template for accessing a bucket: ceph-rgw.local.com/%(bucket) Encryption password: 123.com Path to GPG program: /usr/bin/gpg Use HTTPS protocol: True HTTP Proxy server name: HTTP Proxy server port: 0 Test access with supplied credentials? [Y/n] y Please wait, attempting to list all buckets... Success. Your access key and secret key worked fine :-) Now verifying that encryption works... Success. Encryption and decryption worked fine :-) Save settings? [y/N] y Configuration saved to '/root/.s3cfg'查看认证文件root@k8s01:~/helm/traefik/traefik-helm-chart-35.4.0/traefik# cat /root/.s3cfg [default] access_key = OD3YSHLB5J11CGNIJLLO access_token = add_encoding_exts = add_headers = bucket_location = CN ca_certs_file = cache_file = check_ssl_certificate = False check_ssl_hostname = True cloudfront_host = cloudfront.amazonaws.com connection_max_age = 5 connection_pooling = True content_disposition = content_type = default_mime_type = binary/octet-stream delay_updates = False delete_after = False delete_after_fetch = False delete_removed = False dry_run = False enable_multipart = True encoding = UTF-8 encrypt = False expiry_date = expiry_days = expiry_prefix = follow_symlinks = False force = False get_continue = False gpg_command = /usr/bin/gpg gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_passphrase = 123.com guess_mime_type = True host_base = ceph-rgw.local.com host_bucket = ceph-rgw.local.com/%(bucket) human_readable_sizes = False invalidate_default_index_on_cf = False invalidate_default_index_root_on_cf = True invalidate_on_cf = False kms_key = limit = -1 limitrate = 0 list_md5 = False log_target_prefix = long_listing = False max_delete = -1 mime_type = multipart_chunk_size_mb = 15 multipart_copy_chunk_size_mb = 1024 multipart_max_chunks = 10000 preserve_attrs = True progress_meter = True proxy_host = proxy_port = 0 public_url_use_https = False put_continue = False recursive = False recv_chunk = 65536 reduced_redundancy = False requester_pays = False restore_days = 1 restore_priority = Standard secret_key = vIRD7vv8RXuxiJme41JJmmQn1r3hoS7BKXqrHiJQ send_chunk = 65536 server_side_encryption = False signature_v2 = False signurl_use_https = False simpledb_host = sdb.amazonaws.com skip_existing = False socket_timeout = 300 ssl_client_cert_file = ssl_client_key_file = stats = False stop_on_error = False storage_class = throttle_max = 100 upload_id = urlencoding_mode = normal use_http_expect = False use_https = True use_mime_magic = True verbosity = WARNING website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/ website_error = website_index = index.html三、s3cmd常见使用命令 使用s3cmd --help查看帮助信息，详细使用方法见：http://s3tools.orgroot@ceph-client:~# s3cmd --help Usage: s3cmd [options] COMMAND [parameters]四、s3cmd 使用 4.1创建bucket存储空间Bucket是用于存储对象Object的容器，在上传任意类型的Object前，需要先创建Bucket，并指定区域 bucket/object授权参考：https://docs.amazonaws.cn/AmazonS3/latest/userguide/access-policy-language-overview.htmlroot@ceph-client:~# s3cmd mb s3://mybucket --region=default Bucket 's3://mybucket/' created root@ceph-client:~# s3cmd ls --region=default 2024-12-17 10:08 s3://mybucket4.2上传文件root@ceph-client:~# s3cmd put /var/log/syslog s3://mybucket upload: '/var/log/syslog' -> 's3://mybucket/syslog' [1 of 1] 1252189 of 1252189 100% in 2s 532.45 kB/s done4.3查看文件root@ceph-client:~# s3cmd ls s3://mybucket 2024-12-17 10:11 1252189 s3://mybucket/syslog4.4下载文件root@ceph-client:~# s3cmd get s3://mybucket/syslog /root/ download: 's3://mybucket/syslog' -> '/root/syslog' [1 of 1] 1252189 of 1252189 100% in 0s 74.33 MB/s done root@ceph-client:~# ls -lh total 1.2M drwx------ 3 root root 4.0K Oct 28 15:28 snap -rw-r--r-- 1 root root 1.2M Dec 17 10:11 syslog4.5删除文件root@ceph-client:~# s3cmd ls s3://mybucket/ 2024-12-17 10:11 1252189 s3://mybucket/syslog root@ceph-client:~# s3cmd rm s3://mybucket/syslog delete: 's3://mybucket/syslog' root@ceph-client:~# s3cmd ls s3://mybucket/4.6目录上传与下载上传整个目录 root@ceph-client:~# s3cmd put --recursive /etc/apt s3://mybucket/ upload: '/etc/apt/apt.conf.d/01-vendor-ubuntu' -> 's3://mybucket/apt/apt.conf.d/01-vendor-ubuntu' [1 of 24] 92 of 92 100% in 0s 9.30 kB/s done upload: '/etc/apt/apt.conf.d/01autoremove' -> 's3://mybucket/apt/apt.conf.d/01autoremove' [2 of 24] 630 of 630 100% in 0s 85.25 kB/s done upload: '/etc/apt/apt.conf.d/10periodic' -> 's3://mybucket/apt/apt.conf.d/10periodic' [3 of 24] 129 of 129 100% in 0s 19.73 kB/s done …… 查看验证 root@ceph-client:~# s3cmd ls s3://mybucket/apt/ DIR s3://mybucket/apt/apt.conf.d/ DIR s3://mybucket/apt/preferences.d/ DIR s3://mybucket/apt/sources.list.d/ DIR s3://mybucket/apt/trusted.gpg.d/ 2024-12-17 10:17 2883 s3://mybucket/apt/sources.list 2024-12-17 10:17 2743 s3://mybucket/apt/sources.list.curtin.old 2024-12-17 10:17 2717 s3://mybucket/apt/sources.list.save 2024-12-17 10:17 3939 s3://mybucket/apt/trusted.gpg 2024-12-17 10:17 2760 s3://mybucket/apt/trusted.gpg~ 下载整个目录 root@ceph-client:~# mkdir /tmp/apt root@ceph-client:~# s3cmd get --recursive s3://mybucket/apt/ /tmp/apt root@ceph-client:/var/tmp# s3cmd get --recursive s3://mybucket/apt/ /tmp/apt download: 's3://mybucket/apt/apt.conf.d/01-vendor-ubuntu' -> '/tmp/apt/apt.conf.d/01-vendor-ubuntu' [1 of 24] 92 of 92 100% in 0s 47.39 kB/s done download: 's3://mybucket/apt/apt.conf.d/01autoremove' -> '/tmp/apt/apt.conf.d/01autoremove' [2 of 24] 630 of 630 100% in 0s 200.08 kB/s done download: 's3://mybucket/apt/apt.conf.d/10periodic' -> '/tmp/apt/apt.conf.d/10periodic' [3 of 24] 129 of 129 100% in 0s 2.91 kB/s done download: 's3://mybucket/apt/apt.conf.d/15update-stamp' -> '/tmp/apt/apt.conf.d/15update-stamp' [4 of 24] 108 of 108 100% in 0s 35.32 kB/s done download: 's3://mybucket/apt/apt.conf.d/20apt-esm-hook.conf' -> '/tmp/apt/apt.conf.d/20apt-esm-hook.conf' [5 of 24] …… 查看验证 root@ceph-client:/tmp/apt# pwd /tmp/apt root@ceph-client:/tmp/apt# ls apt.conf.d sources.list sources.list.d trusted.gpg trusted.gpg.d preferences.d sources.list.curtin.old sources.list.save trusted.gpg~五、项目实践 5.1创建bucketroot@ceph-client:~# s3cmd mb s3://images --region=default Bucket 's3://images/' created5.2上传文件root@ceph-client:~# s3cmd put 2024_12_16_22_35_54_347-1734359754493.jpg s3://images upload: '2024_12_16_22_35_54_347-1734359754493.jpg' -> 's3://images/2024_12_16_22_35_54_347-1734359754493.jpg' [1 of 1] 502213 of 502213 100% in 0s 17.88 MB/s done5.3授权授权bucket匿名用户只读权限，用于浏览器访问 https://docs.amazonaws.cn/AmazonS3/latest/userguide/example-bucket-policies.html # 编辑授权文件，对images bucket用户授予读权限 root@ceph-client:~# cat > images-bucket-single-policy.json << EOF { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow" , "Principal": "*", "Action": "s3:GetObject", "Resource": [ "arn:aws:s3:::images/*" ] }] } EOF # 执行授权 root@ceph-client:~# s3cmd setpolicy images-bucket-single-policy.json s3://images s3://images/: Policy updated5.4访问测试浏览器访问 vip:端口/bucket/name测试六、配置nginx反向代理 nginx服务器为192.168.10.95，实现访问www.cuiliangblog.cn 的图片内容将请求转向访问ceph 对象存储中的 image 资源root@ceph-client:~# cat /etc/nginx/conf.d/demo.conf server { listen 80; server_name www.cuiliangblog.cn; location / { root html; index index.html index.htm; } location ~* \.(jpg|png)$ { # 以jpg或png结尾的请求转向192.168.10.90（VIP） proxy_pass https://192.168.10.90; } }启动nginxsystemctl start nginx浏览器访问图片资源 -
RadosGW部署 一、创建 radosgw服务root@ubuntu01:~# ceph orch apply rgw default es --placement="count:3" Scheduled rgw.default update...<realm-name>：RadosGW 的域名，可以自定义，例如 default. <zone-name>：RadosGW 的区域名，可以自定义，例如 es. --placement="count:1"：表示 RadosGW 实例的数量。如果你想在多个节点上部署，可以增加 count 的值，或者指定节点列表二、验证radosgw服务状态root@ubuntu01:~# ceph orch ps | grep rgw rgw.default.ubuntu01.lusoxe ubuntu01 *:80 starting - - - - <unknown> <unknown> known> rgw.default.ubuntu02.nvamia ubuntu02 *:80 starting - - - - <unknown> <unknown> known> rgw.default.ubuntu03.rkgoya ubuntu03 *:80 starting - - - - <unknown> <unknown> known> root@ubuntu01:~# ceph -s cluster: id: 5b0e9b94-e6bb-11ef-a18c-274714e73e14 health: HEALTH_WARN clock skew detected on mon.ubuntu03 1 pool(s) do not have an application enabled services: mon: 3 daemons, quorum ubuntu01,ubuntu02,ubuntu03 (age 8h) mgr: ubuntu02.jmaxlt(active, since 8h), standbys: ubuntu01.exrhij mds: 1/1 daemons up, 1 standby osd: 3 osds: 3 up (since 8h), 3 in (since 2w) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 10 pools, 181 pgs objects: 9.37k objects, 2.5 GiB usage: 8.5 GiB used, 111 GiB / 120 GiB avail pgs: 181 active+clean io: client: 214 KiB/s rd, 5.1 KiB/s wr, 247 op/s rd, 141 op/s wr三、验证 radosgw 存储池资源#查看存储池列表 root@ubuntu01:~# ceph osd pool ls .mgr kubernetes k8s-rbd k8s cephfs_data cephfs_metadata .rgw.root default.rgw.log default.rgw.control default.rgw.meta#查看默认radosgw存储池信息 root@ubuntu01:~# radosgw-admin zone get --rgw-zone=default --rgw-zonegroup=default { "id": "7f883cbd-8ada-48aa-9358-f2c09aee0ca7",# 区域的唯一标识符 "name": "default",# 默认区域的名称 "domain_root": "default.rgw.meta:root",# 区域的根域名 "control_pool": "default.rgw.control",# 系统控制池，在有数据更新是，通知其他RGW更新缓存 "gc_pool": "default.rgw.log:gc", # 用于垃圾回收的存储池 "lc_pool": "default.rgw.log:lc",# 用于存储日志的存储池 "log_pool": "default.rgw.log",# 存储日志信息，用于记录各种log信息 "intent_log_pool": "default.rgw.log:intent", "usage_log_pool": "default.rgw.log:usage", "roles_pool": "default.rgw.meta:roles",# default.rgw.meta:元数据存储池，通过不同的名称空间分别存储不同的rados对象 "reshard_pool": "default.rgw.log:reshard", "user_keys_pool": "default.rgw.meta:users.keys",# 用户的密钥名称空间users.keys "user_email_pool": "default.rgw.meta:users.email",# 用户的email名称空间users.email "user_swift_pool": "default.rgw.meta:users.swift",# 用户的subuser的名称空间users.swift "user_uid_pool": "default.rgw.meta:users.uid", # 用户UID "otp_pool": "default.rgw.otp", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "default.rgw.buckets.index",# 存放bucket到object的索引信息 "storage_classes": { "STANDARD": { "data_pool": "default.rgw.buckets.data"# 存放对象的数据 } }, "data_extra_pool": "default.rgw.buckets.non-ec",# 数据的额外信息存储池 "index_type": 0, "inline_data": true } } ], "realm_id": "", "notif_pool": "default.rgw.log:notif" }#查看默认配置信息 root@ubuntu01:~# ceph osd pool get default.rgw.meta crush_rule crush_rule: replicated_rule root@ubuntu01:~# ceph osd pool get default.rgw.meta size size: 3 root@ubuntu01:~# ceph osd pool get default.rgw.meta pgp_num pgp_num: 16 root@ubuntu01:~# ceph osd pool get default.rgw.meta pg_num pg_num: 16四、访问radosgw服务访问radosgw服务所在的节点 IP+默认 80 端口既可。五、RadosGW https 5.1生成自签证书root@ubuntu01:/# mkdir -p /etc/ceph/rgw/cert root@ubuntu01:/# openssl req -newkey rsa:2048 -x509 -days 3650 -nodes -out /etc/ceph/rgw/cert/rgw-cert.pem -keyout /etc/ceph/rgw/cert/rgw-key.pem -subj "/CN=ceph-rgw.local.com" Generating a RSA private key ...............................+++++ ..................................+++++ writing new private key to '/etc/ceph/rgw/cert/rgw-key.pem' ----- root@ubuntu01:/# tree /etc/ceph/rgw/cert/ /etc/ceph/rgw/cert/ ├── rgw-cert.pem └── rgw-key.pem 0 directories, 2 files5.2创建配置文件#将公钥和私钥内容添加到配置文件中 rgw.yaml service_type: rgw service_id: default placement: hosts: - ubuntu01 - ubuntu02 - ubuntu03 spec: rgw_frontend_ssl_certificate: | -----BEGIN PRIVATE KEY----- MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC/q3zdOnRPSsRz OlP1Z0rBYJUn9VzZF+Rlnt7IN3q+6Ir/jCb6jtMYP1A+KiTcv8OaTdre4OxOjIFG iK1bZpEl16q+phKGbnfbphrptCVwpw7vV3n7Pjz57awXM728S7ql62JutIfrDWqN YY0EnMeO7xFrXdy4ow8d3GobhsfiTcU38nyWXZ5Qy7vd43kJElMxbbsXJrhPO43x 6gJCWdvIztdjewtFW2rqy8cDPaKxcXubYAOlJUzN53eAdGxC6Br6HWsyVn8VnsPI UQigwSzOx6DAa9QKjVIIi454pb3C+se1TD3el1JhZ5ZDav5g0Fp9zB1DetkR3vh7 3ixqwddRAgMBAAECggEADohJElkBQpXPqVDt1rh7MYhKJtpyrL8kARR3ncSfGOR2 zYNp3SuBE+CRC/WUD+y2PvfSNX3mTNpYXum0Ay8WqEDe4E+lLe4oRk4k0j1nbVAt ULZYOFVyuBxuJOA2bZVsVHIxZ2VmvMqqnoeb8pKUiuDTeEmIl7M9TS1OGkIw25aa VUU+kFLNwhVXQPAYu17dApl5GWI32gNcfZD7QF3fwNMz+u4h6dLCBiI+IQmXjtzX OCPs2exwfj9NaZ6jxZPQU1i2YoMRh2AuephP7hW+zeTUpgd4A6VHkX5ASKvs89U2 qel2639RLqdEg1bu7f5kHOlqYVmwuj6rw+EAyfxpQQKBgQDl81/euRc7NKj3uLoh yJqe/vQkvLKgKwVDAM6aKdI8/FhG1LNYbm5QJ83EuSmXSPm20bJjZWOjmQuIXSV2 V2UvmygHcQOeZsipJPgc2Yza6b3MDCKbBq/Kb4YrTtVLogjgKbbJXg99sSPKgJFG B4g4OGs8H2inkE3tKOn4CXUF2QKBgQDVYfXnMuGcZ8LfBiD1zfX1WGt8jKsoDI2/ LGGwQa/0Lo7xgqrN47hOFKMDGWglJPT06G2JR3gsPIvJlZNo5m6tYoLtZ0vEERlK +YzTVmTA/ThGAK7rePQv7GOGWubF6WYA8Bdyfqnm0yv8QtptcXq4bzyLEGMTKh4a mNOBw0eaOQKBgQDh4IczyKzBQ6EWdSahOORpehdoXtM38gphHtHTLXGO5nHwYc8p 9o/1vKOrshbgvJZOtv18FYQil5t0f8rkzERg1xAi/yiG2IATZnUyIaDzCNefL+xu S8syFwq8T9vFR41gZJlghDzDQtXdfR02pP9x+jmb/V5X+rARXjc1plSFQQKBgAvG 0UyGSV9Zdq7aZr1KNbXpwFzqYpPeRYB0kZuptG1UmH0JyiV82PIuP2TvZQkPxhky LsFx2VcPrGNexvj2JsuY8ULq/Yp/qxaxOS18yijAkPeEGCNU1J1EfaWvpKbtn7yT g6fFB9l+dCIDCo0Zwz0knoHKUL2BCJJNNvclcPE5AoGAXCuMEjSk54OUXoKj355l NbbpnrGofy7Pmi9H7dKxgKiN7wonjhV9Rztrbsms9lD4Ab/B1CrIRz6Hp5si+bw/ mDDboXmgEbNGMccxknye2p4S34R6RoERbNlXvor3Z46aWMP9Suqwa11WdklXQiaW wLC0RkGadfimQSxSGuFaJDE= -----END PRIVATE KEY----- -----BEGIN CERTIFICATE----- MIIDGzCCAgOgAwIBAgIUYkIzcxcGB4FBt1qTLXE9tOKNeMUwDQYJKoZIhvcNAQEL BQAwHTEbMBkGA1UEAwwSY2VwaC1yZ3cubG9jYWwuY29tMB4XDTI1MDYxNDE5MTEy MloXDTM1MDYxMjE5MTEyMlowHTEbMBkGA1UEAwwSY2VwaC1yZ3cubG9jYWwuY29t MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv6t83Tp0T0rEczpT9WdK wWCVJ/Vc2RfkZZ7eyDd6vuiK/4wm+o7TGD9QPiok3L/Dmk3a3uDsToyBRoitW2aR JdeqvqYShm5326Ya6bQlcKcO71d5+z48+e2sFzO9vEu6petibrSH6w1qjWGNBJzH ju8Ra13cuKMPHdxqG4bH4k3FN/J8ll2eUMu73eN5CRJTMW27Fya4TzuN8eoCQlnb yM7XY3sLRVtq6svHAz2isXF7m2ADpSVMzed3gHRsQuga+h1rMlZ/FZ7DyFEIoMEs zsegwGvUCo1SCIuOeKW9wvrHtUw93pdSYWeWQ2r+YNBafcwdQ3rZEd74e94sasHX UQIDAQABo1MwUTAdBgNVHQ4EFgQUbL6TyHoVR1jUBE+mT0ETcvJNwq0wHwYDVR0j BBgwFoAUbL6TyHoVR1jUBE+mT0ETcvJNwq0wDwYDVR0TAQH/BAUwAwEB/zANBgkq hkiG9w0BAQsFAAOCAQEAZ/8toy7FIUG4uq+SxP4dxcC/P/njpCklzoA5BGc8aEQ+ M+g/0eTR60ib6HWXp2PtezQ5fK1mZLImSeuHCcdAXddEq0opXaS3wEMs8N27fDLU jMjilBhzDlp7YnxZ64YzF3HzP2qHbDwoJjz/MqFSovkFEb4m8RYZNl6t/5U8XSmx vxWdypbmmd+Zr07BQ1l1ldeGi0CD9gxSYK3exF6Gdr7G/J7vC8Up0xHKnaZSqKOH vP8e/WL7T+p0s0ypjAIR29M1E9XfULt8xNQc3KtiEcvAZbxE3HWG7vnp93S/42vA errp5uKlZdaOaA1OD0/nmrP36hz6RnuUqIF88p7NnQ== -----END CERTIFICATE----- ssl: true rgw_frontend_port: 4435.3更新RGW配置root@ubuntu01:~# ceph orch apply -i rgw.yaml Scheduled rgw.default update... root@ubuntu01:~# ceph orch ps | grep rgw rgw.default.ubuntu01.lusoxe ubuntu01 *:80 running (31m) 12s ago 31m 101M - 18.2.4 2bc0b0f4375d 510eb15ca011 rgw.default.ubuntu02.abhedc ubuntu02 *:443 starting - - - - <unknown> <unknown> <unknown> rgw.default.ubuntu02.nvamia ubuntu02 *:80 running (31m) 13s ago 31m 106M - 18.2.4 2bc0b0f4375d 4a733c6c4488 rgw.default.ubuntu03.pvvalj ubuntu03 *:443 starting - - - - <unknown> <unknown> <unknown> rgw.default.ubuntu03.rkgoya ubuntu03 *:80 running (31m) 13s ago 31m 101M - 18.2.4 2bc0b0f4375d d341d13d6bac 5.4验证访问在本地host添加域名解析六、RadosGW 高可用 6.1自定义https 端口与副本数#修改默认端口并指定运行在 ceph1 和 ceph2 节点 root@ceph-1:/etc/ceph/rgw# cat > rgw.yaml << EOF service_type: rgw service_id: default placement: hosts: - ceph-1 - ceph-2 - ceph-3 spec: rgw_frontend_ssl_certificate: | -----BEGIN PRIVATE KEY----- MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC8EYtfCItzKsoy zh9AeiYMuJAebFaBkEpF00AYGvnviwOBUx8f4CjStnXAUtNq7nFrhTldai1B+pdL eZomzgfFTO6oN0tz6KBXEDJG+yoMqrce0RlGJH6i5ML1znWmWsVynU1xrqGC2D1J jIERsoyx+1vIuYHR8EBpKf7ldLYDqZ79bAEEj1ayzzPOIS4X3g9AAWo3EMCLRzCU ZY1GIfqXuPi2xMJKYfovXM1agU178qCqNlGFrjtAHOSYpmMssGM638Jh8ExaJxBU 4UvKcxc54Efypancj6ZffgEx07A6tVFk4zQrFp+2vbYTkcs3tF5KrRBvN1ExDqsb fofCTI+7AgMBAAECggEAaiYE4giynLgkE/TfEsdevoNVZLaFRO+p3CtV28UuGJP/ 0HiX8qfUosm1QG3/QjV+8s7pB96r2LeVuVXTOd/D5wp7EZrUDYHZLgrINeQBYdDh NpWSjFKA33P6zj5PjStikkRSt713D6D5Ro/1MYXzf2l97pc1vMa7tB+t7Nio+vtE PCzTsRZynbNCY3UoIKfQlbA/fMqyayU05GAJyJT0kHl8M1H5PD5czJjUqztKQEv0 aJGmXN77drBC/qznGfpTPaAi1l6Gh7eBBj8/7yWXDSyI3n7SSAz0e4eDjZRhBZrH hi2f6+xwpuTuOehUw0xo2rtJxl3o9qzNjwN05A8yoQKBgQDdOzuiAZtqBd7ERXZp sYev17tLuLpiCHQ5a5ljh4jVrP1/bh3mNiTnFMk3TwJLIwBPlo2Ugbd/vsl+Db3+ EKDFTC2md5CaH04/1QSmFFjsHfN1ZoLdmedVntlR4Hzah44jNdWvHl0WYRjzfxcM I782oawPYm73J7oGboiEfgtW8wKBgQDZoBNxcvjFZ+0LO5ms+ymWKwTnPpwGhW3M D0DcPcF08GSnmyHvZoB51FG4GEIjMkWRpOX+pg2fxNjO0Y5QxS5vBg0A5KztDg4r Kdy+McCZIWjCeF9O0mvhIXOKLFdUMrfp+s6GqJCIoht8QkxUuMn0TM3eq3p/WWts xVCs/tMmGQKBgBSyPOLsCZECmZN8+BXtMMdnhDMSRgVzywOwKDpibI+ozlJEh/GI cS1ZCXXuI0XKMXZAnGAfPn5p58muGW8SOSgb901SdCmm8hgQoo2y65qzNppuC6IV ism8wZHiUWvUMJzkpWfrjEPSs5Xb9tkA4xuGRmVuDPl8Mu/1GTpj3EW3AoGBAMJI 0pLZ3ZX+7fS1RMDViY7y4PHBR3Ha9ObUR0dYKrnHU1T+fhFIJTKehkYgAguB+fHI kTwB6u/TwOsC0lbxcj7T3BAMFwWbIrMOMG/r4tHSrb/PzuaDnKPkRU35wAz/KonM y0wUeNRCRN9uIM8SGdnsJ26/ECFZJzp3/Uo0RTUhAoGAG8X00lkMTiVHuAZmP7PO 4lYfUQA8PZ6i/7A/SnHuwWI0MyWKLw3T/4mCdHyw9YwPshdVWCddY59L1GKxdzI5 V87lNmdkH7l6jDm7IwY5KX0voZ8uLB1zQ9lIakQxPTj5ydO2lPsJGE8784suwAhY Y7UxYWWOAl7Pu0TfGXZjg+I= -----END PRIVATE KEY----- -----BEGIN CERTIFICATE----- MIIDGzCCAgOgAwIBAgIUN4U/CuGb78PO+vzw370d0hz1aBUwDQYJKoZIhvcNAQEL BQAwHTEbMBkGA1UEAwwSY2VwaC1yZ3cubG9jYWwuY29tMB4XDTI0MTIxNzA3MDAz M1oXDTM0MTIxNTA3MDAzM1owHTEbMBkGA1UEAwwSY2VwaC1yZ3cubG9jYWwuY29t MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvBGLXwiLcyrKMs4fQHom DLiQHmxWgZBKRdNAGBr574sDgVMfH+Ao0rZ1wFLTau5xa4U5XWotQfqXS3maJs4H xUzuqDdLc+igVxAyRvsqDKq3HtEZRiR+ouTC9c51plrFcp1Nca6hgtg9SYyBEbKM sftbyLmB0fBAaSn+5XS2A6me/WwBBI9Wss8zziEuF94PQAFqNxDAi0cwlGWNRiH6 l7j4tsTCSmH6L1zNWoFNe/KgqjZRha47QBzkmKZjLLBjOt/CYfBMWicQVOFLynMX OeBH8qWp3I+mX34BMdOwOrVRZOM0Kxaftr22E5HLN7ReSq0QbzdRMQ6rG36HwkyP uwIDAQABo1MwUTAdBgNVHQ4EFgQUUxgqlKlO+Pmvr+1QYv0bzf8BY4wwHwYDVR0j BBgwFoAUUxgqlKlO+Pmvr+1QYv0bzf8BY4wwDwYDVR0TAQH/BAUwAwEB/zANBgkq hkiG9w0BAQsFAAOCAQEAJf0D5Wy3BS9fUWqqgxLgvUSuK9EzfVHyyBeAzW+AYzus Iqv3KscFnJFkl8U7tfy0E/03z6LzA91Ok/6IvlsQA9w5agJF++nqNSatVbEin4Fr h4hu1HFMDFLkaQeGLcaHBgmMWOgK0DonitYEJZMbHBBYY5W7IzoZfduaOsJaXVoG rYCsoYlH2JHwIu3hXelzCLPfZhdBpvcgWIsQCnVy2n4y2WLRif0R+zPPZ4ZIz0qT en7C+vmvtP9SrpI9eIPUC3VAcTKxftvzyOHqBIB0+BzDa8lk0b4MMmaJkzt7Uq1k EJmfIFBAfER1wHb2vVPKd5/zi3h55T3D366M8yLx9Q== -----END CERTIFICATE----- ssl: true rgw_frontend_port: 8443 EOF root@ceph-1:/etc/ceph/rgw# ceph orch apply -i rgw.yaml Scheduled rgw.default update... root@ceph-1:/etc/ceph/rgw# ceph orch ps | grep rgw rgw.default.ceph-1.rsdmtv ceph-1 *:9443 running (29s) 23s ago 29s 87.6M - 18.2.4 2bc0b0f4375d c4d059ef4eeb rgw.default.ceph-2.hyhuzv ceph-2 *:9443 running (29s) 13s ago 29s 88.8M - 18.2.4 2bc0b0f4375d ff7f86383ba1 rgw.default.ceph-3.wnnkpd ceph-3 *:9443 running (30s) 24s ago 30s 88.4M - 18.2.4 2bc0b0f4375d 3d198beac816 6.2HaProxy部署以下操作在 ceph1、2、3 机器执行安装haproxy root@ceph-1:~# apt install haproxy -y#修改配置文件 root@ceph-1:~# cat > /etc/haproxy/haproxy.cfg << EOF # 开启管理员监控页面 listen admin_stats bind *:8888 # 监听的IP和端口号 mode http # 开启HTTP模式，stats功能需要 stats enable stats refresh 30s # 统计页面自动刷新时间 stats uri /admin # 访问的uri ip:8888/admin stats realm haproxy stats auth admin:admin # 认证用户名和密码 stats hide-version # 隐藏HAProxy的版本号 stats admin if TRUE # 管理界面，如果认证成功了，可通过webui管理节点 timeout client 5s # 客户端超时 timeout connect 3s # 连接超时 timeout server 5s # 后端服务器超时 # 配置前端监听 frontend main # 监听地址 bind *:443 # 匹配后端服务 default_backend rgw # 客户端超时 timeout client 5s # 配置后端代理 backend rgw # 连接超时 timeout connect 3s # 后端服务器超时 timeout server 5s server rgw1 192.168.10.91:9443 check server rgw2 192.168.10.92:9443 check server rgw3 192.168.10.93:9443 check EOF6.3启动服务root@ceph-1:~# systemctl start haproxy root@ceph-1:~# systemctl enable haproxy Synchronizing state of haproxy.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable haproxy root@ceph-1:~# ss -tunlp | grep haproxy tcp LISTEN 0 4096 0.0.0.0:443 0.0.0.0:* users:(("haproxy",pid=61839,fd=7)) tcp LISTEN 0 4096 0.0.0.0:8888 0.0.0.0:* users:(("haproxy",pid=61839,fd=6))确认无误后 ceph-2 和 ceph-3 服务器同样的步骤配置。七、KeepAlived部署 以下操作在ceph-1、2、3 机器执行，设备网卡名称为ens33，VIP为192.168.10.90。 安装软件包root@ceph-1:~# apt install keepalived -y新增haproxy检测脚本root@ceph-1:~# vim /etc/keepalived/check_port.sh #!/bin/bash # 检查指定端口是否正常 PORT=443 if netstat -tuln | grep -q ":${PORT}\b"; then echo "${PORT}端口正常: 服务正在监听" exit 0 else echo "${PORT}端口异常: 未发现监听服务" exit 1 fi root@ceph-1:~# chmod u+x /etc/keepalived/check_port.sh修改配置文件root@ceph-1:/etc/keepalived# cat > /etc/keepalived/keepalived.conf << EOF global_defs { script_user root enable_script_security } vrrp_script chk_port { script "/etc/keepalived/check_port.sh" # 自定义检测脚本路径 interval 1 # 检测间隔，单位为秒 weight -2 # 如果检测失败，权重降低2 } vrrp_instance VI_1 { state MASTER # 设置为master节点 interface ens33 # 物理网卡名称 virtual_router_id 51 # 虚拟路由ID，主备保持一致 priority 100 # 优先级，主大于备 advert_int 1 # 关播间隔 authentication { # 认证信息，主备一致 auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.10.90/24 # 虚拟IP信息 } track_script { chk_port # 引用上面定义的脚本 } } EOF启动服务root@ceph-1:/etc/keepalived# systemctl start keepalived.service root@ceph-1:/etc/keepalived# systemctl enable keepalived.service Synchronizing state of keepalived.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable keepalived root@ceph-1:/etc/keepalived# ip a | grep 192.168.10.90 inet 192.168.10.90/24 scope global secondary ens33此时可以看到vip 192.168.10.90绑定到了 ceph-1服务器ens33 网卡上。同样的操作配置 ceph-2服务器，配置文件如下：global_defs { script_user root enable_script_security } vrrp_script chk_port { script "/etc/keepalived/check.sh" interval 2 weight -2 } vrrp_instance VI_1 { state BACKUP # 主备类型 interface ens33 virtual_router_id 51 priority 99 # 优先级低于主 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.10.90/24 } track_script { chk_port # 引用上面定义的脚本 } }ceph-3 配置如下global_defs { script_user root enable_script_security } vrrp_script chk_port { script "/etc/keepalived/check.sh" interval 2 weight -2 } vrrp_instance VI_1 { state BACKUP # 主备类型 interface ens33 virtual_router_id 51 priority 98 # 优先级低于主 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.10.90/24 } track_script { chk_port # 引用上面定义的脚本 } }八、高可用测试 接下来停止 ceph-1服务，模拟异常故障，查看 ceph-2服务器，vip已经成功飘移过来root@ceph-1:~# systemctl stop haproxy.service root@ceph-2:~# ip a | grep 192.168.10.90 inet 192.168.10.90/24 scope global secondary ens33访问vip的 443 端口，可正常提供服务
-
ceph ubuntu系统改网络配置/etc/netplan/..network: version: 2 ethernets: ens33: dhcp4: no addresses: - 192.168.3.131/24 gateway4: 192.168.3.1 nameservers: addresses: - 8.8.8.8 - 114.114.114.114 - 8.8.4.4账号：admin密码123456Abc@用https访问添加路由目录sudo ip route add default via 192.168.3.1root@ubuntu01:~# sudo resolvectl dns ens33 8.8.8.8 8.8.4.4
