perrynzhou

专注于系统组件研发

0%

glusterfs小文件调优指南

发表于

glusterfs 小文件调优指南

获取glusterfs参数说明和默认值

  • glusterfs获取所有可用参数

    1
    # gluster volume set help

  • 获取指定参数

    1
    # gluster volume set help|grep "cache-min-file-size" -A7
    glusterfs volume调优参数

  • 1.目录操作性能

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    Option: performance.readdir-ahead
    Default Value: on
    Description: enable/disable readdir-ahead translator in the volume.

    Option: performance.rda-cache-limit
    Default Value: 10MB
    Description: maximum size of cache consumed by readdir-ahead xlator. This value is global and total memory consumption by readdir-ahead is capped by this value, irrespective of the number/size of directories cached

    Option: cluster.readdir-optimize
    Default Value: off
    Description: This option if set to ON enables the optimization that allows DHT to requests non-first subvolumes to filter out directory entries.

    Option: cluster.lookup-unhashed
    Default Value: on
    Description: This option if set to ON, does a lookup through all the sub-volumes, in case a lookup didn't return any result from the hash subvolume. If set to OFF, it does not do a lookup on the remaining subvolumes.

    Option: performance.parallel-readdir
    Default Value: off
    Description: If this option is enabled, the readdir operation is performed in parallel on all the bricks, thus improving the performance of readdir. Note that the performance improvement is higher in large clusters

    gluster volume set dht_vol performance.readdir-ahead on
    gluster volume set dht_vol cluster.readdir-optimize on
    gluster volume set dht_vol cluster.lookup-unhashed off
    gluster volume set dht_vol performance.parallel-readdir on
    //默认是关闭
    gluster volume set dht_vol group metadata-cache

  • 2.inode缓存大小

    1
    2
    3
    4
    5
    6
    7
    //官方解释
    Option: network.inode-lru-limit
    Default Value: 16384
    Description: Specifies the limit on the number of inodes in the lru list of the inode cache.

    //设置
    gluster volume set dht_vol network.inode-lru-limit 100000

  • 3.实际IO操作线程调整

    1
    2
    3
    4
    5
    6
    7
    //官方解释
    Option: performance.io-thread-count
    Default Value: 16
    Description: Number of threads in IO threads translator which perform concurrent IO operations

    // 设置值小于等于可用CPU的合数
    gluster volume set dht_vol performance.io-thread-count 32

  • 4.客户端rpc请求吞吐量设置

    1
    2
    3
    4
    5
    Option: server.outstanding-rpc-limit
    Default Value: 64
    Description: Parameter to throttle the number of incoming RPC requests from a client. 0 means no limit (can potentially run out of memory)

    gluster volume set dht_vol server.outstanding-rpc-limit 512

  • 5.event线程数设置（提高性能,降低响应时间)

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    //官方解释
    Option: client.event-threads
    Default Value: 2
    Description: Specifies the number of event threads to execute in parallel. Larger values would help process responses faster, depending on available processing power. Range 1-32 threads.

    Option: server.event-threads
    Default Value: 2
    Description: Specifies the number of event threads to execute in parallel. Larger values would help process responses faster, depending on available processing power.

    //设置超过可用CPU核数会导致context切换严重
    gluster volume set dht_vol  client.event-threads  8
    gluster volume set dht_vol  server.event-threads  8

  • 6.io-cache调整

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    //官方解释
    # gluster volume set help|grep "io-cache" -A7
    Option: performance.cache-min-file-size
    Default Value: 0
    Description: Minimum file size which would be cached by the io-cache translator.

    Option: performance.cache-min-file-size
    Default Value: 0
    Description: Minimum file size which would be cached by the io-cache translator.

    Option: performance.cache-refresh-timeout
    Default Value: 1
    Description: The cached data for a file will be retained for 'cache-refresh-timeout' seconds, after which data re-validation is performed.

    Option: performance.io-cache-pass-through
    Default Value: false
    Description: Enable/Disable io cache translator

    Option: performance.io-cache
    Default Value: on
    Description: enable/disable io-cache translator in the volume.

    Option: performance.open-behind
    Default Value: on
    Description: enable/disable open-behind translator in the volume.


    //当前我们采用opencas作为glusterfsd的后端存储，同时我们业务场景又是存储10亿+的小文件，因此IO-cache开启的意义不大,默认是开启（针对大文件效果好)，需要关闭才可以
    gluster volume set dht_vol  performance.io-cache  off
gluster volume调优样例
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# gluster volume info warm_vol1 
 
Volume Name: warm_vol1
Type: Distribute
Volume ID: d36874f3-60a0-458a-88b3-7f5ed18c645e
Status: Started
Snapshot Count: 0
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 172.21.73.96:/glusterfs/warmvol1/data1/brick1
Brick2: 172.21.73.96:/glusterfs/warmvol1/data2/brick1
Options Reconfigured:
performance.rda-cache-limit: 1024MB
client.event-threads: 8
server.outstanding-rpc-limit: 512
performance.io-thread-count: 32
server.event-threads: 8
network.inode-lru-limit: 500000
performance.read-ahead-page-count: 16
cluster.min-free-inodes: 25%
performance.readdir-ahead: on
cluster.readdir-optimize: on
cluster.lookup-optimize: on
performance.io-cache: off
cluster.lookup-unhashed: off
performance.parallel-readdir: on
storage.fips-mode-rchecksum: on
linux 网络参数调优
1
2
3
4
5
6
7
8
9
//vi /etc/sysctl.conf 添加如下内容
net.core.rmem_max=67108864
net.core.wmem_max=67108864
net.ipv4.tcp_wmem=33554432
net.ipv4.tcp_rmem=33554432
net.core.netdev_max_backlog=30000
net.ipv4.tcp_congestion_control=htcp

// sysctl -p 生效