基本介绍
系统调用例子
1 | #include <stdio.h> |
glusterfs client客戶端出现RPC_CLNT_DISCONNECT问题和kernel出现task xxx blocked for more than 120 seconds
故障现象
- glusterfs客户端无法连接某个节点的brick，一直出现rpc无法连接
- kernel 出现hung_task_timeout_secs超时问题
故障分析
kernel层面分析
kernel出现sync flood，原因是客户端断掉后不断重新尝试导致，后面CallTrace出现 hung_task_timeout_secs关键字，该关键字含义是，异步刷新IO到磁盘，如果在hung_task_timeout_secs时间内IO没有刷完，就会报出这个错误，解决方法需要调整vm.dirty_ratio和vm.dirty_backgroud_ratio参数，两个参数分别控制脏页刷磁盘的比例，如果后端磁盘非常慢，超过操作系统默认的timeout会出现这样的calltrace.建议设置vm.dirty_ratio=5和vm.dirty_backgroud_ratio=5
客户端日志分析
日志中大量报出0-speech_vol_v2-client-55对应的brick无法建立rpc连接(通过网络telnet 主机和端口无异常),从日志来看可能是sync flood导致，客户端无法连接到这个glusterfsd
服务端日志
服务端的日志也很明显出现read/write出现告警
解决办法
- 调整kernel中 vm.dirty_ratio=5和 vm.dirty_backgroud_ratio=10（该问题仅仅关闭kernel报错)
- 重启节点的glusterd的进程或者重启节点(sync flood的问题)
glusterfs version
- 服务端版本
1 | [root@ai-storage-prd-141-165-65-141.v-bj-4.vivo.lan:/root] |
- 客户端版本
1 | [root@storage-center-prd-10-194-8-133.v-bj-4.vivo.lan:/var/log/glusterfs] |
cluster volume
volume info
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87# gluster volume info speech_vol_v2
Volume Name: speech_vol_v2
Type: Distributed-Replicate
Volume ID: e1b28e82-0d0b-4c87-bb6a-45306ec4a568
Status: Started
Snapshot Count: 0
Number of Bricks: 24 x 3 = 72
Transport-type: tcp
Bricks:
Brick1: 141.165.65.141:/data1/brick
Brick2: 141.165.65.142:/data1/brick
Brick3: 141.165.65.144:/data1/brick
Brick4: 141.165.65.141:/data2/brick
Brick5: 141.165.65.142:/data2/brick
Brick6: 141.165.65.144:/data2/brick
Brick7: 141.165.65.141:/data3/brick
Brick8: 141.165.65.142:/data3/brick
Brick9: 141.165.65.144:/data3/brick
Brick10: 141.165.65.141:/data4/brick
Brick11: 141.165.65.142:/data4/brick
Brick12: 141.165.65.144:/data4/brick
Brick13: 141.165.65.141:/data5/brick
Brick14: 141.165.65.142:/data5/brick
Brick15: 141.165.65.144:/data5/brick
Brick16: 141.165.65.141:/data6/brick
Brick17: 141.165.65.142:/data6/brick
Brick18: 141.165.65.144:/data6/brick
Brick19: 141.165.65.141:/data7/brick
Brick20: 141.165.65.142:/data7/brick
Brick21: 141.165.65.144:/data7/brick
Brick22: 141.165.65.141:/data8/brick
Brick23: 141.165.65.142:/data8/brick
Brick24: 141.165.65.144:/data8/brick
Brick25: 141.165.65.141:/data9/brick
Brick26: 141.165.65.142:/data9/brick
Brick27: 141.165.65.144:/data9/brick
Brick28: 141.165.65.141:/data10/brick
Brick29: 141.165.65.142:/data10/brick
Brick30: 141.165.65.144:/data10/brick
Brick31: 141.165.65.141:/data11/brick
Brick32: 141.165.65.142:/data11/brick
Brick33: 141.165.65.144:/data11/brick
Brick34: 141.165.65.141:/data12/brick
Brick35: 141.165.65.142:/data12/brick
Brick36: 141.165.65.144:/data12/brick
Brick37: 141.165.181.146:/data1/brick
Brick38: 141.165.181.147:/data1/brick
Brick39: 141.165.181.148:/data1/brick
Brick40: 141.165.181.146:/data2/brick
Brick41: 141.165.181.147:/data2/brick
Brick42: 141.165.181.148:/data2/brick
Brick43: 141.165.181.146:/data3/brick
Brick44: 141.165.181.147:/data3/brick
Brick45: 141.165.181.148:/data3/brick
Brick46: 141.165.181.146:/data4/brick
Brick47: 141.165.181.147:/data4/brick
Brick48: 141.165.181.148:/data4/brick
Brick49: 141.165.181.146:/data5/brick
Brick50: 141.165.181.147:/data5/brick
Brick51: 141.165.181.148:/data5/brick
Brick52: 141.165.181.146:/data6/brick
Brick53: 141.165.181.147:/data6/brick
Brick54: 141.165.181.148:/data6/brick
Brick55: 141.165.181.146:/data7/brick
Brick56: 141.165.181.147:/data7/brick
Brick57: 141.165.181.148:/data7/brick
Brick58: 141.165.181.146:/data8/brick
Brick59: 141.165.181.147:/data8/brick
Brick60: 141.165.181.148:/data8/brick
Brick61: 141.165.181.146:/data9/brick
Brick62: 141.165.181.147:/data9/brick
Brick63: 141.165.181.148:/data9/brick
Brick64: 141.165.181.146:/data10/brick
Brick65: 141.165.181.147:/data10/brick
Brick66: 141.165.181.148:/data10/brick
Brick67: 141.165.181.146:/data11/brick
Brick68: 141.165.181.147:/data11/brick
Brick69: 141.165.181.148:/data11/brick
Brick70: 141.165.181.146:/data12/brick
Brick71: 141.165.181.147:/data12/brick
Brick72: 141.165.181.148:/data12/brick
Options Reconfigured:
network.ping-timeout: 120
cluster.read-hash-mode: 3
storage.fips-mode-rchecksum: on
nfs.disable: onvolume status
1 | # gluster volume status speech_vol_v2 |
客户端无法连接的Brick的信息
1 | //这些信息存在于/var/log/glusterfs/mnt-{挂载名称}.log中的Final graph: |
客户端挂载Debug日志
操作系统kernel日志(/var/log/message)
1 | May 26 12:20:18 storage-center-prd-141-165-181-147 kernel: [4036004.156726] TCP: request_sock_TCP: Possible SYN flooding on port 24007. Sending cookies. Check SNMP counters. |
客户端Mount Debug日志详情
1 | [2020-05-27 03:47:34.303569] D [logging.c:1718:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5 |
0-speech_vol_v2-client-55对应服务端节点(147下的/data7/brick)日志
1 | [2020-05-27 01:02:22.421765] W [socket.c:774:__socket_rwv] 0-tcp.speech_vol_v2-server: readv on 141.165.32.25:47645 failed (No data available) |
glusterfs客户端挂载init流程
Glusterfs 原理
Glusterfs 基本原理
Glusterfs 是基于fuse的分布式存储，功能上支持分布式/3副本/EC三种存储方式。Glusterfs采用堆栈式的架构设计，服务端和客户端采用translator.
GlusterFS概念中，由一系列translator构成的完整功能栈称之为Volume，分配给一个volume的本地文件系统称为brick，被至少一个translator处理过的brick称为subvolume。客户端是由于volume类型来加载对应的translator，服务端也是一样，根据不同的volume的类型加载translator。客户端(glusterfs)通过挂载时候提供节点IP地址，很对应节点的服务端管理进程通信，获取brick源信息、客户单需要加载的配置，客户端根据配置初始化xlator，后续IO的流程按照xlator的顺序经过每个xlator的fop函数，然后直接和对应的glusterfsd的进程交互IO操作。glusterfsd也是一样，根据服务端配置文件，初始化服务端需要加载xlator，进行每个xlator的fop的操作，最终执行系统IO函数进行IO操作。节点的管理服务(glusterd),仅仅加载一个管理的xlator,处理来自glusterfs/gluster的请求，不会处理对应的IO操作操作。
Glusterfs环境
1.Glusterfs版本
1 | glusterfs 7.5 |
2.volume info
1 | Volume Name: rep3_vol |
Mount 命令
1.通过mount命令挂载
1 | mount -t glusterfs -o acl 10.193.189.153:/rep3_vol /mnt/rep3_vol2 |
2.直接使用glusterfs二进制挂载
1 | /usr/local/sbin/glusterfs --acl --process-name fuse --volfile-server=10.193.189.153 --volfile-id=rep3_vol /mnt/rep3_vol |
Mount Glusterfs 整个流程
Glusterfs 交互架构
Glusterfs 客户端实现分析
1.glusterfsd.c中的main方法入口函数
1
2
3
4
5int main(int argc, char *argv[])
{
create_fuse_mount(ctx);
glusterfs_volumes_init(ctx);
}2.create_fuse_mount初始化mount/fuse的模块，具体是加载/usr/local/lib/glusterfs/2020.05.12/xlator/mount/fuse.so,会去执行fuse.so中的init方法
1
2
3
4
5int create_fuse_mount(glusterfs_ctx_t *ctx)
{
xlator_set_type(master, "mount/fuse")
xlator_init(master);
}3.加载glusterfs fuse模块以后会去执行 glusterfs_volumes_init，该函数主要是在客户端初始化针对客户端操作的volume需要的xlator.
1
2
3
4
5
6int glusterfs_volumes_init(glusterfs_ctx_t *ctx)
{
glusterfs_mgmt_init(ctx);
glusterfs_process_volfp(ctx, fp);
}4.针对我们参数传递的volfile_server为某个节点的ip,代码会走到glusterfs_mgmt_init，该函数会根据连接请求去fetch spec
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17int glusterfs_mgmt_init(glusterfs_ctx_t *ctx)
{
rpc_clnt_register_notify(rpc, mgmt_rpc_notify, THIS);
rpcclnt_cbk_program_register(rpc, &mgmt_cbk_prog, THIS);
ctx->notify = glusterfs_mgmt_notify;
ret = rpc_clnt_start(rpc);
}
static int mgmt_rpc_notify(struct rpc_clnt *rpc, void *mydata, rpc_clnt_event_t event,
void *data)
{
switch (event) {
case RPC_CLNT_DISCONNECT:
//
case RPC_CLNT_CONNECT:
ret = glusterfs_volfile_fetch(ctx);
}
//spec 信息如下参见附件- glusterfs_volfile_fetch 会调用glusterfs_volfile_fetch_one 获取spec
1 | int glusterfs_volfile_fetch(glusterfs_ctx_t *ctx) |
- 6.glusterfs_volfile_fetch_one是实质获取客户端brick元数据和translator的函数、同时通过mgmt_getspec_cbk回调函数处理获取这些信息后的处理函数
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20static int glusterfs_volfile_fetch_one(glusterfs_ctx_t *ctx, char *volfile_id)
{
ret = mgmt_submit_request(&req, frame, ctx, &clnt_handshake_prog,
GF_HNDSK_GETSPEC, mgmt_getspec_cbk,
(xdrproc_t)xdr_gf_getspec_req);
}
//mgmt_submit_request函数会调用服务端的gluster_handshake_actors[GETSPEC]对应的函数，获取spec信息后会调用mgmt_getspec_cbk回调函数.
//在gdb参数中设置--volfile-server=10.193.189.154 这个节点上gdb attach glusterd进程，然后设置在server_getspec，然后客户端请求，然后10.193.189.154节点的glusterd在server_getspec响应
rpcsvc_actor_t gluster_handshake_actors[GF_HNDSK_MAXVALUE] = {
[GF_HNDSK_GETSPEC] = {"GETSPEC", GF_HNDSK_GETSPEC, server_getspec, NULL, 0, DRC_NA}
};
//重新获取spec初始化客户端的translator的，这个函数就是构造xlator的图，同时初始化每个xlator
int
mgmt_getspec_cbk(struct rpc_req *req, struct iovec *iov, int count,
void *myframe)
{
glusterfs_volfile_reconfigure(tmpfp, ctx);
glusterfs_process_volfp(ctx, tmpfp);
}Debug方法
1.调试的注意点
glusterfs/glusterd/glusterfsd是同一个二进制进程，仅仅是区别是glusterfsd和glusterfs软链了glusterd的进程。glusterfs使用同一个二进制程序，根据不同的业务功能，fork或者走不同逻辑分别来实现glusterfs/glusterfsd/glusterd的功能。
2.调试的注意点
3.具体调试方法
- 1.执行命令gdb /usr/local/sbin/glusterfs
- 2.进入gdb的交互控制界面，需要设置参数,参数设置如下
1
2
3
4
5gdb /usr/local/sbin/glusterfs
set args --acl --process-name fuse --volfile-server=10.193.189.153 --volfile-id=rep3_vol /mnt/rep3_vol
(gdb) set print pretty on
(gdb) br main
(gdb) br create_fuse_mount - 3.当断点执行到create_fuse_mount，该函数通过xlator_init加载mount/fuse的translor。
1
2
3
4
5
6
7
8(gdb)
Detaching after fork from child process 37741.
Breakpoint 3, create_fuse_mount (ctx=0x63e010) at glusterfsd.c:719
//中间过程
(gdb)
Detaching after fork from child process 39472.
770 if (ret) {
} - 4.等create_fuse_mount执行完毕以后，需要设置调试进程的模式,这样不至于进程一致停留在父进程中，子进程是需要和调试参数中设置的ip的节点通信，获取相关的bricks信息和客户端需要加载的translator
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25Detaching after fork from child process 39472.
770 if (ret) {//}
(gdb) set follow-fork-mode child
(gdb) set detach-on-fork off
(gdb)
main (argc=7, argv=0x7fffffffe368) at glusterfsd.c:2875
2875 if (ret)
(gdb)
2878 ret = daemonize(ctx);
(gdb)
[New process 39570]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7fffeee4d700 (LWP 39573)]
[New Thread 0x7fffee64c700 (LWP 39574)]
[Switching to Thread 0x7ffff7fe74c0 (LWP 39570)]
main (argc=7, argv=0x7fffffffe368) at glusterfsd.c:2879
2879 if (ret)
Missing separate debuginfos, use: debuginfo-install glibc-2.17-260.el7_6.3.x86_64 libuuid-2.23.2-59.el7.x86_64 openssl-libs-1.0.2k-16.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb)
2887 mem_pools_init();
(gdb)
(gdb) set print elements 0
(gdb) show print elements - 5.在子进程设置glusterfs客户端执行逻辑的函数,通过rpc请求，向服务端glusterd拉取bricks元数据和客户端需要加载的translator的信息
1
2
3
4
5
6
7(gdb) mgmt_getspec_cbk
(gdb) glusterfs_volfile_fetch_one
(gdb) glusterfs_volfile_fetch
(gdb) glusterfs_mgmt_init
(gdb) glusterfs_volumes_init
(gdb) glusterfs_process_volfp
(gdb) glusterfs_graph_construct附件
1 | volume rep3_vol-client-0 |
OpenCAS 消耗过多内存问题
OpenCase 内存耗尽的问题
1.现象
- 使用3.5T SSD 初始化opencas初始化，宿主机内存为120G，初始化成功后，运行一段时间出现内存全部消耗完。
2.opencas 版本信息
1 | opencas version:19.9 |
3.opencas 出问题时候日志
1 | use command: grep vmalloc /proc/vmallocinfo |grep cas_cache |
3.opencas 相关配置
cache mode: WT
cache config
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82casadm -L
type id disk status write policy device
cache 1 /dev/sda Running wt -
└core 1 /dev/sdd Active - /dev/cas1-1
cache 2 /dev/sdb Running wt -
└core 1 /dev/sde Active - /dev/cas2-1
casadm -P -i 1
Cache Id 1
Cache Size 926351414 [4KiB Blocks] / 3533.75 [GiB]
Cache Device /dev/sda
Core Devices 1
Inactive Core Devices 0
Write Policy wt
Eviction Policy lru
Cleaning Policy alru
Promotion Policy always
Cache line size 4 [KiB]
Metadata Memory Footprint 46.7 [GiB]
Dirty for 0 [s] / Cache clean
Metadata Mode normal
Status Running
╔══════════════════╤═══════════╤══════╤═════════════╗
║ Usage statistics │ Count │ % │ Units ║
╠══════════════════╪═══════════╪══════╪═════════════╣
║ Occupancy │ 500156640 │ 54.0 │ 4KiB blocks ║
║ Free │ 426194774 │ 46.0 │ 4KiB blocks ║
║ Clean │ 500156640 │ 54.0 │ 4KiB blocks ║
║ Dirty │ 0 │ 0.0 │ 4KiB blocks ║
╚══════════════════╧═══════════╧══════╧═════════════╝
╔══════════════════════╤════════════╤═══════╤══════════╗
║ Request statistics │ Count │ % │ Units ║
╠══════════════════════╪════════════╪═══════╪══════════╣
║ Read hits │ 2628479218 │ 61.6 │ Requests ║
║ Read partial misses │ 0 │ 0.0 │ Requests ║
║ Read full misses │ 64 │ 0.0 │ Requests ║
║ Read total │ 2628479282 │ 61.6 │ Requests ║
╟──────────────────────┼────────────┼───────┼──────────╢
║ Write hits │ 1579075690 │ 37.0 │ Requests ║
║ Write partial misses │ 4237509 │ 0.1 │ Requests ║
║ Write full misses │ 58309934 │ 1.4 │ Requests ║
║ Write total │ 1641623133 │ 38.4 │ Requests ║
╟──────────────────────┼────────────┼───────┼──────────╢
║ Pass-Through reads │ 0 │ 0.0 │ Requests ║
║ Pass-Through writes │ 0 │ 0.0 │ Requests ║
║ Serviced requests │ 4270102415 │ 100.0 │ Requests ║
╟──────────────────────┼────────────┼───────┼──────────╢
║ Total requests │ 4270102415 │ 100.0 │ Requests ║
╚══════════════════════╧════════════╧═══════╧══════════╝
╔══════════════════════════════════╤═════════════╤═══════╤═════════════╗
║ Block statistics │ Count │ % │ Units ║
╠══════════════════════════════════╪═════════════╪═══════╪═════════════╣
║ Reads from core(s) │ 356 │ 0.0 │ 4KiB blocks ║
║ Writes to core(s) │ 2954907028 │ 100.0 │ 4KiB blocks ║
║ Total to/from core(s) │ 2954907384 │ 100.0 │ 4KiB blocks ║
╟──────────────────────────────────┼─────────────┼───────┼─────────────╢
║ Reads from cache │ 0 │ 0.0 │ 4KiB blocks ║
║ Writes to cache │ 0 │ 0.0 │ 4KiB blocks ║
║ Total to/from cache │ 0 │ 0.0 │ 4KiB blocks ║
╟──────────────────────────────────┼─────────────┼───────┼─────────────╢
║ Reads from exported object(s) │ 12708231250 │ 81.1 │ 4KiB blocks ║
║ Writes to exported object(s) │ 2954907028 │ 18.9 │ 4KiB blocks ║
║ Total to/from exported object(s) │ 15663138278 │ 100.0 │ 4KiB blocks ║
╚══════════════════════════════════╧═════════════╧═══════╧═════════════╝
╔════════════════════╤═══════╤═════╤══════════╗
║ Error statistics │ Count │ % │ Units ║
╠════════════════════╪═══════╪═════╪══════════╣
║ Cache read errors │ 0 │ 0.0 │ Requests ║
║ Cache write errors │ 0 │ 0.0 │ Requests ║
║ Cache total errors │ 0 │ 0.0 │ Requests ║
╟────────────────────┼───────┼─────┼──────────╢
║ Core read errors │ 0 │ 0.0 │ Requests ║
║ Core write errors │ 0 │ 0.0 │ Requests ║
║ Core total errors │ 0 │ 0.0 │ Requests ║
╟────────────────────┼───────┼─────┼──────────╢
║ Total errors │ 0 │ 0.0 │ Requests ║
╚════════════════════╧═══════╧═════╧══════════╝cache line size
1
2
3
4
5
6
7
8
9
10
11
12
13
14# dmesg |grep "Cache line size"
2751 [ 1505.783016] cache1: Hash offset : 44427904 kiB
2752 [ 1505.783017] cache1: Hash size : 904644 kiB
2753 [ 1505.783018] cache1: Cache line size: 4 kiB
2754 [ 1505.783019] cache1: Metadata capacity: 47803 MiB
2755 [ 1521.649327] cache1: OCF metadata self-test PASSED
2756 [ 1527.389763] Thread cas_clean_cache1 started
2893 [ 1823.699193] cache2: Hash offset : 44427904 kiB
2894 [ 1823.699194] cache2: Hash size : 904644 kiB
2895 [ 1823.699195] cache2: Cache line size: 4 kiB
2896 [ 1823.699197] cache2: Metadata capacity: 47803 MiB
2897 [ 1839.660385] cache2: OCF metadata self-test PASSED
2898 [ 1845.359600] Thread cas_clean_cache2 started
5.解决方法
- opencas在初始化时候casadm 需要添加–cache-line-size 参数，默认是4k，针对宿主机器消耗内存的计算公式为 内存消耗 = SSD_size / Cache_Line_size * 70 byte
- 官方给的简要说明
1
2
3
4
5
6It looks quite normal to me. You have two huge cache devices (2 x 3.5TB) and size of CAS metadata is proportional to number of cache lines. CAS allocates about 70 bytes of metadata per cache line, so in your case it is about 60GiB of metadata per single cache, giving ~120GiB in total. That matches pretty well with your numbers.
You can decrease memory consumption by choosing bigger cache line size. You can select cache line size up to 64kiB, which would decrease memory usage by factor of 16.
I'd also recommend you, if it's possible, to switch to CAS v20.3.
CAS v19.9 was tested only with basic set of tests, while v20.3 was thoroughly validated with extensive set of tests, thus it's much more stable than any previous version.
6.关于opencas cache line官方说明
Why does Open CAS Linux use some DRAM space?
Open CAS Linux uses a portion of system memory for metadata, which tells us where data resides. The amount of memory needed is proportional to the size of the cache space. This is true for any caching software solution. However with Open CAS Linux this memory footprint can be decreased using a larger cache line size set by the parameter –cache-line-size which may be useful in high density servers with many large HDDs.
Configuration Tool Details
The Open CAS Linux product includes a user-level configuration tool that provides complete control of the caching software. The commands and parameters available with this tool are detailed in this chapter.
To access help from the CLI, type the -H or --help parameter for details. You can also view the man page for this product by entering the following command:
# man casadm
Usage: casadm –start-cache –cache-device
Example:
# casadm –start-cache –cache-device /dev/sdc
or
# casadm -S -d /dev/sdc
Description: Prepares a block device to be used as device for caching other block devices. Typically the cache devices are SSDs or other NVM block devices or RAM disks. The process starts a framework for device mappings pertaining to a specific cache ID. The cache can be loaded with an old state when using the -l or –load parameter (previous cache metadata will not be marked as invalid) or with a new state as the default (previous cache metadata will be marked as invalid).
Required Parameters:
[-d, –cache-device ] : Caching device to be used. This is an SSD or any NVM block device or RAM disk shown in the /dev directory.
Optional Parameters:
[-i, –cache-id ]: Cache ID to create; <1 to 16384>. The ID may be specified or by default the command will use the lowest available number first.
[-l, –load]: Load existing cache metadata from caching device. If the cache device has been used previously and then disabled (like in a reboot) and it is determined that the data in the core device has not changed since the cache device was used, this option will allow continuing the use of the data in the cache device without the need to re-warm the cache with data.
- Caution: You must ensure that the last shutdown followed the instructions in section Stopping Cache Instances. If there was any change in the core data prior to enabling the cache, data would be not synced correctly and will be corrupted.
[-f, –force]: Forces creation of a cache even if a file system exists on the cache device. This is typically used for devices that have been previously utilized as a cache device.
- Caution: This will delete the file system and any existing data on the cache device.
[-c, –cache-mode ]: Sets the cache mode for a cache instance the first time it is started or created. The mode can be one of the following:
wt: (default mode) Turns write-through mode on. When using this parameter, the write-through feature is enabled which allows the acceleration of only read intensive operations.
wb: Turns write-back mode on. When using this parameter, the write-back feature is enabled which allows the acceleration of both read and write intensive operations.
- Caution: A failure of the cache device may lead to the loss of data that has not yet been flushed to the core device.
wa: Turns write-around mode on. When using this parameter, the write-around feature is enabled which allows the acceleration of reads only. All write locations that do not already exist in the cache (i.e. the locations have not be read yet or have been evicted), are written directly to the core drive bypassing the cache. If the location being written already exists in cache, then both the cache and the core drive will be updated.
pt: Starts cache in pass-through mode. Caching is effectively disabled in this mode. This allows the user to associate all their desired core devices to be cached prior to actually enabling caching. Once the core devices are associated, the user would dynamically switch to their desired caching mode (see ‘-Q | –set-cache-mode’ for details).
wo: Turns write-only mode on. When using this parameter, the write-only feature is enabled which allows the acceleration of write intensive operations primarily.
- Caution: A failure of the cache device may lead to the loss of data that has not yet been flushed to the core device.
[-x, –cache-line-size ]: Set cache line size {4 (default), 8, 16, 32, 64}. The cache line size can only be set when starting the cache and cannot be changed after cache is started.
lustre-2.13部署
chapter 1.lustre部署
0.节点信息
| node | role |
|---|---|
| centos1(mgs_node/mds_node) | mgs/mds |
| centos2(oss_node) | oss |
| centos3(client_node) | client |
1.kernel版本信息
1 | [root@CentOS1 ~]# cat /etc/redhat-release |
2.配置离线lustre 安装包安装源
1 | // lustre-2.13.0 内核刚好匹配 kernel 3.10.0-1062.el7.x86_64 |
3.下载lustre安装包到本地
1 | [root@CentOS1 ~]# cd ~/lustre |
4.安装lustre版本kernel
1 | //after download all lustre package |
5.安装zfs
1 | [root@CentOS1 lustre]# cd lustre-server/RPMS/x86_64/ && pwd |
6.安装lustre server 核心包
1 | [root@CentOS1 ~] yum --nogpgcheck --disablerepo=* --enablerepo=lustre-server install kmod-lustre-2.13.0-1.el7.x86_64.rpm kmod-lustre-osd-ldiskfs-2.13.0-1.el7.x86_64.rpm lustre-2.13.0-1.el7.x86_64.rpm lustre-osd-ldiskfs-mount-2.13.0-1.el7.x86_64.rpm lustre-osd-zfs-mount-2.13.0-1.el7.x86_64.rpm lustre-resource-agents-2.13.0-1.el7.x86_64.rpm |
7.安装lustre client核心包(客户端节点)
1 | //lustre server和lustre client版本版本冲突，客户端必须在其他节点安装 |
8.deploy msg/mst
1 | [root@CentOS1 ~]# fdisk -l|grep sd |
9.deploy mds/mdt
1 | //mkfs.lustre --fsname=lustrefs --mgsnode=msg_node@tcp0 --mdt --index=0 /dev/sdc |
10.deploy oss/ost
1 | //mkfs.lustre --ost --fsname=lustrefs --mgsnode=mgs_node@tcp0 --index=1 /dev/sdb |
11.mount on client node
1 | [root@CentOS3 ~]# mkdir /mnt/lustrefs |
chapter 2. lustre process
1 | //show luste mgs info |
chapter 3.script
- start mgs/mdt
1 | modprobe zfs |
- start ost
1 | modprobe zfs |
- mount
1 | mount -t lustre 10.211.55.3@tcp0:/lustrefs /mnt/lustrefs |
glusterfs集群glusterd显示Disconnect状态导致集群IO请求变慢
glusterfs集群glusterd显示Disconnect状态导致集群IO请求变慢
172.25.78.16节点的集群状态
1 |
|
- 172.25.78.242节点在172.25.78.16的状态明显不对，查看其它的节点242的认证信息
172.25.78.240节点的集群状态
1 | [root@szdpl1543 ~]# gluster pool list |
172.25.78.16节点的集群状态
1 | [root@szdpl1491 ~]# gluster pool list |
- 修改每个节点的var/lib/glusterd/peers/bde7b9e2-af2a-419e-8242-6a8f8e18bb8a 中的hostname1修改为真实的IP，然后重启每个节点的glusterd的服务(systemctl restart glusterd)
安装glusterfs指定版本
1.glusterfs7 repo源
- 添加此源到/etc/yum.repo.d/glusterfs.repo中
1
2
3
4
5
6
7
8
9[centos-gluster7]
gpgcheck = 0
mirrorlist = http://mirrorlist.centos.org?arch=$basearch&release=$releasever&repo=storage-gluster-7
name = CentOS-$releasever - Gluster 7
[centos-gluster6]
gpgcheck = 0
mirrorlist = http://mirrorlist.centos.org?arch=$basearch&release=$releasever&repo=storage-gluster-6
name = CentOS-$releasever - Gluster 62.更新系统yum 源
1
yum check-update
3.安装脚本(glusterfs_install.sh)
- 脚本内容
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27#!/bin/bash
version="$1"
[[ "$version" == "" ]] && echo "version not provided" && exit 1
rpm_packages=(
glusterfs-server
glusterfs-events
glusterfs-extra-xlators
glusterfs-geo-replication
glusterfs-libs
glusterfs-rdma
glusterfs
glusterfs-api
glusterfs-api-devel
glusterfs-cli
glusterfs-client-xlators
glusterfs-fuse
python2-gluster
)
for index in ${!rpm_packages[@]}; do
rpm_version_packages[$index]=${rpm_packages[$index]}"-"$version
done
yum install -y ${rpm_version_packages[*]} - 脚本执行
1
2// 7.2是版本号
./glusterfs_install.sh 7.2
Go语言调用C函数例子
- go access c struct
1 | // go run cgo.go |
- go access c memory
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61// go build -o cgo_test cgo.go
package main
/*
#include <stdlib.h>
void *alloc() {
static int count = 0;
void *d = malloc(sizeof(int));
*((int *)d) = count++;
return d;
}
*/
import "C"
import (
"fmt"
"runtime"
"sync"
"time"
"unsafe"
)
type CStruct struct {
sync.Mutex
name string
allocCnt int
memory unsafe.Pointer
}
func (cs *CStruct) alloc(name string, id int) {
cs.name = fmt.Sprintf("CStruct-%s-%d", name, id)
cs.Lock()
defer cs.Unlock()
cs.allocCnt++
fmt.Printf("%s begin with alloc,count=%d\n", cs.name, cs.allocCnt)
cs.memory = C.alloc()
runtime.SetFinalizer(cs, free)
}
func free(cs *CStruct) {
C.free(unsafe.Pointer(cs.memory))
cs.Lock()
defer cs.Unlock()
cs.allocCnt--
fmt.Printf("%s end with free count=%d\n", cs.name, cs.allocCnt)
}
func CStructTest(i int) {
var c1, c2 CStruct
c1.alloc("c1", i)
c2.alloc("c2", i)
}
func main() {
for i := 0; i < 10; i++ {
CStructTest(i)
time.Sleep(time.Second)
}
runtime.GC()
time.Sleep(time.Second)
fmt.Println("done..")
}
实现简单的shell
- shell实现代码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73/*************************************************************************
> File Name: shell.c
> Author:perrynzhou
> Mail:perrynzhou@gmail.com
> Created Time: Thu 20 Jun 2019 09:15:59 PM CST
************************************************************************/
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <glob.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>
static const char *delimiter = " \t\n";
typedef struct
{
glob_t gt;
int (*cmd_cd_fn)(char **argv);
int (*cmd_exit_fn)(char **argv);
int (*cmd_help_fn)(char **argv);
} cmd_t;
static void prompt()
{
fprintf(stdout, "zsh-0.1$ ");
}
void parsed_cmd(char *line, cmd_t *cmd)
{
char *token;
int flag = 0;
while (1)
{
token = strsep(&line, delimiter);
if (token == NULL)
{
break;
}
if (*token == '\0')
{
continue;
}
glob(token, GLOB_NOCHECK | GLOB_APPEND * flag, NULL, &cmd->gt);
flag = 1;
}
}
int main(int argc, char *argv[])
{
char *line = NULL;
size_t line_size = 0;
cmd_t cmd;
pid_t pid;
while (1)
{
prompt();
if (getline(&line, &line_size, stdin) < 0)
{
break;
}
parsed_cmd(line, &cmd);
pid = fork();
switch (pid)
{
case -1:
perror("fork()");
exit(1);
case 0:
execvp(cmd.gt.gl_pathv[0], cmd.gt.gl_pathv);
exit(0);
default:
wait(NULL);
}
}
}
关于CPU字节序的理解
为什么会有字节序这一说法
- 对于单字节处理器不存在字节序，对于大于1个字节的位数的CPU，其寄存器的宽度也大于1个字节，所以存储字节排序的问题
大端字节序排序
- 大端：高字节位在低地址存储，低字节位在高地址存储
- 小端: 高字节位存储在高地址，低字节位存储在低地址
1
2
3
4例如 int x = 0x0001,假设是在64位的CPU上，它的地址空间为：0xff13,其字节排序可能有两种可能：
oxff13: 0 0 0 1 --->第一种情况 大端
0xff13: 1 0 0 0 --->第二种情况 小端
在x这个值中0001从左往右是高字节位到低字节位，也就是 0,0 是高字节，0 1 是低字节位。如何验证CPU的字节序呢？
- C语言中有union结构，这个结构的数据存储是从低地址向高地址依次存储值，可以通过这个特性来验证CPU的字节序
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37/*************************************************************************
> File Name: byte_order.c
> Author:perrynzhou
> Mail:perrynzhou@gmail.com
> Created Time: Fri 15 Nov 2019 03:42:37 PM CST
************************************************************************/
#include <stdio.h>
typedef union object_t {
int a;
char b;
} object;
enum
{
big_endian_type = 0,
small_endian_type,
};
int checkCpu(object *obj)
{
return obj->b == 1 ? small_endian_type : big_endian_type;
}
int main()
{
int v = 0x0001;
object obj;
obj.a = v;
fprintf(stdout, "value:%d,value_string:%s,object address:%p\n", v, "0x0001", &obj);
if (checkCpu(&obj) == big_endian_type)
{
fprintf(stdout, "store plan: |%d|%d|%d|%d|,big endian\n", 0, 0, 0, 1);
}
else
{
fprintf(stdout, "store plan: |%d|%d|%d|%d|,small endian\n", 1, 0, 0, 0);
}
return 0;
}1
2
3
4[perrynzhou@localhost ~/Source/vivo/linux_kernel_study/chapter_01]$ ./test
value:1,value_string:0x0001,object address:0x7ffcaf058a08
store plan: |0|0|0|1|,big endian
[perrynzhou@localhost ~/Source/vivo/linux_kernel_study/chapter_01]$