本文基于asan的官方文章阅读后,对asan调试内存问题的方法做一次上手,一方面能够清楚asan的工作原理,另一方面后面遇到内存问题的时候,可以留一份记忆使用asan来定位内存问题。此文章力求浅显易懂。
为了检测内存问题,asan做了两方面事情
对于1,preload的目的是从glibc的底层api上对函数堆栈进行符号化,例如打印如下格式
#0xabcdf function_name file_name.cc:1234
对于2,影子内存的作用是监听内存是否越界访问等,其思路是:
对系统的内存,按照1/8的方式创建一个影子区域 然后在影子区域的值按照如下方式进行标记 1. 0x0 代表此8个字节均正常访问 2. 0x1-0x7 代表部分字节可正常访问,其值是 8-k 3. 负数(0xf1-0xfe,等) 代表此内存不可访问原因如下
这样,我们通过影子内存可以知道内存的错误访问,然后通过ld preload可以将错误的内存的堆栈及线程相关信息打印出来。从而定位内存问题
asan能够定位的问题如下:
关于影子内存的值的解释,如下
Shadow byte legend (one shadow byte represents 8 application bytes): Addressable(可正常访问): 00 Partially addressable(部分可访问): 01 02 03 04 05 06 07 Heap left redzone(堆左边红区): fa Freed heap region(已释放区域): fd Stack left redzone(栈左边红区): f1 Stack mid redzone(栈中间红区): f2 Stack right redzone(栈右边红区): f3 Stack after return(返回后访问): f5 Stack use after scope(作用域之外访问栈): f8 Global redzone(全局变量红区): f9 Global init order(全局变量初始化顺序): f6 Poisoned by user(被用户下毒/污染): f7 Container overflow(容器溢出): fc Array cookie(数组越界访问): ac Intra object redzone(内部对象红区): bb ASan internal(asan内部访问): fe Left alloca redzone(alloca左红区): ca Right alloca redzone(alloca右红区): cb Shadow gap(间隔区域): cc
关于出错地址到影子区域的内存换算,其公式如下
Shadow = (Mem >> 3) + offset; [0x10007fff8000, 0x7fffffffffff] HighMem [0x02008fff7000, 0x10007fff7fff] HighShadow [0x00008fff7000, 0x02008fff6fff] ShadowGap [0x00007fff8000, 0x00008fff6fff] LowShadow [0x000000000000, 0x00007fff7fff] LowMem
故python转换程序如下
import sys def asan_shadow_addr(addr): result = ((addr >> 3) + 0x0000100000000000) & 0xFFFFFFFFFFFF return hex(result) if __name__ == "__main__": addr_str = sys.argv[1] addr = int(addr_str, 0) output = asan_shadow_addr(addr) print(f"shadow_addr:{output}")
我们创建一个run.sh,这样就不用每次都使用LD_PRELOAD环境变量了
# cat ./run.sh export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libasan.so.5.0.0 exec $@
下面通过测试代码来上手实验。
对于使用释放后区域,代码示例如下
int main(int argc, char **argv) { int *array = new int[100]; delete [] array; return array[argc]; // BOOM }
运行
# ./run.sh ./use_after_free ================================================================= ==64466==ERROR: AddressSanitizer: heap-use-after-free on address 0x007fa8803e44 at pc 0x0000004007fc bp 0x007fd9d45800 sp 0x007fd9d45820 READ of size 4 at 0x007fa8803e44 thread T0 #0 0x4007f8 in main /root/asan/use_after_free.cpp:4 #1 0x7fac394d8c in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x20d8c) #2 0x4006a8 (/root/asan/use_after_free+0x4006a8) 0x007fa8803e44 is located 4 bytes inside of 400-byte region [0x007fa8803e40,0x007fa8803fd0) freed by thread T0 here: #0 0x7fac611d54 in operator delete[](void*) (/usr/lib/aarch64-linux-gnu/libasan.so.5.0.0+0xefd54) #1 0x400798 in main /root/asan/use_after_free.cpp:3 #2 0x7fac394d8c in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x20d8c) #3 0x4006a8 (/root/asan/use_after_free+0x4006a8) previously allocated by thread T0 here: #0 0x7fac610f3c in operator new[](unsigned long) (/usr/lib/aarch64-linux-gnu/libasan.so.5.0.0+0xeef3c) #1 0x400780 in main /root/asan/use_after_free.cpp:2 #2 0x7fac394d8c in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x20d8c) #3 0x4006a8 (/root/asan/use_after_free+0x4006a8) SUMMARY: AddressSanitizer: heap-use-after-free /root/asan/use_after_free.cpp:4 in main Shadow bytes around the buggy address: 0x001ff5100770: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x001ff5100780: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x001ff5100790: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x001ff51007a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x001ff51007b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa =>0x001ff51007c0: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd 0x001ff51007d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x001ff51007e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x001ff51007f0: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa 0x001ff5100800: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x001ff5100810: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa ==64466==ABORTING
报错解析可以看到在栈区地址0x007fa8803e44读取了4字节,这里0x007fa8803e44是一个400字节的区域
READ of size 4 at 0x007fa8803e44 thread T0 0x007fa8803e44 is located 4 bytes inside of 400-byte region [0x007fa8803e40,0x007fa8803fd0)
pc的位置如下,其值是0x0000004007fc
#0 0x4007f8 in main /root/asan/use_after_free.cpp:4
内存释放的地方在测试代码第3行,如下:
freed by thread T0 here: #0 0x7fae794d54 in operator delete[](void*) (/usr/lib/aarch64-linux-gnu/libasan.so.5.0.0+0xefd54) #1 0x400798 in main /root/asan/use_after_free.cpp:3
内存申请的地方在测试代码第2行
previously allocated by thread T0 here: #0 0x7fae793f3c in operator new[](unsigned long) (/usr/lib/aarch64-linux-gnu/libasan.so.5.0.0+0xeef3c) #1 0x400780 in main /root/asan/use_after_free.cpp:2
对于影子区域内存转换如下
# python3 asan_shadow_addr.py 0x007fa8803e44 shadow_addr:0x100ff51007c8
可以看到0x100ff51007c8的值是0xfd,我们查表如下
Freed heap region: fd
返回代码中看,我们访问了数组的第1项,但访问之前,我们已经delete掉这个数组了。
delete [] array; return array[argc];
代码如下
int main(int argc, char **argv) { int *array = new int[100]; array[0] = 0; int res = array[argc + 100]; // BOOM delete [] array; return res; }
运行
# ./run.sh ./heap_out_of_bounds ================================================================= ==65270==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x007fb3303fd4 at pc 0x0000004008a4 bp 0x007fedcddd30 sp 0x007fedcddd50 READ of size 4 at 0x007fb3303fd4 thread T0 #0 0x4008a0 in main /root/asan/heap_out_of_bounds.cpp:4 #1 0x7fb6e60d8c in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x20d8c) #2 0x400708 (/root/asan/heap_out_of_bounds+0x400708) 0x007fb3303fd4 is located 4 bytes to the right of 400-byte region [0x007fb3303e40,0x007fb3303fd0) allocated by thread T0 here: #0 0x7fb70dcf3c in operator new[](unsigned long) (/usr/lib/aarch64-linux-gnu/libasan.so.5.0.0+0xeef3c) #1 0x4007e0 in main /root/asan/heap_out_of_bounds.cpp:2 #2 0x7fb6e60d8c in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x20d8c) #3 0x400708 (/root/asan/heap_out_of_bounds+0x400708) SUMMARY: AddressSanitizer: heap-buffer-overflow /root/asan/heap_out_of_bounds.cpp:4 in main Shadow bytes around the buggy address: 0x001ff66607a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x001ff66607b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x001ff66607c0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00 0x001ff66607d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ff66607e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x001ff66607f0: 00 00 00 00 00 00 00 00 00 00[fa]fa fa fa fa fa 0x001ff6660800: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x001ff6660810: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x001ff6660820: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x001ff6660830: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x001ff6660840: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
报错信息可以看到从0x007fb3303fd4读取了4字节,0x007fb3303fd4属于400字节的区域范围外+8的字节处 0x007fb3303fd0处是小括号,不包含0x007fb3303fd0
READ of size 4 at 0x007fb3303fd4 thread T0 0x007fb3303fd4 is located 4 bytes to the right of 400-byte region [0x007fb3303e40,0x007fb3303fd0)
pc的地址是0x0000004008a4,指向代码第四行的下一条指令
#0 0x4008a0 in main /root/asan/heap_out_of_bounds.cpp:4
影子区域内存转换
# python3 asan_shadow_addr.py 0x007fb3303fd4 shadow_addr:0x100ff66607fa
对于出错原因是堆左红区
Heap left redzone: fa
返回代码看,这里访问了第101项(访问0x007fa7303fd4),因为0x100ff66607fa包含8字节的影子描述,所以此问题和访问第100项问题报错(访问0x007fa7303fd0)是一致的
出错代码如下
int *array = new int[100]; int res = array[argc + 100]; // BOOM
栈越界的测试代码如下
int main(int argc, char **argv) { int stack_array[100]; stack_array[1] = 0; return stack_array[argc + 100]; // BOOM }
运行
==113972==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x007fd34029f4 at pc 0x000000400a94 bp 0x007fd34027c0 sp 0x007fd34027e0 READ of size 4 at 0x007fd34029f4 thread T0 #0 0x400a90 in main /root/asan/stack_of_bounds.cpp:4 #1 0x7fa46bcd8c in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x20d8c) #2 0x400858 (/root/asan/stack_of_bounds+0x400858) Address 0x007fd34029f4 is located in stack of thread T0 at offset 452 in frame #0 0x400928 in main /root/asan/stack_of_bounds.cpp:1 This frame has 1 object(s): [48, 448) 'stack_array' (line 2) <== Memory access at offset 452 overflows this variable HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork (longjmp and C++ exceptions *are* supported) SUMMARY: AddressSanitizer: stack-buffer-overflow /root/asan/stack_of_bounds.cpp:4 in main Shadow bytes around the buggy address: 0x001ffa6804e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ffa6804f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ffa680500: 00 00 00 00 00 00 f1 f1 f1 f1 f1 f1 00 00 00 00 0x001ffa680510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ffa680520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x001ffa680530: 00 00 00 00 00 00 00 00 00 00 00 00 00 00[f3]f3 0x001ffa680540: f3 f3 f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00 0x001ffa680550: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ffa680560: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ffa680570: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ffa680580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
从报错可以了解0x007fd34029f4处读取了4字节,并且其位于frame的第452字节,其中stack_array范围是[48, 448)
,那么越界访问了第4-8个字节,错在stack_of_bounds.cpp的第4行。摘出错误信息如下
READ of size 4 at 0x007fd34029f4 thread T0 Address 0x007fd34029f4 is located in stack of thread T0 at offset 452 in frame [48, 448) 'stack_array' (line 2) <== Memory access at offset 452 overflows this variable SUMMARY: AddressSanitizer: stack-buffer-overflow /root/asan/stack_of_bounds.cpp:4 in main
影子区域内存转换如下
# python3 asan_shadow_addr.py 0x007fd34029f4 shadow_addr:0x100ffa68053e
可以看到其值是f3,也就是栈右红区
=>0x001ffa680530: 00 00 00 00 00 00 00 00 00 00 00 00 00 00[f3]f3 Stack right redzone: f3
返回代码中看,这里访问了stack_array[101]
正好是栈区的第4-8字节。
当然,如果想复现栈左红区的报错,我们访问数组的-1
项即可stack_array[-1]
,那么错误如下
[48, 448) 'stack_array' (line 2) <== Memory access at offset 44 underflows this variable =>0x001ffc07f0f0: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 f1[f1]00 00 0x001000082240: f9 f9 f9 f9 00 00 f9 f9 f9 f9 f9 f9 f9 f9 00 00 Stack left redzone: f1
测试代码如下
int global_array[100] = {-1}; int main(int argc, char **argv) { return global_array[argc + 100]; // BOOM }
运行
==115097==ERROR: AddressSanitizer: global-buffer-overflow on address 0x0000004111f4 at pc 0x000000400814 bp 0x007ff0360e80 sp 0x007ff0360ea0 READ of size 4 at 0x0000004111f4 thread T0 #0 0x400810 in main /root/asan/global_out_of_bounds.cpp:3 #1 0x7f8fecad8c in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x20d8c) #2 0x4006d8 (/root/asan/global_out_of_bounds+0x4006d8) 0x0000004111f4 is located 4 bytes to the right of global variable 'global_array' defined in 'global_out_of_bounds.cpp:1:5' (0x411060) of size 400 SUMMARY: AddressSanitizer: global-buffer-overflow /root/asan/global_out_of_bounds.cpp:3 in main Shadow bytes around the buggy address: 0x0010000821e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0010000821f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001000082200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001000082210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001000082220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x001000082230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00[f9]f9 0x001000082240: f9 f9 f9 f9 00 00 f9 f9 f9 f9 f9 f9 f9 f9 00 00 0x001000082250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001000082260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001000082270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001000082280: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
通过上面报错信息可以看出
[0x411060, 0x4111f0)
影子区域内存转换如下
# python3 asan_shadow_addr.py 0x0000004111f4 shadow_addr:0x10000008223e =>0x001000082230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00[f9]f9 Global redzone: f9
对应代码验证确实是访问了全局变量之外的第4-8字节。
return global_array[1 + 100];
这里再多提一个细节,并不是所有的内存问题asan都能检测,我们留意影子区域布局如下
=>0x001000082230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00[f9]f9 0x001000082240: f9 f9 f9 f9 00 00 f9 f9 f9 f9 f9 f9 f9 f9 00 00
可以看到在0x001000082244-0x001000082245两个影子字节处是0,那么意味着如果全局变量溢出的问题在这对应的16字节,asan将不起作用,我们修改代码验证。
我们计算当前错误离为0的影子字节个数为5,那么需要额外越界 5*8/4+1
.此时原来的越界数组值是101,我们加上刚刚计算的11,得出112。那么代码如下
return global_array[112];
编译验证,发现asan检测不出这个溢出问题了。
测试代码如下
int *ptr; __attribute__((noinline)) void FunctionThatEscapesLocalObject() { int local[100]; ptr = &local[0]; } int main(int argc, char **argv) { FunctionThatEscapesLocalObject(); return ptr[argc]; }
值得注意的是:
ASAN_OPTIONS=detect_stack_use_after_return=1
此时运行
# ASAN_OPTIONS=detect_stack_use_after_return=1 ./run.sh ./use_after_return ==115526==ERROR: AddressSanitizer: stack-use-after-return on address 0x007f9e1e6034 at pc 0x000000400b24 bp 0x007fee1358f0 sp 0x007fee135910 READ of size 4 at 0x007f9e1e6034 thread T0 #0 0x400b20 in main /root/asan/use_after_return.cpp:18 #1 0x7fa192cd8c in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x20d8c) #2 0x4008b8 (/root/asan/use_after_return+0x4008b8) Address 0x007f9e1e6034 is located in stack of thread T0 at offset 52 in frame #0 0x400988 in FunctionThatEscapesLocalObject() /root/asan/use_after_return.cpp:11 This frame has 1 object(s): [48, 448) 'local' (line 12) <== Memory access at offset 52 is inside this variable HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork (longjmp and C++ exceptions *are* supported) SUMMARY: AddressSanitizer: stack-use-after-return /root/asan/use_after_return.cpp:18 in main Shadow bytes around the buggy address: 0x001ff3c3cbb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ff3c3cbc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ff3c3cbd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ff3c3cbe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ff3c3cbf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x001ff3c3cc00: f5 f5 f5 f5 f5 f5[f5]f5 f5 f5 f5 f5 f5 f5 f5 f5 0x001ff3c3cc10: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 0x001ff3c3cc20: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 0x001ff3c3cc30: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 0x001ff3c3cc40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ff3c3cc50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
通过上面报错信息可以看出
local[1]
影子区域内存转换如下
# python3 asan_shadow_addr.py 0x007f9e1e6034 shadow_addr:0x100ff3c3cc06 0x001ff3c3cbf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x001ff3c3cc00: f5 f5 f5 f5 f5 f5[f5]f5 f5 f5 f5 f5 f5 f5 f5 f5 Stack after return: f5
根据上面的信息,有个细节值得注意的是
下面演示asan无法检测的问题,我们修改代码,根据推算我们需要访问推前 -6*8/4+1
的地址,那么下毒就没有意义了。
那么修改代码如下
return ptr[-13];
此时运行可以发现asan检测不到问题
测试代码如下
volatile int *p = 0; int main() { { int x = 0; p = &x; } *p = 5; return 0; }
运行
# ./run.sh ./use_after_scope ================================================================= ==117169==ERROR: AddressSanitizer: stack-use-after-scope on address 0x007fd2d42760 at pc 0x000000400a88 bp 0x007fd2d426f0 sp 0x007fd2d42710 WRITE of size 4 at 0x007fd2d42760 thread T0 #0 0x400a84 in main /root/asan/use_after_scope.cpp:15 #1 0x7fb7b40d8c in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x20d8c) #2 0x400858 (/root/asan/use_after_scope+0x400858) Address 0x007fd2d42760 is located in stack of thread T0 at offset 32 in frame #0 0x400928 in main /root/asan/use_after_scope.cpp:10 This frame has 1 object(s): [32, 36) 'x' (line 12) <== Memory access at offset 32 is inside this variable HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork (longjmp and C++ exceptions *are* supported) SUMMARY: AddressSanitizer: stack-use-after-scope /root/asan/use_after_scope.cpp:15 in main Shadow bytes around the buggy address: 0x001ffa5a8490: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ffa5a84a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ffa5a84b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ffa5a84c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ffa5a84d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x001ffa5a84e0: 00 00 00 00 00 00 00 00 f1 f1 f1 f1[f8]f3 f3 f3 0x001ffa5a84f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ffa5a8500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ffa5a8510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ffa5a8520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001ffa5a8530: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
报错信息如下
那么就是在x的作用域之外访问了x
影子区域内存转换如下
# python3 asan_shadow_addr.py 0x007fd2d42760 shadow_addr:0x100ffa5a84ec =>0x001ffa5a84e0: 00 00 00 00 00 00 00 00 f1 f1 f1 f1[f8]f3 f3 f3 Stack use after scope: f8
这里f8的位置的0-4个字节就是对应实际内存的x的位置
#include <stdlib.h> void *p; int main() { p = malloc(7); p = 0; // The memory is leaked here. return 0; }
上面代码分配了7个字节给p,然后对指针p赋予空指针,这样代码就存在7字节的内存泄漏,运行如下
# ./run.sh ./memory-leak ================================================================= ==118406==ERROR: LeakSanitizer: detected memory leaks Direct leak of 7 byte(s) in 1 object(s) allocated from: #0 0x7f8ee4025c in __interceptor_malloc (/usr/lib/aarch64-linux-gnu/libasan.so.5.0.0+0xed25c) #1 0x400798 in main /root/asan/memory-leak.cpp:6 #2 0x7f8ebc5d8c in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x20d8c) #3 0x4006c8 (/root/asan/memory-leak+0x4006c8) SUMMARY: AddressSanitizer: 7 byte(s) leaked in 1 allocation(s).
这个堆栈浅显易懂,就不解释了。
本文根据官方的文档,将几种asan定位内存问题进行了上手实验,能够加深对asan工具的使用细节的理解。但故意遗漏了关于全局对象构造顺序相关的问题讨论,这块问题后面会专门提,就不在此篇幅解释了。
单例的目的是保证一个类只有一个实例,这样代码在调用的时候不需要频繁的初始化,但是在调试的过程中,单例可能出现因为多线程访问导致的概率性bug,本文主要讨论单例出现的这种问题
为了支持单例,最明显的代码就是实现一个getInstance函数,这个函数中在第一次访问的时候new自己的对象,之后所有的调用通过getInstance来调用类的函数,代码示例如下:
GlobalConfig *GlobalConfig::getInstance() { if(instance == nullptr) // a instance = new GlobalConfig; // b return instance; }
对于单例,如果在单线程环境下,上述代码不会有任何问题,如果instance是nullptr,则新建一个,如果不是,则返回这个instance。
但是如果是多线程的情况,满足instance都没有创建的情况下,两个线程同时调用了getInstance函数,那么可能出现:
第二个线程判断了instance此时还是nullptr,所以再一次构造了GlobalConfig,这样子我们对这个单例构造了两次。这是不必要的开销。
为了解决多线程环境下出现的单例构造两次的问题,我们可以对这段代码加锁,如下
GlobalConfig *GlobalConfig::getInstance() { LOCK(); if(instance == nullptr) // a instance = new GlobalConfig; // b UNLOCK(); return instance; }
此时我们通过锁解决了上面提到的单例构造两次的问题,因为当第一个线程正在执行时会上锁,这样第二个线程就不会重入代码内部。因为锁也是原子的,所以我们似乎解决了单例构造两次的问题。
但是,上面代码会引入新的问题,我们知道getInstance函数在99%的情况下instance是有值的,而在1%的情况下是需要构造的,这也就导致了这个锁在99%的情况下是不需要的。那么怎么优化这个问题呢?
对于上面的代码,我们需要解决锁带来的性能开销问题,因为我们不能因为1%的场景去让代码执行99%的任务。所以我们需要进行double-check,那么具体是什么样的呢,如下代码
GlobalConfig *GlobalConfig::getInstance() { if(instance == nullptr) { LOCK(); if(instance == nullptr) // a instance = new GlobalConfig; // b UNLOCK(); } return instance; }
这段代码通过重复检测instance的值,从而避免了99%的不必要加锁操作。它看似会正常工作了。
但是我们对构造GlobalConfig这个类需要再分析一下,它主要做如下事情
如果上面步骤1,2,3都是顺序执行的,那么这一切都是正常的,那么假设步骤2和步骤3的顺序颠倒呢?
那么程序是否崩溃就取决于是否访问到没有权限不可访问的区域了,如果访问到了,就崩溃,如果不是,那么可能修改错误的值,或者存在了一个隐藏的bug。
根据上面提到的,步骤2和步骤3在体系架构中真会出现,这主要原因是cpu的乱序执行。
我们知道一条指令在cpu的流水线执行包含如下几个部分
取指---译码---执行---访存---写回
这些步骤是顺序的,但是一般来说CPU会有多级流水线,这样就会出现如下情况。以三级流水线为例
取指---译码---执行---访存---写回 取指---译码---执行---访存---NOP 取指---译码---执行---NOP---写回
而且,我们知道CPU访问内存的速度是慢于CPU自身时钟频率的,所以在取指和访存以及写回三个步骤上会比较慢。所以为了加快CPU的指令执行,CPU乱序执行的就出现了。其含义是:
对于上面的例子,我们知道步骤2是调用构造函数,步骤3是将内存地址返回,对于它们而言,其执行是理论可以乱序的,所以就可以出现运行步骤是 1-->3-->2
这样的现象。
根据上面提到的CPU乱序问题,我们也可以很好解决,那就是让两行代码出现数据依赖就行了,那么代码如下
GlobalConfig *GlobalConfig::getInstance() { if(instance == nullptr) { LOCK(); if(instance == nullptr) // a GlobalConfig* temp = new GlobalConfig; // b instance = temp //c UNLOCK(); } return instance; }
从代码的第b行和第c行看,我们将构造的对象赋值给了temp,然后再将temp传递给instance。它能够完美规避CPU乱序的问题。
那还有什么问题没有呢? 实则不然,我们知道编译器会将指令乱序,这段代码仍可能出现乱序执行的问题。
虽然我们在代码b和代码c行特地构造了数据依赖,让我们人眼直观来看CPU不会针对这行指令进行乱序了,但是我们知道通常情况下代码编译成汇编的时候,编译器会优化代码,如果关闭优化如O(0),则代码执行效率低,所以通常编译默认的优化等级是O(2)。在默认情况下,上述代码编译器会认为temp是没有意义的变量,故合并两行代码。
这样,我们看似自行做的优化,实际上在编译阶段就已经被编译器进行了负优化了。那么我们只能寻找其他办法。
barrier的含义就是栅栏的意思,它同样适用于编译器,在aarch64中,有三种barrier指令,如下
对于上述的问题,我们知道编译器也会执行乱序,所以为了让编译器不做乱序优化,我们在其数据依赖的中间添加任意的barrier指令,避免编译器的错误优化,代码如下
GlobalConfig *GlobalConfig::getInstance() { if(instance == nullptr) { LOCK(); if(instance == nullptr) // a GlobalConfig* temp = new GlobalConfig; // b barrier(); //d instance = temp //c UNLOCK(); } return instance; }
如上,我们添加了第d行,这个barrier可以是任意的barrier指令,例如DMB,它可以使得编译器不进行乱序优化。这样就真正的解决了单例在多线程中的所有问题。
根据上面的介绍,我们为了解决单例在多线程访问中并发的问题做了很多的思考,使得代码看起来非常的乱,甚至是过度优化,对于这种情况,实际上还有最优解。那就是资源最早初始化。
这里表达的意思就是:
如果把getInstance函数放在初始化的单线程中,这样所有的多线程访问都不可能进入instance == nullptr
的逻辑内。
如果就将约束程序员在代码的第一次进入getInstance时是单线程的,那或许比较困难(谁知道其他程序员怎么想~),那么另一种方法或许风险更小,就是在每个线程开始的时候,主动调用如下
GlobalConfig* const instance = GlobalConfig::getInstance();
这样这个instance的值引用的instance并且保存在cache中。从而使得后面调用所有的getInstance都不需要考虑多线程重入的问题。
本文基于线程安全的角度介绍了单例在多线程中的bug,如果我们为了解决一个一个多线程重入导致的问题,那么代码可读性就会很差,并且容易出现负优化,如果我们在这种情况下,对每个线程的最开始主动增加一次getInstance调用,后面的调用实际上利用了cache,那么反而其整体性能最高。
https://www.aristeia.com/Papers/DDJ_Jul_Aug_2004_revised.pdf
在处理内存泄漏问题上,我们有个bug就是循环播放14天视频,会出现内存上升大概20M的样子,也就是这个问题,发现了QThread的隐藏问题:如果QThread完成之后,不显式的连接finished信号并执行deleteLater,那么QThread就会造成内存泄漏。这个内存泄漏用任意的内存泄漏检测工具都可以轻易的检测出来。关于本问题,连续播放视频一周,那么意味着创建了无数个QThread结构实例,但是没有被回收。
本文基于asan获取的堆栈如下
Indirect leak of 272000 byte(s) in 1700 object(s) allocated from: #0 0x7f83cc0d5c in operator new(unsigned long) (/usr/lib/aarch64-linux-gnu/libasan.so.5.0.0+0xeed5c) #1 0x7f7fb9e89c in QThread::QThread(QObject*) (/lib/aarch64-linux-gnu/libQt5Core.so.5+0xa189c) #2 0x7f7fb9ea60 in QThread::createThreadImpl(std::future<void>&&) (/lib/aarch64-linux-gnu/libQt5Core.so.5+0xa1a60) #3 0x5564d9e098 in create<MpvCore::LoadFileInfo()::<lambda()> > /usr/include/aarch64-linux-gnu/qt5/QtCore/qthread.h:235 #4 0x5564d9e098 in MpvCore::LoadFileInfo() core/mpvcore.cpp:1163 #5 0x5564da679c in MpvCore::event(QEvent*) core/mpvcore.cpp:1670 #6 0x7f809088e8 in QApplicationPrivate::notify_helper(QObject*, QEvent*) (/lib/aarch64-linux-gnu/libQt5Widgets.so.5+0x15e8e8) #7 0x7f80911eec in QApplication::notify(QObject*, QEvent*) (/lib/aarch64-linux-gnu/libQt5Widgets.so.5+0x167eec) #8 0x7f7fd6a9c0 in QCoreApplication::notifyInternal2(QObject*, QEvent*) (/lib/aarch64-linux-gnu/libQt5Core.so.5+0x26d9c0) #9 0x7f7fd6d8d4 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) (/lib/aarch64-linux-gnu/libQt5Core.so.5+0x2708d4) #10 0x7f7fdc89c4 (/lib/aarch64-linux-gnu/libQt5Core.so.5+0x2cb9c4) #11 0x7f80f44708 in g_main_context_dispatch (/lib/aarch64-linux-gnu/libglib-2.0.so.0+0x51708) #12 0x7f80f44974 (/lib/aarch64-linux-gnu/libglib-2.0.so.0+0x51974) #13 0x7f80f44a18 in g_main_context_iteration (/lib/aarch64-linux-gnu/libglib-2.0.so.0+0x51a18) #14 0x7f7fdc7e70 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) (/lib/aarch64-linux-gnu/libQt5Core.so.5+0x2cae70) #15 0x7f7fd6916c in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) (/lib/aarch64-linux-gnu/libQt5Core.so.5+0x26c16c) #16 0x7f7fd71770 in QCoreApplication::exec() (/lib/aarch64-linux-gnu/libQt5Core.so.5+0x274770) #17 0x5564d7ed60 in main src/main.cpp:68 #18 0x7f7f690d8c in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x20d8c) #19 0x5564d886a4 (/home/kylin/kylin-video+0x686a4)
根据上面的堆栈,我们看到泄漏存在于QThread的构造函数中。我们翻一下qtbase的源码
class QThreadCreateThread : public QThread { public: explicit QThreadCreateThread(std::future<void> &&future) : m_future(std::move(future)) { } private: void run() override { m_future.get(); } std::future<void> m_future; }; QThread *QThread::createThreadImpl(std::future<void> &&future) { return new QThreadCreateThread(std::move(future)); } QThread::QThread(QObject *parent) : QObject(*(new QThreadPrivate), parent) { Q_D(QThread); // fprintf(stderr, "QThreadData %p created for thread %p\n", d->data, this); d->data->thread = this; }
可以很明显的看到QThread执行了new操作,和asan给出的报告相符合。
我们参照qt的文档如下
When any QObject in the tree is deleted, if the object has a parent, the destructor automatically removes the object from its parent
如果QObject对象有父对象,那么会在父对象中析构所有的成员
但是我们要知道的是QThread继承于QObject,但是没有父对象
qt的文档隐晦的提示你要调用QObject::deleteLater()。文章链接如下
void QThread::finished() This signal is emitted from the associated thread right before it finishes executing. When this signal is emitted, the event loop has already stopped running. No more events will be processed in the thread, except for deferred deletion events. This signal can be connected to QObject::deleteLater(), to free objects in that thread.
根据上面的信息,我们可以总结出来,QThread创建的线程,如果没有父对象,那么需要显式的连接finished信号来执行deleteLater,否则就需要将其绑定到一个对象上。所以解决问题的方法有两种:
QThread* thread = new QThread; Worker* worker = new Worker; worker->setParent(thread); worker->moveToThread(thread);
这种情况下,其对象是否析构取决于worker的管理。
QThread::connect(thread, &QThread::finished, thread, &QThread::deleteLater);
一般QThread不会频繁的绑定对象父子关系,这样不容易管理,所以对于父子关系不清晰的代码,建议直接连接finished即可
对于bug而言,我解决此问题的方法是连接finished,为了测试,下面提供了复现这个问题的测试代码
#include <QThread> #include <QDebug> int main(int argc, char *argv[]) { Q_UNUSED(argc); Q_UNUSED(argv); for (int i = 0; i < 100; ++i) { QThread* thread = QThread::create([i]{ qDebug() << "Thread Started:" << i; }); thread->start(); // QThread::connect(thread, &QThread::finished, thread, &QThread::deleteLater); } QThread::sleep(3); return 0; }
我创建100个线程,如果不连接finished信号调用deleteLater,那么可以很明显的报告内存泄漏,如果添加信号则代码正常。
================================================================= ==52273==ERROR: LeakSanitizer: detected memory leaks Indirect leak of 16000 byte(s) in 100 object(s) allocated from: #0 0x7fbcb04d5c in operator new(unsigned long) (/usr/lib/aarch64-linux-gnu/libasan.so.5.0.0+0xeed5c) #1 0x7fbbfce89c in QThread::QThread(QObject*) (/lib/aarch64-linux-gnu/libQt5Core.so.5+0xa189c) #2 0x7fbbfcea60 in QThread::createThreadImpl(std::future<void>&&) (/lib/aarch64-linux-gnu/libQt5Core.so.5+0xa1a60) #3 0x55855c38b4 in create<main(int, char**)::<lambda()> > /usr/include/aarch64-linux-gnu/qt5/QtCore/qthread.h:199 #4 0x55855c38b4 in main /tmp/qthread/test.cpp:9 #5 0x7fbbac2d8c in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x20d8c) #6 0x55855c41e0 (/root/test+0x41e0) Indirect leak of 12000 byte(s) in 100 object(s) allocated from: #0 0x7fbcb04d5c in operator new(unsigned long) (/usr/lib/aarch64-linux-gnu/libasan.so.5.0.0+0xeed5c) #1 0x7fbbfce8fc in QThread::QThread(QObject*) (/lib/aarch64-linux-gnu/libQt5Core.so.5+0xa18fc) #2 0x7fbbfcea60 in QThread::createThreadImpl(std::future<void>&&) (/lib/aarch64-linux-gnu/libQt5Core.so.5+0xa1a60) #3 0x55855c38b4 in create<main(int, char**)::<lambda()> > /usr/include/aarch64-linux-gnu/qt5/QtCore/qthread.h:199 #4 0x55855c38b4 in main /tmp/qthread/test.cpp:9 #5 0x7fbbac2d8c in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x20d8c) #6 0x55855c41e0 (/root/test+0x41e0) Indirect leak of 10400 byte(s) in 100 object(s) allocated from: #0 0x7fbcb04d5c in operator new(unsigned long) (/usr/lib/aarch64-linux-gnu/libasan.so.5.0.0+0xeed5c) #1 0x7fbbfd6e40 in QWaitCondition::QWaitCondition() (/lib/aarch64-linux-gnu/libQt5Core.so.5+0xa9e40) #2 0x7fbbfce8f0 in QThread::QThread(QObject*) (/lib/aarch64-linux-gnu/libQt5Core.so.5+0xa18f0) #3 0x7fbbfcea60 in QThread::createThreadImpl(std::future<void>&&) (/lib/aarch64-linux-gnu/libQt5Core.so.5+0xa1a60) #4 0x55855c38b4 in create<main(int, char**)::<lambda()> > /usr/include/aarch64-linux-gnu/qt5/QtCore/qthread.h:199 #5 0x55855c38b4 in main /tmp/qthread/test.cpp:9 #6 0x7fbbac2d8c in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x20d8c) #7 0x55855c41e0 (/root/test+0x41e0) Indirect leak of 3200 byte(s) in 100 object(s) allocated from: #0 0x7fbcb04d5c in operator new(unsigned long) (/usr/lib/aarch64-linux-gnu/libasan.so.5.0.0+0xeed5c) #1 0x7fbbfcea54 in QThread::createThreadImpl(std::future<void>&&) (/lib/aarch64-linux-gnu/libQt5Core.so.5+0xa1a54) #2 0x55855c38b4 in create<main(int, char**)::<lambda()> > /usr/include/aarch64-linux-gnu/qt5/QtCore/qthread.h:199 #3 0x55855c38b4 in main /tmp/qthread/test.cpp:9 #4 0x7fbbac2d8c in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x20d8c) #5 0x55855c41e0 (/root/test+0x41e0) SUMMARY: AddressSanitizer: 41600 byte(s) leaked in 400 allocation(s).
最近在调试程序内存的问题时,通过asan挂上内存监听,遇到了一个stack-use-after-scope的错误,asan调试其他问题的介绍后面有空再介绍,本文通过asan发现了C++一个关于生命周期的bug,它会导致程序错误的使用栈。问题比较经典,故特定分享一下。
在使用 播放器播放视频的时候,如果不断的移动播放器进度条,有非常小的概率导致移动失败。
再加上调试过程中,移动进度条播放时,相关代码存在内存泄漏,故相应代码部位添加了asan用于检测。
开启asan的方式简单介绍如下
QMAKE_CXXFLAGS += -fsanitize=address -g QMAKE_LFLAGS += -fsanitize=address
然后挂上asan的库运行即可
LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libasan.so.5.0.0 ./kylin-video
其他问题我们先不关注,这里只关注本文章讨论的stack-use-after-scope
,故asan检测到的错如下
==88163==ERROR: AddressSanitizer: stack-use-after-scope on address 0x007ffe54b9b0 at pc 0x007f99a74504 bp 0x007ffe54ab40 sp 0x007ffe54abb8 READ of size 3 at 0x007ffe54b9b0 thread T0 #0 0x7f99a74500 (/usr/lib/aarch64-linux-gnu/libasan.so.5.0.0+0x64500) #1 0x7f9753d408 in bstr0 ../misc/bstr.h:61 #2 0x7f9753d408 in set_node_arg ../input/cmd.c:179 #3 0x7f9753dfd8 in cmd_node_array ../input/cmd.c:227 #4 0x7f9753dfd8 in mp_input_parse_cmd_node ../input/cmd.c:301 #5 0x7f9753e638 in mp_input_parse_cmd_strv ../input/cmd.c:502 #6 0x7f9755f2ec in mpv_command_async ../player/client.c:1196 #7 0x557793118c in MpvCore::Seek(int, bool, bool) core/mpvcore.cpp:588 #8 0x5577932a74 in operator() core/mpvcore.cpp:1330 #9 0x5577932a74 in call /usr/include/aarch64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:146 #10 0x5577932a74 in call<QtPrivate::List<int>, void> /usr/include/aarch64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:256 #11 0x5577932a74 in impl /usr/include/aarch64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:439 #12 0x5577932a74 in impl /usr/include/aarch64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:432 #13 0x7f95bd79e4 in QMetaObject::activate(QObject*, int, int, void**) (/lib/aarch64-linux-gnu/libQt5Core.so.5+0x29c9e4) #14 0x5577b02500 in GlobalUserSignal::sigSeek(int) .moc/moc_globalsignal.cpp:1047 #15 0x55779e1174 in GlobalUserSignal::seek(int) global/globalsignal.h:47 #16 0x55779e1174 in operator() widget/contralbar.cpp:647 #17 0x55779e1174 in call /usr/include/aarch64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:146 #18 0x55779e1174 in call<QtPrivate::List<>, void> /usr/include/aarch64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:256 #19 0x55779e1174 in impl /usr/include/aarch64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:439 #20 0x7f95be52b8 (/lib/aarch64-linux-gnu/libQt5Core.so.5+0x2aa2b8) #21 0x7f95bd8264 in QObject::event(QEvent*) (/lib/aarch64-linux-gnu/libQt5Core.so.5+0x29d264) #22 0x7f967468e8 in QApplicationPrivate::notify_helper(QObject*, QEvent*) (/lib/aarch64-linux-gnu/libQt5Widgets.so.5+0x15e8e8) #23 0x7f9674feec in QApplication::notify(QObject*, QEvent*) (/lib/aarch64-linux-gnu/libQt5Widgets.so.5+0x167eec) #24 0x7f95ba89c0 in QCoreApplication::notifyInternal2(QObject*, QEvent*) (/lib/aarch64-linux-gnu/libQt5Core.so.5+0x26d9c0) #25 0x7f95c051ac in QTimerInfoList::activateTimers() (/lib/aarch64-linux-gnu/libQt5Core.so.5+0x2ca1ac) #26 0x7f95c05adc (/lib/aarch64-linux-gnu/libQt5Core.so.5+0x2caadc) #27 0x7f96d82708 in g_main_context_dispatch (/lib/aarch64-linux-gnu/libglib-2.0.so.0+0x51708) #28 0x7f96d82974 (/lib/aarch64-linux-gnu/libglib-2.0.so.0+0x51974) #29 0x7f96d82a18 in g_main_context_iteration (/lib/aarch64-linux-gnu/libglib-2.0.so.0+0x51a18) #30 0x7f95c05e70 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) (/lib/aarch64-linux-gnu/libQt5Core.so.5+0x2cae70) #31 0x7f95ba716c in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) (/lib/aarch64-linux-gnu/libQt5Core.so.5+0x26c16c) #32 0x7f95baf770 in QCoreApplication::exec() (/lib/aarch64-linux-gnu/libQt5Core.so.5+0x274770) #33 0x55778f3dc0 in main src/main.cpp:68 #34 0x7f954ced8c in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x20d8c) #35 0x55778ff55c (/home/kylin/kylin-video+0xdf55c) Address 0x007ffe54b9b0 is located in stack of thread T0 at offset 928 in frame #0 0x557793040c in MpvCore::Seek(int, bool, bool) core/mpvcore.cpp:552 This frame has 28 object(s): [48, 49) '<unknown>' [64, 65) '<unknown>' [80, 81) '<unknown>' [96, 97) '<unknown>' [112, 113) '<unknown>' [128, 129) '<unknown>' [144, 145) '<unknown>' [160, 161) '<unknown>' [176, 184) '<unknown>' [208, 216) '<unknown>' [240, 248) '<unknown>' [272, 280) 'tmp' (line 565) [304, 312) '<unknown>' [336, 344) '<unknown>' [368, 376) '<unknown>' [400, 408) '<unknown>' [432, 440) '<unknown>' [464, 472) '<unknown>' [496, 504) '__dnew' [528, 536) '<unknown>' [560, 568) '__dnew' [592, 624) '<unknown>' [656, 680) 'args' (line 573) [720, 752) 'args' (line 568) [784, 816) '<unknown>' [848, 880) 'args' (line 587) [912, 944) '<unknown>' <== Memory access at offset 928 is inside this variable [976, 1016) 'args' (line 581) HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork (longjmp and C++ exceptions *are* supported) SUMMARY: AddressSanitizer: stack-use-after-scope (/usr/lib/aarch64-linux-gnu/libasan.so.5.0.0+0x64500) Shadow bytes around the buggy address: 0x001fffca96e0: f8 f2 00 00 00 f2 00 00 00 f2 00 00 00 f2 00 00 0x001fffca96f0: 00 f2 00 00 00 f2 00 00 f8 f2 00 00 00 f2 00 00 0x001fffca9700: 00 f2 00 00 f8 f2 00 00 f8 f2 f2 f2 f8 f8 f8 f8 0x001fffca9710: f2 f2 f2 f2 00 00 00 f2 f2 f2 f2 f2 00 00 00 00 0x001fffca9720: f2 f2 f2 f2 00 00 00 00 f2 f2 f2 f2 00 00 00 00 =>0x001fffca9730: f2 f2 f2 f2 f8 f8[f8]f8 f2 f2 f2 f2 00 00 00 00 0x001fffca9740: 00 f3 f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00 0x001fffca9750: 00 00 00 00 00 00 f1 f1 f1 f1 00 f3 f3 f3 00 00 0x001fffca9760: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001fffca9770: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001fffca9780: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==88163==ABORTING
关于asan的一下解释后面更新,这里通过错误我们看到了stack-use-after-scope
,这个什么意思呢?如下
通俗点就是,访问了一个作用域之外的栈变量。
此时,通过反汇编代码,可以看到真正存在问题的汇编如下
0x0000007ff74133f8 <+536>: ldr x23, [x23]
可以看到这是一个取值操作,根据汇编上下文和代码上下文分析,其代码如下
int r = m_option_parse(log, opt, bstr0(cmd->name), bstr0(val->u.string), dst);
对于取值,我们可以看到val->u.string
,随机可以通过gdb打印相关值,如下
(gdb) p val->u.string $6 = 0x7fffffe5b8 "32"
可以看到,这里ldr就是把0x7fffffe5b8里面的值32取走了,符合bstr0(val->u.string)的逻辑。我们看看程序的栈区范围如下
kylin@kylin:~$ cat /proc/198018/maps | grep stack 7ffffdf000-8000000000 rw-p 00000000 00:00 0 [stack]
可以看到0x7fffffe5b8就是栈区的变量。
我们知道问题是播放器点击进度条的时候概率出现的,所以我们回到播放器的代码分析
void MpvCore::Seek(int pos, bool relative, bool osd) const char *args[] = {"seek", QString::number(pos).toStdString().c_str(), "absolute", NULL}; mpv_command_async(m_mpvHandle, MPV_REPLY_COMMAND, args); }
这里32的字符串的值就是QString::number(pos).toStdString().c_str()
传递进去的。
我们还需要留意一个细节,那就是mpv_command_async
,稍微跟踪一下代码如下
mpv_command_async run_async_cmd run_async mp_dispatch_enqueue
到这里我们可以清楚了,mpv的控制是异步接口,默认的命令通过队列管理,异步触发。
根据上面的代码分析,调试和日志,我们知道了这是在一个异步请求上出现的访问了作用域之外的栈变量。
也就是说,args的构造中使用了栈变量,它标记销毁了,所以我们重点看args是如何构造的
const char *args[] = {"seek", QString::number(pos).toStdString().c_str(), "absolute", NULL};
我们可以知道,"seek" "absolute" 都是只读数据区,那么唯一出现在0x7fffffe5b8位置,也就是栈区的值就是QString::number(pos).toStdString().c_str()
。 那么我们可以得出结论
为了解决这个问题,我们就要从c++关于prvalue(pure rvalue:纯右值),也就是临时变量的生命周期谈起,相关文章如下,有兴趣的可以直接翻阅
根据上面文献的描述,我们在如下情况下可以延长生命周期
根据上面的描述,可以选用如下两种来解决此问题
其他的并不符合当前代码上下文状态。
std::string&& str = QString::number(pos).toStdString(); const char *args[] = {"seek", str.c_str(), "absolute", NULL};
这里 &&
是使用右值引用,它等效于const情况下的左值引用,如下
const std::string& str = QString::number(pos).toStdString(); const char *args[] = {"seek", str.c_str(), "absolute", NULL};
这样可以实现对c_str()这个栈区的变量进行延长生命周期
std::string str = QString::number(pos).toStdString(); const char *args[] = {"seek", str.c_str(), "absolute", NULL};
这里直接初始化了一个str,对于args的传递,我们使用初始化完成的str来访问c_str(),这样的方式延长了其生命周期。
至此,我们通过分析了c++关于临时变量延长生命周期的方式,解决了一个概率很低的bug(隐秘的bug),此bug一般不会出现问题,但长时间运行和压力测试就会出现错误,通过前期的分析和借助asan工具,可以很好的地位此问题。
另一方面,此问题是关于c++的string类型的生命周期问题,c_str()是当时栈区的临时变量,QString::number(pos).toStdString()
如果和c_str()
一起写,也就是QString::number(pos).toStdString().c_str()
,那么这个string作为临时变量,它的生命周期本应该在;
结束,所以c_str()指向的栈区会在作用域之外失效,为了延长其生命周期,我们通过右值引用和带const的左值引用,以及完全初始化string变量都可以延长其生命周期。
所谓延长其生命周期,也就是使得string有效访问,直到函数传递后不再访问string类型变量为止,由c++定义回收。否则,其string类型变量使用之后,内容再回收,异步的程序如果再次访问此栈区,可能会在某种情况下,被应用程序某个行为进行修改,从而导致此值的错误,然后从隐秘的bug变成实际出现的bug。
在工作中,通常我们身上挂着几十上百个bug,对应着十多个设备机器,如果每次都是人走过去调试非常不方便。如果是非界面的问题,我们可以通过ssh远程,如果是界面问题,我们只能通过vnc远程连接,但是vnc远程不是很方便,本文介绍一个更常用和方便的方法,novnc。基于此加速我们调试问题的节奏
安装主要是安装Vino和novnc,默认机器可能已经预装了,对于没有预装的版本,可以如下
apt install novnc vino websockify
当然,如不想使用vino,那么tightvncserver也可以,但是本人更常用vino,本文介绍也通过vino
apt install tightvncserver
对于vnc,我们需要默认打开一下配置,我们可以在控制面板里面选,那太麻烦了,人还是得过去一趟
所以下面介绍直接配置
ssh kylin@localhost gsettings set org.ukui.control-center vino-enabled true gsettings set org.gnome.Vino prompt-enabled false vncpasswd systemctl --user restart vino-server
上面配置就是打开了vnc的配置,这样远程可以通过vnc viewer进行连接,例如如下
接下来配置novnc就很简单了
kylin@kylin:~$ /usr/share/novnc/utils/launch.sh Using installed websockify at /usr/bin/websockify Starting webserver and WebSockets proxy on port 6080 WebSocket server settings: - Listen on :6080 - Web server. Web root: /usr/share/novnc - No SSL/TLS support (no cert file) - proxying from :6080 to localhost:5900
然后到自己编程的机器上,打开浏览器,登录网站http://${ip}:6080/vnc.html
就出现如下画面
点击连接,输入密码
浏览器正常登录vnc
多个机器就是多个浏览器网页页面而已。
这里介绍了我平时多个bug同时调试的一种方法,用来提高自己解决问题效率。