捉虫日记 0017:cavium octeon cn68xx mount rootfs failed
1 Phenomenon
环境:
- Cavium CN68xx
- WindRiver Linux 5.0
Linux mainline 尚未有 CN68XX 的支持,从 wrlinux 4.3 移植了大部分代码,启动到 mount rootfs 时,失败
NFS 为 rootfs 时:
VFS: Mounted root (nfs filesystem) on device 0:12. devtmpfs: mounted Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
CF 卡为 rootfs 时:
EXT3-fs (sda1): using internal journal EXT3-fs (sda1): recovery complete EXT3-fs (sda1): mounted filesystem with writeback data mode VFS: Mounted root (ext3 filesystem) on device 8:1. devtmpfs: mounted Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
2 Analysis
不太清楚具体的出错地点,于是在 init/main.c 里加了一些打印:
printk("<0> line %d @ %s File %s\n", __LINE__, __FUNCTION__, __FILE__);
很快定位到出错函数在 run_init_process("/sbin/init"); 这个函数最终会调用 sys_execve -> do_execve 生成第一个进程 init
有 kernel panic,没有 call trace,不像是真的 kernel panic,似乎是 init 运行失败,被内核 kill 了,比较困惑
于是就逆向找,在函数 panic() 里,printk("Kernel panic - not syncing: ...) 前加了 dump_stack(),在 do_notify_resume 里加了 show_register(); dump_tlb_all() 把出错时的上下文打印出来了:
Cpu 0 $ 0 : 0000000000000000 0000000000000001 00000000000906a2 00000000386f1980 $ 4 : 000000007ffcd5f0 0000000000000000 0000000010039ba0 0000000000000000 $ 8 : 00000000386f1980 000000000ffffffe 000000007ffcd700 000000002f2f2f2f $12 : 000000007ffcdfb7 000000007ffcdfb5 ffffffffffffff18 0000000010040000 $16 : 0000000000000000 0000000000000000 0000000000000000 ffffffff81880000 $20 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 $24 : 0000000010039894 00000000385ae400 $28 : 00000000386f1980 000000007ffcd5f0 0000000000000000 0000000038596ec4 Hi : 0000000000000007 Lo : 0000000000000000 epc : 0000000038596ed0 0x38596ed0 Not tainted ra : 0000000038596ec4 0x38596ec4 Status: 00009cf3 KX SX UX USER EXL IE Cause : 00800008 BadVA : 0000000000090722 PrId : 000d9108 (Cavium Octeon II) Modules linked in: Process init (pid: 1, threadinfo=a80000003184c000, task=a800000031850000, tls=00000000771284a0) Stack : 0000000038596ec4 000000007ffcd5f0 0000000000000000 0000000000000000 0000000000000000 ffffffff81880000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 00000000386f1980 0000000000000000 fffffffffffffffc 0000000077122000 0000000000000003 000000007ffcd704 000000007ffcd714 0000000038576010 000000003854f8d4 0000000038576010 0000000010003470 7ffcd70400000003 1000527881880000 0000000010004c98 0000000000000000 0000000000000000 0000000038576010 0000000010003470 0000000038576010 ffffffff81880000 0000000000000000 0000000010041ae0 00000000100034b0 000000037ffcdfad 7ffcdfb87ffcdfc5 000000007ffcdfd0 7ffcdfd77ffcdfe2 0000000000000010 0000000000000006 ... Call Trace: (Bad stack address) Code: 1440002a 8f8692f0 27428b60 <8c430080> 8fa400b4 8fa500b0 afa300a0 8c43007c 8cc60000 Index: 83 pgmask=4kb va=0007ffce000 asid=02 [pa=00000000000 c=0 d=0 v=0 g=0] [pa=00000000000 c=0 d=0 v=0 g=0] Index: 84 pgmask=4kb va=00000090000 asid=02 [pa=00000000000 c=0 d=0 v=0 g=0] [pa=00000000000 c=0 d=0 v=0 g=0] ...... ......
可以看到,init 进程访问了一个 0x90722 的地址,出错原因是 tlb load,出错指令位于: 0x38596ed0,指令编码为 0x8c430080
查了一下这个 0x38596ed0 的地址,发现其位于 libc 库,指令上下文环境为:
...... ...... 38596ebc: 0320f809 jalr t9 38596ec0: 03a0202d move a0,sp 38596ec4: 1440002a bnez v0,38596f70 <__libc_start_main+0x170> 38596ec8: 8f8692f0 lw a2,-27920(gp) 38596ecc: 27428b60 addiu v0,k0,-29856 38596ed0: 8c430080 lw v1,128(v0) 38596ed4: 8fa400b4 lw a0,180(sp) 38596ed8: 8fa500b0 lw a1,176(sp) 38596edc: afa300a0 sw v1,160(sp) 38596ee0: 8c43007c lw v1,124(v0) 38596ee4: 8cc60000 lw a2,0(a2) ...... ......
注意一下,v0 的值来自于上条指令的 k0 的值,这个就有点意思了
于是 grep 了一下 glibc 库的代码找到这么一段:
/* Note: rd must be $v1 to be ABI-conformant. */ # ifndef __OCTEON__ # define READ_THREAD_POINTER() \ ({ void *__result; \ asm volatile (".set\tpush\n\t.set\tmips32r2\n\t" \ "rdhwr\t%0, $29\n\t.set\tpop" : "=v" (__result)); \ __result; }) # else /* OCTEON */ /* The Kernel stores the value of "rdhwr v1,$29" in k0 ($26) register. And it is the Kernel's responsibility to always have the correct value in k0. Replacing rdhwr instruction with k0, as this instruction needs to be emulated by the Kernel. */ # define READ_THREAD_POINTER() ( { register void *__result asm ("$26"); __result; } ) # endif /* OCTEON */
原来优化了 glibc 库,直接从 k0 取 thread_info 的指针
反过来看了一下 kernel 部分,似乎没注意到这个
3 解决
找了 kernel 相应的 patch,移植过来即可