Linux – Hev's Blog

Load shared libraries with preferred base address

For performance, sometimes we need to adjust the address space layout. The following way allows the shared or executable object to be loaded with preferred base address.

When the dynamic link maps the shared object, the virtual address of segment(that in ELF program header) will be used as the mapping address hints.

Type 1
Link with text-segment starting address.

# Adjust base address to 0x10000 */
gcc -shared ... -Wl,-Ttext-segment=0x10000

Type 2
We can adjust the virtual address of loaded segment by linker scripts.

0x1 Get link script template

gcc -shared -Wl,--verbose > ld.script

Clear the contents before and after the ‘equal sign’.

0x2 Adjust the base address

....
SECTIONS
{
    /* Adjust base address to 0x10000 */
    . = SEGMENT_START("text-segment", 0x10000) + SIZEOF_HEADERS;
    ...
}
....

0x3 Link with script

gcc -shared ... -Wl,-T ld.script

Over!

mlock() returns EPERM in systemd-nspawn context

Set caps

systemd-nspawn --capability=CAP_IPC_LOCK ...

Over!

Install Linux 5.x to NanoPi M4

Source code
https://github.com/heiher/linux/tree/nanopi-m4

git clone -b nanopi-m4 https://github.com/heiher/linux

Install prebuilt

git clone --depth 1 https://github.com/heiher/kernel-nanopi-m4
cd kernel-nanopi-m4
 
./generate.sh /dev/mmcblk0
 
sudo dd if=output/boot/param.img of=/dev/mmcblk0 bs=1M seek=4
sudo dd if=output/boot/kernel.img of=/dev/mmcblk0 bs=1M seek=32
sudo dd if=output/boot/resource.img of=/dev/mmcblk0 bs=1M seek=20
sudo cp -a output/modules/* /lib/modules/
sudo chown -R root:root /lib/modules

Additional patches
* arm64: rk3399: increase CPU frequency during early boot

This is a workaround to fix the following issues. The right way should
be to increase frequency of CPU before boot linux kernel in bootloader.

[   92.165850] watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [swapper/0:1]
[   92.166643] Modules linked in:
[   92.167463] CPU: 4 PID: 1 Comm: swapper/0 Not tainted 5.5.10-ARCH #1
[   92.168025] Hardware name: FriendlyElec NanoPi M4 (DT)
[   92.168672] pstate: 20000005 (nzCv daif -PAN -UAO)
[   92.169504] pc : trace_event_eval_update+0xfc/0x348
[   92.170164] lr : trace_event_eval_update+0x144/0x348
[   92.170652] sp : ffff80001004bcf0
[   92.171112] x29: ffff80001004bcf0 x28: 0000000000000001
[   92.171834] x27: ffff8000118d2228 x26: ffff8000111710b0
[   92.172502] x25: ffff80001180db98 x24: 000000000000000e
[   92.173134] x23: ffff800011746428 x22: 00000000000000ec
[   92.173749] x21: 0000000000000264 x20: 0000000000000158
[   92.174352] x19: ffff8000118bc778 x18: 0000000000000001
[   92.174951] x17: 00000000ed34b782 x16: 00000000d54c3662
[   92.175549] x15: 2d20377b1910031c x14: ff00000000000000
[   92.176140] x13: 0000000000000000 x12: 0000000000000007
[   92.176731] x11: 0101010101010101 x10: 0000000000000005
[   92.177322] x9 : 0000000000000003 x8 : 0000000000000008
[   92.177922] x7 : 1c0310197b37202d x6 : 2d20377b1910031c
[   92.178506] x5 : 0000000000000000 x4 : 000000000000000e
[   92.179085] x3 : ffff8000118bbbee x2 : 0000000000000045
[   92.179671] x1 : 000000000000002c x0 : 0000000000000000
[   92.180263] Call trace:
[   92.180921]  trace_event_eval_update+0xfc/0x348
[   92.181652]  tracer_init_tracefs+0x160/0x1e8
[   92.182304]  do_one_initcall+0x4c/0x218
[   92.182949]  kernel_init_freeable+0x1d0/0x240
[   92.183652]  kernel_init+0x18/0x104
[   92.184245]  ret_from_fork+0x10/0x18

[  133.872912] cpufreq: cpufreq_online: CPU4: Running at unlisted freq: 12000 KHz
[  133.891606] cpufreq: cpufreq_online: CPU4: Unlisted initial frequency changed to: 408000 KHz

Over!

Linux recovery移除签名校验

背景
某个设备配套的刷机程序是个Linux recovery kernel，刷机过程会先从U盘加载刷机脚本，仅在签名校验通过后才执行脚本。本文记录了分析和移除签名校验的方法。

分析
刷机程序是一个bzImage文件，从启动的输出来看，内部包含了一个initrd，在initrd中实现了读取U盘中的脚本和签名校验过程。

查看initrd内容
通过增加启动参数(cmdline）rdinit=/bin/sh，可以使Kernel启动后执行/bin/sh，而不是默认的/init程序，有了命令行接口后，就可以查看initrd的内容。

~ # busybox find /
/
/.ash_history
/init
/etc
/etc/shadow
/etc/passwd
/.gnupg
/.gnupg/trustdb.gpg
/.gnupg/secring.gpg
/.gnupg/pubring.gpg~
/.gnupg/pubring.gpg
/bin
/bin/kexec
/bin/gpg2
/bin/busybox
/bin/dd
/bin/umount
/bin/sleep
/bin/rmdir
/bin/rm
/bin/reboot
/bin/mount
/bin/mkdir
/bin/ls
/bin/cat
/bin/sh
/mnt
/sys
/proc
/dev
/dev/pts
/dev/loop0
/dev/tty0
/dev/console

# cat /init
...
gpg2 --ignore-time-conflict --ignore-valid-from --verify $FLASH_FILE_SIG $FLASH_FILE
if [ $? -eq 0 ]; then
    echo "PWR_LED 3" > /proc/BOARD_io
    /bin/busybox sh $FLASH_FILE
    if [ $? -eq 0 ]; then
        echo "PWR_LED 1" > /proc/BOARD_io
        echo "flash success..."
        echo "Please unplug USB drive and power cycle system"
    else
        echo "PWR_LED 4" > /proc/BOARD_io
        echo "flash failed..."
        echo "Please try again or try another board"
    fi
else
    echo "PWR_LED 4" > /proc/BOARD_io
    echo "flash failed..."
    echo "Script verify failed"
fi
...

从initrd的内容来看，由/init调用gpg2对U盘中的刷机脚本执行签名校验，只有公钥集成在initrd中，没有私钥。

到这一步，我们已经清楚了签名校验的实现方法，并且也能使启动过程进入受控的命令行交互状态，其实已经可以手工操作跳过签名过程来刷机。

修改
每次手工操作的确太麻烦，那就来移除initrd中的签名校验过程吧。

从bzImage的结构来看，要想修改initrd，先要从bzImage中提取出vmlinux，再从vmlinux中提取出initrd。

1. 提取vmlinux
从bzImage中提取vmlinux比较简单，有现成的工具，位于Linux源代码中 scripts/extract-vmlinux

./scripts/extract-vmlinux bzImage > vmlinux

2. 提取initrd
initrd的格式可以是cpio archive，也可以是gzip、bzip2、lzma、xz或lzo压缩的，先用binwalk扫描一遍。

binwalk vmlinux

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             ELF, 32-bit LSB executable, Intel 80386, version 1 (SYSV)
3641536       0x3790C0        Linux kernel version "2.6.39 (ubuntu@ubuntu) (gcc version 4.9.3 20150311 (prerelease) (crosstool-NG 1.20.0) ) #24 SMP Fri Jun 7 14:32:37 CST 2019"
3922304       0x3BD980        CRC32 polynomial table, little endian
4318976       0x41E700        Unix path: /home/ubuntu/ce5300/barcelona_kernel/arch/x86/include/asm/desc.h
4321256       0x41EFE8        Unix path: /home/ubuntu/ce5300/barcelona_kernel/arch/x86/include/asm/i387.h
4322244       0x41F3C4        Unix path: /home/ubuntu/ce5300/barcelona_kernel/arch/x86/include/asm/processor.h
4323964       0x41FA7C        Unix path: /x86/kernel/cpu/perf_event_intel.c
4324152       0x41FB38        Unix path: /x86/kernel/cpu/perf_event_intel_ds.c
4325960       0x420248        Unix path: /x86/kernel/cpu/mcheck/mce.c
4326820       0x4205A4        Unix path: /x86/kernel/cpu/mcheck/mce_intel.c
4327124       0x4206D4        Unix path: /x86/kernel/cpu/mcheck/therm_throt.c
4328480       0x420C20        Unix path: /x86/kernel/cpu/mtrr/generic.c
4329752       0x421118        Unix path: /x86/kernel/cpu/mtrr/cleanup.c
4329832       0x421168        Unix path: /x86/kernel/cpu/perfctr-watchdog.c
4336148       0x422A14        Unix path: /x86/kernel/apic/apic_noop.c
4336572       0x422BBC        Unix path: /x86/kernel/apic/io_apic.c
4343276       0x4245EC        Unix path: /home/ubuntu/ce5300/barcelona_kernel/arch/x86/include/asm/fixmap.h
4347540       0x425694        Unix path: /x86/kernel/cpu/common.c
4347663       0x42570F        Unix path: /x86/kernel/cpu/vmware.c
4347911       0x425807        Unix path: /x86/kernel/cpu/intel.c
4350475       0x42620B        Unix path: /x86/kernel/acpi/boot.c
4352464       0x4269D0        Unix path: /x86/kernel/apic/apic.c
4352799       0x426B1F        Unix path: /x86/kernel/apic/ipi.c
4367224       0x42A378        Unix path: /home/ubuntu/ce5300/barcelona_kernel/arch/x86/include/asm/mmu_context.h
4374285       0x42BF0D        Unix path: /sys/kernel/debug/tracing/trace_clock
4383716       0x42E3E4        Unix path: /home/ubuntu/ce5300/barcelona_kernel/arch/x86/include/asm/pgalloc.h
4384752       0x42E7F0        Unix path: /home/ubuntu/ce5300/barcelona_kernel/arch/x86/include/asm/dma-mapping.h
4513864       0x44E048        xz compressed data
4514016       0x44E0E0        Unix path: /home/ubuntu/ce5300/barcelona_kernel/arch/x86/include/asm/syscall.h
4533558       0x452D36        Unix path: /Buffer/String/Package/Ref/Ddb], found [%s] %p
4612622       0x46620E        Unix path: /sys/kernel/debug/dri
4614914       0x466B02        Unix path: /sys/kernel/debug/dri.
4618302       0x46783E        Unix path: /sys/kernel/debug/dri/%s/%s
4618366       0x46787E        Unix path: /sys/kernel/debug/dri/%s
4618509       0x46790D        Unix path: /sys/kernel/debug/dri.
4661219       0x471FE3        Unix path: /S70/S75/505V/F505/F707/F717/P8
4665828       0x4731E4        Unix path: /usr/include/asm/ioctls.h
4678778       0x47647A        Copyright string: "Copyright(c) Pierre Ossman"
4690408       0x4791E8        Unix path: /x86/oprofile/../../../drivers/oprofile/event_buffer.c
5242204       0x4FFD5C        ELF, 32-bit LSB shared object, Intel 80386, version 1 (SYSV)
5243884       0x5003EC        ELF, 32-bit LSB shared object, Intel 80386, version 1 (SYSV)

DECIMAL HEXADECIMAL DESCRIPTION -------------------------------------------------------------------------------- 0 0x0 ELF, 32-bit LSB executable, Intel 80386, version 1 (SYSV) 3641536 0x3790C0 Linux kernel version "2.6.39 (ubuntu@ubuntu) (gcc version 4.9.3 20150311 (prerelease) (crosstool-NG 1.20.0) ) #24 SMP Fri Jun 7 14:32:37 CST 2019" 3922304 0x3BD980 CRC32 polynomial table, little endian 4318976 0x41E700 Unix path: /home/ubuntu/ce5300/barcelona_kernel/arch/x86/include/asm/desc.h 4321256 0x41EFE8 Unix path: /home/ubuntu/ce5300/barcelona_kernel/arch/x86/include/asm/i387.h 4322244 0x41F3C4 Unix path: /home/ubuntu/ce5300/barcelona_kernel/arch/x86/include/asm/processor.h 4323964 0x41FA7C Unix path: /x86/kernel/cpu/perf_event_intel.c 4324152 0x41FB38 Unix path: /x86/kernel/cpu/perf_event_intel_ds.c 4325960 0x420248 Unix path: /x86/kernel/cpu/mcheck/mce.c 4326820 0x4205A4 Unix path: /x86/kernel/cpu/mcheck/mce_intel.c 4327124 0x4206D4 Unix path: /x86/kernel/cpu/mcheck/therm_throt.c 4328480 0x420C20 Unix path: /x86/kernel/cpu/mtrr/generic.c 4329752 0x421118 Unix path: /x86/kernel/cpu/mtrr/cleanup.c 4329832 0x421168 Unix path: /x86/kernel/cpu/perfctr-watchdog.c 4336148 0x422A14 Unix path: /x86/kernel/apic/apic_noop.c 4336572 0x422BBC Unix path: /x86/kernel/apic/io_apic.c 4343276 0x4245EC Unix path: /home/ubuntu/ce5300/barcelona_kernel/arch/x86/include/asm/fixmap.h 4347540 0x425694 Unix path: /x86/kernel/cpu/common.c 4347663 0x42570F Unix path: /x86/kernel/cpu/vmware.c 4347911 0x425807 Unix path: /x86/kernel/cpu/intel.c 4350475 0x42620B Unix path: /x86/kernel/acpi/boot.c 4352464 0x4269D0 Unix path: /x86/kernel/apic/apic.c 4352799 0x426B1F Unix path: /x86/kernel/apic/ipi.c 4367224 0x42A378 Unix path: /home/ubuntu/ce5300/barcelona_kernel/arch/x86/include/asm/mmu_context.h 4374285 0x42BF0D Unix path: /sys/kernel/debug/tracing/trace_clock 4383716 0x42E3E4 Unix path: /home/ubuntu/ce5300/barcelona_kernel/arch/x86/include/asm/pgalloc.h 4384752 0x42E7F0 Unix path: /home/ubuntu/ce5300/barcelona_kernel/arch/x86/include/asm/dma-mapping.h 4513864 0x44E048 xz compressed data 4514016 0x44E0E0 Unix path: /home/ubuntu/ce5300/barcelona_kernel/arch/x86/include/asm/syscall.h 4533558 0x452D36 Unix path: /Buffer/String/Package/Ref/Ddb], found [%s] %p 4612622 0x46620E Unix path: /sys/kernel/debug/dri 4614914 0x466B02 Unix path: /sys/kernel/debug/dri. 4618302 0x46783E Unix path: /sys/kernel/debug/dri/%s/%s 4618366 0x46787E Unix path: /sys/kernel/debug/dri/%s 4618509 0x46790D Unix path: /sys/kernel/debug/dri. 4661219 0x471FE3 Unix path: /S70/S75/505V/F505/F707/F717/P8 4665828 0x4731E4 Unix path: /usr/include/asm/ioctls.h 4678778 0x47647A Copyright string: "Copyright(c) Pierre Ossman" 4690408 0x4791E8 Unix path: /x86/oprofile/../../../drivers/oprofile/event_buffer.c 5242204 0x4FFD5C ELF, 32-bit LSB shared object, Intel 80386, version 1 (SYSV) 5243884 0x5003EC ELF, 32-bit LSB shared object, Intel 80386, version 1 (SYSV)

在vmlinux文件偏移0x44E048处，有一个疑似xz压缩文档，提取出来尝试解压。

dd if=vmlinux of=t.xz bs=$((0x44E048)) skip=1
unxz t.xz
unxz: t.xz: Compressed data is corrupt

唯一的疑似压缩文档解压出错了，这个方法行不通，那就换另外一个方法吧。:)

2.1. 分析启动过程中的initrd加载
从bzImage中提取出的vmlinux是strip掉symbols的，不便于反汇编后定位函数，我们先提取该内核的/proc/kallsyms，直接在rdinit=/bin/sh启动的命令行中cat /proc/kallsyms就可以了。
有了symbols后，首先我们要找populate_rootfs函数，从汇编代码中获得__initramfs_start和__initramfs_size。

c14d03b1 t populate_rootfs
c14d0129 t unpack_to_rootfs

c14d03b1:	55                   	push   %ebp
c14d03b2:	b8 6c 59 51 c1       	mov    $0xc151596c,%eax
c14d03b7:	57                   	push   %edi
c14d03b8:	56                   	push   %esi
c14d03b9:	53                   	push   %ebx
c14d03ba:	8d 64 24 b8          	lea    -0x48(%esp),%esp
c14d03be:	8b 15 70 6f 8e c1    	mov    0xc18e6f70,%edx
c14d03c4:	e8 60 fd ff ff       	call   0xc14d0129

在0xc14d03c4处调用了0xc14d0129这个函数，也就是unpack_to_rootfs，传递了两个参数，%eax就是__initramfs_start，值是0xc151596c，%edx就是__initramfs_size，%edx的值是存储在地址0xc18e6f70处的。

有了__initramfs_start的程序地址后，只需要转换为vmlinux文件的偏移地址后，就可以提取出initrd的内容了。映射关系可以通过readelf获得。

readelf -S vmlinux

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .text             PROGBITS        c1000000 001000 376b28 00  AX  0   0 64
  [ 2] .notes            NOTE            c1376b28 377b28 000024 00  AX  0   0  4
  [ 3] __ex_table        PROGBITS        c1376b50 377b50 000c48 00   A  0   0  4
  [ 4] .rodata           PROGBITS        c1378000 379000 100858 00   A  0   0 64
  [ 5] __bug_table       PROGBITS        c1478858 479858 006588 00   A  0   0  1
  [ 6] .pci_fixup        PROGBITS        c147ede0 47fde0 000b38 00   A  0   0  4
  [ 7] __init_rodata     PROGBITS        c147f940 480940 004040 00   A  0   0 64
  [ 8] __param           PROGBITS        c1483980 484980 000960 00   A  0   0  4
  [ 9] __modver          PROGBITS        c14842e0 4852e0 000d20 00  WA  0   0  4
  [10] .data             PROGBITS        c1485000 486000 048d40 00  WA  0   0 4096
  [11] .init.text        PROGBITS        c14ce000 4cf000 025b13 00  AX  0   0  1
  [12] .init.data        PROGBITS        c14f3b40 4f4b40 3f3434 00  WA  0   0 64
  [13] .x86_trampoline   PROGBITS        c18e7000 8e8000 003328 00   A  0   0 4096
  [14] .x86_cpu_dev.init PROGBITS        c18ea328 8eb328 00001c 00   A  0   0  4
  [15] .altinstructions  PROGBITS        c18ea348 8eb348 002dcc 00   A  0   0  4
  [16] .altinstr_replace PROGBITS        c18ed114 8ee114 000bd9 00  AX  0   0  1
  [17] .exit.text        PROGBITS        c18edcf0 8eecf0 0011d0 00  AX  0   0  1
  [18] .data..percpu     PROGBITS        c18ef000 8f0000 00609c 00  WA  0   0 4096
  [19] .smp_locks        PROGBITS        c18f6000 8f7000 004000 00   A  0   0  4
  [20] .bss              NOBITS          c18fa000 8fb000 04a000 00  WA  0   0 4096
  [21] .brk              NOBITS          c1944000 8fb000 120000 00  WA  0   0  1
  [22] .shstrtab         STRTAB          00000000 8fb000 0000e8 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  p (processor specific)

Section Headers: [Nr] Name Type Addr Off Size ES Flg Lk Inf Al [ 0] NULL 00000000 000000 000000 00 0 0 0 [ 1] .text PROGBITS c1000000 001000 376b28 00 AX 0 0 64 [ 2] .notes NOTE c1376b28 377b28 000024 00 AX 0 0 4 [ 3] __ex_table PROGBITS c1376b50 377b50 000c48 00 A 0 0 4 [ 4] .rodata PROGBITS c1378000 379000 100858 00 A 0 0 64 [ 5] __bug_table PROGBITS c1478858 479858 006588 00 A 0 0 1 [ 6] .pci_fixup PROGBITS c147ede0 47fde0 000b38 00 A 0 0 4 [ 7] __init_rodata PROGBITS c147f940 480940 004040 00 A 0 0 64 [ 8] __param PROGBITS c1483980 484980 000960 00 A 0 0 4 [ 9] __modver PROGBITS c14842e0 4852e0 000d20 00 WA 0 0 4 [10] .data PROGBITS c1485000 486000 048d40 00 WA 0 0 4096 [11] .init.text PROGBITS c14ce000 4cf000 025b13 00 AX 0 0 1 [12] .init.data PROGBITS c14f3b40 4f4b40 3f3434 00 WA 0 0 64 [13] .x86_trampoline PROGBITS c18e7000 8e8000 003328 00 A 0 0 4096 [14] .x86_cpu_dev.init PROGBITS c18ea328 8eb328 00001c 00 A 0 0 4 [15] .altinstructions PROGBITS c18ea348 8eb348 002dcc 00 A 0 0 4 [16] .altinstr_replace PROGBITS c18ed114 8ee114 000bd9 00 AX 0 0 1 [17] .exit.text PROGBITS c18edcf0 8eecf0 0011d0 00 AX 0 0 1 [18] .data..percpu PROGBITS c18ef000 8f0000 00609c 00 WA 0 0 4096 [19] .smp_locks PROGBITS c18f6000 8f7000 004000 00 A 0 0 4 [20] .bss NOBITS c18fa000 8fb000 04a000 00 WA 0 0 4096 [21] .brk NOBITS c1944000 8fb000 120000 00 WA 0 0 1 [22] .shstrtab STRTAB 00000000 8fb000 0000e8 00 0 0 1 Key to Flags: W (write), A (alloc), X (execute), M (merge), S (strings), I (info), L (link order), O (extra OS processing required), G (group), T (TLS), C (compressed), x (unknown), o (OS specific), E (exclude), p (processor specific)

程序地址0xc151596c和0xc18e6f70都隶属于.init.data section，文件偏移计算：

Offset = Addr - Section Base Addr + Section Base Offset
 
0xc151596c: 0xc151596c - 0xc14f3b40 + 0x4f4b40 = 0x51696c
0xc18e6f70: 0xc18e6f70 - 0xc14f3b40 + 0x4f4b40 = 0x8e7f70

2.2. 提取initrd
提取initrd，首先需要知道__initramfs_size，该值位于vmlinux文件的0x8e7f70偏移处，类型是unsigned long。

08e7f70 1600 003d 0000 0000 0000 0000 0000 0000

dd if=vmlinux of=initrd bs=$((0x51696c)) skip=1
truncate -s $((0x3d1600)) initrd

2.3. 分析initrd格式
虽然提取出了initrd，但不是已知的格式，内核支持的格式都有确定的magic number:

static const struct compress_format {
	unsigned char magic[2];
	const char *name;
	decompress_fn decompressor;
} compressed_formats[] = {
	{ {037, 0213}, "gzip", gunzip },
	{ {037, 0236}, "gzip", gunzip },
	{ {0x42, 0x5a}, "bzip2", bunzip2 },
	{ {0x5d, 0x00}, "lzma", unlzma },
	{ {0xfd, 0x37}, "xz", unxz },
	{ {0x89, 0x4c}, "lzo", unlzo },
	{ {0, 0}, NULL, NULL }
};

hexdump initrd

0000000 6fde 40fe 2ee2 5fbf 27e3 e8fe fb88 6a72
0000010 b649 904e 378a 49f4 057f 69b4 f9d9 4d43
0000020 7a8a fe5b 1ba5 2442 3ea5 365e 7945 fd49
0000030 9afb fca6 143c b30d eff8 a715 0982 424c
...

既然这个内核能执行，说明它有一种未知的加载方法，那就看看它是怎么做的吧，我们需要找到unpack_to_rootfs函数。

c14d0129 t unpack_to_rootfs
c1001410 T aes_key_schedule_128
c10017c0 T aes_decrypt_128

c14d0129:	55                   	push   %ebp
c14d012a:	b9 11 00 00 00       	mov    $0x11,%ecx
c14d012f:	89 d5                	mov    %edx,%ebp
c14d0131:	57                   	push   %edi
c14d0132:	56                   	push   %esi
c14d0133:	be ce c6 41 c1       	mov    $0xc141c6ce,%esi
c14d0138:	53                   	push   %ebx
c14d0139:	89 c3                	mov    %eax,%ebx
c14d013b:	8d a4 24 30 ff ff ff 	lea    -0xd0(%esp),%esp
c14d0142:	8d 7c 24 0f          	lea    0xf(%esp),%edi
c14d0146:	8d 54 24 20          	lea    0x20(%esp),%edx
c14d014a:	8d 44 24 0f          	lea    0xf(%esp),%eax
c14d014e:	f3 a4                	rep movsb %ds:(%esi),%es:(%edi)
c14d0150:	e8 bb 12 b3 ff       	call   0xc1001410 // aes_key_schedule_128
c14d0155:	31 f6                	xor    %esi,%esi
c14d0157:	39 ee                	cmp    %ebp,%esi
c14d0159:	73 13                	jae    0xc14d016e
c14d015b:	8d 14 33             	lea    (%ebx,%esi,1),%edx
c14d015e:	8d 44 24 20          	lea    0x20(%esp),%eax
c14d0162:	89 d1                	mov    %edx,%ecx
c14d0164:	83 c6 10             	add    $0x10,%esi
c14d0167:	e8 54 16 b3 ff       	call   0xc10017c0 // aes_decrypt_128
c14d016c:	eb e9                	jmp    0xc14d0157
c14d016e:	a1 7c 74 93 c1       	mov    0xc193747c,%eax

果然是unpack_to_rootfs被修改了，里面调用了aes_key_schedule_128和aes_decrypt_128两个函数，加入了AES128解密过程，这说明我们提取出来的initrd是被加密的。
AES128是对称加密，如果没有使用加硬件密钥不管，极有可能是硬编码在程序中的，试试提出它。从汇编代码中看，在栈上构建了一个crypto上下文，部分内容是从地址0xc141c6ce复制过来的，这会不会就是密钥呢？

041d6b2 65 6D  1C 58 72 35  04 A4 0E DD  53 C5 CC D2  B2 4E 00 69  6E 69 74 2F  69 6E 69 74 em.Xr5....S....N.init/init

一个16字节的数据，与字符串混编在一起，极有可能是128位的密钥。

2.4. 解密initrd

openssl enc -d -aes-128-ecb -in initrd -out initrd.img -K 656D1C58723504A40EDD53C5CCD2B24E

经常一些尝试，使用AES-128-ECB解密成功，还原出了initrd.img，实际为cpio格式。

2.5. 修改initrd

mkdir rootfs && cd rootfs
cpio -id < ../initrd.img
 
rm -rf .gnupg bin/gpg2
vim init # Remove gpg2 verify
 
find . | cpio -H newc -o > ../initrd-noverify.img

2.6. 写回initrd
从unpack_to_rootfs汇编代码可以看出，aes_decrypt将明文写到了对应密文的相同内存空间，我们可以修改代码来跳过解密过程，这样就可以直接把initrd-noverify.img写回到vmlinux中，而不需要再加密来找麻烦了。

c14d0129:	55                   	push   %ebp
c14d012a:	b9 11 00 00 00       	mov    $0x11,%ecx
c14d012f:	89 d5                	mov    %edx,%ebp
c14d0131:	57                   	push   %edi
c14d0132:	56                   	push   %esi
c14d0133:	be ce c6 41 c1       	mov    $0xc141c6ce,%esi
c14d0138:	53                   	push   %ebx
c14d0139:	89 c3                	mov    %eax,%ebx
c14d013b:	8d a4 24 30 ff ff ff 	lea    -0xd0(%esp),%esp
c14d0142:	8d 7c 24 0f          	lea    0xf(%esp),%edi
c14d0146:	8d 54 24 20          	lea    0x20(%esp),%edx
c14d014a:	8d 44 24 0f          	lea    0xf(%esp),%eax
c14d014e:	f3 a4                	rep movsb %ds:(%esi),%es:(%edi)
c14d0150:	e8 bb 12 b3 ff       	call   0xc1001410 // aes_key_schedule_128
c14d0155:	e9 14 00 00 00        	jmp    0xc14d016e
c14d0157:	39 ee                	cmp    %ebp,%esi
c14d0159:	73 13                	jae    0xc14d016e
c14d015b:	8d 14 33             	lea    (%ebx,%esi,1),%edx
c14d015e:	8d 44 24 20          	lea    0x20(%esp),%eax
c14d0162:	89 d1                	mov    %edx,%ecx
c14d0164:	83 c6 10             	add    $0x10,%esi
c14d0167:	e8 54 16 b3 ff       	call   0xc10017c0 // aes_decrypt_128
c14d016c:	eb e9                	jmp    0xc14d0157
c14d016e:	a1 7c 74 93 c1       	mov    0xc193747c,%eax

程序地址0xc14d0155处的代码修改为jmp 0xc14d016e，这样就可以直接跳过aes_decrypt过程了。如果不懂x86的instruction encoding方法，有个简单的方法可以产生正确的jmp 0xc14d016e编码。

    .text
    .globl _start
_start:
    jmp    0xc14d016e

gcc -m32 -o jmp -nostdlib -Wl,-Ttext=0xc14d0155 jmp.S
objdump -d jmp

c14d0155 <_start>:
c14d0155:	e9 14 00 00 00       	jmp    c14d016e <_start+0x19>

现在可以写回initrd-noverify.img了

dd if=initrd-noverify.img of=vmlinux conv=notrunc bs=$((0x51696c)) seek=1

还需要编辑vmlinux文件0x8e7f70偏移处的__initramfs_size，改为initrd-noverify.img的长度。

2.7. 写回vmlinux
extract-vmlinux脚本也是通过搜索magic number和尝试解压来提取vmlinux的，以此就可以获得vmlinux在bzImage中的偏移，参考写回initrd类似的方法，可以将vmlinux写回至bzImage原来的偏移位置。
不幸的是，修改后的bzImage无法启动，启动参数加入earlyprintk=ttyS0,115200后，可以看到LZMA data is corrupt的错误。

bzImage的解压缩过程是从input addr读取压缩数据，解压后写到output addr，而output addr与input addr之间的空间是不够完整存放解压后的数据的，之所以没有问题是因为这个input addr与output addr之间的差值是经过精心计算的，能够保证覆盖发生时，被覆盖的数据已完成解压。

正是因为这个原因，我们修改后的vmlinux由于内容变化，压缩比也发现了变化，已不能适应之前的比例计算出的差值，有两种方法解决这个问题：1. 将output addr向低地址方向移动。2. 将input addr向高地址方向移动。
从反汇编bzImage的代码来看，output addr在之后还有校验，相对修改input addr更为复杂。

objdump -D -b binary -mi386 bzImage

...
  5d91d8:	8d ab 00 10 bf ff    	lea    -0x337000(%ebx),%ebp
  5d91de:	55                   	push   %ebp                // output address
  5d91df:	68 f5 d1 4f 00       	push   $0x5d5747           // input length
  5d91e4:	8d 83 62 00 00 00    	lea    0x62(%ebx),%eax     // input address
  5d91ea:	50                   	push   %eax
  5d91eb:	8d 83 40 7b 5d 00    	lea    0x5d7b40(%ebx),%eax // heap
  5d91f1:	50                   	push   %eax
  5d91f2:	56                   	push   %esi                // rmode
  5d91f3:	e8 14 04 00 00       	call   0x5d960c
  5d91f8:	83 c4 14             	add    $0x14,%esp
  5d91fb:	31 db                	xor    %ebx,%ebx
  5d91fd:	ff e5                	jmp    *%ebp
...

Over!

Run GNOME on NanoPi M4 (RK3399)

Issues

/usr/bin/gnome-shell: symbol lookup error: /lib/libmutter-4.so.0: undefined symbol: gbm_bo_get_offset

Arch Linux
Refer: https://archlinuxarm.org/platforms/armv8/generic

Mali GPU user space drivers
Drivers: https://github.com/heiher/libmali-rk3399

git clone https://github.com/heiher/libmali-rk3399
cd libmali-rk3399
 
# Build gbm wrapper
make
 
# Install to system
sudo cp conf/mali.conf /etc/ld.so.conf.d/
sudo cp -rd lib /usr/lib/mali
 
# Update ld.so cache
sudo ldconfig

Clutter
Force use GLESv2

# /etc/clutter-1.0/settings.ini
[Environment]
Drivers=gles2

Fix crash when maximizing window
Patch: https://gitlab.gnome.org/GNOME/mutter/merge_requests/515

GPU device access permission

# /etc/udev/rules.d/50-mali.rules 
KERNEL=="mali0", MODE="0666"

sudo chmod 0666 /dev/mali0

Start gdm.

Over!

Run QEMU with hardware virtualization on macOS

在macOS上通过虚拟机运行其它操作系统，又不想用商业软件，那么开源的QEMU是一个比较好的选择。QEMU的功能支持还是比较全面的，除了功能以外，使用虚拟机软件的用户最关心的就是性能了，一个好消息是macOS 10.10+版本已经引人了硬件虚拟化支持框架，也就是Hypervisor.framework，另一个好消息是QEMU也已支持该框架，也就是hvf accelerator。

Requirements
1. macOS 10.10+
2. Macports

Issues
已经使用过的用户可能已经发现，QEMU使用hvf accelerator并开启多核是有问题的呀。的确，QEMU使用hvf accelerator以单核运行时没有问题，当使用-smp参数指定多核时，很大概率上虚拟机硬件初始化都完成不了就死机了。
不过，好消息是该问题也已经修复了，导致这个问题的原因是hvf accelerator代码设计没有考虑到虚拟机启动后所有hvf vcpu都在并行执行指令，其中包括硬件初始化的I/O模拟操作，多个CPU同时对同一硬件执行初始化显然是不行的。

Patch (已经合并上游社区)

Install QEMU

cd ~
git clone https://github.com/hevz/macports
sudo vim /opt/local/etc/macports/sources.conf

# Add local repositories
file:///Users/[YOUR USER NAME]/macports
rsync://rsync.macports.org/macports/release/tarballs/ports.tar [default]

cd ~/macports
portindex
sudo port install qemu

Run Arch Linux
1. 下载Arch Linux安装ISO镜像。
2. 创建一个虚拟机磁盘镜像。
3. 开始安装新的系统。
4. 启动安装后的系统。

mkdir ~/system/images
qemu-img create -f qcow2 ~/system/images/arch.qcow2 40G
 
qemu-system-x86_64 -no-user-config -nodefaults -show-cursor \
    -M pc-q35-3.1,accel=hvf,usb=off,vmport=off \
    -cpu host -smp 4,sockets=1,cores=2,threads=2 -m 4096 \
    -realtime mlock=off -rtc base=utc,driftfix=slew \
    -drive file=~/system/images/arch.qcow2,if=none,format=qcow2,id=disk0 \
    -device virtio-blk-pci,bus=pcie.0,addr=0x1,drive=disk0 \
    -netdev user,id=net0,hostfwd=tcp::2200-:22 \
    -device virtio-net-pci,netdev=net0,bus=pcie.0,addr=0x2 \
    -device virtio-keyboard-pci,bus=pcie.0,addr=0x3 \
    -device virtio-tablet-pci,bus=pcie.0,addr=0x4 \
    -device virtio-vga,bus=pcie.0,addr=0x5 \
    -cdrom ~/archlinux-2019.01.01-x86_64.iso -boot d

安装完成后，删除qemu-system-x86_64最后一行命令即可启动新系统。

Over!

Category: Linux

Load shared libraries with preferred base address

Allow perf in systemd-nspawn

mlock() returns EPERM in systemd-nspawn context

Nginx WebDAV Service

Install Linux 5.x to NanoPi M4

Systemd.networkd example

快速清除SSD数据

Linux recovery移除签名校验

Run GNOME on NanoPi M4 (RK3399)

Run QEMU with hardware virtualization on macOS