使用 Ubuntu on ZFS root 大約一個星期之後，覺得 ZFS 在做系統快照的時候，不像 Btrfs 會讓整個系統 CPU Usage 飆高，所有的 UI 反應幾乎都停格，整個卡頓過程大概會有 30 秒，真的有點令人受不了。像我試著嘗試每小時快照一次，每次一到整點，就好像要強迫你要起來走走，真的有點要裝狂。。。
這一次的目標是把之前安裝在分區５的 Btrfs 刪除，將完整的一半 SSD 空間做成 ZFS 給 Ubuntu 使用。暫時對 Btrfs 說聲抱歉，之後成熟度再高些的時候再來看看，畢竟 Btrfs 配上 TimeShift 在 Ubuntu 上還算整合滿好的，安裝設定都很簡單，又有不錯的圖形界面，有些接近 MAC 上的 Time-Machine 體驗。唯一的遺憾就是 Btrfs 模式下 TimeShift 只能把快照放在本機的硬碟，如果想要把快照儲存在外接硬碟的話，就要配合像 btrbk 這樣的工具，也就沒有圖形界面可用了。而且 btrbk 又會產生另外一組快照，總覺得有些重複多餘。
首先使用 Ubuntu 20.04 安裝碟開機，執行 Disks，重裝前的 SSD 分割狀況如下：
第一步就是用 Disks 把分區 5, 6, 7 都刪除，然後用命令列執行 sgdisk 加上需要的 ZFS 分區（因為 sgdisk 可以指定分區的格式碼為 BF01：Solaris)，看到有文件說 bpool 最好不要劃的太小，以免 kernel 版本比較多的時候，/boot 可能會空間不夠放。所以這一次 bpool 預計是 2GB，其餘的都劃給 rpool 使用。
ubuntu@ubuntu:~$ sudo sgdisk -p /dev/nvme0n1
Disk /dev/nvme0n1: 2000409264 sectors, 953.9 GiB
Model: INTEL SSDPEKKW010T7
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): C85DF9DB-279D-4D14-83DC-4D49489E105A
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 2000409230
Partitions will be aligned on 2048-sector boundaries
Total free space is 1025223277 sectors (488.9 GiB)
Number Start (sector) End (sector) Size Code Name
1 2048 2099199 1024.0 MiB EF00 EFI system partition
2 2099200 2361343 128.0 MiB 0C01 Microsoft reserved ...
3 2361344 955197439 454.3 GiB 0700 Basic data partition
4 1980418048 2000408575 9.5 GiB 2700 Basic data partition
ubuntu@ubuntu:~$ sudo sgdisk -n5:0:+2G -t5:BF01 /dev/nvme0n1
The operation has completed successfully.
ubuntu@ubuntu:~$ sudo sgdisk -n6:0:0 -t6:BF01 /dev/nvme0n1
The operation has completed successfully.
ubuntu@ubuntu:~$ sudo sgdisk -p /dev/nvme0n1
Disk /dev/nvme0n1: 2000409264 sectors, 953.9 GiB
Model: INTEL SSDPEKKW010T7
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): C85DF9DB-279D-4D14-83DC-4D49489E105A
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 2000409230
Partitions will be aligned on 2048-sector boundaries
Total free space is 2669 sectors (1.3 MiB)
Number Start (sector) End (sector) Size Code Name
1 2048 2099199 1024.0 MiB EF00 EFI system partition
2 2099200 2361343 128.0 MiB 0C01 Microsoft reserved ...
3 2361344 955197439 454.3 GiB 0700 Basic data partition
4 1980418048 2000408575 9.5 GiB 2700 Basic data partition
5 955197440 959391743 2.0 GiB BF01
6 959391744 1980418047 486.9 GiB BF01
我在這次安裝之前有把系統ZFS快照存在我的一張大容量SD卡上，現在我只須要先把zpool import進來，然後再把整個系統拷貝過去即可。
ubuntu@ubuntu:~$ sudo zpool import -d /dev/mmcblk0 sd256gpool
ubuntu@ubuntu:~$ zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
sd256gpool 240G 85.3G 155G - - 0% 35% 1.00x ONLINE -
ubuntu@ubuntu:~$ ls /media/SD256GB/backup/
BOOT HOME ROOT
建立 ZFS pool
因為之前分區5是Btfrs，分區6是另一個pool所使用，所以在建立zpool時要加上 -f 強制覆蓋。
ubuntu@ubuntu:~$ sudo zpool create -f \
> -o ashift=12 \
> -O acltype=posixacl -O canmount=off -O compression=lz4 \
> -O dnodesize=auto -O normalization=formD -O relatime=on \
> -O xattr=sa -O mountpoint=/ -R /mnt \
> rpool /dev/nvme0n1p6
ubuntu@ubuntu:~$ zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 1.88G 444K 1.87G - - 0% 0% 1.00x ONLINE /mnt
sd256gpool 240G 63.3G 177G - - 0% 26% 1.00x ONLINE -
ubuntu@ubuntu:~$ sudo zpool create -f \
> -o ashift=12 -d \
> -o feature@async_destroy=enabled \
> -o feature@bookmarks=enabled \
> -o feature@embedded_data=enabled \
> -o feature@empty_bpobj=enabled \
> -o feature@enabled_txg=enabled \
> -o feature@extensible_dataset=enabled \
> -o feature@filesystem_limits=enabled \
> -o feature@hole_birth=enabled \
> -o feature@large_blocks=enabled \
> -o feature@lz4_compress=enabled \
> -o feature@spacemap_histogram=enabled \
> -o feature@zpool_checkpoint=enabled \
> -O acltype=posixacl -O canmount=off -O compression=lz4 \
> -O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
> -O mountpoint=/boot -R /mnt \
> bpool /dev/nvme0n1p5
ubuntu@ubuntu:~$ zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
bpool 1.88G 444K 1.87G - - 0% 0% 1.00x ONLINE /mnt
rpool 484G 516K 484G - - 0% 0% 1.00x ONLINE /mnt
sd256gpool 240G 63.3G 177G - - 0% 26% 1.00x ONLINE -
建立 ZFS Dataset 容器
ubuntu@ubuntu:~$ sudo zfs create -o canmount=on -o mountpoint=/ rpool/ROOT
ubuntu@ubuntu:~$ sudo zfs create -o canmount=on -o mountpoint=/home rpool/HOME
ubuntu@ubuntu:~$ sudo zfs create -o canmount=on -o mountpoint=/boot bpool/BOOT
ubuntu@ubuntu:~$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
bpool 516K 1.75G 96K /mnt/boot
bpool/BOOT 96K 1.75G 96K /mnt/boot
rpool 704K 469G 96K /mnt
rpool/HOME 96K 469G 96K /mnt/home
rpool/ROOT 104K 469G 104K /mnt
sd256gpool 63.3G 169G 24K /media/SD256GB
sd256gpool/Pictures 38.2G 169G 38.2G /media/SD256GB/Pictures
sd256gpool/backup 25.1G 169G 27K /media/SD256GB/backup
sd256gpool/backup/BOOT 185M 169G 185M /media/SD256GB/backup/BOOT
sd256gpool/backup/HOME 19.0G 169G 17.6G /media/SD256GB/backup/HOME
sd256gpool/backup/ROOT 5.89G 169G 5.04G /media/SD256GB/backup/ROOT
ubuntu@ubuntu:~$ mount
...
sd256gpool on /media/SD256GB type zfs (rw,xattr,noacl)
sd256gpool/Pictures on /media/SD256GB/Pictures type zfs (rw,xattr,noacl)
sd256gpool/backup on /media/SD256GB/backup type zfs (rw,xattr,noacl)
sd256gpool/backup/HOME on /media/SD256GB/backup/HOME type zfs (rw,xattr,noacl)
sd256gpool/backup/ROOT on /media/SD256GB/backup/ROOT type zfs (rw,xattr,noacl)
sd256gpool/backup/BOOT on /media/SD256GB/backup/BOOT type zfs (rw,xattr,noacl)
rpool/ROOT on /mnt type zfs (rw,relatime,xattr,posixacl)
rpool/HOME on /mnt/home type zfs (rw,relatime,xattr,posixacl)
bpool/BOOT on /mnt/boot type zfs (rw,nodev,relatime,xattr,posixacl)
ubuntu@ubuntu:~$ sudo rsync -av --info=progress2 --no-inc-recursive \
> --human-readable /media/SD256GB/backup/ROOT/ /mnt
building file list ...
...
...
...
var/tmp/systemd-private-96661d6909d9419ba0c1e66f7193125f-upower.service-wvp5Ng/
var/tmp/systemd-private-96661d6909d9419ba0c1e66f7193125f-upower.service-wvp5Ng/tmp/
8.45G 99% 44.37MB/s 0:03:01 (xfr#161021, to-chk=0/216830)
sent 8.47G bytes received 3.23M bytes 46.15M bytes/sec
total size is 8.45G speedup is 1.00
ubuntu@ubuntu:~$ zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
bpool 1.88G 624K 1.87G - - 0% 0% 1.00x ONLINE /mnt
rpool 484G 5.54G 478G - - 0% 1% 1.00x ONLINE /mnt
sd256gpool 240G 63.3G 177G - - 0% 26% 1.00x ONLINE -
ubuntu@ubuntu:~$ zpool status rpool
pool: rpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
nvme0n1p6 ONLINE 0 0 0
errors: No known data errors
拷貝 BOOT
ubuntu@ubuntu:~$ sudo rsync -av --info=progress2 --no-inc-recursive \
> --human-readable /media/SD256GB/backup/BOOT/ /mnt/boot
building file list ... done
./
System.map-5.4.0-31-generic
4.74M 2% 106.81MB/s 0:00:00 (xfr#1, to-chk=300/302)
...
...
grub/x86_64-efi/zstd.mod
207.41M 99% 39.84MB/s 0:00:04 (xfr#293, to-chk=0/302)
sent 207.48M bytes received 5.61K bytes 37.73M bytes/sec
total size is 207.41M speedup is 1.00
ubuntu@ubuntu:~$ zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
bpool 1.88G 187M 1.69G - - 0% 9% 1.00x ONLINE /mnt
rpool 484G 5.54G 478G - - 0% 1% 1.00x ONLINE /mnt
sd256gpool 240G 63.3G 177G - - 0% 26% 1.00x ONLINE -
ubuntu@ubuntu:~$ zpool status bpool
pool: bpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
nvme0n1p5 ONLINE 0 0 0
errors: No known data errors
拷貝 HOME
ubuntu@ubuntu:~$ sudo rsync -av --info=progress2 --no-inc-recursive \
> --human-readable /media/SD256GB/backup/HOME/ /mnt/home
building file list ...
...
...
...
rick/toolchains/gcc-arm-8.3-2019.03-x86_64-arm-eabi/share/man/man7/gpl.7
27.77G 99% 23.46MB/s 0:18:48 (xfr#337485, to-chk=0/359204)
sent 27.80G bytes received 6.48M bytes 23.58M bytes/sec
total size is 27.77G speedup is 1.00
ubuntu@ubuntu:~$ zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
bpool 1.88G 187M 1.69G - - 0% 9% 1.00x ONLINE /mnt
rpool 484G 24.3G 460G - - 0% 5% 1.00x ONLINE /mnt
sd256gpool 240G 63.4G 177G - - 0% 26% 1.00x ONLINE -
ubuntu@ubuntu:~$ zpool status rpool
pool: rpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
nvme0n1p6 ONLINE 0 0 0
errors: No known data errors
安裝 GRUB
準備 chroot
ubuntu@ubuntu:~$ sudo su
root@ubuntu:/home/ubuntu# for d in proc sys dev; do mount --rbind /$d /mnt/$d; done
root@ubuntu:/home/ubuntu# chroot /mnt
root@ubuntu:/# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/nvme0n1p5 during installation
#UUID=123f16e7-43b5-412d-8872-ee464f237c12 / btrfs defaults,subvol=@ 0 1
# /boot/efi was on /dev/nvme0n1p1 during installation
UUID=0E18-3C17 /boot/efi vfat umask=0077 0 1
# /home was on /dev/nvme0n1p5 during installation
#UUID=123f16e7-43b5-412d-8872-ee464f237c12 /home btrfs defaults,compress=lzo,subvol=@home 0 2
#/swapfile none swap sw 0 0
/dev/disk/by-id/wwn-0x5000cca77ff2de2b /media/brtbkhd btrfs noatime,nodiratime,compress=lzo,discard,space_cache,nosuid,nodev,nofail,noauto,x-gvfs-show 0 0
/dev/disk/by-id/usb-PNY_USB_3.0_FD_7FF55046ABD74833890F0B63E0BFD5-0:0-part1 /media/PNY-USB30 btrfs noatime,nodiratime,compress=lzo,discard,space_cache,nosuid,nodev,nofail,noauto,x-gvfs-show 0 0
/dev/disk/by-id/usb-PNY_USB_3.0_FD_7FF55046ABD74833890F0B63E0BFD5-0:0 /media/PNY-USB30 btrfs noatime,nodiratime,compress=lzo,discard,space_cache,nosuid,nodev,nofail,noauto,x-gvfs-show 0 0
root@ubuntu:/# ls /boot/efi/
root@ubuntu:/# mount /boot/efi
root@ubuntu:/# ls -l /boot/efi
total 12
drwx------ 5 root root 4096 Oct 16 2019 EFI
drwx------ 3 root root 4096 Jan 16 11:57 McAfee
-rwx------ 1 root root 68 Oct 15 2019 _SMSTSVolumeID.7159644d-f741-45d5-ab29-0ad8aa4771ca
檢查 grub-probe 結果
root@ubuntu:/# grub-probe /boot
zfs
更新 initrd
root@ubuntu:/# update-initramfs -c -k all
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = "zh_TW.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
update-initramfs: Generating /boot/initrd.img-5.4.0-31-generic
update-initramfs: Generating /boot/initrd.img-5.4.0-33-generic確認 /etc/default/grub 的 GRUB_CMDLINE_LINUX_DEFAULT 包含 init_on_alloc=0
root@ubuntu:/# cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash init_on_alloc=0"
GRUB_CMDLINE_LINUX=""
更新 grub
root@ubuntu:/# update-grub
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
cannot open 'bpool/BOOT/ROOT': dataset does not exist
Found linux image: vmlinuz-5.4.0-31-generic in rpool/ROOT
Found initrd image: initrd.img-5.4.0-31-generic in rpool/ROOT
Found linux image: vmlinuz-5.4.0-33-generic in rpool/ROOT
Found initrd image: initrd.img-5.4.0-33-generic in rpool/ROOT
device-mapper: reload ioctl on osprober-linux-nvme0n1p6 failed: Device or resource busy
Command failed.
grub-probe: error: cannot find a GRUB drive for /dev/sda1. Check your device.map.
Adding boot menu entry for UEFI Firmware Settings
done
root@ubuntu:/# grub-install --target=x86_64-efi --efi-directory=/boot/efi \
> --bootloader-id=ubuntu --recheck --no-floppy
Installing for x86_64-efi platform.Installation finished.
No error reported.
先要確認 /etc/zfs/zfs-list.cache 目錄下有要自動掛載的 pool 檔案，而且內容非空白。
root@ubuntu:/# cat /etc/zfs/zfs-list.cache/bpool
bpool /boot off on on off on off on off -none
bpool/BOOT /boot on on on off on off on off - none
root@ubuntu:/# cat /etc/zfs/zfs-list.cache/rpool
rpool / off on on on on off on off - none
rpool/HOME /home on on on on on off on off - none
rpool/ROOT / on on on on on off on off - none
root@ubuntu:/# cat /etc/zfs/zfs-list.cache/sd256gpool
sd256gpool /media/SD256GB on on off on on off on off - none
sd256gpool/Pictures /media/SD256GB/Pictures on on off on on off on off - none
sd256gpool/backup /media/SD256GB/backup on on off on on off on off - none
sd256gpool/backup/BOOT /media/SD256GB/backup/BOOT on on off on on off on off- none
sd256gpool/backup/HOME /media/SD256GB/backup/HOME on on off on on off on off- none
sd256gpool/backup/ROOT /media/SD256GB/backup/ROOT on on off on on off on off- none
現在可以離開 chroot 環境，最後就是安全解除掛載 zpool。
root@ubuntu:/# exit
exit
root@ubuntu:/home/ubuntu# mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
root@ubuntu:/home/ubuntu# zpool export -a
root@ubuntu:/home/ubuntu# 大工告成！如果一切正常，重新開機後應該要看到 grub 選單，然後開機進入 Ubuntu 桌面～
