LVM Transaction ID mismatch and metadata resize error

I nearly lost my LVM thin pool on my laptop recently due to the metadata nearly filling up (99.4%). I first noticed this when trying to install something and the root filesystem was read only. Rebooting the machine re-mounted the root partition as RW. Trying to get this pool into a healthy shape gave me all sorts of transaction id and meta errors so hopefully this post will assist someone.

My LVM Layout

I use LVM on my Fedora 27 system along with encryption; it is probably useful to share the layout so the rest of the blog makes sense.

Using the LVS command:

Unfortunately I don't have any screenshots from the unhealthy system but the layout is the same.

Running "lvs -a" will show you the hidden meta data volumes:

The meta data is currently 220m but it was only 40m before with the meta percent around 99.3.

Extending the MetaData

I don't have the exact command I used as I was on a LIVE CD but if the syntax isn't quite right then run '--help' with the command. You will will need to do this from a live CD but remember to unlock the disk if it is encrypted otherwise the LVM won't show.

I found a few ways to increase the metadata size on the internet but I found this worked well. This will increase the metadata by an additional 128M.

lvextend vg-fed/thin-fed --poolmetadatasize -L+128M

The next step would be to try and run the repair utility HOWEVER this didn't work for me; it listed some errors about can't automatically run without a spare. If this is the case then you will have to do it manually like I did.

lvconvert --repair vg-fed/thin-fed

Manually extending MetaData

This should work if the metadata is recoverable.

Create a temporary small logical volume to swap the metadata to.

lvcreate -an -Zn -L200M --name temp vg-fed

Now you need to swap the pools metadata with the temp volume.

lvconvert --thinpool vg-fed/thin-fed --poolmetadata temp

Create another volume for the repaired metadata to be stored.

lvcreate -L220M --name repaired vg-fed

Then run the repair command.

thin_repair -i /dev/vg-fed/temp -o /dev/vg-fed/repaired

Now swap back.

lvchange -an vg-fed/repaired

lvconvert --thinpool vg-fed/thin-fed --poolmetadata repaired

Now you should see something similar to the above screen shots when running 'lvs -a'.

The pool should be back and running. Unfortunately I got transaction id mismatch errors when trying to activate the thin pool. This was likely due to the fact I tried to remove two thin snapshots when the metadata was nearly full which threw an error. The volumes disappeared from the volume list but there was likely a mismatch between the kernel and LVM.

Fixing Transaction ID Mismatch

The error will probably state something like "Was expecting transaction id 40 but got transaction id 42".

Backup the lvm volume configuration to a file which can be edited.

vgcfgbackup vg-fed -f /home/<user>/backup

Now open the file in a text editor and find the transaction id against the thin pool, change this to the correct value and save the file.

Restore this backup the LVM configuration.

vgcfgrestore vg-fed -f /home/<user>/backup

Now you can activate the 'root' volume

lvchange -ay vg-fed/root

Fingers crossed; it should work.

References

https://unix.stackexchange.com/questions/290432/extend-thin-lvm-metadata-size

https://www.redhat.com/archives/linux-lvm/2014-October/msg00032.html