Replicated datasets change on reboot

Hi,

Long-time Unix and FreeBSD user. A few years using FreeNAS. I’m running version TrueNAS-12.0-U8.1 on a SuperMicro server.

I hoped to use FreeNAS to replicate three other UNIX systems I run. I didn’t use the GUI, as it seemed dedicated to FreeNAS->FreeNAS replication, and only one of these servers is a FreeNAS installation.

The problem I have found in my tests is that the datasets seem to be changed on a reboot, which causes incremental replications to fail or require -F, which I am loath to use. I have found this to be quite repeatable:

freenas# zfs create zroot/BACKUP

source# zfs send -Rpv zroot/ROOT@230902 | ssh freenas zfs recv -d zroot/BACKUP

If I do an immediate incremental send:

source# zfs send -Rpi @230902 zroot/ROOT@230911 | ssh freenas zfs recv -d zroot/BACKUP

all is well. But if I logout and do a clean shutdown, the incremental replication fails with the “filesystem has changed, use -F” message.

I have prevented this by setting the affected datasets to readonly=on. I’ve never had to do that before, so I’m wondering if I’m doing something wrong or what could be causing this.

I’ve considered switching to a straight-up FreeBSD system, but I have the impression the drivers and software for this sort of thing are better, and more complete from iX systems.

So I guess that’s two questions: Why is it happening? Is there a way to stop it or should I just switch to FreeBSD and will that work OK?

Ok, I guess that’s more than two questions. Sorry.

Thanks so much for any advice you can share,
Pammy