Klara, ZFS, rubber duckies
Do you like ZFS? What about ducks?
Klara Systems hosts free webinars and a knowledge base on OpenZFS and other topics, and you should check them out!
1. Background
Allan Jude is a co-founder of Klara Systems and a FreeBSD and OpenZFS developer. He and Klara offer design, implementation and support services, particularly in the storage systems domain. I’ve mentioned Allan in my recent 2.5 Admins podcast plug and I will surely mention him again.
Author Michael W. Lucas co-hosted with Allan. I’ve enjoyed Michael’s writing, specifically his book SSH Mastery: OpenSSH, PuTTY, Tunnels and Keys. I loved hearing from Michael in this webinar form and appreciated his insights.
2. Notes from the webinar
2.1 ZFS Basecamp
Klara Systems offers a ZFS Basecamp, free for all. Go there; learn things.
2.2 On change management and monitoring
As part of his prepared remarks, Allan suggested the following:
- Consistency in your documentation: note cause/effect, remember the little stuff
- Test your DR process - do you have a backup or a waste of time?
- Ensure snapshots include a timestamp and something indicating the system hostname.
- Use a two-person approval process for destructive changes
For me, a solo flyer, the two-person rule takes the form of a stand-in. For guidance I turn to the wisdom of my partner when available (wise indeed but alas, Not An IT Person) or my literal rubber duckies (reflections of my inner greybeard). Behold;
This very moment, I learned how to invoke CSS properties in JSX. Like this:
<div style={{border: "1px solid gray", borderRadius: "1rem", padding: "1.5rem 1.5rem 0 1.5rem", margin: "2rem"}}>
<h4 style={{textAlign: "center", font: "italic 2rem light serif"}}>~ duckies ~</h4>
<p style={{textAlign: "center", fontStyle: "italic", margin: "1rem"}}>Vulfpeck duck painting by [Rae Whiteley](https://raewhiteley.com/).<br />
Bunny pickle painting by [Will Quinn](https://willquinnart.com/).</p>
</div>I have total faith that this horrific JSX-with-inline-CSS-and-a-markdown-image-call is going to break in a most heinous way, probably soon! But right now it works.
2.3 ZFS checkpoints
And speaking of destructive changes, this talk opened my eyes to ZFS checkpoints. I had no idea these were a thing. Read over this article to get a sense for what they are and how they work.
In very short:
-
Checkpoints ARE:
- Immutable
- A point-in-time image of a pool’s state
- A hedge against breaking changes
- Singular (one checkpoint per pool)
-
Checkpoints NOT:
- A snapshot of a dataset or its children
-
Checkpoints CAN:
- Be automated
- Save you from yourself
Good heavens!
I’ll be testing this out next time I make a change to a dataset.
Since I’m architecting my own home storage on the fly, there’s a lot of learning-by-doing and changes occurring as I revise, refine, re-work. I don’t usually worry about making potentially destructive changes, since I have local snapshots, cloud replication, local cold backups, and off-site cold backups. But I will be using cron-scheduled checkpoints from now on as a best practice.
Klara, says Allan, uses Prometheus to compare the latest snapshot time against the current time and, if I understood him right, to graph the difference over a certain time horizon. This alerts them to missed snapshots and trends in increasing or decreasing snapshot time-to-completion.
On monitoring, Michael stated that
“Monitoring is terrible, there is no good solution; it’s also absolutely necessary. You take your best effort and stop worrying about it until you discover a new failure.”
2.4 Michael on administration
Michael says on his website that
”[ … ] computers were a mistake.”
In the context of problem-solving and operational awareness as a sysadmin, he expanded on that statement during the webinar:
“[The first rule of systems administration is that] computers were a mistake. There are an infinite variety of mistakes, and you have to dig deep.”
This made me feel very seen.
In my previous workplace I was a systems engineer. There was a Bad Day, and I said to my boss “I think computers were a mistake. We should go back to smoke signals.” My boss laughed at me, but I wasn’t entirely joking, and I told her as much. The longer I work in this field the more convinced I become, and the more I aim to reduce my time spent with technology.
The time I invest is to make technology work for me so that when I’m 30, I can spend more time reading and gardening and less time typing.
3. Psychological awareness
Many of us deal with impostor syndrome. Many of us deal with anxiety. For some of us these psychological mechanisms are part of what motivate us to reach higher; part of what helps us see problems before they occur.
Michael’s words remind us that even in high-accountability, high-information roles:
- Nobody is infalliable no matter how broad or deep their expertise.
- There are always unknown unknowns.
- If we don’t acknowledge our limitations, we will be blindsided when those unknown unknowns manifest.
Thanks, Michael 💛
Originally published 19 November, 2025.