So while looking at this PR I was a bit surprised to find out that my 150kb prefs file is costing me about 5.4gb of writes a day from my 33k pref changes. We currently write the whole file out every time we change one value. To me that feels unacceptable, I want my drives to last longer than this.
The first thing that comes to mind is changing the settings database format entirely. That's not going to convince anyone and introduces too much complexity, so that's off the list. We're sticking with the
So, to fix the I/O problem without breaking backward compatibility, we're given a choice: write less often, or write smaller.
I think there's three options.
Periodic Saves (Writing Less Often)
Instead of writing to the disk immediately on every single change, we just update the properties in memory and write the file periodically.
For example, we only flush the changes to
The Upside: This is simple. It requires zero changes to the file format and users interacting with
The downside: Unexpected power loss means you lose whatever prefs changed in that 5 second window. Though, realistically, you'd probably be fine.
Append Delta Log (Writing Smaller)
When a pref changes, we still update it in-memory, but instead of rewriting the entire 80kb+
When the delta log reaches a certain configurable size (e.g 500kb), or when the user logs in / out / exits / scheduled, we flush the current in-memory prefs into
The upside: It drops our daily I/O from prefs alone, from gigabytes down to megabytes, but guarantees no data is lost on a crash.
The downside: It might confuse people's existing setups if they are manually editing
Considerations: There may be some tooling for this, some library or database structure that's not plain text or methodology, I'm not sure. The main idea is that we're appending, it's power loss resistent, and when we load it, we work backwards to apply the last changes and skip any changes that were overwritten later.
So
A callback to temporary prefs
Whichever approach we take, it's worth bringing up a callback to my old temporary prefs proposal. A huge chunk of these I/O writes are for prefs that simply aren't important across sessions. things like
Closing thoughts
We could probably expose more tooling that helps minimize the amount of IO, such as appending to a file.
The first thing that comes to mind is changing the settings database format entirely. That's not going to convince anyone and introduces too much complexity, so that's off the list. We're sticking with the
key=value flat file format.So, to fix the I/O problem without breaking backward compatibility, we're given a choice: write less often, or write smaller.
I think there's three options.
Instead of writing to the disk immediately on every single change, we just update the properties in memory and write the file periodically.
For example, we only flush the changes to
prefs.txt every 5 seconds, or wait until a script finishes running.The Upside: This is simple. It requires zero changes to the file format and users interacting with
prefs.txt won't notice any difference 99% of the time. It collapses rapid writes (like combat) into a single write.The downside: Unexpected power loss means you lose whatever prefs changed in that 5 second window. Though, realistically, you'd probably be fine.
Append Delta Log (Writing Smaller)
When a pref changes, we still update it in-memory, but instead of rewriting the entire 80kb+
prefs.txt file, we just append key=value\n to a small irrat_prefs_delta.log file.When the delta log reaches a certain configurable size (e.g 500kb), or when the user logs in / out / exits / scheduled, we flush the current in-memory prefs into
prefs.txt and clear the log. When logging in, we load prefs.txt and apply the log if it exists, over it.The upside: It drops our daily I/O from prefs alone, from gigabytes down to megabytes, but guarantees no data is lost on a crash.
The downside: It might confuse people's existing setups if they are manually editing
prefs.txt directly while KoLmafia is closed, or their special backup / copying setups. Because of that, we should probably make this opt-in.Considerations: There may be some tooling for this, some library or database structure that's not plain text or methodology, I'm not sure. The main idea is that we're appending, it's power loss resistent, and when we load it, we work backwards to apply the last changes and skip any changes that were overwritten later.
Code:
lastRefreshed=10
lastRefreshed=20
lastRefreshed=30lastRefreshed=30 is stored in memory, the log is cleared.A callback to temporary prefs
Whichever approach we take, it's worth bringing up a callback to my old temporary prefs proposal. A huge chunk of these I/O writes are for prefs that simply aren't important across sessions. things like
_concoctionDatabaseRefreshes, or _lastCombatActions. If we allow some prefs to persist only in memory and skip writing them to disk entirely, we could lower the amount of writes. Inter-script communication can be solved with temporary prefs, a major reason I've never explored that in my scripts is because every change is written to file, both session and prefs.Closing thoughts
We could probably expose more tooling that helps minimize the amount of IO, such as appending to a file.
Last edited:
