So, for my day 4 I have a bunch more feedback to report:
First, I'll comment that I actually made the change I described above, and it appears to have greatly reduced the amount of servant-switching that was going on. Of course, I may also have introduced new bugs, so if (when...) I report...