IronTetsubo
Member
Thanks! I will play further with it tonight.
Feature - Implemented "modernize" item manager tables
- Thread starter roippi
- Start date
Thanks! I will play further with it tonight.
Ensiferum
Member
I am probably in the minority with a pretty old computer - 3.33 GHz Celeron D, 512 RAM, Win Xp, but the new handling of items in the storage makes mafia completely unusable.
Every attempt to type in or delete a filter leads to a complete lockup - from 30 seconds to 5 minutes at time. In fact twice it took even longer so I shut down the process from the task manager.
Any chance to make the new item tables optional? Or am I forever stuck using old builds pre-dating the change?
Every attempt to type in or delete a filter leads to a complete lockup - from 30 seconds to 5 minutes at time. In fact twice it took even longer so I shut down the process from the task manager.
Any chance to make the new item tables optional? Or am I forever stuck using old builds pre-dating the change?
roippi
Developer
I'd rather improve the performance of filtering.
roippi
Developer
That would be just as great, as long as it's possible.
In the meantime, if you need to find something and need something that performs better, ctrl-f brings up the JXTable's search. It's nowhere near as slick as the filter field, but it should be way faster for you.
bordemstirs
Member
I -love- you.
I've wanted adv/fullness info on my creatables tab for so long, and now I have it! If you could add that to storage/closet (mostly storage...) that would be awesome.
Now, I'm not sure what would be "better" but it strikes me as odd that the quantity field for creatable items would consider just purchasing items (71 thousand bottles of vodka, oh my...) but I can't think of a more-honest or apt way to display that info without disabling Purchase Items from Mall when necessary.
Anyway, this update is better than sex. Thank you.
I've wanted adv/fullness info on my creatables tab for so long, and now I have it! If you could add that to storage/closet (mostly storage...) that would be awesome.
Now, I'm not sure what would be "better" but it strikes me as odd that the quantity field for creatable items would consider just purchasing items (71 thousand bottles of vodka, oh my...) but I can't think of a more-honest or apt way to display that info without disabling Purchase Items from Mall when necessary.
Anyway, this update is better than sex. Thank you.
roippi
Developer
Ensiferum- see if 11046 performs any better for you when filtering?
Can you clarify what you mean/ are reporting?
Can you clarify what you mean/ are reporting?
Bale
Minion
bordemstirs is referring to the fact that in the createable tabs, the quantity field gives the amount of each item that includes purchasing items from NPCs. Since he is capable of purchasing bowls of rye sprouts it says that he can create 71 thousand bottles of vodka.
He would like a field that gives the amount he can create without purchases. I'm not sure that number is actually any sort of meaningful so I do not agree with him that it is actually valuable.
He might be confused and think that the 71,000 figure means he is purchasing bottles from the mall instead of rye sprouts from an NPC.
He would like a field that gives the amount he can create without purchases. I'm not sure that number is actually any sort of meaningful so I do not agree with him that it is actually valuable.
He might be confused and think that the 71,000 figure means he is purchasing bottles from the mall instead of rye sprouts from an NPC.
The "createable" tab shows how many of the item you can create - not purchase from the mall or an NPC.
It does include things that you can "purchase" from a coinmaster, since those count as a "creation". That might be a bit odd, but it is required for things like lunar food, coffee pixie sticks, items crafted out of Tr4pz0r skins when all you have is yeti skins, items made with clovers or jabanero peppers from the Hermit, and so on, to appear as "creatable" without having the direct ingredients already in inventory. I think that we can live with that. I think we can't live without it, at this point.
I don't think it's a bug that the "creatable" count includes ingredients you can purchase from NPCs if you have autoSatisfyWithNPCs checked.
And if you see "bottle of vodka" listed on "creatables", I think it's a user error to assume that that means "directly purchase from ... somewhere" rather than "create from ... ingredients".
It does include things that you can "purchase" from a coinmaster, since those count as a "creation". That might be a bit odd, but it is required for things like lunar food, coffee pixie sticks, items crafted out of Tr4pz0r skins when all you have is yeti skins, items made with clovers or jabanero peppers from the Hermit, and so on, to appear as "creatable" without having the direct ingredients already in inventory. I think that we can live with that. I think we can't live without it, at this point.
I don't think it's a bug that the "creatable" count includes ingredients you can purchase from NPCs if you have autoSatisfyWithNPCs checked.
And if you see "bottle of vodka" listed on "creatables", I think it's a user error to assume that that means "directly purchase from ... somewhere" rather than "create from ... ingredients".
Ensiferum
Member
Didn't see a link to 11046 available, so downloaded 11048.
Seems a lot better and the time filtering takes now is about 5 to 10 seconds. Clearing already entered strings is still much slower, most likely because it leads to displaying all the items in the storage. At least nothing locks up for minutes.
Will try to get used to this for a while I suppose. There's always the option to revert to an older build with an instant filter.
Either way - thanks for the quick response.
Bale
Minion
True, but older builds will not support new features and will fail once KoL changes something that breaks mafia. I don't think that old builds are a really good solution to any problem, other than waiting for a bug to be fixed.
roippi
Developer
Again with the fatalism. Geez. I'm still developing this feature, so how about we don't throw up our hands and give up on it entirely.
Humor me and sort your storage by a column that is just an integer value - like quantity - and then try filtering things. Is it faster when you do that?
Ensiferum
Member
Sorry, fatalism is just my nature. I wouldn't be able to count how many times I've been rebuked about that. 
Back on topic, though - sorting by columns doesn't seem to affect it much.
Builds 11049/50 are quite an improvement, but I still suspect it's mainly memory shortage or something of the like on my end. Filtering is instant when I start Mafia and attempt a few searches, but after those 5-6 filters or about 30 seconds later it slows back down. The slowest thing remains clearing the filter when it has to list everything in storage (no idea how to see amount of unique items).
Rest of the filtering is good enough, with the occasional spike up to 10 seconds.
Thumbs up!
Back on topic, though - sorting by columns doesn't seem to affect it much.
Builds 11049/50 are quite an improvement, but I still suspect it's mainly memory shortage or something of the like on my end. Filtering is instant when I start Mafia and attempt a few searches, but after those 5-6 filters or about 30 seconds later it slows back down. The slowest thing remains clearing the filter when it has to list everything in storage (no idea how to see amount of unique items).
Rest of the filtering is good enough, with the occasional spike up to 10 seconds.
Thumbs up!
roippi
Developer
Sounds like there's insufficient heap space, yes. AKA not enough memory available to the application. Without knowing your JRE and whatnot, I'd guess Java is allocating somewhere around 64 MB for your heap - which is just a few MB more than my readout tells me I use with the item manager open. Look at the upper right corner of the main frame and see what the numbers there are.
If you want to temporarily free up some memory, click the little garbage can next to that display. Java will probably gobble up the memory again after a minute or so, but if that temporarily fixes performance for you then you know what the problem is.
(This is a little outside of the scope of this thread, but you can increase java's available heap size by adding an -Xmx flag to the shortcut you use to launch mafia. I'd recommend -Xmx256m for your system.)
If you want to temporarily free up some memory, click the little garbage can next to that display. Java will probably gobble up the memory again after a minute or so, but if that temporarily fixes performance for you then you know what the problem is.
(This is a little outside of the scope of this thread, but you can increase java's available heap size by adding an -Xmx flag to the shortcut you use to launch mafia. I'd recommend -Xmx256m for your system.)
When I see slowness with that filtering (like right now, where it isn't instantaneous but it isn't nearly as bad as the 4 second wait at the moment), memory usage is only getting to around 80 MB out of 256 MB. Unless it's briefly spiking up a huge amount, and then garbage collection cleans it up before that display gets to update.
Ensiferum
Member
JRE is 1.6.0.26 if I'm looking at the right place to figure that out. Do I need an update to Java 7?
Garbage collection doesn't change anything at all.
Adding the flag to the shortcut leads to an Unable to invoke -Xmx256m message.
I've always been curios if there was an option to change the heap size, since default is set to 124736 KB. I think this is the proper way to add the flag? "D:\Nano\Kingdom Of Loathing\KoLmafia-11050.jar" -Xmx256m
Garbage collection doesn't change anything at all.
Adding the flag to the shortcut leads to an Unable to invoke -Xmx256m message.
I've always been curios if there was an option to change the heap size, since default is set to 124736 KB. I think this is the proper way to add the flag? "D:\Nano\Kingdom Of Loathing\KoLmafia-11050.jar" -Xmx256m
roippi
Developer
No, don't update to java 7.
Try instead setting that flag as a runtime parameter in control panel > java. Caveat, I have no idea if that exists on xp.
Theraze
Active member
Try this on the command line:
D:
cd "D:\Nano\Kingdom Of Loathing"
javaw -jar KoLmafia-11050.jar -Xmx256m
roippi
Developer
For the more technically-minded, a breakdown of the issues involved here in further optimization:
Your inventory (and storage, and etc) exist in mafia as a LockableListModel. This is the "data model." When you type something into the filter field, that operates on the data model to set each individual element's visibility. Each time something has its visibility changed, it fires an event that causes anything listening to the model to update() itself.
Now, in a list of some thousands, when you type something into the filter field, you can imagine that's a lot of events firing off. Prior to 11046, item manager tables were trying to update after each individual event. Updating involves calling a sorter to sort the rows according to the selected column (default: item name column). More on that later, but suffice to say that doing it thousands of times was unnecessary - now we just fire an event after the data model has been filtered, and the table can update its display once.
Re: sorting. Sorting a list of comparable items is a thing you can take a class or three on. There's a lot of different ways to do it. But essentially there are two salient issues to discuss here.
One, the sorting algorithm. Swingx uses ShuttleSorter, which if you're curious you can inspect yourself here. The sort() algorithm is too recursive for me to parse at a glance, but it claims n log
worst-case performance. Which is curious since shuttle sort is generally considered to be O(n^2), not O(n logn). I'll do more research, but I'm not sure why it's not using Java's internal collection sorting, which uses merge sort.
Two, the comparator. For a list of things to be sorted, every element in said list must implement Comparable - that is, a.compareTo(b) and b.compareTo(a). It is important that we know what class a and b are - 100 compareto( 1000 ) is different than "100" compareto ( "1000" ). Likewise, if strings are mixed in with integers (see: autosell column), everything has to be treated as a string. What all this means is that you have to implement your own Comparator if you want things to be sorted properly. I've done this for two columns - item name and autosell. I need to do this for item name because the text actually has HTML markup in there, and we don't want font color=blah to influence our sorting. The relevant code:
Both employ regex, which has performance implications.
What all this means
Sorting a few thousand items takes a while. Sorting integers is faster than sorting strings. Regex slows things down. Some numbers (specific to my system, which is really fast. Also only 1400 items in storage.)
10 trials of sorting "quantity", a purely integer column
10 trials of sorting "autosell", an integer column with 2 regex
10 trials of sorting "item name", a column with 2 regex then a string compare
All numbers in ns, so int sorts took around 1.5 ms, autosell sorts took around 15 ms, and item name sorts bounced between 15 and 60. Note that every other sort is a "descending" sort, so for whatever reason the shuttle algorithm performs poorly on descending string sorts. I'm looking into ways ways to optimize either sort() or the comparator, though I think the sort() algorithm is the better target.
Your inventory (and storage, and etc) exist in mafia as a LockableListModel. This is the "data model." When you type something into the filter field, that operates on the data model to set each individual element's visibility. Each time something has its visibility changed, it fires an event that causes anything listening to the model to update() itself.
Now, in a list of some thousands, when you type something into the filter field, you can imagine that's a lot of events firing off. Prior to 11046, item manager tables were trying to update after each individual event. Updating involves calling a sorter to sort the rows according to the selected column (default: item name column). More on that later, but suffice to say that doing it thousands of times was unnecessary - now we just fire an event after the data model has been filtered, and the table can update its display once.
Re: sorting. Sorting a list of comparable items is a thing you can take a class or three on. There's a lot of different ways to do it. But essentially there are two salient issues to discuss here.
One, the sorting algorithm. Swingx uses ShuttleSorter, which if you're curious you can inspect yourself here. The sort() algorithm is too recursive for me to parse at a glance, but it claims n log
worst-case performance. Which is curious since shuttle sort is generally considered to be O(n^2), not O(n logn). I'll do more research, but I'm not sure why it's not using Java's internal collection sorting, which uses merge sort.Two, the comparator. For a list of things to be sorted, every element in said list must implement Comparable - that is, a.compareTo(b) and b.compareTo(a). It is important that we know what class a and b are - 100 compareto( 1000 ) is different than "100" compareto ( "1000" ). Likewise, if strings are mixed in with integers (see: autosell column), everything has to be treated as a string. What all this means is that you have to implement your own Comparator if you want things to be sorted properly. I've done this for two columns - item name and autosell. I need to do this for item name because the text actually has HTML markup in there, and we don't want font color=blah to influence our sorting. The relevant code:
Code:
private final Pattern meatPattern = Pattern.compile( "(-?\\d+) meat" );
private final Pattern itemPattern = Pattern.compile( "(?<=^|>)[^><]+?(?=<|$)" );
protected final Comparator<Object> meatComparator = new Comparator<Object>()
{
public int compare( Object o1, Object o2 )
{
Matcher matcher1 = meatPattern.matcher( o1.toString() );
Matcher matcher2 = meatPattern.matcher( o2.toString() );
if ( !matcher1.find() )
{
return -1;
}
else if ( !matcher2.find() )
{
return 1;
}
// if we're here, both strings are in the format (\d+ meat)
Integer o1val = Integer.valueOf( matcher1.group( 1 ) );
Integer o2val = Integer.valueOf( matcher2.group( 1 ) );
return o1val.compareTo( o2val );
}
};
protected final Comparator<Object> itemComparator = new Comparator<Object>()
{
public int compare( Object o1, Object o2 )
{
Matcher matcher1 = itemPattern.matcher( o1.toString() );
Matcher matcher2 = itemPattern.matcher( o2.toString() );
if ( !matcher1.find() )
{
return -1;
}
else if ( !matcher2.find() )
{
return 1;
}
// if we're here, both matches are the item names stripped of HTML.
String o1val = matcher1.group();
String o2val = matcher2.group();
return o1val.toLowerCase().compareTo( o2val.toLowerCase() );
}
};Both employ regex, which has performance implications.
What all this means
Sorting a few thousand items takes a while. Sorting integers is faster than sorting strings. Regex slows things down. Some numbers (specific to my system, which is really fast. Also only 1400 items in storage.)
10 trials of sorting "quantity", a purely integer column
Code:
sort took 1569391
sort took 1625041
sort took 1550601
sort took 1619622
sort took 1544458
sort took 1617453
sort took 1554214
sort took 1619260
sort took 1541929
sort took 162504110 trials of sorting "autosell", an integer column with 2 regex
Code:
sort took 15966023
sort took 15342314
sort took 16096475
sort took 16538781
sort took 14729807
sort took 16539504
sort took 14815450
sort took 16760295
sort took 15987705
sort took 1528630310 trials of sorting "item name", a column with 2 regex then a string compare
Code:
sort took 79332161
sort took 15465538
sort took 55983889
sort took 15416754
sort took 58153141
sort took 15402299
sort took 58334182
sort took 15449276
sort took 64393127
sort took 15429763All numbers in ns, so int sorts took around 1.5 ms, autosell sorts took around 15 ms, and item name sorts bounced between 15 and 60. Note that every other sort is a "descending" sort, so for whatever reason the shuttle algorithm performs poorly on descending string sorts. I'm looking into ways ways to optimize either sort() or the comparator, though I think the sort() algorithm is the better target.