I'm starting to put together these algorithms for determining how likely items are to be leeches, based on how long you spent on them, your retention rate, and how tough/important they are. The algorithms have seen some tuning, but I don't think they're where I'd like them to be yet. I'm putting them up on beta anyway so that y'all can check them out and give some feedback about how it should work.

Kinds of feedback I'm looking for:
1) Is this (a quick message at the top of the Flash window) a good way to display the leech info?
2) Is the leech score percentage a good representation?
3) Are they showing up too often, not often enough, or just right?
4) Are the leech scores relevant, or do the algorithms not capture it?
5) What do you want to do with leeches?

Other feedback is welcome, too.

Hmm, just been trying it, and for me I would say it doesn't distinguish between ones that are just proving a little trickier than average to learn, and real leeches which are ones I keep learning then forgetting. (Eg you can look at my history for say jizhe journalist, or indeed any other starred ones, vs ones I've just added in last few days. Maybe nothing should be a leech until a certain threshold of time spent on it.)

I think better might than the flashing message might be a sigil next to the star/info etc, with a press action.

I guess in terms of what to do, it would be similar to starred ones. Become a list-like thing somehow.

But then again I do wonder if it will help learning having the "leech alert" flashed up. If shouldn't be a % though I suspect.

Another question is if everyone thinks leeches are the same thing that I do? I'm mainly concerned with words where I forget what the proper characters are, or their order. I guess some small amount of tones too.

I think I tend to learn them to a certain level, then always forget again around the 1 month interval.

Perhaps there should be a settable leech flag that just prevents something being pushed out to very long intervals at all. I would really welcome that for currently starred ones, actually.

I have just been trying it and it seems OK, the little drop down is discreet and it stays long enough to read and short enough that it doesn't get in the way.

I don't know if the % is relevant, but it's a measure that you have and I guess if you know that you're retention rate is 90% and this word is 50% then it's obvious you don't know it.

As Wispfrog said, there are many new words that would seem to be leeches as they are new and I haven't learnt them yet. Can the leech notification only appear for words over one or two weeks old or after x number of reviews?

I think the important question really is "what do you want to do with the leeches?". I know the words that are my own leeches because every time they come up, I stare blankly at the screen with no clue about what I need to write! For me, I just need to practice them again soon - isn't that what the algorithm does anyway?

I don't use starred words, but maybe the leeches could be added to starred words as ones needing attention? Maybe a second kind of starred list just with leeches that could be studied separately so that you can work on the lists and then spend a last 5 minutes hitting the leeches one extra time.

I admit I have lots of words which you might consider leeches, but telling me before I write every single word that I'm not very good at it gets boring fast.

I think the messages could be less repetitive and more informative. Eg "You've got this word wrong 5 out of the last 6 times you've studied it"/"You're struggling to remember this past the 2 week mark". Make it that messages only come up after you've written the character, and only show every now and again, not every time.

You might want to think about a leech list too, where we can choose to ban them for a while if we want. Maybe if items are on this list, and appear in study, there might be a leech icon beside the word, marking it as a leech.

Whether things should be added automatically to this list, I don't know. Given how many of mine seem to be flagged up, I would be hesitant in doing so at present.

I guess I'm a little confused by the percent. Does a high percent mean it's very leech-like or I am doing well writing it?

Maybe it's already enough if you could sort the list of words in MyWords by % leechness, so you can easily spot them and do something about them!

I do not like the message at the top - it's a bit disturbing for me - nor the percentage as number. For the time being, the number would be OK for test purposes, however, would be nice, if we can get a better interpretation for it, so that we could check, whether this also represents our own perception. We could use the feedback button for a while, to report back, where we see a mismatch and why (might be only a limited number of users at different learning levels, so to get enough feedback, but not too much to be overwhelmed with such info).
The most important question for me would be 5), what is the action, which results out of this information (info without resulting activity has no use). I could imagine the following possibilities:
- Do nothing, as Skritter automatically would schedule more reviews for such item and/or, once we see that this is a leech, we would ourselves put more effort into such vocabulary.
- Put leeches into an extra list for review, until the "leech status" is gone (don't like this, as it should be also part of the normal schedule).
- Give an extra option in the study page like: there are n number of leeches, review xy of them now, where we could study a certain number in order to have some extra effort there and we could choose the number, depending on the time we just could afford for this extra study (would be my preferred choice).
For the presentation, I could imagine not to have a number, but to have a bar at the top, which changes gradually from green to red, depending on this percentage.
What would be really cool is, if we could see this for all items - if Skritter judges them not to be a leech, it's just totally green - plus have a slider, where we could manually increase the "leech degree" depending on our own perception, how good we know this item. The drawback definitely is that it might be complex for the user, especially one who is new to Skritter, and some users might "mess up" their review schedule. However, as for me, I really would love to have a possibility for certain vocabulary, to manually influence the review schedule.

Am I unique in admitting that I'm not certain what the leech score really stands for?

I get that it indicates that a word or character might be taking too much time for its usefulness, but what does the number indicate? Is the percentage a reflection of our historical success on the item or is it an indication of how well the item fits an algorithmic definition of "leech"? i.e. is a high number a positive thing or a negative thing? And additionally, if it's calculated from our success rate with the character, does the algorithm give greater value to more recent study iterations, i.e. if I missed a prompt 100 times over a month, but I've gotten it correct 10 times in a row spanning the second month, is it considered a leech?

As a sort of response to a comment from HappyBlue, the dropdown notification seems too small and quick to me. Maybe it's just the result of a 14.5" screen at 1600x900 resolution and lowish brightness and the 250 pixel study box all conspiring against me, but I'm finding the notification to be either too small or too fast to catch. Another second or another few pixels of height and I'd be fine, but the combination of tiny and swift makes it too much for me.

And I'd second Roland's question: now that we have this information, what's next? I like getting it and for now, at least, having it drop down like this while I'm studying keeps the studying a bit fresher, but where do we go from here?

Thanks for the feedback, guys. I'm going to postpone making it better for right now. I'll leave it on, but only for words with really high leech scores using the existing algorithm, like 75% or more.

What the leech score is trying to do is say, "You're spending too long and forgetting too much on this word for how not important it is, so consider ways to not let it suck up so much time." And then we'd display the message for items which score over a certain threshold, like 60%. But the algorithms clearly need a lot of work.

I'm still not sure which of two routes to go in terms of what to do with leeches:

1) Let you know that the item is a leech at review time, and let you decide whether to ban it, star it, make a mnemonic, study it offline, edit the definition, or whatever. (I would just jump for banning it most of the time. Life's too short.)

2) Build more integrated leech support into the system so that you can have it do things for you automatically, put them in lists, etc.

Now I understand, what this figure stands for - previously I misinterpreted it. In this case, it doesn't make much sense for me, as all the new words and characters, which I am learning, are mostly in the category harder. that's why it shows me a lot of leeches, almost every new word. Therefore, I would prefer, if this would be set as an option in the future and not as a permanent feature.

I think it is a great help. Especially now that it only appears with the % > 75. When a character like this comes up I pay more attention to it. I just try to reflect and concentrate more before I start writing. If I still get it wrong I just repeat the character for another 3 times before moving. I find it helps me addressing some real problems.

Roland, can you give me a few examples of words or characters where it's showing leeches but shouldn't be, so I can check them out?

Nick, I think, it's calculating correctly. It's just, that recently, I'm getting a lot of characters from names, which are in the category of harder or hardest; they are also quite hard for me to learn, as they often don't carry a real meaning. But I believe, this will be the same for everybody, once you get above the 3000 character level.
What might be helpful is, if your formula wouldn't take the Level_word only, but the distance (Level_word - Level_user) - just an idea, don't know, whether this is practical.

Has the leach feature just been rolled out to everyone, not only testers? I just started seeing these messages and I wish I could turn it off. Showing the message before I attempt to answer doesn't seem right: in a way it is an extra hint ("oh, this is a hard item"), and I find the red drop-down message distracting. Perhaps it would be better to show this information after answering. It could go on the same line as last seen/percent due. But I don't really find it useful.

"I do not like the message at the top - it's a bit disturbing for me..."

I agree with this. I think this statistic should be displayed somewhere else outside of the writing area. That's really all I have to say so far about the new leech statistic; I'm not really sure yet how useful it is to me.

"Maybe it's already enough if you could sort the list of words in MyWords by % leechness, so you can easily spot them and do something about them!"

I agree with this too. I think this would be very useful, so that I could see which words I routinely forget so I can come up with ways to try and remember them.

And while I'm here, I vote for:
"2) Build more integrated leech support into the system so that you can have it do things for you automatically, put them in lists, etc."
Knowing that a word is a leech when I'm reviewing isn't really that useful to me; I still have to try to remember how to write it or whatever.

Also, are leech scores for all four parts of a word, or are leech scores based on individual parts only? That is to say, can I have a word which is a leech in definition but not in writing (for instance)?

Oh yeah, I'll have to turn that off; it wormed its way from beta to the live site, but it's not ready yet.

Also, please be sure to include an off switch for it.

When I have a leech, and I believe it really is a leech, I try to do one of the following:

1) Create a mnemonic or use someone else's mnemonic.
2) Add a hint to the definition field, perhaps an example word or say what one of the components is. For instance, 呸 (pei1) is a character that I run across mainly as a sound effect in manga, so I've added "有口字旁" to the definition. It's a character I want to study, but it's not important enough so that I would worry about giving myself hints. And if I'm wasting too much time on an item, sometimes it's better to do something like this, I think, even for important items.

Now when leech messages have started to appear in the production site, I found that I like this feature. One thing though, maybe the information is not accurate to the 1/100 of percent... For me, tens of percents are accurate enough.

Is it me or has this feature been disabled (even for Beta users)? I think it's a great idea, can it be reinstated please?

Antimacassar, I think I just need to go back to the drawing board a bit in terms of the algorithm and its implementation. Since I don't have time to do that right now, I turned it off. I'll get it back on eventually...

I am sort of missing it. It would be great to have it as an option and with the possibility to tweak it to the % we want. Before you turned this feature on I was not really aware of the this 'leech' concept. Now I recognize it as something that you need to tackle while you study to prevent spending more time than necessary memorizing characters. Although the feature has been turned off, I am more aware now about these 'leeches'. Whenever I find myself struggling for the xth time with a character I just work on it for a while before moving on. If it is a tone, I hit the speaker button 5 to 10 times and repeat the character or word.
It would be great to have this feature back to help me with this.

Marc

I'll definitely bring the feature back, really well done this next time--just focusing on the iOS app for now. The first implementation of the leech detection just isn't good enough to support.