X-Message-Number: 24120
Date: Thu, 20 May 2004 11:10:02 -0700
Subject: Re: measuring the effectiveness and consequences of the rating system
From: Kennita Watson <>

On Thursday, May 20, 2004, at 07:02 US/Pacific, Dan Hitt wrote:

> As most of us are aware, we have a rating system that Kevin put
> together to control spam-like behavior.
>
> He was responding to a specific problem, in which a poster seemed
> to have cryonet.org in some list of addresses that he sent
> a mass mailing to two or three times a week. ...

BTW, does a poster's "I am a spammer" rating wear off faster
or slower if he/she posts a message that is not off-topic
but annoys (some) people in some other way?

As I recall, there is a "normal" rating.  How many people
bother to rate a message "normal"?  If it's not done already,
every message that is posted to Cryonet that is not rated
something else should be counted as having received a number
of "normal" ratings equal to the number of people who bothered
to rate any message in the digest (minus the number who rated
the message something else, of course).
>
> And even if it weren't an immediate problem, a mature system
> should be robust against spam-like behavior.

This one isn't, BTW -- if "spammer" ratings ever wear off,
the spammer can wait till the rating wears off, send a spam,
then shut up till it wears off again.
>
> Kevin's system allows us to rate each message, and in addition
> to rating messages off-topic, it provides us with a chance to
> easily help each other improve by reminding each other to use
> descriptive subject lines ...

This could be more easily handled at the source -- I think it
would be easy to recognize a subject line that is just
"Re: CryoNet #xxxxx - #yyyyy" and bounce it back to the sender
for a subject-line change, rather than bothering the rest of
the list with rating it "poor subject line" (duh!) and/or with
reading a digest that has poor subject lines.

> and i can't help but worry that we're making cryonet less hospitable
> (although see Kevin's http://www.cryonet.org/cgi-bin/dsp.cgi?msg=24093
> for a contrary view).

"Consistent quality is not a virtue of forums that lack strong
moderation."  Cryonet won't be Cryonet if it's strongly moderated.

> Of course the choice of what to do is solely Kevin's: he's paid
> for cryonet both with his money and with is work, and i owe him
> big-time for making the idea of cryonics as real as breakfast
> to me, and providing contacts with cryonicists.  Posting to
> cryonet is not a free speech right.

Quite so, but people can still feel betrayed when the terms of
the (albeit unwritten) contract under which they signed up to
participate in it are changed midstream.
>
> But at least as an exercise, i think we can come up with measures
> as to how effective the system is, and whether and how much it
> may generate unintended consequences. ...
> One measure would be the mean time between first appearances of a
> newcomer.  That is, in the one year before the rating system came into
> being, how many newcomers were there?  (I guess at least one, Paul P.)

There are a number of things -- how frequently do people subscribe,
post for the first time, post subsequent times, unsubscribe?
>
> If the number of newcomers is significantly smaller, then i think
> there's a negative unintended consequence of the rating system.

Hard to be smaller than one ;-) .
>
> A second measure, this one of the effectiveness, would be the number
> of posts with poor subject lines ...

See my comment above -- I'd rather do that one heavyhandedly,
without ratings.  I don't like seeing my messages show up with
poor subject lines more than anybody else does, and would
appreciate having a chance to fix them before they showed up
in the digest.

> What could be done with such metrics?
>
> Well, Kennita has privately proposed an alternate system for
> discouraging spam-like behavior, and it could be tried out
> (to see, objectively, if it performed better).  Perhaps she
> can post it?

I didn't already? <dig, dig, shuffle, dig...

Do you mean this?
>> Possibility:  if the objective is to keep spammers off
>> the list, how about an entry questionnaire?  Where did
>> you hear about Cryonet, when did you first become
>> interested in cryonics, are you a member of any
>> cryonics organization(s), etc.  Give the subscriber the
>> option of posting the answers to the list by way of
>> introduction, or keeping the answers secret so only the
>> moderator can see them (and posting a free-form text
>> intro message?).  This could be useful in the future for
>> demographic analysis, maybe; it's useful now because
>> someone who just wants to spam will probably not go to
>> the trouble of filling out the questionnaire, much less
>> of introducing him/herself to the membership.  I guess
>> that last will always be optional; I've never wanted to
>> keep my membership in a group secret, but I guess some do.

Or maybe this?

>> How about a reactive system -- if Kevin receives X number of
>> complaints about person Y, and private email to Y pointing
>> this out has no effect, he posts a message to Cryonet
>> saying so, and starts an online poll to decide what to do
>> about Y.  Some free online poll sites (all Web-based) that
>> I found without half trying:
>>
>> http://www.thefreecountry.com/scripthosting/polls.shtml
>> http://cde.athabascau.ca/softeval/ratPOL.htm
>> http://www.zoomerang.com/login/index.zgihttp://www.freepolls.com/
>> http://www.evite.com/polling
>> http://www.misterpoll.com/
>> http://www.dream-tools.com/index.mv
>>
>> This allows dealing with list annoyances without adding
>> extra overhead to every message -- more disk space,
>> more scrolling, more mental overhead (even if the user
>> doesn't rate the message, but especially if they do).
>>
>> The poll doesn't have to be just "boot Y" or "leave Y
>> alone"; there could be "give Y one more chance" or
>> "suspend Y's posting privileges for N days" or
>> something like that.
>>
>> I hope that's enough of an outline for the idea that it
>> can replace the per-message rating system.
...

This issue has taken more of my time than I wanted today;
see you next time.

Rate This Message: http://www.cryonet.org/cgi-bin/rate.cgi?msg=24120