X-Message-Number: 24117
Date: Thu, 20 May 2004 10:02:40 -0400 (EDT)
From: Dan Hitt <>
Subject: measuring the effectiveness and consequences of the rating system

As most of us are aware, we have a rating system that Kevin put
together to control spam-like behavior.

He was responding to a specific problem, in which a poster seemed
to have cryonet.org in some list of addresses that he sent
a mass mailing to two or three times a week.  [Just for
reference, Randy (http://www.cryonet.org/cgi-bin/dsp.cgi?msg=23980)
James (http://www.cryonet.org/cgi-bin/dsp.cgi?msg=23857), 
Michael L (http://www.cryonet.org/cgi-bin/dsp.cgi?msg=23991),
and others have considered the question of whether the content
of the messages had any influence on whether the poster's
could be understood on spamming.  Kevin also discussed this
in http://www.cryonet.org/cgi-bin/dsp.cgi?msg=23989.]

And even if it weren't an immediate problem, a mature system
should be robust against spam-like behavior.

Kevin's system allows us to rate each message, and in addition
to rating messages off-topic, it provides us with a chance to
easily help each other improve by reminding each other to use
descriptive subject lines (something Hank made explicit in
his recent message http://www.cryonet.org/cgi-bin/dsp.cgi?msg=20448).
And some people think it's a good thing to have in place: Steve
J (http://www.cryonet.org/cgi-bin/dsp.cgi?msg=24035) likes it,
and no doubt lots of others.

On the other hand, Kennita and others have pointed out
problems with the system (http://www.cryonet.org/cgi-bin/dsp.cgi?msg=24109),
and i can't help but worry that we're making cryonet less hospitable
(although see Kevin's http://www.cryonet.org/cgi-bin/dsp.cgi?msg=24093
for a contrary view).

Of course the choice of what to do is solely Kevin's: he's paid
for cryonet both with his money and with is work, and i owe him
big-time for making the idea of cryonics as real as breakfast
to me, and providing contacts with cryonicists.  Posting to
cryonet is not a free speech right.

But at least as an exercise, i think we can come up with measures
as to how effective the system is, and whether and how much it
may generate unintended consequences.

I have two measures in mind, but i'm sure there are others and better
ones.

One measure would be the mean time between first appearances of a
newcomer.  That is, in the one year before the rating system came into
being, how many newcomers were there?  (I guess at least one, Paul P.)

If the number of newcomers is significantly smaller, then i think
there's a negative unintended consequence of the rating system.

A second measure, this one of the effectiveness, would be the number
of posts with poor subject lines (say divided by the total number of
posts over some period, or perhaps just from some fixed individual
poster).  Is that number decreasing?  (Measuring the number of
off-topic posts is harder to do objectively, i think, but poor
subject lines are pretty visible.)

What could be done with such metrics?

Well, Kennita has privately proposed an alternate system for
discouraging spam-like behavior, and it could be tried out
(to see, objectively, if it performed better).  Perhaps she
can post it?

In any event, thanks again to Kevin for this substantial
piece of work (cryonet), because rating system or no, it
has been very beneficial to me, and i hope lots of others
in the future.

dan

Rate This Message: http://www.cryonet.org/cgi-bin/rate.cgi?msg=24117