[Competencies] Preliminary results of performance framework review

Paul Schilling paul at schillingconsulting.com
Thu Jun 19 15:04:18 UTC 2014


I would definitely agree with the negatives for exactly the same reasons
that Susan states.

Decimals I can go either way.  Working without decimals is easily worked
around by changing the scale.  Use 0-50 instead of 0-5 w/ decimals.  If we
do allow decimals would we need to define how many places?  If the idea is
portability each system would have to know how many places to expect and
store.


On Thu, Jun 19, 2014 at 9:26 AM, Albright, Susan <susan.albright at tufts.edu>
wrote:

>  Having been creating software for some time - I find that whenever i
> make assumptions about how people will use the software in ways that limit
> the possibilities - i'm usually found to be wrong.
>
>  Leaving the flexibility provides no hardship as far as I can tell.  So
> allowing negative numbers and allowing decimals seems to me like the right
> way to go.
>
>  Susan
>  ------------------------------
> *From:* Valerie Smothers [vsmothers at jhmi.edu]
> *Sent:* Thursday, June 19, 2014 9:38 AM
> *To:* Lesley Southgate; Albright, Susan
> *Cc:* Cohen, Sascha; MedBiq Competencies Group
> *Subject:* RE: [Competencies] Preliminary results of performance
> framework review
>
>   Thank you for bringing this up, Lesley. One aspect of the performance
> framework is that it forces you to define what a particular score means. So
> if someone were to define negative scores on a scale, they would have to
> define what those negative scores mean using behavioral indicators. I think
> for the reasons you’ve cited that would only be done under extreme
> circumstances. But there may be circumstances where it is appropriate to
> define a scale with negative integers and have those negative integers
> defined with clear indicators describing what that level of performance
> looks like.
>
>
>
> Tim and I have corresponded regarding your comments, and he reminded me
> that our policy in the past has been to facilitate practices, whatever they
> may be, and not constrain the use of the specification to what we believe
> to be best practice.
>
>
>
> With regard to Simon’s case for decimals, we did discuss whether or not
> decimals should be allowed in the specification. What we discovered was
> that the interest was in being able to define a single learner’s score
> using a decimal rather than defining the scale used for assessment in terms
> of decimals. The scale itself consists of ordinal anchor points and as such
> should be represented as integers.
>
>
>
> Here is the breakdown of comments so far:
>
>
>
> Allow for negative integers and provide guidelines – 2 (Sascha and Mary Jo)
>
> Disallow negative scores – 2 (Deborah and Lesley)
>
> Allow decimals – 1 (Simon)
>
>
>
> If anyone would like to add their voice to the discussion, or reflect
> further on the comments made, please do!
>
>
>
> Best regards,
>
> Valerie
>
>
>
> *From:* Lesley Southgate [mailto:lesleyjill at gmail.com]
> *Sent:* Sunday, June 15, 2014 9:39 AM
> *To:* Albright, Susan
> *Cc:* Cohen, Sascha; Valerie Smothers; MedBiq Competencies Group
> *Subject:* Re: [Competencies] Preliminary results of performance
> framework review
>
>
>
> I have read and observed this conversation for more than a year. I feel
> impelled to comment on the use of negative scores. What would be the
> purpose of using them. What words might be used to describe their meaning.
> What would be the impact on the learner, and indeed indeed faculty. As a
> clinician who spent many years leading the development of, and using
> assessments for UK physicians who were at risk of losing their license to
> practice, this is dangerous territory. Does 0 mean absence of the relevant
> ability...and if so how will feedback be delivered. Does -5 imply something
> much worse, and if so what? Indeed, it might be so concerning that it could
> not possibly remain confidential.... as the words dangerous, unprofessional
> come to mind.
>
>
>
> I am sure you have all thought this through, but it seems an important
> conversation to have.
>
> Lesley Southgate
>
>
>
> On 14 June 2014 01:19, Albright, Susan <susan.albright at tufts.edu> wrote:
>
> Could we ever imagine a scoring range that ran from -5 through 0 to +5?
>
>
>
> I've never seen it but would it help anyone to be able to use it?
>
>
>
> Susan
>  ------------------------------
>
> *From:* Competencies [competencies-bounces at medbiq.org] on behalf of
> Cohen, Sascha [Sascha.Cohen at ucsf.edu]
> *Sent:* Friday, June 13, 2014 12:11 PM
> *To:* Valerie Smothers; MedBiq Competencies Group
> *Subject:* Re: [Competencies] Preliminary results of performance
> framework review
>
> If scores can be set against increasing or decreasing scales, and defined
> within the context of the scale, then there is little reason to disallow a
> scoring rule that begins or ends in negative integer territory. I wouldn’t
> personally use that construct, but hey, I’m just one voice in the peanut
> gallery.
>
>
>
> I would lean toward the consideration that scores should reflect a
> realistic context, and that one aspect of our discussion in the past has
> been that the inclusion of “0” as a score value provides a meaningful data
> point (a learner has no competence within the evaluated scale); and I’m not
> sure that it would be meaningful to have a score that shows “less than 0
> competence”, if you follow. So it really depends on context and the
> effective definition of both scales and scoring values.
>
>
>
> So, I would vote for leaving the data type as xsd:integer, and providing
> guidelines of best practice regarding the definition of scoring and scale
> modeling.
>
>
>
> Sascha
>
>
>
>
>
> *Sascha Benjamin Cohen*
>
> Director of Strategic Development for Ilios
>
> UCSF School of Medicine
>
> sascha.cohen at ucsf.edu
>
>
>
> 415.704.4521 vox
>
> http://curriculum.ucsf.edu
>
> http://www.iliiosproject.org
>
>
>
>
>
>
>
> *From:* Competencies [mailto:competencies-bounces at medbiq.org] *On Behalf
> Of *Valerie Smothers
> *Sent:* Friday, June 13, 2014 6:46 AM
> *To:* MedBiq Competencies Group
> *Subject:* [Competencies] Preliminary results of performance framework
> review
>
>
>
> Hi, everyone.
>
>
>
> We’ve received some comments on the Performance Framework from the
> MedBiquitous Standards Committee. These are captured on the following wiki
> page:
> http://groups.medbiq.org/medbiq/display/CWG/Standards+Committee+review%2C+June+2014
>
> Please note there is a space for the working group’s recommendations on
> this table.
>
>
>
> Several substantive comments from Dan Rehak relate to whether scores can
> be negative or 0. We had decided that scores could be 0. We never
> exclusively ruled out negative numbers. I emailed Dan to see if he had a
> compelling argument against allowing negative numbers:
>
>
>
> I wanted to share my thoughts on your comments and see if you wanted to
> talk it over by phone. The biggest issue is whether scores can be negative.
> The working group did discuss allowing scores to be 0, but if there were a
> convincing argument to be made for disallowing that, I think they could be
> swayed. The group also decided to allow the direction of the scale to be
> determined by framework authors. So you may have a scale of 1 to 5 where 1
> is most competent, or you may have a scale of 1 to 5 where 1 is least
> competent. The reason we decided to do that was that the ISO specs around
> proficiency levels point to scales of differing direction. They used Judo
> as an example, where the Kyu scale goes from 10 being least competent to 1
> being most competent. After Kyu comes Dan, where 1 is least competent and 9
> is most competent.
>
>
>
> We haven’t seen any examples of negative numbers being used in scores, but
> I don’t know that we want to restrict that. Again, if there is a good
> argument to be made for restricting further, I am all ears.
>
>
>
> Here is how Dan replied:
>
>
>
> I don't have any strong feelings about the "right" answer here.  I just
> wanted to be sure about the valid range of values -- if zero or negative
> values are permitted, that's OK.  But if not, then I was thinking that the
> data type be more restrictive than just xsd:integer.  What triggered this
> was that the names are often associated with more restrictive ranges.
>
> I think the kind of things you mention, e.g., 0, negative, and scale and
> direction of scale should be mentioned in the text, and then just make the
> datatypes align.
>
> Please share your thoughts and recommendations by Thursday June 18. I
> would like to send the Standards Committee our recommendations by June 19.
>
>
>
> Best regards,
>
> Valerie
>
>
>
> Valerie Smothers
>
> Deputy Director
>
> MedBiquitous
>
> vsmothers at jhmi.edu
>
> +1-410-735-6142
>
> www.medbiq.org
>
>
> *Sign up for our newsletter
> <http://visitor.constantcontact.com/d.jsp?m=1103590106851&p=oi> **Follow
> us on Twitter! <https://twitter.com/medbiq>*
>
>
>
>
> _______________________________________________
> Competencies mailing list
> Competencies at medbiq.org
> http://medbiq.org/mailman/listinfo/competencies_medbiq.org
>
>
>
>
>
> --
> Dame Lesley Southgate
> Professor of Medical Education
> St Georges University of London
> best contact email or text to mobile
>
> Gmail reaches me anywhere in the world
> Mobile  44(0)7970 020261
> Tel: 44(0)20 8725 0816
> (messages via St Georges)
>
> lesleyjill at gmail.com
> lsouthga at sgul.ac.uk
>
> p.a. lisabennet at dial.pipex.com (part time)
>
> _______________________________________________
> Competencies mailing list
> Competencies at medbiq.org
> http://medbiq.org/mailman/listinfo/competencies_medbiq.org
>
>


-- 
--
Paul Schilling
Schilling Consulting LLC
paul at schillingconsulting.com
Phone: 1.608.628.6313
Fax: 1.888.696.8405
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://medbiq.org/pipermail/competencies_medbiq.org/attachments/20140619/572d80a6/attachment-0002.html>


More information about the Competencies mailing list