[Competencies] Preliminary results of performance framework review

Scott Kroyer skroyer at e-value.net
Mon Jun 23 15:48:06 UTC 2014

I have an example of previously unknown unknowns and an argument for decimals -- the current ACGME milestones.

An option that residency programs want is to have both an NA and a “not yet achieved Level 1” option in addition to the actual competency levels. “Not yet achieved Level 1” needs to be either 0 or an exceptionally small decimal value (e.g. 0.00001) so that it is counted as part of an aggregate population, but does not contribute significantly toward the aggregate value. In addition, NA does not equate to a true NULL value since it is an affirmative selection. However, it is an affirmative selection that is typically neither expected to contribute to an aggregate value nor to that aggregate’s population — for example, it should add to neither the numerator nor the denominator in percentages.

As such, NA needs to be some value outside of the scale range, and “Not yet achieved” needs to be one of a very narrow set of values outside of the scale range. At least, that’s the best I’ve come up with so far without handling one or both selections as additional metadata rather than as values of the same field.  The argument for not handling the “not yet achieved” option with a negative is that it cannot be handled with a straight aggregation without it penalizing the overall aggregate rather than simply not contributing to the aggregate. Both seem to me to be valid but distinctly different end goals.

Scott Kroyer – Software Development Manager & Architect
E*Value Healthcare Education Solutions
10 Second Street NE, Suite 300 | Minneapolis, MN
d 612-877-4589 | m 612-246-4936 | www.e-value.net<http://www.e-value.net/>

[unknown.png] <http://www.evaluehealthcare.com/conference>

From: Leonard Armstrong <larmstrong at mydatacommons.org<mailto:larmstrong at mydatacommons.org>>
Date: Thursday, June 19, 2014 at 11:50 AM
To: Paul Schilling <paul at schillingconsulting.com<mailto:paul at schillingconsulting.com>>
Cc: Lesley Southgate <lesleyjill at gmail.com<mailto:lesleyjill at gmail.com>>, MedBiq Group <competencies at medbiq.org<mailto:competencies at medbiq.org>>
Subject: Re: [Competencies] Preliminary results of performance framework review

I'm late to the game here but I agree with Simon's and Susan's perspectives. I, like Susan, come at this from the software development assumptions standpoint. Allowing for greater flexibility in the variety of values, in this case, does not limit the use of positive integers and could allow for, as Simon states, broader possibilities without the loss of compatibility. To quote Donald Rumsfeld... "there are unknown unknowns" and this is an important consideration in development of any standard. On the Healthcare Professional Profile working group, we have recently had to loosen constraints for some elements because the data sets and data models used as drivers 10 years ago do not conform to a broader use of data in 2014. I think it likely that the same could be true here.

- Leonard Armstrong

On Thu, Jun 19, 2014 at 11:04 AM, Paul Schilling <paul at schillingconsulting.com<mailto:paul at schillingconsulting.com>> wrote:
I would definitely agree with the negatives for exactly the same reasons that Susan states.

Decimals I can go either way.  Working without decimals is easily worked around by changing the scale.  Use 0-50 instead of 0-5 w/ decimals.  If we do allow decimals would we need to define how many places?  If the idea is portability each system would have to know how many places to expect and store.

On Thu, Jun 19, 2014 at 9:26 AM, Albright, Susan <susan.albright at tufts.edu<mailto:susan.albright at tufts.edu>> wrote:
Having been creating software for some time - I find that whenever i make assumptions about how people will use the software in ways that limit the possibilities - i'm usually found to be wrong.

Leaving the flexibility provides no hardship as far as I can tell.  So allowing negative numbers and allowing decimals seems to me like the right way to go.

From: Valerie Smothers [vsmothers at jhmi.edu<mailto:vsmothers at jhmi.edu>]
Sent: Thursday, June 19, 2014 9:38 AM
To: Lesley Southgate; Albright, Susan
Cc: Cohen, Sascha; MedBiq Competencies Group
Subject: RE: [Competencies] Preliminary results of performance framework review

Thank you for bringing this up, Lesley. One aspect of the performance framework is that it forces you to define what a particular score means. So if someone were to define negative scores on a scale, they would have to define what those negative scores mean using behavioral indicators. I think for the reasons you’ve cited that would only be done under extreme circumstances. But there may be circumstances where it is appropriate to define a scale with negative integers and have those negative integers defined with clear indicators describing what that level of performance looks like.

Tim and I have corresponded regarding your comments, and he reminded me that our policy in the past has been to facilitate practices, whatever they may be, and not constrain the use of the specification to what we believe to be best practice.

With regard to Simon’s case for decimals, we did discuss whether or not decimals should be allowed in the specification. What we discovered was that the interest was in being able to define a single learner’s score using a decimal rather than defining the scale used for assessment in terms of decimals. The scale itself consists of ordinal anchor points and as such should be represented as integers.

Here is the breakdown of comments so far:

Allow for negative integers and provide guidelines – 2 (Sascha and Mary Jo)
Disallow negative scores – 2 (Deborah and Lesley)
Allow decimals – 1 (Simon)

If anyone would like to add their voice to the discussion, or reflect further on the comments made, please do!

Best regards,

From: Lesley Southgate [mailto:lesleyjill at gmail.com<mailto:lesleyjill at gmail.com>]
Sent: Sunday, June 15, 2014 9:39 AM
To: Albright, Susan
Cc: Cohen, Sascha; Valerie Smothers; MedBiq Competencies Group
Subject: Re: [Competencies] Preliminary results of performance framework review

I have read and observed this conversation for more than a year. I feel impelled to comment on the use of negative scores. What would be the purpose of using them. What words might be used to describe their meaning. What would be the impact on the learner, and indeed indeed faculty. As a clinician who spent many years leading the development of, and using assessments for UK physicians who were at risk of losing their license to practice, this is dangerous territory. Does 0 mean absence of the relevant ability...and if so how will feedback be delivered. Does -5 imply something much worse, and if so what? Indeed, it might be so concerning that it could not possibly remain confidential.... as the words dangerous, unprofessional come to mind.

I am sure you have all thought this through, but it seems an important conversation to have.
Lesley Southgate

On 14 June 2014 01:19, Albright, Susan <susan.albright at tufts.edu<mailto:susan.albright at tufts.edu>> wrote:
Could we ever imagine a scoring range that ran from -5 through 0 to +5?

I've never seen it but would it help anyone to be able to use it?

From: Competencies [competencies-bounces at medbiq.org<mailto:competencies-bounces at medbiq.org>] on behalf of Cohen, Sascha [Sascha.Cohen at ucsf.edu<mailto:Sascha.Cohen at ucsf.edu>]
Sent: Friday, June 13, 2014 12:11 PM
To: Valerie Smothers; MedBiq Competencies Group
Subject: Re: [Competencies] Preliminary results of performance framework review
If scores can be set against increasing or decreasing scales, and defined within the context of the scale, then there is little reason to disallow a scoring rule that begins or ends in negative integer territory. I wouldn’t personally use that construct, but hey, I’m just one voice in the peanut gallery.

I would lean toward the consideration that scores should reflect a realistic context, and that one aspect of our discussion in the past has been that the inclusion of “0” as a score value provides a meaningful data point (a learner has no competence within the evaluated scale); and I’m not sure that it would be meaningful to have a score that shows “less than 0 competence”, if you follow. So it really depends on context and the effective definition of both scales and scoring values.

So, I would vote for leaving the data type as xsd:integer, and providing guidelines of best practice regarding the definition of scoring and scale modeling.


Sascha Benjamin Cohen
Director of Strategic Development for Ilios
UCSF School of Medicine
sascha.cohen at ucsf.edu<mailto:sascha.cohen at ucsf.edu>

415.704.4521<tel:415.704.4521> vox

From: Competencies [mailto:competencies-bounces at medbiq.org<mailto:competencies-bounces at medbiq.org>] On Behalf Of Valerie Smothers
Sent: Friday, June 13, 2014 6:46 AM
To: MedBiq Competencies Group
Subject: [Competencies] Preliminary results of performance framework review

Hi, everyone.

We’ve received some comments on the Performance Framework from the MedBiquitous Standards Committee. These are captured on the following wiki page: http://groups.medbiq.org/medbiq/display/CWG/Standards+Committee+review%2C+June+2014
Please note there is a space for the working group’s recommendations on this table.

Several substantive comments from Dan Rehak relate to whether scores can be negative or 0. We had decided that scores could be 0. We never exclusively ruled out negative numbers. I emailed Dan to see if he had a compelling argument against allowing negative numbers:

I wanted to share my thoughts on your comments and see if you wanted to talk it over by phone. The biggest issue is whether scores can be negative. The working group did discuss allowing scores to be 0, but if there were a convincing argument to be made for disallowing that, I think they could be swayed. The group also decided to allow the direction of the scale to be determined by framework authors. So you may have a scale of 1 to 5 where 1 is most competent, or you may have a scale of 1 to 5 where 1 is least competent. The reason we decided to do that was that the ISO specs around proficiency levels point to scales of differing direction. They used Judo as an example, where the Kyu scale goes from 10 being least competent to 1 being most competent. After Kyu comes Dan, where 1 is least competent and 9 is most competent.

We haven’t seen any examples of negative numbers being used in scores, but I don’t know that we want to restrict that. Again, if there is a good argument to be made for restricting further, I am all ears.

Here is how Dan replied:

I don't have any strong feelings about the "right" answer here.  I just wanted to be sure about the valid range of values -- if zero or negative values are permitted, that's OK.  But if not, then I was thinking that the data type be more restrictive than just xsd:integer.  What triggered this was that the names are often associated with more restrictive ranges.
I think the kind of things you mention, e.g., 0, negative, and scale and direction of scale should be mentioned in the text, and then just make the datatypes align.
Please share your thoughts and recommendations by Thursday June 18. I would like to send the Standards Committee our recommendations by June 19.

Best regards,

Valerie Smothers
Deputy Director
vsmothers at jhmi.edu<mailto:vsmothers at jhmi.edu>
Sign up for our newsletter<http://visitor.constantcontact.com/d.jsp?m=1103590106851&p=oi>
Follow us on Twitter!<https://twitter.com/medbiq>

Competencies mailing list
Competencies at medbiq.org<mailto:Competencies at medbiq.org>

Dame Lesley Southgate
Professor of Medical Education
St Georges University of London
best contact email or text to mobile
Gmail reaches me anywhere in the world
Mobile  44(0)7970 020261
Tel: 44(0)20 8725 0816
(messages via St Georges)
lesleyjill at gmail.com<mailto:lesleyjill at gmail.com>
lsouthga at sgul.ac.uk<mailto:lsouthga at sgul.ac.uk>

p.a. lisabennet at dial.pipex.com<mailto:lisabennet at dial.pipex.com> (part time)

Competencies mailing list
Competencies at medbiq.org<mailto:Competencies at medbiq.org>

Paul Schilling
Schilling Consulting LLC
paul at schillingconsulting.com<mailto:paul at schillingconsulting.com>
Phone: 1.608.628.6313<tel:1.608.628.6313>
Fax: 1.888.696.8405<tel:1.888.696.8405>

Competencies mailing list
Competencies at medbiq.org<mailto:Competencies at medbiq.org>

CONFIDENTIALITY NOTICE: This e-mail message including attachments, if any, is intended for the person or entity to which it is addressed and may contain confidential, privileged, and/or proprietary material. Any unauthorized review, use, disclosure or distribution is prohibited.  If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://medbiq.org/pipermail/competencies_medbiq.org/attachments/20140623/18f9f6ee/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 9781610C-B48F-4DAD-8A50-F842427988FA[6].png
Type: image/png
Size: 18493 bytes
Desc: 9781610C-B48F-4DAD-8A50-F842427988FA[6].png
URL: <http://medbiq.org/pipermail/competencies_medbiq.org/attachments/20140623/18f9f6ee/attachment.png>

More information about the Competencies mailing list