Reviewers and “Experts”

At a recent industry presentation, I was asked, “does everyone who reviews a proposal have to be an 'expert' in the area?”  This is not an uncommon question as the use of “experts” and “non-experts” in peer-review is increasing as organizations embrace their stakeholders more broadly.

A well designed evaluation process and criteria should enable a “reasonable” person (not necessarily an expert) to read what the proponent writes and make an assessment.  Experts are valuable in giving opinions either via comments in their assessments or during committee discussion.

Sometimes, you will find that people (reviewers) refuse to do an assessment in an area that they are not “expert” in.  What should you do?  

Although you cannot say so, these people are being unnecessarily cautious.  Usually the matrix of criteria (your evaluation criteria) has only a few criteria that really demands deep expertise.  If the person feels they cannot deal with these – or if any one of the criteria answers by the Proponent just does not make any sense – let the Reviewer leave it blank, or if you are using a ProGrid software tool, tell them to enter an “N” for null, i.e. that criteria will not be included in the score calculations.  Your scoring system should automatically adjust the averaging algorithm and take into account only “non N” ratings (as is the case with ProGrid software tools).  You or the staff or the Chair should look at the tables in the reports that highlights whether reviewers used “N” – and how extensively – to get a feeling for whether there might be an issue with a criterion that needs further discussion.