Home > Funding > Become a Reviewer > Guide for expert’s evaluation

Register for the calls

Become an expert

Guide for expert’s evaluation

GUIDE FOR EXPERT EVALUATION OF PROJECTS SUBMITTED TO NRFU CALLS

As an independent expert in a certain field of science you shall evaluate a project submitted to NRFU calls for promotion of research and developments. Subject to the expert evaluation, the Call Commission makes important decisions on project funding. You shall submit your expert opinion in your personal cabinet in the NRFU database in a timely manner. The submitted expert opinion, after possible adjustment and approval by the Call Commission is not subject to revision or amendments. It is your responsibility to prepare and submit an expert opinion to the Call Commission. Another person may not be authorized thereto. 

Below you will find practical recommendations on: 

EXPERT EVALUATION ALGORITHM – step-by-step guidelines for experts 

ISSUES WORTH YOUR ATTENTION – recommendations on the analysis of application aspects important for adequate assessment of the project’s quality and scientific skills of the group

RATIONALE FOR ASSESSMENTS – recommendations on the arguments and comments on each assessment

ARGUMENTS WORDING – recommendations on arguments stylistics and unwanted vocabulary 

EXPERT EVALUATION ALGORITHM 

The list of projects proposed for the evaluation by the Call Commission to a particular expert appears in the section “Applications for evaluation” of your personal cabinet.

Application details and evaluation form is available in the section “Applications for evaluation” after the moderator accepts your documents.

  • Study thoroughly all the files of the project application subject to evaluation (project description, CV of the Principal investigator and project applicants, budget, and others)
  • Assess the extent to which the proposed project relates to the theme of the Call
  • Verify the information provided in the application if you have any doubts as to its reliability
  • Draw up your expert opinion: substantiate thoroughly each of your assessments on each aspect and criterion, avoiding any repetition
  • Evaluate the project in 15 calendar days upon receipt of the application submitted by the Call Commission. Submit your expert opinion through the personal cabinet in the NRFU database

The Call Commission may send some comments regarding your evaluation (see the field below the online evaluation form). If you did not receive any comments from the Call Commission this means that your evaluation is approved.

ISSUES WORTH YOUR ATTENTION 

  • Take into account that any information you may need to assess a particular aspect or criterion of the project may be set forth in different parts of the application and contained both in the project description and in the CV of the Principal investigator and project applicants, estimates of expenditures, or other submitted documents. Therefore, we propose that you should first study thoroughly all the files and only after that proceed to the assessment and substantiation of your assessments. 
  • Should you have any doubts as to the accuracy of the information provided by the applicant (in particular, regarding the affiliation of the Principal investigator, previous employers of the Principal investigator and project applicants, their important publications, participation in international projects, awards and honors, internships, conferences, and others, or regarding the cost of certain equipment, materials or services), you can verify it in open sources and be sure to inform the Call Commission of the discovered discrepancies (if any).
  • When assessing scientific skills of project applicants, consider the number and quality of scientific publications taking into account: (1) the credibility of the professional periodical or publisher where an article, monograph or other work was published (for example, Scopus/WoS quartile of the periodical), (2) the importance and contribution of the publication to the relevant field of study, (3) the relevance of such publications to the subject of the project under evaluation.
  • Given the marked trend in science development in Ukraine of a significant number of publications in international periodicals containing low-quality scientific content and published in commercial pseudo-scientific periodicals on a paid basis, we recommend that the Principal investigator and project applicants should check the publications in question in the list of the so-called “predatory journals” available at https://predatoryjournals.com/journals/, and publishers with a questionable academic reputation in the list available at https://predatoryjournals.com/publishers/. Such publications may prove the poor quality of the published articles/monographs.
  • It is also necessary to take into account such indicators of the quality of scientific work and recognition of the scientist as the Hirsch index and the number of citations in online scientific publications databases (Scopus, WoS, Google Scholar for the given field of science), international and national awards and honors of the scientist. Another evidence of a skilled scientist is his/her long-term scientific internships and/or contract employment in leading scientific institutions abroad.
  • It is important to assess the research, organizational and managerial potential of the Principal investigator and project applicants. Therefore, in addition to publications, you should pay attention to their participation in international research projects, and experience in arranging and holding scientific events (conferences, symposia, seminars, workshops and others).
  • When assessing the quality of the submitted project, it is necessary to decide whether similar projects have already been implemented in Ukraine and how it differs from previous projects; whether this project has already been implemented (was funded); whether the project in question is based on advances of the world and domestic scientific community and is not pseudo-scientific in nature, as well as has no signs of plagiarism or academic dishonesty

RATIONALE FOR ASSESSMENTS 

  • Assess the actual submitted project rather than its potential in case of certain changes. Assess the actual content, advantages or disadvantages of the project, without comparing it with others and without making assumptions about the missing elements;
  • Consider all available information about the project and its authors;
  • Be as specific as possible, express your opinions clearly and concisely, avoid general senseless phrases, refer to the facts;
  • Do not paraphrase the content of the application;
  • Your comments should refer to the core, be objective and impartial, as well as respectful towards the authors of the project;
  • Do not set forth the same remarks in different parts of the evaluation form;
  • Do not advise on possible project improvements;
  • Avoid ambiguous statements and doubts as to your competence;
  • Do not enter into polemics in absentia with the authors of the project;
  • Avoid moral and ethical assessments of the project or its potential effects. However, indicate, where appropriate, the compliance of the project with current ethical principles or rules of research in the relevant field of science, violation of the principles of academic integrity, etc.

ARGUMENTS WORDING 

  • Avoid expressions that imply partiality of the statement (in my opinion, I think, as far as I understand, I am impressed, I think so, etc.);
  • Avoid evaluative judgments (this is the best/worst project, applicants are not competent scientists, the project does not make sense, the budget is too “swollen”, etc.);
  • Avoid colloquial vocabulary and slang (this is a cool idea, a fashionable/hot topic, stylishly presented, well known, popular scholar, etc.);
  • Avoid hostile and insulting statements, excessive categorical and firm statements (hopeless/unique project, incompetent/ignorant/genius author, has no prospects, is not worthy of attention, will be a revolution in science, etc.)

APPENDIXES 

Excerpt from “The procedure of consideration and expertise of research and development projects, submitted to the National Research Foundation of Ukraine for participation in call selections”

  1. The basic principles of consideration and Expert Evaluation of projects are the following:

– maximum openness and transparency;

– independent and impartial scientific and technological Expert Evaluation of research and developments projects;

– competent and impartial persons conducting Expert Evaluation;

– taking into account international scientific and technological progress;

– responsibility for the accuracy and integrity of the analysis, substantiation of expert recommendations;

– respect for copyright and related rights, as well as adherence to the principles of scientific ethics;

– compliance with the principles of fair Call;

– prevention of conflict of interests during the review and Expert Evaluation of research and development projects.

  1. The purpose of the Expert Evaluation of the application is to compile conclusions and recommendations substantiated by the expert as to the ability of the participant of the Call to implement research and developments project at the appropriate level for the form of project evaluation established by the Scientific Council of the Foundation in its decision with regard to the Call.

Typical evaluation form and criteria for the Expert Evaluation of research and development projects are set forth in Appendix 2 hereto.

  1. Before the evaluation, the expert has, no later than three working days from the date of receipt of the project(s), to express his/her consent to the evaluation and declare the absence of a conflict of interests and the relevance of his/her area of evaluation to the project theme.
  2. Applications are subject to Expert Evaluation by experts within fifteen calendar days from the date of receipt of the application from the Call Commission. Given the findings of the Expert Evaluation, the expert opinion shall be drawn up and submitted to the Call Commission.

Excerpt from the “The Rules of Compliance with the Principles of Scientific Ethics and Conflict of Interests Prevention during the NRFU Projects Evaluation and Competitive Selection”

PRINCIPLES OF SCIENTIFIC ETHICS

  1. The basic principles of professional and scientific ethics and conduct of persons conducting project evaluation.

Experts are obliged to conduct their activities in accordance with the following principles:

  • the rule of law;
  • respect for human rights and freedoms, non-discrimination;
  • independence, autonomy and ideological neutrality;
  • objectivity and impartiality;
  • competence and professionalism;
  • integrity and ethical conduct;
  • confidentiality;
  • social responsibility.

Experts are personally liable for their decisions or actions.

  1. Independence, autonomy and ideological neutrality requirements

During the project evaluation experts may not be subjected to any ideological, political, administrative, business, social influence or pressure, and may not allow external interference in their professional activities.

Experts must resist any encroachment on their independence, uphold professional rights, be confident and principled in the performance of their professional duties.

Experts must be independent in expressing professionally grounded judgments and decisions, preventing any attempts to interfere in project consideration or evaluation.

Experts are not allowed to have contacts with persons who have direct or indirect interests in the outcome of the Call unless such contacts are permitted by the rules of the Call.

  1. Competence and professionalism requirements

Experts must have in-depth and comprehensive knowledge in the relevant field required for professional project consideration and evaluation, including awareness of the latest advances in science and technology.

Experts must objectively assess their qualifications and practical experience, scientific knowledge, skills and relevant competence to make reasoned decisions and provide professional opinions. It is necessary to refuse project evaluation in case an expert does not have sufficient skills for the qualified project evaluation.

  1. Good faith and ethical conduct requirements

Experts should not participate in the activities that could discredit them. They should avoid taking actions that could call into question their honesty and integrity.

Public commenting or assessing the work of other persons involved in the competitive selection and project evaluation is not allowed.

  1. Confidentiality requirements

Experts must respect the confidentiality of the information and may not disclose any information that became known to them during the performance of the expert’s responsibilities. Confidentiality requirements do not cover cases where disclosure of the information is needed to comply with the public interest and/or legal requirements.

CONFLICT OF INTERESTS

  1. Prevention of conflict of interests

Experts should conduct their activities without any actual or potential conflict of interests.

At the stage when the names of participants of the call are disclosed to the experts they must notify of the conflict of interests by submitting a uniform application.

If there are actual or potential interpersonal conflicts, or conflicts of interests between experts involved in the project evaluation, these circumstances, if possible, must be communicated.

In case of conflict of interests that was not known at the initial stage of the competitive selection, experts must notify the organizers of the call of such a conflict and declare their withdrawal from further project evaluation.

The grounds for refusing to involve an expert in the project evaluation, for self-disqualification of an expert, as well as for the removal of a member of the Call Commission or refusal by a member of the Call Commission to votе are the following:

  • family relations with project applicants (hereinafter referred to as “the Institution Participating in the Competitive Selection”);
  • employment in the Institution Participating in the Competitive Selection (for the last three years prior to the announcement of the Call date);
  • membership in supervisory or monitoring bodies of the Institution participating in the Competitive Selection (for the last three years prior to the announcement of the Call date); 
  • an expert or a member of the Call Commission being the scientific supervisor of one of the project applicants or a person among the project applicants who was a scientific supervisor of an expert or a member of the Call Commission (for the last three years prior to the announcement of the Call date); 
  • joint scientific publications with one of the project applicants, or participation in joint scientific projects (for the last three years prior to the announcement of the Call date); 
  • other close scientific or commercial cooperation; 
  • other direct scientific or commercial call.

An expert must also resign in case of personal conflict with participants of the Call.

Under no circumstances should experts conceal personal or other factors that affect or may affect the impartiality of the project evaluation.

LIABILITY 

  1. Liability for valid, comprehensive and reliable opinions and decisions

Experts are liable for valid, comprehensive and reliable opinions.

Expert opinions must be valid, objective, based on a fully-fledged and comprehensive study of the project.

Experts are liable for falsifying expert evaluation opinions.

Any intentional and unreasonable delay in the evaluation process is inadmissible and entails liability under the law and this Guide.

Experts shall be liable for any violations caused by them, in particular with regard to:

  • illegal use of copyright and industrial property rights,
  • breach of confidentiality of information collected in connection with consideration or evaluation of the project.

Experts whose violations of the provisions of this article have been identified shall, in the event of their first violation, be informed of the liability and excluded from consideration or evaluation of the project. In the case of repeated violations, such persons are banned from participating in the competitive selection as experts for the next three years. 

Criteria and types of evaluation of research and developments projects submitted for participation in the competitive selection for grants allocated by the National Research Foundation of Ukraine (summary)

Each project shall be evaluated with regard to the following aspects, namely:

  • Quality of the planned research;
  • Importance of the project for further development of science/technology/society due to the objective of the project;
  • Quality and feasibility of the proposed project implementation plan;
  • Scientific qualifications of the project applicants.

Each aspect includes several evaluation criteria. The expert shall evaluate the project under each criterion with regard to the 5-point scale substantiating every point:

0 – Poor The project fails to address the criterion or cannot be assessed due to missing or incomplete information provided by this criterion
1 – Unsatisfactory (Bad)  The information on the criterion under evaluation is inadequately presented or has serious shortcomings
2 – Satisfactory The project broadly addresses the criteria but there are significant weaknesses
3 – Good The project addresses the criteria well but with a number of shortcomings
4 – Very good The project addresses the criteria very well but improvements are possible
5 – Fully Satisfactory The project successfully addresses the criteria

 

Aspect 1. The quality of the planned research

This section provides an assessment of the research project substantiation, its focus on solving current (urgent) scientific problems, clear formulation of its purpose and objectives, its compliance with current scientific developments, the novelty of scientific ideas, and uniqueness of scientific hypothesis, the correct choice of methodology and research methods.

1.1. Motivation and substantiation of the scientific concept: the current state of research and unresolved issues are adequately described, with relevant references. 

1.2. The novelty of scientific ideas (including from the standpoint of interdisciplinary nature if the research is multidisciplinary)

1.3. Clarity and relevance of the stated purpose and objectives of the research

1.4. Adequacy of the proposed approaches and research methods, their compliance with the purpose and objectives of the project.

Aspect 2. Project importance for further development of science/engineering/technology/society in accordance with the purpose of the project

This section provides an assessment of the clarity of definitions and arguments of the prospects for further use of research findings for the development of science and new technologies, as well as the introduction in the economic and social areas. The integrity and potential efficiency of research findings publication, the possibility of commercialization of project achievements are taken into account.

2.1. The potential importance of the expected findings for the development of new knowledge, new approaches and technologies and/or their importance for solving current practical scientific/technical/social problems.

2.2. Effectiveness and relevance of the planned ways of publishing / using the findings of the research (expected professional publications, speeches at international conferences, sharing the findings with the general public, etc.).

Aspect 3. Quality and feasibility of the proposed project implementation plan

Within this section, the substantiation of the work plan and the clarity of intermediate goals, their logical sequence are assessed; clarity of the description of the planned tasks with the specific results which can be checked; consistency of complex tasks with their time frame; a concurrence of equipment and materials specified as necessary for the project implementation with its purpose and objectives; clarity of description of equipment and materials and adequacy of their budget funding.

3.1. Substantiation of the work plan, compliance of the time frame with the complexity of the established stages and tasks, clarity of intermediate goals, their logical sequence

3.2. Alignment of material and technical support, equipment (available and planned) to the set task

3.3. Balance and substantiation of the total project budget

3.4. Presence and rationale of the assessment of possible risks and anticipation of ways to prevent or address them. 

Aspect 4. Scientific achievements of the project applicants

4.1. Quality of publications of the Principal investigator during the last 5 years

4.2 Quality of publications of project applicants during the last 5 years

4.3. Balanced group of applicants: alignment of the number of staff and the level of their qualification to the purpose and objectives of the project, participation of young scientists. 

4.4. Participation of the Principal investigator and project applicants in research programs funded by domestic and international grants during the last 5 years.