Evaluation of the applications

A thorough evaluation of open call proposals is essential for achieving the open call’s core objectives. It ensures fairness by creating a level playing field and clear communication with applicants. Moreover, it helps identify the highest-quality applications with the strongest potential for success. Additionally, providing valuable feedback to applicants and upholding the credibility of the open call host organisation all contribute to the call’s overall success.

Selection of jury members

Successful open calls rely strongly on expert juries since they are the ones eventually ensuring quality and impact of the open call. These juries act as guardians of innovation, carefully evaluating proposals to identify those with the strongest potential. Their diverse expertise fosters a rigorous process, free from bias, and ensures that only the most well-developed and feasible ideas progress.

Considering this crucial role of expert juries, it is important to invest enough time and resources to pick the best possible jury members according to their professional expertise, evaluation experience and personal qualities. Below are listed some topics to consider when selecting jury members: 

  • Relevance to the programme: Choose jury members with expertise and experience relevant to the open call’s program area or challenge theme. This ensures they can effectively evaluate proposals based on their technical merit and potential impact.

  • Alignment with evaluation criteria: Consider the pre-defined evaluation criteria for the open call. Select jury members whose skills and knowledge align with those criteria, allowing them to assess proposals comprehensively.

  • Diversity of perspectives: Strive for a jury that reflects a diversity of professional backgrounds, experiences, disciplines and viewpoints. This ensures a well-rounded evaluation process that considers different approaches and potential biases.
  • Objectivity and fairness: Look for jury members with a demonstrated ability to be objective and fair in their assessments. They should avoid personal biases and base their decisions solely on the proposals’ merit.

  • Critical thinking and analysis: Effective jury members possess strong critical thinking and analytical skills. They can examine proposals thoroughly, identify strengths and weaknesses, and make sound judgments.

  • Communication and collaboration: Jury members should be able to communicate effectively, both verbally and in writing. They need to collaborate with other jury members, discuss proposals openly, and articulate their rationale clearly. For international open calls, consideration should be given to finding a suitable common language for the jury, typically English.

“The heart of a strong evaluation jury lies in diversity, open minds, and expertise. By hand-picking a jury with different viewpoints and proven know-how, you ensure a fair and profound evaluation, fueling the discovery of truly innovative solutions.”

Once the jury members have been pre-selected through consideration of these questions, the next step is to initiate the nomination process. If, however, jury members are expected to apply, the review of application materials precedes nominations.

Below are more detailed descriptions of these phases:

  • Nomination process: Usually jury members are directly nominated by the organisations running the open call or at least they coordinate the process. However, you can also consider using an open call for jury nominations since this allows potential jury members to express their interest.

  • Application materials: Request resumes, cover letters, or brief statements from potential jury members highlighting their relevant experience, qualifications, and overall understanding of the open call process.

  • Interviews: Especially when potential jury members are not known to the open call host, conducting interviews with shortlisted candidates provides an opportunity to assess their expertise, motivation, communication skills, and understanding of the open call’s objectives.

  • Availability and commitment: Ensure that potential jury members have the time and commitment necessary to participate fully throughout the entire evaluation process. Communicate realistically the timelines, process and the needed resources.

  • Conflict of interest: Implement clear conflict of interest policies to prevent bias. Jury members should not be affiliated with any of the proposals they are evaluating. If a jury member is suspected of bias, it’s important to act promptly and notify the open call organisers immediately for further guidance.

By following these tips, one can assemble a strong jury panel with the necessary skills and expertise to effectively evaluate proposals and select the most impactful projects for the open call.

Evaluation platform

Crucial to the success of any open call is selecting the most appropriate platform to manage the evaluation process. Fortunately, several options exist to facilitate the assessment of submitted applications.

The most commonly adopted platforms and tools include the following:

  • Survey platforms: Leveraging platforms like SurveyMonkey or Typeform allows for the creation of customised evaluation questionnaires directly aligned with the established criteria for the open call. These platforms passively collect reviewer scores and feedback through a streamlined process.

  • Project management platforms: Platforms like Trello, Asana, or Monday.com can be effectively employed to organise applications, assign reviewers to specific submissions, and meticulously track the evaluation process throughout its entirety. These platforms often boast features that facilitate collaboration among reviewers, including file sharing, commenting capabilities, and collaborative editing tools.

  • Content management systems (CMS):  For organisations already possessing a website, specific plugins can be configured within certain CMS platforms (e.g., WordPress) to not only submit the applications but also manage the evaluation process.

  • Specialised platforms: Depending on the specific field of the open call, specialised platforms designed to streamline the application review process within that domain may exist. For example, some film festivals utilise platforms specifically tailored for efficient film submission and evaluation. Additionally, open-source project management platforms could be an option for organisations with the technical expertise to implement them.

It is crucial to note that alongside the chosen platform, a clearly defined evaluation rubric outlining the selection criteria for applications is necessary. This rubric should be disseminated to reviewers in advance to ensure consistent and standardised evaluations across all submissions.

Case: CommuniCity Open Call platform

The CommuniCity Open Call Platform managed both pilots and the application process for the three open calls. It utilised Salesforce for overall management, including contract signing and achieving. In contrast, evaluation was handled by the Applicator Evaluator, a web application built with Django (backend) and React (frontend).

Key content for evaluation included challenge descriptions, applications, evaluation criteria, numeric scores, and open jury feedback. Before jury evaluation began, challenges, applications, and criteria were all saved within the Applicator Evaluator.

Jury members received access to the platform and upon login, could see their assigned challenges and related applications. Each nominated jury member then evaluated applications numerically (0–5) and provided additional written feedback when necessary. Following this, Pilot Managers confirmed and submitted the evaluations. Once complete, the scores and evaluation justifications were imported into the Salesforce platform.

A sample application that has each criterion evaluated by each member of the jury.

Evaluation process

The evaluation process for open calls plays a crucial role in selecting the most promising projects and ensuring the effective use of resources. As mentioned earlier, the open call announcement should clearly outline the evaluation process, including the selection criteria in order to be transparent. While the specifics can vary depending on the organisation and the call itself, here is a breakdown of the common phases involved:

1. Eligibility check

This initial screening acts as a gatekeeper, ensuring applications meet the fundamental criteria outlined in the open call announcement. Applications that don’t fulfil these baseline requirements are typically excluded from further evaluation. This initial filtering helps streamline the process and allows reviewers to focus on proposals that have a strong chance of success. Common factors assessed at this stage include the following questions:

  • Is the applicant type (individual, organisation, etc.) eligible to participate according to the open call rules? Are all required fields within the application completed, and is the information accurate?
  • Does the proposed project align with the call’s thematic area or research focus?
  • Does the requested budget fall within the call’s funding limitations? If self-funding is requested, does the applicant demonstrate that it is available?
  • Does the application package include all required documents and forms? Has the application been written in the required language? Was the application submitted before the open call deadline?

2. Proposal evaluation

Applications that pass the eligibility check move on to a more rigorous evaluation based on pre-defined selection criteria. This in-depth assessment includes the following elements:

  • Selection criteria: The open call announcement should clearly outline the specific evaluation criteria reviewers will use to assess proposals. These criteria typically encompass various aspects of the project, such as its merit (considering originality, innovation, and potential impact), the applicant’s expertise (including their qualifications, relevant skills, and track record), the feasibility of the proposed plan (including clarity, structure, workability of the timeline and methodology), and justification for the proposed budget. Additionally, some open calls may include a criterion for evaluating the plan for dissemination and sustainability of the project.

  • Reviewers: The evaluation is often conducted by external experts in the field relevant to the open call. This ensures that reviewers possess the necessary knowledge and experience to objectively assess the proposals. In some cases, a combination of internal and external reviewers might be used.

  • Evaluation methods: Reviewers typically use a scoring system or rubric to assess each proposal against the established criteria. This rubric assigns point values or specific descriptors to different aspects of the proposal. This standardised approach helps to ensure consistency and fairness in the evaluation process.  Some open calls might involve a more narrative-based evaluation where reviewers provide detailed feedback alongside their scores.

“In open calls, evaluation isn’t just about ticking boxes, it’s also about embracing the unknown and feeding curiosity. Look beyond the familiar and seek the spark of creativity that genuinely pushes boundaries.”

3. Reviewer discussion and ranking 

Enabling discussions and collaboration among expert jury members is often highly beneficial. This fosters a fair and transparent evaluation process, potentially leading to a refinement of the evaluation scores if needed.

  • Depending on the open call structure, individual reviewers may convene to discuss their evaluations. This collaborative discussion serves several purposes. First, it allows reviewers to address potential discrepancies. For example, if reviewers have significantly different scores for a particular proposal, they can discuss the reasoning behind their evaluations and potentially reach a consensus. Second, these discussions can lead to a more nuanced understanding of each proposal, potentially resulting in minor adjustments to the initial scores. Finally, reviewers can learn from each other’s perspectives and expertise, enriching the overall evaluation process.

  • Based on the individual or collective evaluations, proposals are often ranked according to their scores or overall merit. This ranking helps identify the top contenders that best meet the call’s objectives and deserve funding or other forms of support.

4. Selection and notification 

Based on the evaluation results, the organisers select the top proposals that best align with the call’s goals and funding availability. This selection may involve a final decision by a designated committee or panel. Unsuccessful applicants typically receive notification, along with a brief explanation or general feedback depending on the call guidelines. This feedback can be valuable for future applications.

5. Funding and project Initiation 

In the case of open calls offering grants or funding, successful applicants enter into the next phase. Formal agreements outlining the terms and conditions of the funding are established between the organisation and the successful applicants. Additionally, procedures might be implemented to monitor project progress, ensure adherence to the proposal, and evaluate the project’s success in achieving its goals.

Next: Closing the open call