What Cyberinfrastructure Funding Covers (and Excludes)

Measuring Research & Evaluation Success in Cyberinfrastructure Workforce Development

The Research & Evaluation sector plays a vital role in assessing the effectiveness of projects that integrate cyberinfrastructure professionals' services into research, while fostering education, training, and recognition that address CI workforce development needs. As part of the Grants to Cyberinfrastructure Education, Training and Recognition for Workforce Development Needs, this sector is crucial in measuring the outcomes and impact of funded projects.

Defining Research & Evaluation Scope and Boundaries

The scope of Research & Evaluation in this context involves assessing the effectiveness of projects in achieving their stated goals, objectives, and outcomes. This includes evaluating the quality of education and training programs, the impact of CI professionals' services on research, and the overall CI workforce development. Concrete use cases include assessing the efficacy of training programs, evaluating the adoption of CI services, and analyzing the career progression of CI professionals. Applicants should focus on developing robust evaluation methodologies, collecting and analyzing relevant data, and reporting on outcomes. Those who shouldn't apply are those without a clear understanding of evaluation methodologies or those who lack experience in assessing complex projects.

One concrete regulation that applies to this sector is the National Science Foundation's (NSF) requirement for grantees to comply with the NSF's Data Management Plan (DMP) policy, which mandates that researchers provide a plan for managing and sharing their data. Another is the need to adhere to the standards set by the Office of Management and Budget (OMB) for evaluating federal programs.

Trends and Priorities in Research & Evaluation

The Research & Evaluation sector is witnessing a shift towards more rigorous and nuanced evaluation methodologies, driven in part by the NSF's emphasis on using robust evaluation methods to assess the impact of its funded projects. There's also a growing recognition of the importance of evaluating not just the immediate outcomes of projects but also their longer-term impact on the CI workforce. Capacity requirements include developing staff expertise in evaluation methodologies, investing in data collection and analysis infrastructure, and fostering a culture of evaluation within organizations. The NSF's Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs are examples of initiatives that prioritize evaluation and impact assessment.

Operational Challenges in Research & Evaluation

One verifiable delivery challenge unique to this sector is the need to balance the demands of conducting rigorous evaluations with the need to support project implementation and delivery. This can be particularly challenging in the context of CI workforce development, where projects often involve complex interventions and multiple stakeholders. Staffing requirements include hiring personnel with expertise in evaluation methodologies, data analysis, and project management. Resource requirements include investing in data collection and analysis tools, as well as providing ongoing training and support for staff.

Risk Management in Research & Evaluation

Eligibility barriers for applicants include a lack of experience in evaluating complex projects, inadequate evaluation methodologies, and insufficient resources for data collection and analysis. Compliance traps include failing to adhere to the NSF's DMP policy or OMB standards for evaluating federal programs. What's not funded includes projects that lack a clear evaluation plan or fail to demonstrate a commitment to using evaluation findings to improve project delivery.

Measuring Outcomes and Impact

Required outcomes for Research & Evaluation projects include assessing the effectiveness of education and training programs, evaluating the impact of CI professionals' services on research, and analyzing the career progression of CI professionals. Key Performance Indicators (KPIs) might include metrics such as the number of participants in training programs, the percentage of participants who report improved skills or knowledge, and the number of research projects supported by CI professionals. Reporting requirements include submitting regular progress reports to the funder, as well as presenting findings at relevant conferences or workshops. Applicants should be prepared to demonstrate their ability to design and implement robust evaluations, collect and analyze relevant data, and report on outcomes in a clear and transparent manner.

Q: What evaluation methodologies are most suitable for assessing the impact of CI workforce development projects? A: The most suitable evaluation methodologies will depend on the specific project goals and objectives, but may include quasi-experimental designs, mixed-methods approaches, or social network analysis, among others. When designing an evaluation, it's essential to consider the NSF's priorities and the need to assess both immediate outcomes and longer-term impact, as reflected in the NSF's SBIR and STTR programs.

Q: How can applicants ensure that their evaluation plans are robust and feasible? A: Applicants should ensure that their evaluation plans are developed in consultation with relevant stakeholders, including project staff, participants, and external evaluators. They should also ensure that their plans are aligned with the funder's priorities and requirements, such as the NSF's DMP policy, and that they have access to the necessary resources and expertise to implement the evaluation.

Q: What are the key data sources for evaluating the impact of CI workforce development projects? A: Key data sources may include participant surveys, administrative data, and metrics on research productivity or collaboration. Applicants should be prepared to demonstrate their ability to collect and analyze relevant data, and to report on outcomes in a clear and transparent manner, using relevant metrics such as those related to SBIR grants or NSF grants.