Report from URSSI workshop on research software incubators

Nic Weber

March 28, 2019

Summary

At the first URSSI community workshop, a small group of participants started to discuss a model for incubating research software projects. Incubation in this context might include a structured program that helps developers plan a new community-based software project, or improve existing projects that need mentorship, strategy, or other resources in order to sustainably grow.

To further explore this topic we brought together 16 experienced funders, community managers, developers, researchers, and software users. We discussed differences between research software incubators and technology start-up accelerators - identifying important questions about the period of time for incubation, the goals for participants and funders, and potential benefits to a research community. We then heard presentations from 6 different existing programs that have an interest in software development, project management, and incubation. Using these examples we engaged in a design activity to make some initial decisions about an incubator program that URSSI could organize in the future.

The mission of the URSSI conceptualization project is to better understand challenges that individuals face when it comes to developing, maintaining, and using research software. In proposing the development of a new research institute to support these activities, URSSI has convened a number of large and small workshops. Each of these gathering are meant to collect community feedback and try to to reach consensus on emerging topics relevant to an institute focused on software sustainability. The Research Software Incubator Workshop was hosted by the University of Maryland in College Park on February 10-11, 2019. In total, 16 experts participated in a series of discussions and also delivered short presentations on their organization’s efforts in this space.

Start-ups vs Research Software Incubators

We began the workshop with an open discussion of general business and technology incubators, and what is needed (or desired) for an incubator to support research software sustainability. The purpose of a technology or start-up incubator is to launch a new product. These incubator programs provides fiscal support for a founder / product owner to identify market fit, build out or expand software features, and develop a network of interested stakeholders. Successful graduation or exit from these types of incubators are through secured funding or a product launch (often with a culminating event where the founder / product owner “pitches” the product to potential investors). The prototypical organization for technology incubators might be Tech stars or YCombinator.

In shifting the incubator model from business start-ups to research software some preliminary questions were raised :

Where would a research software incubator fit in the traditional software development lifecycle?

Grant funding is often used to develop prototypes. These prototypes are in service of either short term research results (1-2 years in length) or for additional grant seeking in the medium-term (2-4 years). Where in the development cycle incubator participation in an incubator would be helpful will vary from project to project. For prototype research the following recommendations were made:

  • An URSSI incubator could provide time or funding for a project to develop more sustainable prototypes, or to move prototypes from proof of concept to a more stable release..
  • The incubator might also be a place to plan sustainable prototyping - for example when new grants are being proposed. In this case an incubator should help a project be more competitive in the proposal process.
  • The incubator should not simply provide a window of time to shop ideas around to potential funders or users.
  • Regardless of where in the development cycle a project enters the incubator there needs to be a strong sense of the stakeholders for the software. There should also be a demonstrated need for taking time away from research (or other development tasks) to improve the long-term viability of the code / project.

There is currently a disconnect between ecosystems of public, private, and government software projects. What is needed to foster collaboration (within and across) this ecosystem for more sustainable software? What role should the URSSI incubator program play in brokering connections throughout this ecosystem?

Most research software development occurs through funding targeted at specific short-term research results. This funding does not require an environmental scan for what is already available in terms of reusable software, or relevant prototypes. This is a two-fold problem: Software isn’t discoverable because it is not well described; and the burden to reuse software is often too high for a short-term research project to invest time and effort. NSF projects, for example, reflect grant mechanisms (not a lot of coordination between different directorates or different programs). Recommendations to better connect an ecosystem of stakeholders include:

  • URSSI should try to foster ecosystems of software projects that don’t have as much redundancy. This can be accomplished by trying to help projects identify relevant existing software, and by encouraging best practices in descriptive metadata usage.
  • URSSI should try to encourage incubator projects that bring together researchers / developers from two or more sectors. The incubator might even prioritize projects that will coordinate efforts across sectors.
  • URSSI should recruit reviewers of incubator proposals, as well as mentors of accepted projects from multiple sectors. This will help to raise awareness of related software.

Decision making process of moving from small group of users (focused on particular research questions) to generalizable medium-to-large groups of users often introduces problems of coordination, collaboration, and policymaking. How would URSSI support not just overcoming these challenges, but help research software projects learn from existing successes and failures?

The following recommendations were offered:

  • URSSI should provide training around governance development for moving from 0-5 users; 6-15; 15+ , etc. The challenges of coordination and collaboration will be different in each scenario.
  • URSSI might also help projects learn how to scale between different contributor levels.
  • URSSI should recruit experts to consult on projects. For example, industry professionals who had previously worked in science could be effective mentors. There is also the opportunity to build a network of mentors through graduated incubator projects (e.g. there is an expectation that graduating from the incubator comes with the responsibility to mentor future projects).

What does success mean for an incubator? Are projects expected to graduate, and if so what are the criteria for graduation?

In a business incubator, graduation is typically built around a certain deliverable (a beta release of a product) or a successful round of funding. In the Apache incubator, graduation is accomplished by three successful releases - releases are highly coordinated and reviews by project mentors. We decided that, generally, a successful graduation would result in more accessible code, lowered barrier for future contributions, a governance model in place that matches the developers / user community, and a clear description of the funding necessary to maintain the project over the short to medium term.

This initial set of questions and recommendations were used as the basis for an activity on Day 2 to design the URSSI incubator.

Presentations

8 workshop participants gave a presentation. In advance, we asked each presenter to speak to how URSSI might support their organization / incubator. Below, we summarize the ways in which these programs might benefit from or collaborate with URSSI, and provide a link to the set of slides that were used for the presentation.

Incubator Models for Research

UW eScience Incubators - Micaela Parker

  • At the end of an eScience incubator, projects are deposited in a GitHub repository, but this isn’t stable or viable for long-term reuse. URSSI staff could consult with existing projects or even host UW eScience projects as part of a sustainability incubator.
  • URSSI staff could work with UW eScience incubators to understand what of their work is generalizable and then help solicit contributors / collaborators
  • Documentation and communicating maintenance needs were where projects struggled most and where intervention seems necessary for sustainability.
  • An incubator could also be a form of technology transfer that does not mean transfer to corporate sponsorship, but to community development. So, there could be a role for URSSI to play in that initial stepping off point by incubating projects that are graduating from campus to community.

ESIP Incubators - Annie Burgess

  • The visibility of many incubator projects is limited to ESIP, but the developed software is often useful to many domains. URSSI could help bring these projects to light through partnership and promotion. It was also suggested that the projects could have some advising or training from URSSI focused specifically on how to produce a sustainable prototype over the short funding period.
  • One of the struggles to ESIP incubators is formative assessment (projects typically run for 6-12 months). URSSI could help develop metrics for success at incubator stage of development. URSSI could also provide best practices related to formative assessment.
  • Code review is challenging for short-term projects and is likely a source of short-term thinking by incubator projects. URSSI could provide best practices around code review / tests at formative start-up stage.

Apache Software Foundation - Suresh Marru

  • Focus of meritocracy in ASF may be a poor fit for URSSI and for NSF.
  • One important lesson to learn is that mentors should be well matched with the intended stakeholders of the software. In other words, mentors for an URSSI incubator need to understand which “research culture” the software will contribute to.
  • ASF projects are more suited for a “shared reference implementation” - meaning there is a relationship between incubated projects and existing projects that leads to success. URSSI should consider this idea when deciding who to admit / recruit to the incubator, or in recruiting cohorts of incubators.

Research Software and Leadership Institutes

NumFOCUS - Andy Terrel

  • Andy presented a matrix of roles played in software development that are differentiated by forms of credit, techniques for generating software, and resources available for this kind of work (see notes Page 9). This matrix tries to demonstrate a misalignment of incentives between different roles
  • URSSI should, at least, try to align these roles in an incubator program. Practically this requires the codesign of software between different types of infrastructures.
  • NumFOCUS and related projects also have a need to understand grant writing for national funding agencies. This aspect of grant funding was seen as another potential cohort for the URSSI incubator - existing software projects that need to work on building a fundable research pitch, identifying solicitations that would be relevant, and writing grant applications. The exit for a cohort like this could be a grant proposal.

Code for Science and Society - Danielle Robinson

  • Danielle described projects supported by CS&S, and the need for coordination in an ecosystem that includes data, open source infrastructure, and research software
  • Many groups struggle to understand how to maintain existing work, when to abandon previous prototypes and start anew, how to support one another’s contributions, and how to innovate for sustained funding.
  • URSSI could play a coordinating role in either hosting an incubator, or supporting and providing outreach around sustainable practices for an Open Source Alliance incubator.
  • URSSI’s incubator could have a project graduate to fiscal sponsorship at CS&S,
  • A cohort of URSSI incubators could also focus on existing CS&S projects that want to better prepare for sustainability

Mozilla Open Leaders - Abby Cabunoc Mayes

  • This program attracts a broad variety of projects - some of which are related to research and research software.
  • URSSI could be an outlet for Open Leaders’ projects that requires greater support for software engineering.

Research on Software Communities
The workshop also included presentations from two organizational scholars that study open-source and research software development practices.

Some factors of success from OSS - Kevin Crowston
Research Software Governance - Dan Scholler

Designing an Incubator Model for URSSI

Day 2 was spent entirely on a design activity. We asked workshop participants to engage with a series of open-ended questions about design choices for an incubator program focused on research software. For each of the seven topics below we first posed some general questions, and then tried to reach consensus on what URSSI should do with respect to designing an incubator for research software.

Discussion topic 0. Incubators and other URSSI activities

Is going through an incubator necessary for a project to access other URSSI services / benefits?

  • No - URSSi will offer many forms of engagement. There should not be a requirement around participating in the incubator.

Should URSSI provide money (or other resources) to incubator projects?

  • Yes, diverse ecosystems aren’t possible without money. Funding will help give leverage to early career scientists who want to participate but need to win over superiors to get time to do the work necessary of improving software, building community, etc.

What kinds of resources (besides money) should URSSI provide?

  • Goods in kind. such as skilled staff-time from URSSI, legal representation and guidance on forming LLCs / choosing licensing schemes, etc.
  • Software engineering, data science, auditing
  • UX or design consultancy, leadership training, management training
  • Funded travel to work-sprints (bringing people together to work for targeted amounts of time), demo events, and other cohort activities
  • The funding can and should be budgeted by the incubated project, but should be approved or managed by URSSI in some way.

How should URSSI incubation work with other incubator projects and programs? Do we need an alliance/network of such programs?

  • Currently there aren’t good examples of research software incubator specific models.
  • URSSI could do one of two things: 1. Begin its own incubator program (as discussed here). 2. Act as an incubator of incubators - help other organizations design an incubator program and then help run that incubator. This would take pressure off of URSSI designing a one size fits all model of project incubation.
  • There was a strong agreement that there is an opportunity for an institute doing both of these activities.

How would an incubator program benefit URSSI?

  • Opportunity to learn from projects that go through the incubator, uncover emerging problems that overlap or cross domains (e.g. writing a code of conduct is really hard - here is a template). Spin up other consulting services that may help solve problems to that a similar project doesn’t have to go through incubator in the future.
  • In general, URSSI could use an incubator as a laboratory to better understand sustainability, develop and determine best practices, and help the community move forward towards more overall sustainability

What are the benefits to research for a software incubator? (URSSI has to make case to NSF for why incubator is important to science)

  • All of the arguments around importance of software sustainability apply to this question - decrease redundancy, increase impact, coordinate domains of research, etc.
  • Other major benefits: Opportunity for research projects to sustainably mature into broadly used software products, increase quality of funded projects, and develop prototypes that can be used in grant applications (better inputs).

Discussion topic 1. Goals of a Software Sustainability Incubator
What are the incubator’s goals?

  • Training individuals to develop, contribute to, and maintain more sustainable software
  • Training groups to work together more effectively, efficiently, and equitably.
  • Improving the health of a code base through testing, auditing, etc.
  • Improving the ability for a project to engage communities of users and contributors.

What are goals for a project being incubated?

  • A project should want to learn how to sustain itself into the future. This should include a desire to learn about things like leadership, governance, admin, fundraising capacity
  • Lowered technical/community barriers to project participation (code cleanup, documentation, documented processes, open infrastructure). Improve capacity for software to be reused.
  • Increase use cases (more use cases not just more users)
  • Evolve or improve the technical capacity of the project and its developers
  • Improve the technical leadership or recruit people who can act as long-term project mentors.

What types of research software projects should enter an URSSI incubation process, or what is necessary of a project to be eligible for incubation?

  • Projects should have some demonstrated research impact (or clear argument why going through the incubator would lead to research impact)
  • Projects should have an audience that would substantially benefit from incubation (e.g., there are users of the software beyond a single lab, and a single university).
  • Projects that have the capability of improving existing software, replacing ineffective software, or replacing proprietary options.

Discussion topic 2. Applications / Roles
What should an application to the URSSI incubator include?; Who might apply, and what would be their range of motivations?; Are there self-assessment frameworks that should be used in making an application? (e.g. Apache Maturity Model or Openness scales); What are the roles that must be identified by a proposal? (e.g. Project lead, contributor, mentor)

This topic was addressed throughout discussion around goals and criteria for acceptance.

Discussion topic 3. Criteria for Acceptance
What are the criteria for being acceptable for entering the URSSI incubator?

  • Projects should be matched with available funding resources, fit with other related projects (in order to build a cohort model), and relevant mentors
  • The capabilities of the researchers / developers to make progress towards a more sustainable software product (broadly defined) should also be a key criteria for evaluation.

Discussion topic 4. Incubator activities
What should a research software incubator actually do. There was a proposal for 3 stages of work in the incubator:

  1. Pre-incubator work - working with mentors and URSSI staff to scope goals, define budget, and establish evaluation criteria.
  2. In-person / remote incubator work - complete education, training, software consulting, financial planning, etc.
  3. Post-incubator - presentation to URSSI community, mentorship for future URSSI incubator projects

How long should the incubator period last?
This will depend upon the types of projects that are being incubated. The incubator could establish an annual schedule for accepting applications and starting cohorts.

  • University software projects could align with academic calendar? (3-9 months)
  • Open-source projects (non-academic) could happen at any time.

What does an incubator provide?

  • Education / training on project management, software development, community engagement.
  • Governance training- how to grow and/or managing community development
  • Training in how to seek and manage fiscal sponsorship / grants / budgeting
  • Sustainability planning
  • Legal Guidance from pro-bono law school professors
  • Specialized technical help: cyber security guidance, human factors, etc.

Who would manage an URSSI incubator?
The incubator program likely needs a .5 FTE appointment. This person needs to have skills/experience:

  • Organizational development
  • Program manager
  • STEM background
  • Must have software skills or a partner that can play the technical lead
  • Business Development

Would URSSI incubate a project that wants to commercialize?

  • No. There are many other places to explore commercialization.

Discussion topic 5: Evaluation and Progress in the Incubator
What are the milestones or metrics of evaluation for progress, and eventual graduation from an incubator? What are criteria for an unsuccessful exit (not graduating)? We should expect that some projects will fail, but this should not be detrimental to any individual who fails. What is the role of the URSSI incubator in extending credit to a project that has graduated? What sorts of value or markers of sustainability should this signal? How?

  • A successful graduation means more accessible code, lowered barrier for future contributions, a governance model in place that matches the developers / user community, and a clear description of the funding necessary to maintain the project over the short to medium term.
  • The project exiting should still ideally want to be part of the research ecosystem, and of the URSSI incubator mentorship

What does a successful outcome look like (and what is an effective “return” on funder investment")?

  • Success should be defined along key metrics (as of yet undefined). A major contribution of URSSI should be developing those metrics.
  • Long-term evaluation should include assessment of the project in 1, 2, or 3 years removed from the incubator.
  • For a research funder the return on investment might be higher quality submissions (inputs)

What does a “demo day” look like for research software?

  • Given that URSSI will likely be a virtual organization this may require a particular venue (e.g., a conference that makes sense for the cohort)
  • URSSI may need an annual event where graduating projects report their results and future plans. This summit could also be useful for building corporate sponsorship, and building sustainable interaction between cohorts.

Discussion 6: Sustainability
What are the models for funding an incubator?

  • Corporate or private partners could donate time, mentorship, etc. This could be for a single project, or an entire cohort of projects.
  • Graduated projects could donate time or build in future grant funds to support URSSI’s incubator.
  • Fiscal sponsors could provide some overhead to the incubator program if it sponsored one or more of their projects.

How should resources be awarded? Do all projects receive the same amount of funding?
This could be solved in a number of ways:

  • Each cohort member gets the same amount
  • URSSI could develop a funding scale at the application phase - a range of services / funds would then be available at each scale - with projects opting designating which scale their application should be considered under.
  • Incubator projects could also be sponsored by an external group (or matching funds). For example - a fiscal sponsor could match all awards for a particular cohort.

If there are no resources awarded why would a group participate in this incubator?

  • Institutional (project & people) recognition
  • Incubator could be an educational process for projects that were looking to reach more users, grow contributors, etc.
  • Join a network of URSSI members, alumni projects, and others

Can going through an URSSI incubator be a way to credential software for NSF panel reviews?

  • Credentialing will probably be difficult to pull off in a way that doesn’t seem like a guarantee that a project is sustainable if it graduates from the URSSI incubator.