Limitations
- The most important limitation is in the indicator for the likelihood of discovery. This indicator is based on future citation counts. We acknowledge this is an indirect indicator. Future work (such as an analysis of the citances of the cited papers) may be able to address this limitation.
- The paper also assumes that discovery is a goal of research. This is not the primary goal of most funding entities. The charter for NSF focuses more on innovation and corresponding economic impact. The charter for NIH focuses more on diseases and treatment. The charters for state and regional sources of research funding will usually reflect issues that are geographically specific. The charters for non-profit funders have their own unique structure. Discovering new theories, concepts or measurement techniques is a commonly held secondary goal that reflects success in an academic setting (i.e. promotion and tenure).
- We are not advocating that actors (such as a program officer at a funding agency) use this method to select (or reject) individual research proposals. In the spirit of why one does a portfolio analysis, the purpose is to gain a more systemic understanding of underlying strengths, weaknesses, opportunities and threats in the current portfolio of research investments. This approach is especially useful in identifying threats due to blind spots.
- The approach has not yet been subjected to the full range of robustness and sensitivity testing that a production diagnostic would require.