 |
Feature Topic
April-June 2007
RD&D Planning and Evaluation
The ECAI task group assigned to Topic B ("Improve deployment partnerships between industry and government") during the first workshop in 2006 held its first conference call late last year. At the time, the group decided to conduct a survey of member organizations, which would compare similarities and differences among the participating organizations relating to RD&D goal setting, activities, and evaluation. The purpose of the survey was to determine the extent to which members might apply similar and/or successful models and methods – and to learn from other organizations. While interest at the time seemed high, only two organizations completed the online survey. As a result, the task group is pursuing a different path for learning and sharing within this topic area.
This article is intended to share some of the experiences of the New York State Energy Research and Development Authority (NYSERDA) and the Clean Energy States Alliance (CESA) – the team leads for this collaborative effort – as they relate to RD&D practices used by state organizations.
Logic Models. Starting in 2003, logic models were developed for several of NYSERDA's larger RD&D programs and for the portfolio as a whole. The portfolio logic model was instrumental in identifying the different types of projects funded by NYSERDA and the appropriate metrics to use for assessing progress and gauging success. For example, one component of the DG-CHP program was designed to fund demonstrations of CHP equipment that is commercially available but has a high degree of performance uncertainty from the perspective of facility owners. The primary goal of the demonstration component of the program was to increase the number of CHP installations in a variety of settings and report performance data. The second component of the DG-CHP Program was aimed at developing and commercializing distributed generation products manufactured in New York. These two components have different goals and, thus, needed to be evaluated differently.
Peer Review. In 2004, NYSERDA pilot-tested a new approach to evaluating RD&D projects. This approach was a hybrid of the peer-review method and a composite scoring method developed for the Advanced Technology Program (ATP). Information was collected for five projects and presented to five sets of peer reviewers to obtain their assessment on six performance criteria: knowledge creation, knowledge dissemination, commercialization progress, potential energy benefits, potential economic benefits, and potential environmental benefits. The results of the pilot study revealed that the method was applicable to diverse RD&D projects. The method was then applied at the program level using two very different programs – both programs were rated very positively by the peer reviewers. The reviewers also provided lengthy comments that were helpful in understanding their scores and providing insights about program improvements.
Aggregate Analysis. Another evaluation activity used by NYSERDA is the tracking of the RD&D projects as a portfolio over time. Examples of this analysis include the source of co-funding for different types of projects. For example, other government organizations were more likely to co-fund policy research than demonstration projects. Another example is the proportion of NYSERDA funding of project cost for different types of projects. The analysis showed that NYSERDA's share of project cost was lower as products approached commercialization. Another example of an aggregate analysis is the shift in emphasis over time on different technological categories.
In other news, CESA is working with its member states to help identify and share best practices in technology and policy research. Working with NYSERDA and the Massachusetts Renewable Energy Trust, CESA staff worked with the state funds to determine the value in the different tools available to determine the effect of their programs on economic development in the clean energy sector.
CESA, in cooperation with Lawrence Berkeley National Laboratory, has worked with its member states to evaluate the early experiences of the state clean energy funds. A series of Case Studies evaluates specific projects (or policies) through a review of the project's history, the role of the state fund in the project, the project economics, and lessons learned. CESA will also conduct original research and analysis on state clean energy funds from time to time. These reports have analyzed how to design PV incentive programs to promote performance, the impact of state clean energy fund support for utility-scale renewable energy projects, and a survey of state clean energy fund support for earlier-stage renewable energy and fuel cell technology R&D.
Through an informed collegial network of peers, CESA members are able to candidly explore problems and solutions. CESA provides a forum for members to convene and discuss issues and best practices from across the country, avoiding pitfalls and improving the effectiveness of program investments.
For more information, contact Paul DeCotis or Helen Kim at 518-862-1090 or Jennifer DeCesaro at 303-296-0475.
Return to News page.
|
|
