Late last week, the US Office of Management and Budget (OMB) released new guidance on the use of evidence and evaluation in making budget trade-off decisions. The guidance is encourages agencies to use evidence-based decisions when developing their budgets for Fiscal Year 2014 (the essence of the Doing What Works project from the Center for American Progress).
The OMB memo notes (my emphasis):
Budget submissions also should include a separate section on agencies’ most innovative uses of evidence and evaluation, addressing some or all of the issues below. Many potential strategies have little immediate cost, and the Budget is more likely to fund requests that demonstrate a commitment to developing and using evidence. The Budget also will allocate limited resources for initiatives to expand the use of evidence, including but not limited to approaches outlined below.
OMB’s guidance provides details and examples of existing approaches agencies could pursue:
1) Proposing New Evaluations
Areas of potential focus outlined by OMB include
- Low-cost evaluations using administrative data or new technology – the Coalition for Evidence-Based Policy’s recent brief highlights how agencies can often use administrative data (such as data on wages, employment, emergency room visits or school attendance) to conduct rigorous evaluations, including evaluations that rely on random assignment, at low cost.
- Evaluations linked to waivers and performance partnerships – OMB notes that One of the best ways to learn about a program is to test variations and subject them to evaluation, using some element of random assignment or a scientifically controlled design. This is one of the primary measurement techniques utilized in the UK government’s recent Nudge report into behavioural insights.
- Expansion of evaluation efforts within existing programs – agencies can add a general policy and requirements favoring evaluation into existing grants, contracts, or waivers (may require additional legislation)
- Systemic measurement of costs and cost per outcome – Agencies are encouraged to include measurement of costs and costs per outcome as part of the routine reporting of funded programs to allow for useful comparison of cost-effectiveness across programs.
2) Using comparative cost-effectiveness data to allocate resources
Through the Pew Charitable Trust’s Results First initiative, a dozen States are currently adopting a model developed by the Washington State Institute for Public Policy (WSIPP) that ranks programs based on the evidence of their return on investment. The model calculated the return on investment to taxpayers from evidence-based prevention and intervention programs and policies. OMB wants such evidence-based programs to be identified so that a comparative analysis of these can “improve agency resource allocation and inform public understanding.”
3) Infusing evidence into grant-making
OMB outlines how agencies should consider the following approaches to increase the use of evidence in formula and competitive programs:
- Encouraging use of evidence in formula grants – OMB wants agencies to propose ways to increase the use of evidence-based practices within formula grant programs.
- Evidence-based grants: Several agencies – several departments have implemented evidence-based grant programs that apply a tiered framework to assess the evidence supporting a proposed project and to determine appropriate funding levels.
- Pay for Success – OMB notes how the Departments of Justice and Labor will be inviting grant applicants to use a “pay for success” approach, under which philanthropic or private entities (the “investors”) pay providers upfront and are only repaid by the government if certain outcomes are met.
4) Using evidence to inform enforcement
Often rigorous evaluation of strategies for enforcing criminal/environment etc. laws reveal that some approaches are significantly better than securing legal compliance. OMB is encouraging agencies to outline how their allocation of resources among enforcement strategies is informed by such evidence.
5) Strengthening agency evaluation capacity
OMB recommends agencies have a high-level official who is responsible for program evaluation, including developing and managing the agency’s research agenda, along with conducting and overseeing rigorous and objective studies.
OMB Support
As part of the memo, OMB also outlines the support it will provide to agencies to initiative and analyse the roll-out of their evidence-based initiatives. Along with organizing discussions with senior policy officials and research experts, they also plan to “reinvigorate the interagency evaluation working group” in this area.
A lot of the ideas outlined in the memo read like common sense i.e. agencies should measure and evaluate what works through the use of experiments and control groups. These results should then inform policy and decision making. The memo lays out a very specific agenda for program evaluation in agencies and champions the use of structured evaluations to inform future decision-making. The Center for American Progress and other have provided detailed reports and analysis on how to conduct such experiments and quantify the results. This, along with the interesting work by behavioural insights team at the UK Cabinet Office, should make government work more effectively through formulating policy based on evidence and rigorous evaluation of budget and management decisions. As the memo says:
Where evidence is strong, we should act on it. Where evidence is suggestive, we should consider it. Where evidence is weak, we should build the knowledge to support better decisions in the future.
