Glimpsing Forward

Clay Shirky’s recent TED talk, “How the Internet will (one day) transform government,” is a powerful assessment of how governments can utilize the power of the Internet to lower the costs of doing things. He investigates the government’s role in the coordination of collective effort, and how the intrinsic features of the web can mean the dramatic lowering of costs facilitating big changes.

Via TED:

The open-source world has learned to deal with a flood of new, oftentimes divergent, ideas using hosting services like GitHub — so why can’t governments? In this rousing talk Clay Shirky shows how democracies can take a lesson from the Internet, to be not just transparent but also to draw on the knowledge of all their citizens.

The UK Cabinet Office’s behavioural insights team recently published a guidance document on the topic ‘Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials‘. The paper describes how the government should be more rigorous in assessing the impact of its policies to make sure they’re effective, deliver value for money and represent the best use of government resources.

Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials

The central thesis of the guidance is that ‘Randomised controlled trials (RCTs) are the best way of determining whether a policy is working’ and these should be used more to test the effectiveness of public policy interventions. The report says random trials could be applied to “almost all aspects of public policy”. It recommend starting the trials in uncontroversial areas (see Reducing fraud, error and debt) – such as the wording on tax reminders (one trial suggested different wording could improve effectiveness by around 30%)  – before working up to more contentious social issues.

9 Key Steps to Conducting a Randomised Control Trial

The paper then goes on to outline in detail the 9 separate steps required to setup any Randomised Control Trial. These steps represent the Behavioural Insights Teamʼs ʻtest, learn, adaptʼ methodology, and focus on understanding better what works and continually improving policy interventions to reflect learning from different trials.

Test – (ensuring you have put in place robust measures that enable you to evaluate the effectiveness or otherwise of the intervention)

1) Identify two or more policy interventions to compare (e.g. old vs new policy; different variations of a policy).
2) Determine the outcome that the policy is intended to influence and how it will be measured in the trial.
3) Decide on the randomisation unit: whether to randomise to intervention and control groups at the level of individuals, institutions (e.g. schools), or geographical areas (e.g. local authorities).
4) Determine how many units (people, institutions, or areas) are required for robust results.
5)  Assign each unit to one of the policy interventions, using a robust randomisation method.
6) Introduce the policy interventions to the assigned groups

Learn – (is about analysing the outcome of the intervention, so that you can identify ‘what works’ and whether or not the effect size is great enough to offer good value for money)

7) Measure the results and determine the impact of the policy interventions.

Adapt – (using learnings to modify the intervention, that that you are continually refining the way in which the policy is designed and implemented)

8) Adapt your policy intervention to reflect your findings.
9) Return to Step 1 to continually improve your understanding of what works.

Examples

Along with providing detailed steps for how to create a Randomised control Trial, the document outlines a series of examples to illustrate the benefits that can occur from such systematic analysis of public policy interventions.

One such example concerns a controlled experiment to study the impact of three schemes to support incapacity benefit claimants in England: support at work, support focused on their individual health needs, or both. The experiment found that while the extra support cost £1400 on average it found no benefit over the standard support that was already available. Thus, the RCT ultimately saved the taxpayer millions of pounds as it provided unambiguous evidence that the additional support was not having the intended effect.

These kinds of findings provides important feedback for policy-makers who can then search elsewhere for effective solutions to practical problems.

Debunking the myths

Randomised trails are common practice in many areas of development and science, but are used nearly universally as a means of assessing which of two medical treatments works best. This was not always the case, however, and when they were first introduced in medicine they were strongly resisted by some clinicians who believed that personal expert judgement was sufficient to decide if a particular treatment was effective.

The myths around RCTs primarily focus on four areas:

1) We don’t necessarily know ‘what works’ – the paper counters this by explaining how RCTs can still be worthwhile in quantifying the benefit and which aspects of a program have the greatest effect. We should be willing to recognise that confident predictions by policy experts often turn out to be incorrect and RCTs demonstrate where  interventions which were designed to be effective were in fact not.

2) RCTs don’t have to be expensive – Costs involved very much depend on how the RCT is designed and with planning they can often be a cheaper than other forms of evaluation.  This is particularly true when a service is already being delivered, and when outcome data is already being routinely collected. Rather than focusing on what a RCT costs to run, the paper suggests, it might be more appropriate to ask: what are the costs of not doing an RCT?

3) RCTs can be unethical – There is often objects to RCTs on the basis that it is unethical to withhold a new intervention from people who could benefit from it. This is particularly true in terms of spending on programmes which might improve health, wealth or educational attainment by one group. They note how we can not be certain of the effectiveness of an intervention until it is tested robustly and sometimes interventions which were long believed to he effective have turned out to be ineffective or even harmful on further experimentation.

4) RCTs do not have to be complicated or difficult to run – The paper notes that ‘RCTs in their simplest form are very straightforward to run.’ The paper explains in great detail the pitfalls and provides much advice on the steps needed – as briefly outlined above – to create such a trial.

The gathering of evidence to support public policy is increasingly important when government resources are stretched. There is a huge responsibility on public policy leaders to show the effectiveness of their projects and ensure value for money. Only those services delivering proportionate value for money should be funded, while those programmes failing to deliver evidence-based results should face reform.

(more at Cabinet Office and guardian.co.uk)

Late last week, the US Office of Management and Budget (OMB) released new guidance on the use of evidence and evaluation in making budget trade-off decisions. The guidance is encourages agencies to use evidence-based decisions when developing their budgets for Fiscal Year 2014 (the essence of the Doing What Works project from the Center for American Progress).

The OMB memo notes (my emphasis):

Budget submissions also should include a separate section on agencies’ most innovative uses of evidence and evaluation, addressing some or all of the issues below. Many potential strategies have little immediate cost, and the Budget is more likely to fund requests that demonstrate a commitment to developing and using evidence. The Budget also will allocate limited resources for initiatives to expand the use of evidence, including but not limited to approaches outlined below.

OMB’s guidance provides details and examples of existing approaches agencies could pursue:

1) Proposing New Evaluations 

Areas of potential focus outlined by OMB include

• Low-cost evaluations using administrative data or new technology – the Coalition for Evidence-Based Policy’s recent brief highlights how agencies can often use administrative data (such as data on wages, employment, emergency room visits or school attendance) to conduct rigorous evaluations, including evaluations that rely on random assignment, at low cost.
• Evaluations linked to waivers and performance partnerships – OMB notes that One of the best ways to learn about a program is to test variations and subject them to evaluation, using some element of random assignment or a scientifically controlled design. This is one of the primary measurement techniques utilized in the UK government’s recent Nudge report into behavioural insights.
• Expansion of evaluation efforts within existing programs – agencies can add a general policy and requirements favoring evaluation into existing grants, contracts, or waivers (may require additional legislation)
• Systemic measurement of costs and cost per outcome –  Agencies are encouraged to include measurement of costs and costs per outcome as part of the routine reporting of funded programs to allow for useful comparison of cost-effectiveness across programs.

 2) Using comparative cost-effectiveness data to allocate resources

Through the Pew Charitable Trust’s Results First initiative, a dozen States are currently adopting a model developed by the Washington State Institute for Public Policy (WSIPP) that ranks programs based on the evidence of their return on investment. The model calculated the return on investment to taxpayers from evidence-based prevention and intervention programs and policies. OMB wants such evidence-based programs to be identified so that a comparative analysis of these can “improve agency resource allocation and inform public understanding.”

 3) Infusing evidence into grant-making

OMB outlines how agencies should consider the following approaches to increase the use of evidence in formula and competitive programs:

• Encouraging use of evidence in formula grants – OMB wants agencies to propose ways to increase the use of evidence-based practices within formula grant programs.
• Evidence-based grants: Several agencies – several departments have implemented evidence-based grant programs that apply a tiered framework to assess the evidence supporting a proposed project and to determine appropriate funding levels.
• Pay for Success – OMB notes how the Departments of Justice and Labor will be inviting grant applicants to use a “pay for success” approach, under which philanthropic or private entities (the “investors”) pay providers upfront and are only repaid by the government if certain outcomes are met.

4) Using evidence to inform enforcement

Often rigorous evaluation of strategies for enforcing criminal/environment etc. laws reveal that some approaches are significantly better than securing legal compliance. OMB is encouraging agencies to outline how their allocation of resources among enforcement strategies is informed by such evidence.

5) Strengthening agency evaluation capacity

OMB Support

As part of the memo, OMB also outlines the support it will provide to agencies to initiative and analyse the roll-out of their evidence-based initiatives. Along with organizing discussions with senior policy officials and research experts, they also plan to “reinvigorate the interagency evaluation working group” in this area.

A lot of the ideas outlined in the memo read like common sense i.e. agencies should measure and evaluate what works through the use of experiments and control groups. These results should then inform policy and decision making. The memo lays out a very specific agenda for program evaluation in agencies and champions the use of structured evaluations to inform future decision-making.  The Center for American Progress and other have provided detailed reports and analysis on how to conduct such experiments and quantify the results. This, along with the interesting work by behavioural insights team at the UK Cabinet Office, should make government work more effectively through formulating policy based on evidence and rigorous evaluation of budget and management decisions. As the memo says:

Where evidence is strong, we should act on it. Where evidence is suggestive, we should consider it. Where evidence is weak, we should build the knowledge to support better decisions in the future.

(h/t The Business of Government)

A report by the UK Cabinet Office’s behavioural insights team published earlier this year, reveals that by applying behavioral insights, the government could reduce fraud, error and debt by billions. It claims that small changes to government processes, forms and language can have a big impact on behaviour and if applied nationally could save “hundreds of millions of pounds”.

The cabinet office team has close links with Professor Richard Thaler, co-author of Nudge: Improving Decisions About Health, Wealth, and Happiness, with the result that many of the ideas espoused in the book are outlined and implemented in the research paper below.

Behavioural Insights Team Paper on Fraud, Error and Debt

Seven simple steps

The report highlights seven simple steps based on evidence from behavioural science that show how “by going with the grain of how people behave, we can reduce the prevalence of fraud, error and debt”

Lessons

The results of many of the trials indicate the changes in behaviour that can result from highlighting key messages and actions required, presenting information more effectively, and exploring different types of communication:

The insights outlined in this document, applied in a range of different contexts and settings, show that not only is it possible to apply behavioural insights to reduce fraud, error and debt, but also that it can be done in a highly cost-effective way.

The report strikes a cautionary note, however, that not all techniques are feasible. For example, adding a post-it note or a handwritten name to an official communication seems to increase responses, but is not feasible, “given the scale of many government communications”. Thus the effectiveness of interventions will depend heavily on the context in which they are applied:

Policymakers should innovate, but should do so with humility about the limits of current knowledge, and with respect for what is acceptable and helpful to the public whom we serve.

This echos some of the sentiment from the UK House of Lords Science and Technology Sub-Committee’s report last year Behaviour Change in which they found:

…“nudges” used in isolation will often not be effective in changing the behaviour of the population. Instead, a whole range of measures – including some regulatory measures – will be needed to change behaviour in a way that will make a real difference to society’s biggest problems.

Links

• Cabinet Office: Behavioural Insights Team publish paper on fraud, error and debt

• Cabinet Office: Behavioural Insights Team annual update

• House of Lords Science and Technology Sub-Committee’s report, Behaviour Change

• Nudge Blog

A few weeks ago, I lodged an Freedom Of Information (FOI) request with the Department of the Taoiseach for information related to the setup and running of MerrionStreet.ie. I requested the development and running costs of the site since its launch in July 2012.

The site was built for the Government on the WordPress Open Source software platform by Arekibo (an Irish digital media company). The reported cost of the project was €40,000 and the project took about five months to build (from initial RFP to the go-live in mid 2011).

The reason for the request was to see if the use of Open Source software (such as WordPress) can really deliver an efficient cost-effective website, or whether there were any hidden costs associated with its deployment. MerrionStreet.ie is a relatively straightforward website based on the LAMP (Linux, Apache, MySQL, php) stack, and does not require complex interfaces or heightened security. It should therefore serve as a good barometer for whether there is efficient hosting platform available within the Irish Government for such sites.

The costs provided based on the FOI are outlined below and broadly in line with previously released estimates:

The FOI response also detailed the following costs:

• Total running costs for Merrionstreet since inception in July 2010 are € 57,777.35 (as per above € 45,751 in 2010 and an estimated € 12,026 for 2011)
• Total costs paid to Arekibo to date are €35,204
• Total staff costs since July 2010 are €76,965. The team that maintains and updates the site content is drawn from staff in the Government Information Service in the Department of the Taoiseach, with the exception of two temporary staff – journalism graduates requiring relevant experience who were recruited at CO level (although one has recently left the Department).

Hosting spend and cost per request

Out of all the costs above, one of the most expensive is the hosting costs estimated to be €4,500 for this year. This is despite the fact that many of the bandwidth intensive objects on MerrionStreet.ie e.g. videos, are stored on third party platforms such as YouTube. The website is currently hosted by the Local Government Computer Services Board and costs are calculated on a cost sharing basis at the end of each year. The costs for 2011 are expected to be in the range of €4,000 to €5,000.

The site has had 784,336 pageviews (see Google Analytics below – requested through FOI) since its launch in July 2010. With hosting charges of an estimated €6,500, this works out at €8.28 per 1000 page views. In comparison to other sites (i.e. based on an analysis of UK based Gov websites), this appears to be high given the site is a static php site hosted on open-source low cost stack software.

There has been 202,116 unique visitors to the site with an average of 2.47 pageviews per visitor. This works out at a cost per 1000 visitors of around €20.45 – again relatively high in comparison to other sites.

Moving to Cloud computing

Full comprehensive data on the cost and analytics for Irish Government websites is not freely available. Many government websites around the world freely release their web statistics (e.g. NY Senate and UK Dept. Business, Innovation and Skills), but there are no examples of Irish central government departments publicly releasing this data.  This data would make a good addition to the Government’s proposed centralised open data portal.

The Government has said it will continue to enhance the use of cloud computing in the public service and a “Cloud Computing Strategy” for the public service is expected to be published at the end of the first quarter 2012. The recent Public Service Reform plan contains a provision to “Seek, through market exercises, to develop a compelling case over traditional computing provision for infrastructure-as-a-service (IAAS) provision for the public service”.

Moving many sites (particularly relatively simple static sites such as those under the remit of the Dept. of the Taoiseach), to a concurrent cloud hosting environment would reduce the hosting expenditure and provide for improved performance for the site users. A cloud infrastructure would reduce costs by allocating resources as the web sites require them, and should be seen as a priority during this period of economic austerity. MerrionStreet.ie is an optimum site to migrate and its corresponding hosting expenditure should be monitored to see if some a move can significantly reduce costs for 2012.

MerrionStreet.ie 20100712-20111219 Analytics Dashboard Report