What would make corporate evaluations more effective?

(Forthcoming in AfDB’s magazine ‘Evaluation matters’, Q4 2017)

This article makes a case for evaluators and managers to adopt a common management language in order to increase the likelihood that the evaluation recommendations will inform the design of new programs.

Relying on the concepts of social learning, I argue that adopting project management thinking will not only provide a powerful counter-factual analysis tool for program evaluators but also increase their perceived value to managers. Additionally, corporate evaluations are more likely to add value to the organization if they explicitly assess the costs of recommended reforms and the corporate choice architecture, thereby eliminating the misconceived ideas which inhibit change.

Although organizational assessment lies slightly outside the scope of traditional evaluation questions (i.e. relevance, effectiveness, efficiency, impact, and sustainability), evaluators are nevertheless well placed to provide feedback about corporate bottlenecks. Providing insights about the implementing organization, evaluations serve an important accountability role. But the question is: do the independent evaluative findings on corporation practices add value for learning purposes? And what ensures that the reforms do actually take place?

We know from experience that in international development, providing facts about the benefits of a certain action is insufficient for the action to be implemented. In sanitation projects, for example, explaining the benefits of hand washing does not necessarily result in better hygiene.

People are selective social learners, claims Joseph Henrich in his book ‘The secret of our success’. They do not change their habits purely based on “the facts” or “education” (Henrich, 2016). People, he explains, copy the behavior of influential people in their social circles. They are more likely to learn from those whose prestige, success, sex, dialect, and ethnicity they consider important.

The same phenomenon might explain why evaluative recommendations have not entered into the planning phase of new programming or informed organizational redesign to the desired degree. So far, the evaluators’ focus on providing evidence for accountability purposes has not been sufficiently conducive to learning and reform. So, what would make evaluation recommendations more potent, leading to an added value for the organization?

Experimenting with charitable giving, researchers Christine L. Exley and Judd B. Kessler conclude that contrary to our expectations, people may see the potential for improvement as a reason to avoid the action. In their working paper ‘The Better is the Enemy of the Good’ (Exley & Kessler, 2017), they explore a “behavioral phenomenon that may contribute to why information fails to encourage desired behaviors”. They conclude that ‘the better’ is seen as ‘the enemy of the good’, particularly when “agents have a self-serving motive to let it be”. The change agents are inclined to deem the actions costly, which explains why information which encourages actions with future benefits may not be taken into consideration.

“In standard economic theory, information helps agents optimize. But providing agents with information about the benefits of taking an action often fails to encourage that action”. Exley & Kessler, 2017 How can we ensure the content of evaluation recommendations is relevant for managers and is framed properly to encourage action? And further, how can we increase the status and influence of evaluators? In my view, the knowledge management team, evaluators and the operational managers should inch closer to each other and learn to speak a common language.

That would help shake the impression that evaluators are merely aloof critics trained in theoretical economics and social science who have no clear stake in the game.

The process of corporate evaluations, or organizational assessments, should be partially collaborative instead of conducted on the strict independence condition, ie. executed in close dialogue with the departments under evaluation.

I believe that management knowledge or experience makes a more effective evaluator. Revolving more closely around the receiving managers’ core expertise increases the chances of added value to the organization. Common management language can more likely spur reform. But it might mean that evaluators compromise their independence to a certain degree.

Case for digging deeper into projects

Having assessed a wide variety (but admittedly, a small number) of undertakings – projects, programs, emergency evacuation and a global strategy – during my three-year evaluation experience in Canada, two phenomena seem to explain the low effectiveness of donor-funded programming. One has to do with local ownership but the second explanation as to why the intended outcomes have not been fully achieved within the budgetary or time constraints is related to the capacity to manage projects - the main vehicles of international development - of the implementing organization. Namely, their low project management skills.

It is counter-intuitive to dig deeper into the project level when we talk about the organizational assessments or corporate evaluations; However, the evaluators’ unit of analysis on the performance of programs, is nevertheless individual projects.

Projects provide feedback for both managers and evaluators about the program’s or portfolio’s performance. After all, programs are simply groups of synergistic projects, which are managed together based on organizational strategies and priorities. They are grouped together into a program as their expected results are similar enough so that they can be aggregated into a higher-level outcome. Managing similar projects jointly is more efficient.

But what are projects, anyway?

And what can project management principles teach evaluators? Project Management Institute has defined projects as temporary endeavors with a clear beginning and end, which result in a unique product, service or an output (Project Management Institute, 2008). It thus follows from the definition that building a school, organizing a world summit, building a water purification system or creating a new educational network can be planned and executed with similar management tools.

The holy book of project management - Project Management Body of Knowledge or PMBoK – adds that due to the uniqueness of each project, there can be many uncertainties about it and the risks can be high.

Project tasks are most likely new to the team, which necessitates a more rigorous planning process than for routine work. Specific training may often be required. In order to meet the project requirements, PMBoK states, the managers need to apply knowledge, skills, tools, and techniques to project activities during the five distinct phases (or life cycles): initiating, planning, executing, monitoring and controlling and closing.

Managing projects also involves knowing how to control requirements, address stakeholder needs, and balance the scope, quality, schedule, budget, resources, and risk. As these eight knowledge areas apply unevenly across the five project phases, PMBoK has charted them in a 2D matrix in order to assist managers to be more effective and economical in their work. It is crucial to keep an eye on the knowledge areas, as they can be subject to risks such as unexpected events which might negatively influence the project.

Managing projects is a complicated undertaking, requiring constant juggling of distinct stakeholders’ needs within the constraints of scope, time, budget and quality. The trade-off between the constraints means that if the scope of the project is changed it has implications for the budget and schedule. Changing quality requirements can also have implications for cost and duration, as low-quality products tend to be cheaper. Scope add-ons increase the budget and can ultimately postpone the completion date. Similarly, if the project is accelerated (time is crunched), it increases the budget and the risks to completion. If the budget increase is not possible, the scope of the project needs to be reviewed, and perhaps cut back. Different stakeholders might have conflicting views on which element of the project can be cut, and the quality of the end product might not meet the expectations of any stakeholder.

Organizational design has the biggest impact on program effectiveness and efficiency

Assessing program effectiveness requires that evaluators compare the planned project objectives with the ones achieved. The efficiency assessment means that evaluators would compare the resources applied and the results achieved. These necessary tasks assume that projects and programs are first planned and then executed. Yet, some evaluation reports suggest that this is not necessarily the case.

The undertakings I have evaluated reveal that often, the initiation phase and its associated processes have been omitted altogether, whereas the planning phase has often been blended with the execution phase. Lack of planning leads to confusion with project scope (what is the intended outcome?) and severely impacts both project cost and time management, i.e. effectiveness and economic efficiency. Low capability to plan, execute and monitor projects is directly tied to the organizational capacity.

Project management knowledge allows evaluators to chart powerful counter-factual analysis about how the project could have been managed more efficiently and effectively.

Understanding the effort needed to plan and manage projects, the main vehicles of development interventions, thus becomes relevant for the evaluators in order to avoid tradeoffs between efficiency and effectiveness. Corporate evaluations need to assess whether the managers are sufficiently trained to manage projects effectively (so that projects would achieve all the elements in the scope) and efficiently (so that the planned activities were budgeted individually and executed within the given time frame).

Project Management Processes (PMBoK 2013, p. 60)

Lost in Translation

Using and referring to managerial vocabulary in both evaluations and in communicating evaluation findings back to the managers (about the capacity to plan and manage projects) can create a common understanding for executives and evaluators.

IDEV has already dramatically stepped up its knowledge management, dissemination and outreach activities (AfDB IDEV, 2015) ensuring the high quality of evaluations, communications, engagement and follow-up with managers. Sharing a common language, signaling that evaluators have skin in the game, and demonstrating relevance to the corporate mandate, all assist with the learning process.

But the organizational assessments could also include understanding the (perceived) cost of recommended change to managers and the incentives which explain the inaction. In International Financial Institutions, both political interests and bureaucratic incentives can influence the decision to intervene in the mechanical decision-making process.

For example, Lang and Presbitero reviewed the degree of discretion embedded in the World Bank-IMF Debt Sustainability Framework to understand the decision-making process of international financial institutions. After synthesizing debt analyses conducted between 2006 and 2015 for low-income countries, the authors demonstrated that both political interests of beneficiary countries near elections and bureaucratic incentives influence the decision to intervene in the mechanical decision-making process.

To neutralize the negative incentives, small nudges, or re-arranged choice architecture, tend to work better than descriptive knowledge.

Choice architecture is a term borrowed term from behavioral economics. Dr. Richard Thaler has received the 2017 Nobel Memorial Prize in Economic Sciences for research on pitfalls of economic decision making. He describes the concept of nudging people through subtle changes in government policy to do things that are in their self-interest.

Finally, Joseph Henrich’s recommendation to design a ‘variation and selection system’ allowing alternative institutions and organizational forms to compete, can empower managers to discover the best solution for their program. As change agents, they can then decide in which form the recommendation is best implemented. In the knowledge management context, that would consist of preparing the choice architecture and tools the managers need to make the right decisions rather than simply listing the evaluation recommendations.

Bibliography

AfDB IDEV, 2015. Independent Development Evaluation Work Program 2016–2018. [Online]
Available at: http://idev.afdb.org/sites/default/files/documents/files/IDEV%20Work%20Program%202016_2018%20EN%20Web%20final.pdf
[Accessed 22 September 2017].

Exley, C. L. & Kessler, J. B., 2017. Better is the enemy of good. [Online]
Available at: Available at SSRN: https://ssrn.com/abstract=3029231
[Accessed 22 September 2017].

Henrich, J., 2016. The secret to our success. Princeton, NJ: Princeton University Press.

Lang, V. F. & Presbitero, A. F., 2017. Room for discretion? Biased decision-making in international financial institutions, s.l.: Elsevier.

Project Management Institute, 2008. A guide to the Project Management Body of Knowledge. Pennsylvania: PMI.

Marje Aksli is an independent evaluation consultant based in Washington, DC, focusing on international development and security programming. She has over 15 years of experience in international relations and programming in Canada and overseas. Her approach to programming is influenced by her training in Project Management (Algonquin College), and MA studies in Globalization and International Development (University of Ottawa). Her research interests include foreign aid effectiveness and the impact of global governance on local policy-making. She has formerly edited The Rising Africa magazine by the Canadian Council on Africa.

Infusing Project Management techniques into a Results-based Management approach leads to better accountability

Infusing Project Management techniques into a Results-based Management approach leads to better accountability

Marje Aksli 

Two Canadian projects, G-8 Summit in 2002 and resettlement of Syrian Refugees in 2016, exemplify how international development or public policy objectives can be met more effectively and efficiently when a results-based-management (RBM) approach is supported by using project management (PM) tools.  

In short, could government programs, currently running on RBM, be made more effective and efficient if they were managed using PM principles, as well?

The Development Assistance Committee at the Organization for Economic Cooperation and Development (OECD DAC) has outlined five criteria for evaluating projects and programs: relevance, effectiveness, efficiency, sustainability, and impact. Setting the latter two aside, for now, we discuss the multiple sets of tools to adequately meet the relevance, effectiveness and efficiency criteria. 

RBM and PM 

The most widely used management tool in international development, and in public service, is Results-Based-Management, or RBM, which connects the project outputs to desired outcomes in a measurable way. For example, we assume that if a country accepts and resettles tens of thousands of refugees from the Middle East - a project output in the Operation Syrian Refugees - suffering is alleviated, which is the desired outcome. Similarly, coordinating G-8 Summits – project output – would, in theory, lead to an increased global visibility and influence for Canada – the desired outcome. Project relevance is thus established with the logical and causal connection from outputs to outcomes through RBM. 

Setting relevance aside, the managers also need to ensure projects are implemented effectively and efficiently as per OECD requirements. This entails achieving project outputs effectively (within the project scope) with scarce inputs or tax dollars (efficiently). Though RBM is focused on achieving higher level results (outcomes), it provides no tools to manage project outputs in a cost-effective and timely manner. In effect, RBM is not designed to provide the guidance required for achieving effectiveness and financial efficiency – two of the main DAC evaluation criteria.

The systemic managerial approach, specified in the Project Management Body of Knowledge (PMBoK) does provide a logical normative system for managing projects effectively and efficiently. Several lessons learned sections of evaluation reports indicate that using PM tools would help to control project scope, which relates to effectiveness, and cost, which explains the efficiency of a project. PMBoK principles could also assist with stakeholder management, a crucial aspect in a project’s success. Still, the public sector and organizations involved in program development seem to overlook these obvious benefits because they have yet to practice PM tools to the same extent as RBM and deliverology.

For better effectiveness and efficiency in tax payer funded programming, results-based management and project management tools should be used in combination because they are complementary by design. RBM ensures that project outputs are connected to project outcomes through logical and causal connections (relevance) and PM tools ensure the management of projects is focused on scope and budget (effectiveness and efficiency).  

Public servants and NGO programming personnel would benefit from extensive training in PM techniques and they should be encouraged to use them. A change in management styles through PM training would also provide a more business-oriented approach to development or to public sector programming, one that mitigates risks through a judicious use of resources. 

By breaking down project components into a 2D Matrix relating knowledge areas, such as project cost, essential human resources, project duration, scope, risk, stakeholders, and quality, with project phases or life cycles, PM tools lay out a set of manageable tasks in a coherent process. In the project initiation phase, the PM tools instruct managers to work on two knowledge areas only: to specify a project goal and to identify stakeholders based on an assessment of needs. In the proceeding planning phase, managers have to engage with many additional project knowledge areas. The project description is explicitly stated in the planning phase and it becomes the project scope. The work needed to achieve this scope, and only this scope, is then broken down into manageable work packages or activities to which cost is assigned, human resources allocated, duration stipulated; risk level mitigated and quality identified. Only then can the manager know the total cost, the duration, and risks of the project. Similarly, PM tools offer guidance for three consequent phases: execution, monitoring and controlling, and for the closing phases. 

Lessons to be learned 

Comparing project closure or evaluation reports to the PMBoK norms reveals several short comings. The biggest deviations between PM theory (the norm) and management practice in Canadian government departments result from omitting, or shortening, the initiation phase and from blending the planning phase with the execution phase. This leads to confusion in many knowledge areas and severely impacts both project scope and cost management. The final report of Kananaskis Summit from 2002, for example, did not distinguish between words such as ‘planning’ and ‘managing’. The two words were used interchangeably in the meaning ‘organizing’, which seems to indicate that the implementation team did not understand how to sequence or prioritize activities in distinct project phases. This became acutely apparent in the time and scope management areas. Additions to the project scope expanded until late in the project execution phase, which led to ballooning budget and unmet commitments. 

The merged planning and execution phases resulted in several planning documents not being completed, which consequently hindered the controlling of project scope and led to difficulties in estimating the exact costs and necessary human resources. For example, the report regrets that environmental impacts were not addressed at the Summit and suggests, correctly, that these requirements should have been included with the scope requirements in the planning phase. Promises were made about the, so called, legacy projects to First Nations – the most important stakeholders, and event co-hosts, yet these add-on projects were not included in the original project scope and were therefore not part of the plan and were consequently not delivered because of insufficient funds. 

Similarly, the Operation Syrian Refugees (OSR) in 2016, which aimed to resettle 25.000 refugees between November 2015 and February 2016, suffered under the lack of interdepartmental coordination during the initiation and planning phases. This led to significant consultation gaps between the government departments, resulting in unclear responsibilities and competing priorities between different government departments. For example, the planning model used by the lead department IRCC did not sufficiently address the significant efforts required to set up operations abroad during the operation’s earlier phases. The activities abroad were not well planned, consequently not assigned costs, durations, human resources, or risk ratings for and were thus managed poorly. Managers cannot execute activities, which are not planned. The recently published IRCC Rapid Impact Evaluation Report of the Syrian Refugee Initiative adds that there was also a lack of sufficient planning and forward-looking preparations regarding the resettlement and early integration phases in the first few weeks post arrival in Canada. The report admits that ‘the expedited nature of the initiative made it challenging for the settlement sector to effectively plan for the delivery of in-Canada services’, and that ‘the quick pace was difficult on refugees’. 

In short – both projects suffered under the lack of proper planning. 

But the largest disparity between project management norms and actual practice relates to efficiency, i.e. how the total cost of projects is estimated in the Canadian public service. Both, the OSR and the G-8 Summit reports reveal that cost planning is conducted through “guess-estimations” by a budget office as opposed to rigorous activity-based costing by a project team. Budgets are often division-based instead of activity-based. Projects with budgets estimated in this manner cannot be as efficient as those projects which total cost has been assessed based on individual activities. 

Although the cited managerial aim in organizing a G-8 Summit in 2002 was to ‘deliver the best possible value from the available mix of resources’, the project closure report was not able to critically assess whether the over 100M$ for a 2-day event is indeed an efficient way to advance Canada’s policy goals in G-8. Canadian experience of hosting four G-8 meetings previously would offer enough material to draft a blueprint Work Breakdown Structure for the next occasion, which would allow controlling scope, durations, costs and necessary human resources. As the closure report revealed, the lack of proper planning indeed led to inadequately composed project scope, late scope additions, overlapping and conflicting requirements, which were poorly communicated to the main stakeholders. One project goal - to make ‘Canadians proud’ – was not measurable or actionable. 

These examples demonstrate how faulty project planning makes it near impossible to assess whether the project was indeed a success. Despite that, the Summit was considered ‘unprecedently successful’ based on only one concern of cohosts – to be ‘conflict-free’. 

In the case of OSR, the Government of Canada’s aim to resettle 25,000 Syrian refugees between November 2015 and February 2016, was a political commitment made during the election campaign in response to the ongoing humanitarian crisis facing the Middle East and Europe. It became a major issue during the federal election season in 2015 after it was revealed that the family of a drowned Syrian toddler had hoped to come to Canada. All parties presented their competing visions several months before the election about how they would respond to the crisis. The Liberal leader Justin Trudeau presented the most ambitious plan in terms of volumes and timelines, saying that a Liberal government would bring in up to 25,000 refugees from Syria by January 2016, as CBC reported. His vision did not specify the expected cost, government’s capacity to deliver on the commitment or the experience of impacted refugees. The management of this project was thus not thought out and no clear criteria were provided for estimating project’s success rate. 

Accordingly, the IRCC Rapid Impact Evaluation report[1] on OSR concluded that one major area which should be taken into account to help ensure successful resettlement and settlement results includes “the need for end-to-end planning”. Yet, despite these shortcomings, the lessons learned portion of the report declares that “the whole of government initiative was a great success in many regards”. 

Conclusion 

As illustrated above in the two distinct projects, undertaken a decade apart, by following PM managerial logic in initiating, planning, executing and monitoring these unique and temporary endeavors with a goal to achieve a specific deliverable would have avoided several problems with effectiveness and efficiency. Both projects suffered under poorly stated and communicated project scopes, under scope creep through add-ons, which led to poor effectiveness, and poorly managed costs (inefficiency), and partially unsatisfied stakeholders. 

The Kananaskis report reveals that the main stakeholders the G-8 Summit who the project intended to make proud were cooperating reluctantly because the stakeholder identification and outreach were not conducted properly in the initiation and planning phase, respectively. OSR, which was meant to alleviate refugees’ fate was suffering under unrealistic timelines and overwhelming scope and led to at times frustrated refugees. 

The PMBoK offers a systemic approach to managing endeavors that are temporary, unique and have a clear beginning and an end, including measurable guidelines for creating progress reports and closing all tasks. PM tools can thus offer increased efficiency and effectiveness in delivering outputs. 

-xxx-