What would make corporate evaluations more effective?

(Forthcoming in AfDB’s magazine ‘Evaluation matters’, Q4 2017)

This article makes a case for evaluators and managers to adopt a common management language in order to increase the likelihood that the evaluation recommendations will inform the design of new programs.

Relying on the concepts of social learning, I argue that adopting project management thinking will not only provide a powerful counter-factual analysis tool for program evaluators but also increase their perceived value to managers. Additionally, corporate evaluations are more likely to add value to the organization if they explicitly assess the costs of recommended reforms and the corporate choice architecture, thereby eliminating the misconceived ideas which inhibit change.

Although organizational assessment lies slightly outside the scope of traditional evaluation questions (i.e. relevance, effectiveness, efficiency, impact, and sustainability), evaluators are nevertheless well placed to provide feedback about corporate bottlenecks. Providing insights about the implementing organization, evaluations serve an important accountability role. But the question is: do the independent evaluative findings on corporation practices add value for learning purposes? And what ensures that the reforms do actually take place?

We know from experience that in international development, providing facts about the benefits of a certain action is insufficient for the action to be implemented. In sanitation projects, for example, explaining the benefits of hand washing does not necessarily result in better hygiene.

People are selective social learners, claims Joseph Henrich in his book ‘The secret of our success’. They do not change their habits purely based on “the facts” or “education” (Henrich, 2016). People, he explains, copy the behavior of influential people in their social circles. They are more likely to learn from those whose prestige, success, sex, dialect, and ethnicity they consider important.

The same phenomenon might explain why evaluative recommendations have not entered into the planning phase of new programming or informed organizational redesign to the desired degree. So far, the evaluators’ focus on providing evidence for accountability purposes has not been sufficiently conducive to learning and reform. So, what would make evaluation recommendations more potent, leading to an added value for the organization?

Experimenting with charitable giving, researchers Christine L. Exley and Judd B. Kessler conclude that contrary to our expectations, people may see the potential for improvement as a reason to avoid the action. In their working paper ‘The Better is the Enemy of the Good’ (Exley & Kessler, 2017), they explore a “behavioral phenomenon that may contribute to why information fails to encourage desired behaviors”. They conclude that ‘the better’ is seen as ‘the enemy of the good’, particularly when “agents have a self-serving motive to let it be”. The change agents are inclined to deem the actions costly, which explains why information which encourages actions with future benefits may not be taken into consideration.

“In standard economic theory, information helps agents optimize. But providing agents with information about the benefits of taking an action often fails to encourage that action”. Exley & Kessler, 2017 How can we ensure the content of evaluation recommendations is relevant for managers and is framed properly to encourage action? And further, how can we increase the status and influence of evaluators? In my view, the knowledge management team, evaluators and the operational managers should inch closer to each other and learn to speak a common language.

That would help shake the impression that evaluators are merely aloof critics trained in theoretical economics and social science who have no clear stake in the game.

The process of corporate evaluations, or organizational assessments, should be partially collaborative instead of conducted on the strict independence condition, ie. executed in close dialogue with the departments under evaluation.

I believe that management knowledge or experience makes a more effective evaluator. Revolving more closely around the receiving managers’ core expertise increases the chances of added value to the organization. Common management language can more likely spur reform. But it might mean that evaluators compromise their independence to a certain degree.

Case for digging deeper into projects

Having assessed a wide variety (but admittedly, a small number) of undertakings – projects, programs, emergency evacuation and a global strategy – during my three-year evaluation experience in Canada, two phenomena seem to explain the low effectiveness of donor-funded programming. One has to do with local ownership but the second explanation as to why the intended outcomes have not been fully achieved within the budgetary or time constraints is related to the capacity to manage projects - the main vehicles of international development - of the implementing organization. Namely, their low project management skills.

It is counter-intuitive to dig deeper into the project level when we talk about the organizational assessments or corporate evaluations; However, the evaluators’ unit of analysis on the performance of programs, is nevertheless individual projects.

Projects provide feedback for both managers and evaluators about the program’s or portfolio’s performance. After all, programs are simply groups of synergistic projects, which are managed together based on organizational strategies and priorities. They are grouped together into a program as their expected results are similar enough so that they can be aggregated into a higher-level outcome. Managing similar projects jointly is more efficient.

But what are projects, anyway?

And what can project management principles teach evaluators? Project Management Institute has defined projects as temporary endeavors with a clear beginning and end, which result in a unique product, service or an output (Project Management Institute, 2008). It thus follows from the definition that building a school, organizing a world summit, building a water purification system or creating a new educational network can be planned and executed with similar management tools.

The holy book of project management - Project Management Body of Knowledge or PMBoK – adds that due to the uniqueness of each project, there can be many uncertainties about it and the risks can be high.

Project tasks are most likely new to the team, which necessitates a more rigorous planning process than for routine work. Specific training may often be required. In order to meet the project requirements, PMBoK states, the managers need to apply knowledge, skills, tools, and techniques to project activities during the five distinct phases (or life cycles): initiating, planning, executing, monitoring and controlling and closing.

Managing projects also involves knowing how to control requirements, address stakeholder needs, and balance the scope, quality, schedule, budget, resources, and risk. As these eight knowledge areas apply unevenly across the five project phases, PMBoK has charted them in a 2D matrix in order to assist managers to be more effective and economical in their work. It is crucial to keep an eye on the knowledge areas, as they can be subject to risks such as unexpected events which might negatively influence the project.

Managing projects is a complicated undertaking, requiring constant juggling of distinct stakeholders’ needs within the constraints of scope, time, budget and quality. The trade-off between the constraints means that if the scope of the project is changed it has implications for the budget and schedule. Changing quality requirements can also have implications for cost and duration, as low-quality products tend to be cheaper. Scope add-ons increase the budget and can ultimately postpone the completion date. Similarly, if the project is accelerated (time is crunched), it increases the budget and the risks to completion. If the budget increase is not possible, the scope of the project needs to be reviewed, and perhaps cut back. Different stakeholders might have conflicting views on which element of the project can be cut, and the quality of the end product might not meet the expectations of any stakeholder.

Organizational design has the biggest impact on program effectiveness and efficiency

Assessing program effectiveness requires that evaluators compare the planned project objectives with the ones achieved. The efficiency assessment means that evaluators would compare the resources applied and the results achieved. These necessary tasks assume that projects and programs are first planned and then executed. Yet, some evaluation reports suggest that this is not necessarily the case.

The undertakings I have evaluated reveal that often, the initiation phase and its associated processes have been omitted altogether, whereas the planning phase has often been blended with the execution phase. Lack of planning leads to confusion with project scope (what is the intended outcome?) and severely impacts both project cost and time management, i.e. effectiveness and economic efficiency. Low capability to plan, execute and monitor projects is directly tied to the organizational capacity.

Project management knowledge allows evaluators to chart powerful counter-factual analysis about how the project could have been managed more efficiently and effectively.

Understanding the effort needed to plan and manage projects, the main vehicles of development interventions, thus becomes relevant for the evaluators in order to avoid tradeoffs between efficiency and effectiveness. Corporate evaluations need to assess whether the managers are sufficiently trained to manage projects effectively (so that projects would achieve all the elements in the scope) and efficiently (so that the planned activities were budgeted individually and executed within the given time frame).

Project Management Processes (PMBoK 2013, p. 60)

Lost in Translation

Using and referring to managerial vocabulary in both evaluations and in communicating evaluation findings back to the managers (about the capacity to plan and manage projects) can create a common understanding for executives and evaluators.

IDEV has already dramatically stepped up its knowledge management, dissemination and outreach activities (AfDB IDEV, 2015) ensuring the high quality of evaluations, communications, engagement and follow-up with managers. Sharing a common language, signaling that evaluators have skin in the game, and demonstrating relevance to the corporate mandate, all assist with the learning process.

But the organizational assessments could also include understanding the (perceived) cost of recommended change to managers and the incentives which explain the inaction. In International Financial Institutions, both political interests and bureaucratic incentives can influence the decision to intervene in the mechanical decision-making process.

For example, Lang and Presbitero reviewed the degree of discretion embedded in the World Bank-IMF Debt Sustainability Framework to understand the decision-making process of international financial institutions. After synthesizing debt analyses conducted between 2006 and 2015 for low-income countries, the authors demonstrated that both political interests of beneficiary countries near elections and bureaucratic incentives influence the decision to intervene in the mechanical decision-making process.

To neutralize the negative incentives, small nudges, or re-arranged choice architecture, tend to work better than descriptive knowledge.

Choice architecture is a term borrowed term from behavioral economics. Dr. Richard Thaler has received the 2017 Nobel Memorial Prize in Economic Sciences for research on pitfalls of economic decision making. He describes the concept of nudging people through subtle changes in government policy to do things that are in their self-interest.

Finally, Joseph Henrich’s recommendation to design a ‘variation and selection system’ allowing alternative institutions and organizational forms to compete, can empower managers to discover the best solution for their program. As change agents, they can then decide in which form the recommendation is best implemented. In the knowledge management context, that would consist of preparing the choice architecture and tools the managers need to make the right decisions rather than simply listing the evaluation recommendations.

Bibliography

AfDB IDEV, 2015. Independent Development Evaluation Work Program 2016–2018. [Online]
Available at: http://idev.afdb.org/sites/default/files/documents/files/IDEV%20Work%20Program%202016_2018%20EN%20Web%20final.pdf
[Accessed 22 September 2017].

Exley, C. L. & Kessler, J. B., 2017. Better is the enemy of good. [Online]
Available at: Available at SSRN: https://ssrn.com/abstract=3029231
[Accessed 22 September 2017].

Henrich, J., 2016. The secret to our success. Princeton, NJ: Princeton University Press.

Lang, V. F. & Presbitero, A. F., 2017. Room for discretion? Biased decision-making in international financial institutions, s.l.: Elsevier.

Project Management Institute, 2008. A guide to the Project Management Body of Knowledge. Pennsylvania: PMI.

Marje Aksli is an independent evaluation consultant based in Washington, DC, focusing on international development and security programming. She has over 15 years of experience in international relations and programming in Canada and overseas. Her approach to programming is influenced by her training in Project Management (Algonquin College), and MA studies in Globalization and International Development (University of Ottawa). Her research interests include foreign aid effectiveness and the impact of global governance on local policy-making. She has formerly edited The Rising Africa magazine by the Canadian Council on Africa.

Ka riik saab olla ettevõtlik ja luua innovatsiooni

Kui USA riigiasutused ei oleks innovatsioonilisi ideid rahastanud, siis poleks meil täna nutitelefoni, kirjutab rahvusvahelise arengu konsultant Marje Aksli.

Miks on suured ja mõjukad firmad nagu Facebook ja Google tekkinud just Ameerikas? Mis teeb ameeriklased palju innovaatilisemaks kui Euroopa? Kas äkki pakutakse USAs start-up’idele mingeid erapankade finantstooted, mis pakuvad kõrge riskiga ettevõtetele laenusüste? Või arenevadki sealsed gasellid pelgalt selle tõttu, et maksukoormus on väiksem?

Hiljuti maineka Leontiefi auhinna saanud professor Mariana Mazzucato on just seda küsimust uurinud. Ta kirjutas raamatu “Ettevõtlik riik” just selleks, et Eestiski levinud puudulikku narratiivi lollist ja laisast riigist ning nutikast erasektorist ümber lükata. Eesti ettevõtjad, kes õhukest riiki ihalevad, peaks küsima, kes või mis rahastas tõeliselt innovaatiliste tehnoloogiate väljatöötamist USAs. Mazzucato selgitaks neile, et vidinad, mis meie nutitelefonid tegelikult nutikaks teevad – ehk siis internet, mobiilside, GPS, mikrokiibid, Siri ja puutetundlik ekraan – on tegelikult kõik riigi rahastatud.

Kui USA valitsusasutused ei oleks andnud innovatsiooni toetamiseks jaoks raha, siis ei oleks meil täna nutitelefoni. Seega, riik ei olnud mitte pelgalt passiivne äri võimaldaja, vaid aktiivne osaline teadusavastuste muutmisel kommertslikult tasuvaks, ning tihti ka strateegiliste innovatsiooniharude looja.

USA firmasid on oluliselt aidanud valitsuse rahastatud programmid väikeettevõtetele. Nii näiteks aitab Small Business Innovation Research (ehk SBIR) programm teatud väikeseid ettevõtteid teha kallist uurimus- ja arendustööd (R&D). Teine riigiprogramm, mis suures osas Ameerika väledaid gaselle aitab, on Small Business Technology Transfer programm (ehk STTR), mis koordineerib föderaalse innovatsiooni uurimus- ja arendustööd koos erasektoriga. See programm hõlmab erasektori kui avaliku sektori koostööd (PPP partnerlusi), ja õhutab koostööd väikefirmadele mõeldud investeerimiserakapitali ja mittetulunduslike uurimisasutuste vahel. See mudel nõuab, et väikefirma töötaks ametlikult koos uurimisasutusega, et aidata teaduslikke avastusi muuta turukõlbulikuks innovatsiooniks.

Kuna riskikapital otsib väga lühikese tagasitootlusmääraga investeerimisvõimalusi, ei ole nende lühike tähelepanuvõime suurte innovaatiliste läbimurrete jaoks piisav. Riigi eelis erasektori ees innovaatiliste leiutiste toetamisel ongi see, et riigil on palju pikem riskitaluvus: 15-20 aastat. Riigiprogrammid on ulatanud oma abistava käe just siis, kui erakapital ei julge veel investeerida. Neile olid kas riskid liiga suured või ei lubanud investeering kasumit vaid paari kiire aasta jooksul.

Näiteks nanotehnoloogia tõestab, et tihti tuleb just riigilt strateegiline visioon, kus suunas areneda. Sama kehtib ravimite kohta – tervelt 75 protsenti uutest innovaatilistest ravimitest on valitsuse rahade eest leiutatud.

Kui Eesti ärimehed seavad meile USAd innovatsiooni toetamisel eeskujuks, siis rääkigu nad ka ausalt sellest, kuidas sealsed teaduslahendused rahaliselt edukaks innovatsiooniks said. Hoopis riik on käitunud riskialtilt, panustades maksumaksjate raha riskantsetesse investeeringutesse, kus puudub selgus selle võimalikus kommertsväärtuses. See on aga hoopis vastupidine jutt kui see, mida Eestis laisast riigiametist ja hulljulgetest erainvesteerijatest räägitakse.

Praegu on USAst kui õhukese riigi eeskujust üldse kohatu rääkida. Tõi ju soss-sepast ärimehe Donald Trumpi võimule just see protestifenomen, et peaasjalikult innovatsiooni tõttu ja automatiseerimise käigus töö kaotanud inimesed ei leidnud endale enam majanduslikku rolli. Nende sissetulek kahanes, head töökohad haihtusid ja meeleheide oligi platsis. Neile oleks hädasti mõnda riigi pakutavat ümberõppe- või mujale kolimise programmi vaja läinud. Sama kehtib Eestis – ettevõtlikku riiki on vaja rohkem, mitte vähem.

Paljud Eesti ettevõtjad kiruvad süsteemi, mis võtab „inimestelt raha ära ja siis pakub neile tasuta teenustena tagasi“ osutades, et „vahepeal on veel „administreerimiskuludena“ suur osa sellest rahast juba ära kasutatud või tuuri pandud“. Kas on ikka nii, et inimesed ei vaja neid nn tasuta asju? Kas Eestis on tõesti liiga palju haridust ja tervishoidu, kõbusaid vanainimesi ja hästi toidetud lapsukesi? Ka valitsuse rahastatatud innovatsiooniprogrammid kasutasid maksumaksja raha, ning evisid administeerimiskulusid.

Eesti peab aru saama, et ilma kõrge kvaliteediga inimkapitalita ei saa innovatsiooni luua, mis ongi üks majanduse mootoreid. Just need tasuta teenused ongi kindlustanud selle, et väike ja USAst viis korda vaesem Eesti on suutnud USAga võrreldes pakkuda sarnase tasemega sotsiaalset heaolu. Eesti ebavõrdsuse indeks ehk Gini koefitsient annab aga aimu, et riigi rahastatavaid programme peaks veelgi tõhusamaks muutma, et heaolu laotuks endiselt ühtlasemalt Eesti elanike vahel, nii maal kui linnas. Innovatsiooni varjukülg ongi kahjuks sissetulekute kui varanduslik ebavõrdsuse kasv.

Oleks tore, kui Eesti riik panustaks USA eeskujul samuti idufirmadesse ja saaks paarikümne aasta pärast idufirmadesse tehtud kulutused kopsaka protsendiga tagasi. Just selleks, et globaliseerumise käigus kaotajaks jäänud täiendhariduseta kaevureid ja endisi tehasetöölisi 21. sajandisse aidata. Räägitakse USAs ju tõsimeeli sellest, et Facebook, kes meie tasuta saadud andmeid silmagi pilgutamata sularahaks vahetab, võiks meile kõigile palka maksta.

Canada’s feminist international assistance policy is not the most effective way to reduce poverty

In the early 2000s, a director of a Moldovan women’s nongovernmental organization (NGO) told me that she would never publicly associate herself with feminism. I found this statement quite curious as I had just asked her to submit an article on gender issues for the Moldovan family planning association’s website. My goal of promoting gender equality and reproductive health awareness with local voices was clearly a feminist one, wasn’t it? Not really, she explained. If she were to declare publicly that she’s a feminist, her colleagues and friends would stop shaking her hand, she told me. She would lose all credibility, and she needed credibility to run an NGO dealing with gender-based violence.

She then referred to an obscure instance of Western feminists publicly berating Mikhail Gorbachev, the last president of the Soviet Union, as a reason why feminism had lost its appeal for her. They were not respectful, she argued. This scandalous event might not even have been real, but the story was influential nonetheless. It led this local leader in women’s issues to say that not only is she not a feminist but she would be ashamed of being considered one.

Fifteen years later, Canada has adopted a feminist international assistance policy to advance gender equality and the empowerment of women and girls as “the most effective way to reduce poverty and build a more inclusive, peaceful and prosperous world.”

Canada means well. The fact that women often bear the costs in development stands out: women and girls are more likely to be among the poor, the least educated and the ones experiencing violence. There clearly is a need to help women and girls.

Effective aid is locally owned

However, the first principle of effective foreign aid is that development needs to be locally owned. The Paris Declaration on Aid Effectiveness, adopted in 2005, clearly states that developing countries are supposed to set their own strategies for poverty reduction, and the donor countries are expected to align behind these objectives and to use local systems. It’s difficult to imagine that the countries with the gravest conditions for women will “own” feminism, a Western concept.

Evaluators assessing foreign aid programming for relevance, effectiveness, efficiency, impact, and sustainability – the criteria set by the Development Assistance Committee of the Organisation for Economic Co-operation and Development, know that the projects that are implemented with low levels of local ownership are less likely to be sustainable after the donor funding is withdrawn. Feminism has to be relevant to Canada’s development partners, and it would need to be reflected in their governments’ poverty reduction strategies. Additionally, achieving outcomes that are in serious disagreement with local traditions ends up being less efficient economically than running projects that enjoy strong local ownership. Potentially low relevance, high inefficiency, and low long-term sustainability mean that development interventions cannot be considered effective.

The second set of problems related to feminist development initiatives is associated with impact and attribution. In the near future, can Canada truly credibly claim that any success in the partner countries was due to our feminist development interventions, and not caused by other favourable trends beyond Canada’s control?

The truth is, social norms guiding the behaviour of men and women have evolved across centuries, following a local logic, and they relate to people’s innate mindset. They are “sticky”: outsiders’ efforts to change ways of life by rearranging power relationships are doomed to fail if they do not follow the local logic. Tinkering with gender relations is not viewed universally as bringing benefits to everyone. Some high-powered men might perceive Canada’s well-meaning efforts as threatening, and they are not likely to support feminist policies that come from abroad. And yet they are also stakeholders in development, interested in maintaining or increasing their prestige or status in their respective societies.

Sticky cultures

In his book The Secret of Our Success, Joseph Henrich of Harvard University writes that when politicians design new policies, they inevitably import their own assumptions about human nature, often rooted in Enlightenment philosophies. He points out that as we have seen from the sad experience of transplanting Western institutions to countries that have different incentives and standards for judging and punishing people, the new norms need to fit in with people’s social norms, informal institutions and cultural psychology.

People are selective social learners, Henrich warns, which means that they do not change their habits purely based on “the facts” or “education”; rather, they copy the behaviour of influential people in their social circles. People are more likely to learn from those whose prestige, success, sex, dialect and ethnicity they consider important, especially on issues related to food, sex, danger and norm violations. It seems as if the best way to alter gender relations is to work with influential local men, allowing them to lead by example. It’s true that Canada has increased its profile internationally, but its high prestige is probably not strong enough to contribute to change in distinctly different societies. 

Cultures are long-lasting; they have evolved over centuries and they change slowly. That is especially true when influential religious organizations work to maintain the status quo in relations between the sexes and have strong views about sexual minorities. Change in social policies depends on the entire society. Or, as Henrich puts it, change depends on the expansion of the collective brain, which in turn depends on openness to discuss issues and ideas. Instead of artificially transplanting copies of Western institutions to other countries, Henrich recommends designing a “variation and selection system” allowing alternative institutions and organizational forms to compete. Locals then decide which form wins and will be adopted.

Win-win for all?

Canada’s new feminist international assistance policy argues that the feminist approach to development “leads to better development results and benefits everyone, including men and boys.” How this claim works out in practice needs to be substantiated in the program monitoring phase and in evaluations. When men perceive that their traditional social status is threatened, a backlash may follow. Evaluators, therefore, need to be attuned to the unintended consequences.

Another risk of the explicitly feminist approach is that it may ignore cases where men, especially gay men, and boys are disadvantaged and vulnerable. Canadian feminist international development policy sees men and boys predominantly as a target group that should be lectured, educated and engaged but not as direct beneficiaries of assistance.

At yet, as Sean Stevens of New York University points out in an article on gender differences, boys seem to be hurt more than girls by living in deprived neighbourhoods. The cycle of poverty seems to be more persistent for the poorest boys than for girls. Even in Canada, as the Canadian economist Miles Corak found, boys born into the poorest families (the bottom 5 percent) were less likely to escape poverty than girls born into the same conditions.

An article by Diane Halpern, professor emerita at Claremont McKenna College and her coauthors discussing sex differences in science and mathematics adds that male development is more sensitive to environmental conditions, which can explain why there is more variability among males in their academic performance. It seems that in the most deprived conditions, it is boys who would benefit from additional help.

That is why a successful development policy would be an inclusive one that would engage men, women, boys, and girls equally depending on their needs and within the social systems where they live. A development policy designed around Paris Declaration principles, evaluative recommendations on program effectiveness and – most importantly - equal opportunities for all would be more effective than an explicitly feminist one.

-xxx-

Marje Aksli is an independent evaluation consultant based in Washington, DC, focusing on international development and security programming. Her thematic interests include the impact of global governance on policy-making.