BS/SDG Index: Can progress on sustainable development be reduced to a single number?

Denmark, Sweden and Finland are the top ranking countries in terms of sustainable development, while Niger, Chad and the Central African Republic are the worse performers, according to the Sustainable Development Report 2019. The rankings in the report are surprisingly correlated to the outcomes of the Human Development Index.

The Sustainable Development Report 2019 was recently launched by the Bertelsmann Foundation of Germany and the Sustainable Development Solutions Network (SDNS), based in New York and Paris.* It includes 17 “dashboards” with indicators selected by the authors for each of the 17 Sustainable Development Goals, and a Global SDG Index that summarizes them in a single number and allows for the ranking of the 162 countries for which enough data are available.

The numbers of the BS/SDG Index are surprisingly similar to those of the Human Development Index (HDI) that UNDP publishes every year (see graph below) with a correlation coefficient of 0.91 (where 1.0 indicates perfect correlation and 0.0 means no relation at all). In other words, if you know a country’s HDI value, you can forecast its BS/SDG ranking with 91 percent precision. Thus, the new index only adds minor details to what we already know.

BS/SDG index and HDI are highly correlated

Source: Graph by the author with data from the BS/SDG Index and UNDP’s HDI.

This high correlation was not to be expected if we remember that the HDI reflects only three factors (income, health and education), while the BS/SDG index intends to reflect all of the goals, from the well-being and gender dimensions of poverty, hunger, health and education (Goals 1 to 5) to environment (SDGs 12 to 15), governance (SDG 16) and implementation (SDG 17).

The 2030 Agenda that frames the SDGs calls itself “transformational” in its very title and has been hailed as a “paradigm change”. Shouldn’t that be reflected in an index that claims to measure progress towards achieving it?

Development used to be conceived as a synonym of economic growth and the OECD still “graduates” countries out of their condition of potential recipients of ODA based on the World Bank threshold between high middle income and high income countries.

In the nineties, the HDI nuanced per capita GDP by adding well-being considerations and showing that countries with similar income levels could have very different “human development” results. Now “sustainable development” further expands the concept, introducing inequalities, governance and the environment into the picture.

For example, one of the new indicators incorporated in the SDGs and not previously considered in development indexes is the number of homicides per 100,000 deaths. The ranking of countries in that indicator does not correlate at all with per capita income or HDI and among the ten best ‘performers’ we find some of the richest and some of the poorest countries in the world. Among the “bottom ten”, the countries with the highest proportion of homicides, mostly middle income countries are found, with extreme inequalities being the only obvious common denominator.

Source: UNODC United Nations Office on Drugs and Crime

Yet, even when they perform better than the UK or Switzerland in this indicator, Burkina Faso and Indonesia end up ranked 141 and 102 respectively, out of 162 countries in the BS/SDG Index. This is due to their non-violence being averaged away, within SDG 16, by other indicators such as property rights (as assessed by the World Economic Forum) or press freedom (as evaluated by Reporters Sans Frontières).

The Index is built, precisely, by averaging first all the indicators for each SDG and then averaging those averages, giving the same value to each indicator within a goal and to each of the 17 SDGs in the total. This method seems logical in the case of health (SDG 3), where 13 basic health-related indicators, ranging from maternal mortality to the percentage of smokers in the population are computed, as ultimately all of them relate to health policies or service delivery. It is less obvious what the meaning of the cocktail is in the case of Goal 16, which averages the three indicators mentioned (homicides, press freedom and property rights) with the corruption perception index of Transparency International, child labour (measured by UNICEF), arms exports (reported by the Stockholm Peace Research Institute) and the Gallup poll about how safe people feel when walking alone at night.

The selection of which indicators to include or not is conditioned, obviously by the availability of data, but it is also an arbitrary choice of the authors. Thus, for example, three poverty indicators form the index for SDG1 (population under the US$1,90 and US$3.20 poverty lines and, for OECD countries, 50% below the median income). But the coverage of social security is not in the list, even when it is an explicit target of SDG1 and abundant data are provided by the ILO.

As independent research institutions, Bertelsmann and SDSN are free to make any choices they want. The BS/SDG Index and Report are not official UN documents, but some confusion is unavoidable when the SDSN, one of the two institutional authors, calls itself “a global initiative for the UN” (emphasis added) and claims to operate “under the auspices of the UN Secretary-General” and adds the UN acronym to its name on its website: www.unsdsn.org.

But the way in which the BS/SDGs Index accounts for the goals only partially reflect the official 2030 Agenda.

For SDG 10, for example, only one indicator for domestic inequalities is used, the Gini index of income for each country, ignoring that this goal requires to “reduce inequalities within and among countries” (emphasis added).

In the case of climate change (SDG 13), the CO2 emissions indicator is supplemented with an indicator on imported CO2 emissions embedded in traded goods (carbon footprint), but fossil fuel exporters are penalized with the carbon equivalent of their exports, thus double counting the emissions (in the country of production and in the country of consumption) and under counting the damage produced by countries that consume their own fossil fuels. Further, all these indicators are expressed on a per capita basis, and as a result the US, which is the largest fossil fuel producer of the world is listed with per capìta exports of less than one tonne per year, while Ecuador, who is a marginal producer, exports four times more fossil fuels per capita due to its small population and very low local consumption.

To make matters worse, the climate indicators cocktail also includes as an indicator the number of people affected by climate-related disasters. The Philippines, which has low emissions and moderate fossil fuel exports gets an orange colour average on climate traffic light ranking because of the millions of victims of climate-related disasters… caused by the emissions of other countries.

The statistical audit of the SDG report by the Joint Research Centre (JRC), the European Commission’s science and knowledge service, finds out that “some countries that have poor performance on SDG12 (on sustainable production and consumption patters) and SDG13 (on climate) have good performance on all the other goals and vice-versa. (…) The top five countries in the index are ranked among the bottom positions of SDG12 and SDG13. For example, Sweden tops the list on the SDG Index, but is on the 138th position on the SDG12 ranking. On the other direction, Central African Republic which is at the bottom of the SDG Index gets the second best position on SDG13.”

This observation reaffirms what the Commission on the Measurement of Economic Performance and Social Progress, led by Nobel Prize winners Amartya Sen and Joseph Stiglitz had said already in 2009: “The assessment of sustainability is complementary to the question of current well-being or economic performance, and must be examined separately. (…) For instance, confusion may arise when one tries to combine current well-being and sustainability into a single indicator. To make an analogy, when driving a car, a meter that added up in one single number the current speed of the vehicle and the remaining level of gasoline would not be of any help to the driver. Both pieces of information are critical and need to be displayed in distinct, clearly visible areas of the dashboard.”

The BS/SDG Index uses the term “dashboards” to name each of the 17 averages. In the final average of averages, the two SDGs where rich countries perform poorly are outnumbered by 15 others that are shaped to correlate with conventional development rankings. The final ranking is affected both by the decision to weight each of the 17 goals equally in the average, but also by the decision to not include in each goal’s score the implementation targets that usually require from the richest support for those left behind. While the poor performance in well-being indicators of poorer countries is counted on each goal’s average, the failure of rich countries to support them, as required in the implementation target of every one of the SDGs, is only counted only once, in the average for SDG17. Within the many indicators within the SDG 17 “dashboard”, a bad cooperation performance can be compensated by higher domestic spending in health and education.

The SDG Index has no space for the notion of limited “stocks” (of air, water, biodiversity or minerals) that are being depleted in unsustainable ways by a few while the majorities lack the minimum resources for a decent life. But the Index does acknowledge that there are “spillovers”, negative or positive, of national activities over other countries and it creates a country by country “spillover score” averaging indicators that range from tax havens (based on Oxfam data) and financial secrecy (based on Tax Justice Network data) to the amount of accidents at work embodied in imported goods to arms exports. Official Development Assistance is counted for as “positive spillover” as well as contributions to peacekeeping.

Even acknowledging that “environmental spillovers can be generated in two ways: i) transboundary effects embodied in trade; ii) direct cross-border flows in air and water,” the 2019 report “only includes indicators on environmental spillovers into trade” because “generating global measures of cross-border flows available at the country level remains an important research agenda”.

This leaves out problems of cross-border water appropriation or contamination, but also greenhouse gas emissions, which is an issue of enormous international concern. This option by the authors is difficult to understand, since data about emissions are abundant. An indicator on “imported biodiversity threats” is included under SDG15, one on “imported water depletion” is part of the average for SDG6 and one on “imported fatal work accidents” in SDG8, but climate change related deaths are attributed in SDG13 to the countries where they happen, and not to those responsible for the greenhouse gas emissions?

Spillover Score: The worst offenders and the good planetary neighbours

Source: Online database for the BS/Sustainable Development Report 2019

The author of this blog computed a negative correlation factor of -0.5 between the GSI and the spillover score, a number which can be considered as a “relatively strong inverse correlation” in social sciences. The higher a country is ranked in GSI, the worse its negative spillovers, as defined by the very same SDG report. Yet, instead of pointing to that negative correlation, which leads to uncomfortable questions of causality, the authors prefer to comment that “there is high variation in spillovers among countries with a similar per capita income. This suggests that countries can reduce their negative spillover effects without reducing their per capita incomes.”

Thus, the problem is portrayed as one of policy options by each country, ignoring the notion of trade-offs between, for example, economic growth and environmental protection and downplaying the role of multilateral agreements, like the conventions on climate change or on biodiversity in defending the global commons.

Further, the spillover “score” that averages a complex mix of economic, environmental and security indicators is expressed on a per capita basis, and the emphasis on trade and non-inclusion of climate produces a table (see above) where “on a per capita basis, small countries with large trade intensity – such as Luxembourg, Singapore and Switzerland – generate the highest negative spillover effects.” This seems to suggest that trade itself is to blame for the spillover, instead of explaining that this is the result of only computing, for example, the water depletion, without accounting for the domestic depletion, that tends to be larger in larger countries, less dependent on foreign trade.

Similarly, the per capita computation of spillovers can be very useful to attribute comparable moral responsibilities. Yet, whatever the faults of a small country like Cyprus, its global impact is limited by size. The report does not make any attempt to measure the absolute spillovers, which would not exempt the small-sized culprits from their sins but could help to better understand the global problems and where to start tackling them.

“Development” used to be understood as a linear progression from low to high, from poor to rich, judged by a single number, be that per cápita GDP or HDI. The Commission on the Measurement of Economic Performance and Social Progress strongly warned against using a single number to measure sustainability, as that could create “confusion”, even when recognizing that “ there are strong demands to develop a single summary measure”. The SDG report ignores the warning and tries to satisfy such demand by picturing the achievement of the SDGs as a kind of Olympic games, where countries accumulate medals in different disciplines. As in the Marathon, some runners will reach the goal faster than others, but eventually with some effort all will get there and the more advantageous might give some advice, encouragement and good example to those following behind.

The GSI as a summary measure ends up being so similar in its ranking to what we already know from the Human Development Index, that it can only support the continuation of the existing development strategy. Rich countries are encouraged to contribute more ODA and more peacekeeping and given a gentle slap on the wrist for their insistence on some bad practices like exporting arms or attracting illicit financial flows, making it harder for poor countries to become like the Nordics. But, ultimately, if we are not “on track” to reach all goals what we need to do is “accelerate” what we are already doing.

But, what if instead of a quest for more medals, the trade-offs were addressed? As the evidence mounts that “business as usual” is not delivering the expected results by 2030, this is the question that country leaders and the international development system need to address in their review of the 2030 Agenda.


This blog post was first published by Global Policy Watch.


*Sachs, J., Schmidt-Traub, G., Kroll, C., Lafortune, G., Fuller, G. (2019): Sustainable Development Report 2019. New York: Bertelsmann Stiftung and Sustainable Development Solutions Network (SDSN). Available at: http://unsdsn.org/