Is the Environmental Performance Index really faulty?

This artical originally appeared in The Hindu

While the methodology has issues, this is an opportunity for India to study where it stands

Last month, India protested against its ranking on the Environmental Performance Index (EPI) of 2022, prepared by researchers at the Yale and Columbia Universities in the U.S. The report measures 40 performance indicators across 11 categories to measure the “state of sustainability around the world.” India was ranked last (180) with low scores across a range of indicators. The Indian Government as well as environment experts have pointed to the faulty methodology of the index that skews the results in favour of the Global North. Chandra Bhushan, Sharad Lele and Anant Sudarshan discuss the report in a conversation moderated by Sonikka Loganathan. Edited excerpts:

What are the issues with the methodology?

Chandra Bhushan: Rating by its very nature is a subjective exercise. But a good rating is one that tries to reduce subjectivity, normalises all indicators, and then develops consensus around the subjective issues. The first step is to remove subjectivity as much as possible. Every rating will end up comparing apples with oranges, if you don’t normalise the indicators. So, the second step is to normalise indicators. Third, if there is subjectivity, you get experts to generate consensus around it. All three have not been done.

But this was a peer-reviewed study…

Chandra Bhushan: I’m not sure what kind of peer review was done because, if you look at the indicators, even a person with basic knowledge of ratings would tell you that the indicators have not been normalised.

Can you give us an example of where this lack of normalisation has impacted India’s rank in a category?

Chandra Bhushan: EPI has used tree cover loss as an indicator to rate deforestation in a country. Eritrea is the best country [as per the ranking]. The total dense forest cover in Eritrea is only about 50 hectares, which is similar to forest cover in one part of Lutyens’ Delhi. How do you compare absolute tree cover loss of a country with 50 ha dense forest with, say, India with millions of ha of dense forest and a tree cover loss of 1 lakh ha?

Is a rating the right way to be measuring environmental progress? What do you think of the government’s response?

Sharad Lele: There is a difference between an index and a ranking. Indices themselves have very limited value, even if you make them absolute, because they collapse the hugely complex issue of environment into one number. But relative ranking is just useless. For example, you could have all countries between seven and nine out of 10. Some country will still end up at 180 because it is at 7.0 whereas others are 7.1 and above. What does that tell you about environment performance? Nothing.

Now the government, instead of responding and quibbling about details, could have used this occasion to call for a meeting of people within the country who follow these issues, to ask questions about where we are, and put out maybe our own performance index, in a much more nuanced manner that tells us something about where we are with respect to, say, five or 10 years ago

Anant Sudarshan: The EPI has a large data set with a huge amount of information on a whole range of indicators. This is more than just an exercise of coming up with one number— it’s a data collection exercise on a whole range of indicators. Certainly, it would be nice if something similar were produced by our Government. Nevertheless, if you look at every single one of the indicators you’ll find that India does quite badly on most. This shouldn’t come as a surprise to most environmentalists. The point of a rating like this is that it puts together a lot of data and it reminds us that things are not going well on a wide range of environmental outcomes in India.

Chandra Bhushan: But Anant, I also want us to understand how this rating was released and what message it gave out. Its message was: if you are big, if you are middle income or a poor country, if you are in Asia or Africa, you are bad environmentally. But if you are a rich country, you consume a lot, but your local environment is clean, you are the best in the world. I don’t think that’s right. If you want to solve environmental problems, consumption is what you attack. While recognising that India has problems, I am not willing to accept that the Western world is the paragon of environmental performance in the world.

Sharad Lele: Ideally, in an EPI, you would look at outcomes. But in reality, you have very limited data on actual outcomes, so you start using proxies like actions taken towards those outcomes. The main indicator of climate change performance is whether the atmospheric carbon dioxide concentration is increasing or decreasing. We all know it is increasing. The world is doing terribly on climate change. How do you allocate this global performance index on climate change, or this outcome variable, to different countries? You would see who are the biggest polluters, and, on a per capita basis, it’s the Global North. When you have a global outcome such as climate change, you have to first determine who is responsible for doing what and ask what have you done towards meeting that responsibility. Otherwise a country such as the USA gets high marks for reducing emissions from 18 to 16 tCO2/e/capita/year while the global South, which is sitting at 2-3-4 tCO2e/capita/year and increasing emissions marginally gets minus marks.

Similarly, if biodiversity is construed to be a global common good, and if a country has wiped out its biodiversity, why should it be getting higher marks because it then added one more protected area?

Speaking of biodiversity, how is Brazil ranked much higher than India, despite rampant deforestation in the Amazon rainforest?

Sharad Lele: If you want to measure biodiversity performance, you would see how the biodiversity was last year and whether it has changed this year. Then you would ask whether there are flagship species that you could use as an indicator. Instead, the entire focus of the EPI is on habitat. With some combination of percentage and absolute values, you have the West doing well and South Asia doing badly. There’s a real problem because habitat is being measured in terms of what percentage of the country is under protection. Brazil could be doing well because it’s a big country with a relatively low population density. A significant percentage of Brazil is under protected area. But in a densely populated country like India, you are not going to be able to put a high proportion of area under strict protection.

India puts out the State of Forest report. But the definition of a forest is ever-changing, which is why India has seen an increase in forest cover, as per those reports. Can you contextualise this issue?

Sharad Lele: You used the word forest cover. The EPI uses the word tree cover. Therein lies the story of how India itself is playing around with this issue. We have not asked why we care about forest cover. There are different answers to this, but if you focus on the carbon sequestration benefits of forests, you wouldn’t care whether it is palm or eucalyptus or a natural species which is endemic to India, because it’s all carbon. On the other hand, if you care about biodiversity, you would want to look at forests as an association of species which are part of this landscape and not just a random species planted for the sake of making the place look green. So, why we care about forest cover determines what we measure. To take another angle, if you are a local person who is dependent on forest for livelihood, you would prefer an open canopy forest, and may be trimming the trees to get firewood without cutting down the whole tree. In that case, you would see very little tree crown cover, which is what the Forest Survey of India measures through satellites. So, when the EPI looks at tree cover, they are falling into the same trap. Should they look at tree cover or should they look at forest cover, which means natural forests? In the Indian context, this matters because natural forest cover has gone down, while plantations have increased, revealing the fault lines in this issue.

One solution we’ve seen grow in popularity is tree planting. Is this actually effective?

Chandra Bhushan: Planting trees has become like atoning your environmental sin. This is a very dangerous solution to the kind of environmental problems we have, because we are forgetting the role of different ecosystems.

Anant Sudarshan: One thing that is dangerous is letting only the government define the metrics it will use to measure success without independent scientific scrutiny. In India, we’ve had this massive increase in what is called forest cover, which is all driven by plantations, while natural forests are dropping. In this indicator, EPI is using tree cover loss from satellite data, so India is doing better on this than it should by some metrics. But at least it’s a data point that’s being independently collected and that’s similar across countries. The criticism of Brazil for tree cover loss and the praise of the Indian government for “forest gain” are really talking about two very different things. One is the rainforest disappearing there and one is plantations being added here. I think that’s a place where an independent index helps, because if we can agree on the indicators, we can get an objective basis of measurement.

Sharad Lele: There is a funny contradiction here. When it came to biodiversity, because you couldn’t measure the outcome very well, you put a lot of emphasis on process and said protected areas is the way to get to biodiversity conservation. When it comes to ecosystem services it is also well acknowledged that local community involvement and people’s rights is actually a better way to achieve sustainable enhancement of ecosystem services of all these areas. So how come there is no measure on how much have you decentralised rights over trees or forests, in local communities? If you took that as an indicator, we would be a real laggard in spite of having the Forest Rights Act of 2006.

India ranked 179 in air quality. How do we solve this?

Anant Sudarshan: We have failed to control air pollution so far. This is where these indices are useful. It’s not useful to compare India with London, but you could compare India with other countries at the same income level and the same population density, and there are many countries that are doing better. So, once we notice this we can ask, why are we doing worse? A large part of it is regulation. Ultimately air pollution is the sort of problem that gets solved through economy-wide regulation.

Chandra Bhushan: I agree that there is a regulatory problem with air pollution in India, but there is also a fundamental problem with the economy. No country in the world has been able to solve air pollution without getting rid of biomass or solid fuel. India combusts around 2.2 billion tonnes of material, of which 1.6 billion tonnes are coal and biomass. Biomass is a problem of poverty and coal is the problem of energy access. The way India will reduce its air pollution is also the way it will solve its climate challenge. The fundamental reason why India will not be able to resolve a lot of its air pollution challenge is because of our energy mix. For example, tomorrow, if all the vehicles in India move to electric vehicles, we will be able to reduce our air pollution, cumulatively, by 20%, but 80% problem will not be solved.

In preparation for the upcoming COP 27, what should India be doing, especially since we’ve seen an increased coal production target?

Chandra Bhushan: The Russia-Ukraine crisis could have been an opportunity for all of us to start investing massively in renewable energy. But fossil fuel companies have used this short-term deficit in energy supply as an opportunity to open new fossil fuel establishments. In India, fossil fuel consumption is going to increase in the short term. If we are smart, we will try and peak coal as quickly as possible. That would be our roadmap.

Anant Sudarshan is South Asia Director of the Energy Policy Institute at the University of Chicago; Sharachchandra Lele is Distinguished Fellow in Environmental Policy and Governance at ATREE, Bengaluru, and Professor at IISER, Pune and SNU, Delhi; Chandra Bhushan is President and CEO of the International Forum for Environment, Sustainability and Technology (iFOREST)