There is no doubt the current COVID-19 pandemic is a defining moment in our lifetime. How do we support the world after the pandemic to be better than what we will leave behind? This is the key question across many professions as we look towards the future, a future where data can and should drive decision making.
The current focus on data is exciting and necessary. The need for high quality and timely data is more urgent than ever, but at the same time some of the ‘usual ways’ for producing official statistics is made difficult as survey operations are disrupted, priorities are changed and budgets re-purposed.
The present difficulties with traditional data collections, in particular face-to-face surveys, have highlighted what statisticians have known for a while – new methods and data sources will be the key to our futures. COVID-19 therefore gives National Statistical Systems the impetus to adapt and fast-pace the move towards more innovative and non-traditional data sources, often labelled as ‘big data’. As an example, the Australian Bureau of Statistics is exploring administrative and transactions data from the public and private sector to inform social and economic statistics in response to COVID-19.
When using big data, challenges are many and include access to data, capturing data, data storage, data analysis, search, sharing, transfer, visualization, querying, updating, information privacy, confidentiality, and security. Globally, statisticians are working to overcome these challenges through the work of the UN Global Working Group on Big Data for Official Statistics in which ESCAP and its members play active roles.
When thinking about big data and COVID-19, many are concerned about how companies and governments worldwide are tapping the location data of millions of internet and mobile phone users for clues about how the virus spreads and whether physical distancing measures are working. But if adhering to the ten rules of engagement of big data for official statistics, these concerns can be mitigated and new methods and data sources can be used for public good and in the interest of people and development.
Countries in Asia and the Pacific have been experimenting with using big data for many years and sharing this experience with support from ESCAP. In Thailand, for instance, the government-wide importance given to big data is driven by a desire to improve efficiency in public administration and citizens’ quality of life. Evidence of this importance can be seen through the establishment of a National Committee on Driving Policy Operation for Utilizing Big Data, Data Center and Cloud Computing with the Deputy Prime Minister as President.
Meanwhile since 2016, Indonesia has been using mobile phone data for the monthly production of inbound tourism statistics which was previously based on costly cross border surveys and this year, BPS Indonesia, the national statistical office, will start integrating mobile positioning data with digital surveying for domestic and outbound tourism, instead of household surveys. During COVID-19, these new approaches have proven even more valuable as the data collection has not been affected by the pandemic.
A new ESCAP report documents and draws on the experience of three National Statistical Offices (NSOs) who have successfully implemented big data, specifically scanner data, into their Consumer Price Indexes (CPI). Lessons learned from the experience of the Australian Bureau of Statistics, the Statistics Bureau of Japan and Statistics New Zealand are shared in the document to help other NSOs in their efforts to incorporate similar approaches.
In many cases, the introduction of big data sources will mean NSOs can limit the burden on individual respondents and retailers. In addition, as society is becoming more concerned with the environmental impacts of our actions, using big data sources to avoid physically collecting price data could reduce carbon footprints involved in producing CPIs. Similar environmental benefits are possible by reducing face-to-face surveys.
Also, in a dynamic environment where data change frequently, traditional manual data collection methods risk underrepresentation and failure to capture user needs. The traditional data collection methods for CPI usually result in relatively small, but well targeted, samples of products and services. Because of their small size and relatively costliness, it is often hard for these samples to reflect the impact of promotional sales and new varieties of goods and services entering the market.
NSOs no longer enjoy the monopoly on producing the type of information they once did. The production of CPI is being challenged by private producers using non-traditional sources. Sticking with the status quo could be both a quality and a reputational risk that NSOs need to be aware of. User expectations are changing toward more granularity in official statistics, including CPIs. Non-traditional data sources can potentially help NSOs deliver more granular CPI series by significantly increasing sample sizes and the geographical coverage of indexes.
The increasing use of big data in Asia and the Pacific so far shows the value of international cooperation and learning from others. In a few months, the Asia Pacific Statistical community will convene at the 7th session of the Committee on Statistics, giving each other and ESCAP further guidance on the new normal for official statistics post COVID-19.