For a detailed discussion of the methodology used to produce the index, download the Working Draft of our Index Summary & Methodology paper here.

Consider how you might measure a population’s understanding of mathematics. You would want to assess mathematical understanding across various forms, from basic arithmetic through to advanced calculus, as well as measure the quality of the educational infrastructure for making mathematics instruction available to the people. This analogy reflects how our Cyber Risk Literacy and Education Index works. It measures key determinants of how well citizens of the world’s major economic geographies understand the elements of cybersecurity, their motivation to further their knowledge, and the tools available to them.

Our Index has the following components.

Drivers –Factors that drive changes to a population’s average cyber risk literacy today and the potential for future improvement.

Pillars – Items that track trends or changes in the average cyber literacy level of each geography as generated by each of the five drivers.

Objectives – Goals that a geography needs to accomplish to address the needs of each corresponding pillar.

Indicators – One or more datasets that measure or serve as a proxy for a geography’s performance on a specific objective.

The index is structured such that the indicator scores produce an objective score, which produce a pillar score, which then produce driver scores that ultimately come together to provide the overall score for a geography on the index.

Geography selection

We selected index constituents at the geography level based on the following criteria: they are economically, politically, culturally, or militarily influential (or part of an influential regional bloc); and they demonstrate a demand for cyber risk literacy. The availability of reliable data and the greater transparency of national governments in publishing polices and regulations were also contributing factors. Several geographies were initially considered but later dropped due to data constraints, including Pakistan, Iran, Iceland, Luxembourg, Malta, and Liechtenstein. Future versions of this index may explore additional jurisdictions of key geographies.

Research and expert interviews

We conducted secondary research and a review of the academic literature on the definition, importance, and evaluation of cyber risk literacy and education. Scholarship in the field is still new, particularly in methods of assessment and comparison across geographies. To supplement this research, we interviewed top global cybersecurity academics, industry experts, policy makers and think tanks to gain a variety of perspectives on the current state of cyber literacy and education, and best practices for geographies to follow. We used the critical insights from these interviews to set our priorities, identify geography objectives, and guide indicator selection. Finally, we also reviewed the methodologies of prominent indexes to collect best practices and compare aggregation and scoring techniques.

Drivers, pillars, and objectives development

The five drivers can be thought of as sub-indices that measure specific changes related to public motivation, government policies, educational systems, labor markets, and population inclusivity. Each driver is comprised of one to two pillars that differentiate the measurement. These pillars measure the extent to which our selected geographies have fulfilled designated objectives underneath them, which our experts believe are crucial for developing a cyber-resilient population. These various elements were developed in conjunction with research and expert interviews, and tested by our internal team and Steering Committee members. Objectives were sorted and categorized based on a natural order/ progression of achievement by geographies (i.e., more readily achievable objectives are listed first). Under these objectives sits our selection of indicators, which measure the extent to which each objective is fulfilled in each geography.

Indicator selection

Indicators aim to measure the stated objective. We aimed to select indicators with data that both covered all or the vast majority of ranked geographies, and that would likely be updated over time. Additionally, indicators needed to reflect independent academic rigor and be relevant to the objective being measured, as judged by our internal advisors. Data for these indicators was collected by the Oliver Wyman Forum through both existing statistical data, as well as through independent analysis. Independent analysis was conducted to evaluate both National Cybersecurity Strategies, as well as National (or proxy) Curricula for Cybersecurity Instruction. Assessment frameworks were developed in both instances with guidance from the Oliver Wyman Forum and the Index Governance Committee. National Cybersecurity Strategies and National Curricula for Cybersecurity Instruction were given scores using these frameworks, which ultimately fed into the scoring for the index.

Data imputation

Not all indicators have complete datasets across the geographies selected for the index. We considered a few imputation techniques to estimate missing data values, each with its own strengths and weaknesses. We aim to ensure that imputed data reflect a geography’s actual population levels. Our primary choice was to use a second source of data directly comparable to the primary source. However, this was rarely possible due to small variances in how survey questions were asked or how metrics were compiled, which could lead to large incompatibility between two datasets. When a second data source was not available, we followed methods similar to ITU’s ICT Development Index (IDI). We utilized hot-deck imputation, which uses data from geographies with similar characteristics, such as GDP per capita and geographic location or cultural similarity.

Data normalization

The indicators selected for the index are often based on different units of measurements or scales. Thus, we needed to normalize the indicators such that they become comparable between geographies and allow geographies to understand their progress over time. A review of the OECD Handbook as well as existing indices revealed two methods that we assessed to be most relevant for our index: The “distance to frontier” approach and the “standardization” (or z-scores) approach. Ultimately, we chose to utilize the distance to frontier approach. It is used by several well-known indices including the Legatum Prosperity Index (2019), and the World Economic Forum (WEF) Global Competitiveness Report (2019). Upon discussion with our internal experts, we also believe that the distance to frontier approach is more intuitive for the reader to understand the scoring gap between various geographies. For a detailed discussion on and data from our comparison of these two approaches, please see the full methodology paper.

Weighting and aggregation

We used an expert-informed weighting, adjusted for indicator quality. We also considered and discussed many other logical approaches, such as equal weighting. Pillars have different relative weights in the Index, and objectives have different relative weights within their respective pillars. In general, objectives that used indicators our governance group believed to be more aligned to the objective measured and less biased carried creater weighting. We emphasized greater weights on pillars that had stronger indicators or were more relevant for assessing the overall national cyber risk literacy picture. As we broaden our ability to capture the highest quality indicators that are most relevant to each objective, we may tweak the weightings of objectives in future versions of the index. For a more detailed discussion on and data from our weighting and aggregation reasoning, please see the full methodology paper.