Metrics, KRIs & Reporting interview questions

• KPI (Key Performance Indicator): measures how well a process or programme is performing against its goals — e.g. percentage of staff completing training on time.
• KRI (Key Risk Indicator): an early warning signal that risk exposure is increasing — e.g. number of critical vulnerabilities older than the patching SLA.
• KCI (Key Control Indicator): measures whether a specific control is operating effectively — e.g. percentage of leavers whose access was removed within the required time.

The same metric can sometimes serve more than one purpose; what matters is that it is linked to an objective, risk or control and has thresholds.

• Percentage of critical/high vulnerabilities remediated within SLA; ageing of open vulnerabilities.
• MFA coverage for remote, privileged and cloud access.
• Percentage of assets with EDR / encryption / up-to-date patches.
• Privileged accounts reviewed on schedule; orphaned accounts found.
• Phishing simulation report rate vs click rate trends.
• Mean time to detect and respond to incidents.
• Backup success rate and results of restore tests.
• Number of high-risk vendors without a current assessment.
• Open audit findings and overdue remediation actions by severity.
• Number of risks above appetite and accepted risks past their review date.

• Measure: a raw value collected, such as '37 critical vulnerabilities'.
• Metric: a measure put in context so it means something, such as '37 critical vulnerabilities older than 30 days, up from 22 last month'.
• Target or threshold: the value you are aiming for or must not exceed, such as 'zero critical vulnerabilities older than 30 days' or 'amber above 10, red above 25'.

Good reporting shows the metric, the target and the trend together.

A dashboard displays key metrics visually so different audiences can see status at a glance:

• Board and executives: top risks against appetite, trends and decisions needed.
• Risk committee and CISO: KRIs, programme progress, major findings and exceptions.
• Control owners and operational teams: detailed control health, overdue actions and tasks.

Each audience needs a different level of detail. One dashboard for everyone usually serves no one well.

• MTTD: the average time between an incident starting (or the first malicious activity) and its detection.
• MTTR: the average time from detection to containment or resolution. Define which one you mean, because 'respond' is used for both.
• Lower values mean less time for attackers to cause damage.
• Measure them consistently with clear start and end points, and look at the distribution, because one very long incident can distort an average.

1. Start from objectives and top risks. Ask what leadership needs to know.
2. Pick a small set (5–10) of metrics that can be measured reliably now, such as overdue findings, critical patch age, MFA coverage, access review completion and vendor assessments on time.
3. Define each metric clearly with a data source and an owner.
4. Collect a baseline for a few months before setting targets.
5. Report simply and improve the set over time.

• Linked to a specific risk in the register and to risk appetite.
• Leading (predictive) where possible, not only lagging.
• Measurable, consistent and ideally automated from reliable data sources.
• Has defined thresholds (e.g. green / amber / red) and an owner who acts when thresholds are breached.
• Easy to understand by its audience.
• Reviewed periodically to make sure it still reflects the risk.

• Lagging indicators measure outcomes that have already happened — e.g. number of incidents, breaches or audit findings last quarter.
• Leading indicators predict future risk — e.g. rising patching backlog, falling training completion, increasing number of policy exceptions.

A good dashboard combines both: lagging shows results, leading lets you intervene early.

A maturity model rates how developed a capability is on a scale, often from initial/ad hoc to optimised. Examples include CMMI-style 0–5 scales, the NIST CSF Tiers (as an indicator of rigour) and C2M2 (Cybersecurity Capability Maturity Model) for energy and other sectors.

Used to:

• Baseline current state and set a target state.
• Prioritise improvements and build a roadmap.
• Communicate progress to leadership over time.

Ratings should be evidence-based and calibrated to avoid optimism bias.

• Too much technical detail, too little business context.
• Vanity metrics (e.g. 'number of attacks blocked') that do not drive decisions.
• No thresholds or targets, so nobody knows whether a number is good or bad.
• Inconsistent definitions between periods, making trends meaningless.
• Manual, error-prone data collection.
• Reporting status without clear asks or decisions for the audience.

• Start from risk appetite and tolerance: red should mean 'outside appetite, escalation needed'.
• Use historical data to understand normal ranges, then set amber as an early warning before red.
• Agree thresholds with risk owners and the risk committee, and document them.
• Define the action at each level: amber means the owner investigates, red means escalation to the committee with a plan.
• Review thresholds annually or when the environment changes.

• Go beyond completion rates, which show activity but not effect.
• Behaviour metrics: phishing simulation click rate and, more importantly, report rate and time to report.
• Real-world outcomes: real phishing reported by staff, incidents caused by human error, policy violations.
• Targeted measurement: high-risk groups (finance, executives, administrators) separately.
• Knowledge checks and surveys, used carefully.

A balanced scorecard groups metrics into a few perspectives so reporting is not dominated by technical numbers:

• Risk: top risks and KRIs against appetite.
• Compliance: audit findings, certification status, regulatory obligations.
• Operations: control health, incident metrics, vulnerability management.
• Programme delivery: roadmap progress, budget, resourcing.
• People and culture: awareness behaviour, staff turnover in the security team, training.

Pick 3–5 metrics per perspective at most.

• Summarise open findings by rating, owner and age, with trends.
• Highlight overdue items and repeat findings, which signal governance problems.
• Show the root-cause themes behind the findings (for example identity management) rather than a long list.
• Show progress against agreed dates, and any requests for extensions with reasons.
• Ask for specific decisions or support where needed.

• A headline: overall position against appetite in one or two sentences.
• Top 5 risks with a rating, trend arrow and one line on what is being done.
• 3–5 KRIs with thresholds, highlighting any amber or red.
• Significant events: incidents, audit results, regulatory changes.
• Roadmap status: on track, at risk, late.
• Decisions or support needed from the committee.

Use plain language and avoid technical jargon.

• Check that the KRI is still valid. Is the threshold realistic and the data accurate?
• If it is valid, the problem is governance: the escalation is not leading to action.
• Present it to the risk committee with the consequence in business terms, the cause, and options with costs.
• Ask the committee for a decision: fund a fix, change the appetite consciously, or formally accept the risk.
• Record the decision. Continued red without a decision is unmanaged risk.

• Find the cause: different data sources, definitions, timing or scope.
• Agree one metric definition: what is counted, from which source, when, and how it is calculated.
• Assign a metric owner responsible for its accuracy.
• Document it in a metric catalogue and automate collection where possible.
• Reconcile historic numbers so trends are not broken, or mark the change clearly.

• Explain that zero incidents is unrealistic and can be harmful: it encourages under-reporting and hiding events.
• Suggest better targets: faster detection and containment, no high-severity incidents caused by known unpatched vulnerabilities, and 100% of incidents reported within the policy timeframe.
• Encourage reporting: a rise in reported minor incidents can be a sign of better visibility and culture.
• Keep severity in view: what matters is limiting the impact.

• Speak in business terms: impact on strategy, revenue, operations, patients/customers, regulation and reputation — not tool names.
• Focus on top risks versus appetite, trend direction and what decisions or support you need.
• Keep it concise: a one-page summary with drill-down appendices.
• Show progress against the roadmap and investment outcomes.
• Include significant incidents, regulatory changes and audit results briefly.
• Use quantification where possible (e.g. loss ranges) and consistent visuals over time.
• Be honest about gaps — credibility matters more than a green dashboard.

• Pick a few top risk scenarios and estimate loss ranges and frequency (for example, with FAIR).
• Show annualised loss exposure and a loss exceedance curve or a simple range ('there is a 10% chance of losing more than X in a year').
• Compare risk with appetite stated in financial terms, and show how proposed investments change the exposure.
• Explain the assumptions and uncertainty clearly. Ranges are honest; precise single numbers are not.
• Start small and build credibility, combining quantified and qualitative views.

• Watermelon metrics look green at summary level but hide real problems underneath.
• Causes: averaging across areas, measuring activity rather than outcome, gaming targets, or excluding hard-to-measure assets.
• Fixes: measure coverage (what percentage of assets is included in the metric), show worst performers, not just averages, and combine automated data with independent testing.
• Regularly ask 'what would make this metric misleading?' and validate it against incidents and audit results.

• Be careful: industry benchmarks are often based on different definitions, self-reported surveys or small samples.
• Use credible sources where available (industry ISACs, maturity benchmarks from assessments, insurer feedback) and say what they can and cannot show.
• Focus on your own risk against appetite and your trend, which matter more than peer averages.
• If useful, compare maturity using a common framework (for example, CSF profiles) with peers through an independent assessor.
• Avoid claiming 'we are better than others'. Attackers do not grade on a curve.

• The risk story: top scenarios, current exposure and trend.
• Evidence of gaps from metrics and testing: for example, recovery test results, coverage gaps or overdue findings.
• For each request: the cost, the expected risk reduction (quantified where possible) and the metrics that will show success.
• What happens if not funded: residual risk, regulatory or contractual consequences.
• Results from previous investments, to build credibility.