Main Content

Validation of Credit Models in ECB Templates

This example uses:

Open Live Script

This example shows how to fill in the European Central Bank (ECB) model validation templates using Modelscape™ software.

The ECB has published a suite of model validation templates covering a wide array of credit models, including probability of default (PD) models. To access the ECB templates, navigate to the Related Documents section of Ongoing model monitoring. For instructions on how to fill in the templates in the document, see [1].

Fill in ECB Template Input Data Using Modelscape

The ECB validation templates these inputs:

  • Model and organization metadata, such as model and institution names, and the start and end dates of the review period

  • Directly observable quantitative data, such as the number of customers who defaulted during the observation period

  • Model performance metrics, such as metrics that measure the discriminatory power of the model or credit rating migrations

You can fill in the qualitative data and performance metrics by using the fillValidationTemplate function. You can then fill in the model names, identifiers, and other variables in Excel. 

fillValidationTemplate(fileName, ...
      "InitialValidationData", initialValidationData, ...
      "InitialPortfolioData", reviewStartPortfolioData, ...
      "TerminalPortfolioData", reviewEndPortfolioData);

  • fileName is the name of the spreadsheet you fill in. Use the standard format required by ECB: "[LEICode]_[ModelType]_[ModelID]_[ReferenceDate]_[VersionNumber].xlsx”. For more details, see the final paragraph of Section 2.2 of the ECB instructions.

  • reviewStartPortfolioData and reviewEndPortfolioData are mrm.data.validation.reporting.pd.OperationalData objects with information about the portfolio at the beginning and the end of the current review period. Examples include the model PDs, exposures, and ratings for the customer populations, as well as flags to indicate credit transferals and overrides within the populations.

  • initialValidationData is an mrm.data.validation.reporting.pd.ValidationData object with the information used in the initial validation of the model, potentially many years before the current review period. Examples of such information include the original model PDs and default indicators for some validation data set.

Set Operational Data

The ECB templates require operational data as a table containing one item per customer or entity, together with the observation date and flags such as whether the model allows technical defaults. The term name indicates a single obligor such as a customer.

To fill in an ECB template, specify each of these variables as vectors with the same length:

  • IDs — Identifier for each name in the data set, specified as a string array.

  • PDs — Probabilities of default predicted by the model, specified as a numeric array.

  • Ratings — Ratings assigned to the customers. You must store the ratings as categorical variables with a fixed ascending order to ensure that all the data quality checks in the templates pass. Define categorical variables in MATLAB using the following syntax:

ratingLabels = categorical(["CCC", "B", "BB", "BBB", "A", "AA", "AAA"], ... 
    ["CCC", "B", "BB", "BBB", "A", "AA", "AAA"], 'Ordinal', true);

  • Exposures — Current exposure for each name. Although the ECB instructions do not specify a particular currency, specify this data in a single currency to ensure that the validation measures can calculate the total exposures within all classes of names.

The operational data must contain these logical vectors with the same length as the identifier vector. These vectors indicate anomalous states of the names. If you do not specify these vectors, the OperationalData class sets these values to false.

Because the flags determine the correct in-scope sample, it is important that you set them correctly.

  • InDefault — Indication of whether the corresponding name is currently in default. This criterion corresponds to definition 2.3 (c) in [1].

  • TechnicalDefaultIndicators — Indication of whether the corresopnding name is currently in technical default. This criterion corresponds to definition 2.3 (d) in [1].

  • OtherModel — Indication of whether a different model is used for the name for rating or regulatory capital purposes. This criterion corresponds to definition 2.3 (e) in [1].

  • NonDefaultablePositive — Indication of whether the corresponding name is non-defaultable. This criterion corresponds to footnote 18 on page 10 of [1].

  • Deficient — Indication of whether the name should be excluded from the sample due to process deficiency. This criterion corresponds to definition 2.5.1 (e) in [1].

  • Outdated — Indication of whether the name has an outdated rating or financial statements. This criterion corresponds to definition 2.5.1 (d) in [1].

  • Overridden — Indication of whether the rating is overridden. This criterion corresponds to definition (2.5.1 (e) in [1].

  • Transferred — Indication of whether the name is transferred to the rating of a third party. This criterion corresponds to definition 2.5.1 (f) in [1].

Finally, the OperationalData class must also contain these properties:

  • OutdatedAllowed, TransferralAllowed, OverridesAllowed, and TechnicalDefaultsAllowed — Logical scalars that indicate whether the model and assignment process allow for outdated accounts, rating transferals, overrides, or technical defaults. These flags are used in Sheet 2.0 of the PD model validation template.

  • Date: The as-of date for the above data.

In practice, saving the one-datum-per-name data as a table and supplying the final four logical scalars and the as-of date separately is a convenient approach to storing the required information.

Set Validation Data

Operational data objects contain snapshots of the model predictions and other relevant data at given time points. Validation data objects, on the other hand, carry information about how the status of the names changes over time. Validation metrics assess how these changes relate to the PDs that the model predicts.

Each ValidationData object carries this data.

  • ObsStartDate and ObsEndDate — Initial and terminal dates of the observation period.

  • IDs — String identifier for each name in the data set.

  • PDs — Model PDs as predicted at ObsStartDate.

  • Exposures — Exposures of the names in some appropriate reference currency, as of ObsStartDate.

  • DefaultIndicators — Logical flags indicating the names that default during the observation period.

  • InitialRatings and TerminalRatings — Rating grades that the model predicts at the beginning and the end of the observation period (as ordered categorical variables). The terminal ratings must have an ascending order again.

In addition, ValidationData uses an augmented set of possible ratings. These extra ratings are allowed according to point 2.5.1 (h) of [1]:

  • Default — Rating for names that default during the observation period

  • Migrated — Rating for non-defaulted names that move to a different model during the observation period

  • Gone — Rating for non-defaulted names that terminate their business relationship with the credit institution during the observation period

The validation data containers can have data only for the names that are pre-screened to be in scope for validation. Given operational data containers operationalData1 and operationalData2 for some reference dates, use a Modelscape helper function to construct the associated validation data. This function performs the appropriate screening and augments the terminal rating categories. 

validationData = mrm.data.validation.reporting.pd.validationDataFromOperationalData(operationalData1, operationalData2);

Reference

[1] IRB Pillar I models for credit risk. "Instructions for reporting the validation results of internal models." European Central Bank -- Banking Supervision.