# Troubleshooting Safety Rules The Safety Reporting Rules Engine comes with _Safety Rule Sets_ for certain standard agencies and enables Admins to create custom _Safety Rule Sets_. When the _Evaluate Reporting Obligations_ action successfully runs for a _Case_ or _Localized Case_, the Safety Rule Engine creates a Submission Rule Log. The Log details the list of considered _Destinations_ and whether those _Destinations_ passed or failed any of the _Safety Rules_. However, if all of the _Safety Rules_ failed, this report does not indicate why. To investigate further, you can use the Safety Rules Troubleshooting tool to diagnose why specific _Safety Rules_ failed in order to fix them. The sections below describe how Admins can use the Safety Rules Troubleshooting tool to diagnose and fix issues with _Safety Rules_ that are failing, especially custom _Safety Rules_. ## Optional Prerequisite To include _Case Products_ with the _Drug Role_ of _Drug Not Administered_ when using the Safety Rules Troubleshooting tool, your Admin must have enabled Extend Definition of Suspect to Drug Not Administered. Otherwise, the Safety Rules Troubleshooting tool considers only _Case Products_ with the _Drug Role_ of _Suspect_ or _Interacting_. ## What to Look for When Evaluating the Reporting Obligations for a Case After running the _Evaluate Reporting Obligations_ action for a _Case_ or _Localized Case_, if Vault did not generate a _Transmission_ for a Destination when that was expected, examine the Submission Rule Log. * If the Destination does not appear in the report, verify that the _Case Product_ or _Case Study_ is registered to that Destination or has a Reporting Family configured. * If the Destination appears in the log but is marked as a failure (no Rules passed), then the issue is likely with the Rule. You can manually determine why the Rule did not pass, or use the Safety Rules Troubleshooting described in the following sections to diagnose the problem. ## Obtain the Rule Engine Troubleshooting Report for a Case The first step in diagnosing an issue with a _Safety Rule_ is to obtain a **Rule Engine Troubleshooting Report** for a _Case_ or _Localized Case_ that did (or did not) generate a _Transmission_ as expected. 1. Go to **Admin > Settings > Safety Rules Troubleshooting**. 2. Enter the **ID** of the _Case_ or _Localized Case_ you want to analyze. You may need to use the **All Actions > Edit Columns** feature on your _Cases_ or _Localized Cases_ view to add the **ID** column. 3. Select **Run Rule Engine**. 4. A notification appears when the **Rule Engine Troubleshooting Report** is ready to download. You can download the report either from the **Notifications** area, or from the system-generated email notification. The file is named using the Case ID you entered, plus the date and time you ran the report.
## Analyze the Rule Engine Troubleshooting Report for a Case Once you have the **Rule Engine Troubleshooting** report for a _Case_ or _Localized Case_, the next step is to analyze the results to determine why Vault did (or did not) generate a _Transmission_ as expected. The report is in CSV format. You can use a text or spreadsheet editor to examine the contents of the report. The report contains a row for each _Safety Rule Parameter_ that the Rule Engine evaluated for the _Case_ or _Localized Case_ and includes the following columns:
Column Name Description
ID The ID number of the Safety Rule Parameter evaluation. This is an incrementing number for each row, starting with one.
DateTime The date and time that the Safety Rule Parameter was evaluated.
Transmission Type For a Case, whether the evaluation was for an Early Notification, Submission, or a Distribution.
Reporting Scenario Whether the evaluation was for General Reporting or a Cross Reporting scenario.
Registration (Cross Reporting) For a Case, if the evaluation was for Cross Reporting, the name and ID of the corresponding Product or Study Registration that triggered the evaluation.
Registration ID (Cross Reporting)
Reporting Family (Distribution) For a Case, if the evaluation was for a Distribution, the name and ID of the Reporting Family that triggered the evaluation.
Reporting Family ID (Distribution)
Destination Name The name and ID of the Destination.
Destination ID
Transmission Profile ID For a Case with a Submission, the ID of the associated Transmission Profile.
Transmission Profile Selection Reason For a Case with a Submission, Vault's reason for the selected Transmission Profile.
RuleSet The name of the Safety Rule Set to which the evaluated Safety Rule belongs.
RuleName The name of the Safety Rule that was evaluated.
Parameter The Safety Rule Parameter that was evaluated.

Parameter Execution Status The outcome of the Safety Rule Parameter evaluation.
  • If the Rule Engine evaluated this parameter without error, the result will be either Pass or Fail.
  • If the Rule Engine encountered an error while evaluating this parameter, the result will show Error, with the description of the error shown in the Error Log column.
Rule Execution Status The outcome of the evaluation of the Safety Rule (Pass or Fail).
Eligible Product IDs 1 For a Case, the ID list of all the Products that were eligible for evaluation for the Safety Rule Parameter in that row.

In the event of multiple eligible Products where only a subset of the Products passes the parameter, the subset of passing Products is listed on the following row as Eligible Products.

Eligible Assessment IDs 1 For a Case, the ID list of all the Assessments that were eligible for evaluation for the Safety Rule Parameter in that row.

In the event of multiple eligible Assessments where only a subset of the Assessments passes the parameter, the subset of passing Assessments is listed on the following row as Eligible Assessments.

Passing Assessment IDs 1 For a Case, the ID list of all the Assessments that passed all of the Safety Rule Parameters within the Safety Rule.
Localized Case Assessment IDs 1 For a Localized Case, the ID list of all the Localized Case Assessments that were eligible for evaluation for the Safety Rule Parameter in that row.
Passing Localized Assessment IDs 1 For a Localized Case, the ID list of all the Localized Case Assessments that passed all of the Safety Rule Parameters within the Safety Rule.
Error Log The description of the error (if available) if the Rule Engine could not successfully evaluate the Safety Rule Parameter.
ExpectednessCalculation 1 For a Case, if the Rule uses the Expected parameter, this column contains the following details of the Expectedness calculation:
  1. The Assessment ID
  2. The result of the calculation (either 'Expected' or 'Unexpected')
  3. The source of the calculation (a Case Assessment ID, a Case Assessment Expectedness ID, or a Datasheet ID)

The report displays this information in the format <AssessmentID>-<ExpectednessResult>-<ExpectednessSourceID>.

1. Multiple values in these columns are separated by a semicolon.
Using this information, you can easily locate and fix problems with the _Safety Rules_ in a _Safety Rule Set_. For _Cases_ and _Localized Cases_ where the Rule Engine completed successfully but did not generate a _Transmission_ as expected, you can use the information in the report to determine why. For each _Safety Rule_ within a _Safety Rule Set_ that the Rule Engine evaluates for a _Case_ or _Localized Case_, you can see the initial list of Eligible Product and Assessment IDs (or the Localized Case Assessment IDs column for _Localized Cases_) in the first _Safety Rule Parameter_ row for that _Safety Rule_. If the _Safety Rule Parameter_ is evaluated as a `Pass`, the Rule Engine likewise evaluates the next _Safety Rule Parameter_ in the _Safety Rule_. If all the _Safety Rule Parameters_ pass, the _Safety Rule_ also passes. This is indicated by the Rule Execution Status column showing as `Pass` and the Passing Assessment IDs column (or the Localized Case Assessment IDs column for _Localized Cases_) showing which of the Eligible Assessments passed the _Safety Rule Parameter_. If a _Safety Rule Parameter_ evaluates as a `Fail`, the Rule Engine marks the _Safety Rule_ as a `Fail` and moves on to the next _Safety Rule_ in the _Safety Rule Set_. The following section gives examples of the possible scenarios. These will help you when analyzing the Rule Engine Troubleshooting report for your _Cases_ and _Localized Cases_. ## Rule Engine Troubleshooting Report Examples The Rule Engine Troubleshooting report examples in this section illustrate how the results in the report appear for each of the following reporting scenarios: * [A Passing Safety Rule][1] * [A Failing Safety Rule][2] * [Products and Assessments eliminated by a Safety Rule Parameter][3] * [A Safety Rule that resulted in an error][4] * [An error that resulted in the failure of the Reporting Rule Engine][5] The Rule Engine Troubleshooting report excerpts below show only those columns required to illustrate each scenario. ### Passing Safety Rule Example {#passing-rule} In the example below, the Serious Unexpected rule for the FDA ICSR Rule Set results in a `Pass` as all of the Rule Parameters within the Rule (Serious, Report Type, Expected, and Product Registration Type) all evaluated as a `Pass` for all of the initial Eligible Products and Assessments. As a result, Vault sent a _Transmission_ to the corresponding Destination (the FDA).
|Destination|Rule Set|Rule Name|Parameter|Parameter Execution Status|Rule Execution Status|Eligible Product IDs|Eligible Assessment IDs|Passing Assessment IDs|Expectedness Calculation| |--- |--- |--- |--- |--- |--- |--- |--- |--- |--- | |FDA|FDA ICSR|Serious Unexpected|Serious|Pass| |V2D000000001001; V2D000000001002|V2J000000001001; V2J000000001002| | | |FDA|FDA ICSR|Serious Unexpected|Report Type|Pass| |V2D000000001001; V2D000000001002| V2J000000001001; V2J000000001002| | | |FDA|FDA ICSR|Serious Unexpected|Expected|Pass| |V2D000000001001; V2D000000001002|V2J000000001001; V2J000000001002| |V2J000000001001-Unexpected-V2J000000001001; V2J000000001002-Unexpected-V2J000000001002| |FDA|FDA ICSR|Serious Unexpected|Product Registration Type|Pass|Pass|V2D000000001001; V2D000000001002|V2J000000001001; V2J000000001002|V2J000000001001; V2J000000001002| ### Failing Safety Rule Example {#failing-rule} In this example for the Downgrade to Serious rule for the FDA ICSR Rule Set, the first two parameters (Serious and Report type) resulted in a Pass. However, the third parameter (Expected) resulted in a `Fail` because the Expectedness calculation was 'Unexpected'. Consequently, the Downgrade to Serious rule also resulted in a failure, so the _Case_ is not reportable to the corresponding Destination (FDA) for this Rule. |Destination|Rule Set|Rule Name|Parameter|Parameter Execution Status|Rule Execution Status|Eligible Product IDs|Eligible Assessment IDs|Passing Assessment IDs|Expectedness Calculation| |--- |--- |--- |--- |--- |--- |--- |--- |--- |--- | |FDA|FDA ICSR|Downgrade to Serious Expected|Serious|Pass| |V2D000000001001; V2D000000001002|V2J000000001001; V2J000000001002| | |FDA|FDA ICSR|Downgrade to Serious Expected|Report Type|Pass| |V2D000000001001; V2D000000001002|V2J000000001001; V2J000000001002| | |FDA|FDA ICSR|Downgrade to Serious Expected|Expected|Fail|Fail|V2D000000001001; V2D000000001002|V2J000000001001; V2J000000001002| |V2J000000001001-Unexpected-V2J000000001001; V2J000000001002-Unexpected-V2J000000001002| ### Safety Rule Parameter-Eliminated Products and Assessments Example {#eliminated} As the Safety Rule Engine evaluates each _Safety Rule Parameter_ of a Rule, if an Eligible Product or Assessment fails the parameter, it is eliminated from the list of Eligible Products or Assessments and does not appear in the following row (parameter) of the report. In the example below, Eligible Product ID `V2D000000003001` has two Eligible Assessment IDs (`V2J000000003001` and `V2J000000003002`). These passed the evaluation for the first parameter (Study Type) and so appear in the following row for the next parameter (Expected). The second parameter (Expectedness) was evaluated as follows: * Assessment ID `V2J000000003002` passed because its Expectedness calculation was evaluated as 'Unexpected' from the Case Assessment Expectedness ID `V3Y000000002003`. * Assessment ID `V2J000000003001` failed because its Expectedness calculation was evaluated as 'Expected' from the Case Assessment ID `V2J000000003001`. Consequently, Assessment ID `V2J000000003002` appears in the following row for the next parameter (Fatal), but Assessment ID `V2J000000003001`, having been eliminated, does not. For the third and final parameter (Fatal), the remaining Eligible Product and Assessment failed. Because no Eligible _Products_ or Assessments passed all three (3) parameters in this Rule, the Rule also failed (the **Rule Execution Status** column shows 'Fail' and the **Passing Assessment IDs** column is empty). |Destination|Rule Set|Rule Name|Parameter|Parameter Execution Status|Rule Execution Status|Eligible Product IDs|Eligible Assessment IDs|Passing Assessment IDs|Expectedness Calculation| |--- |--- |--- |--- |--- |--- |--- |--- |--- |--- | |FDA|FDA ICSR|SUSAR (Death)|Study Type|Pass| |V2D000000003001|V2J000000003001; V2J000000003002| | | |FDA|FDA ICSR|SUSAR (Death)|Expected| Pass | |V2D000000003001|V2J000000003001; V2J000000003002| |V2J000000003001-Expected-V2J000000003001; V2J000000003002-Unexpected-V3Y000000002003| |FDA|FDA ICSR|SUSAR (Death)|Fatal|Fail|Fail|V2D000000003001|V2J000000003002| | | ### A Rule Resulting in an Error {#error-rule} In this example for the Partner Distribution Rule Set, the Non-Serious rule caused an error because it did not specify a Due in Days parameter. |Destination Name|Rule Set|Rule Name|Parameter|Parameter Execution Status|Rule Execution Status|Eligible Product IDs|Eligible Assessment IDs|Passing Assessment IDs|Error Log| |--- |--- |--- |--- |--- |--- |--- |--- |--- |--- | |Partner 1|Partner Distribution Rule Set|Non-Serious|Product Registration Type|Pass|Pass|V2D000000001001|V2J000000001001|V2J000000001001| | |Partner 1|Partner Distribution Rule Set|Non-Serious|Product Registration Type|Error| |V2D000000001001|V2J000000001001| |[SafetyRuleLog] [RULESET:'Partner Distribution Rule Set', RULE:'Non-Serious', FUNCTION: 'NA']No due in days configured| ### Abnormal Termination of the Reporting Rule Engine {#abnormal-engine-termination} If the Rule Engine exited abnormally while evaluating the Rules for a _Case_ or _Localized Case_, the final row of the report includes the names of the Rule Set, Rule, and Function that caused the Rule Engine to fail, along with a description of the reason for the failure. Abnormal termination is often caused by custom _Safety Rule Sets_ with configuration errors. The information in the Rule Engine Troubleshooting report can help you locate and fix configuration errors in your custom _Safety Rule Sets_.
In the example below, the first Rule in the Custom ICSR Rule Set caused the Reporting Rule Engine to terminate because the Rule did not specify an Evaluation Function. |ID|Date Time|Transmission Type|Rule Set|Rule Name|Parameter|Parameter Execution Status|Rule Execution Status|Error Log| |--- |--- |--- |--- |--- |--- |--- |--- |--- | |1|2023-22-23T19:16:30.744|Submission|Custom ICSR Rule Set|Custom Rule|N/A|Error| |Unable to create rule engine on rule set: [SafetyRuleLog] [RULESET: 'Custom ICSR Rule Set', RULE: 'Custom Rule', FUNCTION: 'NA']Failed to load evaluation function| |Rule Engine exited abnormally: [SafetyRuleLog] [RULESET: 'Custom ICSR Rule Set', RULE: 'Custom Rule', FUNCTION: 'NA']Failed to load evaluation function| [0]: #error-log [1]: #passing-rule [2]: #failing-rule [3]: #eliminated [4]: #error-rule [5]: #abnormal-engine-termination