Learn how to use the Safety Rules Troubleshooting tool to diagnose and fix the Safety Rules in your Vault.
Note: Depending on your Admin’s configuration, object, field, and section labels, lifecycle states, and workflows may differ from the general information on this page. Refer to your organization’s business processes for guidance.
About the Safety Rules Troubleshooting Tool
The Vault Safety Reporting Rules Engine comes with Safety Rule Sets for certain standard agencies and enables Admins to create custom Safety Rule Sets. When the Evaluate Reporting Obligations action successfully runs for a Case, the Safety Rule Engine creates a Submission Rule Log. The Log details the list of considered Destinations and whether those Destinations passed or failed any of the Rules. However, if all of the Rules failed, this report does not indicate why. To investigate further, you can use the Safety Rules Troubleshooting tool to diagnose why specific Rules failed.
The sections below describe how Vault Admins can use the Safety Rules Troubleshooting tool to diagnose and fix issues with Safety Rules that are failing, especially custom Safety Rules.
Optional Configuration
To include Case Products with the Drug Role of Drug Not Administered when using the Safety Rules Troubleshooting tool, your Admin must have enabled Extend Definition of Suspect to Drug Not Administered. Otherwise, the Safety Rules Troubleshooting tool considers only Case Products with the Drug Role of Suspect or Interacting.
What to Look for When Evaluating the Reporting Obligations for a Case
After running the Evaluate Reporting Obligations action for a Case, if Vault did not generate a Transmission for a Destination when that was expected, examine the Submission Rule Log for the Case.
- If the Destination does not appear in the report, verify that the Case Product or Case Study is registered to that Destination or has a Reporting Family configured.
- If the Destination appears in the log but is marked as a failure (no Rules passed), then the issue is likely with the Rule. You can manually determine why the Rule did not pass, or use the Safety Rules Troubleshooting described in the following sections to diagnose the problem.
Obtain the Rule Engine Troubleshooting Report for a Case
The first step in diagnosing an issue with a Safety Rule is to obtain a Rule Engine Troubleshooting Report for a Case that did (or did not) generate a Transmission as expected.
- Go to Admin > Settings > Safety Rules Troubleshooting.
- Enter the ID of the Case you want to analyze.
You may need to use the All Actions > Edit Columns feature on your Cases view to add the ID column. - Select Run Rule Engine.
- A notification appears when the Rule Engine Troubleshooting Report is ready to download. You can download the report either from the Notifications area, or from the system-generated email notification. The file is named using the Case ID you entered, plus the date and time you ran the report.
Note: Once generated, a Rule Engine report is available for 24 hours, then it is deleted. If a generated Case report is no longer available and you do not have a downloaded copy, you can rerun the Troubleshooting tool for that Case.
Analyze the Rule Engine Troubleshooting Report for a Case
Once you have the Rule Engine Troubleshooting report for a Case, the next step is to analyze the results to determine why the Case did (or did not) generate a Transmission as expected.
The report is in CSV format. You can use a text or spreadsheet editor to examine the contents of the report.
The report contains a row for each Safety Rule Parameter that the Rule Engine evaluated for the Case and includes the following columns:
| Column Name | Description |
|---|---|
| ID | The ID number of the Rule Parameter evaluation. This is an incrementing number for each row, starting with one (1). |
| DateTime | The date and time that the Rule Parameter was evaluated. |
| Transmission Type | Whether the evaluation was for a Submission or a Distribution. |
| Reporting Scenario | Whether the evaluation was for General Reporting or a Cross Reporting scenario. |
| Registration (Cross Reporting) | If the evaluation was for Cross Reporting, the name and ID of the corresponding Product or Study Registration that triggered the evaluation. |
| Registration ID (Cross Reporting) | |
| Reporting Family (Distribution) | If the evaluation was for a Distribution, the name and ID of the Reporting Family that triggered the evaluation. |
| Reporting Family ID (Distribution) | |
| Destination Name | The name and ID of the Destination. |
| Destination ID | |
| RuleSet | The name of the Rule Set to which the evaluated Rule belongs. |
| RuleName | The name of the Rule that was evaluated. |
| Parameter |
The Rule Parameter that was evaluated.
Note: The Reporting Rule Engine evaluates the Product Registration Type parameter for every Rule in a Rule Set, even if the Rule does not specify this parameter. Consequently, the Rule Engine Troubleshooting report includes this result for every Rule in a Rule Set. |
| Parameter Execution Status |
The outcome of the Rule Parameter evaluation.
|
| Rule Execution Status | The outcome of the evaluation of the Rule ('Pass' or 'Fail'). |
| Eligible Product IDs 1 |
The ID list of all the Products that were eligible for evaluation for the Rule Parameter in that row.
In the event of multiple eligible Products where only a subset of the Products passes the parameter, the subset of passing Products is listed on the following row as Eligible Products. |
| Eligible Assessment IDs 1 |
The ID list of all the Assessments that were eligible for evaluation for the Rule Parameter in that row.
In the event of multiple eligible Assessments where only a subset of the Assessments passes the parameter, the subset of passing Assessments is listed on the following row as Eligible Assessments. |
| Passing Assessment IDs 1 | The ID list of all the Assessments that passed all of the Parameters within the Rule. |
| Error Log | The description of the error (if available) if the Rule Engine could not successfully evaluate the Rule Parameter. |
| ExpectednessCalculation 1 |
If the Rule uses the Expectedness parameter, this column contains the following details of the Expectedness calculation:
The report displays this information in the format
Note: Case Assessment IDs always start with |
| 1. Multiple values in these columns are separated by a semicolon. | |
Using this information, you can easily locate and fix problems with the Rules in a Rule Set.
For Cases where the Rule Engine completed successfully but did not generate a Transmission as expected, you can use the information in the report to determine why.
For each Rule within a Rule Set that the Rule Engine evaluates for a Case, you can see the initial list of Eligible Product and Assessment IDs in the first Rule Parameter row for that Rule. If the Rule Parameter is evaluated as a Pass, the Rule Engine likewise evaluates the next Rule Parameter in the Rule.
If all the Rule Parameters pass, the Rule also passes. This is indicated by the Rule Execution Status column showing as Pass and the Passing Assessment IDs column showing which of the Eligible Assessments passed the Rule Parameter.
If a Rule Parameter evaluates as a Fail, the Rule Engine marks the Rule as a Fail and moves on to the next Rule in the Rule Set.
The following section gives examples of the possible scenarios. These will help you when analyzing the Rule Engine Troubleshooting report for your Cases.
Rule Engine Troubleshooting Report Examples
The Rule Engine Troubleshooting report examples in this section illustrate how the results in the report appear for each of the following reporting scenarios:
- A Passing Safety Rule
- A Failing Safety Rule
- Products and Assessments eliminated by a Rule Parameter
- A Safety Rule that resulted in an error
- An error that resulted in the failure of the Reporting Rule Engine
The Rule Engine Troubleshooting report excerpts below show only those columns required to illustrate each scenario.
Passing Safety Rule Example
In the example below, the Serious Unexpected rule for the FDA ICSR Rule Set results in a Pass as all of the Rule Parameters within the Rule (Serious, Report Type, Expected, and Product Registration Type) all evaluated as a Pass for all of the initial Eligible Products and Assessments. As a result, Vault sent a Transmission to the corresponding Destination (the FDA).
Note: For the Expectedness parameter, the Expectedness Calculation column contains the details of the Expectedness
calculation. For both of the eligible assessments (V2J000000001001 and V2J000000001002), the outcome was
‘Unexpected’, with their source Assessment being V2J000000001001 and V2J000000001002 respectively.
| Destination | Rule Set | Rule Name | Parameter | Parameter Execution Status | Rule Execution Status | Eligible Product IDs | Eligible Assessment IDs | Passing Assessment IDs | Expectedness Calculation |
|---|---|---|---|---|---|---|---|---|---|
| FDA | FDA ICSR | Serious Unexpected | Serious | Pass | V2D000000001001; V2D000000001002 | V2J000000001001; V2J000000001002 | |||
| FDA | FDA ICSR | Serious Unexpected | Report Type | Pass | V2D000000001001; V2D000000001002 | V2J000000001001; V2J000000001002 | |||
| FDA | FDA ICSR | Serious Unexpected | Expected | Pass | V2D000000001001; V2D000000001002 | V2J000000001001; V2J000000001002 | V2J000000001001-Unexpected-V2J000000001001; V2J000000001002-Unexpected-V2J000000001002 | ||
| FDA | FDA ICSR | Serious Unexpected | Product Registration Type | Pass | Pass | V2D000000001001; V2D000000001002 | V2J000000001001; V2J000000001002 | V2J000000001001; V2J000000001002 |
Failing Safety Rule Example
In this example for the Downgrade to Serious rule for the FDA ICSR Rule Set, the first two (2) parameters (Serious and Report type) resulted in a Pass. However, the third parameter (Expected) resulted in a Fail because the Expectedness calculation was ‘Unexpected’. Consequently, the Downgrade to Serious rule also resulted in a failure, so the Case is not reportable to the corresponding Destination (FDA) for this Rule.
| Destination | Rule Set | Rule Name | Parameter | Parameter Execution Status | Rule Execution Status | Eligible Product IDs | Eligible Assessment IDs | Passing Assessment IDs | Expectedness Calculation |
|---|---|---|---|---|---|---|---|---|---|
| FDA | FDA ICSR | Downgrade to Serious Expected | Serious | Pass | V2D000000001001; V2D000000001002 | V2J000000001001; V2J000000001002 | |||
| FDA | FDA ICSR | Downgrade to Serious Expected | Report Type | Pass | V2D000000001001; V2D000000001002 | V2J000000001001; V2J000000001002 | |||
| FDA | FDA ICSR | Downgrade to Serious Expected | Expected | Fail | Fail | V2D000000001001; V2D000000001002 | V2J000000001001; V2J000000001002 | V2J000000001001-Unexpected-V2J000000001001; V2J000000001002-Unexpected-V2J000000001002 |
Rule Parameter-Eliminated Products and Assessments Example
As the Safety Rule Engine evaluates each Rule Parameter of a Rule, if an Eligible Product or Assessment fails the parameter, it is eliminated from the list of Eligible Products or Assessments and does not appear in the following row (parameter) of the report.
In the example below, Eligible Product ID V2D000000003001 has two (2) Eligible Assessment IDs (V2J000000003001 and V2J000000003002). These passed the evaluation for the first parameter (Study Type) and so appear in the following row for the next parameter (Expected).
The second parameter (Expectedness) was evaluated as follows:
- Assessment ID
V2J000000003002passed because its Expectedness calculation was evaluated as ‘Unexpected’ from the Case Assessment Expectedness IDV3Y000000002003. - Assessment ID
V2J000000003001failed because its Expectedness calculation was evaluated as ‘Expected’ from the Case Assessment IDV2J000000003001.
Consequently, Assessment ID V2J000000003002 appears in the following row for the next parameter (Fatal), but Assessment ID V2J000000003001, having been eliminated, does not.
For the third and final parameter (Fatal), the remaining Eligible Product and Assessment failed.
Because no Eligible Products or Assessments passed all three (3) parameters in this Rule, the Rule also failed (the Rule Execution Status column shows ‘Fail’ and the Passing Assessment IDs column is empty).
| Destination | Rule Set | Rule Name | Parameter | Parameter Execution Status | Rule Execution Status | Eligible Product IDs | Eligible Assessment IDs | Passing Assessment IDs | Expectedness Calculation |
|---|---|---|---|---|---|---|---|---|---|
| FDA | FDA ICSR | SUSAR (Death) | Study Type | Pass | V2D000000003001 | V2J000000003001; V2J000000003002 | |||
| FDA | FDA ICSR | SUSAR (Death) | Expected | Pass | V2D000000003001 | V2J000000003001; V2J000000003002 | V2J000000003001-Expected-V2J000000003001; V2J000000003002-Unexpected-V3Y000000002003 | ||
| FDA | FDA ICSR | SUSAR (Death) | Fatal | Fail | Fail | V2D000000003001 | V2J000000003002 |
A Rule Resulting in an Error
In this example for the Partner Distribution Rule Set, the Non-Serious rule caused an error because it did not specify a Due in Days parameter.
| Destination Name | Rule Set | Rule Name | Parameter | Parameter Execution Status | Rule Execution Status | Eligible Product IDs | Eligible Assessment IDs | Passing Assessment IDs | Error Log |
|---|---|---|---|---|---|---|---|---|---|
| Partner 1 | Partner Distribution Rule Set | Non-Serious | Product Registration Type | Pass | Pass | V2D000000001001 | V2J000000001001 | V2J000000001001 | |
| Partner 1 | Partner Distribution Rule Set | Non-Serious | Product Registration Type | Error | V2D000000001001 | V2J000000001001 | [SafetyRuleLog] [RULESET:’Partner Distribution Rule Set’, RULE:’Non-Serious’, FUNCTION: ‘NA’]No due in days configured |
Abnormal Termination of the Reporting Rule Engine
If the Rule Engine exited abnormally while evaluating the Rules for a Case, the final row of the report includes the names of the Rule Set, Rule, and Function that caused the Rule Engine to fail, along with a description of the reason for the failure.
Abnormal termination is often be caused by Custom Safety Rule Sets with configuration errors. The information in the Rule Engine Troubleshooting report can help you locate and fix configuration errors in your custom Rule Sets.
Note: You cannot edit the standard Safety Rule Sets.
In the example below, the first Rule in the Custom ICSR Rule Set caused the Reporting Rule Engine to terminate because the Rule did not specify an Evaluation Function.
| ID | Date Time | Transmission Type | Rule Set | Rule Name | Parameter | Parameter Execution Status | Rule Execution Status | Error Log |
|---|---|---|---|---|---|---|---|---|
| 1 | 2023-22-23T19:16:30.744 | Submission | Custom ICSR Rule Set | Custom Rule | N/A | Error | Unable to create rule engine on rule set: [SafetyRuleLog] [RULESET: ‘Custom ICSR Rule Set’, RULE: ‘Custom Rule’, FUNCTION: ‘NA’]Failed to load evaluation function | |
| Rule Engine exited abnormally: [SafetyRuleLog] [RULESET: ‘Custom ICSR Rule Set’, RULE: ‘Custom Rule’, FUNCTION: ‘NA’]Failed to load evaluation function |