| Internet-Draft | Network Fault Terminology | November 2024 |
| Davis, et al. | Expires 7 May 2025 | [Page] |
This document sets out some terms that are fundamental to a common understanding of network fault and problem management within the IETF.¶
The purpose of this document is to bring clarity to discussions and other work related to network fault and problem management in particular YANG models and management protocols that report, make visible, or manage network faults and problems.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 7 May 2025.¶
Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
Successful operation of large or busy networks depends on network management. Network management comprises a virtuous circle of network control, network observability, network analytics, network assurance, and back to network control. Network fault and problem management is an important aspect of network management and control solutions. It deals with the reporting, inspection, correlation, and management of events within the network. The intention is to focus on those events have a negative effect on the network's ability to forward traffic in an optimal way. Fault and problem management extends to include actions taken to determine the causes of problems and to work toward recovery of optimal network behavior.¶
A number of work efforts within the IETF seek to provide components of a fault management system, such as YANG models or management protocols. It is important that a common terminology is used so that there is a clear understanding of how the elements of the management and control solutions fit together, and how faults and problems will be handled.¶
This document sets out some terms that are fundamental to a common understanding of network fault and problem management. While "faults" and "problems" are concepts that apply at all levels of technology in the Internet, the scope of this document is restricted to the network layer and below, hence this document is specifically about "network fault and problem management."¶
The terms defined in this document are principally intended for consistent use within the IETF. Where similar concepts are described in other bodies, an attempt has been made to harmonize with those other descriptions, but there is care needed where terms are not used consistently between bodies or where terms are applied outside the network layer. If other bodies find the terminology defined in this document useful, they are free to use it.¶
Note that some useful terms are defined in [RFC3877] and [RFC8632]. The definitions in this document are informed by those documents, but they are not dependent on that prior work.¶
This section contains key terms. It is split into three subsections.¶
Section 2.1 contains terms that help set the context for the incident and fault management systems.¶
Section 2.2 includes specific and detailed core terms that will be used in other documents that describe elements of the fault management systems.¶
Section 2.3 provides two further terms that may be helpful.¶
This section includes some terminology that helps describe the context for the rest of this work. The definitions are deliberately kept relatively terse. Further documents may expand on these terms without loss of specificity.¶
This is defined in [RFC9232] and describes the process of collecting operational network data categorized into network planes. Data collected through the Network Telemetry process does not contain network or device configuration information. Nor does it contain any data related to service definition (i.e., intent per [RFC9315].¶
This is the process of keeping a continuous record of a resource, function, or connectivity service. The term 'monitoring' focuses on one single dimension and measurement in dimensional data modelling ([wikipedia]). This could be a measurement of the service state, a network function measurement, or the state of a network function of a resource as an example.¶
Network analytics is the process of deriving analytical insights into or from operational network data. A process, could be a software, a system, or a human that analyzes operational data and outputs new analytical data, ideally metadata (a symptom for example), which is related to the operational data.¶
This is the enablement of network behavioral assessment through analysis of observed operational network data (logs, alarms, traces, etc.) with the aim of detecting symptoms of, and to identify, anomalies and their causes. Network observability begins with information gathered using conventional network monitoring tools and enriched with other operational data (e.g., change records). The expected outcome of the observability processes is identification and analysis of deviations in observed state versus the expected state of a network.¶
Thus, there is a cascaded sequence where:¶
The terms are presented below in an order that is intended to flow such that it is possible to gain understanding reading top to bottom. The figures and explanations in Section 3 may aid understanding the terms set out here.¶
An assembly of components that exhibits some behavior.¶
A system that includes elements that are beyond the scope of the control system.¶
An external system that is of interest to and is influenced by the control system. Viewed as a collection of resources.¶
A component, commodity, service, or capability that can be used to support the delivery of some function. Such function can be local (e.g., within a node) or distributed (e.g., at the level of a network).¶
Resource is a recursive concept so that a resource may be a collection of other resources (for example, a network node is a collection of interfaces).¶
Observable or measurable aspect or behavior associated with a resource.¶
A measurable amount which may be in the form of an integer (e.g., a count) or on a continuous variable (e.g., an analogue measurement) associated with a characteristic.¶
The interpretation of the values of a set of characteristics of the resource (with respect to working order or some other aspect relevant to the resource purpose/application).¶
In the context of monitoring network resources, the variation in values associated with a characteristic of a resource at a specific time or over time.¶
To notice the presence of something (state, change, activity, form, etc.).¶
Hence also to notice a change (from the perspective of the viewer).¶
The change in value (of a characteristic of a resource) at a measurable instant in time (i.e., the period is negligible).¶
Compared with a change, which is over a period of time, an event happens at a measurable instant.¶
A particular condition that something (e.g., a resource) is in (at a specific time).¶
While a state may be observed at a specific moment in time, it is actually achieved by summarizing the measurement over time in a process sometimes called state compression.¶
Consideration of an event, state, or value (through the application of policy, relative to a specific viewpoint/perspective, intent, and in relation to other events, states, and values) to determine whether it is of note to the control system.¶
A relevant event.¶
A particular relevant change.¶
An occurrence that is not desired/required (as it may be indicative of a current or future undesired State). A fault can generally be associated with a known cause. See [RFC8632] for a more detailed discussion of network faults.¶
A state regarded as undesirable and may require remedial action. A problem cannot necessarily be associated with a cause. The resolution of a problem does not necessarily act on the thing that has the problem.¶
Note that there is a historic aspect to the concept of a problem. The current state may be operational, but there could have been a failure that is unexplained, and the fact of that unexplained recent failure is a problem.¶
Note that whilst a problem is unresolved it may continue to require attention. A record of resolved problems may be maintained in a log.¶
Note that there may be a state which is considered to be a problem from several perspectives (e.g., a loss of light state may cause multiple services to fail). A state change (so that the light recovers) may cause the problem to be resolved from one perspective (the services are operational once more), but may leave the problem as unresolved (because the loss of light has not been explained). There could be a further development (the reason for the temporary loss of light is traced to a microbend in the fiber that is repaired) resulting in that unresolved problem is now resolved. But this leaves a further problem still unresolved (why did the microbend occur in the first place?).¶
A network incident is an undesired occurrence such as an unexpected interruption of a network service, degradation of the quality of a network service, or the below-target health of a network service. An incident results from one or more problems, and a problem may give rise to or contribute to one or more incidents. Greater discussion of network incidents, including incident management, can be found in [I-D.ietf-nmop-network-incident-yang].¶
A (network) anomaly is an unusual or unexpected event or pattern in network data in the forwarding plane, control plane, or management plane that deviates from the normal, expected behavior. See [I-D.ietf-nmop-network-anomaly-architecture] for more details.¶
An observable characteristic/state/condition considered as an indication of a problem or potential problem.¶
The events (detected or otherwise) that gave rise to a fault/problem.¶
The process of considering multiple problems, symptoms, and their causes to determine the underlying causes.¶
The indication of a fault.¶
Per [RFC8632], an alarm signifies an undesirable state in a resource that requires corrective action. From a management point of view, an alarm can be as a state in its own right and the transition to this state is a fault and may result in an alert being issued. The receipt of this alert may give rise to a continuous indication (to a human operator) highlighting the potential or actual presence of a problem.¶
Two other terms may be helpful:¶
The relationship between system, resource, and characteristics is shown in Figure 1. A Controlled External System is comprised of Resources, and Resources have Characteristics.¶
Characteristics
^
|
Resource
^
|
Controlled External System
^
|
External System
The Value of a Characteristic of a Resource is expected to change over time. Specific changes in value may be noticed at a specific time (as digital changes), Detected, and treated as Events. This is shown on the left of Figure 2.¶
The center of Figure 2 shows how the Value of a Characteristic may change over time. The value may be Detected at specific times or periodically and give rise to States (and consequently State changes).¶
In practice, the Characteristic may vary in an analog manner over time as shown on the right hand side of Figure 2. The Value can be read or reported (i.e., Detected) periodically leading to Analogue Values that may be deemed Relevant Values, or may be evaluated over time as shown in Figure 6.¶
Event State Value
^ ^ ^
Detect : Detect : Detect :
: : :
^ ^ ^ ^ ^ /\
: : : : : / \
: : : : : /\ / \
__ __ _____ / \/
| | | | /\/
__| |__ ____| |____ /
Change at a time Change over time Change over time
Figure 3 shows the workflow progress for Events. As noted above, an Event is a Change in the Value of a Characteristic at a time. The Event may be evaluated (considering policy, relative to a specific viewpoint/perspective, with a view to intent, and in relation to other Events, States, and Values) to determine if it is an Occurrence and possibly to indicate a change of State. An Occurrence may be undesirable (a Fault) and that can cause an Alert to be generated, may be evidence of a Problem and could directly indicate a Cause.¶
Alert- - - - > Alarm
^
|
| -----> Cause
| |
|----------> Problem
|
|
Fault
^
|
|
|
Occurrence
^
|
|----------> State
|
|
Event
Parallel to the workflow for Events, Figure 4 shows the workflow progress for States. As shown in Figure 2, Change noted at a particular time gives rise to State. The State may be deemed relevant (via Relevance) considering policy, relative to a specific viewpoint/perspective, with a view to intent, and in relation to other Events, States, and Values. A Relevant State may be deemed a Problem, or may indicate a Problem.¶
Problems may be considered as Symptoms and may map directly or indirectly to Causes. An Alarm may be raised as the result of a Problem. An Incident results from one or more Problems.¶
Alarm
^
| ------> Incident
| |
| | ---> Cause
| | |
Problem---------> Symptom
^
|
|
|
Relevant State
^
|
|
|
State
Figure 5 shows how Faults and Problems may be consolidated to determine the Causes.¶
A Cause can be indicated by or determined from Faults, Problems and Symptoms. It may be that one Cause points to another, and can also be considered as a Symptom. The determination of Causes can consider multiple inputs. An Incident results from one or more Problems.¶
---------
------------- | |
| ----------> | Symptom |
| | | |
| | ---------
v | ^
--------- |
------->| Cause |<--------- |
| --------- | |
| ^ | | |
| | | | |
| --- | |
| | |
--------- --------- ----------
| Fault |------------------->| Problem |------->| Incident |
--------- --------- ----------
The final figure in this section (Figure 6) shows how thresholds are important in the consideration of Analogue Values and Events. The use of threshold-driven events and states (and the alerts that they might give rise to) must be treated with caution to dampen any "flapping" (so that consistent states may be observed) and to avoid overwhelming management processes or systems. Analogue Values may be read or notified from the Resource and could transition a threshold, be deemed Relevant Values, or evaluated over time. Events may be counted, and the Count may cross a threshold or reach a Relevant Value.¶
The Threshold Process may be implementation-specific and subject to policies. When a threshold is crossed and any other conditions are matched, an Event may be determined, and treated like any other Event.¶
Occurrence
^
|
|---------------------> State
|
| -------
|------>| Count |-------------------------> Relevant Value
| ------- | ^
| | | |
| | | |
| | v |
| | ----------- ----------------
Event | | Evaluated | | |
^ | | over time |<--------| Analogue Value |
| v ----------- | |
| ----------- | | |
| | Threshold | | | |
|<----| Process |<------ | |
| | |<----------------------| |
| ----------- ----------------
| ^
| |
| Detect Detect |
| |
Change at a Time Change over Time
This document specifies terminology and has no direct effect on the security of implementations or deployments. However, protocol solutions and management models need to be aware of several aspects:¶
The exposure of information pertaining to faults may make available knowledge of the internal workings of a network (in particular its vulnerabilities) that may be of use to an attacker.¶
Systems that generate management information (messages, notifications, etc.) when faults occur, may be attacked by causing them to generate so much information that the management system is swamped an unable to properly manage the network.¶
Reporting false information about faults (or masking reports of faults) may cause the management system to function incorrectly.¶
In general, Fault Management should not expose information about end-user activities or user data. The main privacy concern is for a network operator to keep control of all information about faults to protect their privacy and the details of how they operate their network.¶
This document makes no requests for IANA action.¶
The authors would like to thank Med Boucadair, Wanting Du, Joe Clarke, Javier Antich, Benoit Claise, Christopher Janz, Sherif Mostafa, and Kristian Larsson for their helpful comments.¶
Special thanks to the team that met at a side meeting at IETF-120 to discuss some of the thorny issues:¶