A Routing Architecture for Satellite Networks

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶

This Internet-Draft will expire on 3 March 2025.¶

Copyright Notice

Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶

Table of Contents

1. Introduction

Satellite networks present some interesting challenges for packet networking. The entire topology is continually in motion, with links far less reliable than what is common in terrestrial networks. Some changes to link connectivity can be anticipated due to orbital dynamics.¶

This document proposes a scalable routing architecture for satellite networks based on existing routing protocols and mechanisms, enhanced with scheduled link connectivity change information. This document proposes no protocol changes.¶

Large-scale satellite networks are being deployed, presenting an unforeseen application for conventional routing protocols. The high rate of intentional topological change and the extreme scale are unprecedented in terrestrial networking. Links between satellites can utilize free-space optics technology that allows liberal connectivity. Still, there are limitations due to the range of the links and conjunction with the sun, resulting in links that are far less reliable than network designers are used to. In addition, links can change their endpoints dynamically, resulting in structural changes to the topology.¶

Current satellite networks are proprietary and little information is generally available for analysis and discussion. This document is based on what is currently accessible.¶

This document proposes one approach to provide a routing architecture for such networks utilizing current standards-based routing technology and providing a solution for the scalability of the network while incorporating the rapid rate of topological change. This document intends to provide some initial guidance for satellite network operators, but without specific details, this document can only provide the basis for a more complete analysis and design.¶

This document presents the author's view and is neither the product of the IETF nor a consensus view of the community.¶

2. Overview

  User           +-----------------+    Local
  Stations   --- |   Satellites    |--- Gateway --- Internet
                 +-----------------+

                Figure 1: Overall network architecture

3. Forwarding Plane

The end goal of a network is to deliver traffic. In a satellite network where the topology is in a continual state of flux and the user stations frequently change their association with the satellites, having a highly flexible and adaptive forwarding plane is essential. Toward this end, we propose to use MPLS as the fundamental forwarding plane architecture [RFC3031]. Specifically, we propose to use a Segment Routing (SR) [RFC8402] based approach with an MPLS data plane [RFC8660], where each satellite is assigned a node Segment Identifier (SID). This allows the architecture to support both IPv4 and IPv6 concurrently. A path through the network can then be expressed as a label stack of node SIDs. IP forwarding is not used within the internals of the satellite network, although each satellite may be assigned an IP address for management purposes. Existing techniques may be used to limit the size of the SR label stack so that it only contains the significant waypoints along the path [Giorgetti]. The label stack operates as a loose source route through the network. If there is an unexpected topology change in the network, the IGP will compute a new path to the next waypoint, allowing packet delivery despite ISL failures. While the IGP is converging, there may be micro-loops in the topology. These can be avoided by using TI-LFA alternate paths [I-D.ietf-rtgwg-segment-routing-ti-lfa], or traffic will loop until discarded based on its TTL.¶

We assume that there is a link-layer mechanism for a user station to associate with a satellite. User stations will have an IP address assigned from a prefix managed by its local gateway. The mechanisms for this assignment and its communication to the end station are not discussed herein but might be similar to DHCP [RFC2131]. User station IP addresses change infrequently and do not reflect their association with their first-hop satellite. Gateways and their supporting terrestrial networks advertise prefixes covering all its local user stations into the global Internet.¶

User stations may be assigned a node SID, in which case MPLS forwarding can be used for all hops to the user station. Alternatively, if the user station does not have a node SID, then the last hop from the satellite to the end station can be performed based on the destination IP address of the packet. This does not require a full longest-prefix-match lookup as the IP address is merely a unique identifier at this point.¶

Similarly, gateways may be assigned a node SID. A possible optimization is that a single SID value be assigned as a global constant to always direct traffic to the topologically closest gateway. If traffic engineering is required for traffic that is flowing to a gateway, a specific path may be encoded in a label stack that is attached to the packet by the user station or by the first-hop satellite.¶

Gateways can also perform traffic engineering using different paths and label stacks for separate traffic flows. Routing a single traffic flow across multiple paths has proven to cause performance issues with transport protocols, so that approach is not recommended. Traffic engineering is discussed further in Section 6.¶

4. IGP Suitability and Scalability

As discussed in Section 2.3, IS-IS is architecturally the best fit for satellite networks, but does require some novel approaches to achieve the scalability goals for a satellite network. In particular, we propose that all nodes in the network be L1L2 so that local routing is done based on L1 information and then global routing is done based on L2 information.¶

IS-IS has the interesting property that it does not require interface addresses. This feature is commonly known as 'unnumbered interfaces'. This is particularly helpful in satellite topologies because it implies that ISLs may be used flexibly. Sometimes an interface might be used as an L1 link to another satellite and a few orbits later it might be used as an L1L2 link to a completely different satellite without any reconfiguration or renumbering.¶

Scalability for IS-IS can be achieved through a proposal known as Area Proxy [I-D.ietf-lsr-isis-area-proxy]. With this proposal, all nodes in an L1 area combine their information into a single L2 Link State Protocol Data Unit (LSP). This implies that the size of the L1 Link State Database (LSDB) scales as the number of nodes in the L1 area and the size of the L2 LSDB scales with the number of L1 areas.¶

With Area Proxy, topological changes within an L1 area will not be visible to other areas, so the overhead of link state changes will be greatly reduced.¶

The Area Proxy proposal also includes the concept of an Area SID. This is useful because it allows traffic engineering to construct a path that traverses areas with a minimal number of label stack entries.¶

Suppose, for example, that a network has 1,000 L1 areas, each with 1,000 satellites. This would then mean that the network supports 1,000,000 satellites, but only requires 1,000 entries in its L1 LSDB and 1,000 entries in its L2 LSDB; numbers that are easily achievable today. The resulting MPLS label table would contain 1,000 node SIDs from the L1 (and L2) LSDB and 1,000 area SIDs from the L2 LSDB. If each satellite advertises an IP address for management purposes, then the IP routing table would have 1,000 entries for the L1 management addresses and 1,000 area proxy addresses from L2.¶

In this proposal, IS-IS does not carry IP routes other than those in the satellite topology. In particular, there are no IP routes for user stations or the remainder of the Internet.¶

5. Stripes and Areas

A significant problem with any link state routing protocol is that of area partition. While there have been many proposals for automatic partition repair, none has seen notable production deployment. It seems best to avoid this issue and ensure areas have an extremely low probability of partitioning.¶

As discussed above, intra-orbit ISLs are assumed to have higher reliability and persistence than inter-orbit ISLs. However, even intra-orbit ISLs are not sufficiently reliable to avoid partition issues. Therefore, we propose to group a small number of adjacent orbits as an IS-IS L1 area, called a stripe. We assume that for any given reliability requirement, there is a small number of orbits that can be used to form a stripe that satisfies the reliability requirement.¶

Stripes are connected to other adjacent stripes using the same ISL mechanism, forming the L2 topology of the network. Each stripe should have multiple L2 connections and never become partitioned from the remainder of the network.¶

By using a stripe as an L1 area, in conjunction with Area Proxy, the overhead of the architecture is greatly reduced. Each stripe contributes a single LSP to the L2 LSDB, completely hiding all the details about the satellites within the stripe. The resulting architecture scales proportionately to the number of stripes required, not the number of satellites.¶

Groups of MEO and GEO satellites with interconnecting ISLs can also form an IS-IS L1L2 area. Satellites that lack intra-constellation ISLs are better as independent L2 nodes.¶

6. Traffic Forwarding and Traffic Engineering

Forwarding in this architecture is straightforward. A path from a gateway to a user station on the same stripe only requires a single node SID for the satellite that provides the downlink to the user station.¶

             \
 Gateway -->  X
               \
                \
                 X
                  \
                   \
                    X ---> x  User Station
                     \

                         Figure 2: On-stripe forwarding

Similarly, a user station returning a packet to a gateway need only provide a gateway node SID.¶

For off-stripe forwarding, the situation is a bit more complex. A gateway would need to provide the area SID of the final stripe on the path plus the node SID of the downlink satellite. For return traffic, user stations or first-hop satellites would want to provide the area SID of the stripe that contains the satellite that provides access to the gateway as well as the gateway SID.¶

 Source S
    |
    |
    V
 Internet
    |
    |
    V          \
 Gateway L -->  X
                 \
                  \     \
                   X --- X
                    \     \
                           \     \    Area A
                            X --- X
                             \     \
                                    \
                                     U ---> D   User Station
                                      \

                         Figure 3: Off-stripe forwarding

As an example, consider a packet from an Internet source S to a user station D. A local gateway L has injected a prefix covering D into BGP and advertised it globally. The packet is forwarded to L using IP forwarding. When L receives the packet, it performs a lookup in a pre-computed forwarding table. This contains a SID list for the user station that has already been converted into a label stack. Suppose the user station is currently associated with a different stripe so that the label stack will contain an area label A and a label U for the satellite associated with the user station, resulting in a label stack (A, U).¶

The local gateway forwards this into the satellite network. The first-hop satellite now forwards based on the area label A at the top of the stack. All area labels are propagated as part of the L2 topology. This forwarding continues until the packet reaches a satellite adjacent to the destination area. That satellite pops label A, removing that label and forwarding the packet into the destination area.¶

The packet is now forwarded based on the remaining label U, which was propagated as part of the L1 topology. The last satellite forwards the packet based on the destination address D and forwards the packet to the user station.¶

The return case is similar. The label stack, in this case, consists of a label for the local gateway's stripe/area, A', and the label for the local gateway, L, resulting in the stack (A', L). The forwarding mechanisms are similar to the previous case.¶

Very frequently, access networks congest due to oversubscription and the economics of access. Network operators can use traffic engineering to ensure that they get higher efficiency out of their networks by utilizing all available paths and capacity near any congestion points. In this particular case, the gateway will have information about all of the traffic it is generating and can use all of the possible paths through the network in its topological neighborhood. Since we're already using SR, this is easily done just by adding more explicit SIDs to the label stack. These can be additional area SIDs, node SIDs, or adjacency SIDs. Path computation can be performed by Path Computation Elements (PCE) [RFC4655].¶

Each gateway or its PCE will need topological information from the areas it will route through. It can do this by participating in the IGP directly, via a tunnel, or another delivery mechanism such as BGP-LS [RFC9552]. User stations do not participate in the IGP.¶

Traffic engineering for packets flowing into a gateway can also be provided by an explicit SR path. This can help ensure that ISLs near the gateway do not congest with traffic for the gateway. These paths can be computed by the gateway or PCE and then distributed to the first-hop satellite or user station, which would apply them to traffic. The delivery mechanism is outside of the scope of this document.¶

7. Scheduling

The most significant difference between terrestrial and satellite networks from a routing perspective is that some of the topological changes that will happen to the network can be anticipated and computed. Both link and node changes will affect the topology and the network should react smoothly and predictably.¶

The management plane is responsible for providing information about scheduled topological changes. The exact details of how the information is disseminated are outside of the scope of this document but could be done through a YANG model [I-D.ietf-tvr-schedule-yang]. Scheduling information needs to be accessible to all of the nodes that will make routing decisions based on the topological changes in the schedule, so data about an L1 topological change will need to be circulated to all nodes in the L1 area and information about L2 changes will need to propagate to all L2 nodes, plus the gateways and PCEs that carry the related topological information.¶

There is very little that the network should do in response to a topological addition. A link coming up or a node joining the topology should not have any functional change until the change is proven to be fully operational based on the usual IS-IS liveness mechanisms. Nodes may pre-compute their routing table changes but should not install them before all relevant adjacencies are received. The benefits of this pre-computation appear to be very small. Gateways and PCEs may also choose to pre-compute paths based on these changes, but should not install paths using the new parts of the topology until they are confirmed to be operational. If some path pre-installation is performed, gateways and PCEs must be prepared for the situation where the topology fails to become operational and may need to take alternate steps instead, such as reverting any related pre-installed paths.¶

The network may choose not to pre-install or pre-compute routes in reaction to topological additions, at a small cost of some operational efficiency.¶

Topological deletions are an entirely different matter. If a link or node is to be removed from the topology, the network should act before the anticipated change to route traffic around the expected topological loss. Specifically, at some point before the topology change, the affected links should be set to a high metric to direct traffic to alternate paths. This is a common operational procedure in existing networks when links are taken out of service, such as when proactive maintenance needs to be performed. This type of change does require some time to propagate through the network, so the metric change should be initiated far enough in advance that the network converges before the actual topological change. Gateways and PCEs should also update paths around the topology change and install these changes before the topology change occurs. The time necessary for both IGP and path changes will vary depending on the exact network and configuration.¶

Strictly speaking, changing to a high metric should not be necessary. It should be possible for each router to exclude the link and recompute paths. However, it seems safer to change the metric and use the IGP methods for indicating a topology change, as this can help avoid issues with incomplete information dissemination and synchronization.¶

8. Future Work

This architecture needs to be validated by satellite operators, both via simulation and operational deployment. Meaningful simulation hinges on the exact statistics of ISL connectivity, and that information is not publicly available currently.¶

Current available information about ISLs indicates that links are mechanically steered and will need to track the opposite end of the link continually. The angles and distances that can be practically supported are unknown, as are any limitations about the rate of change.¶

It is expected that intra-orbit and inter-orbit ISL links will have very different properties. Intra-orbit links should be much more stable, but still far less stable than terrestrial links. Inter-orbit links will be less stable. Links between satellites that are roughly parallel should be possible, but will likely have a limited duration. Two orbits may be roughly orthogonal, resulting in a limited potential for connectivity. Finally, in some topologies there may be parallel orbits where the satellites move in opposite directions, giving a relative speed between satellites around 34,000mph (55,000kph). Links in this situation may not be possible or may be so short-lived as to be impractical.¶

The key question to address is whether the parameters of a given network can yield a stripe assignment that produces stable, connected areas that work within the scaling bounds of the IGP. If links are very stable, a stripe could be just a few parallel orbits, with only a few hundred satellites. However, if links are unstable, a stripe might have to encompass dozens of orbits and thousands of satellites, which might be beyond the scaling limitations of a given IGP's implementation.¶

9. Deployment Considerations

The network behind a gateway is expected to be a normal terrestrial network. Conventional routing architectural principles apply. An obvious approach would be to extend IS-IS to the terrestrial network, applying L1 areas as necessary for scalability.¶

The terrestrial network may have one or more BGP connections to the broader Internet. Prefixes for user stations should be advertised to the Internet near the associated gateway. If gateways are not interconnected by the terrestrial network, then it may be advisable to use one autonomous system per gateway as it might simplify the external perception of the network and subsequent policy considerations. Otherwise, one autonomous system may be used for the entire terrestrial network.¶

10. Security Considerations

This document discusses one possible routing architecture for satellite networks. It proposes no new protocols or mechanisms and thus has no new security impact. Security for IS-IS is provided by [RFC5304] and [RFC5310].¶

User stations will interact directly with satellites, potentially using proprietary mechanisms, and under the control of the satellite operator who is responsible for the security of the user station.¶

11. Acknowledgements

The author would like to thank Dino Farinacci, Olivier De jonckere, Eliot Lear, Adrian Farrel, Acee Lindem, Erik Kline, Sue Hares, Joel Halpern, Marc Blanchet, and Dean Bogdanovic for their comments.¶

12. IANA Considerations

This document makes no requests for IANA.¶

Author's Address