Higher-DAL UDP Network Stack

From DDCIDeos
Jump to navigationJump to search


Higher-DAL Network Stack. Project is also called "UDP Stack". Long-term goal is our own high-DAL stack with UDP... and TSN... and TCP, etc.


Overview:

The High-DAL Network Stack (also sometimes called the "UDP Stack") has two goals:

  1. High design assurance level (likely DAL-A) to permit use of networking in high-DAL applications.
    • Current lwip is DAL-E -- it's designed for use at development time.
  2. Improved performance via use of smart networking hardware.
    • The smart networking hardware will inspect inbound frames and route them to designated per-application queues ("rx flow classification").
    • Current lwip uses a client/server model that does inter-process API calls via Deos mailboxes. The need to schedule a network server (lwip) in a window causes surprising latencies: the time to complete an API call (eg a socket call) can be unexpectedly long due to scheduling across windows. Smart networking hardware will address this.
    • If the high-DAL stack is to use smart networking hardware on a given platform, then a high-DAL network driver that utilizes the particular complex smart hardware on that platform must be developed (implication: paid for).
    • If a desired platform does not have sufficiently powerful smart networking hardware, it will be possible to use a software-only frame classifier that (a) must run on a dedicated core, and (b) will have reduced performance in comparison to the smart networking hardware.

The High-DAL Network stack will initially only support UDP, not TCP.

  • Adding TCP will add complexity to the stack architecture and a large amount of additional code that must be verified.

Over the long term, the High-DAL Network stack will also support TSN.

For more introductory info, see High DAL Network Stack Overview for Geekfest 2024


Planned releases:

  1. UDP Software-only Demo (2025q1)
    • Runs on "dumb hardware"
    • Components:
      1. slimstack -- streamlined network stack running in app's partition
      2. software-only classifier
      3. shim software-only frame-routing driver
  2. UDP Rev 1: Beta Release for Customers (dumb hardware) (2025q2)
  3. UDP Rev 2: Smart Hardware (2025q3)
    • No work toward verification included in this release.
    • Components:
      1. slimstack
      2. frame-routing driver (interacts with particular smart networking hardware)
        • Presumably first hardware target is DPAA2
        • Make sure we understand how to interface with the hardware packet-classifier and verify that our approach for queue management is correct.
        • Determine hardware's capability for offloading , e.g., checksums, packet fragmentation/reassembly, header management (how to get zero-copy to work), etc
        • Understand how much of the firmware interface we can/should use and what the relationship is between bare register writes and the firmware. E.g., "flow steering", table management.
      3. configuration tool



Project endpoints

Multiple project endpoints have been mentioned.

  • Endpoint A. From the roadmap: High DAL UDP and/or TSN, on kismet.
    • Customers' requests are for UDP, or TSN, or both.
    • "High" is likely DAL-A.
    • Is on the roadmap.
    • Has a likely first customer. Customer is interested in UDP, not TSN.
  • Endpoint B. DAL-D UDP stack.
    • People have asked about DAL-D, DAL-C.
    • No near-term customer for this.

PM tasks

  • UDP design discussions (Deos R&D)
  • UDP initial implementation (Deos R&D)
  • TSN - identify Deos Infrastructure + impact to all Deos Components (switching to new stack) (Deos R&D)

Project Endpoint A: High DAL UDP and/or TSN, on kismet

  • Current high-probability customer is asking for UDP DAL-B (we'll likely do DAL-A in that case).
  • So current focus is UDP. TSN is secondary.

Current presumption is this effort will involve taking an existing network stack (open source or licensed), updating it, and adding verification artifacts.

Tasks (draft) for Endpoint A (High DAL UDP stack)

  1. Identify list of protocols to be supported and other essential desiderata.
  2. Identify stack to be the basis.
  3. Develop requirements.
  4. Tag the existing source code.
  5. Modify existing source code where it doesn't follow software checklist.
  6. Develop test cases.
  7. Develop test infrastructure.
  8. Develop test procedures.

For "Develop requirements", inputs are: relevant RFCs, existing source code.

For "Identify list of protocols", known protocols to be included:

  • Yes: UDP.
  • Presumably yes: IPv4 (IP required for UDP). ICMP (ping et al). IGMP (multicast).
  • Presumably no: TCP (initially)
  • Unknown: DHCP? DNS? SNMP? IPv6?
  • NB: TSN is (primarily) a collection of Ethernet protocols -- so it's at the datalink layer, down below UDP/TCP/IP/etc.

For "Identify list of protocols", known stack desiderata are:

  • Thread-safe, multicore-safe.
  • Amenable to TSN.
  • Not too entangled in another OS.
  • Scalable.

For "Test infrastructure"

  • Stub driver
  • More.

"Amenable to TSN" means (at least):

  • Its ethernet layer can be extended to support TSN protcols.

For sizing

  • Using LwIP as example. Method: Built LwIP without TCP, DHCP, DNS, SMTP, PPP. Using debug variant, counted what source lines contributed instructions to binary.
    • lines: 7823
    • semicolons: 4678
    • source files: 53

Project Endpoint B: DAL-D UDP stack

  • Have heard chatter about requests for DAL-D, DAL-C, DAL-B.
  • Currently no PO for this one.
  • Deferring this path.

Notes:

TSN

TSN is a set of Ethernet protocols. Therefore, TSN is at layer 2, the datalink layer; it is below things like IP, UDP, TCP.

Divide up the bus bandwidth. Assign time slots to the standard ethernet priorities. Some classes of frames designated as scheduled (time sensitive) traffic, others are designated as best-effort traffic.

TSN aims to be vendor agnostic deterministic ethernet, using standardized Ethernet protocols. Not fully vendor agnostic yet.

A working TSN system requires nodes serving multiple roles. End nodes, bridges/switches, something to create and distribute the schedule, something to provide precise time. All nodes sharing a medium must obey the schedule if guarantees are to be preserved.

Note need for:

  • Precise distributed timing software/hardware.
  • Configuration software.
  • Schedule creation and distribution software.

Questions:

  • How possible is it to support TSN without being wedded to particular vendors?
    • May have to be wedded to a particular chipset architecture?

Can we do something analogous to our current model for handling architecture/vendor dependencies with lwip, ie:

  • Stack is architecture specific
  • Bottom of stack (driver) is specific to particular network adapter hardware

Investigate how this is handled in Linux's TSN support. 


 Gary G: workedTSN demo with Dornerworks on the nai68ppc2; Dornerworks has a TSN stack
 Used shared memory, not Deos IST component (since they viewed this as a proprietary implementation)
 Used NAI h/w: pcie device flowed data from the ARM board to the PPC board
 Time sensitive portion of the bandwidth, as well as best-effort portion; demonstrated a general solution, since it supports TSN and UDP
 Next Step:  provide an open-source solution to remove OS dependency
 Document the interface
 Publish the solution; ie, shared memory feature of IOI
 Long-term goal: DDCI develop TSN, TCP/IP and UDP stack but will likely be tied to h/w (architecture-specific)

 If we have multiple stacks, can we avoid multiple socket libraries etc?

 Dornerworks demo was tsn stack plus shared mem interface, a la ist.
 nai hw plus dornerworks stack.
 nai hw. 1ppc, 1 arm w/ tsn connectivity, pcie between.


 H/W vendors:
 TTTech's TTE device
 NAI's TSN module (with a Linux solution)
 Dornerworks TSN stack - planned for high DAL
 Honeywell's UDP stack - ToDo Kelly: ask Larry M. for access to the source
 ToDo Bill: review recent requests for TSN and UDP to identify initial and long-term features/requirements; follow up with Sales Team for additional customer details/requests

 There is TSN support in
 intel i225
 intel 226

Meeting Notes

2024-09-25

Chris, Matt, Richard, Aaron
Brainstorming notes for how to estimate level of effort and topics for TSN implementation. Hope is that TSN interface could utilize the High-DAL stack. 

Time Sensitive Network
open tsn
socket api

PTP  presume handled by firmware/hardware.

802as
Parts to quote:
  Getting bits to the hardware
  Does hw manage queue priorities?

  Managing configuration data
    config tools
    verification tools
    downloading configuration

  Frame synch logic between network and kernel scheduler.
    continuous sync
    waitForNextSystemTick() may be necessary as well.

  Some sort of "stack".
    may be as simple as memcpy() or might require framing etc.

  Is there a switch?  Who gets the switch responsibilities.
    If there is no switch, how is master 

  How do we describe what the product contains?
    i.e., how many subservient protocols are supported, e.g.,
    bandwidth allocation.

More research:
  Tiger lake TSN capabilities?
  S32G274a.  NXP
     SJA1110

HW we would need:
  boards containing the TSN devices
  A TSN switch.
  Assume we can use wireshark to analyze packets.

UDP Stack: options to build on

  • What should be the basis for a higher-DAL UDP stack?
  • Primary aim is kismet, but nice if worked on jupiter too.
  • Consider a non-client/server architecture -- libusb.so.
    • Goal: Customers don't want to schedule a server partition; thus, networking executes in requestor's timeline.
  • Consider a "pluggable protocol" architecture -- ability to add in other protocols, a la libdhcp.so, libtcp.so.

option as basis for stack: LwIP

  • Pro:
    • We have experience
    • Designed to enable ifdefing-out undesired features
    • Many organizations are actively using it.
  • Con:
    • If TSN is needed: TSN is a set of Ethernet protocols. LwIP doesn't have TSN in its Ethernet implementation. Much would need to be added.
    • Sizeable amount of code to be "reverse engineered". (Depends on how many features needed.)
      • Probably faster than writing it from scratch.
  • Notes
    • If use LwIP for high-DAL, consider upgrading LwIP to a more recent release if use it.
      • Would be primarily bug fixes; just a few new features.
    • Consider: clean up driver interface to ease upgrading. If practical provide a shim for old interface.


Others

Besides LwIP', other stacks. Most are deeply entangled with some other RTOS.

Notes

Protocols and Features of stacks (LwIP, Tuxera SafeTCPIP, MORE)

Feature LwIP Deos LwIP Tuxera Notes
Protocols
IP Y Y Y (lwip: includes IP forwarding over multiple network interfaces)
IPv6 Y
ICMP Y Y Y
ND Y Neighbor Discovery, for IPv6
MLD Y Multicast Listener Discovery for IPv6
UDP Y Y Y
TCP Y Y Y (lwip: includes TCP congestion control, RTT estimation and fast recovery/fast retransmit)
IGMP Y Y Y Internet Group Messaging Protocol: multicast
ARP Y Y Y
PPPoS Y
PPPoE Y
Ethernet Y Y Y
Sockets Y Y Y
Clients
DHCP client Y Y
DNS client Y Y (lwip: includes mDNS hostname resolver)
AutoIP/APIPA (Zeroconf) Y self configure in absence of DHCP
SNMP agent Y Y Y (lwip: includes v1, v2c, v3, private MIB support & MIB compiler)
Addon apps
HTTP(S) server Y Y
SNTP client Y Simple NTP: subset of Network Time Protocol
SMTP(S) client Y
ping Y
NetBIOS nameserver Y
mDNS responder Y multicast DNS, in absense of nameserver
MQTT client Y Y Message Queuing Telemetry Transport, messaging for IoT.
TFTP server Y
Etc
Certifiability Y (Tuxera: ASIL-B: ISO 26262 Automotive Safety Integrity Level)
  • Tuxera developed to safety standards: TCP, UDP, ARP, ICMP, IGMP, Sockets, Ethernet
  • Tuxera extra modules (not developed to safety standards): IPSec/IKEv2, MACsec, SSH, CryptoCore library


Protocols supported by current DAL-E LwIP:

The following protocols are supported:

  • IPv4 (Internet Protocol version 4)
  • ICMP (Internet Control Message Protocol)
  • TCP (Transmission Control Protocol)r
  • UDP (User Datagram Protocol)
  • IGMP (Internet Group Management Protocol -- IP multicast)
  • DHCP (Distributed Host Configuration Protocol
  • DNS (Domain Name System)
  • SNMP (Simple Network Management Protocol version 2c)

The following RFCs are supported:

  • [[1][RFC0768 User Datagram Protocol]]
  • [[2][RFC0791 Internet Protocol]]
  • [[3][RFC0792 Internet Control Message Protocol]]
  • RFC0793 Transmission Control Protocol
  • RFC0826 Converting Network Protocol Address
  • [[4][RFC1112 Host Extensions for IP multicast]]
  • RFC1157 A Simple Network Management Protocol (SNMP)
  • RFC1122 Requirements for Internet Hosts -- Communication Layers
  • RFC2001 TCP Slow Start, Congestion Avoidance, Fast Retransmit, and Fast Recovery Algorithms
  • RFC2113IP Router Alert Option
  • RFC2131 Dynamic Host Configuration Protocol
  • RFC2132 DHCP Options and BOOTP Vendor Extensions
  • RFC1035 Domain Names - Implementation and Specification
  • RFC2181 Clarifications to the DNS Specification
  • RFC2581 TCP Congestion Control
  • [[5][RFC3376 Internet Group Management Protocol, Version 3]]
  • RFC3390 Increasing TCP's Initial Window
  • RFC3416 Version 2 of the Protocol Operations for the Simple Network Management Protocol (SNMP)
  • [[6][RFC4604 Using Internet Group Management Protocol Version 3 (IGMPv3) and Multicast Listener Discovery Protocol Version 2 (MLDv2) for Source-Specific Multicast]]

LwIP flows

Lwip receive flow

Lwip TCP input flow

Lwip TCP output flow

2023-04-11 Telecon Notes (AL BC JJK MF MV)

  • Goals f/ High-DAL UDP stack:
    • Meets DAL-A objectives.
    • Provides UDP functionality typical in FACE market space.
      • multicast
      • (others?) (TODO Bill will go over recent RFQs, get superset of protocols)
    • Architecture of high-DAL stack within the system must accomodate TSN.
  • Competitors have high-DAL stacks; it is needed for sales.
  • NB: The point of High-DAL UDP cannot be determinism. (A high-DAL stack cannot provide determinstic delivery, just by its DAL; UDP is fire-and-forget, and provides no congestion control. The system design must have mechanisms to avoid collisions (affects at least physical configuration) and to mitigate congestion.)
    • TSN is one mechanism.
  • We cannot require the customers to use TSN networks, or only use TSN-capable hardware, under the high-DAL stack.
    • Some customers only want vanilla-ethernet NIC hardware, not TSN or TTE or whatever. "Keep it simple". (EG, the WAS-respin customer.)
  • We can't require the customers to dedicate a core to the high-DAL stack.
    • Some customers don't want multiple cores.
    • A dedicated core is not needed if a lot of TSN is offloaded to an FPGA.
    • So: for TSN, desire hardware that supports it; we are not doing a software-only stack for TSN.
  • We expect the first user-level API to be provided is sockets, second is queuing ports. Later likely add 653 part 2 service access points.
  • We want to develop a high-level architecture outline to show at geekfest.
    • Which protocols.
    • A guess at an implementation strategy.
  • Long term we'd like to have solutions for TSN, TTE, 664 (AFDX).
  • Can we look at TSNs vendors and see commonalities among the interfaces you must use to talk to them? They likely have no significant commonalities, though they do mostly the same thing. All have FPGA to support TSN, all have different interfaces.
  • TODO JJK will put together lists re what protocols/features lwip supports (which ones we provide, which ones we've thought of adding, and the others availalbe.)
  • TODO JJK set up a followup meeting for next week.


2024-04-16 JK AL Telecon Notes

There will be one or more processes on the current Deos instatnce communicating with other processors (and potentially self) using various network protocols, at least UDP.

Questions:

  1. What is the view of a Deos process w.r.t. the network?
    1. A client of a local network stack.
      • This is essentially what LwIP provides today.
      • The issue is that there is lots of shared state, e.g., dynamic routing tables that are shared between all the client processes. Without a client server architecture it would be difficult for clients to have a consistent view of that state.
    2. A network stack running on a (virtualized) NIC.
      • Potentially something like virtio.
  2. Alternately: Is the anticipated model of a single IP per Deos instance, or per Deos process?

What is the goal?

  1. It seems like clients should have a view of the interface at the socket level.
    1. Can we place restrictions on them, e.g., Deos process A gets UDP ports 1000-1999, B gets 2000-2999, etc.
  2. Application processes should not be "surprisingly" blocked by the software architecture.
    1. Surprising would be, for example, needing to wait for the server process to run.
    2. Surprising would NOT be current process induced queuing latencies, perhaps not even sibling process induced latencies.
  3. Do we need to have dynamic routing protocols?
    1. Trying to limit the amount of "server data" that is dynamically changing.
  4. Can we depend on some high level static configuration?
    1. E.g., UDP ports 1000-1999 are reserved for process A.

We would like to assume the Underlying hardware supports packet classification.

  1. E.g., can we program the hardware to use DMA to deliver all packets destined to ports 1000-1999 to process A?
  2. Classification could be either:
    1. The underlying hardware directly has that capability.
    2. There is a core available that could be dedicated to simulate that capability.
      1. The "core" notion is to eliminate "surprising" blocking.
      2. If this is the solution, what is the multi-core interference argument?
        1. If the server is doing primarily classification (updating queue pointers), perhaps its working set would be modest, and depend on application processes to do any requisite data movement.

AL follow up with Matt (MV)

  1. DPAA and DPAA2 have appropriate classification hardware.
  2. MV thought it was likely that similar capabilities would exist on x86 hw.
  3. No data on ARM or other PPC hardware.
  4. Aaron's sense was that MV thought this approach was something to pursue.

Week of 04-22

UDP stack with Servlet/Slim-Stack diagram 1

UDP stack with Servlet/Slim-Stack diagram 2

Week of 04-28

High DAL Network Stack Overview for Geekfest 2024]]

https://deos.ddci.com/svn/DDCI/products/network/high-dal-network-stack/docs/architecture-design-discussions/HighDAL_UDP_2.drawio

elements of the proposed solution (proposed nomenclature)

  1. libslimstack.so
    1. its initialization function (starts libslimstack task)
    2. its exported APIs (including socket APIs)
    3. its internal state: timers; "PCBs" (with per-socket recv-queues); more..
  2. libslimstack task in client partition (an RMA thread or 653 process or pthread)
    • created by libslimstack.so's initialization function
    • handles timers and (probably?) rx-queue
  3. classifier (normally hardware)
    • all net-using partitions, both clients and lite-server, must register for: [ (ipAddr,protocol,portRange), (ipAddr,protocol,portRange), .. ]
  4. queues
    1. rx-queue for each client libslimstack
      • is backed up by resources owned by client (buffers; descriptors)
      • frame flow: NICs → classifier → rx-queues
    2. tx-queue
      • frame flow: tx-queue → NICs
    • if classifier is hardware, queues are accessed via PRL
  5. lite-server
    • code likely the same as libslimstack.so
    • has registered for: arp, icmp, igmp.

remaining questions (add to this list)

  1. Where does the lite-server live in the schedule?
    1. can it "get scheduled" how we need?
  2. how does libslimstack task in client get created?
    1. can a library init function in 653 create a process?
      • 05/06 RLF> Yes. The 653p2 library has something similar where it has a logbook engraver process. Configuration sets its priority so they can choose where it fits in background processes. ... Networking would be Deos extension, but I don't see an issue there for partitions using network.
    2. can a libslimstack process in a 653 partition "get scheduled" how we need? can it handle the timers? can it handle the rx-queue?
    3. presuming rtems is okay -- pthread_create() etc
  3. how does a client's libslimstack task get notified that something's been added to its rx-queue?
    1. can we poll without making a kernel PRL call?
    2. (can we get a pulseEventPPI() ?)
  4. can the hardware classifiers in the relevant SoCs and NICs do (ipAddr,protocol,portRange)→process parsing and send to one of N queues?
    1. Are the classifiers programmable enough?
    2. Are there enough queues? Can they be used this way?
      • 05/15 Number of queues probably in the 10k's (?).
        • Can presume enough queues for a queue per partition. A queue per socket? Likely enough for that sometimes.
  5. can the classifier route multicast packets to multiple queues?
    • 05/08 Yes
  6. (are there broadcast packets that the clients should see, or only the lite-server?)
    1. 05/15 Broadcast UDP packets certainly.
  7. when does the classifier get configured?
    • likely near box startup
    • 05/15 Note that if a queue per socket, then classifier configuration is likely not static.
  8. does this architecture buy us the performance we want?
  9. how does libslimstack protect access to its data structures (given multiple threads in client partition may be doing network calls)?
    1. What are the data structures?
  10. what state data does libslimstack in client need to expose to lite-server?
  11. are we covering all the things the stack must do? and: how to refactor those among (libslimstack, libslimstack task, lite-server)?
    • what are the responsibilities of eg lwip? (what does the tcpip_thread do each go-around?)
  12. is it valuable to do a proof-of-concept prototype? how much of the real needs to be there to be useful?
    1. what questions must it answer?
    2. note that prototype likely needs (eventually?) a rma, 653, and rtems client
  13. (is it true that we need the libslimstack client task to be running for udp?
    1. is it true that we want that task to be who drains the rx-queue?
    2. 05/15 IP fragmentation/reassembly needs a timer, so UDP needs libslimstack client task?
      • But: note that may be able to have hardware accelerator handle frag/reass.
  14. what does TSN demand? what's needed to accomodate that?
    1. (what we know it needs:
      • time sync: PTP
      • schedule: some way to get the schedule and load it on end nodes' hardware (also on switches)
      • likely an extended/customized ethernet layer
      • more?)
  15. What's the basis for the stack code?
    • any better alternatives than lwip? or, write from scratch? (or, tuxera? no response from them yet.)
    • 05/13, 05/15 DPDK? Can possibly use ideas (eg interfaces), some code, but not use as a basis.
  16. how does a client libslimstack send a request to the lite-server?

Week of 05-06

  1. libslimstack.so
    1. its initialization function (starts libslimstack task) (not depicted)
    2. its exported APIs (including socket APIs)
    3. its internal state: "PCBs" (with per-socket recv-queues); (eventually) timers; more..
  2. libslimstack task in client partition (an RMA thread or 653 process or pthread)
    • created by libslimstack.so's initialization function
    • handles (probably?) rx-queue and (eventually) timers
  3. classifier (normally hardware)
    • all net-using partitions, both clients and lite-server, must register for: [ (ipAddr,protocol,portRange), (ipAddr,protocol,portRange), .. ]
  4. queues
    1. rx-queue for each client libslimstack
      • is backed up by resources owned by client (buffers resource; descriptors resource)
      • frame flow: MACs → classifier → rx-queues
    2. tx-queue
      • frame flow: tx-queue → MACs
    • if classifier is hardware, queues are accessed via PRL
  5. lite-server
    • is code the same as libslimstack.so?
    • has registered for: arp, icmp, (eventually) igmp.


Updated diagram

Candidate questions for potential launch customer



These are the current draft questions:

Draft questions:

  • Does the question from the customer imply they have settled on the LX2160A?
    • 05/16 Probably so.
      • Customer wants to make recommendations to FAA by end of June. Maybe a couple of options, us as one. Aiming for verf complete in Sept '25.
  • The LX2160A has a lot of networking capability. How much of that hardware will they be using? What kind of performance are they expecting
    • 05/16 What data rate (packet processing rate) do they need?
  • Will the system be a closed network or a dynamic network? (EG, can nodes join and leave?)
  • Would it be acceptable to provide development support via a second (non high DAL) NIC, or to run development support only when not in high-DAL mode?
  • Does ARP and ICMP need to be high DAL? (Would it be acceptable if all critical functions that depend on ARP and ICMP must be preconfigured?)
  • Do they expect to use RMA, 653, or RTEMS?
    • 05/16 Considering, and leaning toward, 653.
  • Do they have an expected API environment? (EG, sockets, 653 sampling/queuing ports, etc.)
  • Do they expect to be able to specify QoS parameters? Do they have any expectations regarding how? (EG, socket options, boot-time config file, etc)
  • Will they have a set of predefined port assignments, or are port numbers flexible?
  • Will there be routers? And, can the topology change (eg a new network segment appear)?


Internal-only comments:

  • Developing and verifying a complex new component in 12 (15) months is sporty.
  • No security features have been mentioned by the customer. Assuming that'll remian true.
  • We must be clear that verifying the LX2160A's DPAA2 firmware is Not Our Job.
  • Is the ROM that was mentioned just for the stack, or for the stack and driver?



Week of 05-20

APIs

  • // API signatures should be the same, but code implementing them will be different, for: RMA, 653, RTEMS clients.

1. component: libslimstack.so (high-DAL stack, in client partition and probably liteserver partition)

  • // It does UDP and IP.
    • // The frame-routing hardware may do IP fragmentation/reassembly, more.
  • // It includes the libslimstack task (is that needed for version 1?).
    • // It does timers (eg for fragmentation/reassembly)?
    • // It does rx queue draining?
  • API: to user apps
    • Berkeley sockets: socket(), bind(), getsockopt(), setsockopt(), sendto(), recvfrom(), closesocket(), more.
      • //Socket APIs may cause configuration activity via the frame-routing driver (consider bind(), setsockopt(), eg).
    • 653: Facility to bind UDP ports to queuing ports so as to enable sending/receiving via 653 port calls. (Binding may not be a runtime API.)
    • library initialization call
      • Starts the libslimstack task if appropriate.
      • Notifies frame-routing driver that client is online, ready to process its queues.
    • // more

2. component: frame-routing driver -- queues-and-classifier (driver with PRL that interacts with dataplane accelerators)

  • // Queues are backed up by resources owned by client partition (buffers; descriptors).
  • // rx frame flow: NICs -> classifier -> rx-queues ; tx frame flow: tx-queue(s) -> ?classifier -> NICs
  • // We may want to also write a stub-queue and stub-classifier in sw.
  • 2a. user-mode so
    • API: to lwip or similar (Shim that provides the driver-to-stack API that a traditional stack like lwip expects.)
      1. Means for generic configuration of NIC like lwip.config's "interface=", ethernet, gateway, nameserver, phyinit.
        • //When DAL-E lwip is using frame-routing driver as its NIC, interface configuration will primarily be performed elsewhere (eg, slimstack).
      2. Functions to interact with stack: driver_init(netif) ; link_output(netif,pbuf) and (maybe) output(netif,pbuf,ipaddr) ; (*input)(netif,pbuf) ; more.
    • API: access to queues and classifier
      • // Goal is: expose relevant dpaa2 capabilities but be generic to similar hardware
      • queue -- actions:
        • (a) Create a queue. (b) Make it rx or tx:
          sstatus = createQueue( RX_Q|TX_Q, numSlots, dtorResourceHdl, &q )
        • (d) Bind it to (ipaddr,protocol,ports). potentially repeatedly:
          sstatus = bindQueue( q, localIpAddr|localMacAddr, protocolID, portNum, portMask )
          • // This must fit within the configuration of the classifier (via config tool) for this partition
          • // For noninterference, do we ensure that a tx queue doesn't accept outbound frames which state they originate at an (addr,proto,port) that someone else owns?
        • (c) Associate it with a partition's libslimstack.so
        • (e) ? Bind it to a particular PCB (a la particular socket)?
        • (f) Dequeue a frame.
          • // how to wait on queue to become nonempty?
        • (g) Enqueue a frame.
        • (h) Delete a queue.
        • // Are the TSN-configuring APIs APIs on queues?
      • classifier?
        • API: ... TBD
      • buffers?
        • API: ... TBD
  • 2b. kernel-mode so
    • API: ... TBD

3. component: liteserver (task in its own partition)

  • // liteserver registers for: arp, icmp, igmp.
  • // Define facility for partitions to make requests.
  • // Define facility to expose dynamically updated data (eg ARP table, routing table).
  • // Where does it live in the schedule?
  • // What other responsibilities does lwip's tcpip_thread handle? where are they in this architecture?
    • API: TBD

4. component: config tool

  • // Create a binary file that frame-routing driver maps in.



Diagram

Updated diagram



Week of 05-27

Assumptions (mostly assumed goals)

  1. (programmatic) Develop and verify to DAL-A.
  2. (programmatic) Support kismet, and support jupiter if practical.
  3. Do not count on having pulseEventPPI() or PRL interrupts.
  4. Provide UDP.
  5. Provide IPv4.
    1. IPV4: Support future use of hardware accelerators to do IPv4's checksum, fragmentation/reassembly.
  6. Aim for performance significantly better than RPC-based lwip.
  7. Eliminate surprising blocking.
  8. Architecture must accommodate TSN later.
    • Note: references to frame classes are significant.
  9. Architecture should support replacing hardware accelerators with software implementation (eg on a dedicated core).
  10. (rev 0) Stack and driver will live in RMA environment only (not written to use the scheduling facilities in 653 or pthreads).
  11. (rev 0) Stack will export socket API only (eg, no 653 ports interface).
  12. (rev 0) Driver will support DPAA2 only.
  13. (rev 0) Network topology will be static and closed, permitting: ARP table will be static, routing table will be static.
  14. (rev 0) IP fragmentation/reassembly will not be supported.
  15. (rev 0) Do not make ARP requests. Do ARP responses if can't avoid.
  16. (rev 0) Do not support multicast. Architect so as to in the future support multicast using hardware's facilities.
  17. (rev 0) Don't support ICMP.
  18. (rev 0) Don't support IGMP.
  19. (rev 0) Classifier rule configuration is static
    • (Adding TCP may require dynamic updates to classifier rules.)
  20. (rev 0) Avoid client libslimstack task if possible.
    • (But may need it to drain the rx-queue(s), and future support of IPv4's fragmentation/reassembly will need a timer -- when we do it in sofwware).
  21. (rev 0) Client will poll the rx-queue(s). (Is this adequate?)
  22. (rev 0) Config file is manually constructed.
  23. (rev 0) No security features.
  24. (rev 2) Support: ARP, ICMP, IGMP.
  25. (rev N) TSN.
  26. (rev N) Consider DHCP, DNS, SNMP, TCP, IPv6, PPP.
  27. Support running DAL-E lwip as a client of the frame-routing driver during development. (Is this required?)
  28. Support gigabit NIC but the point of that is response speed. Will not need to process gigabit/packet-length-in-bits packets/second.
  29. Thread-safe, multicore-safe.
  30. Scalable.
  31. The hardware classsifier on any target SoC can classify incoming frames according to (NIC,destIpAddr,protocol,portRange) and route frames into rx-queues per the configuration.
  32. Customer practices re port assignments don't clash with static partitioning of ranges of ports (range may be specified by masking).
  33. The hardware classifier on any target SoC supports a sufficient number of queues
    • Assuming classifer supports rx-queue per partition; if more queues avaiable, rx-queue per socket/PCB may be possible
    • Also see question below re DPAA's limit on number of software portals.
  34. The hardware classifier on any target SoC can route multicast and broadcast frames into multiple rx-queues.

Questions

  1. Proposed: Queues have system extent. Partitions can be given access rights to queues.
  2. Is DPAA's limit on number of software portals an issue?
  3. Do we need to support active/inactive queues (eg, queue is inactive until its associated partition initializes)
  4. Flesh out the config tool.
    1. Must the config file be driver specific?
      • Can the config tool take generic inputs from user and generate driver specific output?
  5. Config tool generates config file for initializing NICs, buffer pools, queues, etc. Who does that at runtime?
  6. (future?) Can client libslimstack task get scheduled as we need in: 653, RMA, RTEMS.
  7. When the hardware does fragmentation/reassembly (eg DPAA2), it's crossing layers, needing IP-layer knowledge. How is that done?
  8. If we start from the existing DPAA2 driver
    1. Does the existing driver handle multiple interfaces?
    2. It needs to properly handle multiple client partitions attaching, and related management.
  9. For each environment (653, etc), how does libslimstack protect its shared data given multiple tasks in partition.
  10. Are we covering all the things the stack must do? And: how to refactor those among (libslimstack, libslimstack task, lite-server)? (What does lwip's tcpip_thread do each go-around?)
  11. For noninterference, do we ensure that a tx queue doesn't accept outbound frames which state they originate at an (addr,proto,port) that someone else owns?
  12. Does the buffer model of DPAA* and lwip clash significantly?
    • Note: if necessary, possible to have adapter at driver interface which memcpy()'s between the two.
  13. Lite-server (future)
    1. Where does the lite-server live in the schedule, how is it scheduled?
    2. Define facility for partitions to make requests.
    3. Define facility to expose dynamically updated data (eg ARP table, routing table).

Notes

JJK> Capture this: Going down the rabbit hole of references to "DPA offloading driver API", in "QorIQ SDK v2.0-1703 Documentation", I saw:

(For later) Tasks

Description Priority Assignee Phase Estimate (days) Comments
25 xxx 10

''Legend''

Description: Short description of task to be worked

Priority: A number 1 - 99

  • 1 Highest

Assignee: The person responsible for doing the work associated with the component.

Phases:

  • Blank - Not yet started
  • Dev - In Development
  • Test - Component unreleased and ready to test
  • Delta - A Delta baseline must be established
  • Audit - One (or more) SQA Audits must be performed (e.g., a Verification and/or Software Conformity Audit)
  • Stable - You are done! All steps (including test report) in the Deos Software Release HowTo or OpenArbor Development HowTo have been followed.

Comments: Free form text.

>

Retrieved from "https://ddci.zapto.org/ddciWiki/index.php?title=Higher-DAL_UDP_Network_Stack&oldid=69055"