OpenSolaris

You are not signed in. Sign in or register.

DTrace Network Providers

The following is a design proposal for a collection of DTrace Networking Providers. These providers aim to provide networking observability and troubleshooting information for Solaris users. The first prototype TCP provider was demonstrated at CEC 2006.

This document will list the probes that may be made available, their arguments, and link prototypes and examples. Feedback is welcome, please post to dtrace-discuss. NOTE: this is a work in progress, and these network providers are not yet available.

Who is talking to my web server?

# dtrace -n 'tcp:::receive /args[2]->tcp_dport == 80/ {
        @pkts[args[1]->ip_daddr] = count();
}'
dtrace: description 'tcp:::receive' matched 1 probe
^C

  192.168.1.8                                                       9
  fe80::214:4fff:fe3b:76c8                                         12
  192.168.1.51                                                     32
  10.1.70.16                                                       83
  192.168.7.3                                                     121
  192.168.101.101                                                 192

What ports are people connecting to?

# dtrace -n 'tcp:::accept-established { @[args[2]->tcp_dport] = count(); }'
dtrace: description 'tcp:::accept-established' matched 1 probe
^C

       79                2
       22               14
       80              327

Contents

Prototypes

The following prototypes demonstrate older designs than what is described on this main page, however what they demonstrate should still be doable albiet with slightly different probe names and arguments.
  • Prototype #1, Oct 2006. TCP, UDP and IP providers. Also see the CEC2006 Demo.
  • Prototype #2, Jun 2007. IP, IPv4, IPv6 providers.
  • Prototype #3, Dec 2007. IP provider (covers IPv4 and IPv6).
      • Similar to Prototype #2, but with the probe arguments as described in the arguments section.

Aims

The main aims of the network providers are:

  • Support for common network protocols: TCP, UDP, IP.
  • Everyday observability: connections/packets/bytes by address/port.
  • Performance observability: throughputs, network latencies.
  • Socket level tracing: associate events to PIDs.
  • Further troubleshooting support: TCP flag analysis, header info.

Future enhancements and additions to the providers include:

  • Other protocol support: ICMP, SCTP, ...
  • Kernel code path latency probes.

News

Work has recently restarted on this project (see dtrace-discuss), with the intent of completing a design for the IP providers, before moving onto other protocols (TCP, UDP, ...).

The IP provider details on this page have been updated to reflect recent prototyping. Other provider details (especially TCP) will change as they are reprototyped.

Providers

The Network provider will be a collection of several DTrace providers, such as ipv4, ipv6 and tcp (there is no monolithic net provider). The proposed providers are,

providerdescription
gldThis traces the generic LAN device layer, and shows link layer activity such as Ethernet frames. The probes allow frame by frame tracing.
arpThis traces ARP and RARP packets.
icmpThis traces ICMP packets, and provides the type and code from the ICMP header.
ipThis traces high level IP details for IPv4 and IPv6.
tcpThis traces the TCP layer, and shows what TCP activity is occuring.
udpThis traces UDP events.
sctpThis traces SCTP events.
socketThis traces the socket layer, close to the application. These probes fire in the same context as the corresponding process, and show data sent to or received from sockets.

Other providers will be added as required.

Probes

The following is the master plan for the net providers - what probes could be exported, what their arguments could be, and how they fit together. See the prototype pages for implementation details.

probesargs[0]args[1]args[2]args[3]args[4]args[5]
gld:::send
gld:::receive
pktinfo_t *NULLipinfo_tillinfo_t *etherinfo_t *.
ip:::send
ip:::receive
pktinfo_t *csinfo_tipinfo_t *illinfo_t *ipv4info_t *ipv6info_t *
tcp:::send
tcp:::receive
pktinfo_t *csinfo_t *ipinfo_t *tcpinfo_t *..
tcp:::accept-*
tcp:::connect-*
pktinfo_t *csinfo_t *ipinfo_t *tcpinfo_t *..
tcp:::state-*
pktinfo_t *.....
tcp:::fuse-*
pktinfo_t *csinfo_t *NULLtcpfuseinfo_t *size_t.
udp:::send
udp:::receive
pktinfo_t *csinfo_t *ipinfo_t *udpinfo_t *..
udp:::stream-*
pktinfo_t *.....
icmp:::send
icmp:::receive
pktinfo_t *NULLipinfo_t *illinfo_t *icmpinfo_t *.

Other probes will be added as required.

Arguments

The role of the arguments listed above is as follows:
  • pktinfo_t * - packet info; After task 7 in the plan, this will provide packet IDs to allow a packet to be followed as it passes through the network stack. For now, it will use mblk pointers as packet IDs.
  • csinfo_t * - connection state info; After task 13 in the plan, this will provide connection IDs to assist associating packets with user-level process IDs. For now, it will use conn_t pointers as packet IDs.
  • ipinfo_t * - IP info; basic IP info that is available throughout the stack.
  • illinfo_t * - IP lower level info; details about the network interface.
  • etherinfo_t * - Ethernet header info.
  • ipv4info_t * - IPv4 header info.
  • ipv6info_t * - IPv6 header info.
  • tcpinfo_t * - TCP header info.
  • tcpfuseinfo_t * - TCP fusion details.
  • udpinfo_t * - UDP header info.
  • icmpinfo_t * - ICMP header info.

Other arguments will be added as required.

Plan

Network stack instrumentation with DTrace will be achieved through two projects, one to create customer-orientated providers, the other for engineer-orintated providers:

  • public/stable/customer providers for generic observability
      • ip:::, tcp:::, udp:::, ...
  • private/unstable/engineer interface for code path observability
      • sdt:::ip*

Customer Provider Plan

The following are rough steps for creating a suite of customer-orientated network providers. Each step may be integrated separately. 
1. IP Provider
        ip:::send
        ip:::receive

2. TCP Provider
        tcp:::send
        tcp:::receive
        tcp:::accept-*
        tcp:::connect-*
        tcp:::retransmit
        tcp:::state-*
        tcp:::timeout-*
        sdt:::tcpfuse-*

3. UDP Provider
        udp:::send
        udp:::receive
        udp:::stream-*

4. DTrace Enhancements
        write() action
        msgdcopy() action

5. IP Provider
        ip:::drop-in
        ip:::drop-out
        (usr/src/uts/common/inet/ip/ipdrop.c)

6. IP Provider
        ip:::request
        ip:::deliver

7. IP packet ids
        ip:::packet-free
        perhaps a new mblk_t member: (uint64_t)mblk_t->b_id;
        future, any suitible uint64_t: (uint64_t)nifty_t->n_id;

8. Socket Provider
        socket:::send
        socket:::receive

9. ICMP Provider
        icmp:::send
        icmp:::receive

10. ARP Provider
        arp:::send
        arp:::receive

11. SCTP Provider
        sctp:::send
        sctp:::receive

12. GLD Provider
        gld:::send
        gld:::receive

13. TCP/UDP connection ids
        /connfs

Engineer Provider Plan

Many probes have already been integrated into Solaris Nevada for detailed observability of network stack behaviour: 
dtrace -ln 'sdt:::ip*' | awk 'BEGIN { OFS=":::" } { print $2,$5 }'
PROVIDER:::NAME
sdt:::ip-ire-del
sdt:::ip-ire-del-origin
sdt:::ip6-physical-out-end
sdt:::ip6-physical-out-end
sdt:::ip6-physical-out-start
sdt:::ip6-loopback-in-end
sdt:::ip6-loopback-in-end
sdt:::ip6-loopback-in-start
sdt:::ip6-loopback-in-start
sdt:::ip6-forwarding-end
sdt:::ip6-forwarding-start
sdt:::ip6-physical-in-end
sdt:::ip6-physical-in-start
sdt:::ip-xmit-incomplete
sdt:::ip-xmit-v4
sdt:::ip-wput-ipsec-bail
sdt:::ip6-loopback-out-end
sdt:::ip6-loopback-out-end
sdt:::ip6-loopback-out-end
sdt:::ip6-loopback-out-start
sdt:::ip6-loopback-out-start
sdt:::ip6-loopback-out-start
sdt:::ip4-loopback-in-end
sdt:::ip4-loopback-in-end
sdt:::ip4-loopback-in-start
sdt:::ip4-loopback-in-start
sdt:::ip4-loopback-out_end
sdt:::ip-xmit-1
sdt:::ip-xmit-2
sdt:::ip4-loopback-out-end
sdt:::ip4-loopback-out-end
sdt:::ip4-loopback-out-end
sdt:::ip4-loopback-out-end
sdt:::ip4-loopback-out-start
sdt:::ip4-loopback-out-start
sdt:::ip4-loopback-out-start
sdt:::ip4-loopback-out-start
sdt:::ip4-dhcpinit-pkt
sdt:::ip4-physical-in-end
sdt:::ip4-physical-in-start
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-forwarding-end
sdt:::ip4-forwarding-end
sdt:::ip4-forwarding-start
sdt:::ip4-forwarding-start
sdt:::ip-newroute-drop
sdt:::ip6-physical-out_start
In particular, the probes added as part of ipf (eg, sdt:::ip4-physical-out-start) form a good model to build upon.

Testing

Ongoing testing of the network providers will be needed for maintaining their accuracy. It is expected to be achieved through two forms of testing:

1. Customer testing. The SUNWdtrt (DTrace Test Suite) package will include basic tests that can be initiated from a single host, as with the other tests in SUNWdtrt. These tests will generate simple network traffic which will be traced, confirming that the probes and arguments work as expected. These tests are expected to be frequently executed by engineers as part of existing ongoing testing of SUNWdtrt, and by customers when troubleshooting DTrace. There are limitations with single host testing, and some code paths such as those that require special hardware (crypto accelerator cards, NICs that support LSO, etc.) will usually not be tested.

2. Engineer testing. Using multiple hosts with a variety of different hardware, tests can be executed by hand by generating traffic, tracing details, and confirming that all values are as expected. This will happen during development of each task from the plan and if/whenever bugs are lodged. A test team at Sun with access to a test farm could take this onboard as an ongoing task.

Solutions

Some of the network obserability problems that this provide will solve include:

  • Who is connecting to my web server? (TCP accepts by IP address)
  • Which services are busy? (bytes by destination port)
  • Which hosts are responsible for network load? (bytes by IP address)
  • Are hackers/crackers port scanning my server? (TCP flag matching by IP address)
  • Why are outbound connections slow? (Connect latency, 1st byte latency, throughput latency)

Comments

This design has been publically open for comments for several months, and discussed a number of times on the dtrace-discuss mailing list. Time is now running out before these probes will begin to be integrated.

Updated: 05-Mar-2008