OpenSolaris

You are not signed in. Sign in or register.

DTrace Network Providers - Prototype #2

The following describes a design proposal and an updated prototype for a collection of DTrace Networking Providers. These providers aim to provide networking observability and troubleshooting information for Solaris users.

Who is sending IP packets to us?

# dtrace -n 'ip:::receive { @[args[1]->ip_saddr] = count(); }'
dtrace: description 'ip:::receive ' matched 2 probes
^C

  192.168.1.1                                                       2
  192.168.1.5                                                       3
  fe80::214:4fff:fe3b:76c8                                          8
  192.168.1.219                                                    24
  192.168.1.188                                                   108
  192.168.1.109                                                   959

Tracing IP send/receives,

# ./ipio01.d
 NAME       SOURCE             DEST            BYTES      INT  PROTO
 receive    192.168.1.109   -> 192.168.1.108      68     nge0    TCP
 send       192.168.1.108   -> 192.168.1.109      68     nge0    TCP
 receive    192.168.1.109   -> 192.168.1.108      20     nge0    TCP
 receive    192.168.1.109   -> 192.168.1.108      64     nge0   ICMP
 send       192.168.1.108   -> 192.168.1.109      64     nge0   ICMP
 receive    192.168.1.1     -> 224.0.0.9          52     nge0    UDP
 receive    192.168.1.11    -> 224.0.1.1           8     nge0   IGMP

This page is now obsolete. Check the NetworkProvider page for more recent updates.

Author: Brendan Gregg and others, 22-Jun-2007

Contents

Probes

The following probes were made available in this prototype,

ip:::send
ip:::receive
ip:::local-send
ip:::local-receive
ipv4:::send
ipv4:::receive
ipv4:::local-send
ipv4:::local-receive
ipv4:::drop-in
ipv4:::drop-out
ipv4:::read
ipv4:::write
ipv6:::send
ipv6:::receive
ipv6:::local-send
ipv6:::local-receive
ipv6:::drop-in
ipv6:::drop-out
ipv6:::read
ipv6:::write
  • The send and receive probes fire at the bottom of the IP layer (when IP communicates with the network device driver interface).
  • The read and write probes fire when IP communicates with an upper layer protocol, such as TCP.
  • The drop-in and drop-out probes fire whenever IP itself drops a packet (work in progress).

Arguments

The following information is made available for the probes, where possbile and appropriate:

probesargs[0]args[1]args[2]args[3]args[4]
ip:::send
ip:::receive
ip:::local-send
ip:::local-receive
packetidipinfo_t *illinfo_t *ipv4info_t *ipv6info_t *
ipv4:::send
ipv4:::receive
ipv4:::local-send
ipv4:::local-receive
packetidipinfo_t *illinfo_t *ipv4info_t *.
ipv4:::read
ipv4:::write
packetidipinfo_t *illinfo_t *..
ipv4:::drop-in
ipv4:::drop-out
packetidipinfo_t *kstatinfo_t *ipv4info_t *.
ipv6:::send
ipv6:::receive
ipv6:::local-send
ipv6:::local-receive
packetidipinfo_t *illinfo_t *ipv6info_t *.
ipv6:::read
ipv6:::write
packetidipinfo_t *illinfo_t *..
ipv6:::drop-in
ipv6:::drop-out
packetidipinfo_t *kstatinfo_t *ipv6info_t *.

The arguments contain the following members:

argtypecontentstypesnotes
args[0]packetid
 arg0 uint64_t This is a packet ID, a unique number for each packet as it passes
through the TCP/IP stack. The packet ID may change as the packet is
processed, and so another probe will be provided to trace such changes
(perhaps net:::packetid-change).
args[1]ipinfo_t *
 args[1]->ip_ver
args[1]->ip_plength
args[1]->ip_saddr
args[1]->ip_daddr		 uint8_t
uint16_t
string
string		 These fields are valid for both IPv4 and IPv6, and have been
provided for convienence. ip_saddr and ip_daddr are
stringified versions of the IPv4 or IPv6 addresses.
args[2]ipv4info_t * args[2]->ipv4_ver
args[2]->ipv4_tos
args[2]->ipv4_length
args[2]->ipv4_ident
args[2]->ipv4_flags
args[2]->ipv4_offset
args[2]->ipv4_ttl
args[2]->ipv4_protocol
args[2]->ipv4_src
args[2]->ipv4_dst
args[2]->ipv4_hdr		 uint8_t
uint8_t
uint16_t
uint16_t
uint8_t
uint16_t
uint8_t
uint8_t
ipaddr_t
ipaddr_t
ipha_t		 See RFC 791
args[2]ipv6info_t * args[2]->ipv6_ver
args[2]->ipv6_tclass
args[2]->ipv6_flow
args[2]->ipv6_plen
args[2]->ipv6_next
args[2]->ipv6_hlim
args[2]->ipv6_src
args[2]->ipv6_dst
args[2]->ipv6_hdr		 uint8_t
uint8_t
uint32_t
uint16_t
uint8_t
uint8_t
in6_addr_t
in6_addr_t
struct ip6_hdr		 See RFC 2460
args[3]illinfo_t * args[3]->ill_name
args[3]->ill_ipstack
args[3]->ill_addr		 string
int
uint64_t		 ill_name is the interface name, such as "nge0".

The info structures are new, and will be provided by DTrace. They have been based on the RFCs so that their contents are widely understood.

Testing

The following outputs were gathered while testing the IP providers. Usually both a DTrace script and the snoop utility are run to examine the same packets, to confirm observability.

Outbound TCP over IPv4, ipv* send/receive probes, 
   # ./ipio01.d 
    NAME       SOURCE             DEST            BYTES      INT  PROTO
    send       192.168.1.108   -> 192.168.1.109      68     nge0    TCP
    receive    192.168.1.109   -> 192.168.1.108      68     nge0    TCP
    send       192.168.1.108   -> 192.168.1.109      20     nge0    TCP
   ^C

   # snoop -r
   Using device nge0 (promiscuous mode)
   192.168.1.108 -> 192.168.1.109 TCP D=22 S=53122 Push Ack=1137186463 Seq=1...
   192.168.1.109 -> 192.168.1.108 TCP D=53122 S=22 Push Ack=1895485433 Seq=1...
   192.168.1.108 -> 192.168.1.109 TCP D=22 S=53122 Ack=1137186511 Seq=189548...
Inbound TCP over IPv4, ipv* send/receive probes, 
   # ./ipio01.d 
    NAME       SOURCE             DEST            BYTES      INT  PROTO
    send       192.168.1.108   -> 192.168.1.109      68     nge0    TCP
    receive    192.168.1.109   -> 192.168.1.108      68     nge0    TCP
    send       192.168.1.108   -> 192.168.1.109      20     nge0    TCP
   ^C

   # snoop -r
   Using device nge0 (promiscuous mode)
   192.168.1.183 -> (broadcast)  ARP C Who is 192.168.1.183, 192.168.1.183 ?
   192.168.1.109 -> 192.168.1.108 TCP D=22 S=58335 Push Ack=1147792155 Seq=4...
   192.168.1.108 -> 192.168.1.109 TCP D=58335 S=22 Push Ack=400189612 Seq=11...
   192.168.1.109 -> 192.168.1.108 TCP D=22 S=58335 Ack=1147792203 Seq=400189...
Outbound ICMP over IPv4, ipv* send/receive probes, 
   # ./ipio01.d 
    NAME       SOURCE             DEST            BYTES      INT  PROTO
    send       192.168.1.108   -> 192.168.1.109      64     nge0   ICMP
    receive    192.168.1.109   -> 192.168.1.108      64     nge0   ICMP
   ^C

   # snoop -r
   Using device nge0 (promiscuous mode)
   192.168.1.108 -> 192.168.1.109 ICMP Echo request (ID: 36060 Sequence numb...
   192.168.1.109 -> 192.168.1.108 ICMP Echo reply (ID: 36060 Sequence number...
Inbound ICMP over IPv4, ipv* send/receive probes, 
   # ./ipio01.d 
    NAME       SOURCE             DEST            BYTES      INT  PROTO
    receive    192.168.1.109   -> 192.168.1.108      64     nge0   ICMP
    send       192.168.1.108   -> 192.168.1.109      64     nge0   ICMP
   ^C

   # snoop -r
   Using device nge0 (promiscuous mode)
   192.168.1.109 -> 192.168.1.108 ICMP Echo request (ID: 2085 Sequence numbe...
   192.168.1.108 -> 192.168.1.109 ICMP Echo reply (ID: 2085 Sequence number:...
Outbound UDP over IPv4, ipv* send/receive probes, 
   # ./ipio01.d 
    NAME       SOURCE             DEST            BYTES      INT  PROTO
    send       192.168.1.108   -> 192.168.1.5        53     nge0    UDP
    receive    192.168.1.5     -> 192.168.1.108     105     nge0    UDP
   ^C

   # snoop -r
   Using device nge0 (promiscuous mode)
   192.168.1.108 -> 192.168.1.5  DNS C deimos.sf.fishworks.jpn.com. Internet...
    192.168.1.5 -> 192.168.1.108 DNS R deimos.sf.fishworks.jpn.com. Internet...
Outbound TCP over IPv4 to closed port, ipv* send/receive probes, 
   # ./ipio01.d 
    NAME       SOURCE             DEST            BYTES      INT  PROTO
    send       192.168.1.108   -> 192.168.1.109      32     nge0    TCP
    receive    192.168.1.109   -> 192.168.1.108      20     nge0    TCP
   ^C

   # snoop -r
   Using device nge0 (promiscuous mode)
   192.168.1.108 -> 192.168.1.109 TCP D=81 S=45099 Syn Seq=2149577849 Len=0...
   192.168.1.109 -> 192.168.1.108 TCP D=45099 S=81 Rst Ack=2149577850 Win=0...
Inbound TCP over IPv4 to closed port, ipv* send/receive probes, 
   # ./ipio01.d 
    NAME       SOURCE             DEST            BYTES      INT  PROTO
    receive    192.168.1.109   -> 192.168.1.108      32     nge0    TCP
    send       192.168.1.108   -> 192.168.1.109      20     nge0    TCP
   ^C

   # snoop -r
   Using device nge0 (promiscuous mode)
   192.168.1.109 -> 192.168.1.108 TCP D=81 S=48811 Syn Seq=1404105261 Len=0...
   192.168.1.108 -> 192.168.1.109 TCP D=48811 S=81 Rst Ack=1404105262 Win=0...
Outbound TCP over IPv4 IPsec tunnel, ipv* send/receive probes, 
   # ./ipio01.d 
    NAME       SOURCE             DEST            BYTES      INT  PROTO
    send       10.7.250.24     -> 192.146.17.75      68  ip.tun0    TCP
    send       192.168.1.108   -> 172.16.10.1       140     nge0    UDP
    receive    172.16.10.1     -> 192.168.1.108     140     nge0    UDP
    receive    192.146.17.75   -> 10.7.250.24        68  ip.tun0    TCP
    send       10.7.250.24     -> 192.146.17.75      20  ip.tun0    TCP
    send       192.168.1.108   -> 172.16.10.1        92     nge0    UDP
   ^C

   # snoop -r
   Using device nge0 (promiscuous mode)
   192.168.1.108 -> 172.16.10.1  UDP D=4500 S=4500 LEN=140
    172.16.10.1 -> 192.168.1.108 UDP D=4500 S=4500 LEN=140
   192.168.1.108 -> 172.16.10.1  UDP D=4500 S=4500 LEN=92

   Now things get interesting. The snoop output above matches the DTrace
   output for the interface "nge0"; however DTrace also provides visibility
   for packets being sent by "ip.tun0", which have the original un-NAT'd
   IP addresses.

   Huh? wasn't the send/receive probes for the bottom of IP - when IP
   delivers packets to the interface? Wouldn't those "ip.tun0" events
   occur half-way through IP?... Well, "ip.tun0" *is* an interface that
   the bottom of IP is sending to; it just happens to resend the packets
   back through IP again.

   The send/receive probes trace when the bottom of IP sends to an
   interface; that interface could be for a NIC or a tunnel interface.
Inbound TCP over IPv4 IPsec tunnel, ipv* send/receive probes, 
   # ./ipio01.d 
    NAME       SOURCE             DEST            BYTES      INT  PROTO
    receive    172.16.10.1     -> 192.168.1.108     156     nge0    UDP
    receive    192.146.17.75   -> 10.7.250.24        84  ip.tun0    TCP
    send       10.7.250.24     -> 192.146.17.75      20  ip.tun0    TCP
    send       192.168.1.108   -> 172.16.10.1        92     nge0    UDP
   ^C

   # snoop -r
   Using device nge0 (promiscuous mode)
    172.16.10.1 -> 192.168.1.108 UDP D=4500 S=4500 LEN=156
   192.168.1.108 -> 172.16.10.1  UDP D=4500 S=4500 LEN=92
Outbound TCP over IPv4, ipv* read/write probes, 
   # ./ipio02.d 
    NAME       SOURCE             DEST            BYTES      INT
    write      192.168.1.108   -> 192.168.1.109      68     nge0
    read       192.168.1.109   -> 192.168.1.108      68     nge0
    write      192.168.1.108   -> 192.168.1.109      20     nge0
   ^C
Inbound TCP over IPv4, ipv* read/write probes, 
   # ./ipio02.d 
    NAME       SOURCE             DEST            BYTES      INT
    read       192.168.1.109   -> 192.168.1.108      68     nge0
    write      192.168.1.108   -> 192.168.1.109      68     nge0
    read       192.168.1.109   -> 192.168.1.108      20     nge0
   ^C
Inbound ICMP, ipv* read/write probes, 
   # ./ipio02.d 
    NAME       SOURCE             DEST            BYTES      INT
    read       192.168.1.109   -> 192.168.1.108      64     nge0
    write      192.168.1.108   -> 192.168.1.109      64     nge0
   ^C
Outbound TCP over IPv4 IPsec tunnel, ipv* read/write probes, 
   # ./ipio02.d 
    NAME       SOURCE             DEST            BYTES      INT
    write      10.7.250.24     -> 192.146.17.75      68  ip.tun0
    write      192.168.1.108   -> 172.16.10.1        88     nge0
    read       172.16.10.1     -> 192.168.1.108     140     nge0
    read       192.146.17.75   -> 10.7.250.24        68  ip.tun0
    write      10.7.250.24     -> 192.146.17.75      20  ip.tun0
    write      192.168.1.108   -> 172.16.10.1        40     nge0
   ^C
Outbound TCP over IPv4, ipv* read/write/send/receive probes, 
   # ./ipio03.d 
    NAME       SOURCE             DEST            BYTES      INT
    write      192.168.1.108   -> 192.168.1.109      68     nge0
     send      192.168.1.108   -> 192.168.1.109      68     nge0
     receive   192.168.1.109   -> 192.168.1.108      68     nge0
    read       192.168.1.109   -> 192.168.1.108      68     nge0
    write      192.168.1.108   -> 192.168.1.109      20     nge0
     send      192.168.1.108   -> 192.168.1.109      20     nge0
   ^C
Inbound TCP over IPv4, ipv* read/write/send/receive probes, 
   # ./ipio03.d 
    NAME       SOURCE             DEST            BYTES      INT
     receive   192.168.1.109   -> 192.168.1.108      68     nge0
    read       192.168.1.109   -> 192.168.1.108      68     nge0
    write      192.168.1.108   -> 192.168.1.109      68     nge0
     send      192.168.1.108   -> 192.168.1.109      68     nge0
     receive   192.168.1.109   -> 192.168.1.108      20     nge0
    read       192.168.1.109   -> 192.168.1.108      20     nge0
   ^C
Inbound ICMP, ipv* read/write/send/receive probes, 
   # ./ipio03.d 
    NAME       SOURCE             DEST            BYTES      INT
     receive   192.168.1.109   -> 192.168.1.108      64     nge0
    read       192.168.1.109   -> 192.168.1.108      64     nge0
    write      192.168.1.108   -> 192.168.1.109      64     nge0
     send      192.168.1.108   -> 192.168.1.109      64     nge0
Outbound UDP over IPv4, ipv* read/write/send/receive probes, 
   # ./ipio03.d 
    NAME       SOURCE             DEST            BYTES      INT
    write      192.168.1.108   -> 192.168.1.5        53     nge0
     send      192.168.1.108   -> 192.168.1.5        53     nge0
     receive   192.168.1.5     -> 192.168.1.108     105     nge0
    read       192.168.1.5     -> 192.168.1.108     105     nge0
   ^C
Outbound TCP over IPv4 IPsec tunnel, ipv* read/write/send/receive probes, 
   # ./ipio03.d 
    NAME       SOURCE             DEST            BYTES      INT
    write      10.7.250.24     -> 192.146.17.75      68  ip.tun0
     send      10.7.250.24     -> 192.146.17.75      68  ip.tun0
    write      192.168.1.108   -> 172.16.10.1        88     nge0
     send      192.168.1.108   -> 172.16.10.1       140     nge0
     receive   172.16.10.1     -> 192.168.1.108     140     nge0
    read       172.16.10.1     -> 192.168.1.108     140     nge0
     receive   192.146.17.75   -> 10.7.250.24        68  ip.tun0
    read       192.146.17.75   -> 10.7.250.24        68  ip.tun0
    write      10.7.250.24     -> 192.146.17.75      20  ip.tun0
     send      10.7.250.24     -> 192.146.17.75      20  ip.tun0
    write      192.168.1.108   -> 172.16.10.1        40     nge0
     send      192.168.1.108   -> 172.16.10.1        92     nge0
   ^C

   here is the same output but with newlines manually inserted around
   each physical packet,
   
   # ./ipio03.d 
    NAME       SOURCE             DEST            BYTES      INT

    write      10.7.250.24     -> 192.146.17.75      68  ip.tun0
     send      10.7.250.24     -> 192.146.17.75      68  ip.tun0
    write      192.168.1.108   -> 172.16.10.1        88     nge0
     send      192.168.1.108   -> 172.16.10.1       140     nge0

     receive   172.16.10.1     -> 192.168.1.108     140     nge0
    read       172.16.10.1     -> 192.168.1.108     140     nge0
     receive   192.146.17.75   -> 10.7.250.24        68  ip.tun0
    read       192.146.17.75   -> 10.7.250.24        68  ip.tun0

    write      10.7.250.24     -> 192.146.17.75      20  ip.tun0
     send      10.7.250.24     -> 192.146.17.75      20  ip.tun0
    write      192.168.1.108   -> 172.16.10.1        40     nge0
     send      192.168.1.108   -> 172.16.10.1        92     nge0
   ^C
Outbound TCP over IPv6, ipv* send/receive probes, 
   # ./ipio01.d 
    NAME       SOURCE             DEST            BYTES      INT  PROTO
    send       fe80::2e0:81ff:fe5e:8308 -> fe80::214:4fff:fe3b:76c8    68     nge0    TCP
    receive    fe80::214:4fff:fe3b:76c8 -> fe80::2e0:81ff:fe5e:8308    68     nge0    TCP
    send       fe80::2e0:81ff:fe5e:8308 -> fe80::214:4fff:fe3b:76c8    20     nge0    TCP

   # snoop -r
   Using device nge0 (promiscuous mode)
   fe80::2e0:81ff:fe5e:8308 -> fe80::214:4fff:fe3b:76c8 TCP D=22 S=46004 Push Ack=3433398788 Seq=4184304355 Len=48 Win=50400
   fe80::214:4fff:fe3b:76c8 -> fe80::2e0:81ff:fe5e:8308 TCP D=46004 S=22 Push Ack=4184304403 Seq=3433398788 Len=48 Win=50400
   fe80::2e0:81ff:fe5e:8308 -> fe80::214:4fff:fe3b:76c8 TCP D=22 S=46004 Ack=3433398836 Seq=4184304403 Len=0 Win=50400

   I'll need to rewrite these scripts to keep the output lines < 80 chars
   where possible...
Inbound TCP over IPv6, ipv* send/receive probes (excuse > 80 chars for now), 
   # ./ipio01.d 
    NAME       SOURCE             DEST            BYTES      INT  PROTO
    receive    fe80::214:4fff:fe3b:76c8 -> fe80::2e0:81ff:fe5e:8308    68     nge0    TCP
    send       fe80::2e0:81ff:fe5e:8308 -> fe80::214:4fff:fe3b:76c8    68     nge0    TCP
    receive    fe80::214:4fff:fe3b:76c8 -> fe80::2e0:81ff:fe5e:8308    20     nge0    TCP
   
   # snoop -r
   Using device nge0 (promiscuous mode)
   fe80::214:4fff:fe3b:76c8 -> fe80::2e0:81ff:fe5e:8308 TCP D=22 S=50768 Push Ack=4179173848 Seq=3428121375 Len=48 Win=50400
   fe80::2e0:81ff:fe5e:8308 -> fe80::214:4fff:fe3b:76c8 TCP D=50768 S=22 Push Ack=3428121423 Seq=4179173848 Len=48 Win=50400
   fe80::214:4fff:fe3b:76c8 -> fe80::2e0:81ff:fe5e:8308 TCP D=22 S=50768 Ack=4179173896 Seq=3428121423 Len=0 Win=50400
Outbound TCP over IPv6, ipv* read/write/send/receive probes, 
   # ./ipio03.d 
    NAME       SOURCE             DEST            BYTES      INT
    write      fe80::2e0:81ff:fe5e:8308 -> fe80::214:4fff:fe3b:76c8    68     nge0
     send      fe80::2e0:81ff:fe5e:8308 -> fe80::214:4fff:fe3b:76c8    68     nge0
     receive   fe80::214:4fff:fe3b:76c8 -> fe80::2e0:81ff:fe5e:8308    68     nge0
    read       fe80::214:4fff:fe3b:76c8 -> fe80::2e0:81ff:fe5e:8308    68     nge0
    write      fe80::2e0:81ff:fe5e:8308 -> fe80::214:4fff:fe3b:76c8    20     nge0
     send      fe80::2e0:81ff:fe5e:8308 -> fe80::214:4fff:fe3b:76c8    20     nge0
 ^C
Inbound TCP over IPv6, ipv* read/write/send/receive probes, 
   # ./ipio03.d 
    NAME       SOURCE             DEST            BYTES      INT
     receive   fe80::214:4fff:fe3b:76c8 -> fe80::2e0:81ff:fe5e:8308    68     nge0
    read       fe80::214:4fff:fe3b:76c8 -> fe80::2e0:81ff:fe5e:8308    68     nge0
    write      fe80::2e0:81ff:fe5e:8308 -> fe80::214:4fff:fe3b:76c8    68     nge0
     send      fe80::2e0:81ff:fe5e:8308 -> fe80::214:4fff:fe3b:76c8    68     nge0
     receive   fe80::214:4fff:fe3b:76c8 -> fe80::2e0:81ff:fe5e:8308    20     nge0
    read       fe80::214:4fff:fe3b:76c8 -> fe80::2e0:81ff:fe5e:8308    20     nge0
   ^C

Source Code

A webrev has been generated (which includes patches) to show what this instrumentation will mean for the TCP/IP stack. This isn't intended as a code review (yet, anyway); rather a view of my workspace for anyone interested.

Really determined readers should be able to patch the latest OpenSolaris or Solaris Nevada from the webrev, build, install, and try this for themselves. (if you haven't patched and rebuilt a Solaris kernel before, be warned that it can take several hours).

Prototype #2.0 webrev, original take-2 webrev.
Prototype #2.1 webrev, updated, and is for ip:::*send/*receive probes only.

Feedback is of course welcome to either dtrace-discuss or brendan at sun dot com. However, please check the list below first.

Known Issues

  • The IPv4 and IPv6 provider names don't work as "ipv4:::" or "ipv6:::" (due to pid provider matching); a workaround is to use them as "ipv4*:::" and "ipv6*:::". This will be fixed.
  • The IPv4 and IPv6 providers can't be globbed as "ipv*:::". Well, they can, but the translators always pick IPv6 by mistake. The workaround is to specify them both with identical actions, eg "ipv4*::: { action1 }", "ipv6*::: { action1 }". Either this will be fixed in DTrace (it is complex) or the translator will be coded with a workaround.
  • The packet IDs (currently based on mblk_t *s, but that detail is private) need some tweaking to be more consistant (choosing to skip M_CTL headers in the code, etc).
  • The read/write probes are new additions, and their locations will be tweaked as codepaths that they don't instrument properly are discovered.
  • The drop-in/drop-out probes don't match much beyond IPsec drops, due to RFE 6321434 (they instrument ip_drop_packet()). A kstat_named_t -> kstatinfo_t translator also needs to be written.
  • IPv4 options and IPv6 extension headers need consideration for how best to export these from a DTrace translator.
  • More testing is needed.

Bearing the above workarounds in mind, the following are two of the scripts that were using during testing (these scripts will improve as those bugs are fixed),

ipio01.d
#!/usr/sbin/dtrace -s

#pragma D option quiet
#pragma D option switchrate=10hz

dtrace:::BEGIN
{
        printf(" %-10s %-15s    %-15s %5s %8s %6s\n", "NAME", "SOURCE",
            "DEST", "BYTES", "INT", "PROTO");
}

ipv4*:::send,
ipv4*:::receive
{
        self->protocol = args[3]->ipv4_protocol;
}

ipv6*:::send,
ipv6*:::receive
{
        self->protocol = args[3]->ipv6_next;
}

ipv*:::send,
ipv*:::receive
{
        this->protostr =
            self->protocol == IPPROTO_IP      ? "IP"     :
            self->protocol == IPPROTO_ICMP    ? "ICMP"   :
            self->protocol == IPPROTO_IGMP    ? "IGMP"   :
            self->protocol == IPPROTO_TCP     ? "TCP"    :
            self->protocol == IPPROTO_EGP     ? "EGP"    :
            self->protocol == IPPROTO_UDP     ? "UDP"    :
            self->protocol == IPPROTO_IPV6    ? "IPV6"   :
            self->protocol == IPPROTO_ROUTING ? "ROUTE"  :
            self->protocol == IPPROTO_ESP     ? "ESP"    :
            self->protocol == IPPROTO_AH      ? "AH"     :
            self->protocol == IPPROTO_ICMPV6  ? "ICMPV6" :
            self->protocol == IPPROTO_OSPF    ? "OSPF"   :
            self->protocol == IPPROTO_SCTP    ? "SCTP"   :
            self->protocol == IPPROTO_RAW     ? "RAW"    :
            lltostr((uint64_t)self->protocol);
        self->protocol = 0;
}

ipv4*:::send,
ipv4*:::receive
{
        printf(" %-10s %-15s -> %-15s %5d %8s %6s\n", probename,
            args[1]->ip_saddr, args[1]->ip_daddr, args[1]->ip_plength,
            args[2]->ill_name, this->protostr);
}

ipv6*:::send,
ipv6*:::receive
{
        printf(" %-10s %-15s -> %-15s %5d %8s %6s\n", probename,
            args[1]->ip_saddr, args[1]->ip_daddr, args[1]->ip_plength,
            args[2]->ill_name, this->protostr);
}
ipio03.d
#!/usr/sbin/dtrace -s

#pragma D option quiet
#pragma D option switchrate=10hz

dtrace:::BEGIN
{
        printf(" %-10s %-15s    %-15s %5s %8s\n", "NAME", "SOURCE", "DEST",
            "BYTES", "INT");
}

ipv4*:::send,
ipv4*:::receive
{
        printf("  %-9s %-15s -> %-15s %5d %8s\n", probename,
            args[1]->ip_saddr, args[1]->ip_daddr, args[1]->ip_plength,
            args[2]->ill_name);
}

ipv4*:::read,
ipv4*:::write
{
        printf(" %-10s %-15s -> %-15s %5d %8s\n", probename,
            args[1]->ip_saddr, args[1]->ip_daddr, args[1]->ip_plength,
            args[2]->ill_name);
}

ipv6*:::send,
ipv6*:::receive
{
        printf("  %-9s %-15s -> %-15s %5d %8s\n", probename,
            args[1]->ip_saddr, args[1]->ip_daddr, args[1]->ip_plength,
            args[2]->ill_name);
}

ipv6*:::read,
ipv6*:::write
{
        printf(" %-10s %-15s -> %-15s %5d %8s\n", probename,
            args[1]->ip_saddr, args[1]->ip_daddr, args[1]->ip_plength,
            args[2]->ill_name);
}