Jonathan Stanton1Spring 2004 / Lecture 9Network IICS 184Kernel NetworkingDepartment of Computer ScienceGeorge Washington UniversityJonathan Stanton2Spring 2004 / Lecture 9Application - OS• Typically the Application has a standard interface toan OS.• BSD Sockets on Winsock.• API specified• Usually extensible to support many protocols andchanging models.• Concurrency models key.Jonathan Stanton3Spring 2004 / Lecture 9OS - Network• How an OS interacts with the network protocols anddevices.• Usually OS specific for devices.• Some commonality for protocols (standards, sharedimplementations)• Changes coupled to OS changes.• Very opaque to applications.Jonathan Stanton4Spring 2004 / Lecture 9OS Internal Protocol StructureBSDSocketsINETSocketsTCP UDPuserkernelIPNetworkApplicationsPPP SLIP ETHERNETARPJonathan Stanton5Spring 2004 / Lecture 9Linux Network Structures• Sk_buff– Manages the data making up a packet• Socket structure– Setup by ‘socket’ call• Sock structure– Network specific parts of socket• Proto structure– Operations and implementation of UDP and TCPJonathan Stanton6Spring 2004 / Lecture 9Sk_buff structureskstampdevhnhmacdstcblencsumprotocoltruesizedestructorendtailheaddataPointer to owning socketArrival timePointer to receiving/transmitting devicePointer to transport layer headerChecksumPointer to network layer headerPacket network protocolBuffer sizePointer to destruct functionPointer to link layer headerPointer to dst_entryTCP per-packet control informationActual data lengthdataJonathan Stanton7Spring 2004 / Lecture 9Sock structuredatasocketstruct socketstruct sockJonathan Stanton8Spring 2004 / Lecture 9Creation of a socketdatastruct socket{ } struct sock{ }socketprot{udp, tcp, raw}_protBSD socketINET socketprotocolIPPROTO_{UDP, TCP, RAW}struct proto{ }familyPF_INETinit()tcp_v4_init_sock()… moreinitializationJonathan Stanton9Spring 2004 / Lecture 9Buffer Managementstruct sock*next, *prevstruct device*prev*nextwrite_queue struct sk_buff_head{ }*sk*next, *prev*sk*next, *prev*skbuffs[ ]*dev *dev *devSocket buffer  TCP/UDP IP  devicestruct sk_buff struct sk_buff struct sk_buffJonathan Stanton10Spring 2004 / Lecture 9Path of a message: App -> NetWrite( )sys_write ( )sock_write( )sock_sendmsg( )inet_sendmsg( )tcp_sendmsg( )tcp_transmit_skb( )ip_queue_xmit( )Jonathan Stanton11Spring 2004 / Lecture 9Sockets API -> Kernelsocket()bind()connect()listen()accept()setsockopt()getsockopt()send()sendto()recv()recvfrom()shutdown()socketpair()……socketcall()system callsys_socket()sys_bind()sys_connect()sys_listen()sys_accept()sys_setsockopt()sys_getsockopt()sys_send()sys_sendto()sys_recv()sys_recvfrom()sys_shutdown()sys_socketpair()……socketcall(SYS_SOCKET, 3, args);kernelJonathan Stanton12Spring 2004 / Lecture 9Writing datasock_write(struct file *file, const char *ubuf, size_t size, loff_t *ppos)sock_sendmsg(struct socket *sock, struct msghdr *msg, int size)• Sock_write:• Looks up socket structure.• Fills in message header.• Sock_sendmsg:• Selects correct protocol operation: inet_sendmsg()Jonathan Stanton13Spring 2004 / Lecture 9Protocol write• Inet_sendmsg:• Extracts sock structure• Verifies status of socket• Calls {udp,tcp}_sendmsginet_sendmsg(struct socket *sock, struct msghdr *msg, int size, struct scm_cookie *scm)Jonathan Stanton14Spring 2004 / Lecture 9TCP sending• Tcp_sendmsg:• Allocates sk_buff and initializes it.• Tcp_transmit_skb:• Build TCP header.• Compute checksum.tcp_sendmsg(struct sock *sk, struct msghdr *msg, int size)tcp_transmit_skb(struct sock *sk, struct sk_buff *skb)Jonathan Stanton15Spring 2004 / Lecture 9Transmit to Device• ip_queue_xmit:• Build IP header.• Calculate IP checksum• Fragment packet.• Calculate route.• Call dev_queue_xmit() for appropriate device.– Adds sk_buff to device queue, and if device idle, callshard_start_xmit() to initiate send.ip_queue_xmit(struct sk_buff *skb)Jonathan Stanton16Spring 2004 / Lecture 9Device Sendhard_start_xmit()edev_start_xmit()Network packetSent to edev interfaceEthernet networkJonathan Stanton17Spring 2004 / Lecture 9Path of a message: Net -> Appinet_recvmsg( )net_rx_action( )tcp_rcvmsg( )ip_rcv( )tcp_rcv_established( )read( )sys_read( )netif_rx( )Run by schedulerJonathan Stanton18Spring 2004 / Lecture 9Network Device Receiveedev_interrupt( )edev_rx( )netif_rx ()Ethernet frame that containsIP PacketinterruptJonathan Stanton19Spring 2004 / Lecture 9Soft IRQ handlingnet_rx_action(struct softirq_action *h)• Net_rx_action:• Check packet type.• Call ip_rcv.• Check packet for errors and defragment.• Call ip_route_input()• Call {udp/tcp}_rcvip_rcv(struct sk_buff *skb, struct net_device *dev, struct packet_type *pt)Jonathan Stanton20Spring 2004 / Lecture 9TCP protocol processing• Tcp_rcv_established:• TCP specific processing.• Send acks.• Queue packet in socket receive queue.tcp_rcv_established(struct sock *sk, struct sk_buff *skb,struct tcphdr *th, unsigned len)Jonathan Stanton21Spring 2004 / Lecture 9Process Receive Packet• Process calls read/recvmsg.• Translates to sys_read systemcall.• The socket structure maps areceive to the INET protocol.• Tcp_recvmsg handles the actualpacket preparation and buffer tobe copied into user buffer.inet_recvmsg( )tcp_rcvmsg( )read( )sys_read( )Jonathan Stanton22Spring 2004 / Lecture 9Application level Networking• Minimal OS support• Network protocols implemented by application:– Examples:• TCP/IP• Reliable datagrams• Multicast– Techniques• System call intercept.• OS supported packet interception.• Special library interface• Flexible, but usually lower performance.Jonathan Stanton23Spring 2004 / Lecture 9Limitations of StandardInterfaces• Performance.– Force copies (read/write)– Fragments application packets (write vs. writev)• Control.– Packet interception.– User-specific protocols.– Use of kernel routing table.Jonathan Stanton24Spring 2004 / Lecture 9Solution• Improve existing interfaces behavior– Zero-copy TCP– Wake-one accept• Add new interfaces– Sendfile– Socket option TCP_CORK– Channels and Buffers in JavaJonathan Stanton25Spring 2004 / Lecture 9Rules of thumb• Optimize common case• Watch out for bottlenecks• Fine tune inner loops• Choose good data structures• Beware of data touching• Minimize # packets sent• Send largest packets possible• Cache hints• Use hardware• Exploit