Socket tools - netstat, ss, lsof
Thu, Mar 23, 2017Sometimes you might ask: what’s listening on port X? Specifically, you may want to find out which process has a socket bound to that port, where the corresponding binary lives on disk, and where you might find relevant logs for the process.
Processes, Ports, and Sockets
When a process wants to accept inbound TCP connections, it typically does so by binding a known address and port to a socket it has created.
The process first creates a socket(2), then calls the bind(2) syscall to bind an address (IP address & port) to the socket, and finally calls listen(2) to specify the socket as accepting inbound connections.
A server that’s listening on port X can be said to maintain a listening socket bound to <ip>:<port>, where <ip> is one (or all) of the IP addresses belonging to network interfaces on the host. A socket is the endpoint of a connection and is represented as a file. Processes access their sockets via the corresponding file descriptor.
Note that there are a variety of socket communication domains, including “Unix sockets” (AF_UNIX), used for local communication between processes. The domains we’re interested here are AF_INET and AF_INET6, representing IPv4 and IPv6, respectively. Sockets are created with a specific domain which defines the address families that are valid for use with bind().
Surveying Listening Sockets with netstat
When it comes to inspecting ports in linux, you’re really inspecting sockets. For this, netstat is your friend. We can use some flags to filter output:
- -l show only listening sockets
- -t restrict output to TCP sockets only
- -n ensure the numeric IP and port number are shown
- -p include the PID that owns the socket
❯ sudo netstat -tnlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 259/sshd
tcp 0 0 192.168.2.49:80 0.0.0.0:* LISTEN 1160/nginx -g daem
tcp6 0 0 :::22 :::* LISTEN 259/sshd
Above we can see that two different PIDs are listening on a total of three addresses (ip:port tuples).
For example, PID 259 (sshd) is bound to 0.0.0.0:22 and :::22. The former is the IPv4 representation of “all local interfaces”, meaning that sshd will accept inbound connections to TCP port 22 regardless of whether they’re pointed at eth0, eth1, loopback, or some other network interface on the machine. Similarly, :: indicates that sshd will accept inbound connections on TCP port 22 for any of the many local IPv6 IP address.
By contrast, PID 1160 (nginx) has been configured only to listen to a single IP address, 192.168.2.49, meaning attempts to connect to it at a loopback address (say, 127.0.0.1) will fail:
❯ curl -I 127.0.0.1:80
curl: (7) Failed to connect to 127.0.0.1 port 80: Connection refused
❯ curl -I 192.168.2.49:80
HTTP/1.1 200 OK
...
Note that elevated privilege is necessary to use the -p flag. Otherwise, an unprivileged user will be unable to access the file descriptor list of processes they don’t own as netstat scans /proc.
❯ strace -e trace=openat netstat -tnlp
...
openat(AT_FDCWD, "/proc/1/fd", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = -1 EACCES (Permission denied)
openat(AT_FDCWD, "/proc/2/fd", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = -1 EACCES (Permission denied)
openat(AT_FDCWD, "/proc/3/fd", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = -1 EACCES (Permission denied)
...
ss: A netstat Alternative
The lesser-known utility ss is also a great tool for inspecting sockets. Here, the flags are similar:
❯ sudo ss -ltnp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:* users:(("sshd",pid=259,fd=3))
LISTEN 0 128 192.168.2.49:80 *:* users:(("nginx",pid=1165,fd=6),("nginx",pid=1163,fd=6),("nginx",pid=1162,fd=6),("nginx",pid=1161,fd=6),("nginx",pid=1160,fd=6))
LISTEN 0 128 :::22 :::* users:(("sshd",pid=259,fd=4))
In contrast to netstat, I find ss to be less multi-tool-ish and thus easier to use (ie, the man page is shorter). It can do some neat things, like show you the length of socket send and receive queues (see above) and the value of various TCP timers (-o), as well as apply very complex filters (see debian package iproute2-doc or below example).
Inspecting a Specific Socket
If you know the PID or port number you’re interested in, you could grep the output of netstat. However, we can also use ss’s filtering capabilities to show TCP listening sockets bound to local port 80:
❯ sudo ss -tlnp '( sport = :80 )'
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 192.168.2.49:80 *:* users:(("nginx",pid=20883,fd=6),("nginx",pid=20881,fd=6),("nginx",pid=20880,fd=6),("nginx",pid=20879,fd=6),("nginx",pid=20878,fd=6))
Alternatively, you could use lsof(8) to look at AF_INET/AF_INET6 socket files with a pattern filter that includes the port of interest:
❯ sudo lsof -i :80
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nginx 1160 root 6u IPv4 30472 0t0 TCP 192.168.2.49:http (LISTEN)
nginx 1161 www-data 6u IPv4 30472 0t0 TCP 192.168.2.49:http (LISTEN)
nginx 1162 www-data 6u IPv4 30472 0t0 TCP 192.168.2.49:http (LISTEN)
nginx 1163 www-data 6u IPv4 30472 0t0 TCP 192.168.2.49:http (LISTEN)
nginx 1165 www-data 6u IPv4 30472 0t0 TCP 192.168.2.49:http (LISTEN)
In contrast to the netstat output, we now see five processes with access to the socket bound to 192.169.2.49:80. What gives?
It turns out that nginx uses fork(2) to create multiple worker processes that can handle clients as they connect. fork() creates new processes, and these children inherit the file descriptors of the parent. As a result, we see the parent (1160) and all four children listed as maintaining the listening socket FD.
From PID to Binary
Discovering further details about the server process is simple once you have the PID thanks to the /proc filesystem:
The binary lives at /usr/sbin/nginx:
❯ sudo file /proc/1160/exe
/proc/1160/exe: symbolic link to /usr/sbin/nginx
The current working directory is /:
❯ sudo file /proc/1160/cwd
/proc/1160/cwd: symbolic link to /
You can even use lsof to find which log files the process writes to by ANDing (-a) two filters: (1) show open files in the /var/log directory (+D) (2) belonging to PID 1161:
❯ sudo lsof -a +D /var/log -p 1161
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nginx 1161 www-data 2w REG 0,16 0 172844 /var/log/nginx/error.log
nginx 1161 www-data 4w REG 0,16 0 172844 /var/log/nginx/error.log
nginx 1161 www-data 12w REG 0,16 0 172843 /var/log/nginx/access.log