Find out short living UDP requests (e.g. DNS requests) with auditd

Think about the following situation. You need to modify the address of your internal DNS server and therefore you need to change /etc/resolv.conf on all your Linux boxes. It might be that this change will not be recognized by every service running on your Linux box. Oracle for instance is a good example where you have to restart your database after such change because Oracle loads the DNS information once the database is started. So Oracle would querry your old DNS server IP unless you restart the database.

There are several ways to find out which process requests your old DNS server. Here I will explain how to utilize auditd on Red Hat derivatives to find out short living DNS requests via UDP. First you need to install auditd and enable the service:

yum install audit
systemctl enable auditd
systemctl start auditd

Then you need to write the following auditd rule:

auditctl -a always,exit -F arch=b64 -S connect -S sendto -S sendmsg -S socket -k udp_connections

To test you auditd rule you can do either a

dig @OLD-DNS-SERVER-IP google.de

or a

nslookup google.de OLD-DNS-SERVER-IP

Hint: Please replace OLD-DNS-SERVER-IP the according IP address of your old DNS server IP. Afterwards you can search the audit log as follows:

ausearch -i -ts today | grep OLD-DNS-SERVER-IP

This command shows you the following entries:

type=PROCTITLE msg=audit(05/28/2018 09:34:57.751:1960068) : proctitle=nslookup google.de OLD-DNS-SERVER-IP type=SOCKADDR msg=audit(05/28/2018 09:34:57.751:1960068) : saddr={ fam=inet laddr=OLD-DNS-SERVER-IP lport=53 } 
type=PROCTITLE msg=audit(05/28/2018 09:34:58.909:1960070) : proctitle=dig @OLD-DNS-SERVER-IP google.de type=SOCKADDR msg=audit(05/28/2018 09:34:58.909:1960070) : saddr={ fam=inet laddr=OLD-DNS-SERVER-IP lport=53 }

To find out more details about the process which tries to reach the old DNS server you can simply search for a specific message ID:

ausearch -a 1960070 -ts today

This gives you the following output:

time->Mon May 28 09:34:58 2018
type=PROCTITLE msg=audit(1527500098.909:1960070): proctitle=646967004031302E36312E342E31353400676F6F676C652E6465
type=SOCKADDR msg=audit(1527500098.909:1960070): saddr=HEX-CODE-OF-YOUR-OLD-DNS-SERVER-IP
type=SYSCALL msg=audit(1527500098.909:1960070): arch=c000003e syscall=46 success=yes exit=38 a0=14 a1=7f763ab9aa70 a2=0 a3=7f763ab9a4a0 items=0 ppid=52831 pid=83098 auid=60001 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=988 comm="dig" exe="/usr/bin/dig" key="udp_connections"

Here you can see that the command /usr/bin/dig tried to establish a connection to saddr=HEX-CODE-OF-YOUR-OLD-DNS-SERVER-IP where saddr is the HEX code of your old DNS server IP.

If you found out the processes which requests your old DNS server you need to remove the auditd rule again to reduce audit logs:

auditctl -d always,exit -F arch=b64 -S connect -S sendto -S sendmsg -S socket -F key=udp_connections