why does files in /var/tmp/.oracle disappear?At least this question I am now able to answer.
By the help of Linux kernel's audit system and a proper rule:
-a always,exit -F arch=b64 -S rename,rmdir,unlink,unlinkat,renameat -F path=/var/tmp/.oracle -F key=ora.delete
I was able to see the culprit process:
---- type=PROCTITLE msg=audit(08/06/2018 17:13:48.382:58122) : proctitle=/usr/bin/systemd-tmpfiles --clean type=PATH msg=audit(08/06/2018 17:13:48.382:58122) : item=1 name=sCRSD_IPC_SOCKET_11 inode=50924979 dev=00:25 mode=socket,777 ouid=root ogid=root rdev=00:00 nametype=DELETE type=PATH msg=audit(08/06/2018 17:13:48.382:58122) : item=0 name=/ inode=45518 dev=00:25 mode=dir,sticky,777 ouid=root ogid=root rdev=00:00 nametype=PARENT type=CWD msg=audit(08/06/2018 17:13:48.382:58122) : cwd=/ type=SYSCALL msg=audit(08/06/2018 17:13:48.382:58122) : arch=x86_64 syscall=unlinkat success=yes exit=0 a0=0x5 a1=0x556fb11fb17b a2=0x0 a3=0x3 items=2 ppid=1 pid=25307 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=systemd-tmpfile exe=/usr/bin/systemd-tmpfiles key=ora.delete ----
So it was a feature of systemd: systemd-tmpfiles.
In detail it comes from
/usr/lib/tmpfiles.d/tmp.confthis particular line:
v /var/tmp 1777 root root 30dy
The fix for this is easy: create a proper file (copy) in
/etc/tmpfiles.dand add these lines at the end:
# Exclude /var/tmp/.oracle x /var/tmp/.oracleBut as a research at MOS and google did not come up with any issue with systemd-tmpfiles and crsd.bin, oraagent.bin or any other keyword, something else must be a problem here as well.
This (and the other question from my previous post) are questions for further investigations.
So much for now.