Dienstag, 12. September 2017

wrong permission on shm kills JAVA_JIT

We found a lot of trace files from several DBs on one of our DB Servers.
They look like:

*********_ora_26444.trc or *********_m000_5598.trc
*** 2017-09-08 15:11:29.181
*** SESSION ID:(632.5995) 2017-09-08 15:11:29.181
*** CLIENT ID:(SYSADMIN) 2017-09-08 15:11:29.181
*** SERVICE NAME:(****_****) 2017-09-08 15:11:29.181
*** MODULE NAME:(*::***:******.****.***.****.******) 2017-09-08 15:11:29.181
*** ACTION NAME:(/) 2017-09-08 15:11:29.181

Map_Length = 4096
Map_Protection = 7
Flags = 1
File_Offset = 0
mmap failed with error 1
error message:Operation not permitted

*** 2017-09-08 15:11:29.181
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x0] [PC:0x33BEF41, ioc_pin_shared_executable_object()+1509] [flags: 0x0, count: 1]
DDE: Problem Key 'ORA 7445 [ioc_pin_shared_executable_object()+1509]' was flood controlled (0x6) (incident: 61088)
ORA-07445: exception encountered: core dump [ioc_pin_shared_executable_object()+1509] [SIGSEGV] [ADDR:0x0] [PC:0x33BEF41] [Address not mapped to object] []
ssexhd: crashing the process...
Shadow_Core_Dump = PARTIAL
ksdbgcra: writing core file to directory '/***/diag/rdbms/***/***/cdump'

A quick search on MOS (and a opened SR) showedd as top result Ora‑7445 [Ioc_pin_shared_executable_object()] (Doc ID 1316906.1)

But the suggestions there did not solve the issue. (and we could not set java_jit_enabled = false due to application requirements).

But the Note was good enough to make me search more regarding /dev/shm, mmap and Operation not permitted.  This led to Shared executable memory on StackExchange. Again not a perfect fit, but it makes enough sense to guess:
with java_jit_enabled, the m000 process is doing the compilation (in time). To let the server process execute the compiled code, it's put into shared memory in a /tmp/ file and mapped in the private memory [ 2019-11-15 - fixed based on a comment by Nenad Noveljic ].
This shared memory must be executable, otherwise the server process can not use it. So the memory is mapped with PROT_EXEC.
I checked on the affected host, if there is a reason against this:
> mount|grep shm    
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,seclabel)

noexec prevents shared memory to be executed, so the memory mapping fails.

It's stated in Oracle Database Preinstallation Tasks
rw and execute permissions must be set, but noexec and nosuid must not be set.

this was changed after the DBs were installed. Probably for good intentions but with bad effects.

With the proper changes of the mount options, the test statement
SELECT dbms_java.longname('TEST') long_name FROM  dual;
completes without any error.

Keine Kommentare: