Dienstag, 15. Juli 2008

Shadow process leak on ASM instance when diskspace exhausted (ORA-20)

Once again one of 'my' bugs:
Facts:
  • 10.2.0.3
  • HP-UX
  • archive_log_dest in ASM
  • ASM-Diksgroup full
Symptoms:
  • ORA-20
  • archivelogs doesn't get archived, even the whole backup-subsystem works properly.
Explanation:
If the DG is full he database will try to connect to ASM for every retry of archiver to create archivelog. Even after the attempt to create archivelog fails, this process is not closed.
This will eat up all 'processes' in ASM, so also the DB-process which tries to backup (and delete) the archivelogs can not connect to ASM.
Even if you freed up space in the ASM-DG in the meantime, this problem still persists.

WorkAround:
find OS-process of archiver (ps -ef | grep arc | grep $ORACLE_SID) and kill this process. this will also release all its childs (which holds processes in ASM) and solve the problem.

Reproduce:
  1. fill up your diskGroup
  2. force archiver to respawn a lot of processes (alter system archive log current;) or wait at least for the factor of the REOPEN parameter of your log_archive_dest_x multiplied with PROCESSES from your ASM.
  3. check if there are no more ASM-processes available (e.g. a select name, free_mb from v$asm_diskgroup; will return no rows selected even there are diskgroups available and in use!)
  4. free some space in the DiskGroup
  5. check again if there are ASM-processes available
  6. kill archiver
  7. check again

The good newas after all: there is no corruption, just a ugly hang situation.
Kommentar veröffentlichen