Dienstag, 13. Januar 2009

multiple ASM instances on one node (11gR1)

I just searched a way to circumvent the limit of 63 ASM DiskGroups in a storage system.
My first thought was to create a 2nd (or multiple) ASM instances on the same node which manage seperated DGs. This is quite easily possible: Just create a spfile for the 2nd ASM, just make sure to add *.DB_UNIQUE_NAME to avoid ORA-15150. I also recommend to use different ASM_DISKSTRINGs, to avoid multiple mount-attempts of the same DG on all ASM instances.
So I could create different DGs in different ASM-instances.
But now the troubles starts: In my test-RDBMS I could only see the DGs of the first ASM.
So I created a SR(7275580.994) at MetaLink to ask how to do it and wether or not it's supported in single instance or RAC.
To summarize the answers:
  • It is possible to run multiple ASMs on a node
  • One RDBMS can onlybe served by one ASM
  • all the GUIs are not aware of multiple ASMs
  • it's not supported in RAC
Even these answers are worth some forther testcases.
I have 2 ASMs: +ASM (default) and +ASM2 (2nd, with changed DB_UNIQUE_NAME and ASM_DISKSTRING) and one RDBMS. DG ASM1_DG created in +ASM, DG ASM2_DG created in +ASM2.

  1. Test 1
    +ASM up
    +ASM2 up
    => only ASM1_DG1 visible in RDBMS .
  2. Test 2
    +ASM down
    +ASM2 up
    => ASM2_DG1 visible
    ! create tablespace in ASM2_DG1
  3. Test 3
    +ASM up
    +ASM2 up
    - restart RDBMS
    => only ASM2_DG1 visible in RDBMS
  4. Test 4
    +ASM up
    +ASM2 up
    => DBCA only shows ASM1_DG1: 
  5.  Test 5
    • shutdown +ASM
    • switch ASM1_DG to +ASM2 (some fiddling with ASM_DISKSTRING and symlinks)
    • mount ASM1_DG in +ASM2
    • create TS in ASM1_DG
    • shutdown RDBMS and +ASM2
    • switch ASM1_DG back to +ASM
    • startup +ASM, +ASM2 and rdbms
    • this constelation (ASM1_DG mounted on +ASM, ASM2_DG mounted on +ASM2) led to this error-message:

      ORA-01157: cannot identify/lock data file 8 - see DBWR trace file
      ORA-01110: data file 8: '+ASM1_DG/berx2/datafile/asm1_dg.256.676065339'
      ORA-17503: ksfdopn:2 Failed to open file +ASM1_DG/berx2/datafile/asm1_dg.256.67
      ORA-15001: diskgroup "ASM1_DG" does not exist or is not mounted
      ORA-15001: diskgroup "ASM1_DG" does not exist or is not mounted
Test5 led me to one guess: At startup a RDBMS can potentially access all ASMs, but the first attempt to access a DG pins the RDBMS to the ASM. Unfortunately I do not know how to prove this.
Within the SR Bug:6991236 was mentioned.
This Bug is not visible right now, but interpreting the header I can guess it's an enhancement request to allow or support multiple ASM-instances with differen OS-users in a cluster-environment. It's shown to be fixed (implemented) in 11gR2 and has status 98 (Suggestion Implemented). We will see when 11gR2 is out.

1 Kommentar:

Alex Gorbachev hat gesagt…

You might look into the mechanism of CSSD daemon that manages group membership. It's somewhere there that ASM instances are registered with and I think status of mounted ASM diskgroups is stored.