2023-05-06

Outer joins - where to filter?

Last week a colleague asked me during a coffee break why sometimes some outer joins does not show lines with NULL values for parents without children. 

Of course we had no sample schema at hand, but the rough situation he described was 

If I do a outer ANSI join, but then filter in the WHERE for a childs column, the outer join ( parents without children) disappears. 

The way he described it, it doesn't look right. But before doing the headless chicken run, I decided to read the docu and do some testcases. 

My results are simple: the SQL is correct - it just needs to be written correct! 


Here is my little testcase. 

All the tables and queries can be found on livesql

I have 2 Tables: PAR - and CLD, where CLD has a column P_ID with a foreign key to PAR.ID

The description of my query can be something like 

Give me all Parents and their Children with an AGE_RANGE greater than 10.

The first approach is 


select /* ANSI 1 */ pr.*, cd.*
from par pr left outer join cld cd on 
    (pr.id = cd.p_id and age_range > 10)
;

ID P_NAME ID_1 P_ID C_NAME AGE_RANGE
1 P_one        
2 P_two        
3 P_three        
4 P_four 4 4 4_20 20

That was the expected output. 

But the filter age_range > 10 in the WHERE part of the SQL changes the result: 

select /* ANSI 2 */  pr.*, cd.*
from par pr left outer join cld cd on pr.id = cd.p_id 
where age_range > 10
;  

ID P_NAME ID_1 P_ID C_NAME AGE_RANGE
4 P_four 4 4 4_20 20


The difference is obvious, the filter in the WHERE part made all parents without children disappear
A small clue can be found in the explain plans. 

The Plan for ANSI 1


Plan hash value: 2247011116
---------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
---------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 4 | 344 | 6 (0)| 00:00:01 |
|* 1 | HASH JOIN OUTER | | 4 | 344 | 6 (0)| 00:00:01 |
| 2 | TABLE ACCESS FULL| PAR | 4 | 120 | 3 (0)| 00:00:01 |
|* 3 | TABLE ACCESS FULL| CLD | 1 | 56 | 3 (0)| 00:00:01 |
---------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - access("PR"."ID"="CD"."P_ID"(+))
3 - filter("AGE_RANGE"(+)>10)


The Plan for ANSI 2


Plan hash value: 3984166157
---------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
---------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 86 | 4 (0)| 00:00:01 |
| 1 | NESTED LOOPS | | 1 | 86 | 4 (0)| 00:00:01 |
| 2 | NESTED LOOPS | | 1 | 86 | 4 (0)| 00:00:01 |
|* 3 | TABLE ACCESS FULL | CLD | 1 | 56 | 3 (0)| 00:00:01 |
|* 4 | INDEX UNIQUE SCAN | PAR_PK | 1 | | 0 (0)| 00:00:01 |
| 5 | TABLE ACCESS BY INDEX ROWID| PAR | 1 | 30 | 1 (0)| 00:00:01 |
---------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
3 - filter("CD"."AGE_RANGE">10)
4 - access("PR"."ID"="CD"."P_ID")

In ANSI 1 we have a HASH JOIN OUTER and on CLD a filter("AGE_RANGE"(+)>10).  

In ANSI 2 instead there are NESTED LOOPS and a filter("CD"."AGE_RANGE">10)

Both are correct, but they answer different requirements! 

ANSI 1: 

Give me all Parents; show me their Children also, but only if the Childrens AGE_RANGE is greater than 10. 

 

ANSI 2:

Create a result set with all Parrents and their children if any exist; then filter this resultset for all rows which match AGE_RANGE is greater than 10. 

To be more explicit, I also show another variation of the ANSI 1 which generates the same result:


select /*+ ANSI 3 */  pr.*, cd.*
from par pr left outer join 
     (select /*+ NO_MERGE */ * from cld where age_range > 10 ) cd 
                                   on pr.id = cd.p_id 
;

 



The same can be applied in Oracle notation, and here a little trap exists: 
A "simple" translation of ANSI 1 to Oracle syntax might look like


select /* Oracle 2 */  pr.*, cd.*
from par pr , cld cd
where pr.id = p_id (+)
  and cd.age_range > 10
;

But it is labeled Oracle 2 as it generates the identical result set as ANSI 2! 


For a correct translation of ANSI 1, this query is required: 


select /* Oracle 1 */  pr.*, cd.*
from par pr , cld cd
where pr.id = p_id (+)
  and cd.age_range (+)> 10 
;


So be precise which query you write - it should match the question you ask!


All the tables, inserts, queries and explain plans are visible on livesql; the statements also on my git.  

2023-05-01

GoldenGates credentialstore content

 Oracle GoldenGate is a software which enables logical data extract and replication among different databases. It's not limited to Oracle but also supports several other databases and even streaming platforms like Kafka. 

To extract, but also apply data from or to a database, credentials (and of course grants) are required in the targets. To avoid having credentials as clear text in some scripts, GoldenGate uses a credential store. Technically it's a well known wallet in the directory $GG_HOME/dircrd. To use these credentials, in the commandline tool ggsci commands like



GGSCI (myhost) 1> ALTER CREDENTIALSTORE ADD USER usr5 PASSWORD pwd5 ALIAS usr5@SRV5 domain dom1

Credential store altered.
and

GGSCI (myhost) 2> info credentialstore 

Reading from credential store:

Default domain: OracleGoldenGate

  Alias: ogg@dg
  Userid: ogg@dg

  Alias: c##ogg@dgpdb1
  Userid: c##ogg@dgpdb1

  Alias: c##ogg@dgc
  Userid: c##ogg@dgc


Other domains: 

  dom1

To view other domains, use INFO CREDENTIALSTORE DOMAIN <domain>

can be used.

To see what's really stored in the wallet - in case I need to extract some data from there, a well known mkstore (from my oracle binary installation) can be used: 


  oracle@myhost:/u00/app/oracle/product/gg21_ora/dircrd/ [DGC] mkstore -wrl . -list
Oracle Secret Store Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2022, Oracle and/or its affiliates. All rights reserved.

Oracle Secret Store entries: 
dom1@#3#@16908a4d-ali1
OracleGoldenGate@#3#@55928a5d-c##ogg@dgc
OracleGoldenGate@#3#@c478f3a2-c##ogg@dgpdb1
OracleGoldenGate@#3#@fabe8b1-ogg@dg
  
And to get the content of an element:

oracle@myhost:/u00/app/oracle/product/gg21_ora/dircrd/ [DGC] mkstore -wrl . -viewEntry OracleGoldenGate@#3#@fabe8b1-ogg@dg
Oracle Secret Store Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2022, Oracle and/or its affiliates. All rights reserved.

OracleGoldenGate@#3#@fabe8b1-ogg@dg = {pwd_cred_type}@#4#@NEVER_EXPIRE@#5#@ogg@dg_tns_alias@#111#@ogg_pwd


The first detail to notice here: no password is required to read the wallet. This makes it easy to have a centralized wallet and distribute it to all GoldenGate instances which require these credentials. I did not find any method to enforce a wallets password in GoldenGate. 

Another detail which catches my eyes are some syntax and keywords used in the wallets entries:

OracleGoldenGate@#3#@fabe8b1-ogg@dg = {pwd_cred_type}@#4#@NEVER_EXPIRE@#5#@ogg@dg_tns_alias@#111#@ogg_pwd

At first, there are some delimiters within the text: @#3#@. The meaning of the numbers isn't documented anywhere, but they can be deduced by the syntax of the line. 
OracleGoldenGate matches the domainogg@dg is the aliasogg@dg_tns_alias the username and of course ogg_pwd is the password. All relevant bits of information can be retrieved from the wallet without significant problem. 
Unfortunately, all the keywords are not defined anywhere, but at least I could find them in the binary libggnnzitp.so:

  $  strings libggnnzitp.so | grep -B 20  NEVER_EXPIRE
ldap username not set.
No key in bootstrap wallet with key name=
...for map name=
No map in bootstrap wallet with name=
ldap rootname not set.
ldap farmname not set.
ldap url not set.
bootstrap wallet entry key not set.
bootstrap wallet entry map (alias) not set.
bootstrap wallet path not set.
@#3#@
{pwd_cred_type}
@#5#@
@#111#@
nzcsfCCS_CloseCredStore
Error closing Wallet store
Error closing LDAP store
Error freeing cred store list
No Wallet specified to close cred store
@#4#@
NEVER_EXPIRE

All these bits should be a good starting point for everyone who wants to dig deeper in GoldenGates credential management.