Posts mit dem Label datapatch werden angezeigt. Alle Posts anzeigen
Posts mit dem Label datapatch werden angezeigt. Alle Posts anzeigen

2019-11-06

Oracle Datapatch - out of place patches simplified

datapatch was introduced with Oracle 12.1 to simplify management of post-patching steps when a patch (or patchset) is applied.
As most software, it was written with best intentions but little knowledge how customers would use it.
One of these unforseen used case was parallel patching of multiple DBs in the same ORACLE_HOME at the same time.
Another problem was missing rollback files due to out of place patching.

Oracle addressed this in Bug 23170620.
The solution there is interesting:
the table sys.registry$sqlpatch is extended by a column patch_directory. It is a BLOB which contains the Zipped contents of patch directory.
By this patch, each database contains the sql files required to rollback a change done in the database. If datapatch does not find the proper rollback files in ORACLE_HOME/sqlpatch, it unzips the content of registry$sqlpatch.patch_directory into ORACLE_HGOME/sqlpatch and executes the scripts. For every new patch, a zip file is loaded into the table by datapatch.
With 18c an equal mechanism was introduced in sys.registry$sqlpatch_ru_info respectively DBA_REGISTRY_SQLPATCH_RU_INFO.

By it's own this feature is very nice. But it shows the importance of serious security settings.
Only SYS can write to these tables by default, and that should be limited for some good reasons:
If anyone can write to this table with evil intentions, he can create a zipfile with a content like
.
├── 1234.xml
├── 1234_apply.sql
├── 1234_rollback.sql
└── rollback_files
    └── rdbms
        └── admin
            ├── berx_bug1234_apply.sql
            └── berx_bug1234_rollback.sql

The XML contains some basic content
<?xml version="1.0" encoding="utf-8"?>
<sqlPatch ID="1234" uniquePatchID="5678" applicationPatch="NO" bundle="NO" jvm="NO" bundleSeries="NONE" bundleID="0"   startupMode="normal" estimatedTime="5">
  <patchDescription>berx' datapatch-escape</patchDescription>
  <component>CATALOG</component>
  <component>CATPROC</component>
  <sqlFiles>
    <file mode="rollback" new="yes" estimatedTime="5" component="CATALOG">rdbms/admin/berx_bug1234_rollback.sql</file>
    <file mode="apply" new="yes" estimatedTime="5" component="CATALOG">rdbms/admin/berx_bug1234_apply.sql</file>
  </sqlFiles>
</sqlPatch>

and the 1234_rollback.sql file can contain:
SET SERVEROUTPUT ON

REM Apply script for patch 1234/5678
DEFINE description = &1
DEFINE logfiledir = &2
DEFINE flags = &3
DEFINE bundle_series = &4

COLUMN sqlpatch_logfile NEW_VALUE full_logfile


SELECT '&logfiledir' || '1234_apply_' ||
       CASE WHEN (sys_context('userenv', 'cdb_name') IS NULL) 
            THEN name
            ELSE name || '_' || replace(sys_context('userenv', 'con_name'), '$')
             END || TO_CHAR(systimestamp, '_YYYYMonDD_HH24_MI_SS') ||
                    '.log' AS sqlpatch_logfile
  FROM v$database;

SPOOL &full_logfile

SET PAGESIZE 0
SELECT 'Starting rollback for patch 1234/5678 on ' ||
       SYSTIMESTAMP FROM dual;
SET PAGESIZE 10

BEGIN
    dbms_sqlpatch.patch_initialize(p_patch_id      => 1234,
                                   p_patch_uid     => 5678,
                                   p_flags         => '&flags',
                                   p_description   => '&description',
                                   p_action        => 'APPLY',
                                   p_logfile       => '&full_logfile',
                                   p_bundle_series => '&bundle_series');
END;
/

COLUMN install_file NEW_VALUE sql_script


ALTER SESSION SET CURRENT_SCHEMA = SYS;

SET PAGESIZE 0
SELECT 'Calling rdbms/admin/berx_bug1234_apply.sql on ' || SYSTIMESTAMP FROM dual;
SET PAGESIZE 10

PROMPT IGNORABLE ERRORS: NONE
select "berx_was_here!" from dual;
create table berx_was_here as select  * from dual;

! touch /tmp/berx_was_here


BEGIN dbms_sqlpatch.patch_finalize; END;
/

SET PAGESIZE 0
SELECT 'Finished apply for patch 1234/5678 on' ||
       SYSTIMESTAMP FROM dual;
SET PAGESIZE 10

SPOOL off

This specific script creates only one table and a file in /tmp but it could do worse.

At the next run of datapatch (might it be due to a regular patch session or an one-off patch with post-patching steps), will execute this script. And as the script then is located in ORACLE_HOME/sqlpatch, all other databases which instances use this ORACLE_HOME and get patched will be affected also (and get the zip-file added to their sys.registry$sqlpatch).

2015-10-10

12c datapatch can fail with manual out of place patching

datapatch can fail if doing manual out of place patching, when the new ORACLE_HOME is not cloned from previous one but a fresh installation. The problem is a one-of patch in the old ORACLE_HOME, which is included in something else (like a PSU) in the new ORACLE_HOME.
In my specific situation the old O_H had PSU 12.1.0.2.3 + some one-off + Patch:21355879.
This Patch is not needed anymore in my new O_H (12.1.0.2.4 + some other one-off) - I checked on My Oracle Support - Patches & Updates:

So no need (and no simple way) to add this patch to the new O_H at all.

But when running datapatch within the new O_H, it throws this error and refuses to continue:
Error: prereq checks failed!
  patch 21355879: rollback script /appl/oracle/product/rdbms_121024Jc/sqlpatch/21355879/19095143/21355879_rollback.sql does not exist
Prereq check failed, exiting without installing any patches.

It's obvious the patch  21355879 is not installed within the new O_H. But somehow ( I did not check the exact internal function) datapatch checks the view DBA_REGISTRY_SQLPATCH and finds PATCH_ID:21355879. With a statement similar to
SELECT XMLSerialize(CONTENT dbms_sqlpatch.opatch_registry_state INDENT) from dual;
it finds the patchuid 19095143. The base table seems to be OPATCH_XINV_TAB.  (the other patches within 21355879 are not in this XML and doesn't require a rollback script).

So the database repository says a rollback script should be present, but it's not installed in the new O_H as Oracle calls it not required anymore.

The clean solution would be to patch every new O_H in the exact sequence of all it predecessors - this can be time consuming and also challenging when different branches with conflicting one-offs needs to be merged - even all these conflicting one-offs are merged in a later PSU and not needed anymore.

We decided to do the easier way and just copy all patch directories from old onto new O_H.

12c datapatch - take care of parallel patching

datapatch is a nice new feature in recent Oracle database installations. It helps to ensure the databases objects match the binaries after any kind of patching and so avoid situations which can be an operational and support nightmare - and very hard to identify.

Problem


Unfortunately it has some drawbacks as well.
One of those I hit recently when running datapatch on 2 instances which uses the same ORACLE_HOME.
At some time in it's progress datapatch uses dbms_qopatch.get_opatch_lsinventory, which uses the external table SYS.OPATCH_XML_INV. This has the preprocessor $ORACLE_HOME/QOPatch/qopiprep.bat. This script executes
$ORACLE_HOME/OPatch/opatch lsinventory -xml  $ORACLE_HOME/QOpatch/xml_file.xml -retry 0 -invPtrLoc $ORACLE_HOME/oraInst.loc >> $ORACLE_HOME/QOpatch/stout.txt
`echo "UIJSVTBOEIZBEFFQBL" >> $ORACLE_HOME/QOpatch/xml_file.xml`
echo `cat $ORACLE_HOME/QOpatch/xml_file.xml`
rm $ORACLE_HOME/QOpatch/xml_file.xml
rm $ORACLE_HOME/QOpatch/stout.txt
Maybe you see already what will happen when 2 different instances execute this step at the same time:
2 different opatch lsinventory -xml instances are writing into the same file. A very efficient was to corrupt the xml file, as they write to the same file $ORACLEHOME/QPatch/xml_file.xml

in one db I got this error:
Determining current state...DBD::Oracle::st execute failed: ORA-20001: Latest xml inventory is not loaded into table
ORA-06512: at "SYS.DBMS_QOPATCH", line 1937
ORA-06512: at "SYS.DBMS_QOPATCH", line 1259 (DBD ERROR: error possibly near <*> indicator at char 143 in 'SELECT description, startup_mode
               FROM XMLTable('/InventoryInstance/patches/patch[patchID=21573304]'
                    PASSING <*>dbms_qopatch.get_opatch_lsinventory
                    COLUMNS description VARCHAR2(100) PATH 'patchDescription',
                            startup_mode VARCHAR2(7) PATH 'sqlPatchDatabaseStartupMode')') [for Statement "SELECT description, startup_mode
               FROM XMLTable('/InventoryInstance/patches/patch[patchID=21573304]'
                    PASSING dbms_qopatch.get_opatch_lsinventory
                    COLUMNS description VARCHAR2(100) PATH 'patchDescription',
                            startup_mode VARCHAR2(7) PATH 'sqlPatchDatabaseStartupMode')"] at /appl/oracle/product/rdbms_121024Jc/sqlpatch/sqlpatch.pm line 1368, <LOGFILE> line 73.

in the other
verify_queryable_inventory returned ORA-31011: XML parsing failed
ORA-19202: Error occurred in XML processing
LPX-00229: input source is empty
ORA-06512: at "SYS.XMLTYPE", line 272
ORA-06512: at line 1

Queryable inventory could not determine the current opatch status.
Execute 'select dbms_sqlpatch.verify_queryable_inventory from dual'

But there might be different errors possible as well - all based on the concurrency issue here.

Workaround

In our environment the it's quite simple to workaround such issues: We jave scripts which do all the steps during patching, so the "apply datapatch" module just needs a small extension which ckecks for a "latch" (existence of a file in our case) and only continues if it can grab this file. Otherwise it sleeps for a short time.

Solution

Oracle could easily use a filename like  $ORACLEHOME/QPatch/xml_file.$$.xml instead. I'm not sure if it's worth the effort to fight through the perimeters in MOS.

Update

(2015-12-28)
I found 12.1:Parallel Execution Of Datapatch during Patching or Manual upgrade fails with error " Queryable inventory could not determine the current opatch status " (Doc ID 2054286.1) today which describes exactly this behavior.
Patch:19215058 solves the issue by implementing
DBSID=$ORACLE_SID
...
$ORACLE_HOME/QOpatch/xml_file_$DBSID.xml
...