Posts mit dem Label parallel werden angezeigt. Alle Posts anzeigen
Posts mit dem Label parallel werden angezeigt. Alle Posts anzeigen

2026-05-21

Single Server DFO

 Another improvement similar to last post is Single Server DFO. While in previous post the serial work was done by the query coordinator, that activity can also be executed by a single member of a parallel group set. 

The SQL again: 
select -- BX_13
       /*+ OPT_PARAM('parallel_degree_policy' 'auto') parallel(4) GATHER_PLAN_STATISTICS monitor */ 
       OBJECT_TYPE, count(*) c
from A_O
where rownum < 25000000 
group by OBJECT_TYPE
fetch first 7 rows only; 
select * from dbms_xplan.display_cursor( format =>'ALLSTATS LAST +PARALLEL');

The execution plan now changes from 

Plan hash value: 3726916758
 
---------------------------------------------------------------------------------------------------
| Id  | Operation                                         | Name     |    TQ  |IN-OUT| PQ Distrib |
---------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                                  |          |        |      |            |
|*  1 |  COUNT STOPKEY                                    |          |        |      |            |
|   2 |   PX COORDINATOR                                  |          |        |      |            |
|   3 |    PX SEND QC (RANDOM)                            | :TQ20001 |  Q2,01 | P->S | QC (RAND)  |
|*  4 |     COUNT STOPKEY                                 |          |  Q2,01 | PCWC |            |
|   5 |      VIEW                                         |          |  Q2,01 | PCWP |            |
|*  6 |       SORT GROUP BY STOPKEY                       |          |  Q2,01 | PCWP |            |
|   7 |        PX RECEIVE                                 |          |  Q2,01 | PCWP |            |
|   8 |         PX SEND HASH                              | :TQ20000 |        | S->P | HASH       |
|*  9 |          COUNT STOPKEY                            |          |        |      |            |
|  10 |           PX COORDINATOR                          |          |        |      |            |
|  11 |            PX SEND QC (RANDOM)                    | :TQ10000 |  Q1,00 | P->S | QC (RAND)  |
|* 12 |             COUNT STOPKEY                         |          |  Q1,00 | PCWC |            |
|  13 |              PX BLOCK ITERATOR                    |          |  Q1,00 | PCWC |            |
|* 14 |               TABLE ACCESS STORAGE FULL FIRST ROWS| A_O      |  Q1,00 | PCWP |            |
---------------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   1 - filter(ROWNUM<=7)
   4 - filter(ROWNUM<=7)
   6 - filter(ROWNUM<=7)
   9 - filter(ROWNUM<25000000)
  12 - filter(ROWNUM<25000000)

to 

Plan hash value: 2053989564
 
----------------------------------------------------------------------------------------------------
| Id  | Operation                                          | Name     |    TQ  |IN-OUT| PQ Distrib |
----------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                                   |          |        |      |            |
|*  1 |  COUNT STOPKEY                                     |          |        |      |            |
|   2 |   PX COORDINATOR                                   |          |        |      |            |
|   3 |    PX SEND QC (RANDOM)                             | :TQ10002 |  Q1,02 | P->S | QC (RAND)  |
|*  4 |     COUNT STOPKEY                                  |          |  Q1,02 | PCWC |            |
|   5 |      VIEW                                          |          |  Q1,02 | PCWP |            |
|*  6 |       SORT GROUP BY STOPKEY                        |          |  Q1,02 | PCWP |            |
|   7 |        PX RECEIVE                                  |          |  Q1,02 | PCWP |            |
|   8 |         PX SEND HASH                               | :TQ10001 |  Q1,01 | S->P | HASH       |
|   9 |          BUFFER SORT                               |          |  Q1,01 | SCWP |            |
|* 10 |           COUNT STOPKEY                            |          |  Q1,01 | SCWP |            |
|  11 |            PX RECEIVE                              |          |  Q1,01 | SCWP |            |
|  12 |             PX SEND 1 SLAVE                        | :TQ10000 |  Q1,00 | P->S | 1 SLAVE    |
|* 13 |              COUNT STOPKEY                         |          |  Q1,00 | PCWC |            |
|  14 |               PX BLOCK ITERATOR                    |          |  Q1,00 | PCWC |            |
|* 15 |                TABLE ACCESS STORAGE FULL FIRST ROWS| A_O      |  Q1,00 | PCWP |            |
----------------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   1 - filter(ROWNUM<=7)
   4 - filter(ROWNUM<=7)
   6 - filter(ROWNUM<=7)
  10 - filter(ROWNUM<25000000)
  13 - filter(ROWNUM<25000000)

We can see 3 consecutive DFOs now: TQ10000, TQ10001 and TQ10002. But there is a new Operation in LineID:12: PX SEND 1 SLAVE. This indicates the data is sent only to 1 worker in TQ10001. There is also no 20.000 TFOs anymore. The BUFFER SORT in LineID:9 is kind of a NOOP - there is nothing to SORT or BUFFER, still it's a requirement in this type of parallel plan pieces.

The IN-OUT now shows SCWP to indicate the difference.

SQL Monitor also show these changes:
But you have to know what to look for. Only the PX SEND 1 SLAVE indicates the special treatment of  that DFO. 
In plan_table.OTHER_TAG its SINGLE_COMBINED_WITH_PARENT.







In SQL Monitors Parallel-tab it can look as if there is skew in Parallel Server activity; which is technically true but in this case it's on purpose. Don't start corrective actions in this case!

It might look as if there is no reason for this improvement: Only one process is working for this plan lines anyhow. Still, with Single Server DFO, the additional resources of the parallel worker (e.g. dedicated PGA) can be used and the QC is free for other tasks in the execution. 
Of course there is also an underscore-parameter to control this behavior individually. This feature is enabled by default but requires Back to Parallel.



2026-05-20

Back to Parallel

Recently I stumbled across a feature which exists for quite some time already. It was implemented in 12.1.

The plan showed a parallel execution which somewhere in the middle had a serializing execution. But later on there were obvious other parallel activity. 
That made me research - and then generate a testcase.
I have generated a table A_O as SELECT * FROM ALL_OBJECTS - just to get some data and keep the plan simple. 
First the SQL as it was executed pre 12.1

select -- BX_08
       /*+ OPT_PARAM('parallel_degree_policy' 'limited') parallel(4) GATHER_PLAN_STATISTICS monitor */ 
       OBJECT_TYPE, count(*) c
from A_O
where rownum < 25000000 
group by OBJECT_TYPE
fetch first 7 rows only; 
select * from dbms_xplan.display_cursor( format =>'ALLSTATS LAST +PARALLEL');
with a plan
Plan hash value: 2817860580
 
-------------------------------------------------------------------------------------------------------
| Id  | Operation                                    | Name     | E-Rows |    TQ  |IN-OUT| PQ Distrib |
-------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                             |          |        |        |      |            |
|*  1 |  COUNT STOPKEY                               |          |        |        |      |            |
|   2 |   VIEW                                       |          |      7 |        |      |            |
|*  3 |    SORT GROUP BY STOPKEY                     |          |      7 |        |      |            |
|*  4 |     COUNT STOPKEY                            |          |        |        |      |            |
|   5 |      PX COORDINATOR                          |          |        |        |      |            |
|   6 |       PX SEND QC (RANDOM)                    | :TQ10000 |      7 |  Q1,00 | P->S | QC (RAND)  |
|*  7 |        COUNT STOPKEY                         |          |        |  Q1,00 | PCWC |            |
|   8 |         PX BLOCK ITERATOR                    |          |      7 |  Q1,00 | PCWC |            |
|   9 |          TABLE ACCESS STORAGE FULL FIRST ROWS| A_O      |      7 |  Q1,00 | PCWP |            |
-------------------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   1 - filter(ROWNUM<=7)
   3 - filter(ROWNUM<=7)
   4 - filter(ROWNUM<25000000)
   7 - filter(ROWNUM<25000000)
   
 
Note
-----
   - Degree of Parallelism is 4 because of hint
We can see in lineID:6 the P->S clearly shows: all data is delivered to the query coordinator. This does the filter for  rownum < 25000000 (this must be in serial mode), and afterwards does the GROUP BY in LineID:3 ← this could be executed in parallel again, but as the QC is already involved, no further parallel executions occur. 


This all changes with 12.1: 

Plan hash value: 3726916758
 
---------------------------------------------------------------------------------------------------
| Id  | Operation                                         | Name     |    TQ  |IN-OUT| PQ Distrib |
---------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                                  |          |        |      |            |
|*  1 |  COUNT STOPKEY                                    |          |        |      |            |
|   2 |   PX COORDINATOR                                  |          |        |      |            |
|   3 |    PX SEND QC (RANDOM)                            | :TQ20001 |  Q2,01 | P->S | QC (RAND)  |
|*  4 |     COUNT STOPKEY                                 |          |  Q2,01 | PCWC |            |
|   5 |      VIEW                                         |          |  Q2,01 | PCWP |            |
|*  6 |       SORT GROUP BY STOPKEY                       |          |  Q2,01 | PCWP |            |
|   7 |        PX RECEIVE                                 |          |  Q2,01 | PCWP |            |
|   8 |         PX SEND HASH                              | :TQ20000 |        | S->P | HASH       |
|*  9 |          COUNT STOPKEY                            |          |        |      |            |
|  10 |           PX COORDINATOR                          |          |        |      |            |
|  11 |            PX SEND QC (RANDOM)                    | :TQ10000 |  Q1,00 | P->S | QC (RAND)  |
|* 12 |             COUNT STOPKEY                         |          |  Q1,00 | PCWC |            |
|  13 |              PX BLOCK ITERATOR                    |          |  Q1,00 | PCWC |            |
|* 14 |               TABLE ACCESS STORAGE FULL FIRST ROWS| A_O      |  Q1,00 | PCWP |            |
---------------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   1 - filter(ROWNUM<=7)
   4 - filter(ROWNUM<=7)
   6 - filter(ROWNUM<=7)
   9 - filter(ROWNUM<25000000)
  12 - filter(ROWNUM<25000000)

Here we see the serial execution between lineID:10 and lineID:8, but the GROUP BY is done in TQ20002 again. You can see the top 2 TQs are 20.000+ whereas the bottom one is 10.000. 
Also the IN-OUT (P->S and S->P) column shows quite fine the switch from parallel execution to serial one. In plan_table.OTHER_TAG its PARALLEL_TO_SERIAL and  PARALLEL_FROM_SERIAL.
SQL Monitor Report reflects this behavior properly. 

This feature is enabled by default (and it's cost is calculated) with parallel_degree_policy=auto
Of course there is also an underscore-parameter to control this behavior individually. 

2022-11-20

dynamic paramater change when CPUs (dis)appear

 In my last 2 blogposts I covered how Oracle instance parameters change based on cpu_count at startup, and also what's the minimum requirement for SGA size

But there is more to discover! Modern systems can change the number of CPUs while running. This feature is quite popular in cloud environments but in fact was already available more than 20 years ago

To see if this has any impact on my (21c) sandbox instance, I used a small "trick" in Linux: 
As root I can set a CPU offline (or back online) by 

echo 0 > /sys/devices/system/cpu/cpu7/online

This CPU becomes invisible to the processes - it also "disappears" in /proc/cpuinfo


At every change, the alert.log shows an entry like (The sampling frequency was 70 sec on my system)

2022-11-20T14:09:12.762038+00:00
Detected change in CPU count to 7
* Load Monitor used for high load check
* New Low - High Load Threshold Range = [0 - 0]

In this case my test was quite small (only 1..8 PCUs) but at least in that range only a small set of parameters changed:

PARAMETER
cpu_count
cpu_min_count
job_queue_processes
parallel_max_servers
parallel_servers_target


cpu_count and cpu_min_count makes sense, the remaining parameters are only related to


parallel and job processes. All of them are following the simple formula cpu_count * 20

This follows the "normal" derived values for these parameters as shown before. 


Again this is nothing tremendous spectacular, but worth to know for performance engineering or analysis in volatile systems: Some parameters change and might lead to unexpected behavior, if the change isn't taken into consideration. 

2015-10-10

12c datapatch - take care of parallel patching

datapatch is a nice new feature in recent Oracle database installations. It helps to ensure the databases objects match the binaries after any kind of patching and so avoid situations which can be an operational and support nightmare - and very hard to identify.

Problem


Unfortunately it has some drawbacks as well.
One of those I hit recently when running datapatch on 2 instances which uses the same ORACLE_HOME.
At some time in it's progress datapatch uses dbms_qopatch.get_opatch_lsinventory, which uses the external table SYS.OPATCH_XML_INV. This has the preprocessor $ORACLE_HOME/QOPatch/qopiprep.bat. This script executes
$ORACLE_HOME/OPatch/opatch lsinventory -xml  $ORACLE_HOME/QOpatch/xml_file.xml -retry 0 -invPtrLoc $ORACLE_HOME/oraInst.loc >> $ORACLE_HOME/QOpatch/stout.txt
`echo "UIJSVTBOEIZBEFFQBL" >> $ORACLE_HOME/QOpatch/xml_file.xml`
echo `cat $ORACLE_HOME/QOpatch/xml_file.xml`
rm $ORACLE_HOME/QOpatch/xml_file.xml
rm $ORACLE_HOME/QOpatch/stout.txt
Maybe you see already what will happen when 2 different instances execute this step at the same time:
2 different opatch lsinventory -xml instances are writing into the same file. A very efficient was to corrupt the xml file, as they write to the same file $ORACLEHOME/QPatch/xml_file.xml

in one db I got this error:
Determining current state...DBD::Oracle::st execute failed: ORA-20001: Latest xml inventory is not loaded into table
ORA-06512: at "SYS.DBMS_QOPATCH", line 1937
ORA-06512: at "SYS.DBMS_QOPATCH", line 1259 (DBD ERROR: error possibly near <*> indicator at char 143 in 'SELECT description, startup_mode
               FROM XMLTable('/InventoryInstance/patches/patch[patchID=21573304]'
                    PASSING <*>dbms_qopatch.get_opatch_lsinventory
                    COLUMNS description VARCHAR2(100) PATH 'patchDescription',
                            startup_mode VARCHAR2(7) PATH 'sqlPatchDatabaseStartupMode')') [for Statement "SELECT description, startup_mode
               FROM XMLTable('/InventoryInstance/patches/patch[patchID=21573304]'
                    PASSING dbms_qopatch.get_opatch_lsinventory
                    COLUMNS description VARCHAR2(100) PATH 'patchDescription',
                            startup_mode VARCHAR2(7) PATH 'sqlPatchDatabaseStartupMode')"] at /appl/oracle/product/rdbms_121024Jc/sqlpatch/sqlpatch.pm line 1368, <LOGFILE> line 73.

in the other
verify_queryable_inventory returned ORA-31011: XML parsing failed
ORA-19202: Error occurred in XML processing
LPX-00229: input source is empty
ORA-06512: at "SYS.XMLTYPE", line 272
ORA-06512: at line 1

Queryable inventory could not determine the current opatch status.
Execute 'select dbms_sqlpatch.verify_queryable_inventory from dual'

But there might be different errors possible as well - all based on the concurrency issue here.

Workaround

In our environment the it's quite simple to workaround such issues: We jave scripts which do all the steps during patching, so the "apply datapatch" module just needs a small extension which ckecks for a "latch" (existence of a file in our case) and only continues if it can grab this file. Otherwise it sleeps for a short time.

Solution

Oracle could easily use a filename like  $ORACLEHOME/QPatch/xml_file.$$.xml instead. I'm not sure if it's worth the effort to fight through the perimeters in MOS.

Update

(2015-12-28)
I found 12.1:Parallel Execution Of Datapatch during Patching or Manual upgrade fails with error " Queryable inventory could not determine the current opatch status " (Doc ID 2054286.1) today which describes exactly this behavior.
Patch:19215058 solves the issue by implementing
DBSID=$ORACLE_SID
...
$ORACLE_HOME/QOpatch/xml_file_$DBSID.xml
...

2008-08-29

the 'Magic of 2'

There is a very popular pdf about parallel queries.
Why I quote it here?
to create another link , give you something to read and as a starting-point for my own tests.