0.背景 :生产环境出现gc块等待,需要补丁更新

oracle linux 7.9

RAC 双节点,  版本19.12

----Bug 32685989 - PX Slave Session Hang While Waiting For 'gc cr request', 'gc cr block lost' continuously ( Doc ID 32685989.8 )

---Bug 33676842 - Invalid Message Discarding 'gc block lost' Cause Waiting For 'gc cr request' and Subsequent DB Open Time Out ( Doc ID 33676842.8 )

----Bug 34482043 - Some "gc cr multi block mixed" wait outliers cause a performance regression in 19c ( Doc ID 34482043.8 )

Bug 的 Merge 补丁:

https://updates.oracle.com/download/34622987.html


1.启动crs报错如下:

[root@rac2 ~]#/u01/app/grid/product/19.0.0/crs/install/rootcrs.sh  -postpatch

CRS-6706: Oracle Clusterware Release patch level ('3329448500') does not match Software patch level ('2845761567'). Oracle Clusterware cannot be started.

CRS-4000: Command Start failed, or completed with errors.

[root@rac2 ~]# /oracle/app/19c/grid/bin/crsctl start crs

CRS-6706: Oracle Clusterware Release patch level ('3329448500') does not match Software patch level ('2845761567'). Oracle Clusterware cannot be started.

CRS-4000: Command Start failed, or completed with errors.

2.查看补丁应用日志:

[root@rac2 ~]# more  /u01/app/grid/base/crsdata/szpmsb02cimrac-db07/crsconfig/crs_postpatch_apply_inplace_szpmsb02cimrac-db07_2025-01-01_09-24-05AM.log

2025-01-01 09:24:06: The 'ROOTCRS_PATCHINFO' status is SUCCESS

2025-01-01 09:24:06: Invoking "/u01/app/grid/product/19.0.0/grid/bin/cluutil -ckpt -global -oraclebase /u01/app/grid/base -listckpt -name ROOTCRS_PATCHINFO -pname NODENAME_LIST"

2025-01-01 09:24:06: trace file=/u01/app/grid/base/crsdata/szpmsb02cimrac-db07/crsconfig/cluutil2.log

2025-01-01 09:24:06: Running as user grid: /u01/app/grid/product/19.0.0/grid/bin/cluutil -ckpt -global -oraclebase /u01/app/grid/base -listckpt -name ROOTCRS_PATCHINFO -pname NODENAME_LI ST

2025-01-01 09:24:07: Removing file /tmp/wUcdA39bG1

2025-01-01 09:24:07: Successfully removed file: /tmp/wUcdA39bG1

2025-01-01 09:24:07: pipe exit code: 0

2025-01-01 09:24:07: /bin/su successfully executed

2025-01-01 09:24:07: NODENAME_LIST=szpmsb02cimrac-db07,szpmsb02cimrac-db08

2025-01-01 09:24:07: Succeeded to get property value:NODENAME_LIST=szpmsb02cimrac-db07,szpmsb02cimrac-db08

2025-01-01 09:24:07: doing checks for postpatch rerun

2025-01-01 09:24:07: Invoking "/u01/app/grid/product/19.0.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid/base -chkckpt -name ROOTCRS_PREPATCH -status"

2025-01-01 09:24:07: trace file=/u01/app/grid/base/crsdata/szpmsb02cimrac-db07/crsconfig/cluutil3.log

2025-01-01 09:24:07: Running as user grid: /u01/app/grid/product/19.0.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid/base -chkckpt -name ROOTCRS_PREPATCH -status

2025-01-01 09:24:07: Removing file /tmp/kJ_Kqil2zm

2025-01-01 09:24:07: Successfully removed file: /tmp/kJ_Kqil2zm

2025-01-01 09:24:07: pipe exit code: 0

2025-01-01 09:24:07: /bin/su successfully executed

2025-01-01 09:24:07: FAIL

2025-01-01 09:24:07: The 'ROOTCRS_PREPATCH' is either in START/FAILED state

2025-01-01 09:24:07: Invoking "/u01/app/grid/product/19.0.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid/base -chkckpt -name ROOTCRS_PREPATCH -status"

2025-01-01 09:24:07: trace file=/u01/app/grid/base/crsdata/szpmsb02cimrac-db07/crsconfig/cluutil4.log

2025-01-01 09:24:07: Running as user grid: /u01/app/grid/product/19.0.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid/base -chkckpt -name ROOTCRS_PREPATCH -status

2025-01-01 09:24:07: Removing file /tmp/TVuFPBlBm0

2025-01-01 09:24:07: Successfully removed file: /tmp/TVuFPBlBm0

2025-01-01 09:24:07: pipe exit code: 0

2025-01-01 09:24:07: /bin/su successfully executed

2025-01-01 09:24:07: FAIL

2025-01-01 09:24:07: The 'ROOTCRS_PREPATCH' status is FAILED

2025-01-01 09:24:07: Executing cmd: /u01/app/grid/product/19.0.0/grid/bin/clsecho -p has -f clsrsc -m 739

2025-01-01 09:24:07: Executing cmd: /u01/app/grid/product/19.0.0/grid/bin/clsecho -p has -f clsrsc -m 739

2025-01-01 09:24:07: Command output:

>  CLSRSC-739: cannot run the postpatch operation because the prepatch operation did not complete successfully  

>End Command output

2025-01-01 09:24:07: CLSRSC-739: cannot run the postpatch operation because the prepatch operation did not complete successfully

2025-01-01 09:24:07: ###### Begin DIE Stack Trace ######

2025-01-01 09:24:07:     Package         File                 Line Calling    

2025-01-01 09:24:07:     --------------- -------------------- ---- ----------

2025-01-01 09:24:07:  1: main            rootcrs.pl            357 crsutils::dietrap

2025-01-01 09:24:07:  2: crspatch        crspatch.pm          2905 main::__ANON__

2025-01-01 09:24:07:  3: crspatch        crspatch.pm          2289 crspatch::postPatchRerunCheck

2025-01-01 09:24:07:  4: crspatch        crspatch.pm          2101 crspatch::crsPostPatchCkpts

2025-01-01 09:24:07:  5: crspatch        crspatch.pm           421 crspatch::crsPostPatch

2025-01-01 09:24:07:  6: main            rootcrs.pl            370 crspatch::new

2025-01-01 09:24:07: ####### End DIE Stack Trace #######

2025-01-01 09:24:07:  checkpoint has failed

3.检查补丁情况

[root@rac2 ~]# /oracle/app/19c/grid/bin/crsctl query crs releasepatch

[root@rac2 ~]# /oracle/app/19c/grid/bin/kfod op=patches

[root@rac2 ~]# su - grid

[grid@rac2 ~]$ opatch lsinventory

oracle:

[oracle@szrac01 34622987]$ $ORACLE_HOME/OPatch/opatch lsinventory|grep 34622987

Patch  34622987     : applied on Fri Nov 01 15:36:19 CST 2024

[oracle@szrac01 34622987]$  

grid:

[grid@szrac01 tmp]$ $ORACLE_HOME/OPatch/opatch lsinventory|grep 34622987

Patch  34622987  : applied on Fri Nov 01 15:34:23 CST 2024

4.mos 文章参考解决

[root@rac2 ~]# /u01/app/grid/product/19.0.0/grid/bin/crsctl stat res -t

[root@rac2 ~]# /u01/app/grid/product/19.0.0/grid/bin/crsctl start crs

[root@rac2 ~]# /u01/app/grid/product/19.0.0/grid/crs/install/rootcrs.sh  -prepatch

[root@rac2 ~]# /u01/app/grid/product/19.0.0/grid/crs/install/rootcrs.sh  -unlock

[root@rac2 ~]# /u01/app/grid/product/19.0.0/grid/bin/clscfg -localpatch

[root@rac2 ~]# /u01/app/grid/product/19.0.0/grid/crs/install/rootcrs.sh  -lock

[root@rac2 ~]# /u01/app/grid/product/19.0.0/grid/bin/crsctl start crs


CRS-6706: Oracle Clusterware Release patch level ('nnn') does not match Software patch level ('mmm') (Doc ID 1639285.1)


CRS-6706: Oracle Clusterware Release patch level (

CRS-6706: Oracle Clusterware Release patch level (

CRS-6706: Oracle Clusterware Release patch level (

CRS-6706: Oracle Clusterware Release patch level (