0.背景 :生产环境出现gc块等待,需要补丁更新
oracle linux 7.9
RAC 双节点, 版本19.12
----Bug 32685989 - PX Slave Session Hang While Waiting For 'gc cr request', 'gc cr block lost' continuously ( Doc ID 32685989.8 )
---Bug 33676842 - Invalid Message Discarding 'gc block lost' Cause Waiting For 'gc cr request' and Subsequent DB Open Time Out ( Doc ID 33676842.8 )
----Bug 34482043 - Some "gc cr multi block mixed" wait outliers cause a performance regression in 19c ( Doc ID 34482043.8 )
Bug 的 Merge 补丁:
https://updates.oracle.com/download/34622987.html
1.启动crs报错如下:
[root@rac2 ~]#/u01/app/grid/product/19.0.0/crs/install/rootcrs.sh -postpatch
CRS-6706: Oracle Clusterware Release patch level ('3329448500') does not match Software patch level ('2845761567'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.
[root@rac2 ~]# /oracle/app/19c/grid/bin/crsctl start crs
CRS-6706: Oracle Clusterware Release patch level ('3329448500') does not match Software patch level ('2845761567'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.
2.查看补丁应用日志:
[root@rac2 ~]# more /u01/app/grid/base/crsdata/szpmsb02cimrac-db07/crsconfig/crs_postpatch_apply_inplace_szpmsb02cimrac-db07_2025-01-01_09-24-05AM.log
2025-01-01 09:24:06: The 'ROOTCRS_PATCHINFO' status is SUCCESS
2025-01-01 09:24:06: Invoking "/u01/app/grid/product/19.0.0/grid/bin/cluutil -ckpt -global -oraclebase /u01/app/grid/base -listckpt -name ROOTCRS_PATCHINFO -pname NODENAME_LIST"
2025-01-01 09:24:06: trace file=/u01/app/grid/base/crsdata/szpmsb02cimrac-db07/crsconfig/cluutil2.log
2025-01-01 09:24:06: Running as user grid: /u01/app/grid/product/19.0.0/grid/bin/cluutil -ckpt -global -oraclebase /u01/app/grid/base -listckpt -name ROOTCRS_PATCHINFO -pname NODENAME_LI ST
2025-01-01 09:24:07: Removing file /tmp/wUcdA39bG1
2025-01-01 09:24:07: Successfully removed file: /tmp/wUcdA39bG1
2025-01-01 09:24:07: pipe exit code: 0
2025-01-01 09:24:07: /bin/su successfully executed
2025-01-01 09:24:07: NODENAME_LIST=szpmsb02cimrac-db07,szpmsb02cimrac-db08
2025-01-01 09:24:07: Succeeded to get property value:NODENAME_LIST=szpmsb02cimrac-db07,szpmsb02cimrac-db08
2025-01-01 09:24:07: doing checks for postpatch rerun
2025-01-01 09:24:07: Invoking "/u01/app/grid/product/19.0.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid/base -chkckpt -name ROOTCRS_PREPATCH -status"
2025-01-01 09:24:07: trace file=/u01/app/grid/base/crsdata/szpmsb02cimrac-db07/crsconfig/cluutil3.log
2025-01-01 09:24:07: Running as user grid: /u01/app/grid/product/19.0.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid/base -chkckpt -name ROOTCRS_PREPATCH -status
2025-01-01 09:24:07: Removing file /tmp/kJ_Kqil2zm
2025-01-01 09:24:07: Successfully removed file: /tmp/kJ_Kqil2zm
2025-01-01 09:24:07: pipe exit code: 0
2025-01-01 09:24:07: /bin/su successfully executed
2025-01-01 09:24:07: FAIL
2025-01-01 09:24:07: The 'ROOTCRS_PREPATCH' is either in START/FAILED state
2025-01-01 09:24:07: Invoking "/u01/app/grid/product/19.0.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid/base -chkckpt -name ROOTCRS_PREPATCH -status"
2025-01-01 09:24:07: trace file=/u01/app/grid/base/crsdata/szpmsb02cimrac-db07/crsconfig/cluutil4.log
2025-01-01 09:24:07: Running as user grid: /u01/app/grid/product/19.0.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid/base -chkckpt -name ROOTCRS_PREPATCH -status
2025-01-01 09:24:07: Removing file /tmp/TVuFPBlBm0
2025-01-01 09:24:07: Successfully removed file: /tmp/TVuFPBlBm0
2025-01-01 09:24:07: pipe exit code: 0
2025-01-01 09:24:07: /bin/su successfully executed
2025-01-01 09:24:07: FAIL
2025-01-01 09:24:07: The 'ROOTCRS_PREPATCH' status is FAILED
2025-01-01 09:24:07: Executing cmd: /u01/app/grid/product/19.0.0/grid/bin/clsecho -p has -f clsrsc -m 739
2025-01-01 09:24:07: Executing cmd: /u01/app/grid/product/19.0.0/grid/bin/clsecho -p has -f clsrsc -m 739
2025-01-01 09:24:07: Command output:
> CLSRSC-739: cannot run the postpatch operation because the prepatch operation did not complete successfully
>End Command output
2025-01-01 09:24:07: CLSRSC-739: cannot run the postpatch operation because the prepatch operation did not complete successfully
2025-01-01 09:24:07: ###### Begin DIE Stack Trace ######
2025-01-01 09:24:07: Package File Line Calling
2025-01-01 09:24:07: --------------- -------------------- ---- ----------
2025-01-01 09:24:07: 1: main rootcrs.pl 357 crsutils::dietrap
2025-01-01 09:24:07: 2: crspatch crspatch.pm 2905 main::__ANON__
2025-01-01 09:24:07: 3: crspatch crspatch.pm 2289 crspatch::postPatchRerunCheck
2025-01-01 09:24:07: 4: crspatch crspatch.pm 2101 crspatch::crsPostPatchCkpts
2025-01-01 09:24:07: 5: crspatch crspatch.pm 421 crspatch::crsPostPatch
2025-01-01 09:24:07: 6: main rootcrs.pl 370 crspatch::new
2025-01-01 09:24:07: ####### End DIE Stack Trace #######
2025-01-01 09:24:07: checkpoint has failed
3.检查补丁情况
[root@rac2 ~]# /oracle/app/19c/grid/bin/crsctl query crs releasepatch
[root@rac2 ~]# /oracle/app/19c/grid/bin/kfod op=patches
[root@rac2 ~]# su - grid
[grid@rac2 ~]$ opatch lsinventory
oracle:
[oracle@szrac01 34622987]$ $ORACLE_HOME/OPatch/opatch lsinventory|grep 34622987
Patch 34622987 : applied on Fri Nov 01 15:36:19 CST 2024
[oracle@szrac01 34622987]$
grid:
[grid@szrac01 tmp]$ $ORACLE_HOME/OPatch/opatch lsinventory|grep 34622987
Patch 34622987 : applied on Fri Nov 01 15:34:23 CST 2024
4.mos 文章参考解决
[root@rac2 ~]# /u01/app/grid/product/19.0.0/grid/bin/crsctl stat res -t
[root@rac2 ~]# /u01/app/grid/product/19.0.0/grid/bin/crsctl start crs
[root@rac2 ~]# /u01/app/grid/product/19.0.0/grid/crs/install/rootcrs.sh -prepatch
[root@rac2 ~]# /u01/app/grid/product/19.0.0/grid/crs/install/rootcrs.sh -unlock
[root@rac2 ~]# /u01/app/grid/product/19.0.0/grid/bin/clscfg -localpatch
[root@rac2 ~]# /u01/app/grid/product/19.0.0/grid/crs/install/rootcrs.sh -lock
[root@rac2 ~]# /u01/app/grid/product/19.0.0/grid/bin/crsctl start crs