节点1数据库软件顺利打完补丁，但是节点2在打软件补丁的过程中，由于虚拟机异常被迫重启，导致应用补丁中断，开机后检查节点2数据库软件补丁已应用，但无法正常开启节点2数据库实例
注：打补丁前，一定记得要先把软件目录tar一份作为备份。以下的问题就是没备份，不得已的办法。
数据库启动报错
[grid@rac2 ~]$ srvctl start instance-dorcl-iorcl2 PRCR-1013:Failed to start resource ora.orcl.db PRCR-1064:Failed to start resource ora.orcl.db onnoderac2 CRS-5017: The resource action"ora.orcl.db start"encountered the following error: ORA-03113: end-of-file on communication channel Process ID:0Session ID:0Serial number:0.For details refer to"(:CLSN00107:)"in"/u01/grid/11.2.0.4/log/rac2/agent/crsd/oraagent_oracle/oraagent_oracle.log".CRS-2674: Start of'ora.orcl.db'on'rac2'failed -------------------------------------------------------------------[oracle@rac2 ~]$ sqlplus / as sysdba SQL*Plus: Release11.2.0.4.0 Production on Tue Jul1410:18:362026Copyright(c)1982,2013, Oracle. All rights reserved. Connected to an idle instance. SQL>startup nomount ORA-03113: end-of-file on communication channel SQL>exitDisconnected查看数据库日志
数据库日志没有任何报错信息，oraagent_oracle.log 也没有明确的信息
Tue Jul1410:18:402026Starting ORACLE instance(normal)************************ Large Pages Information ******************* Per process system memlock(soft)limit=UNLIMITED Total Shared Global RegioninLarge Pages=0KB(0%)Large Pages used by this instance:0(0KB)Large Pages unused system wide=0(0KB)Large Pages configured system wide=0(0KB)Large Page size=2048KB RECOMMENDATION: Total System Global Area size is1906MB. For optimal performance, prior to the next instance restart:1. Increase the number of unused large pages by at least953(page size2048KB, total size1906MB)system wide to get100% of the System Global Area allocated with large pages ******************************************************************** LICENSE_MAX_SESSION=0LICENSE_SESSIONS_WARNING=0Initial number of CPU is1Number of processor coresinthe system is1Number of processor socketsinthe system is1集群状态检查
[grid@rac2 ~]$ crsctlstatres-t-------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.FRA.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.LISTENER.lsnr ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.OCR.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.asm ONLINE ONLINE rac1 Started ONLINE ONLINE rac2 Started ora.gsd OFFLINE OFFLINE rac1 OFFLINE OFFLINE rac2 ora.net1.network ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.ons ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.registry.acfs ONLINE ONLINE rac1 ONLINE ONLINE rac2 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr1ONLINE ONLINE rac1 ora.cvu1ONLINE ONLINE rac1 ora.oc4j1ONLINE ONLINE rac2 ora.orcl.db1ONLINE ONLINE rac1 Open2ONLINE OFFLINE Instance Shutdown ora.rac1.vip1ONLINE ONLINE rac1 ora.rac2.vip1ONLINE ONLINE rac2 ora.scan1.vip1ONLINE ONLINE rac1通过检查环境变量、集群日志、操作系统日志、数据库日志、系统参数等均未发现任何问题
检查 bin/oracle 文件
紧接着检查一个最容易出问题但被忽略的文件$ORACLE_HOME/bin/oracle
节点1：
[oracle@rac1 ~]$ ll$ORACLE_HOME/bin/oracle -rwsr-s--x1oracle asmadmin239987343Jul1021:27 /u01/app/oracle/product/11.2.0.4/db_1/bin/oracle[oracle@rac1 ~]$ md5sum$ORACLE_HOME/bin/oracle c4f967050767e21c783133afd56dd08a /u01/app/oracle/product/11.2.0.4/db_1/bin/oracle节点2：
[oracle@rac2 ~]$ ll$ORACLE_HOME/bin/oracle -rwsr-s--x1oracle asmadmin239626641Dec12025/u01/app/oracle/product/11.2.0.4/db_1/bin/oracle[oracle@rac2 ~]$ md5sum$ORACLE_HOME/bin/oracle 01e996bdb9127b1d65724331f7203dfe /u01/app/oracle/product/11.2.0.4/db_1/bin/oracle发现 两边节点的 oracle 二进制文件不一致，节点2的二进制文件还停留在 未打补丁的状态
将节点2 oracle 文件替换为节点1相同 oracle 文件
[oracle@rac2 ~]$mv$ORACLE_HOME/bin/oracle /home/oracle/[oracle@rac2 ~]$scprac1:$ORACLE_HOME/bin/oracle$ORACLE_HOME/bin/oracle oracle100% 229MB114.4MB/s 00:02[oracle@rac2 ~]$ md5sum$ORACLE_HOME/bin/oracle c4f967050767e21c783133afd56dd08a /u01/app/oracle/product/11.2.0.4/db_1/bin/oracle## 使用 setasmgidwrap 工具修复权限[root@rac2 ~]# /u01/grid/11.2.0.4/bin/setasmgidwrap oracle_binary_path=/u01/app/oracle/product/11.2.0.4/db_1/bin/oracle## 确认 oracle 二进制文件与 rac1 一致[oracle@rac2 ~]$ ll$ORACLE_HOME/bin/oracle -rwsr-s--x1oracle asmadmin239987343Jul1410:27 /u01/app/oracle/product/11.2.0.4/db_1/bin/oracle[oracle@rac2 ~]$ md5sum$ORACLE_HOME/bin/oracle c4f967050767e21c783133afd56dd08a /u01/app/oracle/product/11.2.0.4/db_1/bin/oracle开启节点2实例，可以正常启动
[oracle@rac2 ~]$ srvctl start instance-dorcl-iorcl2[oracle@rac2 ~]$ps-ef|grepsmon root33461010:33 ? 00:00:09 /u01/grid/11.2.0.4/bin/osysmond.bin oracle157891010:53 ? 00:00:00 ora_smon_orcl2[grid@rac2 ~]$ crsctlstatres-tora.orcl.db1ONLINE ONLINE rac1 Open2ONLINE ONLINE rac2 Open
