记一次存储扩容数据库启动恢复

客户电话告之，数据库存储要扩容，扩容之后需要将部分数据文件迁移到新存储上，迁移之前顺便把数据库巡检做了。先简单介绍一下客户数据库环境。AIX版本为5304，A、B数据库HA互备，由于B主机有部分存储多余，考虑到最近A数据库磁盘较为繁忙，使用经常达到100%，于是建议将B主机多余存储划分至A主机作为pdisk1，然后将pdisk1加入到A主机datavg中，在datavg上pdisk1上建立文件系统/oradata2，等以上步骤就绪之后，晚上抽空做一下数据迁移。了解客户的需求之后，心想小事一桩。悠哉悠哉从杭州坐大巴去宁波，中途还买了本书，打发时间。谁料一到客户现场，告之，A数据库启动不了。心里咯噔一下，这下又有活干了。
观察alert日志显示，由于存储不可读写，分别由ckpt和pmon尝试将数据库实例宕掉。最终由pmon进程将实例宕掉。

引用

Thu Mar 25 13:12:45 2010
Errors in file /oracle/app/admin/zzrk/bdump/zzrk_ckpt_331808.trc:
ORA-00202: controlfile: '/zzrkdata/zzrk/control01.ctl'
ORA-27091: skgfqio: unable to queue I/O
ORA-27072: skgfdisp: I/O error
IBM AIX RISC System/6000 Error: 5: I/O error
Additional information: 1
Thu Mar 25 13:12:45 2010
Errors in file /oracle/app/admin/zzrk/bdump/zzrk_lgwr_958654.trc:
ORA-00345: redo log write error block 181236 count 4
ORA-00312: online log 4 thread 1: '/zzrkdata/zzrk/redo04.log'
ORA-27063: skgfospo: number of bytes read/written is incorrect
IBM AIX RISC System/6000 Error: 5: I/O error
Additional information: -1
Additional information: 2048
Thu Mar 25 13:12:45 2010
Errors in file /oracle/app/admin/zzrk/bdump/zzrk_arc1_581840.trc:
ORA-00202: controlfile: '/zzrkdata/zzrk/control01.ctl'
ORA-27091: skgfqio: unable to queue I/O
ORA-27072: skgfdisp: I/O error
IBM AIX RISC System/6000 Error: 5: I/O error
Additional information: 1
Thu Mar 25 13:12:45 2010
KCF: write/open error block=0x42372 online=1
     file=13 /zzrkdata/zzrk/ZZRK2004_IDX_02.dbf
     error=27063 txt: 'IBM AIX RISC System/6000 Error: 5: I/O error
Additional information: -1
Additional information: 16384'
Thu Mar 25 13:12:45 2010
Errors in file /oracle/app/admin/zzrk/bdump/zzrk_ckpt_331808.trc:
ORA-00204: error in reading (block 1, # blocks 1) of controlfile
ORA-00202: controlfile: '/zzrkdata/zzrk/control01.ctl'
ORA-27091: skgfqio: unable to queue I/O
ORA-27072: skgfdisp: I/O error
IBM AIX RISC System/6000 Error: 5: I/O error
Additional information: 1
Thu Mar 25 13:12:45 2010
CKPT: terminating instance due to error 204
Thu Mar 25 13:12:45 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_991450.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:45 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_4169856.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:45 2010
Errors in file /oracle/app/admin/zzrk/bdump/zzrk_lgwr_958654.trc:
ORA-00340: IO error processing online log 4 of thread 1
ORA-00345: redo log write error block 181236 count 4
ORA-00312: online log 4 thread 1: '/zzrkdata/zzrk/redo04.log'
ORA-27063: skgfospo: number of bytes read/written is incorrect
IBM AIX RISC System/6000 Error: 5: I/O error
Additional information: -1
Additional information: 2048
Thu Mar 25 13:12:45 2010
Errors in file /oracle/app/admin/zzrk/bdump/zzrk_dbw0_466966.trc:
ORA-00600: internal error code, arguments: [ksfdgfnm1], [0x7000000FE571118], [], [], [], [], [], []
Thu Mar 25 13:12:45 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_2683076.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:45 2010
Errors in file /oracle/app/admin/zzrk/bdump/zzrk_arc1_581840.trc:
ORA-00204: error in reading (block 1, # blocks 1) of controlfile
ORA-00202: controlfile: '/zzrkdata/zzrk/control01.ctl'
ORA-27091: skgfqio: unable to queue I/O
ORA-27072: skgfdisp: I/O error
IBM AIX RISC System/6000 Error: 5: I/O error
Additional information: 1
Thu Mar 25 13:12:45 2010
Errors in file /oracle/app/admin/zzrk/bdump/zzrk_arc1_581840.trc:
ORA-00204: error in reading (block 1, # blocks 1) of controlfile
ORA-00202: controlfile: '/zzrkdata/zzrk/control01.ctl'
ORA-27091: skgfqio: unable to queue I/O
ORA-27072: skgfdisp: I/O error
IBM AIX RISC System/6000 Error: 5: I/O error
Additional information: 1
Thu Mar 25 13:12:45 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3711100.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:45 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_2912488.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:45 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_4268134.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:46 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3768444.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:46 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3645480.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:46 2010
System state dump is made for local instance
Thu Mar 25 13:12:46 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3338340.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-01115: IO error reading block from file 18 (block # 323615)
ORA-01110: data file 18: '/zzrkdata/zzrk/ZZRK2004_IDX_03.dbf'
ORA-27091: skgfqio: unable to queue I/O
ORA-27072: skgfdisp: I/O error
IBM AIX RISC System/6000 Error: 5: I/O error
Additional information: 323614
Thu Mar 25 13:12:46 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3264658.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:46 2010
Errors in file /oracle/app/admin/zzrk/bdump/zzrk_lmd0_458862.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:46 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_2945276.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:47 2010
Errors in file /oracle/app/admin/zzrk/bdump/zzrk_lmon_487452.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:47 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3940452.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:47 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_4464878.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:47 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_4169856.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:47 2010
Errors in file /oracle/app/admin/zzrk/bdump/zzrk_pmon_929890.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:48 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3711100.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:48 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3768444.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:48 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3686494.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:48 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_4661270.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:49 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_4563142.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:49 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3465310.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:50 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_2900032.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:50 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3678250.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:51 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3088542.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:51 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3817474.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:52 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3432540.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:53 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_4317288.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-01115: IO error reading block from file 28 (block # 89898)
ORA-01110: data file 28: '/zzrkdata/zzrk/ZZRK2004_QTXX_05.dbf'
ORA-27091: skgfqio: unable to queue I/O
ORA-27072: skgfdisp: I/O error
IBM AIX RISC System/6000 Error: 5: I/O error
Additional information: 89897
Thu Mar 25 13:12:53 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3997780.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:53 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3334256.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-01115: IO error reading block from file 14 (block # 159530)
ORA-01110: data file 14: '/zzrkdata/zzrk/ZZRK2004_QTXX_02.dbf'
ORA-27091: skgfqio: unable to queue I/O
ORA-27072: skgfdisp: I/O error
IBM AIX RISC System/6000 Error: 5: I/O error
Additional information: 159529
Thu Mar 25 13:12:54 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3903662.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:54 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3817474.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:54 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_4059174.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:55 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3797076.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:55 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_4526242.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:56 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_2687186.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:56 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_4493398.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:57 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_4178066.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:57 2010
Trace dumping is performing id=[cdmp_20100325131246]
Thu Mar 25 13:12:57 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3629252.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:59 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_4579364.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:12:59 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3252266.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:13:00 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_2969624.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:13:00 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3084314.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:13:00 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3629252.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:13:02 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3592410.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:13:02 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3170412.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:13:02 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3575816.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:13:02 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3084314.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:13:03 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_4292672.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:13:04 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3985602.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:13:04 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3723476.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:13:04 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3592410.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:13:04 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_999510.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:13:05 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_2936836.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:13:06 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_4251652.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:13:06 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_4632702.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:13:06 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_999510.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:13:06 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_3723476.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
Thu Mar 25 13:13:07 2010
Errors in file /oracle/app/admin/zzrk/bdump/zzrk_dbw0_466966.trc:
ORA-00600: internal error code, arguments: [ksfdgfnm1], [0x7000000FE571118], [], [], [], [], [], []
Thu Mar 25 13:13:09 2010
Errors in file /oracle/app/admin/zzrk/udump/zzrk_ora_4632702.trc:
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile
ORA-00204: error in reading (block , # blocks ) of controlfile

引用

Thu Mar 25 13:13:57 2010
PMON: terminating instance due to error 204
Instance terminated by PMON, pid = 929890

尝试将数据库mount，alert日志报错：

引用

Thu Mar 25 14:04:34 2010
ORA-00204: error in reading (block 1, # blocks 1) of controlfile
ORA-00202: controlfile: '/zzrkdata/zzrk/control01.ctl'
ORA-27091: skgfqio: unable to queue I/O
ORA-27072: skgfdisp: I/O error
IBM AIX RISC System/6000 Error: 5: I/O error
Additional information: 1

可是奇观的是控制文件和数据文件可以在操作系统中看到

引用

$ ls -l control02.ctl
-rw-rw-rw- 1 ora10g oinstall 41172992 Mar 25 14:05 control02.ctl

进一步用od -x命令查看controlfile,只能查看到部分信息，至此可以确定由于存储问题导致数据库无法识别控制文件，数据文件。

引用

$ od -x control02.ctl
0000000

经确定是由于存储链路挂接错误，导致存储出现问题。