Friday, May 29, 2009

Installing Veritas 5.0 MP2 patches leave Solaris 10 system unbootable.... Un-encapsulated

After installing Veritas 5.0 MP2 patches set, Solaris 10 system become unbootable. Boot process failed and system goes back to "ok>" prompt.
Following is the system console log:
{1} ok boot
Resetting...

Rebooting with command: boot
SunOS Release 5.10 Version Generic_118833-36 64-bit
Copyright 1983-2006 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
sp diff: name finddevice, nargs 1, nret 1,sp 0xf05d35b8 osp 0xf05d35a8
Hardware watchdog enabled
Unretrieved lom log history follows ...

4/29/09 2:39:04 PM Domain Reboot A: Initiating keyswitch: on, domain A.
4/29/09 2:39:04 PM Domain Reboot A: Initiating keyswitch: on, domain A.
Booting to milestone "milestone/single-user:default".
Hostname: sun1

syncing file systems... done
NOTICE: f_client_exit: Program terminated!
debugger entered.

{2} ok


Just heard that it could be an issue with boot disk encapsulation. The decision then is to recover the server by boot with normal disk slice.
First, disable boot disk encapsulation (un-encapsulated) by modify /etc/system and /etc/vfstab. Since server unbootable, reboot from alternate media (cdrom)
ok> boot cdrom -s
root # fsck /dev/rdsk/c0t0d0s0
root # mount /dev/dsk/c0t0d0s0 /a
root # cd /a/etc
root # cp system system.ORG
root # cp vfstab vfstab.ORG


Modify /etc/system (in this case /a/etc/system). Mark "rootdev and vxio"
* vxvm_START (do not remove)
forceload: drv/vxdmp
forceload: drv/vxio
forceload: drv/vxspec
***rootdev:/pseudo/vxio@0:0
***set vxio:vol_rootdev_is_volume=1
* vxvm_END (do not remove)


Modify /etc/vfstab (in this case /a/etc/vfstab) to normal disk slice.
root # cat /a/etc/vfstab
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/dsk/c1t0d0s1 - - swap - no -
/dev/dsk/c1t0d0s0 /dev/rdsk/c1t0d0s0 / ufs 1 no -
/dev/dsk/c1t0d0s3 /dev/rdsk/c1t0d0s3 /var ufs 2 no -
/dev/dsk/c1t0d0s4 /dev/rdsk/c1t0d0s4 /export/home ufs 2 yes -
swap - /tmp tmpfs - yes -
root #


Then reboot server.
root # umount /a
root # /etc/halt
ok> boot


With luck the server now bootable into multi-user mode. When everything is back to normal, then proceed with boot disk encapsulation and re-attach the mirror.

There is Sun Alert Document (Doc ID: 250446) described an issue during upgrade to Veritas 5.0 MP3. However, the impact is very similar. Server unbootable after patches installation.
Below is copy a portion of the document. Please refer to http://sunsolve.sun.com/search/document.do?assetkey=1-66-250446-1 for detail.

Document ID: 250446
Title: During Installation of Veritas VxVM 5.0 Patches the System may Become Unbootable
Solution 250446 : During Installation of Veritas VxVM 5.0 Patches the System may Become Unbootable

During installation of Veritas VxVM 5.0 patches the system may become unbootable (see below for details). Rebooting after installing the patches mentioned below, the system may fail to boot to multiuser mode. This issue is described by Symantec at:
http://seer.entsupport.symantec.com/docs/315817.htm

This alert described such problem and its workaround/resolution.
Following is (cut and paste) steps to recover and re-install the patch:

1. Boot the system from an alternate root. (mirror, net or CDROM)
2. Run format and look at all of the disks to determine which one was the original boot disk. You may need to mount some and see if VRTS patches are there as this is a good indicator that it was the boot disk.
3. Mount the original boot disk to /mnt
4. # bootadm update-archive -R /mnt
(updates the original boot archive, applicable for s10u6 and later)
5. # /etc/vx/bin/vxunroot
(to un-encapsulate root)
6. # reboot
(boot using normal slices)
7. # /etc/vx/bin/vxdg destroy rootdg
(destroy rootdg and all volumes associated with rootdg)
8. # /etc/vx/bin/vxdiskunsetup -C c#t#d#
(c#t#d# of rootdisk) (remove rootdisk from VxVM control)
9. # pkgrm VRTSvxvm
(in order to remove the patch from the system, as the patch cannot be backed out and patchinfo of 122058-11 is inconsistent.)
10. # reboot
11. # install VRTSvxvm and Run *vxinstall* to add/verify licenses and initialize Veritas
12. # vxdiskadm - encapsulate/mirror root and other partitions on the root disk.
13. # install patch 122058-11 and ignore the console prompt referenced above.
14. # reboot

There is good related article on Sun BigAdmin (Enda O'Connor, April 2009) stated:
The recommended way prior to patching is break the mirror, and then patch only on one half of the mirror. If a problem occured, it should be possible to boot from the other half of the mirror (the unpatched half). It is also strongly advised to use Solaris Volume Manager for mirroring root file systems, as opposed to using Veritas Volume Manager (VxVM) mirroring of root file systems. This is due to VxVM disk encapsulation, which, depending on the disk layout and free partitions on the root disk, might create a hard-to-manage disk layout that causes issues when trying to rescue such a layout. Also, there is the common misconception in the sys admin world that encapsulating a disk in VxVM equates to mirroring. This is a major mistake. Encapsulation is only the first step towards mirroring a root file system, whereby the currently installed root file system disk is given over to VxVM control and the data is preserved encapsulated). You then need to mirror the encapsulated root disk using further VxVM commands.


No comments:

Post a Comment