Solaris 10 won't boot. Entering System Maintenance Mode.... mount -m -o remount,rw /

After installing system related patches, system won't boot to multi-user mode. Its stopped in single-user mode (System Maintenance Mode). The root filesystem (/) flagged as Read-Only filesystem. Fsck doesn't work. Even with fsck from booting from cdrom. The problem persist. Later on found that "mount -m -o remount,rw /" fixed the problem.

Following is the system console log during recovery:
ok> boot -s
Rebooting with command: boot -s
SunOS Release 5.10 Version Generic_118833-36 64-bit
Copyright 1983-2006 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Hardware watchdog enabled

svc.configd: smf(5) database integrity check of:
/etc/svc/repository.db
failed. The database might be damaged or a media error might have
prevented it from being verified. Additional information useful to
your service provider is in:
/etc/svc/volatile/db_errors
The system will not be able to boot until you have restored a working
database. svc.startd(1M) will provide a sulogin(1M) prompt for recovery
purposes. The command:
/lib/svc/bin/restore_repository
can be run to restore a backup version of your repository. See
http://sun.com/msg/SMF-8000-MY for more information.
Requesting System Maintenance Mode
svc.configd exited with status 102 (database initialization failure)

Root password for system maintenance (control-d to bypass):
single-user privilege assigned to /dev/console.
Entering System Maintenance Mode

Apr 30 09:05:31 su: 'su root' succeeded for root on /dev/console
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
Sourcing //.profile-EIS.....
root@ #

As descibed above, the system won't start. It need the working database to be restored.
root@ # /lib/svc/bin/restore_repository
No luck. System complaining that the root filesystem ( / ) is in Read-Only mode. Its need to be fix before proceed.
root@ # mount -m -o remount,rw /
Now the root filesystem become writeable, and we can start recovery.
root@ # /lib/svc/bin/restore_repository
Succeeded, then proceed restart the server with boot in normal mode.

Note:
Option on mount -m -o remount,rw
-m: Mount the file system without making an entry in /etc/mnttab.

-o remount,rw : Remount a file system with a new set of read-write options

Backup and copy boot/system disk into another disk.... flarcreate

Solaris Operating System have many utilities or commands for backup and copy bootdisk. We can use paired ufsdump-ufsrestose, fssnap, dd, flarcreate, and so on. Following is a list of steps when we copy bootdisk using flarcreate. The purpose of this step is making a server bootdisk archive's file (flash archive). It is useful when we do rolling server upgrade and/or rolling server migration into another datacenter site. By restoring the flash archive into another server, we then have a cloning server.

Preparation:
- Create exclude file. A file containing list of filesystem which will excluded during archiving.

- File house keeping. Delete or move (backup) un-necessary files (eg; coredump, log files, messages, etc)

- Run explorer to capture the latest system configuration.

root@sun1: # df -k
Filesystem kbytes used avail capacity Mounted on
/dev/md/dsk/d10 33053541 3731851 28991155 12% /
/proc 0 0 0 0% /proc
mnttab 0 0 0 0% /etc/mnttab
fd 0 0 0 0% /dev/fd
swap 33409688 168 33409520 1% /var/run
swap 33409992 472 33409520 1% /tmp
/dev/md/dsk/d40 35308580 37056 34918439 1% /oracle
/dev/md/dsk/d30 25206312 2474452 22479797 10% /export/home
/dev/md/dsk/d51 10325804 5117813 5104733 51% /app/rdbms
/dev/md/dsk/d54 986359 13409 913769 2% /ora/control
/dev/md/dsk/d57 986359 103513 823665 12% /ora/redo
/dev/md/dsk/d58 206531907 154964219 49502369 76% /ora/data1
/dev/md/dsk/d59 206531907 194515035 9951553 96% /ora/data2
/dev/md/dsk/d63 206531907 10320826 194145762 6% /ora/index
/dev/md/dsk/d64 77449353 782820 75892040 2% /ora/archieve

root@sun1: # cat /tmp/exclude_file
/ora/control
/ora/redo
/ora/data1
/ora/data2
/ora/index
/ora/archive

Execute flarcreate command with following options;
flarcreate -n "Clone_system" -c -X /fs_exclude_file -R / /fs_desination/archive_file_name
Note:
-n: name of archived / image
-c: compress
-I: ignore integrity check
-R: the image start from /
-S: skip the disk space check
-X: flag for exclude
/fs_exclude_file: a file containing list of filesystem to be excluded from archive
/fs_destination/archive_file_name: flash (archive) file

root@sun1: # flarcreate -n "Clone_sun1" -c -X /exclude_file -R / /ora/index/sun1-flar-2009
Full Flash
Checking integrity...
Integrity OK.
Running precreation scripts...
Precreation scripts done.
Determining the size of the archive...
cpio: "var/tmp/.oracle/s#28286.1" ?
cpio: "var/tmp/.oracle/s#14887.1" ?
cpio: "var/tmp/.oracle/s#3727.1" ?
cpio: "var/tmp/.oracle/s#3742.1" ?
cpio: "var/tmp/.oracle/sDBPROC" ?
11558133 blocks
5 error(s)
The archive will be approximately 5.91GB.
Creating the archive...
cpio: "var/tmp/.oracle/s#28286.1" ?
cpio: "var/tmp/.oracle/s#14887.1" ?
cpio: "var/tmp/.oracle/s#3727.1" ?
cpio: "var/tmp/.oracle/s#3742.1" ?
cpio: "var/tmp/.oracle/sDBPROC" ?
11558133 blocks
5 error(s)
Archive creation complete.
(Note: the errors can be ignored)

File integrity and disk space check will take a longer times, if necessary these check can be omitted by execute command with the following option:

root@sun1: # flarcreate -n "Clone_sun1" -c -S -I -X /exclude_file -R / /ora/index/sun1-flar-2009
Full Flash
No integrity check. Option "-I" set.
Running precreation scripts...
Precreation scripts done.
Creating the archive...
cpio: "var/tmp/.oracle/s#28286.1" ?
cpio: "var/tmp/.oracle/s#14887.1" ?
cpio: "var/tmp/.oracle/s#3727.1" ?
cpio: "var/tmp/.oracle/s#3742.1" ?
cpio: "var/tmp/.oracle/sDBPROC" ?
11558133 blocks
5 error(s)
Archive creation complete.
(Note: the errors can be ignored)

The next step is restoring the flash (archive) file into another server, either by:
1. Install from cdrom by select for Flash install, or
2. Install through Jumpstart Enterprise Toolkit (JET)

Note:

During archive process (flarcreate), /etc/shadow file won't be backup (excluded). As such we need to copy it into new server once the restore process have been done. Otherwise we may need to reset the existing user password manually.

/dev/null, what it is about.... My hero

From Wikipedia:
In Unix-like operating systems, /dev/null or the null device is a special file that discards all data written to it (but reports that the write operation succeeded), and provides no data to any process that reads from it (yielding EOF immediately). It may also be called the bit bucket or Black hole.

From www.rwc.uc.edu:

A special file referred to as the null device or informally as the bit bucket. The most common use of this virtual device is to discard any data written (directed) to it. Additionally, it provides an End of File (EOF) when used as an input stream (read from).

The null device is typically used for disposing of unwanted output streams of a process, or as a convenient empty file for input streams. This is usually done by redirection.

In practice, /dev/null can be use as describe in the following examples;

1. In conjunction with dd command, to locate the disk drive within JBOD farm.
By running dd command, we can locate the disk drive by physically scan it on JBOD . It indicated by disk led indicator which "steadily blinking". (led is steadily blinking because of excesive I/O activities generated by dd command)

root@sun1: # dd if=/dev/c2t3d0s2 of=/dev/null

2. Use for netbackup performance test.

Besides as performance test, the scenario also good for filesystem integrity check. By writing into /dev/null, netbackup can perform whole filesystem backup with out occupied any real tape media. Once netbackup succeeded with whole filesystem backup, its indicated that the filesystem have good integrity.

Reference:
www.mass.dk (Enabling Netbackup to use /dev/null as storage unit)
Information is provided on a as-is basic. Use at own risk.

!! Creating the bpdm_dev_null file redirect ALL disk based backup to /dev/null on the media server where the touch is placed, not just the newly created one in the example below. Disable all production classes and delete the touch file when testing is complete. !!

a) touch /usr/openv/netbackup/bpdm_dev_null on a media server

b) Create a new disk storage unit (STU), using /tmp or some other directory as the image directory path

c) Create policy that points to the new disk stu.
d) Run a backup using this policy.
NetBackup will create an image files as if it was a real backup image. The image file(s) will be zero bytes long.

e) To clean up the netbackup catalog use bpexpdate -d 0 -backupid {hostname_epoctime} (the backup id is part of the file name). This command remove the zero byte image file and delete the backup in the Netbackup catalog (it hard to restore anyway). Do not delete the file manual.

3. Most of cron jobs script sink the output to /dev/null, unless the owner expected emails for every cron jobs done.

Note:
In case for any reason /dev/null was missing, here is step to recover:

1. If only the soft link was missing:
root@sun1: # devlinks
2. In case the original device in /devices/... was missing:
root@sun1: mknod /devices/pseudo/mm@0:null c 13 2
root@sun1: chown root:sys /devices/pseudo/mm@0:null
root@sun1: chmod 666 /devices/pseudo/mm@0:null
root@sun1: cd /dev
root@sun1: ln -s ../devices/pseudo/mm@0:null null

** Do not bother sending complaints email. The mailbox have been set to /dev/null **

Disk was replaced, however the serial number doesn't changed.... cfgadm

It is a normal procedure to replace faulty disk while system is up and running. By pull-out the faulty disk, and insert the replacement disk, the new disk can be recognized by the OS. If Solaris Volume Manager (SVM) is being used, then we may need to proceed with resync or rebuild the broken sub-mirror. However, it is not an appropriate way. Even though the new disk has been recognized by OS and SVM went through with rebuild mirror, it will left problem behind. With "iostat -En" command, the disk serial number doesn't indicated that the disk has been replaced. The old disk serial number still intact.

root@sun1: # iostat -En
snip
c0t1d0 Soft Errors: 1 Hard Errors: 42 Transport Errors: 10 Vendor: SEAGATE Product: ST373207LSUN72G Revision: 045A Serial No: 053432A5HL Size: 73.40GB <73400057856> Media Error: 36 Device Not Ready: 0 No Device: 6 Recoverable: 1 Illegal Request: 0 Predictive Failure Analysis:0
snip
Note:
Serial No: 053432A5HL was an old disk serial number.

Furthermore when performing system reboot the errors below may be seen:
Sep 22 10:15:25 sun1 metadevadm: [ID 209699 daemon.error] Invalid device relocation information detected in Solaris Volume Manager

and as a result, the sub-mirror broke again, and manual resync (rebuild mirror) should be perform. After system has been rebooted, the new disk shown its correct serial number;
root@sun1: # iostat -En
snip
c0t1d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 Vendor: FUJITSU Product: MAW3073NCSUN72G Revision: 1703 Serial No: 0749B0PDPJ Size: 73.40GB <73400057856> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 Illegal Request: 0 Predictive Failure Analysis:0
snip
Note:
Serial No: 0749B0PDPJ is a new disk serial number

Following is an illustrated how to replace a faulty disk.
Configuration:
- all disk are SCSI disks
- disks mirror by Solaris Volume Manager
- each sub-mirror protected by hot spare disk (hot spare pool)
- faulty disk was c1t0d0, and the other sub-mirror disk was c0t0d0

The following steps and set of commands can be followed during disk replacement:
1. Check whether there are any replicas on faulty disk, then remove them if any;
root@sun1: metadb
root@sun1: metadb -d c1t0d0s7
Verify if there are no existing replicas left on faulty disk;
root@sun1: # metadb | grep c1t0d0
2. Run "cfgadm" command to remove the failed disk.
root@sun1: # cfgadm -c unconfigure c1::dsk/c1t0d0
3. Insert and configure the new disk.
root@sun1: # cfgadm -c configure c1::dsk/c1t0d0
Verify that disk is properly configured;
root@sun1: # cfgadm -al
If necessary run following related disk commands:
root@sun1: # devfsadm
root@sun1: # format (verifying new disk)
4. Create desired partition table on the new disk with prtvtoc command;
root@sun1: # prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c1t0d0s2
5. Recreate replicas on new disk:
root@sun1: # metadb -a c1t0d0s7
6. Run metareplace to enable and resync the new disk.
metareplace -e (mirror-md) cxtyd0sz
root@sun1: # metareplace -e d20 c1t0d0s0

7. In case SVM device-id not up-to-date, run "metadevadm" which will update the new disk device-id.

root@sun1: # metadevadm -u c1t0d0

Reference:
sunsolve.sun.com
=================================================================
Document ID: 208671
Title: Solaris Volume Manager software: Replacing Disks
=================================================================

Description
Beginning with the Solaris[TM] 9 Operating System, Solaris[TM] Volume Manager(VM) software uses a new feature called Device-ID (DevID). This feature identifies each disk not only by its c#t#d# name, but by a unique ID which is generated by the disk's WWN or serial number.
Solaris Volume Manager(VM) relies on the Solaris OS to supply it with each disk's correct DevID. When a disk fails and is replaced, a specific procedure is required for disks to make sure thatSolaris OS is updated with the new disk's DevID.

If this procedure is not followed exactly, the errors below may be seen:

Jun 22 18:22:57 host1 metadevadm: [ID 209699 daemon.error] Invalid device relocation information detected in Solaris Volume Manager

As a result, Solaris OS will not update the DevID until the next reboot, meaning that although a NEW disk is in the system, the DevID being reported by Solaris OS to the Solaris VM software is still the OLD disk's DevID.
(..truncated..)

Sewaktu ganti disk, semuanya berjalan normal. Tapi saat cek serial number pake iostat -En kok serial number nya masih punya disk yang lama?. Bingung dech. Beberapa hari kemudian saat server di reboot, ternyata disk yang baru diganti "lepas" lagi dari mirror-nya. Perlu di resync ulang. Wah agak mubasir lemburan yang kemarin. Tapi lembur ya tetap lembur.... lumayan.

Modifying live Kernel tuning parameters values.... mdb

Sometimes we need to modify Kernel tuning parameters values by modify /etc/system file and then performing system reboot to make the changes take effect. However in most datacenter site, its restricted to perform such reboot at any time during office-hour. It should be carried out after office-hour, in the middle of the night, or the worse during week-end days.

Following is the procedure which can be applied to minimize system downtime and/or after office-hour jobs. The tuning parameters value changes will take immediate effect by executing following commands:

root@sun1: # echo 'lotsfree/W 4096' | mdb -kw
lotsfree: 0xb9e = 0x4096
root@sun1: # echo 'dosynctodr/W 0' | mdb -kw
dosynctodr: 0x1 = 0x0
root@sun1: # echo 'ncsize/W 0t42000' | mdb -kw
ncsize: 0xafc8 = 0xa410
root@sun1: # echo 'shminfo_shmmax/W 0t8500000' | mdb -kw
shminfo_shmmax: 0x800000 = 0x81b320

To verify the changes, we can run mdb utility to read from text input files. First, we create text file containing instructions as;
root@sun1: cat /tmp/mdb_val.cmd
lotsfree/D
dosynctodr/D
ncsize/D
shminfo_shmmax/D
$q

Then redirect the file into mdb utility
root@sun1: mdb -k < /tmp/mdb_val.cmd
lotsfree:
lotsfree: 16534
dosynctodr:
dosynctodr: 0
ncsize:
ncsize: 42000
shminfo_shmmax:
shminfo_shmmax: 8500000

Caution:

The system might crash when we use mdb utility to change the tuning parameters values. This is because the live Kernel relies on the parameters to perform the majority of tasks. An improper setting could cause the Kernel to panic. In addition, any changes using mdb utility are lost when system rebooted, unless appropriate changes have also been made to /etc/system file.

Further reading:
Solaris System Performance Management (SA-400) - Student Guide.

mdb utility use switch or flag to display (D and E) or write (W and Z) the live Kernel tuning parameter value;

/D; Display a 32-bit(4 bytes) parameter value, eg: lotsfree/D
/E; Display a 64-bit (8 bytes) parameter value, eg: buffer_lock/E
/W; Write a 32-bit (4 bytes) parameter value, eg: lotsfree/W 4096
/Z; Write a 64-bit (8 bytes) parameter value, eg: buffer_lock/Z 0t8192

We should used correct switch/flag to display or write the parameter values. The following command used to determine whether the parameter has 4 bytes (32 bit) or 8 bytes (64 bit) value;

"/usr/ccs/bin/nm /dev/ksyms"

example:
root@sun1: # /usr/ccs/bin/nm /dev/ksyms|grep -i ncsize
(Index)|(Value)|(Size)|(Type)|(Bind)|(Other)|(Shndx)|(Name)
[25655] |4274292852| 4| OBJT| GLOB| 0| ABS|ncsize
Coloum 3; Size = 4; its indicated an 4 bytes (32 bit) parameter

root@sun1: # /usr/ccs/bin/nm /dev/ksyms|grep -i buffer_lock
[7916] |4274633176| 8| OBJT| LOCL| 0| ABS|buffer_lock
Coloum 3; Size = 8; its indicated an 8 bytes (64 bit) parameter

Untuk merubah tuning parameter kernel dengan mdb, sebaiknya dilakukan jika kita sudah betul-betul fasih dengan cara penggunaannya. Sebab cara ini berisiko. Jika salah dengan nilai parameter, dapat mengakibatkan system tidak bekerja normal atau bahkan mengakibatkan system crash atau panic. Lebih aman dengan mengubah paramaeter pada file /etc/system, kemudian melakukan reboot. Toh lemburan masih lebih baik ketimbang nungguin iklan.

Worm in Solaris 10. An unexpected Broadcast messages (in.telnetd vulnerability)

While working with Solaris 10 terminal window, got following an unexpected broadcast message from adm (???) as follows. Its harmless, but wondering how can Unix system got infected such worm.

asr@tks> Broadcast Message from adm (???) on tks Mon Oct 13 12:10:00...
__
{" "}
*-*
( Y )
()"*"()
(_)-(_)

use@sapp> Broadcast Message from adm (???) on sapp Fri Jun 13 12:10:00...
Hi, I'm Casper , I am a bored Sun developer and I wrote this piece of code.

Broadcast Message from adm (???) on jtksf Fri Mar 13 12:10:00...

W O R M S A G A I N S T N U C L E A R K I L L E R S ______________________________________________________

\ _ _______ __________________ __ ____ ___ __ __ ____/
\ \ \ /\ / / / /\ \ | \ \ | | | | / / /
\ \ \ / \ / / / /_\ \ | |\ \ | | | | / / /
\ \ \/ /\ \/ / / ___ \ | | \ \ | | | | \ \ /
\ _\ /__\ /__/ /____\ \__| |__\ | |_| | _\ \_/
\_____________________________________________/
\ /
\ Your System Has Been Officically WANKed /
\_______________________________________/

You talk of times of peace for all, and then prepare for war

Following is an explanation as described in Sun Alert 102802 which useful for anticipated such worm as well as for relief the infected system.

Sun Microsystems is aware of an active worm which exploits the in.telnetd vulnerability. Here are a few steps to help determine if a Solaris 10 or Nevada system may be infected:

$ ls -la /var/adm/wtmpx

If the permissions are:
-rw-r--rw- 1 adm adm 1116 Feb 28 12:03 wtmpx

the system may be infected. Next the following command can be run:
$ ls -la /var/adm/sa

If there is directory named .adm the system is probably infected. Other possible indications include the existence of the files:

/var/adm/.profile
/var/spool/lp/.profile

Additionally possible indications include modified crontab entries for users adm and lp.

# cd /var/spool/cron/crontabs
# grep PATH=\. *
adm:#10 1 * * * (cd /var/adm/sa/ && cd .adm && [ -x sysadm ] && PATH=. sysadm) >/dev/null 2>&1 &
lp:#10 1 * * * (cd /var/spool/lp/admins/ && cd .lp && [ -x lpsystem ] && PATH=. lpsystem) >/dev/null 2>&1 &

The following Korn shell script, inoculate.local, can be run locally on an infected system to remove the worm and prevent further re-infection by disabling the telnet service.
Copy the script into a file (for example, in /tmp or /var/tmp) and run the script as the root user.

#!/bin/ksh -p
#
# Save this script as "inoculate.local" (for example, in /tmp or /var/tmp)
# and run the script as the root user
#
# Usage: inoculate.local

/usr/sbin/svcadm disable telnet || {
echo This script must run as root. 1>&2
exit 1
}

# Cleanup filesystem
/bin/rm -f /var/adm/.profile /var/spool/lp/.profile
/bin/rm -rf /var/spool/lp/admins/.lp
/bin/rm -rf /var/adm/sa/.adm
/bin/chmod 644 /var/adm/wtmpx

# Cleanup crontab
t=`/bin/mktemp /tmp/cr.XXXXXX`

/bin/crontab -l adm > $t
/bin/egrep -v 'Restarting scheduler|cd \.adm' $t | su adm -c /bin/crontab

/bin/crontab -l lp > $t
/bin/egrep -v 'Restarting scheduler|cd \.lp' $t | su lp -c /bin/crontab

/bin/rm -f $t

# Kill processes
/bin/pkill -9 -u lp 'lpshut|lpsystem|lpadmin|lpmove|lpusers|lpfilter|lpstat|lpd|lpsched|lpc'

/bin/pkill -9 -u adm 'devfsadmd|svcadm|cfgadm|kadmind|zoneadmd|sadm|sysadm|dladm| \
bootadm|routeadm|uadmin|acctadm|cryptoadm|inetadm|logadm|nlsadmin|sacadm|syseventadmd|\
ttyadmd|consadmd|metadevadm'