Rebuilding Veritas vxdg and vxdisk

Categories System Administration

Ran in to this issue a while back, and at the time couldn’t find many resources online to help with this since I was in a rush to get this back up and running. If you’ve mounted LUNs and the disk and group doesn’t show up, and the status is online and invalid as shown below, here is the procedure to do it with Veritas Volume manager. To access it cat the .cfgrec file and pipe it to /usr/sbin/vprint -D – -ht. You can compare it the current config by running vxprint -ht. This issue occured to us when the storage was removed while a database was online, and the disks were then re-added hot. Did this recovery running as root since all of the commands do require root access.

The basic outline is:
– Get veritas to start managing the disks
– Init the disk group, if disk group created add disks to disk group
– Make subdisk
– Make plex
– Make volume
– Turn on volume
– fsck, mount and verify volume

Old config data is stored in /etc/vx/cbr/bk.
<code>[root@labdb01v bk]# vxdisk list
DEVICE TYPE DISK GROUP STATUS
emc_clariion0_105 auto:none – – online invalid
emc_clariion0_106 auto:none – – online invalid
emc_clariion0_107 auto:none – – online invalid</code>

First we will need to tell veritas to start managing the disks using vxdisksetup -i. This will get veritas to initialize the disk and clear the invalid status.
<code>/opt/VRTS/bin/vxdisksetup -i emc_clariion0_105</code>
You can see below that it cleared the invalid status, continue and do this for all disks.
<code>[root@labdb01v bk]# vxdisk list
DEVICE TYPE DISK GROUP STATUS
emc_clariion0_105 auto:cdsdisk – – online
emc_clariion0_106 auto:none – – online invalid
emc_clariion0_107 auto:none – – online invalid</code>

Once we have ran vxdisksetup on all of our disks, the 3 are online and ready for the disk groups.
<code>[root@labdb01v bk]# vxdisk list
DEVICE TYPE DISK GROUP STATUS
emc_clariion0_105 auto:cdsdisk – – online
emc_clariion0_106 auto:cdsdisk – – online
emc_clariion0_107 auto:cdsdisk – – online thinrclm</code>

If you need to create new disk groups, as we do above, run the following commands. This initializes the new disk groups and adds the disks mentioned after the disk group name in to the disk group.
<code>vxdg init datdg emc_clariion0_107</code>
<code>vxdg init redodg emc_clariion0_105 emc_clariion0_106</code>
If you have existing disk groups and the disks just need to be added, you can run this command:
<code>vxdg adddisk -g GROUPNAME</code>

Once we’ve initialized the disk groups, we can now take a look at our disks again and see they have been added:
<code>[root@labdb01v bk]# vxdisk list
DEVICE TYPE DISK GROUP STATUS
emc_clariion0_105 auto:cdsdisk emc_clariion0_105 redodg online
emc_clariion0_106 auto:cdsdisk emc_clariion0_106 redodg online
emc_clariion0_107 auto:cdsdisk emc_clariion0_107 datdg online thinrclm</code>

Now for the fun part. Using vxmake, we will create the subdisks, plexes, and volumes for each disk group we have. We’ll pipe the info from our cfrec to vxprint. The information that I have put in bold is the part that we care about.

<code>[root@labdb01v bk]# cat datdg.1357922182.17.ldev1db01v/1357922182.17.ldev1db01v.cfgrec |/usr/sbin/vxprint -D – -ht
Disk group: datdg

DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME RVG KSTATE STATE NVOLUME
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO
EX NAME ASSOC VC PERMS MODE STATE
SR NAME KSTATE

<strong>dg datdg default default 18000 1357922182.17.ldev1db01v

dm emc_clariion0_107 emc_clariion0_107 auto 65536 838671792 –

v datavol01 – ENABLED ACTIVE 838670336 SELECT – fsgen
pl datavol01-01 datavol01 ENABLED ACTIVE 838670336 CONCAT – RW
sd emc_clariion0_107-01 datavol01-01 emc_clariion0_107 0 838670336 0 emc_clariion0_107 ENA</strong></code>

First we need the subdisk. The syntax will be <code>vxmake -g $GROUPNAME sd $SUBDISKNAME $DISKNAME,$DISKOFFSET,$LENGTH</code>
This is from the output above: <code>SD SUBDISKNAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
sd <strong>emc_clariion0_107-01</strong> datavol01-01 <strong>emc_clariion0_107 0 838670336</strong> 0 emc_clariion0_107 ENA</code>
<code>vxmake -g datdg sd emc_clariion0_107-01 emc_clariion0_107,0,838670336</code>

Next we make the plex. The syntax will be <code>vxmake -g $GROUPNAME plex $PLEXNAME layout=$LAYOUT sd=$SUBDISK</code>
<code>PL PLEXNAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
pl datavol01-01 datavol01 ENABLED ACTIVE 838670336 CONCAT – RW</code>
<code>vxmake -g datdg plex datavol01-01 layout=concat sd=emc_clariion0_107-01
</code>

Now we will add the volume. The syntax will be <code>vxmake -g $GROUPNAME -U $UTYPE vol $VOLNAME plex=</code>
<code>V VOLNAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
v datavol01 – ENABLED ACTIVE 838670336 SELECT – fsgen</code>

<code>vxmake -g datdg -U fsgen vol datavol01 plex=datavol01-01</code>

Now we start the volume we just created. <code>vxvol -g datdg start datavol01</code>
Once we have all of that done, we’ll do an fstyp to take a look at our partition to make sure it’s still there
<code>[root@labdb01v bk]# /opt/VRTS/bin/fstyp -v /dev/vx/rdsk/datdg/datavol01
vxfs
magic a501fcf5 version 7 ctime Fri 11 Jan 2013 08:45:18 AM PST
logstart 0 logend 0
bsize 8192 size 52416896 dsize 52416896 ninode 0 nau 0
defiextsize 0 ilbsize 0 immedlen 96 ndaddr 10
aufirst 0 emap 0 imap 0 iextop 0 istart 0
bstart 0 femap 0 fimap 0 fiextop 0 fistart 0 fbstart 0
nindir 2048 aulen 32768 auimlen 0 auemlen 1
auilen 0 aupad 0 aublocks 32768 maxtier 15
inopb 32 inopau 0 ndiripau 0 iaddrlen 1 bshift 13
inoshift 5 bmask ffffe000 boffmask 1fff checksum f6005aca
oltext1 11 oltext2 9988 oltsize 1 checksum2 0
free 18285230 ifree 0
efree 228 51 1 0 0 1 1 0 1 0 0 0 0 0 0 2 2 2 0 2 2 1 1 1 0 0 0 0 0 0 0 0</code>

Things look good, so lets run an fsck on it.
<code>[root@labdb01v bk]# fsck -t vxfs -o full /dev/vx/rdsk/datdg/datavol01
fsck 1.39 (29-May-2006)
log replay in progress
pass0 – checking structural files
pass1 – checking inode sanity and blocks
pass2 – checking directory linkage
pass3 – checking reference counts
pass4 – checking resource maps
OK to clear log? (ynq)y
flush fileset headers? (ynq)y
set state to CLEAN? (ynq)y</code>

The disk looks good, so now you can mount it and check if you have data. If you do, you’ve successfully recovered!
Rinse and repeat for any other disk groups you need to do.

 

Leave a Reply

Your email address will not be published. Required fields are marked *