Oracle Database

Oracle Database 9i, 10g,11g, 12c

Cleared 1Z0-997 Oracle Cloud Infra. Certified Architect Professional…

Posted on May 6, 2020 by Cloud Oracle Cloud Oracle Database 1

Cleared following 3 Oracle Cloud Infrastructure exams in a row in three days. Thanks to great Oracle COVID offers. wish to attempt more exams but could not find more slots serially..

Oracle Cloud Infrastructure 2019 Certified Architect Professional
https://www.youracclaim.com/badges/ba7f88e4-25a4-4b2e-92af-dd4c6dd081f1/public_url
Oracle Autonomous Database Cloud 2019 Certified Specialist.
https://www.youracclaim.com/badges/787ba425-6a23-495c-a2f3-ab6082f026fe/public_url
Oracle Cloud Infrastructure 2019 Architect Associate
https://www.youracclaim.com/badges/2b87f825-d1b8-4746-8745-40975ca49d7f/public_url

Though i cleared all three exams in 3 days, it does not mean its that simple. I do have 14+ years experience with Oracle products and Oracle Cloud Infrastructure since its inception. Also i had to do weeks of preparation to get them cleared.

Since some one recently asked how to clear 1Z0-997, thought of putting below few points i followed. Starting with 1Z0-997 Oracle has raised the bar of toughest level of their exams. About 50 questions all with practical scenarios based.

I’ve taken the sample exam given by oracle , and practiced those sample exams by changing answers and following the related content in Oracle cloud document.
All i would suggest grab 50 sample questions from oracle. Provided one example below where i saw this question appeared two times with different options for answer in the real exam

A sample alike question appeared in exam with different options or slightly modified question.

Majority of questions were based on sample practice questions but were modified drastically.
Another example would be in sample question it looks like “an eCommerce company is running on OCI and setup Load balancer and Compute instances….”
In Real exams everything remained same and the question changed to “A fashion industry (which deals with fashion, bags, leathers etc.) running 4 Compute instances behind a Load balancer ” and few change in answer options…
Hopefully this might help someone.

Special thanks to Rohit Rathi, Flavio Pereira and Respective Oracle Product managers for Level-100/200 videos.. But to be honest in order to clear 1Z0-997 the video/training is not sufficient you have to imagine and practice a lot of scenarios.

How to Clean up ODA Failed Deployment.

Posted on November 4, 2019 by Oracle Database Oracle Database Appliance 0

CLEANING UP FAILED ODA DEPLOYMENT.

You can find cleanup.pl under /opt/oracle/oak/onecmd location. Repeat the steps on both ODA nodes.


[root@db-odand01 ~]# cd /opt/oracle/oak/onecmd/
[root@db-odand01 onecmd]# perl cleanup.pl -griduser oragrid -dbuser dbadmin -f
INFO: *******************************************************************
INFO: ** Starting process to cleanup provisioned host db-odand01      **
INFO: *******************************************************************
WARNING: The force mode is specified for cleanup of the provisioned
WARNING: system. The cleanup process will erase 4KB of ASM header on
WARNING: all ASM disks. It may cause ASM disk groups to be unmountable
WARNING: and may also cause data loss. Backup your data or exit the
WARNING: cleanup process.
Do you want to continue (yes/no) : yes
INFO: nodes will be rebooted
Do you want to continue (yes/no) : yes
INFO: /u01/app/18.0.0.0/oragrid/bin/crsctl.bin

INFO: /u01/app/18.0.0.0/oragrid/.patch_storage/28300849_Sep_11_2018_03_21_23/files/bin/crsctl.bin

INFO: /u01/app/18.0.0.0/oragrid/.patch_storage/28977457_Dec_1_2018_09_47_25/files/bin/crsctl.bin

INFO: GI bits /u01/app/18.0.0.0/oragrid found on system under /u01/app directory...

SUCCESS: cmd : parted -s /dev/mapper/SSD_E0_S00_2348916120 unit s print| grep primary | awk '{print $2}' | awk -Fs '{print $1}'
 retStatus : 0 output :
 4096 5626138624
SUCCESS: Found starting sector for all disk partitions

FAILURE: cmd : rpm -q tfa-oda
 retStatus : 1 output :
 package tfa-oda is not installed

INFO: No need to uninstall tfa-oda rpm
FAILURE: cmd : rpm -q orachk
 retStatus : 1 output :
 package orachk is not installed

INFO: No need to uninstall orachk rpm
SUCCESS: cmd : export ORACLE_BASE=/tmp; export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/asmcmd --nocp afd_state
 retStatus : 0 output :

SUCCESS: cmd : export ORACLE_BASE=/tmp; export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/asmcmd --nocp afd_lsdsk
 retStatus : 0 output :

SUCCESS: cmd : export ORACLE_BASE=/tmp; export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/asmcmd --nocp afd_filter -d --all
 retStatus : 0 output :

SUCCESS: cmd : export ORACLE_BASE=/tmp; export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/asmcmd --nocp afd_lsdsk
 retStatus : 0 output :

SUCCESS: cmd : export ORACLE_BASE=/tmp; export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/asmcmd --nocp afd_lsdsk | grep ENABLED | awk '{print $3}'
 retStatus : 0 output :

SUCCESS: AFD filering disabled on all devices
FAILURE: cmd : partprobe 2> /dev/null
 retStatus : 1 output :
 Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy).  As a result, it may not reflect all of your changes until after reboot.
 Warning: WARNING: the kernel failed to re-read the partition table on /dev/sdb (Device or resource busy).  As a result, it may not reflect all of your changes until after reboot.

SUCCESS: cmd : dd if=/dev/zero of=/dev/sdc seek=500 bs=512 count=1 oflag=direct status=noxfer
 retStatus : 0 output :
 1+0 records in
 1+0 records out


 octets=0 0 0 0
SUCCESS: cmd : echo 0100000000001000 | dd of=/dev/sdc seek=256000 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out


 octets=0 0 0 1
SUCCESS: cmd : echo 020000014F581000 | dd of=/dev/sdc seek=256016 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out

SUCCESS: cmd : echo FF00000000000000 | dd of=/dev/sdc seek=256032 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out

SUCCESS: cmd : dd if=/dev/zero of=/dev/sdd seek=500 bs=512 count=1 oflag=direct status=noxfer
 retStatus : 0 output :
 1+0 records in
 1+0 records out


 octets=0 0 0 0
SUCCESS: cmd : echo 0100000000001000 | dd of=/dev/sdd seek=256000 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out


 octets=0 0 0 1
SUCCESS: cmd : echo 020000014F581000 | dd of=/dev/sdd seek=256016 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out

SUCCESS: cmd : echo FF00000000000000 | dd of=/dev/sdd seek=256032 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out

SUCCESS: cmd : dd if=/dev/zero of=/dev/sde seek=500 bs=512 count=1 oflag=direct status=noxfer
 retStatus : 0 output :
 1+0 records in
 1+0 records out


 octets=0 0 0 0
SUCCESS: cmd : echo 0100000000001000 | dd of=/dev/sde seek=256000 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out


 octets=0 0 0 1
SUCCESS: cmd : echo 020000014F581000 | dd of=/dev/sde seek=256016 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out

SUCCESS: cmd : echo FF00000000000000 | dd of=/dev/sde seek=256032 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out

SUCCESS: cmd : dd if=/dev/zero of=/dev/sdf seek=500 bs=512 count=1 oflag=direct status=noxfer
 retStatus : 0 output :
 1+0 records in
 1+0 records out


 octets=0 0 0 0
SUCCESS: cmd : echo 0100000000001000 | dd of=/dev/sdf seek=256000 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out


 octets=0 0 0 1
SUCCESS: cmd : echo 020000014F581000 | dd of=/dev/sdf seek=256016 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out

SUCCESS: cmd : echo FF00000000000000 | dd of=/dev/sdf seek=256032 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out

SUCCESS: cmd : dd if=/dev/zero of=/dev/sdg seek=500 bs=512 count=1 oflag=direct status=noxfer
 retStatus : 0 output :
 1+0 records in
 1+0 records out


 octets=0 0 0 0
SUCCESS: cmd : echo 0100000000001000 | dd of=/dev/sdg seek=256000 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out


 octets=0 0 0 1
SUCCESS: cmd : echo 020000014F581000 | dd of=/dev/sdg seek=256016 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out

SUCCESS: cmd : echo FF00000000000000 | dd of=/dev/sdg seek=256032 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out

INFO: Partitions not found on disk, skip writing of partition info on empty5
INFO: Partitions not found on disk, skip writing of partition info on empty6
INFO: Partitions not found on disk, skip writing of partition info on empty7
INFO: Partitions not found on disk, skip writing of partition info on empty8
INFO: Partitions not found on disk, skip writing of partition info on empty9
INFO: Partitions not found on disk, skip writing of partition info on empty10
INFO: Partitions not found on disk, skip writing of partition info on empty11
INFO: Partitions not found on disk, skip writing of partition info on empty12
INFO: Partitions not found on disk, skip writing of partition info on empty13
INFO: Partitions not found on disk, skip writing of partition info on empty14
INFO: Partitions not found on disk, skip writing of partition info on empty15
INFO: Partitions not found on disk, skip writing of partition info on empty16
INFO: Partitions not found on disk, skip writing of partition info on empty17
INFO: Partitions not found on disk, skip writing of partition info on empty18
INFO: Partitions not found on disk, skip writing of partition info on empty19
SUCCESS: cmd : dd if=/dev/zero of=/dev/sdh seek=500 bs=512 count=1 oflag=direct status=noxfer
 retStatus : 0 output :
 1+0 records in
 1+0 records out


 octets=0 0 0 0
SUCCESS: cmd : echo 0100000000001000 | dd of=/dev/sdh seek=256000 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out

SUCCESS: cmd : echo FF00000000000000 | dd of=/dev/sdh seek=256016 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out

SUCCESS: cmd : dd if=/dev/zero of=/dev/sdi seek=500 bs=512 count=1 oflag=direct status=noxfer
 retStatus : 0 output :
 1+0 records in
 1+0 records out


 octets=0 0 0 0
SUCCESS: cmd : echo 0100000000001000 | dd of=/dev/sdi seek=256000 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out

SUCCESS: cmd : echo FF00000000000000 | dd of=/dev/sdi seek=256016 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out

SUCCESS: cmd : dd if=/dev/zero of=/dev/sdj seek=500 bs=512 count=1 oflag=direct status=noxfer
 retStatus : 0 output :
 1+0 records in
 1+0 records out


 octets=0 0 0 0
SUCCESS: cmd : echo 0100000000001000 | dd of=/dev/sdj seek=256000 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out

SUCCESS: cmd : echo FF00000000000000 | dd of=/dev/sdj seek=256016 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out

SUCCESS: cmd : dd if=/dev/zero of=/dev/sdk seek=500 bs=512 count=1 oflag=direct status=noxfer
 retStatus : 0 output :
 1+0 records in
 1+0 records out


 octets=0 0 0 0
SUCCESS: cmd : echo 0100000000001000 | dd of=/dev/sdk seek=256000 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out

SUCCESS: cmd : echo FF00000000000000 | dd of=/dev/sdk seek=256016 bs=1 count=16 status=noxfer 2>/dev/null
 retStatus : 0 output :
 16+0 records in
 16+0 records out

INFO: ASM disk /dev/mapper/SSD_E0_S00_2348916120p1 found, proceed with cleanup
SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S00_2348916120p1 | egrep "kfbh.type|dskname|grpname|fgname|HOUR|USEC|hdrsts|kfdhdb.driver|grpname|grptyp"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID

SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S00_2348916120p1 | egrep "kfbh.type|kfdhdb.hdrsts"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
INFO: KFBTYP_INVALID found in ASM header on /dev/mapper/SSD_E0_S00_2348916120p1
INFO: Invalid ASM header reported by kfed
INFO: Disk /dev/mapper/SSD_E0_S00_2348916120p1 looks clean of asm header, erase not required.
INFO: ASM disk /dev/mapper/SSD_E0_S00_2348916120p2 found, proceed with cleanup
SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S00_2348916120p2 | egrep "kfbh.type|dskname|grpname|fgname|HOUR|USEC|hdrsts|kfdhdb.driver|grpname|grptyp"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID

SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S00_2348916120p2 | egrep "kfbh.type|kfdhdb.hdrsts"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
INFO: KFBTYP_INVALID found in ASM header on /dev/mapper/SSD_E0_S00_2348916120p2
INFO: Invalid ASM header reported by kfed
INFO: Disk /dev/mapper/SSD_E0_S00_2348916120p2 looks clean of asm header, erase not required.
INFO: ASM disk /dev/mapper/SSD_E0_S01_2348916124p1 found, proceed with cleanup
SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S01_2348916124p1 | egrep "kfbh.type|dskname|grpname|fgname|HOUR|USEC|hdrsts|kfdhdb.driver|grpname|grptyp"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID

SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S01_2348916124p1 | egrep "kfbh.type|kfdhdb.hdrsts"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
INFO: KFBTYP_INVALID found in ASM header on /dev/mapper/SSD_E0_S01_2348916124p1
INFO: Invalid ASM header reported by kfed
INFO: Disk /dev/mapper/SSD_E0_S01_2348916124p1 looks clean of asm header, erase not required.
INFO: ASM disk /dev/mapper/SSD_E0_S01_2348916124p2 found, proceed with cleanup
SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S01_2348916124p2 | egrep "kfbh.type|dskname|grpname|fgname|HOUR|USEC|hdrsts|kfdhdb.driver|grpname|grptyp"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID

SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S01_2348916124p2 | egrep "kfbh.type|kfdhdb.hdrsts"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
INFO: KFBTYP_INVALID found in ASM header on /dev/mapper/SSD_E0_S01_2348916124p2
INFO: Invalid ASM header reported by kfed
INFO: Disk /dev/mapper/SSD_E0_S01_2348916124p2 looks clean of asm header, erase not required.
INFO: ASM disk /dev/mapper/SSD_E0_S02_2348925144p1 found, proceed with cleanup
SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S02_2348925144p1 | egrep "kfbh.type|dskname|grpname|fgname|HOUR|USEC|hdrsts|kfdhdb.driver|grpname|grptyp"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID

SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S02_2348925144p1 | egrep "kfbh.type|kfdhdb.hdrsts"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
INFO: KFBTYP_INVALID found in ASM header on /dev/mapper/SSD_E0_S02_2348925144p1
INFO: Invalid ASM header reported by kfed
INFO: Disk /dev/mapper/SSD_E0_S02_2348925144p1 looks clean of asm header, erase not required.
INFO: ASM disk /dev/mapper/SSD_E0_S02_2348925144p2 found, proceed with cleanup
SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S02_2348925144p2 | egrep "kfbh.type|dskname|grpname|fgname|HOUR|USEC|hdrsts|kfdhdb.driver|grpname|grptyp"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID

SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S02_2348925144p2 | egrep "kfbh.type|kfdhdb.hdrsts"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
INFO: KFBTYP_INVALID found in ASM header on /dev/mapper/SSD_E0_S02_2348925144p2
INFO: Invalid ASM header reported by kfed
INFO: Disk /dev/mapper/SSD_E0_S02_2348925144p2 looks clean of asm header, erase not required.
INFO: ASM disk /dev/mapper/SSD_E0_S03_2348918204p1 found, proceed with cleanup
SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S03_2348918204p1 | egrep "kfbh.type|dskname|grpname|fgname|HOUR|USEC|hdrsts|kfdhdb.driver|grpname|grptyp"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID

SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S03_2348918204p1 | egrep "kfbh.type|kfdhdb.hdrsts"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
INFO: KFBTYP_INVALID found in ASM header on /dev/mapper/SSD_E0_S03_2348918204p1
INFO: Invalid ASM header reported by kfed
INFO: Disk /dev/mapper/SSD_E0_S03_2348918204p1 looks clean of asm header, erase not required.
INFO: ASM disk /dev/mapper/SSD_E0_S03_2348918204p2 found, proceed with cleanup
SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S03_2348918204p2 | egrep "kfbh.type|dskname|grpname|fgname|HOUR|USEC|hdrsts|kfdhdb.driver|grpname|grptyp"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID

SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S03_2348918204p2 | egrep "kfbh.type|kfdhdb.hdrsts"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
INFO: KFBTYP_INVALID found in ASM header on /dev/mapper/SSD_E0_S03_2348918204p2
INFO: Invalid ASM header reported by kfed
INFO: Disk /dev/mapper/SSD_E0_S03_2348918204p2 looks clean of asm header, erase not required.
INFO: ASM disk /dev/mapper/SSD_E0_S04_2348932952p1 found, proceed with cleanup
SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S04_2348932952p1 | egrep "kfbh.type|dskname|grpname|fgname|HOUR|USEC|hdrsts|kfdhdb.driver|grpname|grptyp"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID

SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S04_2348932952p1 | egrep "kfbh.type|kfdhdb.hdrsts"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
INFO: KFBTYP_INVALID found in ASM header on /dev/mapper/SSD_E0_S04_2348932952p1
INFO: Invalid ASM header reported by kfed
INFO: Disk /dev/mapper/SSD_E0_S04_2348932952p1 looks clean of asm header, erase not required.
INFO: ASM disk /dev/mapper/SSD_E0_S04_2348932952p2 found, proceed with cleanup
SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S04_2348932952p2 | egrep "kfbh.type|dskname|grpname|fgname|HOUR|USEC|hdrsts|kfdhdb.driver|grpname|grptyp"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID

SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S04_2348932952p2 | egrep "kfbh.type|kfdhdb.hdrsts"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
INFO: KFBTYP_INVALID found in ASM header on /dev/mapper/SSD_E0_S04_2348932952p2
INFO: Invalid ASM header reported by kfed
INFO: Disk /dev/mapper/SSD_E0_S04_2348932952p2 looks clean of asm header, erase not required.
INFO: ASM disk /dev/mapper/SSD_E0_S20_2181083740p1 found, proceed with cleanup
SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S20_2181083740p1 | egrep "kfbh.type|dskname|grpname|fgname|HOUR|USEC|hdrsts|kfdhdb.driver|grpname|grptyp"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID

SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S20_2181083740p1 | egrep "kfbh.type|kfdhdb.hdrsts"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
INFO: KFBTYP_INVALID found in ASM header on /dev/mapper/SSD_E0_S20_2181083740p1
INFO: Invalid ASM header reported by kfed
INFO: Disk /dev/mapper/SSD_E0_S20_2181083740p1 looks clean of asm header, erase not required.
INFO: ASM disk /dev/mapper/SSD_E0_S21_2181069888p1 found, proceed with cleanup
SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S21_2181069888p1 | egrep "kfbh.type|dskname|grpname|fgname|HOUR|USEC|hdrsts|kfdhdb.driver|grpname|grptyp"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID

SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S21_2181069888p1 | egrep "kfbh.type|kfdhdb.hdrsts"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
INFO: KFBTYP_INVALID found in ASM header on /dev/mapper/SSD_E0_S21_2181069888p1
INFO: Invalid ASM header reported by kfed
INFO: Disk /dev/mapper/SSD_E0_S21_2181069888p1 looks clean of asm header, erase not required.
INFO: ASM disk /dev/mapper/SSD_E0_S22_2181082540p1 found, proceed with cleanup
SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S22_2181082540p1 | egrep "kfbh.type|dskname|grpname|fgname|HOUR|USEC|hdrsts|kfdhdb.driver|grpname|grptyp"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID

SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S22_2181082540p1 | egrep "kfbh.type|kfdhdb.hdrsts"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
INFO: KFBTYP_INVALID found in ASM header on /dev/mapper/SSD_E0_S22_2181082540p1
INFO: Invalid ASM header reported by kfed
INFO: Disk /dev/mapper/SSD_E0_S22_2181082540p1 looks clean of asm header, erase not required.
INFO: ASM disk /dev/mapper/SSD_E0_S23_2181077384p1 found, proceed with cleanup
SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S23_2181077384p1 | egrep "kfbh.type|dskname|grpname|fgname|HOUR|USEC|hdrsts|kfdhdb.driver|grpname|grptyp"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID

SUCCESS: cmd : export ORACLE_HOME=/u01/app/18.0.0.0/oragrid; /u01/app/18.0.0.0/oragrid/bin/kfed read /dev/mapper/SSD_E0_S23_2181077384p1 | egrep "kfbh.type|kfdhdb.hdrsts"
 retStatus : 0 output :
 kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
INFO: KFBTYP_INVALID found in ASM header on /dev/mapper/SSD_E0_S23_2181077384p1
INFO: Invalid ASM header reported by kfed
INFO: Disk /dev/mapper/SSD_E0_S23_2181077384p1 looks clean of asm header, erase not required.
SUCCESS: cmd : /opt/oracle/oak/bin/odaadmcli fstop oak
 retStatus : 0 output :
 2019-10-05 08:39:25.600716172:[init.oak]:[Stopping oakd]
 2019-10-05 08:39:28.692114796:[init.oak]:[Successfully stopped the oakd..]

SUCCESS: cmd : /bin/dd if=/dev/mapper/SSD_E0_S00_2348916120 skip=20 bs=1k count=4 iflag=direct status=none | hexdump -C
 retStatus : 0 output :
 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 *
 00001000

SUCCESS: cmd : /bin/dd if=/dev/mapper/SSD_E0_S00_2348916120 skip=20 bs=1k count=4 iflag=direct status=none | egrep -q "1745NMQ04H";echo $?
 retStatus : 0 output : 1
INFO: OAKD header not found on /dev/mapper/SSD_E0_S00_2348916120
INFO: OAKD header not found
SUCCESS: cmd : /bin/dd if=/dev/mapper/SSD_E0_S01_2348916124 skip=20 bs=1k count=4 iflag=direct status=none | hexdump -C
 retStatus : 0 output :
 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 *
 00001000

SUCCESS: cmd : /bin/dd if=/dev/mapper/SSD_E0_S01_2348916124 skip=20 bs=1k count=4 iflag=direct status=none | egrep -q "1745NMQ04H";echo $?
 retStatus : 0 output : 1
INFO: OAKD header not found on /dev/mapper/SSD_E0_S01_2348916124
INFO: OAKD header not found
SUCCESS: cmd : /bin/dd if=/dev/mapper/SSD_E0_S02_2348925144 skip=20 bs=1k count=4 iflag=direct status=none | hexdump -C
 retStatus : 0 output :
 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 *
 00001000

SUCCESS: cmd : /bin/dd if=/dev/mapper/SSD_E0_S02_2348925144 skip=20 bs=1k count=4 iflag=direct status=none | egrep -q "1745NMQ04H";echo $?
 retStatus : 0 output : 1
INFO: OAKD header not found on /dev/mapper/SSD_E0_S02_2348925144
INFO: OAKD header not found
SUCCESS: cmd : /bin/dd if=/dev/mapper/SSD_E0_S03_2348918204 skip=20 bs=1k count=4 iflag=direct status=none | hexdump -C
 retStatus : 0 output :
 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 *
 00001000

SUCCESS: cmd : /bin/dd if=/dev/mapper/SSD_E0_S03_2348918204 skip=20 bs=1k count=4 iflag=direct status=none | egrep -q "1745NMQ04H";echo $?
 retStatus : 0 output : 1
INFO: OAKD header not found on /dev/mapper/SSD_E0_S03_2348918204
INFO: OAKD header not found
SUCCESS: cmd : /bin/dd if=/dev/mapper/SSD_E0_S04_2348932952 skip=20 bs=1k count=4 iflag=direct status=none | hexdump -C
 retStatus : 0 output :
 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 *
 00001000

SUCCESS: cmd : /bin/dd if=/dev/mapper/SSD_E0_S04_2348932952 skip=20 bs=1k count=4 iflag=direct status=none | egrep -q "1745NMQ04H";echo $?
 retStatus : 0 output : 1
INFO: OAKD header not found on /dev/mapper/SSD_E0_S04_2348932952
INFO: OAKD header not found
SUCCESS: cmd : /bin/dd if=/dev/mapper/SSD_E0_S20_2181083740 skip=20 bs=1k count=4 iflag=direct status=none | hexdump -C
 retStatus : 0 output :
 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 *
 00001000

SUCCESS: cmd : /bin/dd if=/dev/mapper/SSD_E0_S20_2181083740 skip=20 bs=1k count=4 iflag=direct status=none | egrep -q "1745NMQ04H";echo $?
 retStatus : 0 output : 1
INFO: OAKD header not found on /dev/mapper/SSD_E0_S20_2181083740
INFO: OAKD header not found
SUCCESS: cmd : /bin/dd if=/dev/mapper/SSD_E0_S21_2181069888 skip=20 bs=1k count=4 iflag=direct status=none | hexdump -C
 retStatus : 0 output :
 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 *
 00001000

SUCCESS: cmd : /bin/dd if=/dev/mapper/SSD_E0_S21_2181069888 skip=20 bs=1k count=4 iflag=direct status=none | egrep -q "1745NMQ04H";echo $?
 retStatus : 0 output : 1
INFO: OAKD header not found on /dev/mapper/SSD_E0_S21_2181069888
INFO: OAKD header not found
SUCCESS: cmd : /bin/dd if=/dev/mapper/SSD_E0_S22_2181082540 skip=20 bs=1k count=4 iflag=direct status=none | hexdump -C
 retStatus : 0 output :
 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 *
 00001000

SUCCESS: cmd : /bin/dd if=/dev/mapper/SSD_E0_S22_2181082540 skip=20 bs=1k count=4 iflag=direct status=none | egrep -q "1745NMQ04H";echo $?
 retStatus : 0 output : 1
INFO: OAKD header not found on /dev/mapper/SSD_E0_S22_2181082540
INFO: OAKD header not found
SUCCESS: cmd : /bin/dd if=/dev/mapper/SSD_E0_S23_2181077384 skip=20 bs=1k count=4 iflag=direct status=none | hexdump -C
 retStatus : 0 output :
 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 *
 00001000

SUCCESS: cmd : /bin/dd if=/dev/mapper/SSD_E0_S23_2181077384 skip=20 bs=1k count=4 iflag=direct status=none | egrep -q "1745NMQ04H";echo $?
 retStatus : 0 output : 1
INFO: OAKD header not found on /dev/mapper/SSD_E0_S23_2181077384
INFO: OAKD header not found
SUCCESS: cmd : sync; echo 3 > /proc/sys/vm/drop_caches
 retStatus : 0 output :

INFO: Executing </u01/app/18.0.0.0/oragrid/bin/acfsload stop>
FAILURE: cmd : /u01/app/18.0.0.0/oragrid/bin/acfsload stop
 retStatus : 126 output :
 /u01/app/18.0.0.0/oragrid/bin/acfsload: line 34: /lib/acfstoolsdriver.sh: No such file or directory
 /u01/app/18.0.0.0/oragrid/bin/acfsload: line 34: exec: /lib/acfstoolsdriver.sh: cannot execute: No such file or directory

INFO: Executing </u01/app/18.0.0.0/oragrid/bin/afdload stop>
FAILURE: cmd : /u01/app/18.0.0.0/oragrid/bin/afdload stop
 retStatus : 126 output :
 /u01/app/18.0.0.0/oragrid/bin/afdload: line 34: /lib/afdtoolsdriver.sh: No such file or directory
 /u01/app/18.0.0.0/oragrid/bin/afdload: line 34: exec: /lib/afdtoolsdriver.sh: cannot execute: No such file or directory

INFO: Executing </u01/app/18.0.0.0/oragrid/bin/acfsroot uninstall>
SUCCESS: cmd : /u01/app/18.0.0.0/oragrid/bin/acfsroot uninstall
 retStatus : 0 output :
 ACFS-9313: No ADVM/ACFS installation detected.

INFO: Executing </u01/app/18.0.0.0/oragrid/bin/afdroot uninstall>
SUCCESS: cmd : /u01/app/18.0.0.0/oragrid/bin/afdroot uninstall
 retStatus : 0 output :
 AFD-634: Removing previous AFD installation.
 AFD-635: Previous AFD components successfully removed.
 AFD-9294: updating file /etc/sysconfig/oracledrivers.conf

INFO: Executing <rm -f /etc/oracleafd.conf>
SUCCESS: cmd : rm -f /etc/oracleafd.conf
 retStatus : 0 output :

INFO: Executing <unlink /etc/init.d/afd>
FAILURE: cmd : unlink /etc/init.d/afd
 retStatus : 1 output :
 unlink: cannot unlink `/etc/init.d/afd': No such file or directory

INFO: Executing <find /etc/rc.d -name "*afd*" -print -exec rm -rf {} \;>
SUCCESS: cmd : find /etc/rc.d -name "*afd*" -print -exec rm -rf {} \;
 retStatus : 0 output :

INFO: Executing </u01/app/18.0.0.0/oragrid/bin/afddriverstate loaded>
FAILURE: cmd : /u01/app/18.0.0.0/oragrid/bin/afddriverstate loaded
 retStatus : 1 output :
 AFD-9206: AFD device driver loaded status: 'false'

INFO: Executing </u01/app/18.0.0.0/oragrid/bin/afddriverstate installed>
FAILURE: cmd : /u01/app/18.0.0.0/oragrid/bin/afddriverstate installed
 retStatus : 1 output :
 AFD-9204: AFD device driver installed status: 'false'

INFO: Executing </u01/app/18.0.0.0/oragrid/bin/afddriverstate  version>
FAILURE: cmd : /u01/app/18.0.0.0/oragrid/bin/afddriverstate  version
 retStatus : 1 output :
 AFD-642: AFD not installed

INFO: Executing </u01/app/18.0.0.0/oragrid/bin/afddriverstate  supported>
SUCCESS: cmd : /u01/app/18.0.0.0/oragrid/bin/afddriverstate  supported
 retStatus : 0 output :
 AFD-9200: Supported

INFO: Executing </opt/oracle/oak/bin/odaadmcli clear config>
SUCCESS: cmd : /opt/oracle/oak/bin/odaadmcli clear config
 retStatus : 0 output :

INFO: Executing </opt/oracle/oak/bin/odaadmcli disable cluster>
FAILURE: cmd : /opt/oracle/oak/bin/odaadmcli disable cluster
 retStatus : 255 output :
 Unable to connect to oakd.

INFO: Executing </opt/oracle/oak/bin/odaadmcli fstop oak>
SUCCESS: cmd : /opt/oracle/oak/bin/odaadmcli fstop oak
 retStatus : 0 output :
 2019-10-05 08:39:44.766867037:[init.oak]:[Stopping oakd]
 2019-10-05 08:39:47.858641809:[init.oak]:[Successfully stopped the oakd..]

INFO: Executing <pkill oakd>
FAILURE: cmd : pkill oakd
 retStatus : 1 output :

INFO: Executing <rm -rf /etc/oracle/ /etc/oratab /etc/oraInst.loc /tmp/.oracle /var/tmp/.oracle /tmp/.oracle/.oracle_master_diskmon>
SUCCESS: cmd : rm -rf /etc/oracle/ /etc/oratab /etc/oraInst.loc /tmp/.oracle /var/tmp/.oracle /tmp/.oracle/.oracle_master_diskmon
 retStatus : 0 output :

INFO: Executing <rm -f /etc/inittab.crs>
SUCCESS: cmd : rm -f /etc/inittab.crs
 retStatus : 0 output :

INFO: Executing <rm -f /etc/rc.d/init.d/*has*>
SUCCESS: cmd : rm -f /etc/rc.d/init.d/*has*
 retStatus : 0 output :

INFO: Executing <rm -f /etc/init.d/*tfa*>
SUCCESS: cmd : rm -f /etc/init.d/*tfa*
 retStatus : 0 output :

INFO: Executing <rm -rf /var/tmp/*>
SUCCESS: cmd : rm -rf /var/tmp/*
 retStatus : 0 output :

INFO: Executing <rm -rf /tmp/OraInstall20*>
SUCCESS: cmd : rm -rf /tmp/OraInstall20*
 retStatus : 0 output :

INFO: Executing <rm -rf /tmp/CVU_11*>
SUCCESS: cmd : rm -rf /tmp/CVU_11*
 retStatus : 0 output :

INFO: Executing <rm -rf /tmp/install*>
SUCCESS: cmd : rm -rf /tmp/install*
 retStatus : 0 output :

INFO: Executing <rm -rf /tmp/logs*>
SUCCESS: cmd : rm -rf /tmp/logs*
 retStatus : 0 output :

INFO: Executing <rm -rf /tmp/hsperfdata*>
SUCCESS: cmd : rm -rf /tmp/hsperfdata*
 retStatus : 0 output :

INFO: Executing <rm -f /etc/rc.d/*/*init.tfa>
SUCCESS: cmd : rm -f /etc/rc.d/*/*init.tfa
 retStatus : 0 output :

INFO: Executing <rm -f /etc/rc.d/*/*ohasd>
SUCCESS: cmd : rm -f /etc/rc.d/*/*ohasd
 retStatus : 0 output :

INFO: Executing <rm -f /var/lock/subsys/ohasd>
SUCCESS: cmd : rm -f /var/lock/subsys/ohasd
 retStatus : 0 output :

INFO: Executing <rm -f /var/lock/subsys/init.tfa>
SUCCESS: cmd : rm -f /var/lock/subsys/init.tfa
 retStatus : 0 output :

INFO: Executing <rm -rf /opt/oracle/tfa>
SUCCESS: cmd : rm -rf /opt/oracle/tfa
 retStatus : 0 output :

INFO: Executing <rm -f /etc/init.d/init.cssd>
SUCCESS: cmd : rm -f /etc/init.d/init.cssd
 retStatus : 0 output :

INFO: Executing <rm -f /etc/init.d/init.crsd>
SUCCESS: cmd : rm -f /etc/init.d/init.crsd
 retStatus : 0 output :

INFO: Executing <rm -f /etc/init.d/init.crs>
SUCCESS: cmd : rm -f /etc/init.d/init.crs
 retStatus : 0 output :

INFO: Executing <rm -f /etc/init.d/init.evmd>
SUCCESS: cmd : rm -f /etc/init.d/init.evmd
 retStatus : 0 output :

INFO: Executing <rm -rf /tmp/deinstall*>
SUCCESS: cmd : rm -rf /tmp/deinstall*
 retStatus : 0 output :

INFO: Executing <rm -rf /tmp/CVU*>
SUCCESS: cmd : rm -rf /tmp/CVU*
 retStatus : 0 output :

INFO: Executing <rm -rf /tmp/OraInstall2*>
SUCCESS: cmd : rm -rf /tmp/OraInstall2*
 retStatus : 0 output :

INFO: Executing <rm -f /opt/oracle/oak/onecmd/silent/*>
SUCCESS: cmd : rm -f /opt/oracle/oak/onecmd/silent/*
 retStatus : 0 output :

INFO: Executing <rm -rf /var/opt/oracle>
SUCCESS: cmd : rm -rf /var/opt/oracle
 retStatus : 0 output :

INFO: Executing <rm -rf /opt/oracle/bin/setasmgid>
SUCCESS: cmd : rm -rf /opt/oracle/bin/setasmgid
 retStatus : 0 output :

INFO: Executing <rm -rf /opt/ORCLfmap>
SUCCESS: cmd : rm -rf /opt/ORCLfmap
 retStatus : 0 output :

INFO: Executing <rm -rf /opt/oracle/oak/conf/.clustered>
SUCCESS: cmd : rm -rf /opt/oracle/oak/conf/.clustered
 retStatus : 0 output :

INFO: Executing <rm -rf /opt/oracle/oak/conf/.configlock>
SUCCESS: cmd : rm -rf /opt/oracle/oak/conf/.configlock
 retStatus : 0 output :

INFO: Executing <rm -rf /opt/oracle/oak/conf/.offline*>
SUCCESS: cmd : rm -rf /opt/oracle/oak/conf/.offline*
 retStatus : 0 output :

INFO: Executing <rm -rf /opt/oracle/oak/conf/oak_storage_conf.xml>
SUCCESS: cmd : rm -rf /opt/oracle/oak/conf/oak_storage_conf.xml
 retStatus : 0 output :

INFO: Executing <rm -rf /opt/oracle/extapi/asmappl.config>
SUCCESS: cmd : rm -rf /opt/oracle/extapi/asmappl.config
 retStatus : 0 output :

INFO: Executing <rm -rf /tmp/oakpatch>
SUCCESS: cmd : rm -rf /tmp/oakpatch
 retStatus : 0 output :

INFO: Executing <echo non-cluster > /opt/oracle/oak/install/oakdrun>
SUCCESS: cmd : echo non-cluster > /opt/oracle/oak/install/oakdrun
 retStatus : 0 output :

INFO: Executing <rm -rf /opt/oracle/oak/conf/.x7_2_*>
SUCCESS: cmd : rm -rf /opt/oracle/oak/conf/.x7_2_*
 retStatus : 0 output :

INFO: Executing <rm -rf /opt/oracle/oak/conf/.storageTrayDetails>
SUCCESS: cmd : rm -rf /opt/oracle/oak/conf/.storageTrayDetails
 retStatus : 0 output :

INFO: Executing <rm -rf /opt/oracle/oak/afd>
SUCCESS: cmd : rm -rf /opt/oracle/oak/afd
 retStatus : 0 output :

INFO: Executing <rm -rf /opt/oracle/oak/conf/.x6_2_*>
SUCCESS: cmd : rm -rf /opt/oracle/oak/conf/.x6_2_*
 retStatus : 0 output :

INFO: Executing <sed -i '/AsmUser/s/default=.*/default="grid">/' /opt/oracle/oak/conf/oak_conf.xml>
SUCCESS: cmd : sed -i '/AsmUser/s/default=.*/default="grid">/' /opt/oracle/oak/conf/oak_conf.xml
 retStatus : 0 output :

INFO: Executing <rm -rf /opt/oracle/oak/conf/.migratedToDcsStack>
SUCCESS: cmd : rm -rf /opt/oracle/oak/conf/.migratedToDcsStack
 retStatus : 0 output :

INFO: Executing <find /opt/oracle/oak/conf -type f -name ".*" -print -exec rm -rf {} \;>
SUCCESS: cmd : find /opt/oracle/oak/conf -type f -name ".*" -print -exec rm -rf {} \;
 retStatus : 0 output :
 /opt/oracle/oak/conf/.topology

INFO: Executing <rm -rf /opt/oracle/oak/log/db-odand01/storagetopology/fmt*>
SUCCESS: cmd : rm -rf /opt/oracle/oak/log/db-odand01/storagetopology/fmt*
 retStatus : 0 output :

INFO: Executing <rm -rf /tmp/log>
SUCCESS: cmd : rm -rf /tmp/log
 retStatus : 0 output :

INFO: Executing <find /tmp -type d  -name "diag" -print -exec rm -rf {} \;>
FAILURE: cmd : find /tmp -type d  -name "diag" -print -exec rm -rf {} \;
 retStatus : 1 output :
 /tmp/diag
 find: `/tmp/diag': No such file or directory

INFO: Executing <find /tmp -type d  -name "asmcmd" -print -exec rm -rf {} \;>
SUCCESS: cmd : find /tmp -type d  -name "asmcmd" -print -exec rm -rf {} \;
 retStatus : 0 output :

INFO: Executing <grep -v init.tfa /etc/inittab > /tmp/inittab_notfa>
SUCCESS: cmd : grep -v init.tfa /etc/inittab > /tmp/inittab_notfa
 retStatus : 0 output :

INFO: Executing </bin/cp -f /tmp/inittab_notfa /etc/inittab>
SUCCESS: cmd : /bin/cp -f /tmp/inittab_notfa /etc/inittab
 retStatus : 0 output :

INFO: Executing <rm -f /tmp/inittab_notfa>
SUCCESS: cmd : rm -f /tmp/inittab_notfa
 retStatus : 0 output :

INFO: Executing <rm -f /etc/init/oracle-ohasd.conf ;>
SUCCESS: cmd : rm -f /etc/init/oracle-ohasd.conf ;
 retStatus : 0 output :

INFO: Executing <rm -f /etc/init/oracle-tfa.conf>
SUCCESS: cmd : rm -f /etc/init/oracle-tfa.conf
 retStatus : 0 output :

INFO: Executing <rm -rf /u01/app/18.0.0.0/oragrid>
SUCCESS: cmd : rm -rf /u01/app/18.0.0.0/oragrid
 retStatus : 0 output :

INFO: Executing <rm -rf /u01/app/oraInventory>
SUCCESS: cmd : rm -rf /u01/app/oraInventory
 retStatus : 0 output :

INFO: Executing <rm -rf /u01/app/oragrid>
SUCCESS: cmd : rm -rf /u01/app/oragrid
 retStatus : 0 output :

INFO: Executing <rm -rf /u01/app/dbadmin>
SUCCESS: cmd : rm -rf /u01/app/dbadmin
 retStatus : 0 output :

INFO: Executing </bin/sync;/bin/sync;/bin/sync>
SUCCESS: cmd : /bin/sync;/bin/sync;/bin/sync
 retStatus : 0 output :

INFO: Executing </bin/kill -9 `/usr/bin/pgrep -u dbadmin`>
FAILURE: cmd : /bin/kill -9 `/usr/bin/pgrep -u dbadmin`
 retStatus : 1 output :
 usage: kill [ -s signal | -p ] [ -a ] pid ...
        kill -l [ signal ]

INFO: checking if the platform is multinode
INFO: checking if the platform is BMIaas Single Node
INFO: checking if the platform is multinode
INFO: checking if the platform is BMIaas Single Node
Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /dcs-request-rw-locks
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1560)
        at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:731)
        at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:599)
        at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:362)
        at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:290)
Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ssh-auth-keys
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1560)
        at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:731)
        at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:599)
        at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:362)
        at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:290)
INFO: checking if the platform is multinode
INFO: checking if the platform is BMIaas Single Node
INFO: Private Network interface : p1p1 : 192.168.16.25

INFO: Private Network interface : p1p2 : 192.168.17.25

INFO: Executing <cp /opt/oracle/dcs/rdbaas/config/configBackups/limits.conf /etc/security/limits.conf>
SUCCESS: cmd : cp /opt/oracle/dcs/rdbaas/config/configBackups/limits.conf /etc/security/limits.conf
 retStatus : 0 output :

INFO: Executing </usr/sbin/userdel -r -f dbadmin>
SUCCESS: cmd : /usr/sbin/userdel -r -f dbadmin
 retStatus : 0 output :

INFO: Executing </usr/sbin/userdel -r -f oragrid>
SUCCESS: cmd : /usr/sbin/userdel -r -f oragrid
 retStatus : 0 output :

INFO: Executing </usr/sbin/groupdel asmadmin>
SUCCESS: cmd : /usr/sbin/groupdel asmadmin
 retStatus : 0 output :

INFO: Executing </usr/sbin/groupdel asmoper>
SUCCESS: cmd : /usr/sbin/groupdel asmoper
 retStatus : 0 output :

INFO: Executing </usr/sbin/groupdel asmdba>
SUCCESS: cmd : /usr/sbin/groupdel asmdba
 retStatus : 0 output :

INFO: Executing </usr/sbin/groupdel dbaoper>
SUCCESS: cmd : /usr/sbin/groupdel dbaoper
 retStatus : 0 output :

INFO: Executing </usr/sbin/groupdel dba>
SUCCESS: cmd : /usr/sbin/groupdel dba
 retStatus : 0 output :

INFO: Executing </usr/sbin/groupdel oinstall>
SUCCESS: cmd : /usr/sbin/groupdel oinstall
 retStatus : 0 output :

INFO: Executing <sed -i s/HOSTNAME=.*/HOSTNAME=oak2/g /etc/sysconfig/network>
SUCCESS: cmd : sed -i s/HOSTNAME=.*/HOSTNAME=oak2/g /etc/sysconfig/network
 retStatus : 0 output :

INFO: Executing <sed -i s/GATEWAYDEV=.*/GATEWAYDEV=/g /etc/sysconfig/network>
SUCCESS: cmd : sed -i s/GATEWAYDEV=.*/GATEWAYDEV=/g /etc/sysconfig/network
 retStatus : 0 output :

INFO: Executing <sed -i s/GATEWAY=.*/GATEWAY=/g /etc/sysconfig/network>
SUCCESS: cmd : sed -i s/GATEWAY=.*/GATEWAY=/g /etc/sysconfig/network
 retStatus : 0 output :

INFO: Executing <rm -f /etc/hosts>
SUCCESS: cmd : rm -f /etc/hosts
 retStatus : 0 output :

INFO: Executing <echo '127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4' >> /etc/hosts>
SUCCESS: cmd : echo '127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4' >> /etc/hosts
 retStatus : 0 output :

INFO: Executing <echo '::1 localhost localhost.localdomain localhost6 localhost6.localdomain6' >> /etc/hosts>
SUCCESS: cmd : echo '::1 localhost localhost.localdomain localhost6 localhost6.localdomain6' >> /etc/hosts
 retStatus : 0 output :

INFO: Executing <echo '127.0.0.1 oak2' >> /etc/hosts>
SUCCESS: cmd : echo '127.0.0.1 oak2' >> /etc/hosts
 retStatus : 0 output :

INFO: Executing <kill -9 `pgrep init.ohasd`>
FAILURE: cmd : kill -9 `pgrep init.ohasd`
 retStatus : 1 output :
 kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]

INFO: Executing <kill -9 `pgrep init.tfa`>
FAILURE: cmd : kill -9 `pgrep init.tfa`
 retStatus : 1 output :
 kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]

INFO: Executing </sbin/initctl stop initdcsagent>
SUCCESS: cmd : /sbin/initctl stop initdcsagent
 retStatus : 0 output :
 initdcsagent stop/waiting

INFO: Executing <rm -rf /opt/oracle/dcs/repo/node_0>
SUCCESS: cmd : rm -rf /opt/oracle/dcs/repo/node_0
 retStatus : 0 output :

INFO: Executing <rm -rf /opt/oracle/dcs/repo/node_1>
SUCCESS: cmd : rm -rf /opt/oracle/dcs/repo/node_1
 retStatus : 0 output :

INFO: Executing <rm -rf /opt/oracle/dcs/dcsagent_wallet>
SUCCESS: cmd : rm -rf /opt/oracle/dcs/dcsagent_wallet
 retStatus : 0 output :

INFO: Executing <rm -rf /opt/oracle/dcs/ft/ft.db>
SUCCESS: cmd : rm -rf /opt/oracle/dcs/ft/ft.db
 retStatus : 0 output :

INFO: Executing </opt/zookeeper/bin/zkCli.sh rmr /dcs-request-rw-locks/019-10-05 08:39:50,821 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x0>
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `/opt/zookeeper/bin/zkCli.sh rmr /dcs-request-rw-locks/019-10-05 08:39:50,821 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x0 2>&1'
FAILURE: cmd : /opt/zookeeper/bin/zkCli.sh rmr /dcs-request-rw-locks/019-10-05 08:39:50,821 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x0
 retStatus : 1 output :

INFO: Executing </opt/zookeeper/bin/zkCli.sh rmr /dcs-request-rw-locks/likely server has closed socket>
FAILURE: cmd : /opt/zookeeper/bin/zkCli.sh rmr /dcs-request-rw-locks/likely server has closed socket
 retStatus : 1 output :
 Connecting to localhost:2181
 2019-10-05 08:39:53,398 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.9-1757313, built on 08/23/2016 06:50 GMT
 2019-10-05 08:39:53,402 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=db-odand01.mtdproducts.com
 2019-10-05 08:39:53,402 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_181
 2019-10-05 08:39:53,404 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
 2019-10-05 08:39:53,405 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.8.0_181-amd64/jre
 2019-10-05 08:39:53,405 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper/bin/../lib/jline-0.9.94.jar:/opt/zookeeper/bin/../zookeeper-3.4.9.jar:/opt/zookeeper/bin/../src/java/lib/*.jar:/opt/zookeeper/bin/../conf:
 2019-10-05 08:39:53,405 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
 2019-10-05 08:39:53,405 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
 2019-10-05 08:39:53,405 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
 2019-10-05 08:39:53,405 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
 2019-10-05 08:39:53,405 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
 2019-10-05 08:39:53,405 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=4.1.12-124.18.6.el6uek.x86_64
 2019-10-05 08:39:53,405 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
 2019-10-05 08:39:53,406 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
 2019-10-05 08:39:53,406 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/opt/oracle/oak/onecmd
 2019-10-05 08:39:53,407 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589e
 2019-10-05 08:39:53,429 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
 2019-10-05 08:39:53,493 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@876] - Socket connection established to localhost/127.0.0.1:2181, initiating session
 2019-10-05 08:39:53,496 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
 Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /dcs-request-rw-locks/likely
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1560)
        at org.apache.zookeeper.ZKUtil.listSubTreeBFS(ZKUtil.java:114)
        at org.apache.zookeeper.ZKUtil.deleteRecursive(ZKUtil.java:49)
        at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:714)
        at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:599)
        at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:362)
        at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:290)

INFO: Executing </opt/zookeeper/bin/zkCli.sh rmr /dcs-request-rw-locks/closing socket connection and attempting reconnec>
FAILURE: cmd : /opt/zookeeper/bin/zkCli.sh rmr /dcs-request-rw-locks/closing socket connection and attempting reconnec
 retStatus : 1 output :
 Connecting to localhost:2181
 2019-10-05 08:39:53,837 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.9-1757313, built on 08/23/2016 06:50 GMT
 2019-10-05 08:39:53,841 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=db-odand01.mtdproducts.com
 2019-10-05 08:39:53,841 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_181
 2019-10-05 08:39:53,844 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
 2019-10-05 08:39:53,844 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.8.0_181-amd64/jre
 2019-10-05 08:39:53,844 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper/bin/../lib/jline-0.9.94.jar:/opt/zookeeper/bin/../zookeeper-3.4.9.jar:/opt/zookeeper/bin/../src/java/lib/*.jar:/opt/zookeeper/bin/../conf:
 2019-10-05 08:39:53,844 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
 2019-10-05 08:39:53,844 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
 2019-10-05 08:39:53,844 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
 2019-10-05 08:39:53,844 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
 2019-10-05 08:39:53,844 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
 2019-10-05 08:39:53,845 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=4.1.12-124.18.6.el6uek.x86_64
 2019-10-05 08:39:53,845 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
 2019-10-05 08:39:53,845 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
 2019-10-05 08:39:53,845 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/opt/oracle/oak/onecmd
 2019-10-05 08:39:53,846 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589e
 2019-10-05 08:39:53,869 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
 2019-10-05 08:39:53,931 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@876] - Socket connection established to localhost/127.0.0.1:2181, initiating session
 2019-10-05 08:39:53,934 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
 Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /dcs-request-rw-locks/closing
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1560)
        at org.apache.zookeeper.ZKUtil.listSubTreeBFS(ZKUtil.java:114)
        at org.apache.zookeeper.ZKUtil.deleteRecursive(ZKUtil.java:49)
        at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:714)
        at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:599)
        at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:362)
        at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:290)

INFO: Executing </opt/zookeeper/bin/zkCli.sh rmr /ssh-auth-keys/019-10-05 08:39:51,255 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x0>
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `/opt/zookeeper/bin/zkCli.sh rmr /ssh-auth-keys/019-10-05 08:39:51,255 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x0 2>&1'
FAILURE: cmd : /opt/zookeeper/bin/zkCli.sh rmr /ssh-auth-keys/019-10-05 08:39:51,255 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x0
 retStatus : 1 output :

INFO: Executing </opt/zookeeper/bin/zkCli.sh rmr /ssh-auth-keys/likely server has closed socket>
FAILURE: cmd : /opt/zookeeper/bin/zkCli.sh rmr /ssh-auth-keys/likely server has closed socket
 retStatus : 1 output :
 Connecting to localhost:2181
 2019-10-05 08:39:54,276 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.9-1757313, built on 08/23/2016 06:50 GMT
 2019-10-05 08:39:54,280 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=canvlp-jb7dom1.mtdproducts.com
 2019-10-05 08:39:54,281 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_181
 2019-10-05 08:39:54,283 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
 2019-10-05 08:39:54,283 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.8.0_181-amd64/jre
 2019-10-05 08:39:54,283 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper/bin/../lib/jline-0.9.94.jar:/opt/zookeeper/bin/../zookeeper-3.4.9.jar:/opt/zookeeper/bin/../src/java/lib/*.jar:/opt/zookeeper/bin/../conf:
 2019-10-05 08:39:54,283 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
 2019-10-05 08:39:54,284 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
 2019-10-05 08:39:54,284 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
 2019-10-05 08:39:54,284 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
 2019-10-05 08:39:54,284 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
 2019-10-05 08:39:54,284 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=4.1.12-124.18.6.el6uek.x86_64
 2019-10-05 08:39:54,284 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
 2019-10-05 08:39:54,284 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
 2019-10-05 08:39:54,284 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/opt/oracle/oak/onecmd
 2019-10-05 08:39:54,285 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589e
 2019-10-05 08:39:54,308 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
 2019-10-05 08:39:54,370 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@876] - Socket connection established to localhost/127.0.0.1:2181, initiating session
 2019-10-05 08:39:54,373 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
 Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ssh-auth-keys/likely
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1560)
        at org.apache.zookeeper.ZKUtil.listSubTreeBFS(ZKUtil.java:114)
        at org.apache.zookeeper.ZKUtil.deleteRecursive(ZKUtil.java:49)
        at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:714)
        at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:599)
        at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:362)
        at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:290)

INFO: Executing </opt/zookeeper/bin/zkCli.sh rmr /ssh-auth-keys/closing socket connection and attempting reconnec>
FAILURE: cmd : /opt/zookeeper/bin/zkCli.sh rmr /ssh-auth-keys/closing socket connection and attempting reconnec
 retStatus : 1 output :
 Connecting to localhost:2181
 2019-10-05 08:39:54,708 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.9-1757313, built on 08/23/2016 06:50 GMT
 2019-10-05 08:39:54,711 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=db-odand01.mtdproducts.com
 2019-10-05 08:39:54,712 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_181
 2019-10-05 08:39:54,714 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
 2019-10-05 08:39:54,714 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.8.0_181-amd64/jre
 2019-10-05 08:39:54,714 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper/bin/../lib/jline-0.9.94.jar:/opt/zookeeper/bin/../zookeeper-3.4.9.jar:/opt/zookeeper/bin/../src/java/lib/*.jar:/opt/zookeeper/bin/../conf:
 2019-10-05 08:39:54,714 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
 2019-10-05 08:39:54,714 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
 2019-10-05 08:39:54,715 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
 2019-10-05 08:39:54,715 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
 2019-10-05 08:39:54,715 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
 2019-10-05 08:39:54,715 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=4.1.12-124.18.6.el6uek.x86_64
 2019-10-05 08:39:54,715 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
 2019-10-05 08:39:54,715 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
 2019-10-05 08:39:54,715 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/opt/oracle/oak/onecmd
 2019-10-05 08:39:54,716 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589e
 2019-10-05 08:39:54,739 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
 2019-10-05 08:39:54,801 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@876] - Socket connection established to localhost/127.0.0.1:2181, initiating session
 2019-10-05 08:39:54,804 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
 Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ssh-auth-keys/closing
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1560)
        at org.apache.zookeeper.ZKUtil.listSubTreeBFS(ZKUtil.java:114)
        at org.apache.zookeeper.ZKUtil.deleteRecursive(ZKUtil.java:49)
        at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:714)
        at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:599)
        at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:362)
        at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:290)

INFO: Executing </opt/zookeeper/bin/zkServer.sh stop>
SUCCESS: cmd : /opt/zookeeper/bin/zkServer.sh stop
 retStatus : 0 output :
 ZooKeeper JMX enabled by default
 Using config: /opt/zookeeper/bin/../conf/zoo.cfg
 Stopping zookeeper ... STOPPED

INFO: Executing <rm -rf /opt/zookeeper/data/version-2/>
SUCCESS: cmd : rm -rf /opt/zookeeper/data/version-2/
 retStatus : 0 output :

INFO: Executing <rm -rf /opt/zookeeper/data/zookeeper_server.pid>
SUCCESS: cmd : rm -rf /opt/zookeeper/data/zookeeper_server.pid
 retStatus : 0 output :

INFO: Executing <echo 'blacklist {' > /etc/multipath.conf
>
SUCCESS: cmd : echo 'blacklist {' > /etc/multipath.conf

 retStatus : 0 output :

INFO: Executing <echo 'devnode "*"' >> /etc/multipath.conf
>
SUCCESS: cmd : echo 'devnode "*"' >> /etc/multipath.conf

 retStatus : 0 output :

INFO: Executing <echo '}' >> /etc/multipath.conf
>
SUCCESS: cmd : echo '}' >> /etc/multipath.conf

 retStatus : 0 output :

INFO: Executing <multipath -r
>
SUCCESS: cmd : multipath -r

 retStatus : 0 output :

INFO: Executing <multipath -F
>
SUCCESS: cmd : multipath -F

 retStatus : 0 output :

INFO: Executing <rm -rf /etc/udev/rules.d/70-persistent-nvme.rules>
SUCCESS: cmd : rm -rf /etc/udev/rules.d/70-persistent-nvme.rules
 retStatus : 0 output :

INFO: Executing <rm -rf /etc/udev/rules.d/55-usm.rules>
SUCCESS: cmd : rm -rf /etc/udev/rules.d/55-usm.rules
 retStatus : 0 output :

INFO: Executing <start_udev>

ODA Bundle Patch: Upgrade to OAK 18.3.0.0.0

Posted on June 6, 2019 by Oracle Database 6

ODA Bundle Patch: Upgrade to OAK 18.3.0.0.0

Here i am upgrading ODA from 12.1.2.12.0 to 18.3.0.0.0.
Backup ODA_BASE before proceeding with ODA Bundle patch.
For ODA_BASE backup refer to post “Backup ODA_BASE: Database Appliance ODA X5“.
Above post applies to ODA X5 or higher versions.

Login to ODA_BASE as root user.
Download & Unpack the ODA bundle patch on both Nodes.

Ensure you have sufficient space under /opt.
Usually the patches are unpacked to /opt/oak/oracle/pkgrepos folder.

# oakcli unpack -package /tmp/p28864520_183000_Linux-x86-64_1of3.zip
# oakcli unpack -package /tmp/p28864520_183000_Linux-x86-64_2of3.zip
# oakcli unpack -package /tmp/p28864520_183000_Linux-x86-64_3of3.zip

Verify the patch before apply. On Both ODA nodes.

On Node 1:

[root@oda1 ~]# oakcli update -patch 18.3.0.0 --verify
INFO: 2019-05-16 20:10:54: Reading the metadata file now...
                Component Name            Installed Version         Proposed Patch Version
                ---------------           ------------------        -----------------
                Controller_INT            4.650.00-7176             Up-to-date
                Controller_EXT            13.00.00.00               Up-to-date
                Expander                  0018                      Up-to-date
                SSD_SHARED {
                [ c1d20,c1d21,c1d22,      A29A                      Up-to-date
                c1d23 ]
                [ c1d16,c1d17,c1d18,      A29A                      Up-to-date
                c1d19 ]
                             }
                HDD_LOCAL                 A7E0                      Up-to-date
                HDD_SHARED                PAG1                      Up-to-date
                ILOM                      3.2.9.23 r116695          4.0.2.26.b r125868
                BIOS                      30110000                  30130500
                IPMI                      1.8.12.4                  Up-to-date
                HMP                       2.3.5.2.8                 2.4.1.0.14
                OAK                       12.1.2.12.0               18.3.0.0.0
                OL                        6.8                       6.10
                OVM                       3.4.3                     3.4.4
                GI_HOME                   12.1.0.2.170814           18.3.0.0.180717
                DB_HOME                   12.1.0.2.5                12.1.0.2.180717
                ASR                       5.5.1                     18.3.1

On Node 2:

[root@oda2 ~]# oakcli update -patch 18.3.0.0 --verify
INFO: 2019-05-16 20:27:49: Reading the metadata file now...
                Component Name            Installed Version         Proposed Patch Version
                ---------------           ------------------        -----------------
                Controller_INT            4.650.00-7176             Up-to-date
                Controller_EXT            13.00.00.00               Up-to-date
                Expander                  0018                      Up-to-date
                SSD_SHARED {
                [ c1d20,c1d21,c1d22,      A29A                      Up-to-date
                c1d23 ]
                [ c1d16,c1d17,c1d18,      A29A                      Up-to-date
                c1d19 ]
                             }
                HDD_LOCAL                 A7E0                      Up-to-date
                HDD_SHARED                PAG1                      Up-to-date
                ILOM                      3.2.9.23 r116695          4.0.2.26.b r125868
                BIOS                      30110000                  30130500
                IPMI                      1.8.12.4                  Up-to-date
                HMP                       2.3.5.2.8                 2.4.1.0.14
                OAK                       12.1.2.12.0               18.3.0.0.0
                OL                        6.8                       6.10
                OVM                       3.4.3                     3.4.4
                GI_HOME                   12.1.0.2.170814           18.3.0.0.180717
                DB_HOME                   12.1.0.2.5                12.1.0.2.180717

The components marked “Up-to-date” does not needs an upgrade.
Components that require upgrade is taken care automatically by the “oakcli update” command.

ODA Bundle patch can be applied in three phases in following order.

# /opt/oracle/oak/bin/oakcli update -patch 18.3.0.0.0 --server
# /opt/oracle/oak/bin/oakcli update -patch 18.3.0.0.0 --storage
# /opt/oracle/oak/bin/oakcli update -patch 18.3.0.0.0 --database
ODA_BASE Bundle patch: Server.

Patching Server component may fail if root mount has less than 12gb of space.
Make sure you free up root mount point to have more than 12gb.

[root@oda1 ~]# /opt/oracle/oak/bin/oakcli update -patch 18.3.0.0.0 --server

****************************************************************************
*****                                                                  *****
*****   For all X5-2 customers with 8TB disks, please make sure to     *****
*****   run storage patch ASAP to update the disk firmware to "PAG1".  *****
*****                                                                  *****
****************************************************************************

INFO: DB, ASM, Clusterware may be stopped during the patch if required
INFO: Both Nodes may get rebooted automatically during the patch if required
Do you want to continue: [Y/N]?: Y
INFO: User has confirmed for the reboot
INFO: Patch bundle must be unpacked on the second Node also before applying the patch
Did you unpack the patch bundle on the second Node? : [Y/N]? : Y
INFO: All the VMs except the ODABASE will be shutdown forcefully if needed
Do you want to continue : [Y/N]? : Y
INFO: Running pre-install scripts
INFO: Running  prepatching on node 0
INFO: Running  prepatching on node 1

For Further output refer below screenshot.

oakcli update -patch 18.3.0.0.0 –server (Image 1)
oakcli update -patch 18.3.0.0.0 –server (Image 2)
oakcli update -patch 18.3.0.0.0 –server (Image 3)
ODA_BASE Bundle patch: Storage

Now patching Storage….


[root@oda1 ~]# /opt/oracle/oak/bin/oakcli update -patch 18.3.0.0.0 --storage
INFO: DB, ASM, Clusterware may be stopped during the patch if required
INFO: Both Nodes may get rebooted automatically during the patch if required
Do you want to continue: [Y/N]?: Y
INFO: User has confirmed for the reboot
INFO: Running pre-install scripts
INFO: Running  prepatching on node 0
INFO: Running  prepatching on node 1
INFO: Completed pre-install scripts
INFO: All Shared Storage components are running with latest firmware
INFO: No patching required. Exiting...
You have new mail in /var/spool/mail/root
oakcli update -patch 18.3.0.0.0 –storage
ODA_BASE Bundle patch: Database

Now Patching database…


[root@oda1 ~]# /opt/oracle/oak/bin/oakcli update -patch 18.3.0.0.0 --database  
INFO: Running pre-install scripts
INFO: Running  prepatching on node 0
INFO: Running  prepatching on node 1
INFO: Completed pre-install scripts
...
...
INFO: 2019-05-17 03:29:42: ------------------Patching DB-------------------------
INFO: 2019-05-17 03:29:43: Getting all the possible Database Homes for patching
...
INFO: 2019-05-17 03:29:51: Patching 12.1.0.2 Database Homes on the Node oda1

Found the following 12.1.0.2 homes possible for patching:

HOME_NAME                      HOME_LOCATION                                          
---------                      -------------                                          
OraDb12102_home1               /u01/app/oracle/product/12.1.0.2/dbhome_1              

[Please note that few of the above Database Homes may be already up-to-date. They will be automatically ignored]

Would you like to patch all the above homes: Y | N ? : Y
INFO: 2019-05-17 03:30:01: Updating OPATCH
Verifying Opatch version for home:.
Expecting version:<12.2.0.1.14>
Opatch version on node  is <12.2.0.1.14>
Opatch version on node  is <12.2.0.1.14>
INFO: 2019-05-17 03:33:15: Performing the conflict checks...
SUCCESS: 2019-05-17 03:33:27: Conflict checks passed for all the Homes
INFO: 2019-05-17 03:33:27: Checking if the patch is already applied on any of the Homes
INFO: 2019-05-17 03:33:31: Home is not Up-to-date
SUCCESS: 2019-05-17 03:33:33: Successfully stopped the Database consoles
SUCCESS: 2019-05-17 03:33:38: Successfully stopped the EM agents
INFO: 2019-05-17 03:33:43: Applying patch on /u01/app/oracle/product/12.1.0.2/dbhome_1 Homes
INFO: 2019-05-17 03:33:43: It may take upto 15 mins. Please wait...
SUCCESS: 2019-05-17 03:44:45: Successfully applied the patch on the Home : /u01/app/oracle/product/12.1.0.2/dbhome_1
SUCCESS: 2019-05-17 03:44:45: Successfully started the Database consoles
SUCCESS: 2019-05-17 03:44:45: Successfully started the EM Agents
INFO: 2019-05-17 03:44:50: Patching 12.1.0.2 Database Homes on the Node oda2

INFO: DB patching summary on node: oda1
SUCCESS: 2019-05-17 03:57:09:  Successfully applied the patch on the Home /u01/app/oracle/product/12.1.0.2/dbhome_1

INFO: DB patching summary on node: oda2
SUCCESS: 2019-05-17 03:57:09:  Successfully applied the patch on the Home /u01/app/oracle/product/12.1.0.2/dbhome_1

INFO: Executing /tmp/pending_actions on both nodes
oakcli update -patch 18.3.0.0.0 –database (Image 1)
oakcli update -patch 18.3.0.0.0 –database (Image 2)

Once patch is completed you can run –verify again to see components patched.
Now you should see all components are “Up-to-date”.

On Node 1:

[root@oda1 ~]# oakcli update -patch 18.3.0.0 --verify
INFO: 2019-05-17 04:06:43: Reading the metadata file now...
                Component Name            Installed Version         Proposed Patch Version
                ---------------           ------------------        -----------------
                Controller_INT            4.650.00-7176             Up-to-date
                Controller_EXT            13.00.00.00               Up-to-date
                Expander                  0018                      Up-to-date
                SSD_SHARED {
                [ c1d20,c1d21,c1d22,      A29A                      Up-to-date
                c1d23 ]
                [ c1d16,c1d17,c1d18,      A29A                      Up-to-date
                c1d19 ]
                             }
                HDD_LOCAL                 A7E0                      Up-to-date
                HDD_SHARED                PAG1                      Up-to-date
                ILOM                      4.0.2.26.b r125868        Up-to-date
                BIOS                      30130500                  Up-to-date
                IPMI                      1.8.12.4                  Up-to-date
                HMP                       2.4.1.0.14                Up-to-date
                OAK                       18.3.0.0.0                Up-to-date
                OL                        6.10                      Up-to-date
                OVM                       3.4.4                     Up-to-date
                GI_HOME                   18.3.0.0.180717           Up-to-date
                DB_HOME                   12.1.0.2.180717           Up-to-date
                ASR                       18.3.1                    Up-to-date
Node 1, Post ODA Bundle patch

On Node 2:

[root@oda2 ~]# oakcli update -patch 18.3.0.0 --verify
INFO: 2019-05-17 04:06:37: Reading the metadata file now...
                Component Name            Installed Version         Proposed Patch Version
                ---------------           ------------------        -----------------
                Controller_INT            4.650.00-7176             Up-to-date
                Controller_EXT            13.00.00.00               Up-to-date
                Expander                  0018                      Up-to-date
                SSD_SHARED {
                [ c1d20,c1d21,c1d22,      A29A                      Up-to-date
                c1d23 ]
                [ c1d16,c1d17,c1d18,      A29A                      Up-to-date
                c1d19 ]
                             }
                HDD_LOCAL                 A7E0                      Up-to-date
                HDD_SHARED                PAG1                      Up-to-date
                ILOM                      4.0.2.26.b r125868        Up-to-date
                BIOS                      30130500                  Up-to-date
                IPMI                      1.8.12.4                  Up-to-date
                HMP                       2.4.1.0.14                Up-to-date
                OAK                       18.3.0.0.0                Up-to-date
                OL                        6.10                      Up-to-date
                OVM                       3.4.4                     Up-to-date
                GI_HOME                   18.3.0.0.180717           Up-to-date
                DB_HOME                   12.1.0.2.180717           Up-to-date
Node 2, Post ODA Bundle patch

Oracle documentation is the best place to find upgrade path.

e.g. You can upgrade from 12.1.2.6- 12.1.2.12 -> 12.2.1.4 -> 18.3.0.0.0 -> 18.5.0.0.

You cannot upgrade from 12.1.2.6 to 18.3.0.0.0.

Check Oracle ODA Documentation here for more details.

Happy Patching. Thanks for visiting my post. Please feel free to comment.

Excuse typos.

Backup ODA_BASE: Database Appliance ODA X5

Posted on June 6, 2019 by Oracle Database Oracle Database Appliance 1

There are situations we may need to backup ODA_BASE, especially before patching ODA_BASE.
This article applies only to virtual deployment of ODA.
In this Article I’ve used ODA-X5-2 appliance to backup.
The procedure is applicable for ODA X5 or higher versions.
ODA_BASE is backed up from the DOM machines, i.e. the parent machine from where ODA_BASE was created.
Login to the DOM machine of the ODA Appliance.

NODE 1:
— ODA_BASE – DOM Server, STOP ODA_BASE —

[root@oda1-dom0 ~]# oakcli show oda_base
ODA base domain
ODA base CPU cores      :16
ODA base domain memory  :244
ODA base template       :/OVS/oda_base_12.1.2.5.gz
ODA base vlans          :['net1', 'net2', 'vbr1']
ODA base current status :Running

[root@oda1-dom0 ~]#  oakcli stop oda_base
INFO: Stopping ODA base domain... It may take several minutes.
INFO: Stopping all the shared repos

[root@oda1-dom0 ~]# oakcli show oda_base
ODA base domain
ODA base CPU cores      :16
ODA base domain memory  :244
ODA base template       :/OVS/oda_base_12.1.2.5.gz
ODA base vlans          :['net1', 'net2', 'vbr1']
ODA base current status :Stopped

Continue reading…

ODA X5 CPU configuration

Posted on October 28, 2018 by Oracle Database Appliance 0

Oracle Database Appliance X5-2 nodes
each contain 2 CPUs and
each CPU has 18 cores.
When you deploy Oracle Database Appliance X5-2, all 72 cores (36 cores in each server) are active with hyper-threading enabled by default.

Below show processor showing each node having two processors and each with 18 cores i.e.

Node 0: 18 Cores x 2 Proc = 36 Cores x 2 (for HT) = 72 vCPU/Threads
Node 1: 18 Cores x 2 Proc = 36 Cores x 2 (for HT) = 72 vCPU/Threads

[root@samodanodep01 ~]# oakcli show processor
  NAME  HEALTH HEALTH_DETAILS PART_NO. LOCATION   MODEL   MAX_CLK_SPEED TOTAL_CORES ENABLED_CORES
  CPU_0 OK     -              060F     P0 (CPU 0) Intel(R) Xeon(R) CPU E5-2699  2.300 GHz       18          18
  CPU_1 OK     -              060F     P1 (CPU 1) Intel(R) Xeon(R) CPU E5-2699  2.300 GHz       18          18

[root@samodanodp01 ~]# oakcli show cpupool -node 0
                     Pool                           Cpu List
    default-unpinned-pool     [0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
                               10, 11, 12, 13, 14, 15, 16, 1
                              7, 18, 19, 20, 21, 22, 23, 24,
                               25, 26, 27, 28, 29, 30, 31, 3
                              2, 33, 34, 35, 36, 37, 38, 39,
                               40, 41, 42, 43, 44, 45, 46, 4
                              7, 48, 49, 50, 51, 52, 53, 54,
                               55, 56, 57, 58, 59, 60, 61, 6
                                      2, 63, 64, 65, 66, 67]
           odaBaseCpuPool                   [68, 69, 70, 71]

[root@samodanode02 ~]# oakcli show processor
  NAME  HEALTH HEALTH_DETAILS PART_NO. LOCATION   MODEL    MAX_CLK_SPEED TOTAL_CORES ENABLED_CORES
  CPU_0 OK     -              060F     P0 (CPU 0) Intel(R) Xeon(R) CPU E5-2699  2.300 GHz       18          18
  CPU_1 OK     -              060F     P1 (CPU 1) Intel(R) Xeon(R) CPU E5-2699  2.300 GHz       18          18

[root@samodanode02 ~]# oakcli show cpupool -node 1
                     Pool                           Cpu List
    default-unpinned-pool     [0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
                               10, 11, 12, 13, 14, 15, 16, 1
                              7, 18, 19, 20, 21, 22, 23, 24,
                               25, 26, 27, 28, 29, 30, 31, 3
                              2, 33, 34, 35, 36, 37, 38, 39,
                               40, 41, 42, 43, 44, 45, 46, 4
                              7, 48, 49, 50, 51, 52, 53, 54,
                               55, 56, 57, 58, 59, 60, 61, 6
                                      2, 63, 64, 65, 66, 67]
           odaBaseCpuPool                   [68, 69, 70, 71]

CRS-0804, PROC-44, CRS-4535, PROC-44: Error in network address

Posted on February 22, 2018 by Real Application Clusters 0

Getting following error during CRS start. we had a server reboot post maintenance.
CRS-0804, PROC-44, CRS-4535, PROC-44: Error in network address

CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager

[crsd(12451930)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-44: Error in network address and interface operations Network address
and interface operations error [7]]. Details at (:CRSD00111:) in /grid/app/11.2.0.3/grid/log/sam-rac01/crsd/crsd.log.
2017-03-20 13:46:39.533:
[CLWAL][1]clsw_Initialize: OLR initlevel [70000]
2017-03-20 13:46:42.946: [ OCRAPI][1]clsu_get_private_ip_addresses: no ip addresses found.
[ OCRAPI][1]a_init_clsss: failed to call clsu_get_private_ip_addr (7)
2017-03-20 13:46:43.149: [ OCRAPI][1]a_init:13!: Clusterware init unsuccessful : [44]
2017-03-20 13:46:43.149: [ CRSOCR][1] OCR context init failure. Error: PROC-44: Error in network address and interface operations Network address and interface operat
ions error [7]
2017-03-20 13:46:43.150: [ CRSMAIN][1] Created alert : (:CRSD00111:) : Could not init OCR, error: PROC-44: Error in network address and interface operations Network ad
dress and interface operations error [7]
2017-03-20 13:46:43.150: [ CRSD][1][PANIC] CRSD exiting: Could not init OCR, code: 44
2017-03-20 13:46:43.150: [ CRSD][1] Done.</pre>

Looking at the logs above this seems to be issue with network.
Verify the same by comparing output of following two commands.

$GRID_HOME/bin/oifcfg iflist -p -n
$GRID_HOME/bin/gpnptool get

On output 1 the interface name for PRIVATE network is bond0 where as from
output 2 the interface name for cluster_interconnect in bond1.
Output 1:

grid@sam-rac01::/u01/app/grid/product/11.2.0/grid_2/bin: oifcfg iflist -p -n
bond0 10.55.9.0 PRIVATE 255.255.255.0
bond0 172.16.250.0 PRIVATE 255.255.255.0
bond0 169.254.156.0 UNKNOWN 255.255.255.0

Output 2:

grid@sam-rac01::/u01/app/grid/product/11.2.0/grid_2/bin: gpnptool get
Warning: some command line parameters were defaulted. Resulting command line:
./gpnptool.bin get -o-

<?xml version="1.0" encoding="UTF-8"?>
<gpnp:GPnP-Profile Version="1.0" xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.xsd" ProfileSequence="6" ClusterUId="53cd88e0368eeffdff7b1aa8bfeb99a3" ClusterName="its-racd" PALocation="">
<gpnp:Network-Profile>
<gpnp:HostNetwork id="gen" HostName="*">
<gpnp:Network id="net1" IP="10.55.9.0" Adapter="bond0" Use="public"/>
<gpnp:Network id="net2" IP="172.16.250.0" Adapter="bond1" Use="cluster_interconnect"/>
</gpnp:HostNetwork>
</gpnp:Network-Profile>
<orcl:CSS-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/>
<orcl:ASM-Profile id="asm" DiscoveryString="" SPFile="+DATA/its-racd/asmparameterfile/registry.253.743595285"/>
<ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
<ds:SignedInfo>
<ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
<ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
<ds:Reference URI="">
<ds:Transforms>
<ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
<ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">
<InclusiveNamespaces xmlns="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="gpnp orcl xsi"/>
</ds:Transform></ds:Transforms><ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
<ds:DigestValue>8HwbUAQu9nZIzAFBq43nRqpaNeM=</ds:DigestValue>
</ds:Reference>
</ds:SignedInfo>
<ds:SignatureValue>RjumSmZayRFBKzZTaD85jBNFCw9/CHJCXI0WRsAMv7UU4BwIRDUDL/A4YBpqWdhD99ka4JDB4hbLlAseG0SfNg/lnSh5cuVL9dWOpcmUrmRH6kZohVy6xeNvdwQ3hPPto9p2A3PlfE+iW5cl7WfBYCUwCXlVCXPOB3m6G5hpbI8=</ds:SignatureValue>
</ds:Signature>
</gpnp:GPnP-Profile>
Success.

Above XML output is formatted for better readanbility.

As per MOS note:

There are 2 possible solutions:
A. Restore the old network configuration and execute the necessary steps in the correct order.
Check with your network administrator if the old network configuration can be restored, this should allow the clusterware to start.
Once the clusterware is running again, follow the instructions in
Note 283684.1 “How to Modify Private Network Information in Oracle Clusterware” to make the subnet change.

B. De-configure and re-configure the clusterware
If for some reason it is not possible to restore the old network configuration, a rebuild of the OCR is necessary – as documented in
Note 1377349.1 “How to Deconfigure/Reconfigure(Rebuild OCR) or De-install Grid Infrastructure”

 

Excuse Typos. Thanks for reaching my blog.

Oracle Database 18c, Database of Year 2018

Posted on February 22, 2018 by Oracle Database 18c 0

Oracle database 18c is now available…it’s about a week since announcement made on 16 Feb.
So we download a copy and try it …

Well wait a sec..its released only on Oracle Cloud & Engineered Systems.

So what does 18c stands for, the ‘c’ of course stands for cloud, but why 18c not 13c. Why there are 6 versions skipped.
From sources like blog.oracle.com, mike dietrich’s blog it clearly states that 18c is a subsequent release of 12.2.0.2 (12c Rel 2).

Oracle has only changed the naming convention to reflect year when product released. So thus indicates now oracle will have yearly releases with quarterly updates.

oracle-database-18c-release-schedule

So we can expect version 19c or 19x in 2019. will see how that goes.

Well its nice to see that many MOS ids/docs has also been appended to reflect 18c release. for e.g. in the upgrade compatibility matrix and many more docs where you can see 18c is officially available.

upgrade-to-oracle-database-18c

You can get more details on Database 18c and updates on MOS Document 2285040.1
For more information on Oracle 18c on Different platforms you can refer Document 742060.1.

Wanna get started with Oracle 18c, so here’s how you can start.
Step 1: Refer Oracle database 18c Documentation .
https://docs.oracle.com/en/database/oracle/oracle-database/18/newft/new-features.html

Step 2: Try out Oracle 18c with LiveSQL.
https://livesql.oracle.com/apex/livesql/file/index.html

 

Excuse typos.

Single Instance to RACOne Node Conversion.

Posted on May 18, 2017 by Oracle Database Oracle Database Appliance Real Application Clusters 0

Convert single instance to RAC using rconfig utility.
Verify the database is added to cluster and is SINGLE Instance.

[oracle@SAMOELNODE01 ~]$ srvctl config database -d ORCL
Database unique name: ORCL
Database name:
Oracle home: /u01/app/oracle/product/11.2.0.4/dbhome_2
Oracle user: oracle
Spfile:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: ORCL
Database instance: ORCL
Disk Groups:
Mount point paths:
Services:
Type: SINGLE
Database is administrator managed

Above output shows Type: SINGLE.

[oracle@SAMOELNODE01 scripts]$ cd /u01/app/oracle/product/11.2.0.4/dbhome_2/assistants/rconfig/sampleXMLs/
[oracle@SAMOELNODE01 sampleXMLs]$ ls -ltr
total 12
-rw-r--r-- 1 oracle oinstall 2804 Mar 22 2010 ConvertToRAC_PolicyManaged.xml
-rw-r--r-- 1 oracle oinstall 2754 Apr 8 14:24 ConvertToRAC_AdminManaged.xml.orig
-rw-r--r-- 1 oracle oinstall 2781 Apr 8 15:00 ConvertToRAC_AdminManaged.xml
[oracle@SAMOELNODE01 sampleXMLs]$ cp ConvertToRAC_AdminManaged.xml ConvertToRACOneNode.xml
[oracle@SAMOELNODE01 sampleXMLs]$ diff ConvertToRAC_AdminManaged.xml ConvertToRACOneNode.xml
7c7
< <n:Convert verify="YES">
---
> <n:Convert verify="ONLY">
9c9
< <n:SourceDBHome>/oracle/product/11.2.0/db_1</n:SourceDBHome>
---
> <n:SourceDBHome>/u01/app/oracle/product/11.2.0.4/dbhome_2</n:SourceDBHome>
11c11
< <n:TargetDBHome>/oracle/product/11.2.0/db_1</n:TargetDBHome>
---
> <n:TargetDBHome>/u01/app/oracle/product/11.2.0.4/dbhome_2</n:TargetDBHome>
13c13
< <n:SourceDBInfo SID="sales">
---
> <n:SourceDBInfo SID="ORCL">
16c16
< <n:Password>oracle</n:Password>
---
> <n:Password>system123</n:Password>
22,23c22,23
< <n:Node name="node1"/>
< <n:Node name="node2"/>
---
> <n:Node name="SAMOELNODE01"/>
> <n:Node name="SAMOELNODE02"/>
26c26
< <!--n:RacOneNode servicename="saleORCLac1service"/-->
---
> <n:RacOneNode servicename="ORCL"/>
28c28
< <n:InstancePrefix>sales</n:InstancePrefix>
---
> <n:InstancePrefix>ORCL</n:InstancePrefix>
33c33
< <n:TargetDatabaseArea>+ASMDG</n:TargetDatabaseArea>
---
> <n:TargetDatabaseArea></n:TargetDatabaseArea>
35c35
< <n:TargetFlashRecoveryArea>+ASMDG</n:TargetFlashRecoveryArea>
---
> <n:TargetFlashRecoveryArea></n:TargetFlashRecoveryArea>

So here I’ve changed only following values
i.e. SourceDBHome, TargetDBHome, SourceDBInfo, Password, Node name, RacOneNode servicename, InstancePrefix
and following values are altered to blank (><) as we are not migrating storage area i.e. TargetDatabaseArea & TargetFlashRecoveryArea.

“Convert verify”: Value of “ONLY” denotes to run rconfig in Test mode, “YES” denotes Live mode conversion.

Now execute rconfig in Test mode only (<n:Convert verify=”ONLY”>). Post success, we can execute with <n:Convert verify=”YES”>.

[oracle@SAMOELNODE01 bin]$ ./rconfig /u01/app/oracle/product/11.2.0.4/dbhome_2/assistants/rconfig/sampleXMLs/ConvertToRACOneNode.xml
<?xml version="1.0" ?>
<RConfig version="1.1" >
<ConvertToRAC>
<Convert>
<Response>
<Result code="0" >
Operation Succeeded
</Result>
</Response>
<ReturnValue type="object">
There is no return value for this step </ReturnValue>
</Convert>
</ConvertToRAC></RConfig>

Here “Operation Succeded” mentions its success in test mode.
Now execute rconfig in Live mode only (change <n:Convert verify=”YES”>). And execute below.

[oracle@SAMOELNODE01 bin]$ ./rconfig /u01/app/oracle/product/11.2.0.4/dbhome_2/assistants/rconfig/sampleXMLs/ConvertToRACOneNode.xml
Converting Database "ORCL" to Cluster Database.
Target Oracle Home: /u01/app/oracle/product/11.2.0.4/dbhome_2. Database Role: PRIMARY.
Setting Data Files and Control Files
Adding Database Instances
Adding Redo Logs
Enabling threads for all Database Instances
Setting TEMP tablespace
Adding UNDO tablespaces
Adding Trace files
Setting Fast Recovery Area
Updating Oratab
Creating Password file(s)
Configuring Listeners
Configuring related CRS resources
Starting Cluster Database
<?xml version="1.0" ?>
<RConfig version="1.1" >
<ConvertToRAC>
<Convert>
<Response>
<Result code="0" >
Operation Succeeded
</Result>
</Response>
<ReturnValue type="object">
<Oracle_Home>
/u01/app/oracle/product/11.2.0.4/dbhome_2
</Oracle_Home>
<Database type="ADMIN_MANAGED" >
<InstanceList>
<Instance SID="ORCL_1" Node="SAMOELNODE01" >
</Instance>
</InstanceList>
</Database> </ReturnValue>
</Convert>
</ConvertToRAC></RConfig>

Now again we see “Operation Succeded” that says its success.
Let’s verify the same.

[oracle@SAMOELNODE01 bin]$ srvctl config database -d ORCL
Database unique name: ORCL
Database name: ORCL
Oracle home: /u01/app/oracle/product/11.2.0.4/dbhome_2
Oracle user: oracle
Spfile: /u02/app/oracle/oradata/datastore/.ACFS/snaps/ORCL/spfileORCL.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: ORCL
Database instances:
Disk Groups:
Mount point paths:
Services: saleORClac1service
Type: RACOneNode
Online relocation timeout: 30
Instance name prefix: ORCL
Candidate servers: SAMOELNODE01,SAMOELNODE02
Database is administrator managed

For troubleshooting refer logs at $ORACLE_HOME/cfgtoollogs/rconfig.
e.g. /u01/app/oracle/product/11.2.0.4/dbhome_2/cfgtoollogs/rconfig

xml_shortcode

add_shortcode( 'xml', `xml_shortcode` );