ORA-15032: not all alterations performed ORA-15260: permission denied on ASM disk group

The following error would occur if you are connect to ASM instance as  sysdba instead of sysasm user.

[root@stdby02 bin]# su grid
[grid@stdby02 bin]$ . oraenv
ORACLE_SID = [grid] ? +ASM2
The Oracle base has been set to /grid/app/grid
[grid@stdby02 bin]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Tue Jun 27 10:41:14 2017

Copyright (c) 1982, 2014, Oracle. All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter diskgroup RED_DATA add disk '/dev/oracleasm/disks/ASMDISK12';
alter diskgroup RED_DATA add disk '/dev/oracleasm/disks/ASMDISK12'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15260: permission denied on ASM disk group

Now let us try logging as sysasm user

ORACLE_SID = [+ASM2] ?
The Oracle base remains unchanged with value /grid/app/grid
[grid@stdby02 bin]$ sqlplus / as sysasm

SQL*Plus: Release 12.1.0.2.0 Production on Tue Jun 27 10:50:24 2017

Copyright (c) 1982, 2014, Oracle. All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter diskgroup RED_DATA add disk '/dev/oracleasm/disks/ASMDISK12';

Diskgroup altered.

SQL> select * from v$asm_operation;

GROUP_NUMBER OPERA PASS STAT POWER ACTUAL SOFAR EST_WORK
------------ ----- --------- ---- ---------- ---------- ---------- ----------
 EST_RATE EST_MINUTES ERROR_CODE CON_ID
---------- ----------- -------------------------------------------- ----------
 1 REBAL REBALANCE RUN 1 1 163 1386
 527 2 0

1 REBAL COMPACT WAIT 1 1 0 0
 0 0 0

It worked.!!

#11g, #12c, #alter, #alter-diskgroup, #asm, #disk-group, #grid, #ora-15032, #ora-15260, #oracle, #permission-denied, #rac, #sysasm, #sysdba

Oracle RAC 12cR1 Grid Infrastructure 12.1.0.2 Installation – [INS-20802] Oracle Cluster Verification Utility failed

After a successful execution Root scripts, the Grid Infrastructure 12.1.0.2 Installation failed at “Configure Oracle Grid Infrastructure for a Cluster” stage with following error –

[INS-20802] Oracle Cluster Verification Utility failed.

When looked into installation log file, Cluster Time Synchronization Service (CTSS) is started in Observer state.
Which means CTSS is expecting NTPD service to be up and running.

INFO: CTSS is in Observer state. Switching over to clock synchronization checks using NTP
INFO: Starting Clock synchronization checks using Network Time Protocol(NTP)...
INFO: NTP configuration file "/etc/ntp.conf" existence check passed
INFO: PRVF-7590 : "ntpd" is not running on node "stdby02"
INFO: PRVF-7590 : "ntpd" is not running on node "stdby01"
INFO: PRVG-1024 : The NTP Daemon or Service was not running on any of the cluster nodes.
INFO: PRVF-5415 : Check to see if NTP daemon or service is running failed
INFO: Clock synchronization check using Network Time Protocol(NTP) failed
INFO: PRVF-9652 : Cluster Time Synchronization Services check failed

So, I enabled NTPD service on both the nodes using following commands.

service ntpd start
Starting ntpd: [ OK ]
chkconfig ntpd on

Clicked “Retry” on the Installation window.

This time it failed with following errors saying that the NTP Time Server is different on both the nodes.

PRVF-9652 : Cluster Time Synchronization Services check failed

It is required to have atleast one NTP Time Server to be common both nodes for a successful Clock Synchronization check.

root@stdby01 logs]# tail -100f installActions2017-06-22_09-33-49PM.log
INFO: CTSS is in Observer state. Switching over to clock synchronization checks using NTP
INFO: Starting Clock synchronization checks using Network Time Protocol(NTP)...
INFO: NTP configuration file "/etc/ntp.conf" existence check passed
INFO: Liveness check passed for "ntpd"
INFO: Check for NTP daemon or service alive passed on all nodes
INFO: ERROR:
INFO: PRVF-5408 : NTP Time Server "104.171.118.241" is common only to the following nodes "stdby02"
INFO: ERROR:
INFO: PRVF-5408 : NTP Time Server "132.239.1.6" is common only to the following nodes "stdby02"
INFO: ERROR:
INFO: PRVF-5408 : NTP Time Server "129.6.15.29" is common only to the following nodes "stdby01"
INFO: ERROR:
INFO: PRVF-5408 : NTP Time Server ".GPS." is common only to the following nodes "stdby02"
INFO: ERROR:
INFO: PRVF-5408 : NTP Time Server "129.6.15.27" is common only to the following nodes "stdby01"
INFO: ERROR:
INFO: PRVF-5408 : NTP Time Server "139.78.97.128" is common only to the following nodes "stdby01"
INFO: PRVF-5416 : Query of NTP daemon failed on all nodes
INFO: Clock synchronization check using Network Time Protocol(NTP) failed
INFO: PRVF-9652 : Cluster Time Synchronization Services check failed

NTP Time servers that are being used can be seen issuing ntpq -pn command. The server with “*” is currently in use.

[root@stdby02 app]# /usr/sbin/ntpq -pn
 remote refid st t when poll reach delay offset jitter
==============================================================================
+173.230.144.109 132.239.1.6 2 u 44 64 177 123.116 6.012 7.763
+104.171.118.254 104.171.118.241 3 u 1 64 177 51.457 7.346 19.068
*185.140.51.3 .GPS. 1 u 55 64 177 70.576 10.809 26.393 <---

[root@stdby01 logs]# /usr/sbin/ntpq -pn
 remote refid st t when poll reach delay offset jitter
==============================================================================
+104.131.53.252 129.6.15.29 2 u 63 64 77 50.038 1.200 4.071
+66.85.74.226 192.5.41.40 2 u - 64 177 82.885 5.497 4.595
*108.61.73.244 129.6.15.27 2 u 33 64 177 39.497 1.523 9.439 <---

If we want to use CTSS anyway instead of NTPD for time synchronization, disable NTPD service and remove /etc/ntp.conf file on all the nodes.

[root@stdby01 bin]# service ntpd stop
Shutting down ntpd: [ OK ]
[root@stdby01 bin]# chkconfig ntpd off
[root@stdby01 bin]# mv /etc/ntp.conf /etc/ntp.conf.back

Before clicking “Retry” on the installation window, just wanted to perform clocksync check using cluvfy utility.

The cluvy utility always looks for NTPD service to be up and running and CTSS demon to be started in Observer Mode. So, ignore the below error.

[root@stdby01 bin]# su grid
[grid@stdby01 bin]$ ./cluvfy comp clocksync -n all -verbose

Verifying Clock Synchronization across the cluster nodes

Checking if Clusterware is installed on all nodes...
Oracle Clusterware is installed on all nodes.

Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
 Node Name Status
 ------------------------------------ ------------------------
 stdby01 passed
 stdby02 passed
CTSS resource check passed

Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed

Check CTSS state started...
Check: CTSS state
 Node Name State
 ------------------------------------ ------------------------
 stdby02 Observer
 stdby01 Observer
CTSS is in Observer state. Switching over to clock synchronization checks using NTP

Starting Clock synchronization checks using Network Time Protocol(NTP)...

Checking existence of NTP configuration file "/etc/ntp.conf" across nodes
 Node Name File exists?
 ------------------------------------ ------------------------
 stdby02 no
 stdby01 no
PRVG-1019 : The NTP configuration file "/etc/ntp.conf" does not exist on nodes "stdby02,stdby01"
PRVF-5414 : Check of NTP Config file failed on all nodes. Cannot proceed further for the NTP tests

Checking daemon liveness...

Check: Liveness for "ntpd"
 Node Name Running?
 ------------------------------------ ------------------------
 stdby02 no
 stdby01 no
PRVF-7590 : "ntpd" is not running on node "stdby02"
PRVF-7590 : "ntpd" is not running on node "stdby01"
PRVG-1024 : The NTP Daemon or Service was not running on any of the cluster nodes.
PRVF-5415 : Check to see if NTP daemon or service is running failed
Result: Clock synchronization check using Network Time Protocol(NTP) failed

PRVF-9652 : Cluster Time Synchronization Services check failed

Verification of Clock Synchronization across the cluster nodes was unsuccessful on all the specified nodes.

Clicked “Retry” on the installation window.

This time installation completed successfully and noticed that CTSS is started in Active state.

[grid@stdby01 bin]$ tail -100f /grid/app/oraInventory/logs/installActions2017-06-22_09-33-49PM.log
INFO: Oracle Clusterware is installed on all nodes.
INFO: CTSS resource check passed
INFO: Query of CTSS for time offset passed
INFO: CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
INFO: Check of clock time offsets passed
INFO: Oracle Cluster Time Synchronization Services check passed
INFO: Checking VIP configuration.
INFO: Checking VIP Subnet configuration.

 

Thanks 🙂

 

#12-1-0-2, #12c, #12cr1, #20802, #clocksync, #cluster, #cluster-time-synchronization-services, #cluster-time-synchronization-services-check-failed, #cluvfy, #crsctl, #ctss, #failed, #grid, #ins, #ins-20802, #installation, #ntp, #ntp-time-server, #ntp-time-server-is-common-only-to-the-following, #ntpd, #oracle, #oracle-grid-infrastructure-12-1-0-2-installation-ins-20802-oracle-cluster-verification-utility-failed, #prvf-5408, #prvf-5416, #prvf-9652, #rac, #service, #utility, #verification

ERROR OGG-01264 The call to the cm_param_set() function from line 378 in cm_sp_init_common() returned an unexpected value.

2016-11-30 18:25:47 INFO OGG-01851 filecaching started: thread ID: 140714592524032.

2016-11-30 18:25:47 WARNING OGG-01262 The call to the ggDirIsWritable() function from line 2673 in cm_cachedir_add() failed with reason 'The directory /u01/goldengate/data
/mydb/cache to hold temporary cachefiles does not exist or is not writable'.

2016-11-30 18:25:47 WARNING OGG-01278 cm_param_dir: cm_memdir_add: /u01/goldengate/data/mydb/cache size: default.

2016-11-30 18:25:47 WARNING OGG-01276 cm_param_arg: BAD DIRECTORY PARAMETER: CACHEDIRECTORY /u01/goldengate/data/mydb/cache.

2016-11-30 18:25:47 WARNING OGG-01271 cm_param_parse: Bad Parameter Argument: CACHEDIRECTORY /u01/goldengate/data/mydb/cache.

2016-11-30 18:25:47 WARNING OGG-01270 cm_param_set: Bad Parameter: CACHEMGR CACHEDIRECTORY /u01/goldengate/data/mydb/cache.

Source Context :
 SourceModule : [ggapp.cachemgr]
 SourceID : [/scratch/aime/adestore/views/aime_adc4150408/oggcore/OpenSys/src/gglib/ggapp/cachemgr/cmgr_restore.c]
 SourceFunction : [cm_sp_init_common]
 SourceLine : [378]
 ThreadBacktrace : [12] elements
 : [/u01/goldengate/oracle/OGG/libgglog.so(CMessageContext::AddThreadContext()+0x1b) [0x7ffac1709f1b]]
 : [/u01/goldengate/oracle/OGG/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x134) [0x7ffac17040a4]]
 : [/u01/goldengate/oracle/OGG/libgglog.so(_MSG_ERR_CACHEOBJ_FUNCTION_FAILED(CSourceContext*, char const*, CMessageFactory::MessageDispos
ition)+0x30) [0x7ffac16e5d6e]]
 : [/u01/goldengate/oracle/OGG/extract() [0x6a189d]]
 : [/u01/goldengate/oracle/OGG/extract(CM_cache_init(pool_info*, char const*, extract_vals*)+0x58) [0x6a2718]]
 : [/u01/goldengate/oracle/OGG/extract() [0x5f3a5a]]
 : [/u01/goldengate/oracle/OGG/extract(ggs::gglib::MultiThreading::MainThread::ExecMain()+0x60) [0x6d05b0]]
 : [/u01/goldengate/oracle/OGG/extract(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::Thread::ThreadArgs*)+0x1
4d) [0x6d179d]]
 : [/u01/goldengate/oracle/OGG/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**)+0xb1) [0x6d1881]]
 : [/u01/goldengate/oracle/OGG/extract(main+0x3b) [0x5f50eb]]
 : [/lib64/libc.so.6(__libc_start_main+0xfd) [0x3409a1ed1d]]
 : [/u01/goldengate/oracle/OGG/extract() [0x563159]]

2016-11-30 18:25:47 ERROR OGG-01264 The call to the cm_param_set() function from line 378 in cm_sp_init_common() returned an unexpected value.

2016-11-30 18:25:47 ERROR OGG-01668 PROCESS ABENDING.

Create the mising directory for Cache and restart the process

[gguser@newhost01 OGG] $ pwd
/u01/goldengate/oracle/OGG
[gguser@newhost01 OGG] $ mkdir -p /u01/goldengate/data/mydb/cache

#abending, #cache, #cachedirectory, #cachefiles, #cachemgr, #cm_param_set, #extract, #goldengate, #ogg-01264, #pump, #replicat

ERROR OGG-02077 Extract encountered a read error in the asynchronous reader thread and is abending: Error code 3113, error message: ORA-03113: end-of-file on communication channel

Integrated Extracts are often abending with the error “ERROR   OGG-02077  Extract encountered a read error in the asynchronous reader thread and is abending: Error code 3113, error message: ORA-03113: end-of-file on communication channel”.

Simply restarting the extract process worked.

Will update this post after further research..

#3133, #apply, #asynchronous, #cature, #channel, #communication, #end, #end-of-file, #error, #extract, #file, #ggsci, #goldengate, #integrated, #integrated-apply, #integrated-capture, #ogg, #ogg-02077, #ora-03113, #oracle, #oracle-goldengate, #pump, #replicat

OGG-00303 Invalid version found in DEFGEN file: 5.0.

Recently came across an issue with the replicats abending with  when the source system Goldengate version is 12.2.0.1.1 whereas the target Goldengate version is 12.1.2.1.2.

OGG-00303  Invalid version found in DEFGEN file: 5.0.
OGG-01668  PROCESS ABENDING.

The source Goldengate version is higher than the target Goldengate version. The DEFGEN file created on 12.2 version cannot be interpreted by 12.1 version of Goldengate. In this situation, create the DEFSFILE using FORMAT RELEASE to specify the version of target Goldengate.

DEFSFILE ./dirdef/dnfsd.def, PURGE, FORMAT RELEASE 12.1
USERID ggadm@testdb, PASSWORD ******
TABLE SAMPLE.MYTABLE;

Refer to Oracle Support (Doc ID 2155977.1) for more information.

#00303, #12-1, #12-2, #5-0, #defgen, #extract, #file, #format, #found, #ggsci, #goldengate, #invalid, #ogg, #ogg-00303, #oracle-goldengate, #pump, #release, #replicat, #version

DBLOGIN – OGG-01117 Received signal: Segmentation violation (11) / Memory fault(coredump)

Encountered “OGG-01117  Received signal: Segmentation violation (11)” when trying to login into database from GGSCI.

Figured out this is because my ORACLE_HOME and LD_LIBRARY_PATH are not set appropriately.

#coredump, #dblogin, #fault, #ggsci, #goldengate, #ld_library_path, #memory, #memory-faultcoredump, #ogg-01117, #oracle, #oracle-goldengate, #password, #received-signal-segmentation-violation