FlexASM Deep Dive – Show Me the Output!!!

Written by Viscosity North America | Sep 23, 2013 9:40:04 PM

If you saw the first FlexASM blog you know we installed and configured FlexASM and a CDB plus a couple of PDBs. Also, this was Policy Managed with a cardinality of 2. Now let's see what the configuration looks like, and we can break it down using the wonderful crsctl and srvctl tools

First let's ensure we are really running in FlexASM mode:

[oracle@rac02 ~]$ asmcmd showclustermode
ASM cluster : Flex mode enabled

[oracle@rac02 ~]$ srvctl status serverpool -serverpool naboo
Server pool name: naboo
Active servers count: 2

[oracle@rac01 trace]$ crsctl get node role status -all
Node 'rac01' active role is 'hub'
Node 'rac03' active role is 'hub'
Node 'rac02' active role is 'hub'
Node 'rac04' active role is 'hub'

[oracle@rac01 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE rac01 STABLE
ONLINE ONLINE rac02 STABLE
ONLINE ONLINE rac03 STABLE
ONLINE ONLINE rac04 STABLE

You notice that we have 4 ASM listeners one on each node in the Cluster. You'll see the process as the following on each node:

[oracle@rac01 ~]$ ps -ef |grep -i asmnet

ooracle 6646 1 0 12:19 ? 00:00:00 /u01/app/12.1.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit

ora.CRSDATA.DATAVOL1.advm
ONLINE ONLINE rac01 Volume device /dev/a
sm/datavol1-194 is o
nline,STABLE
ONLINE ONLINE rac02 Volume device /dev/a
sm/datavol1-194 is o
nline,STABLE
ONLINE OFFLINE rac03 Unable to connect to
ASM,STABLE
ONLINE ONLINE rac04 Volume device /dev/a
sm/datavol1-194 is o
nline,STABLE
The datavol1 ADVM resource runs on all the nodes where indicated it should run. In this case we are seeing that RAC03 is having some issues.
Let's look into that a little later. But I like the fact crsctl tells something is amiss here on node3

ora.CRSDATA.dg
ONLINE ONLINE rac01 STABLE
ONLINE ONLINE rac02 STABLE
ONLINE ONLINE rac03 STABLE
OFFLINE OFFLINE rac04 STABLE

ora.FRA.dg
ONLINE ONLINE rac01 STABLE
ONLINE ONLINE rac02 STABLE
ONLINE ONLINE rac03 STABLE
OFFLINE OFFLINE rac04 STABLE

The crsdata and fra disk groups resource is started on all nodes except node 4

ora.LISTENER.lsnr
ONLINE ONLINE rac01 STABLE
ONLINE ONLINE rac02 STABLE
ONLINE ONLINE rac03 STABLE
ONLINE ONLINE rac04 STABLE

We all know, as in 11gR2, that this is the Node listener.

ora.PDBDATA.dg
ONLINE ONLINE rac01 STABLE
ONLINE ONLINE rac02 STABLE
ONLINE ONLINE rac03 STABLE
OFFLINE OFFLINE rac04 STABLE

The pdbdata disk groups resource is started on all nodes except node 4

ora.crsdata.datavol1.acfs
ONLINE ONLINE rac01 mounted on /u02/app/
oracle/acfsmounts,ST
ABLE
ONLINE ONLINE rac02 mounted on /u02/app/
oracle/acfsmounts,ST
ABLE
ONLINE OFFLINE rac03 (2) volume /u02/app/
oracle/acfsmounts of
fline,STABLE
ONLINE ONLINE rac04 mounted on /u02/app/
oracle/acfsmounts,ST
ABLE

ACFS filesystem resource for datavol1 is started on all nodes except node3.
But I think the following has something to do w/ it :-). Need to debug this a bit later. I even tried:
[oracle@rac03 ~]$ asmcmd volenable --all
ASMCMD-9470: ASM proxy instance unavailable
ASMCMD-9471: cannot enable or disable volumes

ora.net1.network
ONLINE ONLINE rac01 STABLE
ONLINE ONLINE rac02 STABLE
ONLINE ONLINE rac03 STABLE
ONLINE ONLINE rac04 STABLE
ora.ons
ONLINE ONLINE rac01 STABLE
ONLINE ONLINE rac02 STABLE
ONLINE ONLINE rac03 STABLE
ONLINE ONLINE rac04 STABLE

The Network (in my case I only have only Net1) and ONS are same as in previous versions

ora.proxy_advm
ONLINE ONLINE rac01 STABLE
ONLINE ONLINE rac02 STABLE
ONLINE OFFLINE rac03 STABLE
ONLINE ONLINE rac04 STABLE

Yep, since proxy_advm is not started on node3, the filesystems won't come online….but again, i'll look at that later
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac02 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac03 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac04 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE rac01 169.254.90.36 172.16
.11.10,STABLE
ora.asm
1 ONLINE ONLINE rac03 STABLE
2 ONLINE ONLINE rac01 STABLE
3 ONLINE ONLINE rac02 STABLE

Since we have the cardinality of 3 ASM instance we have 3 ASM resources active

ora.cvu
1 ONLINE ONLINE rac01 STABLE
ora.mgmtdb
1 ONLINE ONLINE rac01 Open,STABLE
ora.oc4j
1 ONLINE ONLINE rac01 STABLE
ora.rac01.vip
1 ONLINE ONLINE rac01 STABLE
ora.rac02.vip
1 ONLINE ONLINE rac02 STABLE
ora.rac03.vip
1 ONLINE ONLINE rac03 STABLE
ora.rac04.vip
1 ONLINE ONLINE rac04 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac02 STABLE
ora.scan2.vip
1 ONLINE ONLINE rac03 STABLE
ora.scan3.vip
1 ONLINE ONLINE rac04 STABLE
ora.tatooine.db
1 ONLINE ONLINE rac01 Open,STABLE
2 ONLINE ONLINE rac02 Open,STABLE

As we stated above, I specified a Policy Managed database with cardinality of 2, so I have 2 database instances running
--------------------------------------------------------------------------------

Here's some other important supporting info on FlexASm:

[oracle@rac02 ~]$ srvctl config asm -detail
ASM home: /u01/app/12.1.0/grid
Password file: +CRSDATA/orapwASM
ASM listener: LISTENER
ASM is enabled.
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM

[oracle@rac02 ~]$ srvctl status filesystem
ACFS file system /u02/app/oracle/acfsmounts is mounted on nodes rac01,rac02,rac04

ANd here's what the Database has to say about FlexASM

NOTE: ASMB registering with ASM instance as client 0x10001 (reg:1377584805)
NOTE: ASMB connected to ASM instance +ASM1 (Flex mode; client id 0x10001)
NOTE: ASMB rebuilding ASM server state
NOTE: ASMB rebuilt 2 (of 2) groups
SUCCESS: ASMB reconnected & completed ASM server state

So for the interesting part:
If you notice that ASM is not running node 4:
[oracle@rac02 ~]$ srvctl status asm -v

ASM is running on rac01,rac02,rac03
[oracle@rac02 ~]$ srvctl status asm -detail
ASM is running on rac01,rac02,rac03

So, how does a client (ocrdump, rman, asmcmd, etc..) connect to if ASM if there is no ASM on that node. Well let's test this using asmcmd on node4. You notice that a pipe is created, a connect string is generated and passed to ASMCMD to connect remotely to ASM2 on node2!!!!

22-Sep-13 12:54 ASMCMD Foreground (PID = 14106): Pipe /tmp/pipe_14106 has been found.
22-Sep-13 12:54 ASMCMD Background (PID = 14117): Successfully opened the pipe /tmp/pipe_14106
22-Sep-13 12:54 ASMCMD Foreground (PID = 14106): Successfully opened the pipe /tmp/pipe_14106 in read mode
NOTE: Executing kfod /u01/app/12.1.0/grid/bin/kfod op=getclstype..
22-Sep-13 12:54 Printing the connection string
contype =
driver =
instanceName = <>
usr =
ServiceName = <+ASM>
23-Sep-13 16:23 Successfully connected to ASM instance +ASM2
23-Sep-13 16:23 NOTE: Querying ASM instance to get list of disks
22-Sep-13 12:54 Registered Daemon process.
22-Sep-13 12:54 ASMCMD Foreground (PID = 14106): Closed pipe /tmp/pipe_14106.

View full post