Viscosity's Blog

Oracle ACFS / ASM Filter Driver and Oracle Enterprise Linux

Written by Gary Gordhamer | Jun 17, 2020 4:34:00 PM
In the world of Oracle databases, DBA’s and system administrators don’t always see eye to eye.  Mostly I have found that this is really just about coming at challenges from different angles.  Some of the major changes in Oracle since 10g version has been the introduction of Oracle Automatic Storage Management or ASM.  This has led to many features such as ASMLIB, ASM Clustered Filesystem (ACFS), and now the ASM Filter Driver (AFD or ASMFD).  These features require OS specific compatible libraries, and kernel modules.  Information on compatibility can be confusing.
 

Background References

Let’s start with some background.  For this blog, I’m going to focus on Oracle Enterprise Linux (OEL).  The information here could be assembled for any supported OS.  This information is from a few sources, the main ones being:
Before making any major patching or upgrade decisions please reference these notes and Oracle support in general. 
 
Oracle does a pretty good job of providing information on Linux and Oracle database / ASM compatibility.  Though, the notes are not always clear as to what the goals is.  For example, I have heard many people say that ASM has stopped working during a Linux upgrade.  Or that ASM drivers are not compatible with Linux changes, or that you can’t patch Linux without breaking ASM / AFD / ACFS.  Well, these are antidotal statements, and the truth is a little more complex.
 

Linux Kernel Version

Oracle Enterprise Linux (OEL) recommends using the Unbreakable Enterprise Kernel or UEK.  The main reason behind this is that Oracle controls the destiny of the UEK.  As noted above ACFS and AFD are kernel dependent.  So having the ability to set course for both the Linux Kernel and the ASM related kernel modules is important.  For the scope of this discussion, let’s look at OEL releases 5 through 8 and Oracle Database releases 11gR2 through 19c.  First here is a high level view of OEL and UEK releases.
Support Dates OEL Version            
Jun 2007 - Jun 2017 OEL5            
Feb 2011 - Mar 2021 OEL6            
Jul 2014 - Jul 2024 OEL7            
Jul 2019 - Jul 2029 OEL8            
  UEK YUM Channel UEKR1 UEKR2 UEKR3 UEKR4 UEKR5 UEKR6
  Major UEK Version 2.6.32 2.6.39 3.8.13 4.1.12 4.14.35 5.4.17
  Released   Mar, 2012 Oct, 2013 Jan, 2016 Jun, 2018 Mar, 2020
 
So this is a nice swath of about 10 years of versions.  Please note that the major version number of the kernels does not change if you don’t change UEK channels.  For example, if you move from OEL6 Update 5 to Update 8, but stick with UEKR3 your kernel major number will remain 3.8.13.  The minor numbers beyond the .13 will change, but not the major number.  This is also true for OEL7 and UEKR5 where the major release number is 4.14.35.  Within in this major version, Oracle release new security fixes and patch updates monthly or quarterly for each kernel version that is supported within the support dates.  So for example, OEL6 UEKR2 which was first release in Mar of 2012, just released kernel UEK 2.6.39-400.321.1 in April of 2020.  That is a long history of fixing the same set of code.
 

ACFS / AFD Linux Kernel Compatibility

Now let's move over to the database support for ACFS / AFD on OEL.  Here is another summary now listing Oracle database versions.
  UEK YUM Channel UEKR1 UEKR2 UEKR3 UEKR4 UEKR5 UEKR6
  Major UEK Version 2.6.32 2.6.39 3.8.13 4.1.12 4.14.35 5.4.17
DB Version Release Date   Mar, 2012 Oct, 2013 Jan, 2016 Jun, 2018 Mar, 2020
11.2.0.3 OEL5 - OEL6 2.6.32-100 2.6.39-100        
11.2.0.4 OEL5 - OEL6 2.6.32-100 2.6.39-100 3.8.13-118 4.1.12-112.16.4    
12.1.0.1 OEL5 - OEL6 2.6.32-100 2.6.39-100        
12.1.0.2 OEL6 - OEL7   2.6.39-100 3.8.13-118      
12.2.0.1 OEL6 - OEL7   2.6.39-100 3.8.13-118 4.1.12-112.16.4 4.14.35-1902  
18c OEL6 - OEL7   2.6.39-100 3.8.13-118 4.1.12-112.16.4 4.14.35-1902 TBD
19c OEL7       4.1.12-112.16.4 4.14.35-1902 TBD
 
Here again I’ve covered about 10 years, and you can see clearly what high-level database versions are compatible with what OEL and UEK versions.  You might also see Oracle’s commitment to long term support for 11.2.0.4.  Similar to the expected support for 19c going until 2024.  Also note that UEKR6 is not yet supported.
 
Now the information above still high level.  You may assume that every 19c version of Oracle support all UEK4 and UEK5 kernels out of the box.  That is not true.  Database releases and kernel releases are not specifically synchronized.  So perhaps you have tried to test if the ACFS drivers can be installed:
$ acfsdriverstate supported -v
ACFS-9459: ADVM/ACFS is not supported on this OS version: '4.14.35-1902.10.8.el7uek.x86_64'
ACFS-9201: Not Supported

ACFS-9553: Operating System: linux
ACFS-9554: Machine Architecture: x86_64-linux-thread-multi
ACFS-9555: Operating system name and information: Linux srvr03 4.14.35-1902.10.8.el7uek.x86_64 #2 SMP Thu Feb 6 11:02:28 PST 2020 x86_64 x86_64 x86_64 GNU/Linux
ACFS-9556: Release package: oraclelinux-release-7.7-1.0.5.el7.x86_64
ACFS-9557: Version: ADVM/ACFS is not supported on 4.14.35-1902.10.8.el7uek.x86_64
ACFS-9558: Variable _ORA_USM_NOT_SUPPORTED is defined: no

ASM Required Patches

If your combination of grid software and Linux kernel are not supported, you will see the above message “ACFS-9201: Not Supported”.  How can you fix this?  Oracle provides patches at specific versions of GI to support UEK versions.
DB Version OS Version UEK Channel UEK Minimum Base bug or Base version
11.2.0.3.6 OEL5 - OEL6 UEKR1 2.6.32-100 15986571
11.2.0.3.7 OEL5 - OEL6 UEKR1 2.6.32-100 12983005
11.2.0.4 OEL5 UEKR1 2.6.32-100 Base
11.2.0.4 OEL5 - OEL6 UEKR2 2.6.39-100 Base
11.2.0.4.4 OEL6 UEKR3 3.8.13-118 16318126
11.2.0.4.6 OEL7 UEKR3 3.8.13-35 18321597
11.2.0.4.180717  OEL6 - OEL7 UEKR4 4.1.12-112.16.4 22810422, 28171094, 27463879
12.1.0.1 OEL5 - OEL6 UEKR1 2.6.32-100 Base
12.1.0.1 OEL5 - OEL6 UEKR2 2.6.39-100 Base
12.1.0.2.170718 OEL5 - OEL6 UEKR2 2.6.39-100 Base
12.1.0.2.170718 OEL6 UEKR3 3.8.13-13 Base
12.1.0.2.170718 OEL7 UEKR3 3.8.13-35 18321597
12.1.0.2.181016 OEL6 - OEL7 UEKR4 4.1.12-112.16.4 22810422, 27942938, 28171094, 27942938
12.1.0.2.190716 OEL7 UEKR5 4.14.35-1902 27494830
12.2.0.1 OEL6 UEKR2 2.6.39-100 Base
12.2.0.1 OEL6 UEKR3 3.8.13-13 Base
12.2.0.1.180717 OEL6 UEKR4 4.1.12-112.16.4 27463879, 28171094
12.2.0.1 OEL7 UEKR3 3.8.13-35 Base
12.2.0.1.180717 OEL7 UEKR4 4.1.12-112.16.4 27463879, 28171094, 27463879
12.2.0.181016 OEL7 UEKR5 4.14.35-1902 28069955
18.3.0.0 OEL6 UEKR2 2.6.39-100 Base
18.3.0.0 OEL6 UEKR3 3.8.13-13 Base
18.4.0.0.181016 OEL6 UEKR4 4.1.12-112.16.4 28069955
18.3.0.0 OEL7 UEKR3 3.8.13-35 Base
18.4.0.0.181016 OEL7 UEKR4 4.1.12-112.16.4 27463879, 28171094, 27463879
18.4.0.0.181016 OEL7 UEKR5 4.14.35-1902 27494830
19.3.0.0 OEL7 UEKR4 4.1.12-112.16.4 Base
19.4.190716 OEL7 UEKR5 4.14.35-1902 27494830
 
The example above on my test system is a fresh install of GI 19.3.0.0 on a OEL 7.8 system using UEKR5.  Based on the list above we can tell that for UEKR5 support I need either 19.4.190716 or I need patch 27494830.  Oracle provides many patches where the title contains “ACFS RU” that you can search for in the patches section of MOS.  I decided to use this patch “MERGE ON ACFS RU 19.6.0.0.0 OF 28531803 30685278 (Patch 31055785)” which includes the following merge of patches:
Patch  31055785     : applied on Fri May 22 14:15:34 CDT 2020Unique Patch ID:  23448796Patch description:  "ACFS Interim patch for 31055785"   Created on 20 Mar 2020, 03:53:18 hrs PST8PDT   Bugs fixed:     28531803, 30685278, <font color="#d52c1f">27494830</font>, 27917085, 28064731, 28293236, 28321248     28375150, 28553487, 28611527, 28687713, 28701011, 28740425, 28818513     28825772, 28844788, 28855761, 28860451, 28900212, 28951588, 28960047     28995524, 29001307, 29030106, 29031452, 29039918, 29115917, 29116692     29127489, 29127491, 29167352, 29173957, 29198743, 29229120, 29234059     29250565, 29264772, 29302070, 29313039, 29318169, 29338628, 29339084     29350729, 29363565, 29379932, 29391373, 29411007, 29417321, 29428155     29437701, 29482354, 29484738, 29520544, 29524859, 29527221, 29551699     29560075, 29586338, 29590813, 29604082, 29643818, 29704429, 29705711     29721120, 29760083, 29779338, 29791186, 29848987, 29851205, 29862693     29872187, 29893148, 29929003, 29929061, 29937236, 29941227, 29963428     30003321, 30032562, 30046061, 30051637, 30057972, 30076951, 30093449     30140896, 30179531, 30239575, 30251503, 30264950, 30269395, 30275174     30324590, 30363621, 30655657

Which you can see includes the required patch 27494830 (first row, 3rd column). Depending on your version of Oracle, and quarterly bundle patch there are many options of patches available. The quarterly bundle patches for GI generally contain an updated ACFS patch as well.

 

Details of the ASM Kernel Modules 

In 19c, under the GRID_HOME is a directory called usm. On my test system under here I can find this: 
$ ls $GRID_HOME/usm/install/Oracle/EL7UEK/x86_644.1.12/  4.1.12-112.16.4/  4.14.35-1902/
This is the location of the libraries used to compile the kernel modules for ACFS / AFD.  As expected, there is a directory for each major kernel version that pretty much matches our list above.  The command “ amcmd afd_configure -e” is used to install the AFD kernel modules.  The AFD module use the Kernel Application Binary Interface and are installed under the /lib/modules location on your Linux system.  As they are considered weak modules, they are placed in one location and then symbolically linked into each kernel specific directory when the kernel is updated.
[root@srvr03 sbin]# ls -l /lib/modules/4.14.35-1902*/{extra,weak-updates}/oracle
/lib/modules/4.14.35-1902.0.9.el7uek.x86_64/extra/oracle:
total 9824
-rw-r--r-- 1 root root 10055682 Jun  5 13:16 oracleafd.ko

/lib/modules/4.14.35-1902.10.8.el7uek.x86_64/weak-updates/oracle:
total 0
lrwxrwxrwx 1 root root 69 Jun 5 13:17 oracleafd.ko -> /lib/modules/4.14.35-1902.0.9.el7uek.x86_64/extra/oracle/oracleafd.ko

/lib/modules/4.14.35-1902.11.3.1.el7uek.x86_64/weak-updates/oracle:
total 0
lrwxrwxrwx 1 root root 69 Jun 5 13:44 oracleafd.ko -> /lib/modules/4.14.35-1902.0.9.el7uek.x86_64/extra/oracle/oracleafd.ko

 
Here we can see that the primary module was installed under 4.14.35-1902.0.9.  Then when the kernel 4.14.35-1902.11.3.1 was installed, a symbolic link was created to the original module.
 
The AFD module is normally loaded by a Linux RC script called afd (/etc/init.d/afd).  You can verify it will be started at boot with the following command as the root user:
[root@srvr03 init.d]# chkconfig --list afd

Note: This output shows SysV services only and does not include native
systemd services. SysV configuration data might be overridden by native
systemd configuration.

If you want to list systemd services use 'systemctl list-unit-files'.
To see services enabled on particular target use
'systemctl list-dependencies [target]'.

afd 0:off 1:off 2:off 3:on 4:off 5:on 6:off

Here you can see that the AFD module will attempt to be loaded at run levels 3 and 5.  You can also check what version of the AFD module you have loaded with the following Linux command:

[oracle@srvr03 ~]$ modinfo oracleafdfilename:       /lib/modules/4.14.35-1902.11.3.1.el7uek.x86_64/weak-updates/oracle/oracleafd.kolicense:        Oracle Corporationdescription:    ASM Filter Driverauthor:         Oracle Corporationsrcversion:     533BB7E5866E52F63B9ACCBdepends:        retpoline:      Yname:           oracleafdvermagic:       4.14.35-1902.0.9.el7uek.x86_64 SMP mod_unload modversions signat:         PKCS#7signer:         sig_key:        sig_hashalgo:   md4parm:           oracleafd_use_logical_block_size:Non-zero value implies using logical sectorsize (int)parm:           oracleafd_use_maxio_softlimits:Non-zero value implies using conservative maximum block io limits (int)

Using the lsmod command to see if it is loaded.  If this command returns no lines, then it is not loaded. 

[oracle@srvr03 ~]$ lsmod |grep oracleafdoracleafd             229376  0

 

Conclusion

Ok, so hopefully this has given you a clear picture as to what you need to plan for when using ASM Filter Driver (AFD) or ACFS on your OEL system with UEK.  Once you have a base version of kernel and GI you should have little issues unless you jump UEK channels or do a GI major version change.  As many people are eyeing up 19c upgrades, hopefully the above information is helpful to prepare your Linux OS for the change.  For myself what started as a somewhat simple post about kernel drivers turned into a pretty long delve into the workings of the ASM filter driver.  Please leave your comments and questions.

 
View full post