In the world of Oracle databases, DBA’s and system administrators don’t always see eye to eye. Mostly I have found that this is really just about coming at challenges from different angles. Some of the major changes in Oracle since 10g version has been the introduction of Oracle Automatic Storage Management or ASM. This has led to many features such as ASMLIB, ASM Clustered Filesystem (ACFS), and now the ASM Filter Driver (AFD or ASMFD). These features require OS specific compatible libraries, and kernel modules. Information on compatibility can be confusing.
Background References
Let’s start with some background. For this blog, I’m going to focus on Oracle Enterprise Linux (OEL). The information here could be assembled for any supported OS. This information is from a few sources, the main ones being:
- My Oracle Support (MOS) document “ACFS Support On OS Platforms (Certification Matrix). (Doc ID 1369107.1)”
- "Oracle Linux and Unbreakable Enterprise Kernel (UEK) Releases"By Simon Coter, SENIOR MANAGER, ORACLE LINUX AND VIRTUALIZATION PRODUCT MANAGEMENT https://blogs.oracle.com/scoter/oracle-linux-and-unbreakable-enterprise-kernel-uek-releases
Before making any major patching or upgrade decisions please reference these notes and Oracle support in general.
Oracle does a pretty good job of providing information on Linux and Oracle database / ASM compatibility. Though, the notes are not always clear as to what the goals is. For example, I have heard many people say that ASM has stopped working during a Linux upgrade. Or that ASM drivers are not compatible with Linux changes, or that you can’t patch Linux without breaking ASM / AFD / ACFS. Well, these are antidotal statements, and the truth is a little more complex.
Linux Kernel Version
Oracle Enterprise Linux (OEL) recommends using the Unbreakable Enterprise Kernel or UEK. The main reason behind this is that Oracle controls the destiny of the UEK. As noted above ACFS and AFD are kernel dependent. So having the ability to set course for both the Linux Kernel and the ASM related kernel modules is important. For the scope of this discussion, let’s look at OEL releases 5 through 8 and Oracle Database releases 11gR2 through 19c. First here is a high level view of OEL and UEK releases.
| Support Dates | OEL Version | ||||||
| Jun 2007 - Jun 2017 | OEL5 | ||||||
| Feb 2011 - Mar 2021 | OEL6 | ||||||
| Jul 2014 - Jul 2024 | OEL7 | ||||||
| Jul 2019 - Jul 2029 | OEL8 | ||||||
| UEK YUM Channel | UEKR1 | UEKR2 | UEKR3 | UEKR4 | UEKR5 | UEKR6 | |
| Major UEK Version | 2.6.32 | 2.6.39 | 3.8.13 | 4.1.12 | 4.14.35 | 5.4.17 | |
| Released | Mar, 2012 | Oct, 2013 | Jan, 2016 | Jun, 2018 | Mar, 2020 |
So this is a nice swath of about 10 years of versions. Please note that the major version number of the kernels does not change if you don’t change UEK channels. For example, if you move from OEL6 Update 5 to Update 8, but stick with UEKR3 your kernel major number will remain 3.8.13. The minor numbers beyond the .13 will change, but not the major number. This is also true for OEL7 and UEKR5 where the major release number is 4.14.35. Within in this major version, Oracle release new security fixes and patch updates monthly or quarterly for each kernel version that is supported within the support dates. So for example, OEL6 UEKR2 which was first release in Mar of 2012, just released kernel UEK 2.6.39-400.321.1 in April of 2020. That is a long history of fixing the same set of code.
ACFS / AFD Linux Kernel Compatibility
Now let's move over to the database support for ACFS / AFD on OEL. Here is another summary now listing Oracle database versions.
| UEK YUM Channel | UEKR1 | UEKR2 | UEKR3 | UEKR4 | UEKR5 | UEKR6 | |
| Major UEK Version | 2.6.32 | 2.6.39 | 3.8.13 | 4.1.12 | 4.14.35 | 5.4.17 | |
| DB Version | Release Date | Mar, 2012 | Oct, 2013 | Jan, 2016 | Jun, 2018 | Mar, 2020 | |
| 11.2.0.3 | OEL5 - OEL6 | 2.6.32-100 | 2.6.39-100 | ||||
| 11.2.0.4 | OEL5 - OEL6 | 2.6.32-100 | 2.6.39-100 | 3.8.13-118 | 4.1.12-112.16.4 | ||
| 12.1.0.1 | OEL5 - OEL6 | 2.6.32-100 | 2.6.39-100 | ||||
| 12.1.0.2 | OEL6 - OEL7 | 2.6.39-100 | 3.8.13-118 | ||||
| 12.2.0.1 | OEL6 - OEL7 | 2.6.39-100 | 3.8.13-118 | 4.1.12-112.16.4 | 4.14.35-1902 | ||
| 18c | OEL6 - OEL7 | 2.6.39-100 | 3.8.13-118 | 4.1.12-112.16.4 | 4.14.35-1902 | TBD | |
| 19c | OEL7 | 4.1.12-112.16.4 | 4.14.35-1902 | TBD |
Here again I’ve covered about 10 years, and you can see clearly what high-level database versions are compatible with what OEL and UEK versions. You might also see Oracle’s commitment to long term support for 11.2.0.4. Similar to the expected support for 19c going until 2024. Also note that UEKR6 is not yet supported.
Now the information above still high level. You may assume that every 19c version of Oracle support all UEK4 and UEK5 kernels out of the box. That is not true. Database releases and kernel releases are not specifically synchronized. So perhaps you have tried to test if the ACFS drivers can be installed:
$ acfsdriverstate supported -v
ACFS-9459: ADVM/ACFS is not supported on this OS version: '4.14.35-1902.10.8.el7uek.x86_64'
ACFS-9201: Not Supported
ACFS-9553: Operating System: linux
ACFS-9554: Machine Architecture: x86_64-linux-thread-multi
ACFS-9555: Operating system name and information: Linux srvr03 4.14.35-1902.10.8.el7uek.x86_64 #2 SMP Thu Feb 6 11:02:28 PST 2020 x86_64 x86_64 x86_64 GNU/Linux
ACFS-9556: Release package: oraclelinux-release-7.7-1.0.5.el7.x86_64
ACFS-9557: Version: ADVM/ACFS is not supported on 4.14.35-1902.10.8.el7uek.x86_64
ACFS-9558: Variable _ORA_USM_NOT_SUPPORTED is defined: no
ASM Required Patches
If your combination of grid software and Linux kernel are not supported, you will see the above message “ACFS-9201: Not Supported”. How can you fix this? Oracle provides patches at specific versions of GI to support UEK versions.
| DB Version | OS Version | UEK Channel | UEK Minimum | Base bug or Base version |
| 11.2.0.3.6 | OEL5 - OEL6 | UEKR1 | 2.6.32-100 | 15986571 |
| 11.2.0.3.7 | OEL5 - OEL6 | UEKR1 | 2.6.32-100 | 12983005 |
| 11.2.0.4 | OEL5 | UEKR1 | 2.6.32-100 | Base |
| 11.2.0.4 | OEL5 - OEL6 | UEKR2 | 2.6.39-100 | Base |
| 11.2.0.4.4 | OEL6 | UEKR3 | 3.8.13-118 | 16318126 |
| 11.2.0.4.6 | OEL7 | UEKR3 | 3.8.13-35 | 18321597 |
| 11.2.0.4.180717 | OEL6 - OEL7 | UEKR4 | 4.1.12-112.16.4 | 22810422, 28171094, 27463879 |
| 12.1.0.1 | OEL5 - OEL6 | UEKR1 | 2.6.32-100 | Base |
| 12.1.0.1 | OEL5 - OEL6 | UEKR2 | 2.6.39-100 | Base |
| 12.1.0.2.170718 | OEL5 - OEL6 | UEKR2 | 2.6.39-100 | Base |
| 12.1.0.2.170718 | OEL6 | UEKR3 | 3.8.13-13 | Base |
| 12.1.0.2.170718 | OEL7 | UEKR3 | 3.8.13-35 | 18321597 |
| 12.1.0.2.181016 | OEL6 - OEL7 | UEKR4 | 4.1.12-112.16.4 | 22810422, 27942938, 28171094, 27942938 |
| 12.1.0.2.190716 | OEL7 | UEKR5 | 4.14.35-1902 | 27494830 |
| 12.2.0.1 | OEL6 | UEKR2 | 2.6.39-100 | Base |
| 12.2.0.1 | OEL6 | UEKR3 | 3.8.13-13 | Base |
| 12.2.0.1.180717 | OEL6 | UEKR4 | 4.1.12-112.16.4 | 27463879, 28171094 |
| 12.2.0.1 | OEL7 | UEKR3 | 3.8.13-35 | Base |
| 12.2.0.1.180717 | OEL7 | UEKR4 | 4.1.12-112.16.4 | 27463879, 28171094, 27463879 |
| 12.2.0.181016 | OEL7 | UEKR5 | 4.14.35-1902 | 28069955 |
| 18.3.0.0 | OEL6 | UEKR2 | 2.6.39-100 | Base |
| 18.3.0.0 | OEL6 | UEKR3 | 3.8.13-13 | Base |
| 18.4.0.0.181016 | OEL6 | UEKR4 | 4.1.12-112.16.4 | 28069955 |
| 18.3.0.0 | OEL7 | UEKR3 | 3.8.13-35 | Base |
| 18.4.0.0.181016 | OEL7 | UEKR4 | 4.1.12-112.16.4 | 27463879, 28171094, 27463879 |
| 18.4.0.0.181016 | OEL7 | UEKR5 | 4.14.35-1902 | 27494830 |
| 19.3.0.0 | OEL7 | UEKR4 | 4.1.12-112.16.4 | Base |
| 19.4.190716 | OEL7 | UEKR5 | 4.14.35-1902 | 27494830 |
The example above on my test system is a fresh install of GI 19.3.0.0 on a OEL 7.8 system using UEKR5. Based on the list above we can tell that for UEKR5 support I need either 19.4.190716 or I need patch 27494830. Oracle provides many patches where the title contains “ACFS RU” that you can search for in the patches section of MOS. I decided to use this patch “MERGE ON ACFS RU 19.6.0.0.0 OF 28531803 30685278 (Patch 31055785)” which includes the following merge of patches:
Patch 31055785 : applied on Fri May 22 14:15:34 CDT 2020Unique Patch ID: 23448796Patch description: "ACFS Interim patch for 31055785" Created on 20 Mar 2020, 03:53:18 hrs PST8PDT Bugs fixed: 28531803, 30685278, <font color="#d52c1f">27494830</font>, 27917085, 28064731, 28293236, 28321248 28375150, 28553487, 28611527, 28687713, 28701011, 28740425, 28818513 28825772, 28844788, 28855761, 28860451, 28900212, 28951588, 28960047 28995524, 29001307, 29030106, 29031452, 29039918, 29115917, 29116692 29127489, 29127491, 29167352, 29173957, 29198743, 29229120, 29234059 29250565, 29264772, 29302070, 29313039, 29318169, 29338628, 29339084 29350729, 29363565, 29379932, 29391373, 29411007, 29417321, 29428155 29437701, 29482354, 29484738, 29520544, 29524859, 29527221, 29551699 29560075, 29586338, 29590813, 29604082, 29643818, 29704429, 29705711 29721120, 29760083, 29779338, 29791186, 29848987, 29851205, 29862693 29872187, 29893148, 29929003, 29929061, 29937236, 29941227, 29963428 30003321, 30032562, 30046061, 30051637, 30057972, 30076951, 30093449 30140896, 30179531, 30239575, 30251503, 30264950, 30269395, 30275174 30324590, 30363621, 30655657
Which you can see includes the required patch 27494830 (first row, 3rd column). Depending on your version of Oracle, and quarterly bundle patch there are many options of patches available. The quarterly bundle patches for GI generally contain an updated ACFS patch as well.
Details of the ASM Kernel Modules
In 19c, under the GRID_HOME is a directory called usm. On my test system under here I can find this:
$ ls $GRID_HOME/usm/install/Oracle/EL7UEK/x86_644.1.12/ 4.1.12-112.16.4/ 4.14.35-1902/
This is the location of the libraries used to compile the kernel modules for ACFS / AFD. As expected, there is a directory for each major kernel version that pretty much matches our list above. The command “
amcmd afd_configure -e” is used to install the AFD kernel modules. The AFD module use the Kernel Application Binary Interface and are installed under the /lib/modules location on your Linux system. As they are considered weak modules, they are placed in one location and then symbolically linked into each kernel specific directory when the kernel is updated.[root@srvr03 sbin]# ls -l /lib/modules/4.14.35-1902*/{extra,weak-updates}/oracle
/lib/modules/4.14.35-1902.0.9.el7uek.x86_64/extra/oracle:
total 9824
-rw-r--r-- 1 root root 10055682 Jun 5 13:16 oracleafd.ko
/lib/modules/4.14.35-1902.10.8.el7uek.x86_64/weak-updates/oracle:
total 0
lrwxrwxrwx 1 root root 69 Jun 5 13:17 oracleafd.ko -> /lib/modules/4.14.35-1902.0.9.el7uek.x86_64/extra/oracle/oracleafd.ko
/lib/modules/4.14.35-1902.11.3.1.el7uek.x86_64/weak-updates/oracle:
total 0
lrwxrwxrwx 1 root root 69 Jun 5 13:44 oracleafd.ko -> /lib/modules/4.14.35-1902.0.9.el7uek.x86_64/extra/oracle/oracleafd.ko
Here we can see that the primary module was installed under 4.14.35-1902.0.9. Then when the kernel 4.14.35-1902.11.3.1 was installed, a symbolic link was created to the original module.
The AFD module is normally loaded by a Linux RC script called afd (/etc/init.d/afd). You can verify it will be started at boot with the following command as the root user:
[root@srvr03 init.d]# chkconfig --list afd
Note: This output shows SysV services only and does not include native
systemd services. SysV configuration data might be overridden by native
systemd configuration.
If you want to list systemd services use 'systemctl list-unit-files'.
To see services enabled on particular target use
'systemctl list-dependencies [target]'.
afd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
Here you can see that the AFD module will attempt to be loaded at run levels 3 and 5. You can also check what version of the AFD module you have loaded with the following Linux command:
[oracle@srvr03 ~]$ modinfo oracleafdfilename: /lib/modules/4.14.35-1902.11.3.1.el7uek.x86_64/weak-updates/oracle/oracleafd.kolicense: Oracle Corporationdescription: ASM Filter Driverauthor: Oracle Corporationsrcversion: 533BB7E5866E52F63B9ACCBdepends: retpoline: Yname: oracleafdvermagic: 4.14.35-1902.0.9.el7uek.x86_64 SMP mod_unload modversions signat: PKCS#7signer: sig_key: sig_hashalgo: md4parm: oracleafd_use_logical_block_size:Non-zero value implies using logical sectorsize (int)parm: oracleafd_use_maxio_softlimits:Non-zero value implies using conservative maximum block io limits (int)
Using the lsmod command to see if it is loaded. If this command returns no lines, then it is not loaded.
[oracle@srvr03 ~]$ lsmod |grep oracleafdoracleafd 229376 0
Conclusion
Ok, so hopefully this has given you a clear picture as to what you need to plan for when using ASM Filter Driver (AFD) or ACFS on your OEL system with UEK. Once you have a base version of kernel and GI you should have little issues unless you jump UEK channels or do a GI major version change. As many people are eyeing up 19c upgrades, hopefully the above information is helpful to prepare your Linux OS for the change. For myself what started as a somewhat simple post about kernel drivers turned into a pretty long delve into the workings of the ASM filter driver. Please leave your comments and questions.
SUBMIT YOUR COMMENT