In this cloud-era it’s almost weird to explore stuff on-premises, but I did…. In this case a slightly Higher Availibility configuration of Oracle Enterprise Manager (OEM) that doesn’t cost anything, is not disruptive to the existing configuration and gives a tiny bit more confidence during patching of OEM : Always On Monitoring (AOM). Maybe a bit underrated functionality – within Oracle .
When Oracle Management Server (OMS) is down for patching, AOM is capable of mailing you when a critical target is down. It’s a limited functionality, but can be quite useful in some cases. Some details has changed since the start in 2016, when DBIservices wrote about it. At the end of the blogpost my findings.
What have I done in this blogpost
1. Using an existing OEM 13.3 installation – Vagrant build, by Oracle-base.
2. Create a pluggable database for the AOM repository and one for testing purposes, and registered them in OEM.
3. Install / unzip the AOM-software, and create the AOM repository
4. Configure OAM with down-time contacts, critical targets and synchronize AOM with OEM.
5. Run some little tests: bringing down the OMS, the test-database, the agent for simulating a node outage.
6. Some findings at the end
1. Using an existing OEM 13.3 installation – Vagrant build.
The following configuration is my base: OEM 13.3 with a repository on the same server.
Connected to this database:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 – Production
Version 19.3.0.0.0
SQL> show pdbs
CON_ID CON_NAME OPEN MODE RESTRICTED
———- —————————— ———- ———-
2 PDB$SEED READ ONLY NO
3 EMREP READ WRITE NO
2. Create a pluggable database for the AOM repository and one for testing purposes, and registered them in OEM.
Create a seperate PDB OAMREP (sorry, should be AOMREP….) for the Always On Monitoring repository.
SQL> Create pluggable database oamrep admin user oamsys identified by oracle;
Pluggable database created.
SQL> show pdbs
CON_ID CON_NAME OPEN MODE RESTRICTED
———- —————————— ———- ———-
2 PDB$SEED READ ONLY NO
3 EMREP READ WRITE NO
4 OAMREP MOUNTED
SQL> alter session set container=oamrep;
Session altered.
SQL> alter pluggable database open read write;
Pluggable database altered.
Create a seperate PDB TESTDB for test-purposes later on.
SQL> Create pluggable database testdb admin user testdba identified by oracle;
Pluggable database created.
SQL> alter pluggable database testdb open read write;
Pluggable database altered.
SQL> alter pluggable database oamrep open read write;
Pluggable database altered.
SQL> show pdbs
CON_ID CON_NAME OPEN MODE RESTRICTED
———- —————————— ———- ———-
2 PDB$SEED READ ONLY NO
3 EMREP READ WRITE NO
4 OAMREP READ WRITE NO
5 TESTDB READ WRITE NO
Here’s what it looks like
Put the databases in OEM:
Let them automatically start when CDB starts:
alter pluggable database all save state;
According to the documentation, trying to set the parameter:
SQL> ALTER SYSTEM SET OPTIMIZER_ADAPTIVE_FEATURES=FALSE;
ALTER SYSTEM SET OPTIMIZER_ADAPTIVE_FEATURES=FALSE
*
ERROR at line 1:
ORA-25138: OPTIMIZER_ADAPTIVE_FEATURES initialization parameter has been made
obsolete
O.k. this has changed apparently, I’m fine.
Characterset should be AL32UTF8
SQL> SELECT * from nls_database_parameters where parameter=’NLS_CHARACTERSET’;
NLS_CHARACTERSET
AL32UTF8
Put OAMREP for convenience in TNSNAMES.ORA
Create user EMS in OAMREP
create user ems identified by ems;
grant CREATE SESSION, ALTER SESSION, CREATE DATABASE LINK, CREATE MATERIALIZED VIEW, CREATE PROCEDURE, CREATE PUBLIC SYNONYM, CREATE ROLE, CREATE SEQUENCE, CREATE SYNONYM, CREATE TABLE, CREATE TRIGGER, CREATE TYPE, CREATE VIEW, UNLIMITED TABLESPACE, SELECT ANY DICTIONARY to ems;
grant EXECUTE ON SYS.DBMS_CRYPTO to ems;
grant EXECUTE ON SYS.DBMS_AQADM to ems;
grant EXECUTE ON SYS.DBMS_AQ to ems;
grant EXECUTE ON SYS.DBMS_AQIN to ems;
grant EXECUTE on SYS.DBMS_LOCK to ems;
grant EXECUTE ON SYS.DBMS_SCHEDULER to ems;
grant create job to ems;
3. Install / unzip the AOM-software, and create the AOM repository
Unzip the binaries:
[oracle@ol7-emcc ems]$ pwd
/u01/app/oracle/middleware/sysman/ems
[oracle@ol7-emcc ems]$ ls
ems_13.3.0.0.0.zip
[oracle@ol7-emcc ems]$ unzip ems_13.3.0.0.0.zip
Archive: ems_13.3.0.0.0.zip
creating: ems/
creating: ems/lib/
inflating: ems/lib/com.bea.core.apache.commons.lang_1.0.0.0_2-6.jar
inflating: ems/lib/com.oracle.http_client.http_client_12.1.3.jar
inflating: ems/lib/jackson-core-asl-1.9.2.jar
etc…
Put the emkey in the OEM-repository:
[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emctl config emkey -copy_to_repos
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.
After the required operation has been completed, secure the EMKey by running “emctl config emkey -remove_from_repos”.
Go to the script-directory:
cd /u01/app/oracle/middleware/sysman/ems/ems/scripts
Should be Java to run emsca for creating the AOM repository:
[oracle@ol7-emcc scripts]$ ./emsca -createEmsDbUser=false
Either JAVA_HOME environment variable is not set or an incorrect Java version is used. To run emsca, use Java 1.7
Trying seperate settings
export PATH=$PATH:/u01/app/oracle/middleware/oracle_common/jdk/bin
[oracle@ol7-emcc scripts]$ java -version
java version “1.7.0_171”
Java(TM) SE Runtime Environment (build 1.7.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 24.171-b11, mixed mode)
export JAVA_HOME=/u01/app/oracle/product/19.0.0/dbhome_1/jdk/jre/bin
export PATH=$PATH:/u01/app/oracle/product/19.0.0/dbhome_1/jdk/jre/bin
Nope, this won’t fly. Then a seperate install – yum install java. This is working!
Creating OAM repository:
[oracle@ol7-emcc scripts]$ ./emsca -createEmsDbUser=false
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.
—————————————————————
Always-On Monitoring Repository Connection String : localhost:1521:oamrep
Always-On Monitoring Repository Username [ems] :
Always-On Monitoring Repository Password [ems] :
Always-On Monitoring Repository user “ems” has already been created
Enterprise Manager Repository Connection String : localhost:1521:emrep
Enterprise Manager Repository Username : sysman
Enterprise Manager Repository Password :
Agent Registration Password :
Keystore for host ol7-emcc.localdomain created successfully.
Connecting to Always-On Monitoring Repository.
Creating Always-On Monitoring Repository schema
Creating repository storage for Targets data.
Creating repository storage for Alerts and Availability data.
Creating repository storage for Notification Metadata data.
Creating repository storage for Target Metric Metadata data.
Registering Always-On Monitoring instance
Always-On Monitoring Upload URL: https://ol7-emcc.localdomain:8081/upload
Remove emkey from OEM-repository:
[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emctl config emkey -remove_from_repos
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey has been removed from the Management Repository.
The status:
4. Configure OAM with down-time contacts, critical targets and synchronize AOM with OEM.
Setting the mail property for OAM
[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli set_oms_property -property_name=’oracle.sysman.core.events.ems.downtimeContact’ -property_value=’<mail-address>’
Error: The command name “set_oms_property” is not a recognized command.
Run the “help” command for a list of recognized commands.
You may also need to run the “sync” command to synchronize with the current OMS.
How could I forget…
[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli login -username=sysman
Enter password
Login successful
[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli sync
Synchronized successfully
Trying to set the mail-property again:
[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli set_oms_property -property_name=’oracle.sysman.core.events.ems.downtimeContact’ -property_value=’<mail-address>’
Property “oracle.sysman.core.events.ems.downtimeContact” has been successfully set to value “<mail-address>” on all Management Servers.
Server restart is not required for the property change to get reflected
Define downtime contact from the target property, tried it first with the examples I saw.
[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli set_target_property_value -property_records=”TESTDB:oracle_database:Downtime Contact:<mail-address>”
Error: Invalid target in record: TESTDB:oracle_database:Downtime Contact
Allright, it’s a pluggable database, need another name and type. What choices I have:
[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli get_targets -target=oracle_pdb;
Status Status Target Type Target Name
ID
1 Up oracle_pdb emcdb_TESTDB
1 Up oracle_pdb emcdb_CDBROOT
1 Up oracle_pdb emcdb_EMREP
1 Up oracle_pdb emcdb_OAMREP
Another attempt:
[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli set_target_property_value -property_records=”emcdb_TESTDB:oracle_pdb:Downtime Contact:<mail-address>”
Properties updated successfully
Synchronizing for the first time:
[oracle@ol7-emcc scripts]$ pwd
/u01/app/oracle/middleware/sysman/ems/ems/scripts
[oracle@ol7-emcc scripts]$ ./emsctl sync
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.
——————————————————————
Connecting to Always-On Monitoring Repository.
Starting synchronization with Enterprise Manager.
Synchronizing with Enterprise Manager repository: sysman@localhost:1521:emrep
Synchronizing Targets data.
Synchronizing Alerts and Availability data.
Synchronizing Notification Metadata data.
Synchronizing Target Metric Metadata data.
Synchronization complete at : Wed Apr 22 11:43:01 UTC 2020
Merge AOM with OEM by setting the AOM string in the agents:
/u01/app/oracle/middleware/bin/emctl set property -name “oracle.sysman.core.events.ems.emsURL” -value “https://ol7-emcc.localdomain:8081/upload” -sysman_pwd Welcome1
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
Property oracle.sysman.core.events.ems.emsURL has been set to value https://ol7-emcc.localdomain:8081/upload for all Management Servers
OMS restart is not required to reflect the new property value
Check URL in agent:
[oracle@ol7-emcc config]$ pwd
/u01/app/oracle/agent/agent_inst/sysman/config
[oracle@ol7-emcc config]$ grep 8081 emd.properties
EMS_URL=https:// ol7-emcc.localdomain:8081/upload
You can always start and stop OAM individual:
[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/sysman/ems/ems/scripts/emsctl start
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.
——————————————————————
Pinging Always-On Monitoring…
Always-On Monitoring is not running.
Starting Always-On Monitoring.
Waiting for process to start
Retrying…
Retrying…
Notifications Enabled : true
Total Downtime Contacts Configured : 1
Always-On Monitoring is up.
Enable notifications:
[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/sysman/ems/ems/scripts/emsctl enable_notification
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.
——————————————————————
Notifications have been enabled. There are downtime contacts configured.
Connecting to Always-On Monitoring Repository.
Starting synchronization with Enterprise Manager.
Synchronizing with Enterprise Manager repository: sysman@localhost:1521:emrep
Synchronizing Targets data.
Synchronizing Alerts and Availability data.
Synchronizing Notification Metadata data.
Synchronizing Target Metric Metadata data.
Synchronization complete at : Wed Apr 29 09:55:56 UTC 2020
Are the targets in de AOM-repository (user EMS):
5. Run some little tests: bringing down the OMS, the test-database, the agent for simulating a node outage.
Let’s test it by stopping the oms.
[oracle@ol7-emcc scripts]$ $OMS_HOME/bin/emctl stop oms –all. And we’ve got mail.
Other test: stopping the TESTDB pluggable database
SQL> alter pluggable database testdb close immediate;
Result: I’ve got (2) mails, from 2 different ‘event_types’.
Let’s open the PDB testdb :
SQL> alter pluggable database testdb open read write;
Pluggable database altered.
And I’ve got 2 mails again:
– 11:13 AM : emcdb_TESTDB – The pluggable database is in OPEN state.
– 11:13 AM : emcdb_TESTDB – The pluggable database is up.
Let’s start the OMS.
$OMS_HOME/bin/emctl start oms
Let’s see if OMS knows what happened in the time he was down.
That seems to be the case (don’t mind the different time-lines. There’s a two hour difference between my Linux VM and my laptop, so 12:39 is for OEM 10:39) .
Suppose we are closing the agent:
[oracle@ol7-emcc scripts]$ $AGENT_HOME/bin/emctl stop agent
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
Stopping agent … stopped.
Status OAM gives no clue of what is happening.
[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/sysman/ems/ems/scripts/emsctl status
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.
——————————————————————
Always-On Monitoring Version : 13.3.0.0.0
Always-On Monitoring Home : /u01/app/oracle/middleware/sysman/ems/ems
Started At : April 29, 2020 1:36:34 PM UTC
Last Repository Sync : April 29, 2020 6:26:15 PM UTC
Upload URL : https://ol7-emcc.localdomain:8081/upload
Always-On Monitoring Process ID : 26450
Always-On Monitoring Repository : localhost:1521:oamrep
Enterprise Manager Repository : localhost:1521:emrep
Notifications Enabled : true
Total Downtime Contacts Configured : 1
Starting the agent gives us two mail: one to mention that the agent is down (!), and one that he’s up.
6. Some findings
– Installation is not that hard, but requires a few steps in the right order.
– Installation / configuration is not ‘disruptive’ of your current OEM-configuration.
– It’s a limited functionality – in fact the down-time of a target – and only use of mail-communication is possible. But while it’s not disruptive, it could be worthy when you don’t want to be ‘blind’ while patching, and you’ve got a few very critical targets.
– AOM is not very ‘hot’ within OEM and apparently not on the developer’s priorities-list.
– Outage of a node / agent will not be noticed by AOM, although you can see if the agents are communicating with AOM through the command ‘emsctl list_agents’. Must be a way to use this.
– It’s all commandline. To be hated, to be loved.
– You need an extra hole in your firewall for the communication of the agents with the AOM software.
– Configuring a lot of agents should be one command, haven’t tested this.
– Made a mistake in configuring the URL in the agents. No mail, and no clue where I made the mistake. Always check if the agents are communicating with the AOM software with ‘emsctl list_agents’.
Resources
DBI-services: https://blog.dbi-services.com/always-on-monitoring-em-13c/
Oracle blog: https://docs.oracle.com/cd/E63000_01/EMADM/em_mon_svc.htm#EMADM15617
Oracle-base OEM Vagrant build: https://oracle-base.com/blog/2019/06/19/enterprise-manager-cloud-control-13-3-vagrant-build/
Leave A Comment