In this cloud-era it’s almost weird to explore stuff on-premises, but I did….  In this case a slightly Higher Availibility configuration of Oracle Enterprise Manager (OEM) that doesn’t  cost anything, is not disruptive to the existing configuration and gives a tiny bit more confidence during patching of OEM : Always On Monitoring (AOM).  Maybe a bit underrated functionality – within Oracle .

When Oracle Management Server (OMS) is down for patching, AOM is capable of mailing you when a critical target is down. It’s a limited functionality, but can be quite useful in some cases. Some details has changed since the start in 2016, when DBIservices wrote about it. At the end of the blogpost my findings.

What have I done in this blogpost

1. Using an existing OEM 13.3 installation – Vagrant build, by Oracle-base.

2. Create a pluggable database for the AOM repository and one for testing purposes, and registered them in OEM.

3. Install / unzip the AOM-software, and create the AOM repository

4. Configure OAM with down-time contacts, critical targets and synchronize AOM with OEM.

5. Run some little tests: bringing down the OMS, the test-database, the agent for simulating a node outage.

6. Some findings at the end

 

1. Using an existing OEM 13.3 installation – Vagrant build.

The following configuration is my base: OEM 13.3 with a repository on the same server.

Connected to this database:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 – Production

Version 19.3.0.0.0

SQL> show pdbs

CON_ID CON_NAME OPEN MODE RESTRICTED

———- —————————— ———- ———-

2 PDB$SEED READ ONLY NO

3 EMREP READ WRITE NO

image

2. Create a pluggable database for the AOM repository and one for testing purposes, and registered them in OEM.

Create a seperate PDB OAMREP (sorry, should be AOMREP….) for the Always On Monitoring repository.

SQL> Create pluggable database oamrep admin user oamsys identified by oracle;

Pluggable database created.

SQL> show pdbs

CON_ID CON_NAME OPEN MODE RESTRICTED

———- —————————— ———- ———-

2 PDB$SEED READ ONLY NO

3 EMREP READ WRITE NO

4 OAMREP MOUNTED

SQL> alter session set container=oamrep;

Session altered.

SQL> alter pluggable database open read write;

Pluggable database altered.

Create a seperate PDB TESTDB for test-purposes later on.

SQL> Create pluggable database testdb admin user testdba identified by oracle;

Pluggable database created.

SQL> alter pluggable database testdb open read write;

Pluggable database altered.

SQL> alter pluggable database oamrep open read write;

Pluggable database altered.

SQL> show pdbs

CON_ID CON_NAME OPEN MODE RESTRICTED

———- —————————— ———- ———-

2 PDB$SEED READ ONLY NO

3 EMREP READ WRITE NO

4 OAMREP READ WRITE NO

5 TESTDB READ WRITE NO

Here’s what it looks like

image

Put the databases in OEM:

clip_image002

 

Let them automatically start when CDB starts:

alter pluggable database all save state;

According to the documentation, trying to set the parameter:

SQL> ALTER SYSTEM SET OPTIMIZER_ADAPTIVE_FEATURES=FALSE;

ALTER SYSTEM SET OPTIMIZER_ADAPTIVE_FEATURES=FALSE

*

ERROR at line 1:

ORA-25138: OPTIMIZER_ADAPTIVE_FEATURES initialization parameter has been made

obsolete

O.k. this has changed apparently, I’m fine.

Characterset should be AL32UTF8

SQL> SELECT * from nls_database_parameters where parameter=’NLS_CHARACTERSET’;

NLS_CHARACTERSET

AL32UTF8

Put OAMREP for convenience in TNSNAMES.ORA

Create user EMS in OAMREP

create user ems identified by ems;

grant CREATE SESSION, ALTER SESSION, CREATE DATABASE LINK, CREATE MATERIALIZED VIEW, CREATE PROCEDURE, CREATE PUBLIC SYNONYM, CREATE ROLE, CREATE SEQUENCE, CREATE SYNONYM, CREATE TABLE, CREATE TRIGGER, CREATE TYPE, CREATE VIEW, UNLIMITED TABLESPACE, SELECT ANY DICTIONARY to ems;

grant EXECUTE ON SYS.DBMS_CRYPTO to ems;

grant EXECUTE ON SYS.DBMS_AQADM to ems;

grant EXECUTE ON SYS.DBMS_AQ to ems;

grant EXECUTE ON SYS.DBMS_AQIN to ems;

grant EXECUTE on SYS.DBMS_LOCK to ems;

grant EXECUTE ON SYS.DBMS_SCHEDULER to ems;

grant create job to ems;

3. Install / unzip the AOM-software, and create the AOM repository

Unzip the binaries:

[oracle@ol7-emcc ems]$ pwd

/u01/app/oracle/middleware/sysman/ems

[oracle@ol7-emcc ems]$ ls

ems_13.3.0.0.0.zip

[oracle@ol7-emcc ems]$ unzip ems_13.3.0.0.0.zip

Archive: ems_13.3.0.0.0.zip

creating: ems/

creating: ems/lib/

inflating: ems/lib/com.bea.core.apache.commons.lang_1.0.0.0_2-6.jar

inflating: ems/lib/com.oracle.http_client.http_client_12.1.3.jar

inflating: ems/lib/jackson-core-asl-1.9.2.jar

etc…

Put the emkey in the OEM-repository:

[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emctl config emkey -copy_to_repos

Oracle Enterprise Manager Cloud Control 13c Release 3

Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.

Enter Enterprise Manager Root (SYSMAN) Password :

The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.

After the required operation has been completed, secure the EMKey by running “emctl config emkey -remove_from_repos”.

Go to the script-directory:

cd /u01/app/oracle/middleware/sysman/ems/ems/scripts

Should be Java to run emsca for creating the AOM repository:

[oracle@ol7-emcc scripts]$ ./emsca -createEmsDbUser=false

Either JAVA_HOME environment variable is not set or an incorrect Java version is used. To run emsca, use Java 1.7

Trying seperate settings

export PATH=$PATH:/u01/app/oracle/middleware/oracle_common/jdk/bin

[oracle@ol7-emcc scripts]$ java -version

java version “1.7.0_171”

Java(TM) SE Runtime Environment (build 1.7.0_171-b11)

Java HotSpot(TM) 64-Bit Server VM (build 24.171-b11, mixed mode)

export JAVA_HOME=/u01/app/oracle/product/19.0.0/dbhome_1/jdk/jre/bin

export PATH=$PATH:/u01/app/oracle/product/19.0.0/dbhome_1/jdk/jre/bin

Nope, this won’t fly. Then a seperate install – yum install java. This is working!

Creating OAM repository:

[oracle@ol7-emcc scripts]$ ./emsca -createEmsDbUser=false

Oracle Enterprise Manager Cloud Control 13c Release 3

Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.

—————————————————————

Always-On Monitoring Repository Connection String : localhost:1521:oamrep

Always-On Monitoring Repository Username [ems] :

Always-On Monitoring Repository Password [ems] :

Always-On Monitoring Repository user “ems” has already been created

Enterprise Manager Repository Connection String : localhost:1521:emrep

Enterprise Manager Repository Username : sysman

Enterprise Manager Repository Password :

Agent Registration Password :

Keystore for host ol7-emcc.localdomain created successfully.

Connecting to Always-On Monitoring Repository.

Creating Always-On Monitoring Repository schema

Creating repository storage for Targets data.

Creating repository storage for Alerts and Availability data.

Creating repository storage for Notification Metadata data.

Creating repository storage for Target Metric Metadata data.

Registering Always-On Monitoring instance

Always-On Monitoring Upload URL: https://ol7-emcc.localdomain:8081/upload

Remove emkey from OEM-repository:

[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emctl config emkey -remove_from_repos

Oracle Enterprise Manager Cloud Control 13c Release 3

Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.

Enter Enterprise Manager Root (SYSMAN) Password :

The EMKey has been removed from the Management Repository.

The status:

image

4. Configure OAM with down-time contacts, critical targets and synchronize AOM with OEM.

Setting the mail property for OAM

[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli set_oms_property -property_name=’oracle.sysman.core.events.ems.downtimeContact’ -property_value=’<mail-address>’

Error: The command name “set_oms_property” is not a recognized command.

Run the “help” command for a list of recognized commands.

You may also need to run the “sync” command to synchronize with the current OMS.

How could I forget…

[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli login -username=sysman

Enter password

Login successful

[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli sync

Synchronized successfully

Trying to set the mail-property again:

[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli set_oms_property -property_name=’oracle.sysman.core.events.ems.downtimeContact’ -property_value=’<mail-address>’

Property “oracle.sysman.core.events.ems.downtimeContact” has been successfully set to value “<mail-address>” on all Management Servers.

Server restart is not required for the property change to get reflected

Define downtime contact from the target property, tried it first with the examples I saw.

[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli set_target_property_value -property_records=”TESTDB:oracle_database:Downtime Contact:<mail-address>”

Error: Invalid target in record: TESTDB:oracle_database:Downtime Contact

Allright, it’s a pluggable database, need another name and type. What choices I have:

[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli get_targets -target=oracle_pdb;

Status Status Target Type Target Name

ID

1 Up oracle_pdb emcdb_TESTDB

1 Up oracle_pdb emcdb_CDBROOT

1 Up oracle_pdb emcdb_EMREP

1 Up oracle_pdb emcdb_OAMREP

Another attempt:

[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/bin/emcli set_target_property_value -property_records=”emcdb_TESTDB:oracle_pdb:Downtime Contact:<mail-address>”

Properties updated successfully

Synchronizing for the first time:

[oracle@ol7-emcc scripts]$ pwd

/u01/app/oracle/middleware/sysman/ems/ems/scripts

[oracle@ol7-emcc scripts]$ ./emsctl sync

Oracle Enterprise Manager Cloud Control 13c Release 3

Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.

——————————————————————

Connecting to Always-On Monitoring Repository.

Starting synchronization with Enterprise Manager.

Synchronizing with Enterprise Manager repository: sysman@localhost:1521:emrep

Synchronizing Targets data.

Synchronizing Alerts and Availability data.

Synchronizing Notification Metadata data.

Synchronizing Target Metric Metadata data.

Synchronization complete at : Wed Apr 22 11:43:01 UTC 2020

Merge AOM with OEM by setting the AOM string in the agents:

/u01/app/oracle/middleware/bin/emctl set property -name “oracle.sysman.core.events.ems.emsURL” -value “https://ol7-emcc.localdomain:8081/upload” -sysman_pwd Welcome1

Oracle Enterprise Manager Cloud Control 13c Release 3

Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.

Property oracle.sysman.core.events.ems.emsURL has been set to value https://ol7-emcc.localdomain:8081/upload for all Management Servers

OMS restart is not required to reflect the new property value

Check URL in agent:

[oracle@ol7-emcc config]$ pwd

/u01/app/oracle/agent/agent_inst/sysman/config

[oracle@ol7-emcc config]$ grep 8081 emd.properties

EMS_URL=https:// ol7-emcc.localdomain:8081/upload

You can always start and stop OAM individual:

[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/sysman/ems/ems/scripts/emsctl start

Oracle Enterprise Manager Cloud Control 13c Release 3

Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.

——————————————————————

Pinging Always-On Monitoring…

Always-On Monitoring is not running.

Starting Always-On Monitoring.

Waiting for process to start

Retrying…

Retrying…

Notifications Enabled : true

Total Downtime Contacts Configured : 1

Always-On Monitoring is up.

Enable notifications:

[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/sysman/ems/ems/scripts/emsctl enable_notification

Oracle Enterprise Manager Cloud Control 13c Release 3

Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.

——————————————————————

Notifications have been enabled. There are downtime contacts configured.

Connecting to Always-On Monitoring Repository.

Starting synchronization with Enterprise Manager.

Synchronizing with Enterprise Manager repository: sysman@localhost:1521:emrep

Synchronizing Targets data.

Synchronizing Alerts and Availability data.

Synchronizing Notification Metadata data.

Synchronizing Target Metric Metadata data.

Synchronization complete at : Wed Apr 29 09:55:56 UTC 2020

Are the targets in de AOM-repository (user EMS):

clip_image004

5. Run some little tests: bringing down the OMS, the test-database, the agent for simulating a node outage.

Let’s test it by stopping the oms.

[oracle@ol7-emcc scripts]$ $OMS_HOME/bin/emctl stop oms –all. And we’ve got mail.

clip_image006

Other test: stopping the TESTDB pluggable database

SQL> alter pluggable database testdb close immediate;

Result: I’ve got (2) mails, from 2 different ‘event_types’.

clip_image008

clip_image010

Let’s open the PDB testdb :

SQL> alter pluggable database testdb open read write;

Pluggable database altered.

And I’ve got 2 mails again:

– 11:13 AM : emcdb_TESTDB – The pluggable database is in OPEN state.

– 11:13 AM : emcdb_TESTDB – The pluggable database is up.

Let’s start the OMS.

$OMS_HOME/bin/emctl start oms

Let’s see if OMS knows what happened in the time he was down.

clip_image012

That seems to be the case (don’t mind the different time-lines. There’s a two hour difference between my Linux VM and my laptop, so 12:39 is for OEM 10:39) .

Suppose we are closing the agent:

[oracle@ol7-emcc scripts]$ $AGENT_HOME/bin/emctl stop agent

Oracle Enterprise Manager Cloud Control 13c Release 3

Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.

Stopping agent … stopped.

Status OAM gives no clue of what is happening.

[oracle@ol7-emcc scripts]$ /u01/app/oracle/middleware/sysman/ems/ems/scripts/emsctl status

Oracle Enterprise Manager Cloud Control 13c Release 3

Copyright (c) 2016, 2017, Oracle Corporation. All rights reserved.

——————————————————————

Always-On Monitoring Version : 13.3.0.0.0

Always-On Monitoring Home : /u01/app/oracle/middleware/sysman/ems/ems

Started At : April 29, 2020 1:36:34 PM UTC

Last Repository Sync : April 29, 2020 6:26:15 PM UTC

Upload URL : https://ol7-emcc.localdomain:8081/upload

Always-On Monitoring Process ID : 26450

Always-On Monitoring Repository : localhost:1521:oamrep

Enterprise Manager Repository : localhost:1521:emrep

Notifications Enabled : true

Total Downtime Contacts Configured : 1

Starting the agent gives us two mail: one to mention that the agent is down (!), and one that he’s up.

clip_image014

clip_image016

6. Some findings

– Installation is not that hard, but requires a few steps in the right order.

– Installation / configuration is not ‘disruptive’ of your current OEM-configuration.

– It’s a limited functionality – in fact the  down-time of a target – and only use of mail-communication is possible. But while it’s not disruptive, it could be worthy when you don’t want to be ‘blind’ while patching, and you’ve got a few very critical targets.

– AOM is not very ‘hot’ within OEM and apparently not on the developer’s  priorities-list.

– Outage of a node / agent will not be noticed by AOM, although you can see if the agents are communicating with AOM through the command ‘emsctl list_agents’. Must be a way to use this.

– It’s all commandline. To be hated, to be loved.

– You need an extra hole in your firewall for the communication of the agents with the AOM software.

– Configuring a lot of agents should be one command, haven’t tested this.

– Made a mistake in configuring the URL in the agents. No mail, and no clue where I made the mistake. Always check if the agents are communicating with the AOM software with ‘emsctl list_agents’.

 

Resources

DBI-services: https://blog.dbi-services.com/always-on-monitoring-em-13c/

Oracle blog: https://docs.oracle.com/cd/E63000_01/EMADM/em_mon_svc.htm#EMADM15617

Oracle-base OEM Vagrant build: https://oracle-base.com/blog/2019/06/19/enterprise-manager-cloud-control-13-3-vagrant-build/