OEM 13.5 - the missing CMAN metrics (confusion of languages)

 As my journey with Oracle Connection Manager (cman) and Enterprise Manger (OEM) continues, I tried to get used to the metrics Oracle provides for cman out of the box. Even I can not change the Metric Thresholds for cman (Note:2590064.1), at least I'd like to see any telemetry. 

But unfortunately OEM always shows zero. Also other Metrics like the general status shows a lot of empty fields, and even the "Serviced Databases" view is empty. Of course I opened a SR at My Oracle Support (MOS), but the effect was only some back & forth of emctl, cmctl commands and their logfiles. Some basic Tests I provided showed at least the problem might be somewhere on the agents side. 

CMAN shows some active connections:

cmctl show connections -c cman_xxx


CMCTL for Linux: Version 21.0.0.0.0 - Production on 03-MAR-2023 10:21:51

Copyright (c) 1996, 2021, Oracle. All rights reserved.

Current instance cman_xxx is already started
Connecting to (DESCRIPTION=(address=(protocol=tcp)(host=xxx)(port=1521)))
Number of connections: 24.
The command completed successfully.

and also the cmanload.pl script provided some values:

export CMAN_ORACLE_HOME=$ORACLE_HOME
export CMAN_NAME=cman_xxx
export CMAN_ORA_DIR=$TNS_ADMIN
export CMAN_MACHINE=xxx
export CMAN_PROTOCOL="TCP"
export CMAN_PORT="1521"
unset CMAN_PASSWORD

cd $AGENT_HOME/agent_13.5.0.0.0/plugins/oracle.sysman.db.agent.plugin_13.5.1.0.0/scripts
perl cmanload.pl
em_result=1270|0|24

but the agent is not capable of getting these informations properly: 

oracle@xxx:~ $ $AGENT_HOME/agent_inst/bin/emctl getmetric agent cman_xxx,oracle_cman,Load
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation. All rights reserved.
estConnsTotal,refConnsTotal,activeConnsTotal,estConns,refConns
0,0,0,0,0

At least I'm quite confident the problem is somewhere on the agents side, not in OEM or its repository. 

Unfortunately the script cmanload.pl is poorly instrumented, so even setting 
emctl setproperty agent -name 'EMAGENT_PERL_TRACE_LEVEL' -value 'DEBUG', reloading the agent and check emagent_perl.trc did not show any useful insight. 

In my frustration, I suggested to edit cmanload.pl and add some debug code, and the reply was encouraging:

yes we should do that .do you think of anything ? 

Without discussing the whole cmanload.pl, I only show some relevant lines here and the debug line I've added: 

...
my $executable = $ENV{CMAN_ORACLE_HOME} . "/bin/cmctl";
...
  $command = "$executable show gateways -c $name";
...
  $result = executeCmd($command);
...
  my @info = split( /\n/, $result );
...
foreach $line (@info)
{
  EMAGENT_PERL_DEBUG("line: $line");
...
EMAGENT_PERL_DEBUG("em_result=$totalEstablished|$totalRefused|$totalActiveConnections");
...

With this extra line I get everything cmctl show gateways  provides in the tracefile:

cmanload.pl: 2023-03-14 14:23:48,865: DEBUG:  line:  CMCTL for Linux: Version 21.0.0.0.0 - Production on 14-MRZ-2023 14:23:48
cmanload.pl: 2023-03-14 14:23:48,866: DEBUG:  line:  
cmanload.pl: 2023-03-14 14:23:48,866: DEBUG:  line:  Copyright (c) 1996, 2021, Oracle.  All rights reserved.
cmanload.pl: 2023-03-14 14:23:48,866: DEBUG:  line:  
cmanload.pl: 2023-03-14 14:23:48,866: DEBUG:  line:  Aktuelle Instanz cman_xxx ist schon gestartet
cmanload.pl: 2023-03-14 14:23:48,866: DEBUG:  line:  Anmeldung bei (DESCRIPTION=(address=(protocol=tcp)(host=xxx)(port=1521)))
cmanload.pl: 2023-03-14 14:23:48,866: DEBUG:  line:  
cmanload.pl: 2023-03-14 14:23:48,866: DEBUG:  line:  Gateway-ID                     0
cmanload.pl: 2023-03-14 14:23:48,866: DEBUG:  line:  Gateway-Status                  READY
cmanload.pl: 2023-03-14 14:23:48,866: DEBUG:  line:  Anzahl von aktiven Verbindungen   0
cmanload.pl: 2023-03-14 14:23:48,866: DEBUG:  line:  Spitze von aktiven Verbindungen        5
cmanload.pl: 2023-03-14 14:23:48,866: DEBUG:  line:  Gesamte Verbindungen              17
cmanload.pl: 2023-03-14 14:23:48,866: DEBUG:  line:  Gesamte abgelehnte Verbindungen      0
cmanload.pl: 2023-03-14 14:23:48,866: DEBUG:  line:  Empfangene eingehende Byte              128214
cmanload.pl: 2023-03-14 14:23:48,866: DEBUG:  line:  Empfangene ausgehende Byte              337359
cmanload.pl: 2023-03-14 14:23:48,866: DEBUG:  line:  Gesendete eingehende Byte                  337359
cmanload.pl: 2023-03-14 14:23:48,866: DEBUG:  line:  Gesendete ausgehende Byte                 128214
cmanload.pl: 2023-03-14 14:23:48,867: DEBUG:  line:  
cmanload.pl: 2023-03-14 14:23:48,867: DEBUG:  line:  Gateway-ID                     1
cmanload.pl: 2023-03-14 14:23:48,867: DEBUG:  line:  Gateway-Status                  READY
cmanload.pl: 2023-03-14 14:23:48,867: DEBUG:  line:  Anzahl von aktiven Verbindungen   0
cmanload.pl: 2023-03-14 14:23:48,867: DEBUG:  line:  Spitze von aktiven Verbindungen        6
cmanload.pl: 2023-03-14 14:23:48,867: DEBUG:  line:  Gesamte Verbindungen              19
cmanload.pl: 2023-03-14 14:23:48,867: DEBUG:  line:  Gesamte abgelehnte Verbindungen      0
cmanload.pl: 2023-03-14 14:23:48,867: DEBUG:  line:  Empfangene eingehende Byte              260851
cmanload.pl: 2023-03-14 14:23:48,867: DEBUG:  line:  Empfangene ausgehende Byte              832901
cmanload.pl: 2023-03-14 14:23:48,867: DEBUG:  line:  Gesendete eingehende Byte                  832901
cmanload.pl: 2023-03-14 14:23:48,867: DEBUG:  line:  Gesendete ausgehende Byte                 260851
cmanload.pl: 2023-03-14 14:23:48,867: DEBUG:  line:  
cmanload.pl: 2023-03-14 14:23:48,867: DEBUG:  line:  
cmanload.pl: 2023-03-14 14:23:48,867: DEBUG:  line:  Übersicht:
cmanload.pl: 2023-03-14 14:23:48,867: DEBUG:  line:  
cmanload.pl: 2023-03-14 14:23:48,867: DEBUG:  line:  Aktive Verbindungen gesamt              0
cmanload.pl: 2023-03-14 14:23:48,867: DEBUG:  line:  Aktive Spitzenverbindungen gesamt          11
cmanload.pl: 2023-03-14 14:23:48,868: DEBUG:  line:  Verbindungen gesamt              36
cmanload.pl: 2023-03-14 14:23:48,868: DEBUG:  line:  Abgelehnte Verbindungen gesamt      0
cmanload.pl: 2023-03-14 14:23:48,868: DEBUG:  line:  Empfangene eingehende Byte gesamt        389065
cmanload.pl: 2023-03-14 14:23:48,868: DEBUG:  line:  Empfangene ausgehende Byte gesamt       1170260
cmanload.pl: 2023-03-14 14:23:48,868: DEBUG:  line:  Gesendete eingehende Byte gesamt            1170260
cmanload.pl: 2023-03-14 14:23:48,868: DEBUG:  line:  Gesendete ausgehende Byte gesamt           389065
cmanload.pl: 2023-03-14 14:23:48,868: DEBUG:  line:  Der Befehl wurde erfolgreich abgeschlossen.
cmanload.pl: 2023-03-14 14:23:48,868: DEBUG:  em_result=0|0|0

 Here the output of cmctl isn't as expected! 

Lines like Aktive Verbindungen gesamt, Verbindungen gesamt or Abgelehnte Verbindungen gesamt are probably not what cmanload.pl expects in it's pattern matching: 

...
  if($line =~ /^\s*Total Connections\s*[0-9]+/i )
...
  elsif($line =~ /^\s*Total Connections refused\s*[0-9]+/i )
...
  elsif($line =~ /^\s*Number of active connections\s*[0-9]+/i)
...

The explanation for these observations are quite simple: someone at this site decided to have all messages on servers in German and therefore NLS_LANG is set to GERMAN_GERMANY.WE8ISO8859P15. This is also true for the oem agent, and all the programs it calls - in this case cmanload.pl and cmctl. 

With this valuable insight, my workaround is simple. Replace the line 

	my @output = `$cmd 2>&1`;

by 

	my @output = `unset NLS_LANG; $cmd 2>&1`;

 or, as I'm lazy, do

sed -i 's/my @output = `$cmd 2>&1`;/my @output = `unset NLS_LANG; $cmd 2>\&1`;/' cmanload.pl

Now the tracefile show some understandable lines:

cmanload.pl: 2023-03-14 14:39:03,156: DEBUG:  line:  CMCTL for Linux: Version 21.0.0.0.0 - Production on 14-MAR-2023 14:39:03
cmanload.pl: 2023-03-14 14:39:03,156: DEBUG:  line:  
cmanload.pl: 2023-03-14 14:39:03,156: DEBUG:  line:  Copyright (c) 1996, 2021, Oracle.  All rights reserved.
cmanload.pl: 2023-03-14 14:39:03,156: DEBUG:  line:  
cmanload.pl: 2023-03-14 14:39:03,156: DEBUG:  line:  Current instance cman_xxx is already started
cmanload.pl: 2023-03-14 14:39:03,156: DEBUG:  line:  Connecting to (DESCRIPTION=(address=(protocol=tcp)(host=xxx)(port=1521)))
cmanload.pl: 2023-03-14 14:39:03,157: DEBUG:  line:  
cmanload.pl: 2023-03-14 14:39:03,157: DEBUG:  line:  Gateway ID                     0
cmanload.pl: 2023-03-14 14:39:03,157: DEBUG:  line:  Gateway state                  READY
cmanload.pl: 2023-03-14 14:39:03,157: DEBUG:  line:  Number of active connections   6
cmanload.pl: 2023-03-14 14:39:03,157: DEBUG:  line:  Peak active connections        6
cmanload.pl: 2023-03-14 14:39:03,157: DEBUG:  line:  Total connections              23
cmanload.pl: 2023-03-14 14:39:03,157: DEBUG:  line:  Total connections refused      0
cmanload.pl: 2023-03-14 14:39:03,157: DEBUG:  line:  Received IN bytes              148674
cmanload.pl: 2023-03-14 14:39:03,157: DEBUG:  line:  Received OUT bytes             372165
cmanload.pl: 2023-03-14 14:39:03,157: DEBUG:  line:  Sent IN bytes                  372165
cmanload.pl: 2023-03-14 14:39:03,157: DEBUG:  line:  Sent OUT bytes                 148674
cmanload.pl: 2023-03-14 14:39:03,157: DEBUG:  line:  
cmanload.pl: 2023-03-14 14:39:03,157: DEBUG:  line:  Gateway ID                     1
cmanload.pl: 2023-03-14 14:39:03,158: DEBUG:  line:  Gateway state                  READY
cmanload.pl: 2023-03-14 14:39:03,158: DEBUG:  line:  Number of active connections   6
cmanload.pl: 2023-03-14 14:39:03,158: DEBUG:  line:  Peak active connections        6
cmanload.pl: 2023-03-14 14:39:03,158: DEBUG:  line:  Total connections              25
cmanload.pl: 2023-03-14 14:39:03,158: DEBUG:  line:  Total connections refused      0
cmanload.pl: 2023-03-14 14:39:03,158: DEBUG:  line:  Received IN bytes              281311
cmanload.pl: 2023-03-14 14:39:03,158: DEBUG:  line:  Received OUT bytes             867707
cmanload.pl: 2023-03-14 14:39:03,158: DEBUG:  line:  Sent IN bytes                  867707
cmanload.pl: 2023-03-14 14:39:03,158: DEBUG:  line:  Sent OUT bytes                 281311
cmanload.pl: 2023-03-14 14:39:03,158: DEBUG:  line:  
cmanload.pl: 2023-03-14 14:39:03,158: DEBUG:  line:  
cmanload.pl: 2023-03-14 14:39:03,158: DEBUG:  line:  Summary:
cmanload.pl: 2023-03-14 14:39:03,158: DEBUG:  line:  
cmanload.pl: 2023-03-14 14:39:03,159: DEBUG:  line:  Total active cons              12
cmanload.pl: 2023-03-14 14:39:03,159: DEBUG:  line:  Total peak active con          12
cmanload.pl: 2023-03-14 14:39:03,159: DEBUG:  line:  Total Connections              48
cmanload.pl: 2023-03-14 14:39:03,159: DEBUG:  line:  Total connections Refused      0
cmanload.pl: 2023-03-14 14:39:03,159: DEBUG:  line:  Total Received IN bytes        429985
cmanload.pl: 2023-03-14 14:39:03,159: DEBUG:  line:  Total Received OUT bytes       1239872
cmanload.pl: 2023-03-14 14:39:03,159: DEBUG:  line:  Total Sent IN bytes            1239872
cmanload.pl: 2023-03-14 14:39:03,159: DEBUG:  line:  Total Sent OUT bytes           429985
cmanload.pl: 2023-03-14 14:39:03,159: DEBUG:  line:  The command completed successfully.
cmanload.pl: 2023-03-14 14:39:03,159: DEBUG:  em_result=96|0|12

Agent and OEM metric page are also happy now! 

Oracle accepted this unwanted behavior as Bug 35186390 : "Load" Metric it always show values Zero for CMAN target. Let's see how long it will take until an official Patch is available. Until then, at least I know how to help myself.