Adding custom jmx metrics to google cloud monitoring collectd configuration - stackdriver

Question

I've added the JVM Monitoring plugin as described here
That's all working great and I can, but now I'd like to add more JMX metrics. e.g. MemoryPool specific counters
So I've added this config to /opt/stackdriver/collectd/etc/collectd.d/jvm-sun-hotspot.conf
<MBean "jvm_localhost_MemoryPool">
ObjectName "java.lang:type=MemoryPool,name=*"
InstanceFrom "name"
<Value>
Type "gauge"
InstancePrefix "memorypool-usage_used"
Table false
Attribute "Usage.used"
</Value>
</MBean>
and
Collect "jvm_localhost_MemoryPool"
in the Connection section
It seems to be valid collectd config but when it's getting sent to Stackdriver/Google Cloud Monitoring it is rejected.
012 {#012 "index": 261,#012 "valueErrors": [#012 {#012 "error": {#012 "code": 3,#012 "message": "Unsupported
collectd id: plugin: \"jvm\" type: \"gauge\" type_instance: \"memorypool-usagecommitted\""#012 }#012 }#012 ]#012 },#012 {#012
"index": 262,#012 "valueErrors": [#012 {#012 "error": {#012 "code": 3,#012 "message": "Unsupported collectd id: plug
in: \"jvm\" type: \"gauge\" type_instance: \"memorypool-usageinit\""#012 }#012 }#012 ]#012 },#012 {#012 "index": 263,#012
"valueErrors": [#012 {#012 "error": {#012 "code": 3,#012 "message": "Unsupported collectd id: plugin: \"jvm\" type: \"gau
ge\" type_instance: \"memorypool-usagemax\""#012 }#012 }#012 ]#012 },#012 {#012 "index": 264,#012 "valueErrors": [#012
{#012 "error": {#012 "code": 3,#012 "message": "Unsupported
Now from what I understand it needs to be added as a custom metric, but this document suggests it would be auto created.
And indeed when I look at the list of builtin jvm metrics I can't see how they map onto the existing ones in the collectd configuration.
e.g. how is os-open_fd_count mapped to os/open_files ?
It would be helpful to see the actual api request being sent by Google's custom collectd implementation but I can't see a way of increasing logging.
I can see from this post that it's probably custom metrics where I want to be looking but how do I do that in the collectd configuration?
I've tried
InstancePrefix "custom.googleapis.com/memorypool-usage"
but still no joy.
Has anyone done this before or can offer any suggestions on what I'm doing wrong?

Answer 1

To get this logging I needed to add stackdriver_metric_type metadata.
The complete chain is now
<Chain "GenericJMX_jvm">
<Rule "rewrite_custom_jmx">
<Match regex>
Plugin "^GenericJMX$"
PluginInstance "^jvm.*$"
TypeInstance "^memorypool-usage_used$"
</Match>
<Target "set">
MetaData "stackdriver_metric_type" "custom.googleapis.com/jvm/memorypool/usage_used"
MetaData "label:pool" "%{plugin_instance}"
</Target>
<Target "replace">
MetaData "label:pool" "jvm" ""
</Target>
</Rule>
<Rule "rewrite_genericjmx_to_jvm">
<Match regex>
Plugin "^GenericJMX$"
PluginInstance "^jvm.*$"
</Match>
<Target "replace">
PluginInstance "jvm" ""
</Target>
<Target "set">
Plugin "jvm"
</Target>
Target "return"
</Rule>
</Chain>
The plugin instance is the pool name (e.g. G1 Eden Space) which is why I copied it into the 'pool' label.
This did auto-create the metric in Stackdriver but I also used the following body to the projects.metricDescriptors.create method to add description and units.
{
"name": "projects/yourprojecthere/metricDescriptors/custom.googleapis.com/jvm/memorypool/usage_used",
"labels": [
{
"key": "pool",
"description": "Name of the JVM memory pool."
}
],
"metricKind": "GAUGE",
"valueType": "DOUBLE",
"unit": "By",
"description": "Current size in bytes of the memory pool.",
"type": "custom.googleapis.com/jvm/memorypool/usage_used",
"monitoredResourceTypes": [
"gce_instance"
]
}
The resulting graph in metric explorer looks like this

Answer 2

The troubleshooting documents [1] could be helpful to determine what points need to be transformed, as well as to ensure your transformations behave as expected.
[1] https://cloud.google.com/monitoring/agent/custom-metrics-agent#troubleshooting