CloudWatch collectd plugin

https://aws.amazon.com/blogs/aws/new-high-resolution-custom-metrics-and-alarms-for-amazon-cloudwatch/
https://github.com/awslabs/collectd-cloudwatch
https://collectd.org/wiki/index.php/First_steps
$ sudo yum install collectd collectd-python
$ wget https://raw.githubusercontent.com/awslabs/collectd-cloudwatch/master/src/setup.py 
$ chmod a+x setup.py 

$ sudo ./setup.py
Installing dependencies ... OK
Installing python dependencies ... OK
Downloading plugin ... OK
Extracting plugin ... OK
Moving to collectd plugins directory ... OK
Copying CloudWatch plugin include file ... OK

Choose AWS region for published metrics:
  1. Automatic [ap-southeast-2]
  2. Custom
Enter choice [1]:

Choose hostname for published metrics:
  1. EC2 instance id [i-0c06f8a30aaaaaa]
  2. Custom
Enter choice [1]:

Choose authentication method:
  1. IAM Role [mytestrole]
  2. IAM User
Enter choice [1]:

Enter proxy server name:
  1. None
  2. Custom
Enter choice [1]:

Enter proxy server port:
  1. None
  2. Custom
Enter choice [1]:

Include the Auto-Scaling Group name as a metric dimension:
  1. No
  2. Yes
Enter choice [1]:

Include the FixedDimension as a metric dimension:
  1. No
  2. Yes
Enter choice [1]:

Enable high resolution:
  1. Yes
  2. No
Enter choice [2]:

Enter flush internal:
  1. Default 60s
  2. Custom
Enter choice [1]:

Choose how to install CloudWatch plugin in collectd:
  1. Do not modify existing collectd configuration
  2. Add plugin to the existing configuration
  3. Use CloudWatch recommended configuration (4 metrics)
Enter choice [3]:
Plugin configuration written successfully.
Creating backup of the original configuration ... OK
Replacing collectd configuration ... OK
Replacing whitelist configuration ... OK
Stopping collectd process ... NOT OK
Starting collectd process ... OK

If you have problems check that hostname resolves.

Plugin specific configuration

The default location of the configuration file used by collectd-cloudwatch plugin is: /opt/collectd-plugins/cloudwatch/config/plugin.conf. The parameters in this file are optional when plugin is executed on EC2 instance. This file allows modification of the following parameters:

• credentials_path - Used to point to AWS account configuration file
• region - Manual override for region used to publish metrics
• host - Manual override for EC2 Instance ID and Host information propagated by collectd
• proxy_server_name - Manual override for proxy server name, used by plugin to connect aws cloudwatch at *.amazonaws.com.
• proxy_server_port - Manual override for proxy server port, used by plugin to connect aws cloudwatch at *.amazonaws.com.
• enable_high_resolution_metrics - The storage resolution is for high resolution support
• flush_interval_in_seconds - The flush_interval_in_seconds is used for flush interval, it means how long plugin should flush the metrics to Cloudwatch
• whitelist_pass_through - Used to enable potentially unsafe regular expressions. By default regex such as a line containing .* or .+ only is automatically disabled in the whitelist configuration. Setting this value to True may result in a large number of metrics being published. Before changing this parameter, read pricing information to understand how to estimate your bill.
• push_asg - Used to include the Auto-Scaling Group as a dimension for all metrics (see Adding additional dimensions to metrics below for details)
• push_constant - Used to include a Fixed dimension (see constant_dimension_value below) on all metrics. Useful for collating all metrics of a certain type (see Adding additional dimensions to metrics below for details)
• constant_dimension_value - Used to specify the value for the Fixed dimension (see Adding additional dimensions to metrics below for details)
• debug - Provides verbose logging of metrics emitted to CloudWatch

To add new collectd metrics:
vi /etc/collectd.conf # add new metrics

vi /opt/collectd-plugins/cloudwatch/config/whitelist.conf # add to whitelist
service collectd restart 

To help debug, load CSV plugin to write values to plain text CSV files:
vi /etc/collectd.conf # add following

LoadPlugin csv

<Plugin "csv">
  DataDir "/var/lib/collectd/csv"
  StoreRates true
</Plugin>

To add custom metric:
vi /etc/collectd.conf # add following
LoadPlugin exec
<Plugin exec>
  Exec "ec2-user" "/opt/collectd-plugins/local/temperature.sh"
</Plugin>

# file opt/collectd-plugins/local/temperature.sh
#!/bin/bash

HOSTNAME="${COLLECTD_HOSTNAME:-localhost}"

INTERVAL="${COLLECTD_INTERVAL:-60}"

while sleep $INTERVAL; do
  val=$(dd if=/dev/urandom bs=1 count=1 2>/dev/null | hexdump -e "\"%d\n\"")
  echo "PUTVAL \"$HOSTNAME/exec-test/temperature\" interval=$INTERVAL N:$val"
done

# add to whitelist 

vi /opt/collectd-plugins/cloudwatch/config/whitelist.conf # add to whitelist
exec-.*

To get collectd cloudwatch stats via CLI:
aws cloudwatch get-metric-statistics --metric-name memory.percent.used --start-time `date -u --date="1 days ago" +'%Y-%m-%dT%H:%M:00'` --end-time `date -u +'%Y-%m-%dT%H:%M:00'` --period 900 --namespace collectd --statistics "[\"Average\",\"Sum\",\"SampleCount\",\"Maximum\",\"Minimum\"]" --dimensions Name=Host,Value=i-0c06f8a30abaaaaa  Name=PluginInstance,Value=NONE | jq -r  '.Datapoints[] | "\(.Timestamp),\(.Average)"' 2>/dev/null | sort

aws cloudwatch get-metric-statistics --metric-name exec.temperature --start-time `date -u --date="1 days ago" +'%Y-%m-%dT%H:%M:00'` --end-time `date -u +'%Y-%m-%dT%H:%M:00'` --period 60 --namespace collectd --statistics "[\"Average\",\"Sum\",\"SampleCount\",\"Maximum\",\"Minimum\"]" --dimensions Name=Host,Value=i-0c06f8a30ab693830  Name=PluginInstance,Value=test | jq -r  '.Datapoints[] | "\(.Timestamp),\(.Average)"' 2>/dev/null | sort