AWS Cloudwatch Logs

http://stackoverflow.com/questions/27804342/how-do-i-filter-and-extract-raw-log-event-data-from-amazon-cloudwatch

For using AWSCLI (plain one as well as with cwlogs plugin) seehttp://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/SearchDataFilterPattern.html For pattern syntax (plain text, [space separated] as as {JSON syntax}) see:http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/FilterAndPatternSyntax.html For python command line utility awslogs see https://github.com/jorgebastida/awslogs. AWSCLI: aws logs filter-log-events AWSCLI is official CLI for AWS services and now it supports logs too. To show help: $ aws logs filter-log-events help The filter can be based on: log group name --log-group-name (only last one is used) log stream name --log-stream-name (can be specified multiple times) start time --start-time end time --end-time (not --stop-time) filter patter --filter-pattern Only --log-group-name is obligatory. Times are expressed as epoch using milliseconds (not seconds). The call might look like this: $ aws logs filter-log-events \ --start-time 1447167000000 \ --end-time 1447167600000 \ --log-group-name /var/log/syslog \ --filter-pattern ERROR \ --output text It prints 6 columns of tab separated text: 1st: EVENTS (to denote, the line is a log record and not other information) 2nd: eventId 3rd: timestamp (time declared by the record as event time) 4th: logStreamName 5th: message 6th: ingestionTime So if you have Linux command line utilities at hand and care only about log record messages for interval from 2015-11-10T14:50:00Z to 2015-11-10T15:00:00Z, you may get it as follows: $ aws logs filter-log-events \ --start-time `date -d 2015-11-10T14:50:00Z +%s`000 \ --end-time `date -d 2015-11-10T15:00:00Z +%s`000 \ --log-group-name /var/log/syslog \ --filter-pattern ERROR \ --output text| grep "^EVENTS"|cut -f 5 AWSCLI with cwlogs plugin The cwlogs AWSCLI plugin is simpler to use: $ aws logs filter \ --start-time 2015-11-10T14:50:00Z \ --end-time 2015-11-10T15:00:00Z \ --log-group-name /var/log/syslog \ --filter-pattern ERROR It expects human readable date-time and always returns text output with (space delimited) columns: 1th: logStreamName 2nd: date 3rd: time 4th till the end: message On the other hand, it is a bit more difficult to install (few more steps to do plus current pip requires to declare the installation domain as trusted one). $ pip install awscli-cwlogs --upgrade \ --extra-index-url=http://aws-cloudwatch.s3-website-us-east-1.amazonaws.com/ \ --trusted-host aws-cloudwatch.s3-website-us-east-1.amazonaws.com $ aws configure set plugins.cwlogs cwlogs (if you make typo in last command, just correct it in ~/.aws/config file) awslogs command from jorgebastida/awslogs This become my favourite one - easy to install, powerful, easy to use. Installation: $ pip install awslogs To list available log groups: $ awslogs groups To list log streams $ awslogs streams /var/log/syslog To get the records and follow them (see new ones as they come): $ awslogs get --watch /var/log/syslog And you may filter the records by time range: $ awslogs get /var/log/syslog -s 2015-11-10T15:45:00 -e 2015-11-10T15:50:00 Since version 0.2.0 you have there also the --filter-pattern option. The output has columns: 1st: log group name 2nd: log stream name 3rd: message Using --no-group and --no-stream you may switch the first two columns off. Using --no-color you may get rid of color control characters in the output. Docker with Cloudwatch logs: https://start.jcolemorrison.com/how-to-setup-aws-ecs-logs-in-cloudwatch-and-ssm/