Amazon Kinesis (Data Analytics, Data Firehose, Data Streams, Video Streams) monitoring (2023)

Dynatrace ingests metrics for multiple preselected namespaces, including Amazon Kinesis. You can view metrics for each service instance, split metrics into multiple dimensions, and create custom charts that you can pin to your dashboards.

Prerequisites

To enable monitoring for this service, you need

  • ActiveGate version 1.181+, as follows:
    • For Dynatrace SaaS deployments, you need an Environment ActiveGate or a Multi-environment ActiveGate.
    • For Dynatrace Managed deployments, you can use any kind of ActiveGate.
      Note: For role-based access (whether in a SaaS or Managed deployment), you need an Environment ActiveGate installed on an Amazon EC2 host.
  • Dynatrace version 1.182+
  • An updated AWS monitoring policy to include the additional AWS services.
    To update the AWS IAM policy, use the JSON below, containing the monitoring policy (permissions) for all supporting services.

JSON predefined policy for all supporting services

json

{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "acm-pca:ListCertificateAuthorities", "apigateway:GET", "apprunner:ListServices", "appstream:DescribeFleets", "appsync:ListGraphqlApis", "athena:ListWorkGroups", "autoscaling:DescribeAutoScalingGroups", "cloudformation:ListStackResources", "cloudfront:ListDistributions", "cloudhsm:DescribeClusters", "cloudsearch:DescribeDomains", "cloudwatch:GetMetricData", "cloudwatch:GetMetricStatistics", "cloudwatch:ListMetrics", "codebuild:ListProjects", "datasync:ListTasks", "dax:DescribeClusters", "directconnect:DescribeConnections", "dms:DescribeReplicationInstances", "dynamodb:ListTables", "dynamodb:ListTagsOfResource", "ec2:DescribeAvailabilityZones", "ec2:DescribeInstances", "ec2:DescribeNatGateways", "ec2:DescribeSpotFleetRequests", "ec2:DescribeTransitGateways", "ec2:DescribeVolumes", "ec2:DescribeVpnConnections", "ecs:ListClusters", "eks:ListClusters", "elasticache:DescribeCacheClusters", "elasticbeanstalk:DescribeEnvironmentResources", "elasticbeanstalk:DescribeEnvironments", "elasticfilesystem:DescribeFileSystems", "elasticloadbalancing:DescribeInstanceHealth", "elasticloadbalancing:DescribeListeners", "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:DescribeRules", "elasticloadbalancing:DescribeTags", "elasticloadbalancing:DescribeTargetHealth", "elasticmapreduce:ListClusters", "elastictranscoder:ListPipelines", "es:ListDomainNames", "events:ListEventBuses", "firehose:ListDeliveryStreams", "fsx:DescribeFileSystems", "gamelift:ListFleets", "glue:GetJobs", "inspector:ListAssessmentTemplates", "kafka:ListClusters", "kinesis:ListStreams", "kinesisanalytics:ListApplications", "kinesisvideo:ListStreams", "lambda:ListFunctions", "lambda:ListTags", "lex:GetBots", "logs:DescribeLogGroups", "mediaconnect:ListFlows", "mediaconvert:DescribeEndpoints", "mediapackage-vod:ListPackagingConfigurations", "mediapackage:ListChannels", "mediatailor:ListPlaybackConfigurations", "opsworks:DescribeStacks", "qldb:ListLedgers", "rds:DescribeDBClusters", "rds:DescribeDBInstances", "rds:DescribeEvents", "rds:ListTagsForResource", "redshift:DescribeClusters", "robomaker:ListSimulationJobs", "route53:ListHostedZones", "route53resolver:ListResolverEndpoints", "s3:ListAllMyBuckets", "sagemaker:ListEndpoints", "sns:ListTopics", "sqs:ListQueues", "storagegateway:ListGateways", "sts:GetCallerIdentity", "swf:ListDomains", "tag:GetResources", "tag:GetTagKeys", "transfer:ListServers", "workmail:ListOrganizations", "workspaces:DescribeWorkspaces" ], "Resource": "*" } ]}

If you don't want to add permissions to all services, and just select permissions for certain services, consult the table below. The table contains a set of permissions that are required for all services (All monitored Amazon services) and, for each supporting service, a list of optional permissions specific to that service.

Complete list of permissions for cloud services

(Video) Intro to AWS Kinesis | Video Streams | Data Steams | Data Analytics | Kinesis Firehose

NameAdditional permissions
AWS Certificate Manager Private Certificate Authority"acm-pca:ListCertificateAuthorities"
All monitored Amazon services"cloudwatch:GetMetricData",
"cloudwatch:GetMetricStatistics",
"cloudwatch:ListMetrics",
"sts:GetCallerIdentity",
"tag:GetResources",
"tag:GetTagKeys",
"ec2:DescribeAvailabilityZones"
Amazon MQ
Amazon API Gateway"apigateway:GET"
AWS App Runner"apprunner:ListServices"
Amazon AppStream"appstream:DescribeFleets"
AWS AppSync"appsync:ListGraphqlApis"
Amazon Athena"athena:ListWorkGroups"
Amazon Aurora"rds:DescribeDBClusters"
Amazon EC2 Auto Scaling"autoscaling:DescribeAutoScalingGroups"
Amazon EC2 Auto Scaling (built-in)"autoscaling:DescribeAutoScalingGroups"
AWS Billing
Amazon Keyspaces
AWS Chatbot
Amazon CloudFront"cloudfront:ListDistributions"
AWS CloudHSM"cloudhsm:DescribeClusters"
Amazon CloudSearch"cloudsearch:DescribeDomains"
AWS CodeBuild"codebuild:ListProjects"
Amazon Cognito
Amazon Connect
Amazon Elastic Kubernetes Service (EKS)"eks:ListClusters"
AWS DataSync"datasync:ListTasks"
Amazon DynamoDB Accelerator (DAX)"dax:DescribeClusters"
Amazon Database Migration Service"dms:DescribeReplicationInstances"
Amazon DocumentDB"rds:DescribeDBClusters"
AWS Direct Connect"directconnect:DescribeConnections"
Amazon DynamoDB (built-in)"dynamodb:ListTables",
"dynamodb:ListTagsOfResource"
Amazon EBS (built-in)"ec2:DescribeVolumes"
Amazon EC2 API
Amazon EC2 (built-in)"ec2:DescribeInstances"
Amazon EC2 Spot Fleet"ec2:DescribeSpotFleetRequests"
Amazon Elastic Container Service (ECS)"ecs:ListClusters"
Amazon ECS ContainerInsights"ecs:ListClusters"
Amazon ElastiCache (EC)"elasticache:DescribeCacheClusters"
AWS Elastic Beanstalk"elasticbeanstalk:DescribeEnvironments"
Amazon Elastic File System (EFS)"elasticfilesystem:DescribeFileSystems"
Amazon Elastic Inference
Amazon Elastic Map Reduce (EMR)"elasticmapreduce:ListClusters"
Amazon Elasticsearch Service (ES)"es:ListDomainNames"
Amazon Elastic Transcoder"elastictranscoder:ListPipelines"
AWS Elastic Load Balancing (ELB) (built-in)"elasticloadbalancing:DescribeInstanceHealth",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeRules",
"elasticloadbalancing:DescribeTags",
"elasticloadbalancing:DescribeTargetHealth"
Amazon EventBridge"events:ListEventBuses"
Amazon FSx"fsx:DescribeFileSystems"
Amazon GameLift"gamelift:ListFleets"
AWS Glue"glue:GetJobs"
Amazon Inspector"inspector:ListAssessmentTemplates"
AWS Internet of Things (IoT)
AWS IoT Analytics
Amazon Managed Streaming for Kafka"kafka:ListClusters"
Amazon Kinesis Data Analytics"kinesisanalytics:ListApplications"
Amazon Kinesis Data Firehose"firehose:ListDeliveryStreams"
Amazon Kinesis Data Streams"kinesis:ListStreams"
Amazon Kinesis Video Streams"kinesisvideo:ListStreams"
AWS Lambda (built-in)"lambda:ListFunctions",
"lambda:ListTags"
Amazon Lex"lex:GetBots"
AWS Application and Network Load Balancer (built-in)"elasticloadbalancing:DescribeInstanceHealth",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeRules",
"elasticloadbalancing:DescribeTags",
"elasticloadbalancing:DescribeTargetHealth"
Amazon CloudWatch Logs"logs:DescribeLogGroups"
AWS Elemental MediaConnect"mediaconnect:ListFlows"
Amazon MediaConvert"mediaconvert:DescribeEndpoints"
Amazon MediaPackage Live"mediapackage:ListChannels"
Amazon MediaPackage Video on Demand"mediapackage-vod:ListPackagingConfigurations"
Amazon MediaTailor"mediatailor:ListPlaybackConfigurations"
Amazon VPC NAT Gateways"ec2:DescribeNatGateways"
Amazon Neptune"rds:DescribeDBClusters"
AWS OpsWorks"opsworks:DescribeStacks"
Amazon Polly
Amazon QLDB"qldb:ListLedgers"
Amazon RDS (built-in)"rds:DescribeDBInstances",
"rds:DescribeEvents",
"rds:ListTagsForResource"
Amazon Redshift"redshift:DescribeClusters"
Amazon Rekognition
AWS RoboMaker"robomaker:ListSimulationJobs"
Amazon Route 53"route53:ListHostedZones"
Amazon Route 53 Resolver"route53resolver:ListResolverEndpoints"
Amazon S3"s3:ListAllMyBuckets"
Amazon S3 (built-in)"s3:ListAllMyBuckets"
Amazon SageMaker Batch Transform Jobs
Amazon SageMaker Endpoint Instances"sagemaker:ListEndpoints"
Amazon SageMaker Endpoints"sagemaker:ListEndpoints"
Amazon SageMaker Ground Truth
Amazon SageMaker Processing Jobs
Amazon SageMaker Training Jobs
AWS Service Catalog
Amazon Simple Email Service (SES)
Amazon Simple Notification Service (SNS)"sns:ListTopics"
Amazon Simple Queue Service (SQS)"sqs:ListQueues"
AWS Systems Manager - Run Command
AWS Step Functions
AWS Storage Gateway"storagegateway:ListGateways"
Amazon SWF"swf:ListDomains"
Amazon Textract
AWS IoT Things Graph
Amazon Transfer Family"transfer:ListServers"
AWS Transit Gateway"ec2:DescribeTransitGateways"
Amazon Translate
AWS Trusted Advisor
AWS API Usage
AWS Site-to-Site VPN"ec2:DescribeVpnConnections"
Amazon WAF Classic
Amazon WAF
Amazon WorkMail"workmail:ListOrganizations"
Amazon WorkSpaces"workspaces:DescribeWorkspaces"

Example of JSON policy for one single service.

JSON policy for Amazon API Gateway

json

{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "apigateway:GET", "cloudwatch:GetMetricData", "cloudwatch:GetMetricStatistics", "cloudwatch:ListMetrics", "sts:GetCallerIdentity", "tag:GetResources", "tag:GetTagKeys", "ec2:DescribeAvailabilityZones" ], "Resource": "*" } ]}

In this example, from the complete list of permissions you need to select

  • "apigateway:GET" for Amazon API Gateway
  • "cloudwatch:GetMetricData", "cloudwatch:GetMetricStatistics", "cloudwatch:ListMetrics", "sts:GetCallerIdentity", "tag:GetResources", "tag:GetTagKeys", and "ec2:DescribeAvailabilityZones" for All monitored Amazon services.

Enable monitoring

To enable monitoring for this service, you first need to integrate Dynatrace with Amazon Web Services:

  • Set up Dynatrace SaaS integration
  • Set up Dynatrace Managed integration

Add the service to monitoring

In order to view the service metrics, you must add the service to monitoring in your Dynatrace environment.

To add a service to monitoring

  1. In the Dynatrace menu, go to Settings > Cloud and virtualization and select AWS.
  2. On the AWS overview page, scroll down and select the desired AWS instance. Select the Edit button.
  3. Scroll down and select Add service. Choose the service name from the drop-down and select Add service.
  4. Select Save changes.
(Video) Amazon Kinesis Data Streams Fundamentals

Note: Once AWS cloud services are added to monitoring, you might have to wait 15-20 minutes before the metric values are displayed.

Cloud-service monitoring consumption

All cloud services consume Davis data units (DDUs). The amount of DDU consumption per service instance depends on the number of monitored metrics and their dimensions (each metric dimension results in the ingestion of 1 data point; 1 data point consumes 0.001 DDUs).

You can choose to monitor resources based on existing AWS tags, as Dynatrace automatically imports them from service instances. Nevertheless, the transition from AWS to Dynatrace tagging isn't supported for all AWS services. Expand the table below to see which cloud services are filtered by tagging.

Tag filtering per service

NameTags monitoring & filtering
AWS Certificate Manager Private Certificate Authorityyes
Amazon MQ-
Amazon API Gatewayyes
AWS App Runneryes
Amazon AppStreamyes
AWS AppSyncyes
Amazon Athenayes
Amazon Aurorayes
Amazon EC2 Auto Scaling-
Amazon EC2 Auto Scaling (built-in)yes
AWS Billing-
Amazon Keyspacesyes
AWS Chatbot-
Amazon CloudFrontyes
AWS CloudHSMyes
Amazon CloudSearch-
AWS CodeBuildyes
Amazon Cognito-
Amazon Connect-
Amazon Elastic Kubernetes Service (EKS)yes
AWS DataSyncyes
Amazon DynamoDB Accelerator (DAX)yes
Amazon Database Migration Serviceyes
Amazon DocumentDByes
AWS Direct Connectyes
Amazon DynamoDB (built-in)yes
Amazon EBS (built-in)yes
Amazon EC2 API-
Amazon EC2 (built-in)yes
Amazon EC2 Spot Fleet-
Amazon Elastic Container Service (ECS)yes
Amazon ECS ContainerInsightsyes
Amazon ElastiCache (EC)yes
AWS Elastic Beanstalkyes
Amazon Elastic File System (EFS)yes
Amazon Elastic Inferenceyes
Amazon Elastic Map Reduce (EMR)yes
Amazon Elasticsearch Service (ES)yes
Amazon Elastic Transcoder-
AWS Elastic Load Balancing (ELB) (built-in)yes
Amazon EventBridgeyes
Amazon FSxyes
Amazon GameLift-
AWS Glueyes
Amazon Inspectoryes
AWS Internet of Things (IoT)-
AWS IoT Analytics-
Amazon Managed Streaming for Kafkayes
Amazon Kinesis Data Analyticsyes
Amazon Kinesis Data Firehoseyes
Amazon Kinesis Data Streamsyes
Amazon Kinesis Video Streamsyes
AWS Lambda (built-in)yes
Amazon Lexyes
AWS Application and Network Load Balancer (built-in)yes
Amazon CloudWatch Logsyes
AWS Elemental MediaConnect-
Amazon MediaConvertyes
Amazon MediaPackage Liveyes
Amazon MediaPackage Video on Demandyes
Amazon MediaTailoryes
Amazon VPC NAT Gatewaysyes
Amazon Neptuneyes
AWS OpsWorksyes
Amazon Polly-
Amazon QLDByes
Amazon RDS (built-in)yes
Amazon Redshiftyes
Amazon Rekognition-
AWS RoboMakeryes
Amazon Route 53-
Amazon Route 53 Resolveryes
Amazon S3yes
Amazon S3 (built-in)yes
Amazon SageMaker Batch Transform Jobs-
Amazon SageMaker Endpoint Instancesyes
Amazon SageMaker Endpointsyes
Amazon SageMaker Ground Truth-
Amazon SageMaker Processing Jobs-
Amazon SageMaker Training Jobs-
AWS Service Catalog-
Amazon Simple Email Service (SES)-
Amazon Simple Notification Service (SNS)yes
Amazon Simple Queue Service (SQS)yes
AWS Systems Manager - Run Command-
AWS Step Functions-
AWS Storage Gatewayyes
Amazon SWF-
Amazon Textract-
AWS IoT Things Graph-
Amazon Transfer Familyyes
AWS Transit Gatewayyes
Amazon Translate-
AWS Trusted Advisor-
AWS API Usage-
AWS Site-to-Site VPNyes
Amazon WAF Classic-
Amazon WAF-
Amazon WorkMailyes
Amazon WorkSpacesyes
(Video) How to Use Amazon Kinesis to Analyze AWS Streaming Data

To monitor resources based on tags

  1. In the Dynatrace menu, go to Settings > Cloud and virtualization > AWS and select Edit for the desired AWS instance.
  2. For Resources to be monitored, select Monitor resources selected by tags.
  3. Enter the Key and Value.
  4. Select Save.

Configure service metrics

Once you add a service, Dynatrace starts automatically collecting a suite of metrics for this particular service. These are recommended metrics. Apart from the recommended metrics, most services have the possibility of enabling optional metrics. You can remove or edit any of the existing metrics or any of their dimensions, where there are multiple dimensions available. Metrics consisting of only one dimension can't be edited. They can only be removed or added.

Service-wide metrics are metrics for the whole service across all regions. Typically, these metrics include dimensions containing Region in their name. If selected, these metrics are displayed on a separate chart when viewing your AWS deployment in Dynatrace. Keep in mind that available dimensions differ among services.

To change a metric's statistics, you have to recreate that metric by choosing different statistics. You can choose among the following statistics: Sum, Minimum, Maximum, Average, and Sample count. The Average + Minimum + Maximum statistics enable you to collect all three statistics as one metric instead of one statistic for three metrics separately. This can reduce your expenses for retrieving metrics from your AWS deployment.

To be able to save a newly added metric, you need to select at least one statistic and one dimension.

How to add and configure metrics

  1. In the Dynatrace menu, go to Settings > Cloud and virtualization and select AWS.
  2. On the AWS overview page, scroll down and select the desired AWS instance. Select the Edit button.
  3. Scroll down to the Services section and select Manage services.
  4. To add a metric, select the service for which you want to add metrics.
  5. Select Add new metric.
  6. From the menu, select the metric you want.
  7. Select Add metric to add the metric to monitoring.
  8. To configure a metric, select the Edit button associated to it.
  9. Select Apply to save your configuration.

Note: Once AWS cloud services are configured, you might have to wait 15-20 minutes before the metric values are displayed.

View service metrics

You can view the service metrics in your Dynatrace environment either on the custom device overview page or on your Dashboards page.

(Video) Process Data from Kinesis Data Streams with Kinesis Data Analytics | Amazon Web Services

View metrics on the custom device overview page

To access the custom device overview page

  1. In the Dynatrace menu, go to Technologies and processes.
  2. Filter by service name and select the relevant custom device group.
  3. Once you select the custom device group, you're on the custom device group overview page.
  4. The custom device group overview page lists all instances (custom devices) belonging to the group. Select an instance to view the custom device overview page.

View metrics on your dashboard

You can also view metrics in the Dynatrace web UI on dashboards. There is no preset dashboard available for this service, but you can create your own dashboard.

To check the availability of preset dashboards for each AWS service, see the list below.

Preset dashboard availability list

AWS servicePreset dashboard
AWS Certificate Manager Private Certificate Authorityno
Amazon MQyes
Amazon API Gatewayno
AWS App Runnerno
Amazon AppStreamyes
AWS AppSyncyes
Amazon Athenayes
Amazon Aurorano
Amazon EC2 Auto Scalingyes
Amazon EC2 Auto Scaling (built-in)no
AWS Billingyes
Amazon Keyspacesyes
AWS Chatbotyes
Amazon CloudFrontno
AWS CloudHSMyes
Amazon CloudSearchyes
AWS CodeBuildyes
Amazon Cognitono
Amazon Connectyes
Amazon Elastic Kubernetes Service (EKS)yes
AWS DataSyncyes
Amazon DynamoDB Accelerator (DAX)yes
Amazon Database Migration Serviceyes
Amazon DocumentDByes
AWS Direct Connectyes
Amazon DynamoDB (built-in)no
Amazon EBS (built-in)no
Amazon EC2 APIyes
Amazon EC2 (built-in)no
Amazon EC2 Spot Fleetno
Amazon Elastic Container Service (ECS)no
Amazon ECS ContainerInsightsyes
Amazon ElastiCache (EC)no
AWS Elastic Beanstalkyes
Amazon Elastic File System (EFS)no
Amazon Elastic Inferenceyes
Amazon Elastic Map Reduce (EMR)no
Amazon Elasticsearch Service (ES)no
Amazon Elastic Transcoderyes
AWS Elastic Load Balancing (ELB) (built-in)no
Amazon EventBridgeyes
Amazon FSxyes
Amazon GameLiftyes
AWS Glueno
Amazon Inspectoryes
AWS Internet of Things (IoT)no
AWS IoT Analyticsyes
Amazon Managed Streaming for Kafkayes
Amazon Kinesis Data Analyticsno
Amazon Kinesis Data Firehoseno
Amazon Kinesis Data Streamsno
Amazon Kinesis Video Streamsno
AWS Lambda (built-in)no
Amazon Lexyes
AWS Application and Network Load Balancer (built-in)no
Amazon CloudWatch Logsyes
AWS Elemental MediaConnectyes
Amazon MediaConvertyes
Amazon MediaPackage Liveyes
Amazon MediaPackage Video on Demandyes
Amazon MediaTailoryes
Amazon VPC NAT Gatewaysno
Amazon Neptuneyes
AWS OpsWorksyes
Amazon Pollyyes
Amazon QLDByes
Amazon RDS (built-in)no
Amazon Redshiftno
Amazon Rekognitionyes
AWS RoboMakeryes
Amazon Route 53yes
Amazon Route 53 Resolveryes
Amazon S3no
Amazon S3 (built-in)no
Amazon SageMaker Batch Transform Jobsno
Amazon SageMaker Endpoint Instancesno
Amazon SageMaker Endpointsno
Amazon SageMaker Ground Truthno
Amazon SageMaker Processing Jobsno
Amazon SageMaker Training Jobsno
AWS Service Catalogyes
Amazon Simple Email Service (SES)no
Amazon Simple Notification Service (SNS)no
Amazon Simple Queue Service (SQS)no
AWS Systems Manager - Run Commandyes
AWS Step Functionsyes
AWS Storage Gatewayyes
Amazon SWFyes
Amazon Textractyes
AWS IoT Things Graphyes
Amazon Transfer Familyyes
AWS Transit Gatewayyes
Amazon Translateyes
AWS Trusted Advisoryes
AWS API Usageyes
AWS Site-to-Site VPNyes
Amazon WAF Classicyes
Amazon WAFyes
Amazon WorkMailyes
Amazon WorkSpacesyes

Available metrics

Amazon Kinesis Data Analytics

NameDescriptionUnitStatisticsDimensionsRecommended
BytesThe number of bytes read (per input stream) or written (per output stream)BytesSumApplication, Flow, Id✔️
InputProcessing.DroppedRecordsThe number of records returned by a Lambda function that were marked with Dropped statusCountSumApplication, Flow, Id
InputProcessing.DurationThe time taken for each AWS Lambda function invocation performed by Kinesis Data AnalyticsMillisecondsMultiApplication, Flow, Id
InputProcessing.OkBytesThe sum of bytes of the records returned by a Lambda function that were marked with OK statusBytesSumApplication, Flow, Id
InputProcessing.OkRecordsThe number of records returned by a Lambda function that were marked with OK statusCountSumApplication, Flow, Id
InputProcessing.ProcessingFailedRecordsThe number of records returned by a Lambda function that were marked with ProcessingFailed statusCountSumApplication, Flow, Id
InputProcessing.SuccessThe number of successful Lambda invocations by Kinesis Data AnalyticsCountSumApplication, Flow, Id
KPUsThe number of Kinesis Processing Units that are used to run your stream processing applicationCountSumApplication✔️
LambdaDelivery.DeliveryFailedRecordsThe number of successful Lambda invocations by Kinesis Data AnalyticsCountSumApplication, Flow, Id
LambdaDelivery.DurationThe time taken for each Lambda function invocation performed by Kinesis Data AnalyticsMillisecondsMultiApplication, Flow, Id
LambdaDelivery.OkRecordsThe number of records returned by a Lambda function that were marked with OK statusCountSumApplication, Flow, Id
MillisBehindLatestIndicates how far behind from the current time an application is reading from the streaming sourceMillisecondsMultiApplication; Application, Flow, Id
RecordsThe number of records read (per input stream) or written (per output stream)CountSumApplication, Flow, Id✔️
SuccessThe number of successful deliveries. Every successful delivery attempt to the destination configured for your application is marked with 1. Every failed delivery attempt is marked with 0 .CountAverageApplication, Flow, Id✔️

Amazon Kinesis Data Firehose

NameDescriptionUnitStatisticsDimensionsRecommended
BackupToS3.BytesThe number of bytes delivered to Amazon S3 for backup over the specified time period. Kinesis Data Firehose emits this metric when backup to Amazon S3 is enabled.BytesSumRegion
BackupToS3.BytesBytesSumDeliveryStreamName
BackupToS3.DataFreshnessAge (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the Amazon S3 bucket for backup. Kinesis Data Firehose emits this metric when backup to Amazon S3 is enabled.SecondsMaximumRegion
BackupToS3.DataFreshnessSecondsMaximumDeliveryStreamName
BackupToS3.RecordsThe number of records delivered to Amazon S3 for backup over the specified time period. Kinesis Data Firehose emits this metric when backup to Amazon S3 is enabled.CountSumRegion
BackupToS3.RecordsCountSumDeliveryStreamName
BackupToS3.SuccessSum of successful Amazon S3 put commands for backup over sum of all Amazon S3 backup put commands. Kinesis Data Firehose emits this metric when backup to Amazon S3 is enabled.CountCountRegion
BackupToS3.SuccessCountCountDeliveryStreamName
DataReadFromKinesisStream.BytesWhen the data source is a Kinesis data stream, this metric indicates the number of bytes read from that data stream. This number includes rereads due to failovers.BytesSumRegion
DataReadFromKinesisStream.BytesBytesSumDeliveryStreamName
DataReadFromKinesisStream.RecordsWhen the data source is a Kinesis data stream, this metric indicates the number of records read from that data stream. This number includes rereads due to failovers.CountSumRegion
DataReadFromKinesisStream.RecordsCountSumDeliveryStreamName
DeliveryToElasticsearch.BytesThe number of bytes indexed to Amazon ES over the specified time periodBytesSumRegion
DeliveryToElasticsearch.BytesBytesSumDeliveryStreamName
DeliveryToElasticsearch.RecordsThe number of records indexed to Amazon ES over the specified time periodCountSumRegion
DeliveryToElasticsearch.RecordsCountSumDeliveryStreamName
DeliveryToElasticsearch.SuccessThe sum of the successfully indexed records over the sum of records that were attemptedCountCountRegion
DeliveryToElasticsearch.SuccessCountCountDeliveryStreamName
DeliveryToRedshift.BytesThe number of bytes copied to Amazon Redshift over the specified time periodBytesSumRegion
DeliveryToRedshift.BytesBytesSumDeliveryStreamName
DeliveryToRedshift.RecordsThe number of records copied to Amazon Redshift over the specified time periodCountSumRegion
DeliveryToRedshift.RecordsCountSumDeliveryStreamName
DeliveryToRedshift.SuccessThe sum of successful Amazon Redshift COPY commands over the sum of all Amazon Redshift COPY commandsCountCountRegion
DeliveryToRedshift.SuccessCountCountDeliveryStreamName
DeliveryToS3.BytesThe number of bytes delivered to Amazon S3 over the specified time periodBytesSumRegion
DeliveryToS3.BytesBytesSumDeliveryStreamName
DeliveryToS3.DataFreshnessThe age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the S3 bucket.SecondsMaximumRegion
DeliveryToS3.DataFreshnessSecondsMaximumDeliveryStreamName
DeliveryToS3.RecordsThe number of records delivered to Amazon S3 over the specified time periodCountSumRegion
DeliveryToS3.RecordsCountSumDeliveryStreamName
DeliveryToS3.SuccessThe sum of successful Amazon S3 put commands over the sum of all Amazon S3 put commandsCountCountRegion
DeliveryToS3.SuccessCountCountDeliveryStreamName
DeliveryToSplunk.BytesThe number of bytes delivered to Splunk over the specified time periodBytesSumRegion
DeliveryToSplunk.BytesBytesSumDeliveryStreamName
DeliveryToSplunk.DataAckLatencyThe approximate duration it takes to receive an acknowledgement from Splunk after Kinesis Data Firehose sends it dataSecondsAverageRegion
DeliveryToSplunk.DataAckLatencySecondsAverageDeliveryStreamName
DeliveryToSplunk.DataFreshnessAge (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to Splunk.SecondsMaximumRegion
DeliveryToSplunk.DataFreshnessSecondsMaximumDeliveryStreamName
DeliveryToSplunk.RecordsThe number of records delivered to Splunk over the specified time periodCountSumRegion
DeliveryToSplunk.RecordsCountSumDeliveryStreamName
DeliveryToSplunk.SuccessThe sum of the successfully indexed records over the sum of records that were attemptedCountCountRegion
DeliveryToSplunk.SuccessCountCountDeliveryStreamName
DescribeDeliveryStream.LatencyThe time taken per DescribeDeliveryStream operation, measured over the specified time periodMillisecondsMultiRegion
DescribeDeliveryStream.LatencyMillisecondsMultiDeliveryStreamName
DescribeDeliveryStream.RequestsThe total number of DescribeDeliveryStream requestsCountSumRegion
DescribeDeliveryStream.RequestsCountSumDeliveryStreamName
ExecuteProcessing.DurationThe time it takes for each Lambda function invocation performed by Kinesis Data FirehoseMillisecondsMultiRegion
ExecuteProcessing.DurationMillisecondsMultiDeliveryStreamName
ExecuteProcessing.SuccessThe sum of the successful Lambda function invocations over the sum of the total Lambda function invocationsCountCountRegion
ExecuteProcessing.SuccessCountCountDeliveryStreamName
FailedConversion.BytesThe size of the records that could not be convertedBytesSumRegion
FailedConversion.BytesBytesSumDeliveryStreamName
FailedConversion.RecordsThe number of records that could not be convertedCountSumRegion
FailedConversion.RecordsCountSumDeliveryStreamName
IncomingBytesThe number of bytes ingested successfully into the delivery stream over the specified time period after throttlingBytesSumRegion
IncomingBytesBytesSumDeliveryStreamName✔️
IncomingRecordsThe number of records ingested successfully into the delivery stream over the specified time period after throttlingCountSumRegion
IncomingRecordsCountSumDeliveryStreamName✔️
KinesisMillisBehindLatestWhen the data source is a Kinesis data stream, this metric indicates the number of milliseconds that the last read record is behind the newest record in the Kinesis data streamMillisecondsAverageRegion
KinesisMillisBehindLatestMillisecondsAverageDeliveryStreamName
ListDeliveryStreams.LatencyThe time taken per ListDeliveryStream operation, measured over the specified time periodMillisecondsMultiRegion
ListDeliveryStreams.LatencyMillisecondsMultiDeliveryStreamName
ListDeliveryStreams.RequestsThe total number of ListFirehose requestsCountSumRegion
ListDeliveryStreams.RequestsCountSumDeliveryStreamName
PutRecordBatch.BytesThe number of bytes put to the Kinesis Data Firehose delivery stream using PutRecordBatch over the specified time periodBytesSumRegion
PutRecordBatch.BytesBytesSumDeliveryStreamName
PutRecordBatch.LatencyThe time taken per PutRecordBatch operation, measured over the specified time periodMillisecondsMultiRegion
PutRecordBatch.LatencyMillisecondsMultiDeliveryStreamName
PutRecordBatch.RecordsThe total number of records from PutRecordBatch operationsCountSumRegion
PutRecordBatch.RecordsCountSumDeliveryStreamName
PutRecordBatch.RequestsThe total number of PutRecordBatch requestsCountSumRegion
PutRecordBatch.RequestsCountSumDeliveryStreamName
PutRecord.BytesThe number of bytes put to the Kinesis Data Firehose delivery stream using PutRecord over the specified time periodBytesSumRegion
PutRecord.BytesBytesSumDeliveryStreamName
PutRecord.LatencyThe time taken per PutRecord operation, measured over the specified time periodMillisecondsMultiRegion
PutRecord.LatencyMillisecondsMultiDeliveryStreamName
PutRecord.RequestsThe total number of PutRecord requests, which is equal to the total number of records from PutRecord operationsCountSumRegion
PutRecord.RequestsCountSumDeliveryStreamName
SucceedConversion.BytesThe size of the successfully converted recordsBytesSumRegion
SucceedConversion.BytesBytesSumDeliveryStreamName
SucceedConversion.RecordsThe number of successfully converted recordsCountSumRegion
SucceedConversion.RecordsCountSumDeliveryStreamName
SucceedProcessing.BytesThe number of successfully processed bytes over the specified time periodBytesSumRegion
SucceedProcessing.BytesBytesSumDeliveryStreamName
SucceedProcessing.RecordsThe number of successfully processed records over the specified time periodCountSumRegion
SucceedProcessing.RecordsCountSumDeliveryStreamName
ThrottledDescribeStreamThe total number of times the DescribeStream operation is throttled when the data source is a Kinesis data streamCountAverageRegion
ThrottledDescribeStreamCountAverageDeliveryStreamName
ThrottledGetRecordsThe total number of times the GetRecords operation is throttled when the data source is a Kinesis data streamCountAverageRegion
ThrottledGetRecordsCountAverageDeliveryStreamName
ThrottledGetShardIteratorThe total number of times the GetShardIterator operation is throttled when the data source is a Kinesis data streamCountAverageRegion
ThrottledGetShardIteratorCountAverageDeliveryStreamName
UpdateDeliveryStream.LatencyThe time taken per UpdateDeliveryStream operation, measured over the specified time periodMillisecondsMultiRegion
UpdateDeliveryStream.LatencyMillisecondsMultiDeliveryStreamName
UpdateDeliveryStream.RequestsThe total number of UpdateDeliveryStream requestsCountSumRegion
UpdateDeliveryStream.RequestsCountSumDeliveryStreamName
(Video) High Performance Data Streaming with Amazon Kinesis: Best Practices and Common Pitfalls

Amazon Kinesis Data Streams (KDS)

NameDescriptionUnitStatisticsDimensionsRecommended
GetRecords.BytesThe number of bytes retrieved from the Kinesis stream, measured over the specified time period. Minimum, maximum, and average statistics represent the bytes in a single GetRecords operation for the stream in the specified time period.BytesSumStreamName
GetRecords.BytesBytesMultiStreamName
GetRecords.BytesBytesCountStreamName
GetRecords.IteratorAgeMillisecondsThe age of the last record in all GetRecords calls made against a Kinesis stream, measured over the specified time period. Age is the difference between the current time and when the last record of the GetRecords call was written to the stream. The minimum and maximum statistics can be used to track the progress of Kinesis consumer applications. A value of 0 indicates that the records being read are completely caught up with the stream.MillisecondsMultiStreamName✔️
GetRecords.IteratorAgeMillisecondsMillisecondsCountStreamName
GetRecords.LatencyThe time taken per GetRecords operation, measured over the specified time periodMillisecondsMultiStreamName
GetRecords.RecordsThe number of records retrieved from the shard, measured over the specified time period. Minimum, maximum, and average statistics represent the records in a single GetRecords operation for the stream in the specified time period.CountSumStreamName
GetRecords.RecordsCountMultiStreamName
GetRecords.RecordsCountCountStreamName
GetRecords.SuccessThe number of successful GetRecords operations per stream, measured over the specified time periodCountSumStreamName
GetRecords.SuccessCountAverageStreamName✔️
GetRecords.SuccessCountCountStreamName
IncomingBytesThe number of bytes successfully put to the Kinesis stream over the specified time period. This metric includes bytes from PutRecord and PutRecords operations. Minimum, maximum, and average statistics represent the bytes in a single put operation for the stream in the specified time period.BytesSumStreamName; StreamName, ShardId
IncomingBytesBytesMultiStreamName; StreamName, ShardId
IncomingBytesBytesCountStreamName; StreamName, ShardId
IncomingRecordsThe number of records successfully put to the Kinesis stream over the specified time period. This metric includes record counts from PutRecord and PutRecords operations. Minimum, maximum, and average statistics represent the records in a single put operation for the stream in the specified time period.CountSumStreamName; StreamName, ShardId
IncomingRecordsCountMultiStreamName; StreamName, ShardId
IncomingRecordsCountCountStreamName; StreamName, ShardId
IteratorAgeMillisecondsThe age of the last record in all GetRecords calls made against a shard, measured over the specified time period. Age is the difference between the current time and when the last record of the GetRecords call was written to the stream. The minimum and maximum statistics can be used to track the progress of Kinesis consumer applications. A value of 0 indicates that the records being read are completely caught up with the stream.MillisecondsMultiStreamName, ShardId
IteratorAgeMillisecondsMillisecondsCountStreamName, ShardId
OutgoingBytesThe number of bytes retrieved from the shard, measured over the specified time period. Minimum, maximum, and average statistics represent the bytes returned in a single GetRecords operation or published in a single SubscribeToShard event for the shard in the specified time period.BytesSumStreamName, ShardId
OutgoingBytesBytesMultiStreamName, ShardId
OutgoingBytesBytesCountStreamName, ShardId
OutgoingRecordsThe number of records retrieved from the shard, measured over the specified time period. Minimum, maximum, and average statistics represent the records returned in a single GetRecords operation or published in a single SubscribeToShard event for the shard in the specified time period.CountSumStreamName, ShardId
OutgoingRecordsCountMultiStreamName, ShardId
OutgoingRecordsCountCountStreamName, ShardId
PutRecord.BytesThe number of bytes put to the Kinesis stream using the PutRecord operation over the specified time periodBytesSumStreamName
PutRecord.BytesBytesMultiStreamName
PutRecord.BytesBytesCountStreamName
PutRecord.LatencyThe time taken per PutRecord operation, measured over the specified time periodMillisecondsMultiStreamName
PutRecord.SuccessThe number of successful PutRecord operations per Kinesis stream, measured over the specified time period. Average reflects the percentage of successful writes to a stream.CountSumStreamName
PutRecord.SuccessCountAverageStreamName✔️
PutRecord.SuccessCountCountStreamName
PutRecords.BytesThe number of bytes put to the Kinesis stream using the PutRecords operation over the specified time periodBytesSumStreamName
PutRecords.BytesBytesMultiStreamName
PutRecords.BytesBytesCountStreamName
PutRecords.LatencyThe time taken per PutRecords operation, measured over the specified time periodMillisecondsMultiStreamName
PutRecords.RecordsThe number of successful records in a PutRecords operation per Kinesis stream, measured over the specified time periodCountSumStreamName
PutRecords.RecordsCountMultiStreamName
PutRecords.RecordsCountCountStreamName
PutRecords.SuccessThe number of PutRecords operations where at least one record succeeded, per Kinesis stream, measured over the specified time periodCountSumStreamName
PutRecords.SuccessCountAverageStreamName
PutRecords.SuccessCountCountStreamName
ReadProvisionedThroughputExceededThe number of GetRecords calls throttled for the stream over the specified time periodCountSumStreamName
ReadProvisionedThroughputExceededCountMultiStreamName✔️
ReadProvisionedThroughputExceededCountCountStreamName
SubscribeToShardEvent.BytesThe number of bytes received from the shard, measured over the specified time period. Minimum, maximum, and average statistics represent the bytes published in a single event for the specified time period.BytesSumStreamName, ConsumerName
SubscribeToShardEvent.BytesBytesMultiStreamName, ConsumerName
SubscribeToShardEvent.BytesBytesCountStreamName, ConsumerName
SubscribeToShardEvent.MillisBehindLatestThe difference between the current time and when the last record of the SubscribeToShard event was written to the streamMillisecondsMultiStreamName, ConsumerName
SubscribeToShardEvent.MillisBehindLatestMillisecondsCountStreamName, ConsumerName
SubscribeToShardEvent.RecordsThe number of records received from the shard, measured over the specified time period. Minimum, maximum, and average statistics represent the records in a single event for the specified time period.CountSumStreamName, ConsumerName
SubscribeToShardEvent.RecordsCountMultiStreamName, ConsumerName
SubscribeToShardEvent.RecordsCountCountStreamName, ConsumerName
SubscribeToShardEvent.SuccessThis metric is emitted every time an event is published successfully. Only emitted when there's an active subscription.CountSumStreamName, ConsumerName
SubscribeToShardEvent.SuccessCountMultiStreamName, ConsumerName
SubscribeToShardEvent.SuccessCountCountStreamName, ConsumerName
SubscribeToShard.RateExceededThis metric is emitted when a new subscription attempt fails because there already is an active subscription by the same consumer or if you exceed the number of calls per second allowed for this operationCountMinimumStreamName, ConsumerName
SubscribeToShard.SuccessCountMinimumStreamName, ConsumerName
WriteProvisionedThroughputExceededThe number of records rejected due to throttling for the stream over the specified time period. This metric includes throttling from PutRecord and PutRecords operations.CountSumStreamName
WriteProvisionedThroughputExceededCountMultiStreamName✔️
WriteProvisionedThroughputExceededCountCountStreamName

Amazon Kinesis Video Streams

NameDescriptionUnitStatisticsDimensionsRecommended
GetHLSMasterPlaylist.LatencyLatency of the GetHLSMasterPlaylist API calls for the given stream nameMillisecondsMultiStreamName
GetHLSMasterPlaylist.RequestsNumber of GetHLSMasterPlaylist API requests for a given streamCountSumStreamName
GetHLSMasterPlaylist.SuccessThe rate of success, 1 being the value for every successful request, and 0 the value for every failureCountAverageStreamName
GetHLSMediaPlaylist.LatencyLatency of the GetHLSMediaPlaylist API calls for the given stream nameMillisecondsMultiStreamName
GetHLSMediaPlaylist.RequestsNumber of GetHLSMediaPlaylist API requests for a given streamCountSumStreamName
GetHLSMediaPlaylist.SuccessThe rate of success, 1 being the value for every successful request, and 0 the value for every failureCountAverageStreamName
GetHLSStreamingSessionURL.LatencyLatency of the GetHLSStreamingSessionURL API calls for the given stream nameMillisecondsMultiStreamName
GetHLSStreamingSessionURL.RequestsNumber of GetHLSStreamingSessionURL API requests for a given streamCountSumStreamName
GetHLSStreamingSessionURL.SuccessThe rate of success, 1 being the value for every successful request, and 0 the value for every failureCountAverageStreamName
GetMP4InitFragment.LatencyLatency of the GetMP4InitFragment API calls for the given stream nameMillisecondsMultiStreamName
GetMP4InitFragment.RequestsNumber of GetMP4InitFragment API requests for a given streamCountSumStreamName
GetMP4InitFragment.SuccessThe rate of success, 1 being the value for every successful request, and 0 the value for every failureCountAverageStreamName
GetMP4MediaFragment.LatencyLatency of the GetMP4MediaFragment API calls for the given stream nameMillisecondsMultiStreamName
GetMP4MediaFragment.OutgoingBytesTotal number of bytes sent out from the service as part of the GetMP4MediaFragment API for a given streamBytesSumStreamName
GetMP4MediaFragment.RequestsNumber of GetMP4MediaFragment API requests for a given streamCountSumStreamName
GetMP4MediaFragment.SuccessThe rate of success, 1 being the value for every successful request, and 0 the value for every failureCountSumStreamName
GetMedia.ConnectionErrorsThe number of connections that were not successfully establishedCountSumStreamName✔️
GetMediaForFragmentList.OutgoingBytesTotal number of bytes sent out from the service as part of the GetMediaForFragmentList API for a given streamBytesSumStreamName
GetMediaForFragmentList.OutgoingFragmentsTotal number of fragments sent out from the service as part of the GetMediaForFragmentList API for a given streamCountSumStreamName
GetMediaForFragmentList.OutgoingFramesTotal number of frames sent out from the service as part of the GetMediaForFragmentList API for a given streamCountSumStreamName
GetMediaForFragmentList.RequestsNumber of GetMediaForFragmentList API requests for a given streamCountSumStreamName
GetMediaForFragmentList.SuccessThe rate of success, 1 being the value for every successful request, and 0 the value for every failureCountAverageStreamName
GetMedia.MillisBehindNowTime difference between the current server timestamp and the server timestamp of the last fragment sentMillisecondsMultiStreamName
GetMedia.OutgoingBytesTotal number of bytes sent out from the service as part of the GetMedia API for a given streamBytesSumStreamName
GetMedia.OutgoingFragmentsNumber of fragments sent while doing GetMedia for the streamCountSumStreamName
GetMedia.OutgoingFramesNumber of frames sent during GetMedia on the given streamCountSumStreamName
GetMedia.RequestsNumber of GetMedia API requests for a given streamCountSumStreamName
GetMedia.SuccessThe rate of success, 1 being the value for every successful request, and 0 the value for every failureCountAverageStreamName✔️
ListFragments.LatencyLatency of the ListFragments API calls for the given stream nameMillisecondsMultiStreamName✔️
PutMedia.ActiveConnectionsThe total number of connections to the service hostCountSumStreamName
PutMedia.BufferingAckLatencyTime difference between when the first byte of a new fragment is received by Kinesis Video Streams and when the Buffering ACK is sent for the fragmentMillisecondsMultiStreamName
PutMedia.ConnectionErrorsErrors while establishing PutMedia connection for the streamCountSumStreamName✔️
PutMedia.ErrorAckCountNumber of Error ACKs sent while doing PutMedia for the streamCountSumStreamName
PutMedia.FragmentIngestionLatencyTime difference between when the first and last bytes of a fragment are received by Kinesis Video StreamsMillisecondsMultiStreamName
PutMedia.FragmentPersistLatencyTime taken from when the complete fragment data is received and archivedMillisecondsMultiStreamName
PutMedia.IncomingBytesNumber of bytes received as part of PutMedia for the streamBytesSumStreamName
PutMedia.IncomingFragmentsNumber of complete fragments received as part of PutMedia for the streamCountSumStreamName
PutMedia.IncomingFramesNumber of complete frames received as part of PutMedia for the streamCountSumStreamName
PutMedia.LatencyTime difference between the request and the HTTP response from InletService while establishing the connectionMillisecondsMultiStreamName
PutMedia.PersistedAckLatencyTime difference between when the last byte of a new fragment is received by Kinesis Video Streams and when the Persisted ACK is sent for the fragmentMillisecondsMultiStreamName
PutMedia.ReceivedAckLatencyTime difference between when the last byte of a new fragment is received by Kinesis Video Streams and when the Received ACK is sent for the fragmentMillisecondsMultiStreamName
PutMedia.RequestsNumber of PutMedia API requests for a given streamCountSumStreamName
PutMedia.SuccessThe rate of success, 1 being the value for every successful request, and 0 the value for every failureCountAverageStreamName✔️

Videos

1. Create a data Firehose on AWS // Kinesis streaming
(Kahan Data Solutions)
2. Stream Data Analytics with Amazon Kinesis Firehose and Redshift
(Amazon Web Services)
3. Amazon Kinesis Data Streams: Why Streaming Data?
(Amazon Web Services)
4. Getting Started with Amazon Kinesis Data Streams
(Amazon Web Services)
5. AWS Kinesis Tutorial for Beginners [FULL COURSE in 65 mins]
(Johnny Chivers)
6. Query Your Data Streams in Real Time With Kinesis Data Analytics Studio | Amazon Web Services
(Amazon Web Services)
Top Articles
Latest Posts
Article information

Author: Corie Satterfield

Last Updated: 05/08/2023

Views: 5838

Rating: 4.1 / 5 (42 voted)

Reviews: 89% of readers found this page helpful

Author information

Name: Corie Satterfield

Birthday: 1992-08-19

Address: 850 Benjamin Bridge, Dickinsonchester, CO 68572-0542

Phone: +26813599986666

Job: Sales Manager

Hobby: Table tennis, Soapmaking, Flower arranging, amateur radio, Rock climbing, scrapbook, Horseback riding

Introduction: My name is Corie Satterfield, I am a fancy, perfect, spotless, quaint, fantastic, funny, lucky person who loves writing and wants to share my knowledge and understanding with you.