Data collection rules (DCRs) in Azure Monitor

Data collection rules (DCRs) are part of an Extract, transform, and load (ETL)-like data collection process that improves on legacy data collection methods for Azure Monitor. This process uses a common data ingestion strategy for all data sources and a standard method of configuration that's more manageable and scalable than previous collection methods.

For many monitoring scenarios, you don't need to understand how a DCR is created or assigned. You can simply use guidance in the Azure portal to enable and configure data collection, while Azure Monitor creates and configures the DCR for you. This article provides more details about how DCRs work to get you started on creating and configuring them manually so that you can customize the data collection process.

Specific advantages of DCR-based data collection include:

• Consistent method for configuration of different data sources.
• Ability to apply a transformation to filter or modify incoming data before it's sent to a destination.
• Scalable configuration options supporting infrastructure as code and DevOps processes.
• Option of Azure Monitor pipeline in your own environment to provide high-end scalability, layered network configurations, and periodic connectivity.

View DCRs

Data collection rules (DCRs) are stored in Azure so you can centrally deploy and manage them like any other Azure resource. They provide a consistent and centralized way to define and customize different data collection scenarios.

View all of the DCRs in your subscription from the Data Collection Rules option of the Monitor menu in the Azure portal. Regardless of the method used to create the DCR and the details of the DCR itself, this screen lists all DCRs in the subscription.

Replace legacy data collection methods

The data collection process that uses DCRs replaces other data collection methods in Azure Monitor. The following table lists legacy methods and their DCR-based replacements. In the future, DCRs are expected to replace other data collection methods in Azure Monitor.

Data collection process

The data collection process that DCRs support provides a common processing path for incoming data. Each data collection scenario is defined in a DCR. The DCR provides instructions for how Azure Monitor should process the data it receives. Depending on the scenario, DCRs specify all or some of the following items:

• Data to collect and send to Azure Monitor.
• Schema of the incoming data.
• Transformations to apply to the data before it's stored.
• Destination where the data should be sent.

Data collection rule associations (DCRAs)

Create data collection rule associations (DCRAs) between the resource and the DCR to enable certain data collection scenarios. This relationship is many-to-many, where you can associate multiple resources with a single DCR and associate up to 30 DCRs with a single resource. Develop a strategy for maintaining your monitoring across sets of resources with different requirements.

Using a DCR

Once a DCR is created, there are different methods to use it based on the data collection scenario. The following table lists the common scenarios and the method used to collect data in each case. Further details on each are provided below.

Scenarios

The following sections describe common scenarios for using DCRs to collect data in Azure Monitor. They describe the details included in the DCR and the method used to specify which DCR to use for that particular scenario.

Azure Monitor agent (AMA)

Use Azure Monitor agent (AMA) to collect data from virtual machines and Kubernetes clusters. The following diagram shows how AMA collects data when running on a virtual machine. When you install the agent, it connects to Azure Monitor to retrieve any DCRs that are associated with it. In this scenario, the DCRs specify events and performance data to collect. For a Kubernetes cluster, this collection also includes Prometheus metrics. The agent uses that information to determine what data to collect from the machine and optionally apply a client-side transformation (Preview) to filter and transform the data before sending it to Azure Monitor. Once the data is sent, any ingestion-time transformations specified in the DCR run to filter and modify the data further. Then Azure Monitor delivers the data to the specified destination.

For more information, see Collect data from virtual machine client with Azure Monitor and Enable monitoring for Kubernetes clusters.

Event hubs (Preview)

The following diagram shows how data is ingested into a Log Analytics workspace directly from Event Hubs. When the event hub receives data, it delivers the data to Azure Monitor. Azure Monitor then transforms the data and sends it to any destinations specified in any DCRs associated with it.

For more information, see Ingest events from Azure Event Hubs into Azure Monitor Logs (preview).

Platform metrics (Preview)

Azure resources automatically collect platform metrics and send them to Azure Monitor Metrics. The following diagram shows the process of using a DCR to send this data to a Log Analytics workspace for analysis using log queries. This process replaces the current method of using diagnostic settings to perform this function.

When the DCR is created, it specifies the workspace and table where the data should be sent. The DCR also includes a transformation that ensures the data is in the correct format for the target table. The DCR is then associated with the resource from which the platform metrics are collected.

For more information, see Metrics export using data collection rules.

Direct ingestion

Use direct ingestion to specify a particular DCR to process the incoming data. For example, the following diagram illustrates data from a custom application using Logs ingestion API. Each API call specifies the DCR that processes its data. The DCR understands the structure of the incoming data, includes a transformation that ensures the data is in the format of the target table, and specifies a workspace and table to send the transformed data.

For more information, see Logs ingestion API.

Workspace transformation DCR

Workspace transformation DCRs provide transformations for data collection that don't use a DCR. They're applied directly to the Log Analytics workspace and are automatically activated when they're created.

For more information, see Workspace transformation DCR.

Transformations

Transformations are KQL queries included in a DCR that run against each record received. They allow you to modify incoming data before it's stored in Azure Monitor or sent to another destination. Filter unneeded data to reduce your ingestion costs, remove sensitive data that shouldn't be persisted in the Log Analytics workspace, or format data to ensure that it matches the schema of its destination. Transformations also enable advanced scenarios such as sending data to multiple destinations or enriching data with additional information.

Multi-stage transformations (Preview) allow you to string together multiple transformations in a single DCR, where the output of one transformation is the input of the next. The key components of multi-stage transformations are:

• Client-side transformations assigned to the data source
• Ingestion-time transformations which run in Azure Monitor assigned to the data flow

DCR regions

Data collection rules are available in all public regions where Log Analytics workspaces and the Azure Government and China clouds are supported. Air-gapped clouds aren't yet supported. You create and store a DCR in a particular region, and the service backs it up to the paired-region within the same geography. The service is deployed to all three availability zones within the region. For this reason, it's a zone-redundant service, which further increases availability.

Single region data residency is a preview feature to enable storing customer data in a single region and is currently only available in the Southeast Asia Region (Singapore) of the Asia Pacific Geo and the Brazil South (Sao Paulo State) Region of the Brazil Geo. Single-region residency is enabled by default in these regions.

Related content

For more information on how to work with DCRs, see:

• Data collection rule structure for a description of the JSON structure of DCRs and the different elements used for different workflows.
• Sample data collection rules (DCRs) for sample DCRs for different data collection scenarios.
• Create and edit data collection rules (DCRs) in Azure Monitor for different methods to create DCRs for different data collection scenarios.
• Data collection endpoints (DCE) overview for the full context required for direct ingestion using the Logs Ingestion API.
• Azure Monitor service limits for limits that apply to each DCR.