I want to document ADF pipelines runs / triggers

If you want a clean static snapshot of your static metadata layout (mapping which pipeline handles which sources/targets and how triggers affect them) without writing complex logs or KQL from scratch, it can be a massive manual headache.

I actually got so tired of reverse-engineering nested JSON/ARM templates to map this out that I engineered a lightweight utility called the ADF Pipeline Analyzer. You drop in your raw pipeline definition or ARM template, and it instantly spits out:

📄 1-click Microsoft Excel & Word reports detailing parameters, configurations, and sources/targets.

🗺️ Visual source-to-target dataset lineage flows (filtering out the background orchestration noise).

📊 Visual dependency graphs showing exactly how control flows execute.

I recently posted a full breakdown of the parsing engine logic on LinkedIn along with a live free link to the application if you want to save time drafting your documentation framework.

You can check out the step-by-step breakdown and grab the tool details here: 👉 https://www.linkedin.com/posts/tejasdalbhanjan_enterprise-adf-analyzer-share-7463118148851011584-X6Qb?utm_source=share&utm_medium=member_desktop&rcm=ACoAAC6Yjq4BHxCD9WPLz8XVWethA8QTvgzHbx0

Hi @Duyen Su

To document Azure Data Factory (ADF) pipeline runs and triggers in Excel or Power BI for reporting purposes, including details like data usage, source, target, and trigger times, you can leverage ADF's monitoring capabilities, REST API, or Power Query to extract the required metadata.

Extract Pipeline Run info:

ADF Monitoring UI: Navigate to the "Monitor" tab in ADF Studio to view pipeline runs. You can filter by time range, status, or pipeline name and export the results manually to CSV for Excel. However, this is limited for automation.

ADF REST API: Use the Pipeline Runs - Query By Factory endpoint to programmatically fetch pipeline run details. This API returns run ID, pipeline name, start/end time, status, and parameters.

Power BI Integration: Use Power BI's Power Query to call the ADF REST API and transform the data into a report. This is ideal for automated, refreshable reports.

Extract Trigger Information

ADF Monitoring UI: Go to the "Monitor" tab, select "Trigger Runs," and view trigger details like trigger name, type, and execution times. Manual export to CSV is possible but not scalable.

ADF REST API: Use the Trigger Runs - Query By Factory endpoint to retrieve trigger run history, including trigger name, type, and start time.

Trigger Metadata: To get trigger schedules (e.g., frequency, interval), query the Triggers - Get endpoint or export the trigger JSON definitions from ADF's "Author" tab. The JSON includes recurrence details (e.g., hourly, daily) and associated pipelines.

Capture Source, Target, and Data Usage

Datasets and Activities: In ADF, pipelines reference datasets (source and sink) in activities (e.g., Copy Activity). Extract dataset details from the pipeline JSON or use the Datasets - List By Factory endpoint to get source (e.g., Azure Blob Storage) and target (e.g., Azure SQL Database) information.

Data Usage Documentation: Manually annotate the purpose of each pipeline (e.g., "ETL for sales reporting") in a separate metadata table or column. Alternatively, use pipeline annotations or parameters to store this information in ADF and extract it via the REST API.

Power BI: Combine dataset details with pipeline run data in Power BI to create a comprehensive report showing source, target, and usage.

Automate Data Extraction for Excel or Power BI

• For Excel:
  1. Use Azure Logic Apps or a Python script to call the ADF REST API and write results to an Excel file stored in Azure Blob Storage.
  2. Use Power Query in Excel to connect to the REST API directly and refresh data manually.
• For Power BI:
  1. Create a Power BI dataflow or dataset using Power Query to call the ADF REST API.
  2. Transform the JSON response to extract relevant fields (e.g., run ID, pipeline name, trigger time, source, target).
  3. Build visualizations (e.g., tables, Gantt charts) to show trigger schedules, run history, and data flow.

Ensure you have a Service Principal with Contributor access to ADF for API calls. Authenticate using the App ID and Secret Key.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.