Making BPA Results Actionable with Automated PR Comments
Building your Power BI CI/CD pipeline with GitHub Actions (Part 3)
This is Part 3 of a series on building a Power BI CI/CD pipeline with GitHub Actions.
In Part 1, I showed you how to run the Best Practice Analyzer (BPA) with both standard and custom rulesets. In Part 2, we tackled the repetitive nature of BPA checks by using a C# macro to consolidate all findings into a single CSV output.
In this post, I’ll show you how to take that CSV output and make it actually useful by automatically analyzing the results and posting clear feedback directly on pull requests.
The Problem with “Exit Code 1”
When BPA detects issues, the workflow fails with this:
To my mind this creates several issues:
Requires familiarity with jargon - You need to know that “exit code 1” means a Severity 3 violation occurred. Not everyone on your team will understand this, especially junior developers or those new to the workflow.
No indication of what needs fixing - Even if you understand what "exit code 1" means, there's no clear indication of what the Severity 3 issues are (is there one or multiple?), and you still don't know what needs to be corrected and maybe more importantly why it needs to be corrected in the first place.
Requires additional clicking - You need to navigate to the Actions tab, find the workflow run, and if you use a solution similar to what I recommended in my previous post: locate artifacts, download a .zip file, extract it, and open the CSV to see what went wrong.
Even if we have a comprehensive CSV output with all violations as shown in Part 2, if users still need multiple clicks and technical knowledge just to understand what failed, we haven’t really solved the real problem. The information exists, but it’s not accessible.
Solution Overview
So what would actually solve this? The answer is bringing the information directly to where developers already are - in the pull request itself.
Automatically analyzes BPA results and categorizes violations by severity
Posts a structured, readable comment directly on pull requests
Gives developers immediate, clear feedback without requiring them to download artifacts or understand GitHub Actions internals or BPA code output
Provides links to rule descriptions, turning every code review into a learning opportunity
The core idea is simple: meet developers where they already are (in the PR) instead of making them hunt for information in workflow logs and CSV files. Here is a sneak peek example of what developers will see on the PR once the GA workflow has completed the analysis of the BPA result.
Want to see it in action? Check out this test PR to see the actual comment with all the collapsible sections, rule links, and severity categorization.
You can find all the scripts, the files required to build this solution, and the complete workflow in the GitHub repo.
The logical approach is to first analyze and categorize the raw BPA data, then format it into something human-readable. So we’ll start with the analysis script that does the heavy lifting, then move the result to the PR comment script that presents the results.
Step 1: Analyzing BPA Results
The first script enriches our BPA output CSV with custom severity levels from our ruleset. To do that, we need to create a connection between the result and the ruleset, that’s why we’ll add a new attribute to our BPA rule set, let’s call it “CustomSeverity” and more importantly, an ID column as well. The CustomSeverity field will allow us to:
Override default severities
Align rule importance with our organization’s standards
Control which issues block deployment vs. which are warnings
Categorizing by Severity
The script splits results into three categories:
Severity 3: Must Correct 🚨
Critical issues that block deployment. These represent problems that could lead to incorrect calculations, poor performance, or maintenance issues.
Severity 2: Correct ASAP ⚡
Important issues that should be fixed soon but don’t block deployment. These typically represent best practices that improve model maintainability and performance.
Severity 1: Nice to Have 💡
Suggestions for improvement. Quality-of-life enhancements that make the model better but aren’t urgent.
And once the split is done, it saves the result into separate CSV files based on severity.
The script also sets a GitHub Actions environment variable (SEVERITY_3_FOUND=true or false) that we'll use later in the workflow to determine whether to fail the build. Only Severity 3 issues will cause the workflow to fail.
Now we have an artifact at the end of our GA workflow which contains 3 separate CSV files - each with the rule violations by severity. I think this is a step further toward what we want as at least Power BI developers can now focus on what's really important (Severity 3 rule violations).
I think this is a step further toward what we want as at least Power BI developers can now focus on what's really important (Severity 3 rule violations) but if we stopped here in our solution, they still need to download and manually open the CSVs, which is not our goal.
Step 2: Let’s find a way to turn this into something useful for developers
Having categorized CSV files available as artifacts is better than nothing. But it’s still not ideal.
A typical workflow when a PR fails looks something like this:
1. Push changes to feature branch → 2. Create a PR → 3. Wait for checks to complete → 4. See a red X - workflow failed → 5. Click into the Actions tab → 6. Find the correct workflow run → 7. Scroll to find artifacts → 8. Download a .zip file → 9. Extract the .zip file → 10. Open the correct CSV file → 11. Finally see what went wrong → 12. Fix the issue
That's 12 steps just to understand what needs to be fixed, under the assumption that the developer:
Knows where to find artifacts in GitHub Actions
Understands which CSV file to check first when downloaded
Knows what the rule violations mean
Understands how to fix the issues
Based on my (own) experience, this will cause a problem, i.e.: delay and frustration. Developers may start seeing BPA checks as an obstacle. Probably the junior team members get frustrated and discouraged and senior developers will look for workarounds - and most probably find one which you won't appreciate later. The whole initiative loses momentum, purpose and support.
Direct feedback - in the PR discussion
At this stage I felt that I made good progress and it’s only a little thing missing, but that is the most important as well. The most straightforward way to improve the user experience is to take the weight of the analysis of the BPA result off from the shoulders of the developers and directly tell them where and what they should focus on in which priority order. We have all the knowledge (analysis result) we just need to shape it in a more interactable format. The easiert way to do this is to add a comment to the PR.
When BPA results are posted directly as a comment on the pull request, the workflow becomes:
1. Push changes to feature branch → 2. Create a PR → 3. Wait for checks to complete → 4. See a red X - workflow failed → 5. See detailed comment explaining what’s wrong → 5. Fix the issues
From 12 steps down to 5. From “figure it out yourself” to “here’s exactly what needs fixing.”
And I think this is what an automated process should look like. Other than being automated, it must provide users with clear indication and feedback about their work.
Why I think this matters:
1. Feedback on the platform where developers already are
If your team works in pull requests, they're already there to review code changes, read team comments, and check CI/CD status. Putting BPA results there means they don't need to context-switch to the Actions tab, download files, and interpret CSV data.
2. Clear Deployment Status
The comment starts with a direct answer::
❌ Model cannot be deployed until 3 errors are fixed.
or
✅ Model can be deployed. No Severity 3 (critical) issues found.
No interpretation needed of mythical “exit codes” to understand, instead a simple yes or no.
This is important when you have team members who aren’t deeply technical or who are new to Git workflows. They don’t need to become GitHub Actions experts to understand where the model needs more attention. Actually, they don’t need to know about GA workflows at all. They just need to know if their changes are ready.
3. Educational Value
Other than just the result of the analysis I also include details which are already in your BPA ruleset - and you can also add even more deatils, background info to this. On the PR comment, every rule violation includes a link to the rule description:
🔍 Avoid division (use DIVIDE function instead) — [Rule Description 🔗](link)When developers click that link, they see why the rule exists, what problems it prevents, and how to fix violations. To do this you’ll need a .md file with all the rule descriptions and useful resources. This adds an additional layer to BPA. Junior developers learn best practices through their work. More senior developers understand the reasoning behind standards.
Without PR comments, developers might fix the issue to pass the check, but it can take more time and they might not learn why it matters.
4. Less Support Overhead
I can easily imagine a scenario when you or someone who knows needs to spend a significant amount of time helping team members understand BPA failures by answering similar questions:
“My PR failed but I don’t know why”; “Where do I find the artifacts?”; “What does this rule mean?”; “Which issues do I actually need to fix?”
These questions happen less when the information is self-service and immediately visible. The PR comment answers most of the above questions:
Why did it fail? → See the deployment status message
What needs fixing? → See the Must Correct section
Which objects? → See the specific measures/columns listed
Why does it matter? → Click the rule description link
Can I deploy anyway? → No, Severity 3 blocks deployment
5. Built-In prgress tracker of the report status
PR comments create a record of what issues existed when the PR was created, which were fixed before merge, and what the model's health was at each stage. This is useful for onboarding new team members and improving processes over time.
With artifacts alone, this history is scattered across workflow runs. With PR comments, it's in the pull request thread.
Now that we've established why PR comments are essential, let's look at how the second Python script actually builds them.
Step 3: Building the PR Comment
The second Python script takes the categorized CSV files and builds a structured comment that’s posted directly to the pull request. The comment follows a clear hierarchy: deployment status at the top, followed by a summary table showing issue counts by severity, then rule-level summaries, and finally detailed violations in collapsible sections.
Severity 3 sections are expanded by default, while Severity 2 and 1 are collapsed. This keeps critical issues immediately visible without overwhelming the reader. The script also handles GitHub’s 65,000 character limit with a fallback strategy - if the comment is too long, it progressively collapses lower severity details while always keeping Severity 3 issues fully visible.
Additionally, the script automatically labels each PR (bpa-failed, bpa-warning, or bpa-passed) based on the results, making it easy to filter PRs by status.
For the complete comment structure and implementation details, check the README in the GitHub repo.
Wrapping Up
We’ve covered a lot of ground in this series. In Part 1, we were dealing with locally managed version chaos - manual file naming, no real history tracking. In Part 2, we automated BPA checks but still left developers hunting through CSV files to understand failures. Now, in Part 3, we’ve closed that loop: developers get immediate, actionable feedback right where they work.
The difference is about making quality checks feel like a helpful tool. When someone creates a PR now, they don’t need to become a GitHub Actions expert or know where artifacts are hidden. They see a clear answer: can this be deployed? What needs fixing? Why does it matter?
This is what I think good automation should look like. Going beyond “just” running checks automatically, but actually helping people do better work with less frustration. The BPA results were already valuable - we just needed to present them in a way that developers could actually use easily.
What’s Next
We now have automated BPA checks with clear feedback delivered directly on pull requests. But there’s more to consider when you want to scale this across multiple projects. Here is what I’d like to achieve:
In the next post, I’ll cover:
🗃️ Setting up a central repository - Maintaining one source of truth for BPA rules and scripts across all projects
🧰 Creating repository templates - Automatically setting up new projects with the same workflow
🎯 Controlling which checks run - Selectively triggering workflows based on PR content
📋 Pull request templates - Structuring PR information to enable smarter automation
Try It Yourself
Everything shown in this article is available in the GitHub repo:
📂 Repo: powerbi-cicd-with-githubactions-demos
👉Live example: View a test PR (set to fail on purpose) with the automated BPA comment in action.