Azure in the Context of Data Science
Data Science is an intensely broad field, requiring a vast array of skills and presenting some pervasive patterns of challenges. In this blog, I will...
2 min read
Modern DevOps workflows increasingly rely on secure, automated authentication between CI/CD pipelines and cloud providers. GitHub’s OpenID Connect (OIDC) integration offers a powerful way to achieve this without hardcoding long-lived credentials in your repositories.
The CI/CD Pain Point We've All Experienced
If you've ever deployed a Microsoft Fabric workspace across Dev, Test, and Production environments, you know the drill:
- Deploy your notebooks and pipelines
- Manually update every hardcoded connection string
- Find and replace workspace GUIDs in 47 different places
- Update lakehouse references across data pipelines
- Cross your fingers and hope you didn't miss anything
- Watch as your Production pipeline fails because it's still pointing to Dev SQL Server
- Scramble to fix it during a critical deployment window
Sound familiar? There has to be a better way.
Enter Variable Libraries: The Game Changer
Microsoft Fabric's Variable Libraries feature is the secret weapon that transforms CI/CD from a manual nightmare into an automated dream. Think of them as environment-aware configuration stores that follow your workspace through every stage of deployment.
What Are Variable Libraries?
Variable Libraries in Fabric are workspace-level configuration artifacts that:
- Store key-value pairs (variables) organized by environment
- Automatically switch context based on workspace naming conventions
- Are accessible from both notebooks and data pipelines
- Support multiple value sets (Dev, Test, Prod, Custom)
- Deploy seamlessly through Fabric Git integration or deployment pipelines
The magic: Reference a variable once, and it automatically resolves to the correct value based on environment.
Real-World Example: ETL Framework Architecture
Let me show you how we implemented this in our Fabric ETL framework:
- Connections Library - External Source Connections
Problem: We need to connect to different SQL Server instances, Oracle databases, and APIs across environments.
Traditional Approach:
The pipeline example gets data from a SQL source using a Copy Task (red circled item in image).
The Connection to the source would be a fixed value selected from the list of configured connections. In the DEV environment this would be using the dev database connection, but would need to be changed to the appropriate server when deployed to the higher environments like TEST and PROD.This is a manual effort that would need to be done for all pipelines using a data connection, for each environment, every time it is deployed.
Variable Library Approach:
Using Variable Libraries that same pipeline’s Copy Task would look like this:
Now, instead of the hardcoded connection name, the Connection is dynamically set to read from the Variable Library
CI/CD Pipeline: Before and After
Before Variable Libraries
Deployment Steps (Manual,Error-Prone, 30+ minutes):
- Export workspace to Git
- Create release branch
- Find/replace all Dev connection strings with Prod strings (search 50+ files)
- Update lakehouse GUIDs in all the pipelines
- Update workspace references in nested pipeline calls
- Test everything manually
- Fix the things you missed
- Repeat steps 3-7 until it works
- Document all the places that need manual updates
- Train team members on the manual process
Result: Deployments are dreaded,infrequent, and high-risk.
After Variable Libraries
Deployment Steps (Automated,Reliable, 5 minutes):
- Export workspace to Git
- Deploy via Fabric Deployment Pipeline or Git integration
- Update Variable Libraries in target environment (one-time setup per environment)
- Done ✨
Result: Deployments are routine,frequent, and low-risk.
Setting Up Variable Libraries: Step-by-Step
Step 1: Create Variable Librariesin Your Workspace
# Navigate to workspace in FabricPortal
# Click "New" →"Variable Library"
# Name: "ConnectionsLibrary"
Step 2: Define Value Sets
Step 3: Add the connection entries to the variable library
Step 4: Add the variable library entries to the pipeline’s “Library Variables”
Step 5: Update the pipeline replacing the hardcoded connections with the Library Variable
References:
<ahref="https:>https://learn.microsoft.com/en-us/fabric/cicd/variable-library/get-started-variable-libraries?tabs=home-page</ahref="https:>
<ahref="https:>https://learn.microsoft.com/en-us/fabric/cicd/variable-library/tutorial-variable-library</ahref="https:>
<ahref="https:>https://learn.microsoft.com/en-us/fabric/cicd/variable-library/variable-library-overview</ahref="https:>
Understanding GitHub OIDC Tokens for Cloud Deployments
Modern DevOps workflows increasingly rely on secure, automated authentication between CI/CD pipelines and cloud providers. GitHub’s OpenID Connect...
-1.png)
Exploring SQL RAG: Revolutionizing GenAI with SQL Generation
William Richard : Jun 3, 2025 10:41:00 AM
By combining AI for unstructured and structured data assets, we are helping organizations unlock the full potential of their data, drive innovation,...