Bacancy

Bacancy represents the connected world, offering innovative and customer-centric information technology experiences, enabling Enterprises, Associates and the Society to Rise™.

12+

Countries where we have happy customers

1050+

Agile enabled employees

World wide offices

12+

Years of Experience

Agile Coaches

Certified Scrum Masters

2500+

Clients projects

1458

Happy customers

Azure

How to Connect Logic Apps with Blob Storage and Setup Monitoring & Alerting on Application Logs

Last Updated on August 28, 2024

Introduction: Connect Logic Apps with Blob Storage and Setup Monitoring

Logs play a role in an application’s life cycle, providing insights that range from tracking everyday operations to diagnosing problems and understanding user behavior. They reveal what’s happening behind the scenes, but merely storing logs isn’t enough. Therefore, it is essential to keep an eye on these logs and receive alerts or notifications whenever issues arise.

If your application already stores data in a storage service, like Amazon S3 or Azure Blob Storage, you can take advantage of this setup. On my blog today, I want to dwell on how Logic Apps with Blob Storage helps you. By deploying Azure Logic Apps, you will be able to establish a good monitoring and alerting system for your logs to catch problems at the earliest stage and respond immediately to keep your application running smoothly.

In this guide, we will learn how to connect Logic Apps with Blob Storage, create an automated workflow using Azure Logic Apps to monitor your logs for specific events—like errors or performance issues—and instantly alert you when action is needed.

Section 1: Understanding Azure Blob Storage and Azure Logic Apps

Before we deep dive into monitoring and alerting, it’s important to lay some groundwork. In this section, we will explore the significance of Azure Blob Storage in managing logs. Delve into the basics of Azure Logic Apps to gain a grasp of how they interact with each other.

What is Azure Blob Storage?

Microsoft Azure offers Blob Storage as a service tailored to store data quantities in the cloud. The term “blob” represents a “Binary Large Object,” making it suitable for storing data formats like text, images, videos, documents, and binary data. Here’s an in-depth look at Blob Storage and its functionality;

What is Logic Apps?

Containers:
Data in Azure Blob Storage is organized into containers. Each container can hold unlimited blobs, and one Azure storage account can have multiple containers. Containers group up blobs like directories do in a file system.

Blobs:
A blob is the actual object that is stored inside Azure Blob Storage. Blobs fall into three categories:

Block Blobs: These are meant for storing texts and binary files. They are ideal for large files like videos or images because they can be divided into blocks that can be uploaded individually or in parallel.
Append Blobs: These resemble block blobs but are explicitly designed for appending purposes. This type of storage is often used for logging data since it allows you to add new information without altering any existing content.
Page Blobs: These are designed for random write/read operations and usually store Virtual Hard Disk (VHD) files for Azure virtual machines.

Storage Tiers:
Azure Blob Storage offers various access tiers to optimize performance and cost depending on the frequency of data accessed:

Hot Tier: This tier is for data that is frequently accessed. It has low latency but a relatively high price compared to other levels.
Cool Tier: This is for data that is accessed less frequently and stored for at least 30 days. It is cheaper than the Hot Tier but comes with higher access costs.
Archive Tier: This tier is for data that is rarely accessed and stored for a long period (at least 180 days). It offers the lowest storage cost but with higher latency and retrieval costs.

Benefits of Azure Blob Storage

Scalability: Blob Storage can scale to store vast amounts of data without infrastructure limitations, making it suitable for growing needs.
Durability and Availability: Azure spreads data across sites, providing a range of replication choices for resilience and accessibility even in the face of setbacks.
Security: Blob Storage safeguards your information with security elements such as encryption, access control through Azure Active Directory shared access signatures and network limitations.
Cost-Effectiveness: By offering storage tiers and pricing structures, Blob Storage assists companies in managing expenses efficiently by storing data based on usage trends.
Integration and Accessibility: Blob Storage integrates with Azure services and is accessible via REST APIs, SDKs, and the Azure portal, supporting powerful workflows and data processing.
Data Management and Lifecycle Policies: Azure’s lifecycle management policies automate data transitions between storage tiers or deletion, optimizing storage costs and efficiency.

Log File in Blob Storage

Your logs might be stored in various formats such as JSON, CSV, or plain text. For instance, a JSON log file might look like this:

This log entry includes essential details like the timestamp, severity level, and error message, which are crucial for effective monitoring.

What is Logic Apps?

Azure Reasoning Applications is a cloud-based solution that allows you to automate processes and work with jobs throughout systems, applications, and solutions. It offers a user approach that allows individuals to design workflows without extensive coding knowledge. Here’s an in-depth overview;

Key Components of Logic Apps

Workflows:
Logic Apps revolve around workflows, consisting of steps detailing the actions and logic needed to finalize a process. Events like receiving an email, uploading a file, or reaching a time can activate these workflows.

Connectors:
Logic Apps offer a selection of connectors that enable you to connect with a variety of services both, within Azure and externally. These connectors are made of integrations with platforms such as Microsoft 365 Salesforce, Twitter SQL Server and more facilitating smooth communication, between different systems.

Triggers:
A trigger is an occasion that begins a process in Reasoning Application. Induces can be time-based (e.g. every hr), event-based (e.g. when a data is contributed to a storage space account) or handbook (e.g. started by an individual). Each process begins with a solitary trigger.

Actions:
Actions are the steps taken after a trigger fires. These actions encompass tasks such, as sending emails making API calls handling data processing or executing business operations. Actions function as the components that determine what occurs within a workflow.

Benefits of Azure Logic Apps

Visual Designer: Logic Apps provide a drag-and-drop interface in the Azure portal, making it easy to build workflows without extensive coding knowledge.
Pre-Built Templates: A library of pre-built templates helps you quickly create workflows for common tasks like notifications and data integration, speeding up deployment.
Scalability: Logic Apps automatically scale to handle anything from simple tasks to complex processes, ensuring smooth operation as demand fluctuates.
Integration with Azure Services: Seamlessly integrate with Azure services like Functions and Blob Storage, allowing you to create flexible, powerful workflows within the Azure ecosystem.
Monitoring and Management: Built-in tools let you monitor workflows, track logs, and manage updates, ensuring reliability and easy troubleshooting.
Cost-Effectiveness: With pay-as-you-go pricing, you only pay for executed actions, making it an affordable solution without the need for server management.
Security and Compliance: Logic Apps follow Azure’s security protocols, offering encryption and Azure AD integration, while also supporting industry compliance standards.

How Logic Apps Work

Azure Logic Apps automate workflows by starting with a trigger (like an event or schedule), followed by a series of actions. These actions can interact with various services using connectors, which allow Logic Apps to integrate with both Azure and external platforms.

Trigger: Starts the workflow (e.g., event, schedule).
Actions: Tasks executed in sequence (e.g., send email, process data).
Connectors: Integrate with Azure and external services.
Conditions/Loops: Add logic and repetition to workflows.
Execution: Runs automatically based on triggers.
Monitoring: Real-time tracking and management.

Section 2: Connecting Logic Apps with Blob Storage

Creating the Logic App

To begin, connecting Logic Apps with Blob Storage, you’ll need to create a new Logic App in the Azure portal:

1. Go to the Azure portal.
2. Use the search box to find “Logic Apps” – select the option and Press the + sign to create a new Logic App

When you’re getting started with Azure Logic Apps, it’s crucial to select the pricing plan that suits your requirements. Azure provides a range of pricing options based on how you intend to utilize Logic Apps. Here’s a comparison of the pricing models;

Azure Logic Apps offers two distinctive pricing models – Consumption-Based and Standard.

Feature	Consumption-based	Standard
Billing model	Pay-as-you-go	Fixed Cost
Resources	Execution, actions, storage	vCPUs, Memory
Usage Pattern	Unpredictable, cost-sensitive	Consistent, dedicated
Best for	Workflows with unpredictable usage patterns or that are sensitive to cost	Workflows with consistent usage patterns and that require dedicated resources

We will be considering the Consumption plan for this particular how-to connect Logic Apps with Blob Storage guide. It offers flexible pay-as-you-go pricing. This approach is beneficial for situations where your usage may fluctuate, making it cost-effective for monitoring and alerting based on application logs.

3. To create the logic app – provide all necessary details such as Name and Resource group, Hosting Plan (Consumption or Standard).

4. Once you are done with the Logic App creation – you will be redirected to the Logic App Designer.

Setting Up The Blob Storage Trigger

After creating a Logic App, it is time to move on to the next step: setting up a trigger in blob storage. A trigger that will activate the Logic App whenever a new log file is uploaded to Blob Storage;

1. Go to the Logic App Designer and select “Azure Blob Storage” from the multiple triggers available.
2. Choose the “when a blob is added or modified” trigger.
3. Configure the trigger to monitor the specific container where your application logs are stored.

Please input the name of the connection and choose the type of authentication. Based on the chosen authentication method, fill in the information for your Blob Storage. This connection between Logic Apps with Blob Storage guarantees that your Logic App will be activated whenever a new log file is added, or an existing one is modified in Blob Storage.

If you find it difficult to connect Logic Apps with Blob Sotrage you can seeks assistance from our cutting edge Azure Integration Services to perform this integration.

Section 3: Defining Monitoring Conditions

Understanding Your Log Data

Before defining monitoring conditions, it’s essential to understand the structure of your log data. Logs might typically include:

Timestamps: Indicating when the log entry was created.
Log Levels: Severity indicators like INFO, WARNING, or ERROR.
Messages: Descriptions of events or errors.

Example of a Typical JSON Log Entry

{
    "timestamp": "2024-08-21T14:30:00Z",
    "level": "INFO",
    "message": "User login successful",
    "user": {
        "id": "12345",
      },

Accessing and Parsing Log Data

Once the Logic App is triggered, the next step is to access and parse the log data:

1. Get Blob Content: Use the “Get Blob Content” action to retrieve the log file’s contents that triggered the Logic App.

Add a new action of ‘Get blob content’ and select the appropriate Container and blob.

2. Parse the Log Data: If your logs are in JSON format, use the “Parse JSON” action to extract key fields like timestamp, level, and message. For plain text logs, you can use the “Compose” action combined with expressions to break down and analyze the text.

Now create a new action called ‘Parse JSON’ and in that define the Content and JSON Schema.

In the content field we have to provide the content obtained from the above action that is Get Blob Content. By Default Get Blob Content data is Base64-encoded (which is common when dealing with binary or large text files), you’ll need to decode it before you can parse it. To do this, use the following expression in the “Compose” action:

decodeBase64(body('Get_blob_content_(V2)').$content)

This expression decodes the content retrieved by the “Get Blob Content (V2)” action, converting it back into a readable format.

After decoding, if your logs are in JSON format, use the “Parse JSON” action to extract key fields such as timestamp, level, and message. Provide the JSON schema based on your log structure to help Logic Apps parse the data effectively.

Next, you’ll need to define the schema for your JSON log file. You can easily generate this automatically by selecting the ‘Use sample payload to generate schema’ option. Simply paste your JSON data, and the schema will be created for you.

Setting Up Monitoring Conditions

After connecting Logic Apps with Blob Storage, let’s define the conditions that will trigger alerts:

1. Add a Condition Action: Use the “Condition” action to evaluate specific fields in your log data.

Log Level Monitoring: Set up a condition to trigger when the level field is “ERROR” or “WARNING”.

In content, use the Parse JSON data to check for the condition.

Advanced Expressions: For more complex scenarios, use logical operators (AND/OR) to combine multiple conditions.

Example Condition
Condition: If $.level equals “ERROR.”

Handling True or False Conditions

Logic Apps allows you to branch your workflow based on the outcome of your conditions:
True Path: If the condition is met, you might trigger an alert, log additional details, or escalate the issue.
False Path: If the condition isn’t met, you can either end the workflow or continue with other checks.

For example, if an error is detected, you could send an immediate alert to your team, whereas if everything is normal, the Logic App could simply log a confirmation message.

Section 4: Setting Up Alerts

Email alerts are an efficient method to inform your team about any issues found in your application logs. Using Azure Logic Apps with Blob Storage, you can easily set up these notifications to be delivered directly to your Outlook inbox, ensuring that important alerts reach you promptly.

Steps to Configure Outlook Email Alerts:

Add the Outlook Actions
Once you have established the Logic App conditions, include an action for sending an email through Outlook. Look for “Office 365 Outlook” in the Logic Apps Designer. Choose the “Send an email (V2)” function.

Compose the Email Content
? Configure the email details:

To: Input the email address(es) of the Recipient(s), those who need to receive the alert. You can list addresses by separating them with semicolons.
Subject: Craft a clear and concise subject line that indicates the nature of the alert. For example, “Critical Error Detected in Application Logs.”
Body: Customize the email body to include relevant details from the log entry. Use dynamic content to insert fields such as the error message, timestamp, and other metadata.

By setting up email alerts, you ensure that your team is immediately informed about critical issues, allowing for swift action to maintain the health of your application.

Example Notification:

Section 5: Testing and Optimization

Testing Your Logic App

After setting up your Logic App, it’s crucial to test it to ensure it works as expected:
- Upload Test Logs: Upload a log file containing known issues to Blob Storage. Verify that the Logic App triggers and that notifications are sent correctly.
- Review Execution Logs: Check the logs generated by the Logic App itself to ensure all steps are executed properly.
Optimizing for Performance and Cost

To keep your Logic App running efficiently and cost-effectively:
- Minimize Triggers: Only trigger the Logic App when necessary, such as when a new log is added. This reduces unnecessary executions.
- Batch Processing: If your logs are frequent, consider processing multiple logs in a single run to reduce execution frequency.
- Monitor Costs: Regularly review the costs associated with running the Logic App, especially if you’re using multiple integrations or sending numerous notifications.
Conclusion

By utilizing Azure Logic Apps to oversee and notify about your application logs stored in Azure Blob Storage, you can transform your logs into a tool for managing your application’s well-being. This configuration ensures alerts for any issues, enabling responses to maintain the smooth operation of your application.

With this automated monitoring and alert system established, you can reduce time spent sorting through logs and allocate time to enhancing and refining your application. Are you prepared to leverage Azure Logic Apps for the lifting? Leverage our Azure Consulting Serrvices to begin setting up your Logic App today and elevate your log monitoring capabilities!