Getting Started with Splunk in Azure

Splunk in Azure is a powerful combination that can help you unlock the full potential of your data. Splunk is a popular platform for machine data that can be used to monitor, report, and analyze data from various sources.

To get started with Splunk in Azure, you'll need to create a Splunk Cloud instance. This can be done through the Azure Marketplace, where you can choose from different deployment options, including a free trial.

The first step in setting up your Splunk instance is to select the right Azure region. This is important because it will determine the location of your data and the latency of your searches.

A unique perspective: Azure Splunk

In designing the network architecture for Splunk in Azure, it's essential to consider potential future growth to accommodate additional Splunk Enterprise VMs. This requires a network size of at least a /23 size to ensure sufficient IP addresses.

The Splunk Enterprise on Azure reference implementation requires segmenting the network into subnets for different components. Here's a breakdown of the subnets:

Segmenting the network into these subnets allows for the implementation of Network Security Groups to restrict communication between components to the required protocols and ports, reducing the risk of lateral movement in the event of a publicly available component becoming compromised.

Splunk in Azure has several key components that work together to provide a robust and scalable solution. A Search Head is the Splunk Enterprise instance that handles search management functions, directing search requests to Indexers and serving results back to the user.

The recommended VM family for Search Heads and the Search Head Deployer is the general-purpose Dds_v4 family. This is also the case for the Deployment Server, which is a Splunk Enterprise instance that acts as a centralised configuration manager for non-clustered Splunk components.

A Search Head cluster is deployed behind an Application Gateway for HTTP(S) access to the Splunk UI and load balancing of search traffic. The minimum number of Search Head VMs which can be deployed is 3, as this is the minimum requirement for a stable Search Head cluster.

Here's a quick rundown of the main components:

Outbound Connectivity is crucial for the smooth operation of your Splunk setup. If you're using internal load balancers for your Indexers, Deployment Server, and Syslog Receivers, you won't be able to access the internet via the Load Balancer.

This is because internal load balancers don't allow outbound access. To get around this, Splunk Enterprise on Azure reference implementation recommends deploying a separate external Load Balancer for the Indexers, Deployment Server, and Syslog Receiver VMs.

This is in line with Azure best practices, which aim to provide secure and reliable access to your resources.

To enable outbound access, you'll need to set up an external Load Balancer that can handle traffic from the internet. This will allow your Indexers, Deployment Server, and Syslog Receivers to install and configure Splunk without any issues.

Here are some key points to keep in mind when setting up outbound connectivity:

DNS Resolution is a crucial aspect of Splunk VMs, allowing for service to service communication without relying on IP addresses. This simplifies the process of replacing components without reconfiguring Splunk.

All Splunk VMs should be resolvable via DNS, which also enables load balancing connections to Indexers from Universal Forwarders or Heavy Forwarders. This is particularly useful in production deployments.

Check this out: Splunk Elasticsearch

The supplied Splunk Enterprise on Azure reference implementation creates a private DNS zone and automatically registers records for all VMs. This ensures that if a VM is rebuilt or replaced, the corresponding DNS record can be updated without requiring changes to Splunk configuration.

The DNS zone name is configurable with a deployment parameter, giving you flexibility in your setup.

A Search Head is the Splunk Enterprise instance that handles search management functions, directing search requests to the Indexers and then serving the results back to the user.

The recommended VM family for the Search Heads and the Search Head Deployer is the general-purpose Dds_v4 family. This is a crucial configuration decision to ensure maximum availability.

To achieve maximum availability, it is recommended to deploy Splunk Search Head Clusters across 3 Availability Zones. This provides a financially backed SLA of 99.99% uptime of the Search Head cluster.

The minimum number of Search Head VMs which can be deployed is 3, as this is the minimum requirement for a stable Search Head cluster which is resilient against the loss of a single Search Head VM.

The recommended Search Head replication factor is 3 to ensure availability of search artifacts. This is the default value set by Splunk.

Here are the recommended configurations for the Search Heads and the Search Head Deployer:

The Search Head Deployer typically does not need to be highly available, but it's essential to have a recovery plan for re-deploying the Search Head Deployer in case it becomes unavailable.

The Indexer is a crucial component in a Splunk Enterprise setup, responsible for indexing and storing data. It's recommended to enable HTTP Event Collection (HEC) on the Indexers for receiving events pushed via HTTP.

The Indexer configuration options are quite detailed, but some key settings to consider are the Cluster Master Size, which defaults to D16ds_v4, and the Indexer Size, which defaults to D64ds_v4. These settings determine the VM SKU for the Cluster Master and Indexers, respectively.

When deciding on the number of Indexers, it's worth noting that the default is 3. This can be adjusted based on the specific requirements of your setup. The Hot/Warm Volume Size is set to 1024 TB by default, and the Cold Volume Type is set to Standard HDD.

Discover more: Azure Cluster

Here's a summary of the Indexer configuration options:

The Cluster-wide replication factor is set to 3 by default, and the Cluster-wide search factor is set to 2. These settings can be adjusted based on the specific requirements of your setup.

The forwarder is a crucial component in the Splunk Observability Cloud platform, responsible for collecting and forwarding data to the Splunk platform.

You can configure the forwarder to deploy Heavy Forwarders, which are specialized forwarders that can handle large volumes of data. By default, Heavy Forwarders are not deployed, but you can change this setting to provision them.

The number of Heavy Forwarders to deploy is also configurable, with a default setting of 3. This setting can be adjusted to meet your specific needs.

Heavy Forwarder Size refers to the VM SKU (Virtual Machine SKU) used for these forwarders, which is set to D8ds_v4 by default.

You can also configure the number of pipelines per Heavy Forwarder, which is set to 2 by default. This setting determines how many ingestion pipelines are available for the Heavy Forwarders.

Explore further: Azure Dev Ops Pipelines

The forwarder can also be configured to provision Syslog Receivers, which are used to receive syslog data. By default, Syslog Receivers are not deployed, but you can change this setting to provision them.

Here is a summary of the forwarder configuration options:

Splunk in Azure offers a sizing calculator to estimate storage requirements based on expected daily ingest volume and retention. You can also adjust the raw compression factor and metadata size factor for a more accurate estimate if you're deploying an existing platform.

Splunk data will be compressed, but the size of hot/warm storage available will be defined by the size of the available local disk if you're using the Ls_v2 VM family as Indexers.

Storage Accounts are a separate service from Virtual Machines in Azure, and data will live on even after you delete the VM. This is because Storage Accounts have their own security and retention mechanisms.

In Azure, a source service can be configured to dump data into a separate storage account for retrieval, making it easy to access your data even after it's been deleted.

Curious to learn more? Check out: Access Azure Blob Storage

Deploying Splunk components across 3 Availability Zones provides maximum availability and data durability. This is the default configuration in the Splunk Enterprise on Azure reference implementation, with no option to override.

A proven recovery plan is essential for non-critical management components that don't offer native HA configuration. This includes components like Deployment Server, Monitoring Console, Cluster Master, and Search Head Deployer.

All Splunk configuration and knowledge objects should be persisted in a source control repository. This ensures that configuration can be recovered without loss in the event of a failure.

A standby instance of each component or a proven redeployment and restore process can serve as a recovery plan. This is a crucial step in ensuring high availability and disaster recovery.

Deploying Splunk Search Head Clusters across 3 Availability Zones provides a financially backed SLA of 99.99% uptime of the Search Head cluster. This is the recommended approach for maximum availability.

The Search Head replication factor should be set to the Splunk default value of 3 to ensure availability of search artifacts. This is a critical setting for maintaining high availability.

Having a recovery plan for re-deploying the Search Head Deployer is sufficient, as its unavailability will not impact Search Head cluster functionality.

To deploy Splunk in Azure, you'll need to follow a series of steps. First, click the button in the Azure Portal to launch the deployment experience. All configuration items are documented below for reference purposes, in addition to tooltips provided in the Azure Portal.

The recommended deployment process involves launching the deployment experience in the Azure Portal, which will guide you through the configuration process. Once the deployment is completed successfully, you'll receive the appropriate URLs to access the Splunk UI for the Search Head cluster, Monitoring Console, Cluster Master, and Deployment Server.

To automate the deployment process, you can run the script located in the .\scripts\azure-setup.ps1 file. Replace the variables at the top of the script with values from your environment, and then run the script. The script will prompt you to authenticate to your Azure subscription, and the output will provide the necessary configuration information.

Here is a summary of the deployment steps:

Azure Bastion provides secure, browser-based console access for remote administration and management purposes.

Azure Bastion can be used when implementing Splunk Enterprise on Azure without VM public IP addresses.

More information about Azure Bastion is available in its documentation.

The Splunk Enterprise on Azure reference implementation will deploy Azure Bastion by default when deployed without VM public IP addresses to ensure all instances are reachable regardless of private network connectivity.

Check this out: Enterprise Service Bus Azure

A Deployment Server is a Splunk Enterprise instance that acts as a centralised configuration manager for non-clustered Splunk components.

It's a crucial component in managing non-clustered Splunk instances, and it's included in the Splunk Enterprise on Azure reference implementation. The recommended VM family for the Deployment Server is the general-purpose Dds_v4 family.

The Splunk Enterprise on Azure reference implementation deploys a load balancer to manage deployment clients calls to the Deployment Server. Additional Deployment Server instances can be added to the load balancer backend pool based upon availability and scale requirements.

It's worth noting that the web server port for the Deployment Server has been updated to 8002 in the reference implementation to prevent cookie overwrite issues associated with sharing a single frontend host/backend port combination.

If you're planning to deploy a large number of deployment clients, it's essential to scale your Deployment Server accordingly.

Check this out: Azure Port

In an enterprise setting, a solid deployment strategy is crucial for a smooth and efficient experience. To set up Splunk Enterprise, you'll need to install the add-on using the latest package file from the GitHub releases page.

This step is essential to ensure you have the most up-to-date version of the add-on. It's always a good idea to check the GitHub releases page regularly for updates.

To configure the add-on's data inputs, you'll need to use the output from either the .\scripts\azure-setup.ps1 or .\scripts\azure-setup.sh script. This will help you set up the data inputs correctly.

Here's a simple step-by-step guide to help you through this process:

To install Splunk in Azure, you'll need to start by installing the Splunk Distribution of the OpenTelemetry Collector. This will give you access to the full benefits of the Splunk Observability Cloud platform.

Installing the collector is a crucial step, and you can find more information on how to do it by visiting the Get started with the Splunk Distribution of the OpenTelemetry Collector page.

You'll also need to set up Azure resources, such as an Azure Event Hub, Key Vault, and Azure AD Service Principal, to properly integrate Splunk with Azure. This requires some configuration, which can be accomplished using the scripts available in the .\scripts folder.

It's essential to implement secure access controls for your Splunk Enterprise environment in Azure. This means limiting direct access to approved sources only.

For public access, consider using ingress or load balancing solutions, such as HTTP Event Collector with a Public Load Balancer or Search Heads, Monitoring Console, and other management components with Application Gateway.

Check this out: Python Access Azure Blob Storage

Network Security Groups should be applied to limit access to approved sources only, preventing direct access to Splunk Enterprise VMs from the public internet.

Here are some common scenarios for public access:

By default, the Splunk Enterprise on Azure reference implementation deploys all VMs with private IP addresses only, but you can override this to provision public IPs on a per-component basis.

Public access to Splunk Enterprise on Azure should be implemented with caution. It's recommended to use private IPs only, except where access is required over public networks.

For public access, consider using ingress or load balancing solutions for security and high availability purposes. This can be achieved with HTTP Event Collector and a Public Load Balancer, or Search Heads and Monitoring Console with Application Gateway.

To limit access to approved sources only, apply Network Security Groups to prevent direct access to Splunk Enterprise VMs from the public internet.

For your interest: Access Connector for Azure Databricks

By default, the Splunk Enterprise on Azure reference implementation deploys all VMs with private IP addresses only. However, you can override this to provision public IPs on a per-component basis, for example, to simplify access to a test/development environment.

It's not recommended to enable direct access to Splunk VMs via the public internet for security and privacy reasons, unless you have an explicit requirement to receive Splunk data directly from public networks.

To access the Splunk UI on the Search Head cluster, Deployment Server, Monitoring Console, and Cluster Master, an Application Gateway is deployed by default. This will not be publicly accessible, but you can limit access to a specific source IP range.

A Standard Load Balancer will be provisioned if HEC is selected at deployment time, providing an option to expose a public IP for HEC data without directly exposing Indexer VMs to the public internet.

Here are some public access options:

Remember to configure a TLS listener with a valid certificate on the Application Gateway for added security.

Windows Users need to be aware of the different types of accounts available, such as Administrator, Standard User, and Guest accounts.

The Administrator account has full control over the system, including the ability to install software and modify system settings.

A Standard User account, on the other hand, has limited access to system settings and can only install software that is specifically designed for Standard Users.

Guest accounts are temporary accounts that allow visitors to use a computer without having access to any personal files or settings.

Windows Users can also use the User Account Control (UAC) feature to limit the actions that can be performed by Standard Users and Guest accounts.

Consider reading: How to Use Microsoft Azure

Accessing data is a crucial part of working with Splunk. To access Azure data in Splunk, you can use add-ons like the Splunk Add-on for Microsoft Cloud Services and the Microsoft Azure Add-on for Splunk.

These add-ons help make Azure data available to Splunk. The Splunk Add-on for Microsoft Cloud Services is a great option for getting started, as it provides a simple way to access Azure data.

A different take: Microsoft Windows Azure Services

To access live Splunk data in Azure Data Factory, you'll need to follow a series of steps. This involves creating a linked service and setting up the connection settings.

Here are the steps to create a linked service in Azure Data Factory:

After creating the linked service, you can preview the data and see the imported Splunk table. This will give you a chance to verify that the data is being accessed correctly.

By following these steps, you can access live Splunk data in Azure Data Factory and use it to create data flows.

To ingest data into Splunk in Azure, you'll need to use add-ons. The two main add-ons used are the Splunk Add-on for Microsoft Cloud Services and the Microsoft Azure Add-on for Splunk. These add-ons allow you to access Azure data.

For high availability, Splunk recommends using Splunk Connect for Syslog, a container-based solution for collecting syslog messages and forwarding to the HTTP Event Collector. This is especially important for larger environments.

The recommended VM family for Syslog Receivers is the general-purpose Dds_v4 family.

Additional reading: Add Azure

Syslog Receivers are a crucial part of ingesting syslog data into Splunk.

For high availability in larger environments, Splunk recommends using Splunk Connect for Syslog, a container-based solution for collecting syslog messages and forwarding to their HTTP Event Collector.

The recommended VM family for Syslog Receivers is the general-purpose Dds_v4 family.

Splunk Enterprise on Azure reference implementation supports optionally deploying syslog receiver nodes, running Splunk Connect for Syslog which is configured to forward logs to the HTTP Event Collector.

If syslog receivers are required, HTTP event collection must be enabled.

You might enjoy: How to Connect to Aks from Azure Portal

In Azure Data Factory, you can access live Splunk data by following a series of steps. To start, you'll need to log in to Azure Data Factory.

To establish a connection from Azure Data Factory to the CData Connect Cloud Virtual SQL Server API, you'll need to create a linked service. This involves selecting SQL Server from the search bar and creating a new linked service.

On a similar theme: Azure Data Studio Connect to Azure Sql

When creating the linked service, you'll need to enter the connection settings, including the Name and Linked service fields. You'll also need to select the Table name from the available options and import the schema from the connection or store.

After creating the linked service, you can click preview data to see the imported Splunk table. This will give you a sense of what data is available and how it's structured.

Here are the steps to create a linked service in Azure Data Factory:

Azure Integration is a key aspect of setting up Splunk in Azure. To ensure seamless connectivity, it's essential to understand the implications of using Private IPs only for Indexers.

Private IPs only mean Indexers will not be accessible via the public internet, and data cannot be sent via Splunk to Splunk protocol over the internet. To send data from Universal Forwarders to Indexer VMs via public internet connectivity, Indexer VMs need to have Public IPs.

Consider reading: Via Azure

Consideration needs to be given on how to allow the flow of traffic from the virtual network where ExpressRoute or your VPN terminates to the Splunk virtual network. Peering the two virtual networks is a common approach to achieve this connectivity.

A hub-spoke network topology, as seen in the Azure hub-spoke reference architecture, is a common pattern to achieve this connectivity. Here are the main components that can be integrated with Splunk in Azure:

The Splunk Add-on for Microsoft Cloud Services integrates with these components, providing access to activity data, alerts, authentication data, and more.

Private connectivity is a crucial aspect of integrating Splunk Enterprise with your Azure environment.

To use Private IPs only, Indexers won't be accessible via the public internet, meaning data can't be sent via Splunk to Splunk protocol over the internet.

If you need to send data from Universal Forwarders to Indexer VMs via public internet connectivity, Indexer VMs will need to have Public IPs.

If this caught your attention, see: Azure Openai Private Link

Network load balancers aren't supported for load balancing Splunk to Splunk traffic, except for HTTP Event Collector traffic, which can and should be sent via a load balancer for high availability.

To send data from on premises to Splunk Enterprise in Azure via ExpressRoute or VPN, you'll need to consider how to allow traffic flow from the virtual network where ExpressRoute or your VPN terminates to the Splunk virtual network.

Peering the two virtual networks is a simple way to achieve this connectivity.

To configure ADF (Azure Data Factory) connectivity, you'll need to create a connection to your Splunk data source. This is done using CData Connect Cloud, which provides a point-and-click interface to connect to data sources.

First, log into Connect Cloud and click on Connections. From there, click Add Connection to start the process. Select "Splunk" from the Add Connection panel to begin the configuration.

To authenticate requests, you'll need to set the User, Password, and URL properties to valid Splunk credentials. The port used for requests to Splunk is port 8089, and the data provider uses plain-text authentication by default.

If this caught your attention, see: Azure Service Connection

Here's a step-by-step guide to creating the connection:

With the connection configured, you're ready to connect to Splunk data from Azure Data Factory.

The Microsoft Cloud Add-on is a powerful tool for integrating your Azure resources with Splunk. It provides access to a wide range of data, including activity data and alerts, authentication data, and NSG flow logs.

One of the key benefits of the Microsoft Cloud Add-on is its ability to integrate with various data sources, such as Event Hubs, storage accounts, and the activity log. This allows you to collect and analyze data from multiple sources in a single place.

The Splunk Add-on for Microsoft Cloud Services integrates with Event Hubs, storage accounts, and the activity log, providing access to resource data, authentication data, cost and consumption, and metrics. You can even get the activity log via the REST API or Event Hub – it's the same data either way.

Here are some of the data sources that the Microsoft Cloud Add-on integrates with:

This integration enables you to collect and analyze data from your Azure resources, providing valuable insights into your cloud usage and performance.