I recently helped one of my customers integrate Splunk and Azure. I thought it would be worth putting a blog article together to talk about what we did, list out some gotchas, and help others who are trying to accomplish the same thing.

Why use Splunk if Azure has got OMS, Log Analytics, Monitor, and more?

Not every project starts as a pure cloud application on Azure. More often than not you are working in a hybrid scenario where Splunk might already be deployed.

Yes, OMS can monitor on premises machines, but while it fits most hybrid scenarios, its price point might not.
Especially in situations where there is a large machine to people ratio, I see a lot of businesses evaluating other options.
It – however – looks like the OMS team has heard this feedback and is adjusting their pricing to reflect scenarios like that.

Last but not least; it can take a while for businesses to migrate between solutions, so even if OMS is the end goal, there might be a sizeable amount of time that people will spend on their existing solution, which is when integration is key.

It’s important to understand that different Enterprise monitoring solutions fit different scenarios. I am not recommending Splunk or OMS or any other solution in particular in this article. You need to evaluate different options to find the solution that best fits your scenario.

Configuring Splunk on a Windows Server

Splunk runs on both Linux and Windows.
If you are looking to set the Windows “flavour” up in a test environment, make sure the following things are in place.

  • python in PATH variable
    • This does not always get added properly by default
  • pip in PATH variable
    • same as above
  • %SPLUNK_HOME%
    • This should get set during the installation, but it is worth doublechecking whether it is there and set to your Splunk install folder
    • It was not in my case and it meant that add-ins started failing, because their underlying scripts could not find “%SPLUNKHOME%”
  • npm
    • Again install it and make sure you can call it from cmd

If you would like to try Splunk out in a test environment, you can sign up for a free account on the Splunk website and get a free trial of Splunk Enterprise there.
The trial converts to the community version after 30 days. No hidden charges there.

Configuring the Azure Monitor Add on

How it works

This add on is a third party add-on that is designed to work with Azure Monitor, the activity and audit logs, and generally any event hub in Azure.

This means that it is very flexible and can ingest data efficiently via the EventHub.

This would be the option you would pick for getting data into Splunk in a timely fashion.

What needs to be done on Azure

All config information can be found here.

  • You need to setup a KeyVault
  • You need to setup a service principal with access to the KeyVault
  • You need to create at least one EventHub
  • You need to store the key (at least read) for the EventHub in the KeyVault created earlier

 

  • The extension will log in via the Service Principal, go to your KeyVault, get the key and then access the Event Hub with the key
  • For this to work you need to give it the correct secret name
  • Make sure you send the content type as requested in the documentation, otherwise the integration is not going to work

What needs to be done in Splunk

  • Once the add on is installed you will see a variety of new data types appear.
  • There is no UI for the add on itself at the time of writing, which is not a big problem as you can configure the data type separately.
  • Once the data type is configured, content should start arriving within a minute or so.

Troubleshooting

I ran into several difficulties while configuring this add on. Luckily you can just access the add on files in your %SPLUNKHOME% folder and run the command line scripts manually to see if they execute.

In my case I did not have all the environment variables set at first, and then I had provided the wrong service principal secret.

You can also switch on debug level monitoring in Splunk for the ExecProcessor. How? ->

What can’t I get with this add on?

What is this add on good at?

  • Leverages event hubs to get events from Azure with 30-60 seconds delay
  • Can get Diagnostics logs for all resources that can write diagnostics to event hubs
  • Can get the entire Activity Log
  • Can get Metrics if they can be logged to an event hub
    • it can also filter by tags to only get the metrics you want
  • Open Source

What is this add on bad at?

  • it’s third party and not supported by Microsoft or Splunk
  • it’s relatively difficult to configure
  • there is no UI for the app in Splunk

Configuring the Microsoft Cloud Services Add on

https://splunkbase.splunk.com/app/3110/

http://docs.splunk.com/Documentation/AddOns/released/MSCloudServices/About

How it works

This add on is built by Splunk and lets you get:

  • The audit log (resource changes)
  • Data stored in Azure Tables or Blob
    • for example: Web App Diagnostics output

What needs to be done on Azure

You need to configure a Service Principal that has access to the logging information you are trying to pull. Link ->

You can skip this step, if you are only pulling from a storage account. In this case you can use a Shared Access Signature. Link ->

I like using the Azure Storage Explorer to generate shared access signatures. Download it ->

What needs to be done in Splunk

  • Install the Add-In from the above link
  • Access the app
  • Create an Azure account (if required) that references your service principal
  • Create data connectors for the data you would like to pull

What can’t I get with this add on?

What is this add on good at?

  • good user experience
  • easy to configure
  • pulls storage account data
  • it’s written by the Splunk team

What is this add on bad at?

  • pulling from storage accounts is an expensive action and cannot always be completed in a timely manner
    • you shouldn’t really be pulling large quantities of data more often than once an hour
    • the add on offers some filtering conditions to only pull what you need
  • does not support EventHubs at the time of writing