CINQby Quinten Stekelenburg
In this blog we will state some of the changes and challenges you might face when you upgrade from Splunk 8.2x to Splunk 9.0.4. After reading this blog and checking the latest release notes you should have a better understanding of the upcoming changes when you upgrade to Splunk 9.
Starting with Splunk 9 the classic dashboards should now use a version number, if you do not have a version number the user will see the warning below:
All dashboards that contain the <dashboard> tag will automatically receive the version when upgrading to version 9.0.4. If you upgrade to a lower version you have to add the version=”1.1” yourself. Any dashboards that contain dropdowns will contain the <form> tag instead of the <dashboard> tag. These will not get the version=”1.1” and you have to manually add the version to no longer show the warning.
We recommend to use 9.0.4.1 since 9.0.4 contains a known issue for dashboards that contain <set tokens (found this information on the Splunk Slack):
If you have:
After upgrading it will look like this:
And this means that the token will no longer work and the dashboard functionality that depends on that token breaks.
To find out if you have any dashboards that might be affected you should run this search that was shared by Becky Burwell on the Splunk Slack:
The workaround is to manually change these dashboards before upgrading to Splunk 9.0.4 by adding version=”1.1” into the <dashboard or <form tag.
It was stated that this is fixed in version 9.0.4.1 that was released on the 17th of March.
Please note that any dashboards in etc/apps and etc/users will be changed by upgrading, any dashboards that are pushed by deployers in etc/shcluster/apps will not be changed. So pushing apps with dashboards in the default folder without version=”1.1” might undo the changes done on the search heads and show the warning again for users.
With Splunk 9 Dashboard studio now supports the usage of tokens, this is one of the most requested feature for Dashboard studio dashboards. You can now set default tokens or pass tokens when linking to other dashboards , this opens the linked dashboard with the token values so you can see the right information without having to select the options again.
To read all about these changes please see the links at the bottom.
There is a new option to help admin troubleshoot, there is a new index called _configtracker. This index will contain changes to .conf files and their key value pairs. This makes seeing what was changed and when it was changed so much easier without using 3rd party tools.
If you receive ip information in you Splunk environment you can enrich that ip information with the iplocation command. This adds the following fields by default to the results:
Splunk ships its software with a Maxmind database containing this information. It normally gets updated when you upgrade to a new Splunk version. For some use-cases the data needs to be refreshed weekly to prevent the use of inaccurate data and false positive hits when checking for certain countries. To make it easier to update this information you can now update this using the GUI or by placing the file on your search heads and specifying the path in the limits.conf file. For exact instructions please visit the link at the bottom of the page.
Splunk has changed some of the names used in folders and configuration files. An important change is when you have an indexer cluster and you are pushing your apps from the master-apps folder.
With Splunk 9 you should start using manager-apps instead of master-apps since that will be removed in a feature release. In order to use manager-apps you should copy over all files and folders from the master-apps and remove them in that location. When you start copying you also have to check your conf files for hardcoded paths, the indexer path will change from slave-apps to peer-apps, any hardcoded references to slave-apps will break. Usually paths are used for SSL configuration and you do not want that to break after your upgrade.
There are more changes in Splunk 9.0.x, please read the release notes and check out the Splunk slack channel Splunk_9_upgrade_issues to see what other users are encountering or ask for help.
Splunk Enterprise Release notes: https://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes
Splunk slack: https://splunk-usergroups.slack.com
Dashboard studio changes: https://docs.splunk.com/Documentation/Splunk/latest/DashStudio/WhatNew
Configuration change tracker: https://docs.splunk.com/Documentation/Splunk/9.0.4/Troubleshooting/WhatSplunklogsaboutitself#Configuration_Change_Tracker
iplocation command: https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Iplocation