Techology

How to make Okta Oauth2 work on Grafana without outage

Detailed Guide on implementing Okta Oauth2 to simplify user management for Grafana.

Daniel Schwemmlein

Nov 1, 2022 • 8 min read

How to enable Okta OAuth2 on Grafana

This article describes how to set up Okta authentication on Grafana without the risk of locking users out during the cutover. This is done by leaving LDAP authentication enabled while adding configuration for okta authentication. We assume basic familiarity with Grafana, Okta and Kubernetes.

Why use Okta Oauth2 for Grafana? There are several potential reasons - at EverQuote we have been using Google LDAP for authentication, which caused quite a bit of administrative overhead when on or offboarding users. Using Okta instead allows us not to have that overhead. Also, compared to Google LDAP, Okta Oauth2 just is the more secure option.

As a first step, a group or several groups would need to be created and populated with users. We would recommend creating at least 2 groups for later testing (for instance to see if Grafana assigns the correct role to a user depending on the Okta group they are a member of).

Okta group creation

If required, below are the steps that need to be followed to create a group in Okta:

Open the Advanced Server Access dashboard.
Click Groups.
Click Create Group.
From the Create Group window, enter a group name.
Optional. Select any team roles to assign to the group. See Team roles.
Click Create Group.

Okta application creation

In our example here, the plan is to create 2 groups named ‘Grafana - Users’ and ‘Grafana - Admins’. The next step includes creating the actual Okta application that would be used.

Go to the ‘Admin’ section in your Okta portal.
Select Applications, then ‘Add Application’.
Use ‘Web’ as the platform.
Enter a name for your application (Grafana for instance).
Add the Base URI of the application, such as https://grafana.example.com.
Enter values for the Login redirect URI. Use Base URI and append it with /login/okta and /login for example: https://grafana.example.com/login/okta and https://grafana.example.com/login (If we want to keep Grafana sign in working during the cutover https://grafana.example.com/login would need to be added.)
We would recommend actually typing the Login redirect URI to limit the chance of spaces to be copied over. If there is a typo in the Login Redirect URI, this will result in a 400 error when users try to sign in using okta.
Set login initiated by to App
Click Done to finish creating the Okta application.
Assign the groups we had created previously to the application

you would need the Unique ID of the application that has just been created
Also the Unique ID of the group(s) that should be added to the application are required.
If more details are needed, search for ‘Assign group to Application’ in the Okta Help Center

Grafana Okta configuration

Now, we need to think about the Grafana configuration part. This will vary depending on the runtime platform of your Grafana instance. If it is running in its own cluster on a public cloud such as AWS, GCP or Azure, you may want to store the Okta application’s Client ID and Client Secret as a Kubernetes secret. The example below shows roughly how Okta credentials could be stored as a secret in a kubernetes cluster on AWS. Going forward, whenever specific Kubernetes configurations are mentioned, they are referring to AWS EKS. The code below would need to be stored in a yaml file and then applied to the cluster using kubectl apply -f secret.yaml

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: grafana-okta-credentials
  namespace: grafana
stringData:
  client-id: foo
  client-secret: bar

Note: Do not store your actual client-id and client-secret directly in your source code management (SCM) tool. After applying the secret with the real values for client-id and client-secret, there should be a pull request (request to merge code into your main branch) with placeholder values such as ‘foo’ or ‘bar’ to store the secret in SCM but not the value

Once the secret exists in your cluster (you may want to verify by running kubectl get secret grafana-okta-credentials -n grafana -o yaml) we can move on to add configuration to grafana.ini allowing us to use Okta for authentication.

Below is an example of a helm release yaml file referencing the secret we had discussed above.

OKTA_CREDENTIALS_CLIENT_ID:
        secretKeyRef:
          name: grafana-okta-credentials
          key: client-id
OKTA_CREDENTIALS_CLIENT_SECRET:
        secretKeyRef:
          name: grafana-okta-credentials
          key: client-secret

An important fact to realize here is that we are storing the values for our Okta application credentials in two environment variables. It would be quite an easy mistake to make, referencing these variables by just adding a $ before the variable’s name. In order for the environment variables values to still be there when being passed on to another of docker’s layers, they need to be wrapped in curly braces and double quotes as indicated below when referencing them in grafana.ini:

client_secret: "${OKTA_CREDENTIALS_CLIENT_SECRET}"
client_id: "${OKTA_CREDENTIALS_CLIENT_ID}"

We can now look at the configuration that needs to be present in grafana.ini apart from the credentials in order for Okta authentication to work:

[...]
auth.ldap:
    enabled: true  # left enabled to prevent disruption for Grafana logins 
    sync_cron: "0 12 * * * *"

  auth.okta:
    enabled: true
    name: Okta
    icon: okta
    allow_sign_up: true
    client_id: "${OKTA_CREDENTIALS_CLIENT_ID}"
    client_secret: "${OKTA_CREDENTIALS_CLIENT_SECRET}"
    scopes: openid profile email groups
    auth_url: https://yourdomain.okta.com/oauth2/v1/authorize
    token_url: https://yourdomain.okta.com/oauth2/v1/token
    api_url: https://yourdomain.okta.com/oauth2/v1/userinfo
    role_attribute_path: contains(groups[*], 'Grafana - Admins') && 'Admin' || contains(groups[*], 'Grafana - Users') && 'Editor' || 'Viewer'
[...]

In order to avoid any potential issues after adding Okta configuration to your grafana.ini, auth.ldap can be left enabled (as shown above). This way users can still log in the way they are used to in case there is an issue with Okta configuration.

Okta scopes and authentication server config

Quite an important detail about the code example above would be scopes - It lists the scopes that are needed from the Okta auth server and then passed on to Grafana so that depending on a user’s profile, email or group membership they could be assigned a ‘Admin’, ‘Editor’ or ‘Viewer’ role in Grafana.

In order to make sure you don’t have any surprises when first testing out authentication with Okta, we would strongly recommend checking whether or not your default authentication server in Okta features all of these scopes (openid, profile, email, groups) required by Grafana. Authentication Servers are listed in the Okta Admin section under Security → API. If the default authentication server does not feature the scopes required, you may need to create a new custom one. Doing that will entail several steps and a complete guide for this can be found on http://developer.okta.com/docs .

Okta group to Grafana Role mapping

The other important thing to note in our example above would be ‘role_attribute_path’. Here, we are using the JMESPath language to map the content of a user’s group value to a certain role in Grafana. Depending on the circumstances, some previous familiarity with JMESPath could be really good here - if you have not worked with it before, there is a tutorial page named https://jmespath.org/tutorial.html that would allow you to try out your regular expression there instead of deploying this and finding out that a single quote is at the wrong place.

In our example above, the section contains(groups[*], 'Grafana - Admins') && 'Admin' means about as much as ‘if the user’s group scope contains the Okta group ‘Grafana - Admins’, assign the ‘Admin’ role to them’ - in a similar way the section || contains(groups[*], ‘Grafana - Users') && ‘Editor' means something like ‘if the user's group scope does not show membership in the Admin group but instead the ‘Grafana - Users' group, assign the ‘Editor' role'. The last section || ‘Viewer' is similar to a ‘else’ statement in another programming language, because it assigns the ‘Viewer’ permission to anyone without membership in the two Okta groups we have mentioned above. One important thing to note about this is the order in which ‘Admin’ , 'Editor’ and 'Viewer’ roles are mapped in the role_attribute_path. An example: If a user were to be member of both 'Grafana - Admins’ and 'Grafana - Users’ they would be assigned the 'Admin’ role because this is the first condition that is inspected. In order to prevent this, one could have the 'Editor’ role mapping inspected first like this:

role_attribute_path: contains(groups[*], 'Grafana - Users') && 'Editor' || contains(groups[*], 'Grafana - Admins') && 'Admin'

Notice that above we have left the || 'Viewer' bit out. This is because Grafana would grant the user this role by default unless the option role_attribute_strict: true is set.

By default it is set to ‘false’ and means that if the user trying to log in can not be mapped to any existing Grafana Team, they would be able to sign in with a ‘Viewer’ role still. Depending on sensitivity of the data, this could be used to further lock down access because it would block these users from accessing Grafana altogether.

Important: If your authentication server does not feature the ‘groups’ scope, all members of both of our Okta groups will only be able to sign in with ‘Viewer’ role because when Grafana is listening for the ‘groups’ scope from Okta, it will get an empty response - which then means that the first 2 cases in our JMESPath if-statement above are disregarded automatically and only the else statement remains.

Grafana Team creation

One the last steps to get the Okta group mapped to the correct role with Grafana is to create the actual team in Grafana. The Team’s name should be the same as the Okta group’s. The steps do this are indicated below:

In the sidebar, hover over the Configuration (gear) icon and then click Teams.
Click New team.
In Name, enter the name of the team. You do not need to enter an email.
Click Create.
Click on the Teams link at the top of the page to return to teams page and create the second team.

Now, we only need to enable ‘external group sync' from the Grafana side to make sure Grafana also 'looks’ for an Okta group with a particular name.

On the External group sync tab, and click Add group.
Insert the value of the group you want to sync with. In our example that would be the name of the Okta group we want to have synced with this Grafana team. In our example here the Okta group names were ‘Grafana - Users’ or ‘Grafana - Admins’. To avoid conflicts, we recommend just using one Okta group name to sync with.

There is no need to add users to those Grafana Teams 🙂 This will be done by Okta as long as you have added users to your Okta groups.

Granting Okta-synced Teams access to Dashboard Folders

There is one last step that needs to be taken before completely switching over to Okta - we need to ensure that each team can access the dashboard folders they are used to. In order to do this we need to

log in to Grafana as an organization administrator.
on the left hand side we need to hover over the Dashboards (squares) icon and click ‘Browse’.
hover over a folder and click ‘Go to folder’ (We need to be sure which team needs access to which folders).
click the ‘Permissions’ tab, and then click ‘Add Permission’.
go to the ‘Add Permission For’ dropdown menu - there, we need to select ‘Team’, enter the name of our Okta group (in our example ‘Grafana - Users’), and the role we want to grant them over the dashboard folder (in our case ‘Editor’) - then we can click 'Save'.
repeat this procedure for every Okta group that needs access to Grafana.

After having verified that all members of each Okta group are able to sign in and get the correct Grafana role, we can switch off ‘auth.ldap’ in the grafana.ini file as shown below:

[...]
auth.ldap:
    enabled: false
[...]