Security

ENTERPRISE

Configuring secure DC/OS service accounts

This guide describes how to configure secure DC/OS service accounts for DC/OS Data Science Engine. When running in DC/OS strict security mode, both the DC/OS Data Science Engine and Spark applications launched from it must authenticate to Mesos using a DC/OS service account.

Provisioning a service account

This section describes how to configure DC/OS access for DC/OS Data Science Engine. Depending on your security mode, DC/OS Data Science Engine may require service authentication for access to DC/OS.

Security mode Service Account
Disabled Not available
Permissive Optional
Strict Required

If you install a service in permissive mode and do not specify a service account, Metronome and Marathon will act as if requests made by this service are made by an account with the superuser permission.

Prerequisites:

Create a Key Pair

In this step, a 2048-bit RSA public-private key pair is created using the Enterprise DC/OS CLI.

Create a public-private key pair and save each value into a separate file within the current directory.

dcos security org service-accounts keypair <private-key>.pem <public-key>.pem

NOTE: You can use the DC/OS Secret Store to secure the key pair.

Create a Service Account

From a terminal prompt, create a new service account (for example, data-science-engine) containing the public key (<your-public-key>.pem).

dcos security org service-accounts create -p <your-public-key>.pem -d <description> data-science-engine

You can verify your new service account using the following command.

dcos security org service-accounts show data-science-engine

Create a Secret

Create a secret (data-science-engine/<secret-name>) with your service account and private key specified (<private-key>.pem).

NOTE: If you store your secret in a path that matches the service name, for example, service name and secret path are both data-science-engine, then only the service named data-science-engine can access it.

Permissive

dcos security secrets create-sa-secret <private-key>.pem <service-account-id> data-science-engine/<secret-name>

Strict

dcos security secrets create-sa-secret --strict <private-key>.pem <service-account-id> data-science-engine/<secret-name>

You can list the secrets with this command:

dcos security secrets list /

Create and assign permissions

In strict mode, any Spark applications launched by the DC/OS Data Science Engine will require additional permissions for authenticating with Mesos. This includes the launching of executors (worker tasks) on the cluster.

NOTE: Spark applications launched by the DC/OS Data Science Engine do not require an additional service account setup and will reuse the service account created for DC/OS Data Science Engine with additional required permissions.

Use the following DC/OS CLI commands to rapidly provision a service account with the required permissions:

# Allows the default user 'nobody' to execute tasks
dcos security org users grant <service-account-id> dcos:mesos:master:task:user:nobody create
dcos security org users grant <service-account-id> dcos:mesos:agent:task:user:nobody create

# Allows Spark framework to reserve cluster resources using <service-account-id> role and principal
dcos security org users grant <service-account-id> dcos:mesos:master:framework:role:<service-account-id> create
dcos security org users grant <service-account-id> dcos:mesos:master:reservation:role:<service-account-id> create
dcos security org users grant <service-account-id> dcos:mesos:master:reservation:principal:<service-account-id> delete
dcos security org users grant <service-account-id> dcos:mesos:master:volume:role:<service-account-id> create
dcos security org users grant <service-account-id> dcos:mesos:master:volume:principal:<service-account-id> delete

# Allows Spark framework to launch tasks using <service-account-id> role and principal
dcos security org users grant <service-account-id> dcos:mesos:master:task:role:<service-account-id> create
dcos security org users grant <service-account-id> dcos:mesos:master:task:principal:<service-account-id> create
dcos security org users grant <service-account-id> dcos:mesos:master:task:app_id:<service-account-name> create

NOTE: The default service account name is data-science-engine. However, ensure that the <service-account-name> entered here is the same as the service name specified in the options.json or in the UI when DC/OS Data Science Engine is installed.

To configure Spark for using created service account and permissions, add the following configuration under the spark section:

{
  "service": {
      "service_account": "<service-account-id>",
      "service_account_secret": "<service-account-secret>",
  },
  "spark": {
      "spark_mesos_principal": "<service-account-id>",
      "spark_mesos_secret": "<service-account-secret>",
      "spark_mesos_role": "<service-account-id>"
  }
}

Using the secret store

DC/OS Enterprise allows users to add privileged information in the form of a file to the DC/OS Secret Store. These files can be referenced in DSEngine jobs and used for authentication and authorization with various external services (for example, HDFS). For example, you can use this functionality to pass Kerberos keytab files.

Where to place secrets

For a secret to be available to DC/OS Data Science Engine, it must be placed in a path that can be accessed by the DSEngine service. If only DSEngine requires access to a secret, you can store the secret in a path that matches the name of the DSEngine service (for example, data-science-engine/secret).

Limitations

Anyone who has access to the DC/OS Data Science Engine’s notebook has access to all secrets available to it. Do not grant users access to the notebook unless they are also permitted to access its secrets.

Binary secrets

You can store binary files, like a Kerberos keytab, in the DC/OS Secrets Store. In DC/OS 1.11 and later, you can create secrets from binary files directly.

To create a secret called mysecret with the binary contents of kerb5.keytab, run the following command:

dcos security secrets create --file kerb5.keytab mysecret

In DC/OS 1.10 or earlier, files must be base64-encoded–as specified in RFC 4648–before being stored as secrets.

Using Mesos secrets in Spark jobs

Once a secret has been added to the Secret Store, you can include it in the service’s configuration under the security section:

{
  "service": {
      "service_account": "<service-account-id>",
      "service_account_secret": "<service-account-secret>",
  },
  "security": {
    "extra_spark_secrets": {
      "secret_names": "/data-science-engine/my-secret",
      "secret_filenames": ".secrets/my-secret",
	    "secret_envkeys": "MY_SECRET_ENV_VAR"
    }
  }
}

Provided secrets will be automatically mounted to DC/OS Data Science Engine’s sandbox. The secrets will also be made available to all Spark executors by adding them to the following Spark configuration properties: -spark.mesos.executor.secret.names -spark.mesos.executor.secret.<filenames|envkeys>

NOTE: It is mandatory to provide values for service_account and service_account_secret in the service configuration in order to access any secrets.

Limitations

When using environment variables and file-based secrets, there must be an equal number of sinks and secret sources. That is, the keys secret_names, secret_filenames, and secret_envkeys must have the same number of values. For example:

{
  "service": {
      "service_account": "<service-account-id>",
      "service_account_secret": "<service-account-secret>",
  },
  "security": {
    "extra_spark_secrets": {
      "secret_names": "/data-science-engine/my-secret-file,/data-science-engine/my-secret-envvar",
      "secret_filenames": "target-secret-file,placeholder-file",
      "secret_envkeys": "PLACEHOLDER,SECRET_ENVVAR"
    }
  }
}

NOTE: It is mandatory to provide values for service_account and service_account_secret in the service configuration in order to access any secrets.

This configuration places the contents of data-science-engine/my-secret-file into the target-secret-file as well as the PLACEHOLDER environment variable. Additionally, the contents of data-science-engine/my-secret-envvar are exported to the SECRET_ENVVAR and written to the placeholder-file.

NOTE: If the content size of binary secrets is greater than 4KB, Mesos' security module will reject container execution due to the overhead.

Authenticating to your DC/OS Data Science Engine instance

You can run multiple installations of DC/OS Data Science Engine by changing the service.name option during installation. Each instance can have different authentication mechanisms configured.

Password Authentication (default)

The default DC/OS Data Science Engine password is set tojupyter. You can override it with the service.jupyter_password option.

OpenID Connect

You can choose to enable OpenID Connect authentication. The OpenID Connect flow will be triggered if oidc.enabled is true and both oidc.discovery_uri and oidc.client_secret are set, since they are the minimal options.

The default client name is notebook and it can be overridden with the oidc.client_id option.

Here is an example of a simple OpenID Connect configuration for DC/OS Data Science Engine:

{
  "service": {
      "service_account": "<service-account-id>",
      "service_account_secret": "<service-account-secret>",
  },
  "oidc": {
      "enabled": true,
      "discovery_uri": "https://keycloak.example.com/auth/realms/notebook/.well-known/openid-configuration",
      "client_id": "data-science-engine-client",
      "client_secret": "11111111-2222-3333-4444-555555555555"
  }
}

NOTE: It is mandatory to provide values for service_account and service_account_secret in the service configuration in order to access any secrets.

Using Arbitrary Secrets

This feature only works for Apache Toree kernel. To know more click here.

Suppose you want to use credentials for private repositories. First of all, create a credential property file with following content:

user = username
pwd = password
host = repository.com

You can use this credential file in apache toree notebook by using %AddDeps magic as follows:

%AddDeps org.name artifact version --credential /path/to/credentials.prop --repository https://repository.com/artifactory/example-repo-local/

In order to see all the configuration parameters of %AddDeps magic, you should run %AddDeps without any arguments or flag.

Kerberos

ENTERPRISE

Using Kerberos with DC/OS Data Science Engine to retrieve and write data securely…Read More

TLS

ENTERPRISE

Using TLS to encrypt channels for greater security…Read More