Versions

In HDFS 2.2.0-2.6.0-fdh5.11.0 and later, these topics have been divided up among the Getting Started and Security sections. Earlier versions will still need the information below.

Configure DC/OS access for HDFS

This topic describes how to configure DC/OS access for HDFS. Depending on your security mode, HDFS requires service authentication for access to DC/OS.

If you install a service in permissive mode and do not specify a service account, Metronome and Marathon will act as if requests made by this service are made by an account with the superuser permission.

Prerequisites:

• DC/OS CLI installed and be logged in as a superuser.
• Enterprise DC/OS CLI 0.4.14 or later installed.
• If your security mode is permissive or strict, you must get the root cert before issuing the curl commands in this section.

Create a Key Pair

In this step, a 2048-bit RSA public-private key pair is created uses the Enterprise DC/OS CLI.

Create a public-private key pair and save each value into a separate file within the current directory.

dcos security org service-accounts keypair <private-key>.pem <public-key>.pem

Tip: You can use the DC/OS Secret Store to secure the key pair.

Create a Service Account

From a terminal prompt, create a new service account (<service-account-id>) containing the public key (<your-public-key>.pem).

dcos security org service-accounts create -p <your-public-key>.pem -d "HDFS service account" <service-account-id>

Tip: You can verify your new service account using the following command.

dcos security org service-accounts show <service-account-id>

Create a Secret

Create a secret path (hdfs/<secret-name>) with your service account (<service-account-id>) and private key specified (<private-key>.pem).

Tip: If you store your secret in a path that matches the service name (e.g. service name and secret path are hdfs), then only the service named hdfs can access it.

Permissive

dcos security secrets create-sa-secret <private-key>.pem <service-account-id> hdfs/<secret-name>

Strict

dcos security secrets create-sa-secret --strict <private-key>.pem <service-account-id> hdfs/<secret-name>

Tip: You can list the secrets with this command:

dcos security secrets list /

Create and Assign Permissions

Use the following curl commands to rapidly provision the HDFS service account with the required permissions.

Tips:

• Any / character in a resource must be replaced with %252F before it can be passed in a curl command.
• When using the API to manage permissions, you must first create the permissions and then assign them. Sometimes, the permission may already exist. In this case, the API returns an informative message. You can regard this as a confirmation and continue to the next command.

1. Create the permission.

   Important: These commands use the default HDFS role value of hdfs-role. If you’re running multiple instances of HDFS, replace the instances of hdfs-role with the correct name (<name>-role). For example, if you have a HDFS instance named hdfs2, you would replace each role value in the code samples to hdfs2-role.

   Permissive

   Run these commands with your service account name (<service-account-id>) specified.

   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:task:user:nobody \
   -d '{"description":"Allows Linux user nobody to execute tasks"}' \
   -H 'Content-Type: application/json'
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:framework:role:hdfs-role \
   -d '{"description":"Controls the ability of hdfs-role to register as a framework with the Mesos master"}' \
   -H 'Content-Type: application/json'
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:role:hdfs-role \
   -d '{"description":"Controls the ability of hdfs-role to reserve resources"}' \
   -H 'Content-Type: application/json'
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:role:hdfs-role \
   -d '{"description":"Controls the ability of hdfs-role to access volumes"}' \
   -H 'Content-Type: application/json'
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:principal:<service-account-id> \
   -d '{"description":"Controls the ability of <service-account-id> to reserve resources"}' \
   -H 'Content-Type: application/json'
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:principal:<service-account-id> \
   -d '{"description":"Controls the ability of <service-account-id> to access volumes"}' \
   -H 'Content-Type: application/json'
   

   Strict

   Run these commands with your service account name (<service-account-id>) specified.

   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:framework:role:hdfs-role \
   -d '{"description":"Controls the ability of hdfs-role to register as a framework with the Mesos master"}' \
   -H 'Content-Type: application/json'
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:role:hdfs-role \
   -d '{"description":"Controls the ability of hdfs-role to reserve resources"}' \
   -H 'Content-Type: application/json'
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:role:hdfs-role \
   -d '{"description":"Controls the ability of hdfs-role to access volumes"}' \
   -H 'Content-Type: application/json'
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:principal:<service-account-id> \
   -d '{"description":"Controls the ability of <service-account-id> to reserve resources"}' \
   -H 'Content-Type: application/json'
   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:principal:<service-account-id> \
   -d '{"description":"Controls the ability of <service-account-id> to access volumes"}' \
   -H 'Content-Type: application/json'
   

1. Grant the permissions and the allowed actions to the service account using the following commands.

   Run these commands with your service account name (<service-account-id>) specified.

   curl -X PUT --cacert dcos-ca.crt \
   -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:framework:role:hdfs-role/users/<service-account-id>/create
   curl -X PUT --cacert dcos-ca.crt -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:role:hdfs-role/users/<service-account-id>/create
   curl -X PUT --cacert dcos-ca.crt -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:role:hdfs-role/users/<service-account-id>/create
   curl -X PUT --cacert dcos-ca.crt -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:task:user:nobody/users/<service-account-id>/create
   curl -X PUT --cacert dcos-ca.crt -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:principal:<service-account-id>/users/<service-account-id>/delete
   curl -X PUT --cacert dcos-ca.crt -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:principal:<service-account-id>/users/<service-account-id>/delete
   

Create a Configuration File

Create a custom configuration file that will be used to install HDFS and save as config.json.

Specify the service account (<service-account-id>) and secret path (hdfs/<secret-name>).

{
  "service": {
    "service_account": "<service-account-id>",
    "service_account_secret": "hdfs/<secret-name>"
  }
}

Install HDFS

Install HDFS with this command:

dcos package install --options=config.json hdfs