Changing Stored Passwords¶

VOR Stream keeps the passwords it uses to reach other systems in Vault. When one of those passwords changes on the system it belongs to, the copy in Vault has to change too, or VOR Stream keeps connecting with the old one.

Database Node Connections¶

Database nodes (db=pg and db=mssql) connect with a connection secret: a Vault key-value secret under kv/node/ that holds the host, port, database name, and credentials. Go nodes that call ConnectPSQL or ConnectMSSQL read the same secrets, so the steps below change their password too. The steps apply to PostgreSQL and MSSQL databases.

A node finds its connection secret at one of three levels. All three are the same kind of secret, so the procedure is the same at each level; only the path differs:

• Node: the connection option on the node in the process's .strm file, or for a Go node, the path it passes to ConnectPSQL or ConnectMSSQL.
• Playpen: database-conn-path in the playpen's config.yaml.
• Global: the database_conn_path inventory variable. On a running system, consul kv get vor/defaults shows it as database-conn-path.

Each path is relative to kv/node/: a node with connection="reporting/pg" reads kv/node/reporting/pg. See Setting up Database Connections for how the levels override each other.

No service restart is needed. Super reads the connection secret from Vault each time a node asks for it, without caching it, so the next node that connects gets the new password.

Finding the Stored Password¶

Read the connection secret:

vault kv get kv/node/<path>

The secret holds the credentials in one of two ways:

• Inline: the secret has username and password fields. The password changes in this secret.
• Credential path: the secret has a credential_path field instead. Super reads the username and password from the secret that credential_path names, so the password changes there. The value is a full Vault API path, such as kv/data/node/creds/reporting; vault kv commands take it without the data/ segment, as kv/node/creds/reporting.

If credential_path points into Vault's database secrets engine, under database/creds/ or database/static-creds/, Vault issues or rotates the password itself, and there is no stored password for this procedure to change.

Order of Steps¶

1. Change the password on the database server.
2. Change the password in Vault, as soon as possible after step 1.

Between the two steps, Vault still holds the old password, so any job that uses the connection and starts in that window fails. The vor run output and the database node's log, under output/<process>/logs/ in the playpen, show the database's authentication error. For PostgreSQL:

cannot connect to the database: pq: password authentication failed for user "<user>" (28P01)

For MSSQL:

cannot connect to the database: mssql: login error: Login failed for user '<user>'.

Rerun those jobs once Vault holds the new password. To avoid failed jobs, make the change when no jobs that use the database are running or due to start.

Changing the database first also suits vor create secret, whose password prompt checks the password against the database.

Jobs Already Running¶

A db= node fetches its connection secret once, when it starts, and connects to the database then. It does not fetch the secret again, so a change in Vault does not reach a node that is already running. What happens to that node when the password changes on the server depends on the database:

• PostgreSQL does not check the password again on an open session. A node that is already connected keeps working, and its job finishes normally. If the node's session ends for another reason, such as a database restart or a dropped network connection, the node reconnects with the password it fetched at startup and fails with pq: password authentication failed for user "<user>" (28P01).
• MSSQL checks the password again each time a node reuses its open connection. After the password changes on the server, a running node fails the next time it goes back to the database. Its log shows SQL Server had internal error, and the SQL Server error log shows error 18456, Login failed for user '<user>'. Reason: Password did not match, followed by error 18056. Let jobs that use the database finish before changing its password, or rerun the ones that fail.

Changing the Password in Vault¶

vault kv patch works for inline credentials and for a credential_path secret alike: it changes only the fields it names and keeps the rest of the secret as it was. vor create secret writes connection secrets only, so it suits inline credentials, and only when you want to reenter the whole connection.

read -rs NEW_PASSWORD
printf '%s' "$NEW_PASSWORD" | vault kv patch kv/node/<path> password=-
unset NEW_PASSWORD

vor create secret --db-type=postgres --vault-path=<path> --token="$VAULT_TOKEN" \
  --host=<host> --port=<port> --name=<database> --ssl=<ssl> \
  --allowed-groups=<group>,<group> --username=<user>

Checking the Change¶

Run a job that uses the connection, or rerun one that failed between the two steps. A database node that reads or writes its table confirms that the new password works.

Django Database¶

Django keeps its data in a PostgreSQL or MSSQL database, set up with the database_user and database_password in the inventory file. When the inventory also sets database_vault_path, Django gets its database credentials from Vault.

The password of database_user is kept in three places, and all three have to agree:

• The database itself.
• Vault.
• database_password in the inventory file.

Every deployment writes database_password into Vault. When PostgreSQL is deployed by the playbook, meaning Django's database runs on the host in the postgres group of the inventory, every deployment also sets it on the database. If the inventory still holds the old password, the next deployment puts it back.

No other VOR Stream service signs in with this password, so only Django is affected. If a database node connection uses the same database user, change its secret too.

Finding the Credential Type¶

database_vault_path in the inventory file shows how Django gets its credentials. On a running system, VAULT_DATABASE_PATH in /opt/vor/etc/django/config.json on the Django host holds the same value.

Run the vault commands below with VAULT_ADDR set to the Vault server, for example https://<vault host>:8200, and a token that can write to the path, such as the root token.

Changing the Password with the Playbook¶

When PostgreSQL is deployed by the playbook, set database_password in the inventory file to the new password and rerun the playbook. The deployment changes the password on the database, writes it into Vault, and restarts uWSGI, so no manual step is needed. The PostgreSQL play signs in over the local socket, so it does not need the old password. As in any deployment, the web UI is unavailable while the Django play reinstalls Django and restarts uWSGI.

For MSSQL or any other database server, or to avoid a full deployment, follow the procedure for the credential type instead.

Changing the Password on the Database¶

Both procedures start by changing the password of database_user on the database server. For a PostgreSQL server that the playbook did not deploy, change it the way that server is administered.

$ /opt/vor/pgsql/bin/psql -h /tmp -d postgres
postgres=# \password <database_user>

ALTER LOGIN [<database_user>] WITH PASSWORD = '<new password>' OLD_PASSWORD = '<old password>';

Static Credentials¶

With static credentials, a Vault key-value secret at database_vault_path holds the username and password. Django reads the secret once, when uWSGI starts.

1. Change the password on the database server, as described in Changing the Password on the Database.

2. Change the password in the secret, which keeps the username as it is. vault kv commands take the path without its data/ segment, so kv/data/postgres becomes kv/postgres. Read the new password without echoing it, type it and press Enter, then patch it into the secret:

   read -rs NEW_PASSWORD
   printf '%s' "$NEW_PASSWORD" | vault kv patch kv/<path> password=-
   unset NEW_PASSWORD
   

3. Restart uWSGI on the Django host, as the installation user:

   supervisorctl restart uwsgi
   

4. Set database_password in the inventory file to the new password.

Django opens a new database connection for each request, so it stops working as soon as the password changes on the database server, and starts again once uWSGI restarts and reads the new password from Vault. Run steps 1 to 3 back to back, at a time when nobody is using VOR Stream. In between, the web UI sends users to the login page with Session Check Failed, and /opt/vor/log/django/uwsgi.log shows the database's error:

password authentication failed for user "<database_user>"

For MSSQL:

Login failed for user '<database_user>'. (18456)

If uWSGI restarts before step 2, it reads the old password from the secret and keeps it after the secret changes, so restart uWSGI again once step 2 is done. On PostgreSQL, its workers wait at startup for the database in the meantime, the web UI stops responding, and the log repeats Database not ready: ... password authentication failed.

Dynamic Credentials¶

With dynamic credentials, Django does not sign in as database_user. Vault's database secrets engine signs in as database_user and creates a short-lived database user for Django. The password to change is the one Vault signs in with.

1. Change the password on the database server, as described in Changing the Password on the Database.

2. Change the password Vault signs in with. Vault's connection is named after the database, <database_name>-django, which is vor-django by default; the commands below use the default. Read the new password without echoing it, type it and press Enter, then write it into the connection:

   read -rs NEW_PASSWORD
   printf '%s' "$NEW_PASSWORD" | vault write database/config/vor-django password=-
   unset NEW_PASSWORD
   

   Vault keeps the rest of the connection settings, and signs in with the new password before saving it. If the database rejects the password, the command fails with error verifying connection, and Vault keeps the old one.

3. Set database_password in the inventory file to the new password.

No service restart is needed. The users Vault creates for Django have their own passwords, so Django keeps working after step 1.

Vault keeps working after step 1 too, on the database connection it already has open, because PostgreSQL does not check the password again on an open session. The next time Vault connects, for example after Vault restarts, it signs in with the password it has stored, which fails until step 2 is done:

error verifying connection: ping failed: ... password authentication failed for user "<database_user>" (SQLSTATE 28P01)

Django then cannot get new credentials, which it needs when uWSGI restarts and when its current ones expire. If uWSGI restarts in that state, the web UI stops responding, and /opt/vor/log/django/uwsgi.log repeats Database not ready: ... with the same error. Django recovers on its own within seconds of step 2, without another restart.

Checking the Django Password¶

Log in to the web UI. The login and the list of playpens shown after it both read from Django's database.

With dynamic credentials, also make Vault reconnect with the password it has stored, using the connection name from step 2:

vault write -f database/reset/vor-django

Success! means Vault signed in with it; otherwise the command prints the database's error.

RabbitMQ¶

Super and the nodes of every job connect to RabbitMQ. The deployment creates the broker user rabbitmq_user with the password rabbitmq_password from the inventory file. When the inventory also sets rabbitmq_vault_path, VOR Stream gets its RabbitMQ credentials from Vault.

The password of rabbitmq_user is kept in three places, and all three have to agree:

• The broker.
• Vault.
• rabbitmq_password in the inventory file.

Every deployment writes rabbitmq_password into Vault. If the inventory still holds the old password, the next deployment puts it back into Vault. With a managed broker, the deployment first signs in to the broker with rabbitmq_password, and fails there instead if the broker rejects it.

Finding the RabbitMQ Credential Type¶

rabbitmq_vault_path in the inventory file shows how VOR Stream gets its RabbitMQ credentials. On a running system, rabbitmq-vault-path in /opt/vor/etc/super/config.yaml on the Super host holds the same value.

Run the vault commands below with VAULT_ADDR set to the Vault server and a token that can write to the path, such as the root token. Run the rabbitmqctl commands, which apply to a broker the playbook deployed, on a host in the rabbitmq group as the installation user (vrisk, by default). If the inventory sets rabbitmq_node_name, set RABBITMQ_NODENAME to that name for them.

Changing the Password on the Broker¶

Both procedures start by changing the password of rabbitmq_user on the broker.

read -rs NEW_PASSWORD
printf '%s\n' "$NEW_PASSWORD" | /opt/vor/rabbitmq/sbin/rabbitmqctl change_password <rabbitmq_user>
unset NEW_PASSWORD

Static RabbitMQ Credentials¶

With static credentials, a Vault key-value secret at rabbitmq_vault_path holds the username and password of rabbitmq_user. Super reads the secret when it opens its connection to RabbitMQ, and keeps using that password while the connection stays open. Each job's nodes sign in with a copy that Super reads from the secret when it starts the job.

1. Change the password on the broker, as described in Changing the Password on the Broker.

2. Change the password in the secret. vault kv commands take the path without its data/ segment, so kv/data/rabbitmq becomes kv/rabbitmq. Read the new password without echoing it, type it and press Enter, then patch it into the secret:

   read -rs NEW_PASSWORD
   printf '%s' "$NEW_PASSWORD" | vault kv patch kv/<path> password=-
   unset NEW_PASSWORD
   

3. Restart Super on each host in the super group, as the installation user. Restarting Super cancels the jobs that are running.

   supervisorctl restart super
   

4. If any job ran between steps 1 and 3, delete the queues it left behind, as described in Deleting Leftover Queues.

5. Check the change, as described in Checking the RabbitMQ Password.
6. Set rabbitmq_password in the inventory file to the new password.

Run steps 1 to 3 back to back, at a time when no jobs are running or due to start. In between:

• A job that starts before step 2 fails. Its node logs show the broker's error, for a Go node:

  Error getting RabbitMQ connection: unable to establish a TLS connection to RabbitMQ: Exception (403) Reason: "username or password not allowed"
  

  For a Python node:

  pika.exceptions.ProbableAuthenticationError: ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'
  

• A job that is already running keeps the connections it has open, but a node that opens another one fails the same way, and the job can stop making progress until it is canceled.

• Super keeps the connection it has open, but it can no longer delete a job's queues when the job ends, and /opt/vor/log/super.log shows:

  Bad HTTP response from RabbitMQ API: 401 {"error":"not_authorized","reason":"Not_Authorized"}
  

  A later run of the same process can read the messages left in those queues. A node then finishes early, and the job reports success with missing results. Restarting Super does not delete the queues; step 4 does.

Deleting Leftover Queues¶

Super deletes a job's queues when the job ends, so with no jobs running, the queues left in VOR Stream's vhost, rabbitmq_vhost or / by default, are ones that jobs left behind. Make sure no jobs are running, then delete every queue in the vhost. If other applications share the vhost, delete only VOR Stream's queues instead.

/opt/vor/rabbitmq/sbin/rabbitmqctl list_queues --silent name | xargs -r -n 1 /opt/vor/rabbitmq/sbin/rabbitmqctl delete_queue

Dynamic RabbitMQ Credentials¶

With dynamic credentials, neither Super nor jobs sign in as rabbitmq_user. Vault's RabbitMQ secrets engine signs in to the broker's management API as rabbitmq_user and creates a broker user, with its own password, for Super and for each job. The password to change is the one Vault signs in with.

1. Change the password on the broker, as described in Changing the Password on the Broker.

2. Change the password Vault signs in with. Vault takes the connection URI, username, and password together and cannot show the ones it holds, so pass all three. Read the new password without echoing it, type it and press Enter, then write the connection:

   read -rs NEW_PASSWORD
   printf '%s' "$NEW_PASSWORD" | vault write rabbitmq/config/connection \
     connection_uri=https://<host>:<port> username=<rabbitmq_user> password=-
   unset NEW_PASSWORD
   

   The host is rabbitmq_host if the inventory sets it, and otherwise the fully qualified name of the first host in the rabbitmq group. The port is rabbitmq_management_port, 15672 by default. A broker the playbook deployed without TLS certificates uses http:// instead.

   Vault signs in with the new password before saving it. If the broker rejects it, the command fails, and Vault keeps the old one:

   failed to validate the connection: Error: API responded with a 401 Unauthorized
   

3. Check the change, as described in Checking the RabbitMQ Password.

4. Set rabbitmq_password in the inventory file to the new password.

No service restart is needed. The broker users Vault has already created keep their own passwords, so Super and running jobs keep working throughout.

Between steps 1 and 2, Vault cannot sign in to the broker. A job that starts in that window fails, and vor run shows:

cannot read secret "rabbitmq/creds/vor-midtier-role" in Vault--secret does not exist or you do not have permission to view it: ... failed to create a new user with the generated credentials

Vault cannot delete the broker users of leases that end in that window either. Its log shows failed to revoke lease with API responded with a 401 Unauthorized. Vault retries for a limited time, then gives up and leaves the broker user in place, so run steps 1 and 2 back to back.

Checking the RabbitMQ Password¶

With static credentials, check the password in the secret against the broker. On a host that has both the vault command and a broker the playbook deployed, such as a single-host installation, rabbitmqctl authenticate_user reads the password from standard input:

vault kv get -field=password kv/<path> | /opt/vor/rabbitmq/sbin/rabbitmqctl authenticate_user <rabbitmq_user>

It prints Success when the broker accepts the password, and invalid credentials when it does not.

With dynamic credentials, have Vault create a broker user and delete it again. Use the role from rabbitmq_vault_path, vor-midtier-role by default:

LEASE_ID=$(vault read -field=lease_id rabbitmq/creds/vor-midtier-role)
vault lease revoke -sync "$LEASE_ID"

Success! Revoked lease means Vault signed in to the broker both times.

With either type, run a job, and check that its output holds every row it should. The job's nodes sign in with the credentials Super gets from Vault, so a complete result shows that Vault holds a working credential.