Skip to content

we can automate management of TLS certs #400

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
VickieKarasic merged 3 commits into from
Oct 14, 2025
Merged

we can automate management of TLS certs #400

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cba47ad
we can automate management of TLS certs
kayiwa Oct 13, 2025
7e2c1a5
awk commands didn't actually work
kayiwa Oct 14, 2025
29fc84c
we use snapshots from OpenBSD
kayiwa Oct 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
283 changes: 74 additions & 209 deletions services/create_ssl_certs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@
1. If the certificate already exists you will need to revoke it before running your chosen playbook

### For sites outside the Princeton domain
1. Create the CSR (certificate signing request) - can be automated with [playbooks/cert_renewal.yml](https://github.com/pulibrary/princeton_ansible/blob/main/playbooks/incommon_certbot.yml)
2. Submit it to gandi via [this form](https://shop.gandi.net/en/certificate/create)
3. Your TLS/SSL cert will be created and returned to you via a email within 3 hours from gandi.net
1. Create a new entry under [sites](https://github.com/pulibrary/princeton_ansible/blob/dac77a6c2e0f1301201c9b2a63b9ebead5f7b7ac/group_vars/nginxplus/production.yml#L16)
2. Run the [nginxplus playbook](https://github.com/pulibrary/princeton_ansible/blob/main/playbooks/nginxplus.yml)
3. Your TLS/SSL cert will be on the production loadbalancer
4. Verify the files you get back and add them to your server configuration.

## Verifying certbot certificate renewals
Expand All @@ -28,244 +28,109 @@ Our certificate management system is Sectigo. Operations folks can [log into Sec

These certs are not managed by our usual process. These certs cover:
- sites we do not serve from the load balancers
- sites without the '.princeton.edu' extension
- vendor-hosted sites with the '.princeton.edu' extension
Many of these certs must be deployed manually. Some must also be renewed manually. If a private key is kept in princeton_ansible, it is encrypted as a file in the `/keys/` directory of the repo.

cicognara.org
Purpose: public site for the Cicognara collection (a collaborative project)
Managed: on gandi.net, private key is in princeton_ansible
Deployed: on the load balancers
cicognara.org
Purpose: public site for the Cicognara collection (a collaborative project)
Managed: [Lego](https://github.com/pulibrary/princeton_ansible/blob/main/roles/nginxplus/tasks/lego.yml)
Deployed: on the load balancers

dataspace.princeton.edu
Purpose: production site for dspace
Managed: Via [Lego](lego.md)
Deployed: on Google cloud, on prod.pulcloud.io

dataspace-dev.princeton.edu
Purpose: dev/staging site for dspace
Managed: in ServiceNow, private key is on princeton_ansible
Deployed: on Google cloud, on dev.pulcloud.io
dataspace-dev.princeton.edu
Purpose: dev/staging site for dspace
Managed: Via [Lego](lego.md)
Deployed: on Google cloud, on dev.pulcloud.io

dataspace-staging.princeton.edu

dss2.princeton.edu
Purpose: secures dataset downloads from a separate server for DSS via a web browser
Managed: in ServiceNow - John will move to letsencrypt
Deployed: on the dss2 CentOS VM
Purpose: dev/staging site for dspace
Managed: Via [Lego](lego.md)
Deployed: on Google cloud, on dev.pulcloud.io

dss2.princeton.edu
Purpose: secures dataset downloads from a separate server for DSS via a web browser
Managed: in ServiceNow - John will move to letsencrypt
Deployed: on the dss2 CentOS VM
Notes: cannot be a SAN name for the main DSS cert, because we only want to secure this functionality on one machine - can be tricky to maintain because server access requires signing nondisclosure agreements (for protected data)

ezproxy.princeton.edu
Purpose: allows access to journals by confirming Princeton affiliation
Managed: on ezproxy-prod1 by letsencrypt
Deployed: in /etc/letsencrypt/live/ezproxy on the ezproxy-prod1 server
ezproxy.princeton.edu
Purpose: allows access to journals by confirming Princeton affiliation
Managed: on ezproxy-prod1 by letsencrypt
Deployed: in /etc/letsencrypt/live/ezproxy on the ezproxy-prod1 server

imagecat2.princeton.edu
imagecat2.princeton.edu
Philippe will shut down the server once he has copied whatever we need from it. Once it's gone, we can revoke the cert.

lib-aeon.princeton.edu
Purpose: redirects traffic to hosted Aeon service at https://princeton.aeon.atlas-sys.com
Managed: for new site by the vendor
Deployed: to new site by the vendor
lib-aeon.princeton.edu
Purpose: redirects traffic to hosted Aeon service at https://princeton.aeon.atlas-sys.com
Managed: for new site by the vendor
Deployed: to new site by the vendor
Notes: We would like to redirect the old URL on the load balancers and power off the old lib-aeon machine. The templates for printing Aeon call slips, which used to live on the lib-aeon machine, have been moved to a fileshare called aeonprint on lib-fileshare.

lib-gisportal.princeton.edu
Purpose: for maps (Wangyal)
Managed: in ServiceNow
Deployed: in IIS on a physical machine that runs MS HyperV virtualization - cluster of lib-geoserv1 and lib-geoserv2 (not the Lib-Gisportal2 VM) server
lib-gisportal.princeton.edu
Purpose: for maps (Wangyal)
Managed: in ServiceNow
Deployed: in IIS on a physical machine that runs MS HyperV virtualization - cluster of lib-geoserv1 and lib-geoserv2 (not the Lib-Gisportal2 VM) server
Notes: windows physical machine, you must be an admin on the Windows box, expires 2024/07/30

lib-illsql.princeton.edu
Purpose: interlibrary loan
Managed: in ServiceNow
Deployed: in IIS, on the lib-illiad-new VM
lib-illsql.princeton.edu
Purpose: interlibrary loan
Managed: in ServiceNow
Deployed: in IIS, on the lib-illiad-new VM
Notes: Windows VM; cert has a SAN name of lib-illiad.princeton.edu; we hope to migrate this to a hosted platform in 2024

libserv97.princeton.edu
Purpose: Philippe's test machine, may disappear in 2024
Managed: in ServiceNow
libserv97.princeton.edu
Purpose: Philippe's test machine, may disappear in 2024
Managed: in ServiceNow
Deployed: directly on the libserv97 VM (dev environment)

oar.princeton.edu

oar-dev.princeton.edu
Purpose: open access repository
Managed: in ServiceNow, private key is on princeton_ansible
Deployed: on Google cloud at dev.pulcloud.io

oar-staging.princeton.edu
Purpose: open access repository
Managed: in ServiceNow, private key is on princeton_ansible
Deployed: on Google cloud at staging.pulcloud.io

pulmirror.princeton.edu
Purpose: distributing Ubuntu packages
Managed: in ServiceNow, private key is in princeton_ansible
Purpose: production site for oar
Managed: Via [Lego](lego.md)
Deployed: on Google cloud, on prod.pulcloud.io

oar-dev.princeton.edu
Purpose: production site for oar
Managed: Via [Lego](lego.md)
Deployed: on Google cloud, on prod.pulcloud.io

oar-staging.princeton.edu
Purpose: production site for oar
Managed: Via [Lego](lego.md)
Deployed: on Google cloud, on prod.pulcloud.io

pcdm.org
Purpose: Portland Common Data Model
Managed: [Lego](https://github.com/pulibrary/princeton_ansible/blob/main/roles/nginxplus/tasks/lego.yml)

pulmirror.princeton.edu
Purpose: distributing Ubuntu packages
Managed: Via [Lego](lego.md)
Deployed: on Google cloud at pulmirror.princeton.edu

recapgfa.princeton.edu
Purpose: ReCAP inventory management system
Managed: by ACME directly on the VM
recapgfa.princeton.edu
Purpose: ReCAP inventory management system
Managed: by ACME directly on the VM
Deployed: N/A - it automatically renews

simrisk.pulcloud.io
Purpose: experimental application for CDH
Managed: on staging.pulcloud.io by acme-client contacting letsencrypt CA
Deployed: in /etc/ssl/simrisk.pulcloud.io.fullchain.pem on the staging.pulcloud.io server
simrisk.pulcloud.io
Purpose: experimental application for CDH
Managed: on staging.pulcloud.io by acme-client contacting letsencrypt CA
Deployed: in /etc/ssl/simrisk.pulcloud.io.fullchain.pem on the staging.pulcloud.io server
Maintained using `/etc/daily.local` as root

tigris.princeton.edu
Purpose: hosted service for University Records management
Managed: in ServiceNow, private key is in princeton_ansible
tigris.princeton.edu
Purpose: hosted service for University Records management
Managed: in ServiceNow, private key is in princeton_ansible
Deployed: by vendor; to update, email a .pfx file of the cert to [email protected]

### Detailed instructions for sites outside the Princeton domain

#### 1. Create the Certificate Signing Request

To generate a CSR manually:

1. For a site with no Subject Alternative Name (SAN)[1]

* Export an environment variable with the host name for later commands
```
export NEW_HOST_NAME=<new host name>
```
* Create a file named `$NEW_HOST_NAME.cnf` with the following command

```ini
echo "[req]
default_bits = 2048
distinguished_name = dn
prompt = no
[dn]
C=\"US\"
ST=\"New Jersey\"
L=\"Princeton\"
O=\"The Trustees of Princeton University\"
OU="OIT"
emailAddress=\"[email protected]\"
CN=\"$NEW_HOST_NAME\"" > $NEW_HOST_NAME.cnf
```

* Generate the certificate which you will provide to
gandi.net with the following command

```bash
openssl req -out ${NEW_HOST_NAME}.csr -newkey rsa:2048 -nodes -keyout ${NEW_HOST_NAME}_priv.key -config ${NEW_HOST_NAME}.cnf
```

The step :point_up_2: above will create `${NEW_HOST_NAME}.csr` and
`${NEW_HOST_NAME}_priv.key` in your current directory.


2. For a site with a Subject Alternative Name (SAN)

* Export an environment variable with the host name for later commands
```
export NEW_HOST_NAME=<new host name>
```

* Create a file named `${NEW_HOSTNAME}_san.cnf` with the following command:

```ini
echo "[ req ]
default_bits = 4096
distinguished_name = dn
req_extensions = req_ext
prompt = no
[ dn ]
C=\"US\"
ST=\"New Jersey\"
L=\"Princeton\"
O=\"The Trustees of Princeton University\"
OU=\"OIT\"
emailAddress=\"[email protected]\"
CN=\"$NEW_HOST_NAME\"
[ req_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1 = \"${NEW_HOST_NAME}\"
DNS.2 = \"\"" > ${NEW_HOST_NAME}_san.cnf
```
* Edit the file to add your additional Alternative name

* Generate the certificate you will provide to gandi.net
with the following command

```bash
openssl req -out ${NEW_HOST_NAME}.csr -newkey rsa:4096 -nodes -keyout ${NEW_HOST_NAME}_priv.key -config ${NEW_HOST_NAME}_san.cnf
```

The step :point_up_2: above will create `${NEW_HOST_NAME}.csr` and
`${NEW_HOST_NAME}_priv.key` in your current directory.

#### 2. Submit the Certificate request to gandi.net

Submit the CSR to gandi.net. Use the following guidance:

* (SKIP if not SAN) Before submitting it you can check to see if your CSR contains the SAN you
specified in the `${NEW_HOST_NAME}_san.cnf` file by doing.

```bash
openssl req -noout -text -in ${NEW_HOST_NAME}.csr | grep DNS
```

* Provide a `cat`'ed copy of the Certificate Signing Request
with the following command

```
cat ${NEW_HOST_NAME}.csr
```

#### 4. Verify the files you get back

* Gandi.net provides the certificates in the form of an email. You will want the Certificate and Root/Intermediate files:

* `vi ${NEW_HOST_NAME}_cert.cer` and copy and paste including `-----BEGIN CERTIFICATE-----` to `-----END CERTIFICATE-----`
* `vi ${NEW_HOST_NAME}_interm.cer` and copy and paste the rest of the certificates marked as `X.509 Root/Intermediate(s)`. This should have Multiple begin and end certificates, which should be included.

1. Create the chained file from the data returned by gandi.net:

* Concatenate the certificate and the intermediate certificates to create a chained `.pem` file that includes the contents of both files:

```bash
cat ${NEW_HOST_NAME}_cert.cer ${NEW_HOST_NAME}_interm.cer > ${NEW_HOST_NAME}_chained.pem
```

2. Verify the certificates

* Make sure the certificates match (the private key must be unencrypted):

```bash
echo "--Certificate:" && openssl x509 -noout -modulus -in ${NEW_HOST_NAME}_chained.pem && echo "--Key:" && openssl rsa -noout -modulus -in ${NEW_HOST_NAME}_priv.key
```

* Make sure the CN name matches ${NEW_HOST_NAME}:

```bash
openssl x509 -in ${NEW_HOST_NAME}_chained.pem -text
```

3. Save the unencrypted private key for names that are not under ansible control

* Add the unencrypted private key to Shared-SSLCerts directory of LastPass Enterprise

4. Encrypt the private key, and add the encrypted private key and the chained file to princeton-ansible:

* Encrypt the private key with `ansible-vault` and add it to `nginxplus/files/ssl/${NEW_HOST_NAME}_priv.key`

```
cp ${NEW_HOST_NAME}_priv.key roles/nginxplus/files/ssl/
cd roles/nginxplus/files/ssl/
ansible-vault encrypt ${NEW_HOST_NAME}_priv.key
```
* add the chained file to `nginxplus/files/ssl/`

```
mv ${NEW_HOST_NAME}_chained.pem roles/nginxplus/files/ssl/
```

#### Tigris

In July of every year [tigris.princeton.edu](tigris.princeton.edu) will get an automatic renewal. The following steps will be needed to ensure the certificate remains renewed.
In July of every year [tigris.princeton.edu](tigris.princeton.edu) will get an automatic renewal. The following steps will be needed to ensure the certificate remains renewed.
* Open a ticket with tigris (aka Gimmal) support at [email protected] and ask who should receive the new chained file.
* You will need the [vaulted private key](https://github.com/pulibrary/princeton_ansible/blob/main/keys/tigris_princeton_edu_priv.key) and the certificate and intermediate certificate to generate a pfx file that you will ship to the vendor

Expand Down
Loading