Explore ACME on IdM

Redhat IPA (freeipa) AMCE in technical preview

The Automated Certificate Management Environment (ACME) service is now available in Identity Management (IdM) as a Technology Preview. <- RHEL 9.1 https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/9.1_release_notes/index#technology-preview_identity-management

In RHEL 9.2. ACME now supports automatically removing expired certificates, but no mention about it is GA https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/9.2_release_notes/index#technology-preview_identity-management

In RHEL 9.3 ACME available as a Technology Preview

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/9.3_release_notes/index#technology-preview_identity-management

P.s. IdM is a software that comes with RHEL. only need one RHEL subscription. It supports replica (like MS domain controllers replication), but for simple setup, we may consider one node as a VM and backup/snapshot by hypervisor.

Testing Environment

[root@idm ~]# cat /etc/redhat-release

Red Hat Enterprise Linux release 8.9 (Ootpa)

[root@idm ~]# ipa --version

VERSION: 4.9.12, API_VERSION: 2.251

Install IPA ACME (simple) by ipa-server-upgrade

[root@idm ~]# ipa-acme-manage enable

[root@idm ~]# ipa-acme-manage status

ACME is enabled

The ipa-acme-manage command was successful

Then, we should expect an acme page available on ipa servers.

https://idm.lab.example.com/acme/directory

certbot

Next, try if ACME work by certbot.

https://certbot.eff.org/

certbot is not come from RH subscription channel directly. we may follow the following steps to get certbot.

https://tecadmin.net/how-to-install-certbot-on-centos-9/

As it will install python and hope not mess up the working machine, use venv before installation.

python3 -m venv /home/user/venv/certbot

chmod 0755 /home/user/venv/certbot/

source /home/user/venv/certbot/bin/activate

First, we need trust ipa CA certificate, copy it to /etc/pki/ca-trust/source/ of certbot machine and update-ca-trust

[root@idm ~]# cat /etc/ipa/ca.crt <- CA certificate location

-----BEGIN CERTIFICATE-----

MIIEmjCCAwKgAwIBAgIBATANBgkqhkiG9w0BAQsFADA7MRkwFwYDVQQKDBBMQUIu

…

(certbot) [user@user ~]$ certbot register --server https://idm.lab.example.com/acme/directory -m admin@lab.example.com --agree-tos --work-dir certbot/ --logs-dir certbot/var/log/letsencrypt/ --config-dir certbot/etc/letsencrypt/

Saving debug log to /home/user/certbot/var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Would you be willing, once your first certificate is successfully issued, to

share your email address with the Electronic Frontier Foundation, a founding

partner of the Let's Encrypt project and the non-profit organization that

develops Certbot? We'd like to send you email about our work encrypting the web,

EFF news, campaigns, and ways to support digital freedom.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

(Y)es/(N)o: Y

Account registered.

(certbot) [user@user certbot]$ certbot certonly --server https://idm.lab.example.com/acme/directory --manual --preferred-challenges dns -d test-certbot.lab.example.com --work-dir certbot/ --logs-dir certbot/var/log/letsencrypt/ --config-dir certbot/etc/letsencrypt/

Saving debug log to /home/user/certbot/certbot/var/log/letsencrypt/letsencrypt.log

Requesting a certificate for test-certbot.lab.example.com

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Please deploy a DNS TXT record under the name:

_acme-challenge.test-certbot.lab.example.com.

with the following value:

iGKEO8CE_9fDzTiMSqEWiJyDOU1HYwXczyKtb-4_uhQ

Before continuing, verify the TXT record has been deployed. Depending on the DNS

provider, this may take some time, from a few seconds to multiple minutes. You can

check if it has finished deploying with aid of online tools, such as the Google

Admin Toolbox: https://toolbox.googleapps.com/apps/dig/#TXT/_acme-challenge.test-certbot.lab.example.com.

Look for one or more bolded line(s) below the line ';ANSWER'. It should show the

value(s) you've just added.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Press Enter to Continue

<- Before Enter, go the ipa server and add dns txt records,

ipa dnsrecord-add lab.example.com. _acme-challenge.test-certbot.lab.example.com. --txt-data=iGKEO8CE_9fDzTiMSqEWiJyDOU1HYwXczyKtb-4_uhQ

Successfully received certificate.

Certificate is saved at: /home/user/certbot/certbot/etc/letsencrypt/live/test-certbot.lab.example.com/fullchain.pem

Key is saved at: /home/user/certbot/certbot/etc/letsencrypt/live/test-certbot.lab.example.com/privkey.pem

This certificate expires on 2024-02-26.

These files will be updated when the certificate renews.

NEXT STEPS:

- This certificate will not be renewed automatically. Autorenewal of --manual certificates requires the use of an authentication hook script (--manual-auth-hook) but one was not provided. To renew this certificate, repeat this same certbot command before the certificate's expiry date.

We were unable to subscribe you the EFF mailing list because your e-mail address appears to be invalid. You can try again later by visiting https://act.eff.org.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

If you like Certbot, please consider supporting our work by:

* Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate

* Donating to EFF: https://eff.org/donate-le

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Verify the certificate in IdM console.

Authentication > Certificates

https://idm.lab.example.com/ipa/ui/#/e/cert/search

IPA is working with certbot and DNS challenge but both are not yet supported.

Even it said not supported (yet), example in acme page are refer to certbot

Install cert-manager on Openshift

Prepare CA certificate in base64

[root@idm ~]# base64 -w 0 /etc/ipa/ca.crt

LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVtakNDQXdLZ0F3SUJBZ0lCQVRBTkJ…

Create clusterIssuer instead of issuer (can be used not limited to namespace).

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: acme-lab-issuer
spec:
  acme:
    caBundle: >-
      LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0...N4ZFczU1oxTSUNBVEUtLS0tLQ==
    disableAccountKeyGeneration: true
    email: admin@lab.example.com
    preferredChain: ''
    privateKeySecretRef:
      name: ipa-issuer-account-key
    server: 'https://idm.lab.example.com/acme/directory'
    solvers:
      - dns01:
          rfc2136:
            nameserver: 192.168.40.10
            tsigAlgorithm: HMACSHA512
            tsigKeyName: acme-update
            tsigSecretSecretRef:
              key: tsig-secret-key
              name: ipa-tsig-secret

[user@user ~]$ oc project

Using project "cert-manager-operator" on server "https://api.ocp4.example.com:6443".

[user@user ~]$ oc get clusterissuer -o wide

NAME READY STATUS AGE

acme-lab-issuer True The ACME account was registered with the ACME server 3h57m

Test with DNS01 challenge, prepare in IdM.

[root@idm ~]# ipa-dns-install --dnssec-master

The log file for this installation can be found in /var/log/ipaserver-dns-install.log

==============================================================================

This program will setup DNS for the IPA Server.

This includes:

* Configure DNS (bind)

* Configure SoftHSM (required by DNSSEC)

* Configure ipa-dnskeysyncd (required by DNSSEC)

* Configure ipa-ods-exporter (required by DNSSEC key master)

* Configure OpenDNSSEC (required by DNSSEC key master)

* Generate DNSSEC master key (required by DNSSEC key master)

NOTE: DNSSEC zone signing is not enabled by default

Plan carefully, replacing DNSSEC key master is not recommended

To accept the default shown in brackets, press the Enter key.

Do you want to setup this IPA server as DNSSEC key master? [no]: yes

Do you want to configure DNS forwarders? [yes]:

Following DNS servers are configured in /etc/resolv.conf: 127.0.0.1

Do you want to configure these servers as DNS forwarders? [yes]: no

Enter an IP address for a DNS forwarder, or press Enter to skip: 8.8.8.8

DNS forwarder 8.8.8.8 added. You may add another.

Enter an IP address for a DNS forwarder, or press Enter to skip: 8.8.4.4

DNS forwarder 8.8.4.4 added. You may add another.

Enter an IP address for a DNS forwarder, or press Enter to skip:

DNS forwarders: 8.8.8.8, 8.8.4.4

Checking DNS forwarders, please wait ...

Do you want to search for missing reverse zones? [yes]: no

The following operations may take some minutes to complete.

Please wait until the prompt is returned.

Configuring DNS (named)

[1/8]: generating rndc key file

[2/8]: setting up our own record

[3/8]: adding NS record to the zones

[4/8]: setting up kerberos principal

[5/8]: setting up named.conf

[6/8]: setting up server configuration

[7/8]: configuring named to start on boot

[8/8]: changing resolv.conf to point to ourselves

Done configuring DNS (named).

Restarting the web server to pick up resolv.conf changes

Configuring DNS key synchronization service (ipa-dnskeysyncd)

[1/7]: checking status

[2/7]: setting up bind-dyndb-ldap working directory

[3/7]: setting up kerberos principal

[4/7]: setting up SoftHSM

[5/7]: adding DNSSEC containers

[6/7]: creating replica keys

[7/7]: configuring ipa-dnskeysyncd to start on boot

Done configuring DNS key synchronization service (ipa-dnskeysyncd).

Configuring IPA OpenDNSSEC exporter daemon (ipa-ods-exporter)

[1/5]: setting up DNS Key Exporter

[2/5]: setting up kerberos principal

[3/5]: disabling default signer daemon

[4/5]: starting DNS Key Exporter

[5/5]: configuring DNS Key Exporter to start on boot

Done configuring IPA OpenDNSSEC exporter daemon (ipa-ods-exporter).

Configuring OpenDNSSEC enforcer daemon (ods-enforcerd)

[1/7]: setting up configuration files

[2/7]: setting up ownership and file mode bits

[3/7]: generating master key

[4/7]: setting up OpenDNSSEC

[5/7]: setting up ipa-dnskeysyncd

[6/7]: starting OpenDNSSEC enforcer

[7/7]: configuring OpenDNSSEC enforcer to start on boot

Done configuring OpenDNSSEC enforcer daemon (ods-enforcerd).

Restarting ipa-dnskeysyncd

Restarting named

Updating DNS system records

==============================================================================

Setup complete

Global DNS configuration in LDAP server is not empty

The following configuration options override local settings in named.conf:

Global forwarders: 8.8.8.8

Forward policy: only

Allow PTR sync: True

IPA DNS servers: idm.lab.example.com

IPA DNSSec key master: idm.lab.example.com

You must make sure these network ports are open:

TCP Ports:

* 53: bind

UDP Ports:

* 53: bind

[root@idm ~]#

[root@idm ~]# ipa dnszone-mod lab.example.com --dynamic-update=True --update-policy='grant acme-update wildcard * ANY;'

[root@idm ~]# ipa dnszone-show lab.example.com.

Zone name: lab.example.com.

Active zone: True

Authoritative nameserver: idm.lab.example.com.

Administrator e-mail address: hostmaster.lab.example.com.

SOA serial: 1701166300

SOA refresh: 3600

SOA retry: 900

SOA expire: 1209600

SOA minimum: 3600

BIND update policy: grant acme-update wildcard * ANY; grant LAB.EXAMPLE.COM krb5-self * A; grant LAB.EXAMPLE.COM krb5-self * AAAA; grant LAB.EXAMPLE.COM krb5-self * SSHFP;

Dynamic update: True

Allow query: any;

Allow transfer: any;

Allow in-line DNSSEC signing: False

[root@idm ~]#

tsig-keygen -a hmac-sha512 acme-update >> /etc/named/ipa-ext.conf

****

[root@idm KeyID]# dnssec-keygen -a HMAC-SHA512 -b 512 -n HOST acme-update

Kacme-update.+165+35177

[root@idm KeyID]# ls

Kacme-update.+165+35177.key Kacme-update.+165+35177.private

[root@idm KeyID]# cat Kacme-update.+165+35177.private

Private-key-format: v1.3

Algorithm: 165 (HMAC_SHA512)

Key: +GmvX0YPG/hG4ia0YmW4Lw5nCXPMhGtV5eZxb9PyCzpC7weoR8RYFiE37kLIt8oA/jTR0atufGrtaep4lzN39Q==

Bits: AAA=

Created: 20231128110925

Publish: 20231128110925

Activate: 20231128110925

[root@idm KeyID]# vi /etc/named/ipa-ext.conf

[root@idm KeyID]# systemctl restart named-pkcs11.service

[root@idm KeyID]# nsupdate -k Kacme-update.+165+35177.key

> update add www1.lab.example.com 60 txt testing

> send

> update delete www1.lab.example.com txt

> send

> quit

[root@idm KeyID]#

<- verify if the Key work with nsupdate, if it failed, never work in cert-manager.

update ipa-ext.conf and restart (systemctl restart named-pkcs11.service)

[root@idm ~]# cat /etc/named/ipa-ext.conf /* User customization for BIND named * * This file is included in /etc/named.conf and is not modified during IPA * upgrades. * * "options" settings must be configured in /etc/named/ipa-options-ext.conf. * * Example: ACL for recursion access: * * acl "trusted_network" { * localnets; * localhost; * 234.234.234.0/24; * 2001::co:ffee:babe:1/48; * }; */ key "acme-update" { algorithm hmac-sha512; secret "+GmvX0YPG/hG4ia0YmW4Lw5nCXPMhGtV5eZxb9PyCzpC7weoR8RYFiE37kLIt8oA/jTR0atufGrtaep4lzN39Q=="; }; [root@idm ~]#

troubleshoot

[user@user ~]$ oc project cert-manager-operator

Already on project "cert-manager-operator" on server "https://api.ocp4.example.com:6443".

[user@user ~]$ oc create secret generic ipa-tsig-secret --from-literal=tsig-secret-key="+GmvX0YPG/hG4ia0YmW4Lw5nCXPMhGtV5eZxb9PyCzpC7weoR8RYFiE37kLIt8oA/jTR0atufGrtaep4lzN39Q==" ← wrong steps, need to be cert-manager instead of cert-manager-operator

secret/ipa-tsig-secret created

[user@user ~]$

[user@user ~]$ oc get certificate -o yaml

…

message: 'The certificate request has failed to complete and will be retried:

Failed to wait for order resource "test-ssl-shcsz-1318501929" to become ready:

order is in "errored" state: Failed to create Order: 400 urn:ietf:params:acme:error:malformed:

invalid label in dns identifier: ``' <- becareful of dns field input mistake

Normal OrderPending 20m cert-manager-certificaterequests-issuer-acme Waiting on certificate issuance from order test-ssl/test-ssl-jk2lq-1250072317: ""

[user@user ~]$ oc get certificaterequest

NAME APPROVED DENIED READY ISSUER REQUESTOR AGE

test-ssl-fzc6q True False acme-lab-issuer system:serviceaccount:cert-manager:cert-manager 3m3s

OrderPending and never come back. Need to go to the cetr-manger namespace for troubleshooting.

E1128 23:02:02.859322 1 controller.go:167] "cert-manager/challenges: re-queuing item due to error processing" err="secret \"ipa-tsig-secret\" not found" key="test-ssl/test-ssl-lmt2q-1250072317-58234220"

So add the DNS01 challenge secret to the namesapce

[user@user ~]$ oc project cert-manager

Now using project "cert-manager" on server "https://api.ocp4.example.com:6443".

[user@user ~]$ oc create secret generic ipa-tsig-secret --from-literal=tsig-secret-key="+GmvX0YPG/hG4ia0YmW4Lw5nCXPMhGtV5eZxb9PyCzpC7weoR8RYFiE37kLIt8oA/jTR0atufGrtaep4lzN39Q=="

secret/ipa-tsig-secret created

E1128 23:07:09.151897 1 controller.go:208] "cert-manager/challenges: challenge in work queue no longer exists" err="challenge.acme.cert-manager.io \"test-ssl-lmt2q-1250072317-58234220\" not found"

I1128 23:07:09.165481 1 conditions.go:192] Found status change for Certificate "test-ssl" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2023-11-28 23:07:09.16546479 +0000 UTC m=+400104.824942479

I1128 23:07:09.239298 1 controller.go:162] "cert-manager/certificates-issuing: re-queuing item due to optimistic locking on resource" key="test-ssl/test-ssl" error="Operation cannot be fulfilled on certificates.cert-manager.io \"test-ssl\": the object has been modified; please apply your changes to the latest version and try again"

I1128 23:07:09.404872 1 controller.go:162] "cert-manager/certificates-key-manager: re-queuing item due to optimistic locking on resource" key="test-ssl/test-ssl" error="Operation cannot be fulfilled on certificates.cert-manager.io \"test-ssl\": the object has been modified; please apply your changes to the latest version and try again"

[user@user ~]$ oc get certificate

NAME READY SECRET AGE

test-ssl True test-ssl-tls 7m53s

[user@user ~]$ oc get certificaterequest

NAME APPROVED DENIED READY ISSUER REQUESTOR AGE

test-ssl-lmt2q True True acme-lab-issuer system:serviceaccount:cert-manager:cert-manager 8m9s

[user@user ~]$

[user@user ~]$ oc get secret

NAME TYPE DATA AGE

builder-dockercfg-bpblg kubernetes.io/dockercfg 1 17h

builder-token-f657p kubernetes.io/service-account-token 4 17h

default-dockercfg-cg5f6 kubernetes.io/dockercfg 1 17h

default-token-zh2ng kubernetes.io/service-account-token 4 17h

deployer-dockercfg-snl9g kubernetes.io/dockercfg 1 17h

deployer-token-ndmv8 kubernetes.io/service-account-token 4 17h

ipa-tsig-secret Opaque 1 11h

test-ssl-tls kubernetes.io/tls 2 2m53s

[user@user ~]$

Further test, create and remove certificates.

[user@user ~]$ oc get cert

NAME READY SECRET AGE

test-ssl True test-ssl-tls 100m

test-ssl1 True test-ssl1-tls 72m

[user@user ~]$ oc delete cert test-ssl1

certificate.cert-manager.io "test-ssl1" deleted

Need RHEL 9.2 version and Set ACME to automatically remove expired certificates from the CA:

# ipa-acme-manage pruning --enable --cron "0 0 1 * *"

Expired certificates are removed after their retention period. By default, this is 30 days after expiry.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/managing_certificates_in_idm/deploying-and-managing-the-acme-service-in-idm_managing-certificates-in-idm#doc-wrapper

May need to manually revoke it.

Also secrets are not removed, good is safer if you still need the certificate. Otherwise one more step to remove it.

Test with application

[user@user ~]$ oc project test-ssl

Already on project "test-ssl" on server "https://api.ocp4.example.com:6443".

[user@user ~]$ oc new-app quay.io/redhattraining/hello-openshift

--> Found container image 7af3297 (5 years old) from quay.io for "quay.io/redhattraining/hello-openshift"

* An image stream tag will be created as "hello-openshift:latest" that will track this image

--> Creating resources ...

imagestream.image.openshift.io "hello-openshift" created

Warning: would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (container "hello-openshift" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "hello-openshift" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "hello-openshift" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "hello-openshift" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

deployment.apps "hello-openshift" created

service "hello-openshift" created

--> Success

WARNING: No container image registry has been configured with the server. Automatic builds and deployments may not function.

Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:

'oc expose service/hello-openshift'

Run 'oc status' to view your app.

[user@user ~]$

[user@user ~]$ cat test-ssl.yaml apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: test-ssl-tls spec: isCA: false commonName: 'test-ssl.lab.example.com' secretName: test-ssl dnsNames: - test-ssl.lab.example.com issuerRef: name: acme-lab-issuer kind: ClusterIssuer

Observations

Still have a lot of rooms to improve

Document is not clear (or no document at all)
Empty in Certificate page <- suppose it will show the certificate created

Empty in CertificateRequest page <- suppose it will show the CertificateRequest
Misleading privateKeySecretRef <- eventually it does not need

privateKeySecretRef:

name: ipa-issuer-account-key

Seem a pull request to update about it.

https://github.com/cert-manager/cert-manager/issues/1751

Ports

1. It need outgoing port 443 to communicate from the cert-manager pods to ACME server.

2. For ACME challenge, most common is http01 and dns01, somehow it will ask to proof the domain or dns is owned by you.

2.1 For http01, seem like it will temporary create a pod and expose 80 port which AMCE servers get the "Key" from the web page. So Http01 incoming port 80 is required. Assume that dns is correct to resolve that http request.