feat(managesieve): add XOAUTH2 authentication mechanism

docs/modules/servers/partials/configure/sieve.adoc

@@ -86,4 +86,28 @@ Optional integer, defaults to 2 times the count of CPUs.
 | maxExecutorCount
 | Set the maximum count of worker threads. Worker threads takes care of potentially blocking tasks like executing ManageSieve commands.
 Optional integer, defaults to 16.
-|===
+
+| oidc
+| If this property is present, OIDC will be configured and the following properties are mandatory (unless otherwise specified).
+
+| oidc.oidcConfigurationURL
+| Your identity provider's OIDC discovery URL. This is currently not used for managesieve but is still required when OIDC is configured.
+
+| oidc.jwksURL
+| URL to the endpoint for the JSON Web Key Set of your provider. This is used to locally validate tokens.
+
+| oidc.claim
+| Name of the claim in the token you want to use as the identifier for the user (e.g. "email_address").
+
+| oidc.scope
+| OIDC scope. This is currently not used for managesieve but is still required when OIDC is configured.
+
+| oidc.introspection.url
+| URL to your identity provider's introspection endpoint. It is optional and if specified James will use the endpoint to validate the token in addition to local validation.
+
+| oidc.introspection.auth
+| Provide Authorization header for introspection requests (optional, e.g. `Basic xyz`).
+
+| oidc.userinfo.url
+| URL to your identity provider's userinfo endpoint. It is optional and if specified James will use the endpoint to validate the token in addition to local validation. James will ignore this option if `oidc.introspection.url` is already configured.
+|===

examples/oidc/README.md

@@ -6,12 +6,10 @@ This is example of an OIDC setup with James.
 
 The API Gateway for example is [Apisix](https://apisix.apache.org/), we can use Apisix for websocket gateway, horizontal scaling, etc...
 
-This [docker-compose](docker-compose.yml) will start the following services:
+This [docker compose](./compose.yaml) will start the following services:
 
 - apisix: The image `linagora/apisix:3.2.0-debian-javaplugin` was created by Linagora. It based on `apisix:3.2.0-debian`, it already contain apisix plugin for 
   SLO (Single Logout) and rewrite the `X-User` header. 
-  - Dockerfile: [here](https://github.com/linagora/tmail-backend/blob/master/demo/apisix/Dockerfile)
-  - Project `tmail-apisix-plugin-runner`: [here](https://github.com/linagora/tmail-backend/tree/master/demo/apisix/tmail-apisix-plugin-runner)
   - Apisix being the OIDC gateway against James by exposing two endpoints:
       - `POST /jmap` for JMAP requests against James with normal authentication
       - `POST /oidc/jmap` for JMAP request against James with a JWT token issued by the LemonLDAP
@@ -161,21 +159,21 @@ Use websocket with endpoint `ws://apisix.example.com:9080/oidc/jmap/ws` and the
 
 We would use Thunderbird version 91.4.1 as a mail client (above versions should work).
 * Open `/thunderbird/omni.ja` in your host, find and modify `OAuth2Providers.jsm`:
-  * Add James hostname in kHostnames: `["localhost", ["james.local", "email"]],`
+  * Add James hostname in kHostnames: `["localhost", ["james.example.com", "email"]],`
   * Register using `james-thunderbird` Keycloak client in kIssuers:
   ```
   [
-    "james.local",
+    "james.example.com",
     [
       "james-thunderbird", //client_id from keycloak
       "Xw9ht1veTu0Tk5sMMy03PdzY3AiFvssw", // client_secret from keycloak
-      "http://keycloak.local:8080/auth/realms/oidc/protocol/openid-connect/auth",
-      "http://keycloak.local:8080/auth/realms/oidc/protocol/openid-connect/token",
+      "http://sso.example.com:8080/auth/realms/oidc/protocol/openid-connect/auth",
+      "http://sso.example.com:8080/auth/realms/oidc/protocol/openid-connect/token",
     ],
   ] 
   ```
 
-* Adding a line `127.0.0.1 keycloak.local` to your `/etc/hosts` so Thunderbird can resolve the address of keycloak.
+* Adding a line `127.0.0.1 sso.example.com` to your `/etc/hosts` so Thunderbird can resolve the address of keycloak.
 * Run Thunderbird, configure it using `james-user@localhost` account against these IMAP/SMTP settings:
   * IMAP: server: localhost, port: 143, connection security: No, authentication method: OAUTH2
     ![](_media/imap-setting.png)
@@ -189,4 +187,35 @@ We would use Thunderbird version 91.4.1 as a mail client (above versions should
   ![](_media/receive-mail.png)
 
 A remark here is that if you generate a new client_secret for `james-thunderbird` client in Keycloak, you have to modify
-it accordingly in `OAuth2Providers.jsm`.
+it accordingly in `OAuth2Providers.jsm`.
+
+### IMAP on the CLI
+
+You can test logging into IMAP on the CLI by connecting with `telnet localhost 143`. Here are some commands that can be tried:
+
+- `a AUTHENTICATE XOAUTH2 <initial response>` (unauthenticated state)
+- `b AUTHENTICATE OAUTHBEARER <initial response>` (unauthenticated state)
+- `c LOGOUT` (any state)
+
+You can get the initial response from the [test script](./test.sh).
+
+### ManageSieve on the CLI
+
+You can test logging into IMAP on the CLI by connecting with `telnet localhost 4190`. Here are some commands that can be tried:
+
+- `AUTHENTICATE "XOAUTH2" "<initial response>"` (unauthenticated state)
+- `AUTHENTICATE "OAUTHBEARER" "<initial response>"` (unauthenticated state)
+- `CAPABILITY` (any state)
+- `LOGOUT` (any state)
+
+You can get the initial response from the [test script](./test.sh).
+
+### SMTP on the CLI
+
+You can test logging into IMAP on the CLI by connecting with `telnet localhost 587`. Here are some commands that can be tried:
+
+- `AUTH XOAUTH2 <initial response>` (unauthenticated state)
+- `AUTH OAUTHBEARER <initial response>` (unauthenticated state)
+- `QUIT` (any state)
+
+You can get the initial response from the [test script](./test.sh).

examples/oidc/apisix/conf/apisix.yaml

@@ -219,7 +219,7 @@ upstreams:
   -
     id: jmap_upstream
     nodes:
-      "james:80": 1
+      "james.example.com:80": 1
     type: roundrobin
 
 plugin_configs:

examples/oidc/docker-compose.yml → examples/oidc/compose.yaml

@@ -1,5 +1,3 @@
-version: "3"
-
 services:
   apisix:
     container_name: apisix.example.com
@@ -8,59 +6,62 @@ services:
       - ./apisix/conf/apisix.yaml:/usr/local/apisix/conf/apisix.yaml
       - ./apisix/conf/config.yaml:/usr/local/apisix/conf/config.yaml
     environment:
-      - X_USER_SECRET=xusersecret123
+      X_USER_SECRET: xusersecret123
     networks:
       - james
     ports:
-      - "9080:9080/tcp"
+      - "127.0.0.1:9080:9080"
 
   james:
     depends_on:
       - ldap
     networks:
       - james
     image: apache/james:memory-latest
-    container_name: james
-    hostname: james.local
-    command:
-      - --generate-keystore
+    container_name: james.example.com
+    hostname: james.example.com
+    command: [--generate-keystore]
     volumes:
       - ./james/usersrepository.xml:/root/conf/usersrepository.xml
       - ./james/jmap.properties:/root/conf/jmap.properties
+      - ./james/imapserver.xml:/root/conf/imapserver.xml
+      - ./james/smtpserver.xml:/root/conf/smtpserver.xml
+      - ./james/managesieveserver.xml:/root/conf/managesieveserver.xml
     ports:
-      - "8000:8000"
+      - "127.0.0.1:8000:8000"
+      - "127.0.0.1:143:143"
+      - "127.0.0.1:587:587"
+      - "127.0.0.1:4190:4190"
     healthcheck:
       test: ["CMD", "curl", "-f", "http://james:8000/domains"]
 
-  sso.example.com:
+  sso:
     depends_on:
       - ldap
     image: quay.io/keycloak/keycloak:16.1.0
     container_name: sso.example.com
     volumes:
       - ./keycloak/realm-oidc.json:/tmp/realm-oidc.json
     ports:
-      - "8080:8080"
+      - "127.0.0.1:8080:8080"
     environment:
-      - KEYCLOAK_USER=admin
-      - KEYCLOAK_PASSWORD=admin
-      - KEYCLOAK_IMPORT=/tmp/realm-oidc.json
+      KEYCLOAK_USER: admin
+      KEYCLOAK_PASSWORD: admin
+      KEYCLOAK_IMPORT: /tmp/realm-oidc.json
     networks:
-      james:
-        aliases:
-          - keycloak
+      - james
 
   ldap:
-    container_name: ldap
+    container_name: ldap.example.com
     image: osixia/openldap:1.5.0
     ports:
-      - "389:389"
-      - "636:636"
+      - "127.0.0.1:389:389"
+      - "127.0.0.1:636:636"
     command: [--copy-service]
     volumes:
       - ./ldap/populate.ldif:/container/service/slapd/assets/config/bootstrap/ldif/data.ldif
     environment:
-      - LDAP_DOMAIN=localhost
+      LDAP_DOMAIN: localhost
     networks:
       - james
 
@@ -71,7 +72,7 @@ services:
     networks:
       - james
     ports:
-      - "6379:6379"
+      - "127.0.0.1:6379:6379"
 
 networks:
-  james:
+  james:

examples/oidc/james/imapserver.xml

@@ -4,12 +4,6 @@
         <jmxName>imapserver</jmxName>
         <bind>0.0.0.0:143</bind>
         <connectionBacklog>200</connectionBacklog>
-        <tls socketTLS="false" startTLS="false">
-            <keystore>file://conf/keystore</keystore>
-            <keystoreType>PKCS12</keystoreType>
-            <secret>james72laBalle</secret>
-            <provider>org.bouncycastle.jce.provider.BouncyCastleProvider</provider>
-        </tls>
         <connectionLimit>0</connectionLimit>
         <connectionLimitPerIP>0</connectionLimitPerIP>
         <idleTimeInterval>120</idleTimeInterval>
@@ -18,12 +12,12 @@
         <auth>
             <plainAuthEnabled>true</plainAuthEnabled>
             <oidc>
-                <oidcConfigurationURL>http://keycloak:8080/auth/realms/oidc/.well-known/openid-configuration</oidcConfigurationURL>
-                <jwksURL>http://keycloak:8080/auth/realms/oidc/protocol/openid-connect/certs</jwksURL>
+                <oidcConfigurationURL>http://sso.example.com:8080/auth/realms/oidc/.well-known/openid-configuration</oidcConfigurationURL>
+                <jwksURL>http://sso.example.com:8080/auth/realms/oidc/protocol/openid-connect/certs</jwksURL>
                 <claim>email</claim>
                 <scope>openid profile email</scope>
                 <introspection>
-                    <url>http://keycloak:8080/auth/realms/oidc/protocol/openid-connect/token/introspect</url>
+                    <url>http://sso.example.com:8080/auth/realms/oidc/protocol/openid-connect/token/introspect</url>
                     <auth>Basic amFtZXMtdGh1bmRlcmJpcmQ6WHc5aHQxdmVUdTBUazVzTU15MDNQZHpZM0FpRnZzc3c=</auth>
                 </introspection>
             </oidc>

examples/oidc/james/managesieveserver.xml

@@ -0,0 +1,21 @@
+<?xml version="1.0"?>
+<managesieveservers>
+    <managesieveserver enabled="true">
+        <jmxName>managesieveserver</jmxName>
+        <bind>0.0.0.0:4190</bind>
+        <connectionBacklog>200</connectionBacklog>
+        <connectiontimeout>360</connectiontimeout>
+        <connectionLimit>0</connectionLimit>
+        <connectionLimitPerIP>0</connectionLimitPerIP>
+        <oidc>
+            <oidcConfigurationURL>http://sso.example.com:8080/auth/realms/oidc/.well-known/openid-configuration</oidcConfigurationURL>
+            <jwksURL>http://sso.example.com:8080/auth/realms/oidc/protocol/openid-connect/certs</jwksURL>
+            <claim>email</claim>
+            <scope>openid profile email</scope>
+            <introspection>
+                <url>http://sso.example.com:8080/auth/realms/oidc/protocol/openid-connect/token/introspect</url>
+                <auth>Basic amFtZXMtdGh1bmRlcmJpcmQ6WHc5aHQxdmVUdTBUazVzTU15MDNQZHpZM0FpRnZzc3c=</auth>
+            </introspection>
+        </oidc>
+    </managesieveserver>
+</managesieveservers>

examples/oidc/james/smtpserver.xml

@@ -4,26 +4,19 @@
         <jmxName>smtpserver</jmxName>
         <bind>0.0.0.0:587</bind>
         <connectionBacklog>200</connectionBacklog>
-        <tls socketTLS="false" startTLS="false">
-            <keystore>file://conf/keystore</keystore>
-            <keystoreType>PKCS12</keystoreType>
-            <secret>james72laBalle</secret>
-            <provider>org.bouncycastle.jce.provider.BouncyCastleProvider</provider>
-            <algorithm>SunX509</algorithm>
-        </tls>
         <connectiontimeout>360</connectiontimeout>
         <connectionLimit>0</connectionLimit>
         <connectionLimitPerIP>0</connectionLimitPerIP>
         <auth>
             <announce>forUnauthorizedAddresses</announce>
             <plainAuthEnabled>true</plainAuthEnabled>
             <oidc>
-                <oidcConfigurationURL>http://keycloak:8080/auth/realms/oidc/.well-known/openid-configuration</oidcConfigurationURL>
-                <jwksURL>http://keycloak:8080/auth/realms/oidc/protocol/openid-connect/certs</jwksURL>
+                <oidcConfigurationURL>http://sso.example.com:8080/auth/realms/oidc/.well-known/openid-configuration</oidcConfigurationURL>
+                <jwksURL>http://sso.example.com:8080/auth/realms/oidc/protocol/openid-connect/certs</jwksURL>
                 <claim>email</claim>
                 <scope>openid profile email</scope>
                 <introspection>
-                    <url>http://keycloak:8080/auth/realms/oidc/protocol/openid-connect/token/introspect</url>
+                    <url>http://sso.example.com:8080/auth/realms/oidc/protocol/openid-connect/token/introspect</url>
                     <auth>Basic amFtZXMtdGh1bmRlcmJpcmQ6WHc5aHQxdmVUdTBUazVzTU15MDNQZHpZM0FpRnZzc3c=</auth>
                 </introspection>
             </oidc>
@@ -39,5 +32,3 @@
         </handlerchain>
     </smtpserver>
 </smtpservers>
-
-

examples/oidc/james/usersrepository.xml

@@ -22,7 +22,7 @@
 
 <usersrepository name="LocalUsers"
                  class="org.apache.james.user.ldap.ReadOnlyUsersLDAPRepository"
-                 ldapHost="ldap://ldap:389"
+                 ldapHost="ldap://ldap.example.com:389"
                  principal="cn=admin,dc=localhost"
                  credentials="admin"
                  userBase="ou=people,dc=localhost"

examples/oidc/test.sh

@@ -1,8 +1,10 @@
 #!/bin/sh
 
-set -eux
+# You need to start the compose project before running this script!
 
-docker-compose up -d
+set -eu
+# Uncomment the following line to print more information.
+# set -x
 
 GET_TOKEN_RESPONSE=`curl --location 'http://sso.example.com:8080/auth/realms/oidc/protocol/openid-connect/token' \
                                   --header 'Content-Type: application/x-www-form-urlencoded' \
@@ -14,7 +16,9 @@ GET_TOKEN_RESPONSE=`curl --location 'http://sso.example.com:8080/auth/realms/oid
                                   --data-urlencode 'password=secret' 2>/dev/null`
 
 ACCESS_TOKEN=`echo $GET_TOKEN_RESPONSE 2>/dev/null |perl -pe 's/^.*"access_token"\s*:\s*"(.*?)".*$/$1/'`
+echo "Access token: $ACCESS_TOKEN"
 REFRESH_TOKEN=`echo $GET_TOKEN_RESPONSE 2>/dev/null |perl -pe 's/^.*"refresh_token"\s*:\s*"(.*?)".*$/$1/'`
+echo "Refresh token: $REFRESH_TOKEN"
 
 echo "Got an access_token"
 if curl -H "Authorization: Bearer $ACCESS_TOKEN" http://sso.example.com:8080/auth/realms/oidc/protocol/openid-connect/userinfo 2>/dev/null| grep james-user >/dev/null; then
@@ -23,15 +27,55 @@ else
 	echo "ACCESS_TOKEN VERIFICATION FAILED"
 fi
 
-echo -n "Trying James: "
-
+echo -n "Trying James:"
 APISIX_JMAP_ENDPOINT=apisix.example.com:9080/oidc/jmap/session
 if curl -v -H 'Accept: application/json; jmapVersion=rfc-8621' -H "Authorization: Bearer $ACCESS_TOKEN" $APISIX_JMAP_ENDPOINT 2>/dev/null | grep uploadUrl >/dev/null; then
 	echo "OK"
 else
 	echo "Not OK"
 fi
 
+XOAUTH2_INITIAL_CLIENT_RESPONSE=`echo -n -e "user=james-user@localhost\x01auth=Bearer ${ACCESS_TOKEN}\x01\x01" | base64 -w 0`
+echo "XOAUTH2: $XOAUTH2_INITIAL_CLIENT_RESPONSE"
+OAUTHBEARER_INITIAL_CLIENT_RESPONSE=`echo -n -e "n,a=james-user@localhost\x01auth=Bearer ${ACCESS_TOKEN}\x01\x01" | base64 -w 0`
+echo "OAUTHBEARER: $OAUTHBEARER_INITIAL_CLIENT_RESPONSE"
+
+MANAGESIEVE_XOAUTH2_RESPONSE=`(echo "AUTHENTICATE \"XOAUTH2\" \"${XOAUTH2_INITIAL_CLIENT_RESPONSE}\""; echo "CAPABILITY"; echo "LOGOUT"; sleep 3) | telnet 127.0.0.1 4190`
+if echo "$MANAGESIEVE_XOAUTH2_RESPONSE" | grep "\"OWNER\" \"james-user@localhost\"" > /dev/null; then
+	echo "Success: Managesieve XOAUTH2 login"
+else
+	echo "Error: Managesieve XOAUTH2 login"
+fi
+if echo "$MANAGESIEVE_XOAUTH2_RESPONSE" | grep "OK channel is closing" > /dev/null; then
+	echo "Success: Managesieve XOAUTH2 logout"
+else
+	echo "Error: Managesieve XOAUTH2 logout"
+fi
+
+IMAP_XOAUTH2_RESPONSE=`(echo "a AUTHENTICATE XOAUTH2 ${XOAUTH2_INITIAL_CLIENT_RESPONSE}"; echo "c LOGOUT"; sleep 3) | telnet 127.0.0.1 143`
+if echo "$IMAP_XOAUTH2_RESPONSE" | grep "a OK AUTHENTICATE completed" > /dev/null; then
+	echo "Success: IMAP XOAUTH2 login"
+else
+	echo "Error: IMAP XOAUTH2 login"
+fi
+if echo "$IMAP_XOAUTH2_RESPONSE" | grep "c OK LOGOUT completed" > /dev/null; then
+	echo "Success: IMAP XOAUTH2 logout"
+else
+	echo "Error: IMAP XOAUTH2 logout"
+fi
+
+SMTP_XOAUTH2_RESPONSE=`(echo "AUTH XOAUTH2 ${XOAUTH2_INITIAL_CLIENT_RESPONSE}"; echo "QUIT"; sleep 3) | telnet 127.0.0.1 587`
+if echo "$SMTP_XOAUTH2_RESPONSE" | grep "235 Authentication successful" > /dev/null; then
+	echo "Success: SMTP XOAUTH2 login"
+else
+	echo "Error: SMTP XOAUTH2 login"
+fi
+if echo "$SMTP_XOAUTH2_RESPONSE" | grep "221 2.0.0 james.example.com Service closing transmission channel" > /dev/null; then
+	echo "Success: SMTP XOAUTH2 logout"
+else
+	echo "Error: SMTP XOAUTH2 logout"
+fi
+
 # Logout
 
 curl --location 'http://sso.example.com:8080/auth/realms/oidc/protocol/openid-connect/logout' \