Configuring TLS between Domino and Exchange

Introduction

In the modern messaging world Transport Layer Security has become the norm and given the ease at which Microsoft Exchange is able to achieve this out of the box, organisations quite rightly demand this when they are on Coexistence mode between IBM Domino and Microsoft Exchange.

Up to Domino 8.5.x Domino does not actually do genuine TLS – it actually does HTTPS v3 for Message Transport. Apparently Domino 9 will do TLS natively.

Below I have outlined the steps to Configure Domino 8.5.3 and Exchange 2010 for TLS.

Domino Server Configuration

  • Create Domino CSR
  • Create Key Ring
  • Create Certificate Request
  • Sent to CA
  • Install trusted Root Certificate into Key Ring
  • View & Edit Key Rings
  • View Certificate Request Log
  • Install Certificate Into Key Ring
  • Configure Domino
  • Enable SSL on Domino
  • Enable TLS for SMTP Outbound mail
  • Enable TLS for SMTP Inbound mail
  • Configure notes.ini
  • Replicate changes to Domino Environment
  • Restart The Router and SMTP tasks

Exchange Server Configuration

  • Configure Receive Connectors
  • CHECK Config
  • Configure Send Connectors
  • CHECK Config

 

Domino Server Configuration

1. Create Domino CSR

1. Create Key Ring

Create Key Ring

Create Key Ring2

Create Key Ring3

Create Key Ring4

 

 

2. Create Certificate Request

Create Certificate Request1

Create Certificate Request2

Create Certificate Request3

Create Key Ring4

 

—–BEGIN NEW CERTIFICATE REQUEST—–
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
—–END NEW CERTIFICATE REQUEST—–

 

Sent to CA

3. Install trusted Root Certificate into Key Ring

Install trusted Root Certificate into Key Ring1

 

Install trusted Root Certificate into Key Ring2*

Install trusted Root Certificate into Key Ring3

 

—–BEGIN CERTIFICATE—–

***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
—–END CERTIFICATE—–

 

Install trusted Root Certificate into Key Ring4

Install trusted Root Certificate into Key Ring5

Install trusted Root Certificate into Key Ring6

 

4. View & Edit Key Rings

View & Edit Key Rings1

View & Edit Key Rings2

View & Edit Key Rings3

View & Edit Key Rings4

 

5. View Certificate Request Log

View Certificate Request Log1

 

View Certificate Request Log2

 

View Certificate Request Log3

6. Install Certificate Into Key Ring

 

Install Certificate Into Key Ring1

 

Install Certificate Into Key Ring2

 

 

Configure Domino

Enable SSL on Domino

  • Create SSL Cert Request, send to EnTrust, Retrieve CER file, Complete Cert Request
    • Save SSL key file name file to Domino Data directory
  • Server Document: Server Ports, Internet Ports, SSL key File Name, set SSL ciphers to RC$ 128-bit only (MD5 and SHA-1)
  • Restart HTTP Service
  • Don’t need HTTPS running for SSL

 

Enable TLS for SMTP Outbound mail

  • Server Document: Server Ports, Internet Ports, Set TCPIP port status to Negotiated SSL

Enable TLS for SMTP Outbound mail

 

 

Enable TLS for SMTP Inbound mail

  • Server Configuration Document: Router\SMTP, Advanced, Commands and Extensions set SSL negotiated over TCP/IP port: to ENABLED

Enable TLS for SMTP Inbound mail

Configure notes.ini

Check /or set the following notes.ini settings

HTTPJVMMaxHeapSize=64M

  • HTTPJVMMaxHeapSizeSet=1
  • HTTPDisableMethods=TRACE
  • SSL_USE_ADDSESSION2=1
  • SSL_SESSION_SIZE=8192
  • SSLCipherSpec=0405

 

Debug Settings
If Anything Goes wrong, use these:

;SMTPClientDebug=0
;SMTPDebug=0
;SMTPDebugIO=0
;SMTPSaveOutboundToFile=1
;SMTPSaveImportErrors=0

;HTTPDebugLogDirectpry=<path>
;HTTPEnabledThreadDebug=1
;HTTPEnablePostDataLogging=1
;HTTPEnableResponseContentLogging=1
;DEBUG_SSL_ALL=1
;TraceSSLHandshake=1
;ReportSSLHandshakeErrors=1

 

Replicate changes to Domino Environment

  • Replicate changes to Domino affected server if necessary

Restart The Router and SMTP tasks

  • Stop and start the Domino Router and SMTP tasks

 

Exchange Server Configuration

Configure Receive Connectors

Get-ReceiveConnector | ? { $_ -like ‘*domino*’ } | Set-ReceiveConnector -RequireTLS $true -ProtocolLoggingLevel Verbose

Get-ReceiveConnector | ? { $_ -like ‘*domino’ } | Set-ReceiveConnector -AuthMechanism Tls -PermissionGroups AnonymousUsers

Get-ReceiveConnector | ? { $_ -like ‘*domino’ } | Add-ADPermission -User “NT AUTHORITY\ANONYMOUS LOGON” -ExtendedRights “Ms-Exch-SMTP-Accept-Any-Recipient”

CHECK Config

Get-ReceiveConnector | ? { $_ -like ‘*domino*’ } | FL Identity,RequireTLS,ProtocolLoggingLevel

Configure Send Connectors

Get-SendConnector | ? { $_ -like ‘*domino*’ } | Set-SendConnector -TlsAuthLevel $null -IgnoreSTARTTLS $false -RequireTLS $true -ProtocolLoggingLevel Verbose

CHECK Config

Get-SendConnector | ? { $_ -like ‘*domino*’ } | FL Identity,TlsAuthLevel,RequireTLS,IgnoreSTARTTLS,ProtocolLoggingLevel

Restart Transport Service

Restart-Service MSExchangeTransport

 

Conrad Murray

Conrad Murray has been working in IT for over 15 years specializing in the Messaging Arena and in particular IBM Domino and Microsoft Exchange and now of course Office 365. Working with like minded colleagues now specializing in very large scale complex migrations from Lotus Notes and On-Premise Microsoft Exchange to Office 365.

Leave a Reply

Your email address will not be published. Required fields are marked *

Search