White Paper: Continuous Replication Deep Dive

 

Applies to: Exchange Server 2007 SP1

Ross Smith IV, Technology Architect; Scott Schnoll, Principal Technical Writer, Microsoft Exchange Server

July 2008

Summary

This white paper provides the technical details for a technology in Microsoft Exchange Server 2007 called continuous replication.

Applies To

Microsoft Exchange Server 2007 Service Pack 1 (SP1)

Table of Contents

• Introduction

• Continuous Replication Basics

• Replication Components

  • Information Store Service Functions

  • Replication Service Functions

• Replication Service Components

• Replication Service Configuration Information

• Replication Service Communication Coordination

• Log Shipping and Replay

  • Log Inspection

  • Log Replay

• Scheduled Outages

• Unscheduled Outages

• Lost Log Resilience

• Transport Dumpster

• Incremental Reseed

• Database Reseed Scenarios

• Standby Continuous Replication

• Continuous Replication and Backups

• Log Truncation

• Comparing Continuous Replication with Other Replication Solutions

• Conclusion

• Additional Information

Introduction

Continuous replication (also known as log shipping and replay) is new technology in Microsoft Exchange Server 2007 that creates and maintains database copies to provide full high availability and disaster recovery solutions for mailbox databases. With the release of Exchange Server 2007 Service Pack 1 (SP1), there are now three forms of continuous replication.

• Local Continuous Replication (LCR)   LCR is a single-server solution that uses continuous replication to create and maintain a copy of a storage group on a second set of disks connected to the same server as the active storage group copy. For more information, see Local Continuous Replication.

• Cluster Continuous Replication (CCR)   CCR is a high availability feature that provides failover capabilities for e-mail service and mailbox data, using continuous replication to provide one copy of data redundancy with clustering technology for the server redundancy. For more information, see Cluster Continuous Replication.

• Standby Continuous Replication (SCR)   SCR is a disaster recovery feature that employs continuous replication to create and maintain multiple copies of a storage group. It is provided as a feature to facilitate site resiliency. For more information, see Standby Continuous Replication.

Continuous Replication Basics

Continuous replication leverages the Exchange Server database crash recovery support to provide technology that performs asynchronous updating of one or more copies of a mailbox database. Each mailbox server records database updates made on an active database (for example, user e-mail activity) as log records in a sequential set of 1 MB transaction log files. This set of files is referred to as the log stream. The log stream is used for backup and server crash recovery purposes.

In continuous replication, the log stream is also used to asynchronously update one or more copies of a database. This is accomplished by shipping the logs to a location containing a copy of the active storage group and then replaying them into the database copy. If all logs from the active database are replayed against a copy of the database then the two databases are equivalent.

Continuous replication is configured and managed at the storage group level. However, because continuous replication requires that a storage group contain only one database, continuous replication effectively operates at the database level.

This white paper will delve into the internals of continuous replication, and explain key components of the Mailbox server role that make continuous replication possible.

In simple terms, continuous replication follows these steps:

1. Create a target database by seeding the destination with a copy of the source database.

2. Monitor for new logs in source log directory for copying by subscribing to Windows file system notification events.

3. Copy any new log files to the destination inspection log directory.

4. Inspect the copied log files.

5. Upon successful inspection, move the copied log file to the storage group copy’s log path and replay the copied log into the copy of the database.

Replication Components

The two key services responsible for log generation, log shipping, and log replay activity are:

• Microsoft Exchange Information Store service   Responsible for servicing user and application requests, performing write ahead logging, and updating the database file via Extensible Storage Engine (ESE).

• Microsoft Exchange Replication Service   Responsible for log shipping and replay against the database copy.

Information Store Service Functions

The following steps are performed by ESE when data is retrieved from, inserted into, or modified in a database.

1. An operation occurs against the database (a client sends a new message), and the page that requires updating is read from the database file and placed into the ESE cache (assuming the page is not already in memory), while the log buffer is notified and it records the operation in memory.

2. The changes are recorded by the database engine but these changes are not immediately written to the database file on disk. Instead, these changes are held in the ESE cache and they are known as dirty pages because they have not been committed to the database file. The version store is used to keep track of these changes, thus ensuring that isolation and consistency are maintained.

3. As the database pages are changed, the log buffer is notified to commit the change, and the transaction is recorded in a transaction log file, which may or may not require closing the current Exx.log file and starting a new log generation. (Note that ESE is also responsible for closing a log file once it reaches its maximum file size (1 MB) and starting a new generation.)

4. Eventually the dirty database pages are written to the database file on disk.

5. The checkpoint is advanced.

Replication Service Functions

When continuous replication is enabled, the Microsoft Exchange Replication Service (the "Replication service") is responsible for detecting when the current log file is closed by ESE, copying it, inspecting it, and replaying it into the database copy. This service is installed by default on all servers that have the Mailbox server role installed.

The executable for the Replication service is Microsoft.Exchange.Cluster.ReplayService.exe, which is located by default at <install path>\bin. The Replication service depends on the Microsoft Exchange Active Directory Topology Service. The Replication service can be stopped and started using the Services snap-in or from the command line, and it is also configured to automatically restart in case of a failure or exception.

The Replication service stores its diagnostic logging level setting in the Windows registry in the following location: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchange Repl\Diagnostics

For more information on diagnostics logging and the associated settings, see Diagnostic Logging of Exchange Processes.

Return to top

Replication Service Components

The Replication service uses several components to provide continuous replication between an active storage group and one or more copies of the active storage group. The following are a few of the components that are involved in the continuous replication process and pipeline.

• LogCopier   Responsible for copying closed log files from the source storage group to the target server that contains a copy of the storage group. This is an asynchronous operation in which the Replication service continuously monitors the source storage group's log file directory. It does this monitoring by subscribing to Windows file system notification events. When a new log file is closed on the source, an event fires which notifies the Replication service that a new file exists. The LogCopier will then copy the log file to the inspector location on the target server.

• LogInspector   Responsible for verifying that the log files are valid. It checks the destination inspector directory on a regular basis. If a log file is found to be corrupted or unusable for replay, the Replication service will request a re-copy of the file.

• LogReplayer   Responsible for replaying inspected log files into the database copy.

• LogTruncater   Responsible for deleting log files that have been successfully replayed into the database copy. This is especially important because typically after a full or incremental backup, the log files that are below the checkpoint are deleted because the log records contained within those log files have been written to the database. When continuous replication is used, the LogTruncator only deletes log files that are not needed for recovery and replay. Any log files on the active copy that have not been replicated and replayed into the database copy are not deleted by an online backup of the active copy.

• Incremental Reseeder   Responsible for ensuring that the active database and database copies are not diverged after a database restore has been performed, or after a failover has occurred in a CCR environment.

• Seeder   Responsible for creating the baseline content of a storage group used to start replay processing. The Replication service performs automatic seeding for new storage groups, and for any existing storage groups that still have all generated logs (including the first log file, which contains the CreateDB record) available.

• Replay Manager   Responsible for keeping track of all replica instances. It creates and destroys replica instances on-demand, based on the online status of the storage group. The configuration of a replica instance is intended to be static. Therefore, when a replica instance configuration is changed, the replica will be restarted with the updated configuration. In addition, during shutdown of the Replication service, replica instance configuration is not saved. As a result, each time the Replication service starts it has an empty replica instance list. During startup, the replay manager performs discovery of the storage groups that are currently online and creates a "running instance" list.

  The replay manager periodically runs a configupdater thread to scan for newly configured replica instances. The configupdater thread runs in the Replication service process every 30 seconds for LCR or CCR, and every 3 minutes for SCR. It will create and destroy a replica instance based on the current database state (whether the database is online or offline). The configupdater thread uses the following algorithm:

  1. Read instance configuration from Active Directory.

  2. Compare list of configurations found in Active Directory against running storage groups/databases. The Active Directory configuration for an instance is also checked to see if it has changed, if so the instance is put on a restart list.

  3. Produce a list of running instances to stop and a list of configurations to start.

  4. Stop running instances on the stop list.

  5. Start instances on the start list.

  Effectively, therefore, the replay manager always has a dynamic list of the replica instances.

Replication Service Configuration Information

Each storage group and database enabled for LCR has the msExchHasLocalCopy attribute defined. The Replication service uses the following algorithm to search Active Directory for replica information.

1. Find the Exchange Server object in Active Directory using the computer name. If there is no server object then return.

2. Enumerate all storage groups that are on this Exchange server.

   1. For each storage group with msExchHasLocalCopy set to true, retrieve the source and destination paths for the system files, log files, and database file.

In a CCR environment, the Replication service performs the following tasks to retrieve the clustered mailbox server’s configuration.

1. Establishes a connection to the cluster database.

2. Determines which node owns the clustered mailbox server.

3. Enumerates all storage groups that exist on the source and target nodes, as well as various settings used for replication such as:

   1. Source and destination paths for the system files, log files, and database file.

   2. Returns the last owner of the storage group (assuming the storage group has been mounted before).

   3. The network share used for log shipping.

   4. The AutoDatabaseMountDial setting. See "Incremental Reseeder" earlier in this white paper.

   5. The ForcedDatabaseMountAfter setting

   6. Determine the appropriate network path for log shipping.

4. Verifies that the configuration on the source equals that of the target.

For SCR environments, the Replication service uses the msExchStandbyCopyMachines attribute to identify which storage groups are enabled for replication, and it performs the following tasks:

1. Find the Exchange Server object in Active Directory using the computer name. If there is no server object then return.

2. Enumerate all storage groups that are on this Exchange server. For each storage group with msExchStandbyCopyMachines attribute, do the following:

   • Retrieve the ReplayLagTime and TruncationLagTime parameters for each target server.

   • Retrieve the source and destination paths for the system files, log files, and database file.

Replication Service Communication Coordination

The Replication service communicates between the active instance and the passive instance via three mechanisms.

• Log Files   Certain information, like backup status and database header changes, is recorded within the log files on the active instance and replicated to the copy, thereby updating the database copy. For more information, see "Continuous Replication and Backups" later in this white paper.

• Cluster database or registry   The Replication service stores replay state data on a per-storage group basis in either the cluster database (CCR) or the registry (LCR/SCR). The data that is written to and read from these locations is for persistent state information, such as:

  • Dumpster variables that are used when transport dumpster is invoked.

  • Activation variables like whether the database can be mounted if it is configured to allow no loss.

  • Replication status for the storage group; for example, Suspended or Failed.

  In the RTM version of Exchange 2007, the registry and cluster database were also used to store runtime state (where exactly we are in the replication pipeline). However, in SP1 this was changed so that the management tasks communicate directly with the Replication service via RPC to obtain this information. The runtime state data is still written to the registry and cluster database; however the data is no longer read from these locations by the management tasks.

• Store process   The Replication service will generate RPC calls to the Store process on the active instance for log truncation coordination. For more information, see "Log Truncation" later in this white paper.

Return to top

Log Shipping and Replay

Now that we have a basic understanding of the core components that are involved, we can discuss how log shipping works. Before log files can be shipped to a passive copy, the active database copy must first be seeded. This can be accomplished in a few ways.

1. Automatic seeding   An automatic seed produces a copy of a storage group's database in the target location. Automatic seeding requires that all log files, including the very first log file created by the storage group (it contains the database creation log record), be available on the source. Automatic seeding only occurs during the creation of a new server or creation of a new storage group and database.

2. Seeding using the Update-StorageGroupCopy cmdlet   You can use the Update-StorageGroupCopy cmdlet in the Exchange Management Shell to seed a storage group copy. You can also use the Update Storage Group Copy wizard in the Exchange Management Console to seed a storage group copy.

3. Manually copying the offline database   This process dismounts the database and copies the database file to the same location on the passive node. If you use this method, there will be an interruption in service because the procedure requires you to dismount the database.

The second option utilizes the streaming copy backup API to copy the database from the active location to the target location.

The Replication service also ensures all of the relevant paths are created for the log inspector directory, the log directory, and the checkpoint file directory.

In a CCR or SCR environment, there are two Replication services that are involved in copying storage group data. One is on the source node; the other is on the passive node or target. The source node's service is responsible for creating read-only hidden shares for each storage group’s log directory. The share has the following structure:

• Share Name: <Storage Group GUID>$

• Folder Path: <drive>:\<Storage Group Log Folder Path>

• Share Permissions: <root domain>\Exchange Servers – Read

• Folder Permissions: <root domain>\Exchange Servers – Read, <computer>\Administrators – Full Control, SYSTEM – Full Control

  Note

  Although files can be listed and copied, when you view the share via Windows Explorer, you will see that the Exchange Servers security group does not have the 'Read' permissions checkbox checked. This is due to an access flag not being set.

Depending on the environment, the SCR target's service will connect using a particular namespace.

• The Replication service on the CCR passive node always connects to \\nodenameActive to pull logs to the passive copy.

• The Replication service on the SCR target, when connecting to a CCR source environment, connects to \\nodenameActive to pull logs to the SCR target.

• The Replication service on the SCR target, when connecting to a SCC source environment, connects to \\CMSName to pull logs to the SCR target.

• The Replication service on the SCR target, when connecting to a standalone mailbox server, connects to \\ServerName to pull logs to the SCR target.

Each time the Replication service is started, it will check to ensure that each storage group on the mailbox server has the log folder shared. When new storage groups are created, a share is created. If the shares exist, they will not be recreated. If the permissions on the shares are modified, the Replication service will not re-apply the permissions. For this scenario, simply remove the share and cycle the Replication service on the source node to recreate it.

The Replication service on the target uses file system notifications to detect new log files as they are closed. The following steps explain how the Replication service handles the log copy, inspection, and replay. For each of these steps, corresponding markers are updated and maintained by the target node Replication service. These markers are used by the Replication service to track where it is in the log replication process (stored in the cluster database and available via Get-StorageGroupCopyStatus and the target node’s Performance Monitor).

1. A log file is closed in the source log directory. LastLogCopyNotified is updated with the last generation that was seen in the source directory.

   Note

   The store process updates the LastLogGenerated field in the cluster database (or registry) every 20 seconds. The Replication service on the target also watches for file change notifications and updates its internal counter which is reflected in the state (see the section "Replication Service Communication Coordination" for more information). As a result, if the Replication service is not running on the target, then the LastLogGenerated value will be used as the marker and it could be 20 seconds out of date.

2. The target node Replication service utilizes a process at the managed code level (LogCopier) to pull the log files to the target node over Server Message Block (SMB). The copied log files are then placed in an Inspector directory that exists within the storage group’s log folder. The logs are placed in an inspector directory so that they are isolated and not accidentally replayed into the copy database until after they have gone through inspection. LastLogCopied is updated when a log is successfully copied into the Inspector directory.

3. The log files are inspected via LogInspector and moved to the passive log directory. LastLogInspected is updated when a log is successfully inspected. For more information, see "Log Inspection" later in this white paper.

   1. If log copying or the log file inspection fails, the Replication service will attempt to recopy the log file and inspect it again. The Replication service will do this three times before marking the log file as irrecoverable and requiring a database reseed.

4. The log file is replayed into the database copy by the Replication service (LogReplayer). LastLogReplayed is updated when a log is successfully replayed into the database. For more information, see "Log Replay" later in this white paper.

   Note

   In RTM, LCR utilized a batching logic for log replay. By default, the log replay queue batch threshold was set to 10. This was the minimum number of logs to replay at once. If no logs were received during the log replay batching timer duration (default 60 seconds) then a smaller batch size was used. This logic was removed in SP1 due to the architectural change of keeping the passive database instances up between log replay iterations which reduced the I/O load on the passive instance from the RTM load of 2-3 times the active instances to .5-1 times the active instances. For more information, see New High Availability Features in Exchange 2007 SP1.

Return to top

Log Inspection

The following actions are performed by LogInspector:

1. Physical integrity inspection   This validation utilizes ESEUTIL /K against the log file and validates that the checksum recorded in the log file matches the checksum generated in memory.

2. Header inspection   The Replication service validates the following aspects of the log file’s header:

   1. The generation is not higher than the highest generation recorded for the storage group in question.

   2. The generation recorded in the log header matches the generation recorded in the log filename.

   3. The log file signature recorded in the log header matches that of the log file.

3. Removal of Exx.log   Before the inspected log file can be moved into the log folder, the Replication service needs to remove any Exx.log files. These log files are placed into another sub-directory of the log directory, the ExxOutofDate directory. An Exx.log file would only exist on the target if it was previously running as a source. The Exx.log file needs to be removed before log replay occurs because it will contain old data which has been superseded by a full log file with the same generation. If the closed log file is not a superset of the existing Exx.log file, then we will have to perform an incremental or full reseed. For more information on full reseed, see "Database Reseed Scenarios" later in this white paper.

Log Replay

After the log files have been inspected, they are placed within the log directory so that they can be replayed in the database copy. Before the Replication service replays the log files, it performs a series of validation tests.

1. Recalculates the required log generations necessary for log replay to be successful. This check determines the highest and lowest log generations required by inspecting the database header.

   1. The database header contains a log signature and this will be compared with the log file’s signature to ensure they match.

   2. The database header will report one of two conditions: Either the database is in a consistent state (Clean Shutdown) or an inconsistent state (Dirty Shutdown). If the database is consistent, then the next log file to be committed log position (LGPOS) will be used to derive the lowest and highest required log generations. If the database is in an inconsistent state, then the “Required Log” stamp defines the lowest and highest required generations. In addition, the backup information that is stored in the database header is reviewed to determine the last generation that was backed up.

2. Determines the highest generation that is present in the log directory to ensure that a log file exists. If there are no log files, then replay cannot continue.

3. Compare the highest log generation that is present in the directory to the highest log file that is required. As long as the highest log generation that is present is equal to or higher than the highest required log generation, replay can continue.

4. Make sure the logs form the correct sequence. Checking all the log files in the directory is a slow process, so only the required log files are validated. This is done to determine if recovery will fail immediately (the assumption being that if recovery fails on generation N, the database header will require generation N-1). In addition to validating that all the required log files are available, the following additional validation steps are performed:

   1. Ensures the generation of a log file matches the sequence expectation.

   2. Ensures the log file's signature is correct.

   3. Ensures the creation time of the log file forms a sequence with previous log files.

5. Query the checkpoint file, if one exists.

   1. If the checkpoint file is too advanced by defining a generation that is higher than the lowest generation required, then the checkpoint file will be deleted.

   2. If the checkpoint is too far behind and points to a non-existent file, then the checkpoint file will be deleted.

Once these validation checks have been completed, the Replication service will replay the log iteration. This is actually a special recovery mode, which is different from the replay performed by ESEUTIL /R. For more information, see Eseutil /R Recovery Mode. Typically when ESEUTIL /R is executed, it replays all available log files including the Exx.log file, thus ensuring all transactions are either committed to the database or rolled back. However, since in continuous replication the replication engine is always one log file behind (the active Exx.log file), the undo phase of recovery (where uncommitted transactions are rolled back) is skipped to ensure database divergence does not occur.

After the log file is replayed, the following additional steps are performed.

1. Validate that the required generations in the database header were updated. There are certain situations in which the database header will not get updated after a log replay event. This occurs when the log file that was replayed only contains termination and initialization records. In this scenario, however, the database should be in a consistent state.

2. Execute the LogTruncater phase to remove any unnecessary logs from the passive and the active path. This call reads the database header for full and incremental backup information, so replays are blocked for this duration. For more information on how log truncation is performed, see "Log Truncation" later in this white paper.

Scheduled Outages

During a scheduled outage, the active storage group instance is brought down in a clean manner and the passive instance is activated and becomes the new active storage group. In this scenario, no data loss will occur. As the active storage groups are taken offline, the open log file is closed. The Replication service copies any remaining logs it has not yet copied, which includes highest generated log (the open Exx.log file which was previously closed) for each storage group. The logs for each storage group are inspected and then replayed into the corresponding database copies. The database copies are then activated and thus, contain the same data as the source.

Unscheduled Outages

Since continuous replication involves shipping closed log files, the active Exx.log file that is actively being written to by ESE and is still open at the time of a failure, cannot be copied to the passive. As described in the transport dumpster section the vast majority of any data represented by this open log is resubmitted by the Hub Transport servers after failover. If we consider the cluster scenario, there are many different failures that can occur, such as a server hardware failure, storage hardware failure, and operating system failure, to name a few. In the event that an unscheduled outage occurs that affects the active node, the passive node will bring the clustered mailbox server instance online and the Replication service on that node will attempt to copy the missing log files from the node that experienced the failure. If the copy process is successful (for example, because the server is online and the shares and necessary data are accessible), then the storage groups will mount and there will be zero data loss. If the copy process is unsuccessful, then the databases will be mounted based on the clustered mailbox server’s AutoDatabaseMountDial setting and how far behind it is in log replication per storage group. There are three possible values for the server setting AutoDatabaseMountDial.

• Lossless   Lossless is zero logs lost. When the attribute is set to Lossless, under most circumstances the system waits for the failed node to come back online before databases are mounted. Even then the failed system must return with all logs accessible and not corrupted. After the failure, the passive node is made active, and the Microsoft Exchange Information Store service is brought online. It checks to determine whether the databases can be mounted without any data loss. If possible, the databases are mounted. If they cannot be automatically mounted, the system periodically attempts to copy the logs. If the server returns with its logs intact, this attempt will eventually succeed, and the databases will mount. If the server returns without its logs intact, the remaining logs will not be available, and the affected databases will not mount automatically. In this event, administrative action is required to force the database to mount when logs are lost.

• Good availability   Good availability is three logs lost. Good availability provides fully automatic recovery when replication is operating normally and replicating logs at the rate they are being generated.

• Best availability   Best availability is six logs lost, which is the default setting. Best availability operates similarly to Good availability, but it allows automatic recovery when the replication experiences slightly more latency. Thus, the new active node might be slightly farther behind the state of the old active node after the failover, thereby increasing the likelihood that database divergence occurs, which requires a full reseed to correct.

Return to top

Lost Log Resilience

In the case of a lossy failure, there is the potential for the original active and new active copies of a database to diverge, requiring a full reseed of the database on the original active when it recovers. A feature of ESE called lost log resilience (LLR) provides protection of the original copy of the database in order to minimize the need to perform full database reseeds.

In a lossy failure, there is at least one log file missing per storage group (Exx.log), but there also could be additional closed log files missing. This means that if the databases are brought online, data stored on NodeA (the failed node) is different than the data being generated on NodeB. This is referred to as divergence.

Divergence is when a storage group copy has information that is not in the active storage group. Divergence can be in the database or in the log files and can be caused by lossy failures, for example. Administrators can also cause divergence (by performing an offline defrag or hard repair of the active copy).

Divergence is problematic because it means that data has been lost. In particular, database divergence is the worst case because it guarantees the need to reseed, which can be an expensive operation in terms of time and possibly bandwidth. Log file divergence also means data has been lost. However, log file divergence doesn’t necessarily cause database divergence because of LLR.

Remember that the order of write operations of Exchange data is always memory, log file, and then database file. LLR works by delaying writes to the database until the specified number of log generations have been created. LLR delays recent updates to the database file for a short time. The length of time that writes are delayed depends on how quickly logs are being generated.

LLR provides the ability to force database changes to be held in memory until one or more additional log generations are created. In a nutshell, this forces the active database file to remain a few generations behind the log files that are created, thus reducing the likelihood of database divergence between the copies.

LLR introduces a new marker for log files. From a log replay perspective, there have always been two key markers.

• Committed   The committed marker tells ESE the last log generated. The term is somewhat of a misnomer because it does not mean that the log records contained within the log file are actually written to the database file.

• Checkpoint   The checkpoint marker tells ESE the minimum log file required for replay. All log files prior to (below) the checkpoint contain log records that have been written into the database file.

Exchange 2007 added an additional marker for LLR and divergence detection.

• Waypoint   The waypoint marker tells ESE the maximum log file required for replay. All logs prior to (below) the waypoint are required for recovery as some of those logs contain data that has been committed to the database. All log files after (above) the waypoint have not been committed to the database.

To ensure understanding of these markers, let’s look at an example output of a database that was not shut down cleanly.

Initiating FILE DUMP mode...

Database: priv1.edb

...

State: Dirty Shutdown

Log Required: 2-10 (0x2-0xA)

Log Committed: 0-20 (0x0-0x14)

...

The output of this EDB file tells us the three markers.

• Committed is log generation 20 (the last log generation listed in “Log Committed”).

• Checkpoint is log generation 2 (the first log generation listed in “Log Required”).

• Waypoint is log generation 10 (the last log generation listed in “Log Required”).

This means that log generations 11 through 20 have not been committed to the EDB file. But what exactly does this mean? Since log generations 11 through 20 have not been committed to the database, they can be discarded by LLR. As long as we have log generations 2 through 10, we do not have to perform a database reseed.

The LLR depth value depends on the mailbox server configuration. In CCR environments running SP1, the LLR depth is a value of 10, which means that at any given time, data contained in the last 10 log files have not been committed to the database file. For all other SP1 mailbox servers (SCC, and standalone mailbox servers with or without LCR), the LLR depth is a value of 1.

In the absence of user or database activity, ESE now also forces the active log file to close, thus ensuring that if there is data within that log file, it will be replicated to the database copies. The log roll behavior is based on the value of the LLR depth. The log roll mechanism does not generate transaction logs in the absence of user or other database activity. In fact, log roll is designed to occur only when there is a partially filled log. To calculate when log roll should occur (provided the conditions for log rolling are met), the system uses the following formula:

[15 (minutes) ÷ LLR Depth value] = Frequency of log roll activity (in minutes)

The following table lists the maximum number of transaction log files that will be generated each day by an idle storage group as a result of log roll activity.

Maximum number of logs generated each day due to log roll activity

Mailbox server configuration	Maximum number of transaction logs generated per day by an idle storage group
Stand-alone (with or without LCR) Single copy cluster	96
Cluster continuous replication	960

Consider a CCR environment where NodeA owns the clustered mailbox server and NodeB is the passive node. NodeA is generating and closing log files and NodeB is copying the closed log files and replaying them into the passive copy of the database.

NodeA generating log files and shipping them to NodeB

Now let’s assume NodeA fails. In this case, we can see that NodeA created generation 4 and generation 5 log files for storage group 1, but neither log file was copied to NodeB before NodeA failed.

NodeA has failed and NodeB begins generating log files as the new active node, starting with generation 4

As a result of the failure, the following will occur on NodeB:

1. The Replication service will attempt to copy the missing log files from NodeA in order to prevent a lossy activation of the storage groups.

2. If the copy attempt fails, the Replication service will calculate the log loss by subtracting LastLogInspected from LastLogGenerated for each storage group and comparing this value with the clustered mailbox server’s AutoDatabaseMountDial value. If the difference is less than the AutoDatabaseMountDial value, then the storage group will mount. If the difference is more than the AutoDatabaseMountDial value, then the storage group will not mount.

   Note

   If the databases are beyond the AutoDatabaseMountDial value in terms of missing logs, they will not automatically mount. In this scenario, the Replication service will "wake up" every 30 seconds and try to contact the passive node (the original active node that failed) to copy the missing log files. If it can copy enough log files to reduce the "lossy-ness" to an acceptable amount, then the database will come online. Also, there is an option to specify a time by which you want to force the databases to mount via the setting ForcedDatabaseMountAfter, however this feature should only be used with extreme caution as it can cause database divergence.

For each storage group that mounts, the Replication service will initiate a transport dumpster redelivery request to recover recent messages that had been submitted to the clustered mailbox server immediately before the time of failure. This is discussed in detail in the "Transport Dumpster" section.

Also note that as a result of storage group 1 being mounted on NodeB, the storage group will generate log files and will continue with the generation sequence of the log stream, starting with log generation 4. As can be seen from the figure above, storage group 1 generates generation 4 through generation 6 log files. How Exchange deals with the log streams that exist on NodeA and NodeB when NodeA returns is discussed in the "Incremental Reseed" section.

Return to top

Transport Dumpster

Transport dumpster is a feature built into Exchange 2007 designed to minimize data loss by redelivering recently submitted messages back to the mailbox server after a lossy failure.

If we return to our NodeA/NodeB scenario, log generations 1, 2, and 3 have been committed to the database on NodeA. Those logs have also been replicated to NodeB and have been replayed into its database. Then, NodeA failed and log generations 4 and 5 were never replicated to NodeB. NodeB took ownership of the clustered mailbox server, brought the storage group online and initiates the transport dumpster redelivery process that is discussed within this section. The log generations 4 and 5 that were generated on NodeA, but never replicated to NodeB may contain actual data. For example, they might contain the following:

• Data that was submitted to a transport server (messages or appointment invites)

• Non-transport related user data (calendar entries, contacts, tasks, notes, drafts, property update, and so on)

• Database maintenance records (online maintenance)

• Log rolls

For now let’s focus on the first bullet point. In Exchange 2007, all e-mail must go through a Hub Transport server. This was mainly to allow for compliance (regulatory and corporate), but it also provides a means for data recovery.

In Exchange 2007, whenever a Hub Transport server receives a message, it undergoes categorization. Part of the categorization process involves querying Active Directory to determine if the destination storage group containing the recipient’s mailbox is enabled for LCR or CCR. Once the message has been delivered to all recipients, the message is committed to the mail.que file on the Hub Transport server and stored in the transport dumpster inside the mail.que file. The transport dumpster is available for each storage group within each Active Directory site that has LCR or CCR enabled. There are two settings that define the life of a message within the transport dumpster. They are:

• MaxDumpsterSizePerStorageGroup   Defines the size available for each storage group on the Hub Transport server. The recommendation is that this be set to 1.5 times the maximum message size limit within your environment. If you do not have a maximum message size limit set, then you should evaluate the messages that are delivered within your environment and set the value to 1.5 times the average message size in your organization. The default value for this setting is 18 MB.

• MaxDumpsterTime   Defines the length of time that a message remains within the transport dumpster if the dumpster size limit is not reached. The default is seven days.

If either the time or size limit is reached, messages are removed from the transport dumpster by order of first in, first out.

When a failover (unscheduled outage) occurs, the Replication service will attempt to copy the missing log files. If the copy attempt fails, then this is known as a lossy failover and the following steps are taken.

1. If the databases are within the AutoDatabaseMountDial value, they will automatically mount. For LCR, the dumpster trigger is only initiated when Restore-StorageGroupCopy is exercised.

2. The Replication service will record that the storage group requires transport dumpster redelivery in the cluster database (or in the case of LCR, within the registry) by setting the DumpsterRedeliveryRequired key to true.

3. The Replication service will record the Hub Transport servers that exist within the clustered mailbox server’s Active Directory site in the cluster database (or in the case of LCR, within the registry) setting DumpsterRedeliveryServers.

4. The Replication service will calculate the loss window. This is done using the LastLogInspected marker as the start time and the current time as the end time. Since the transport dumpster is based on message delivery times, we generously pad the loss window by expanding it 12 hours back and 4 hours forward. The start time is recorded in DumpsterRedeliveryStartTime and the end time is recorded in DumpsterRedeliveryEndTime.

5. The Replication service makes an RPC call to the Hub Transport servers listed in DumpsterRedeliveryServers requesting dumpster redelivery for the given loss time window. If the Hub Transport server is not available, CCR will try the same request every 30 seconds until the configured MaxDumpsterTime is reached. For LCR, this request is only sent once.

   Note

   In RTM, the redelivery request was hard-coded to be retried for up to seven days. SP1 made it variable based on the MaxDumpsterTime.

6. The Hub Transport server will acknowledge the first redelivery request with a Retry response.

7. The Hub Transport server will redeliver the messages it has within its transport dumpster for the allotted time window. Once the message is resubmitted for delivery, the message is removed from the transport dumpster.

8. The Replication service makes an RPC call to the Hub Transport servers listed in DumpsterRedeliveryServers requesting dumpster redelivery for the given loss time window.

9. The Hub Transport servers that have successfully redelivered the dumpster messages will acknowledge the redelivery request with a Success response. At this point the Replication service will remove those Hub Transport servers from the DumpsterRedeliveryServers key.

10. This process will continue until either all Hub Transport servers have redelivered the mail, or the MaxDumpsterTime has been reached. Again, in the LCR scenario, the request will only be sent once, so Hub Transport servers that are unavailable at the time Restore-StorageGroupCopy is executed will never redeliver dumpster items for this loss window.

    Note

    If multiple lossy failures take place, the new loss window is added to the previous one.

At this point, there is the question of how much data loss can be handled by the transport dumpster. Let’s look at an example.

Consider an environment that utilizes a 15 MB dumpster size; has eight Hub Transport servers within the Active Directory site; each mailbox generates 20 MB of traffic over 10 hours; and there are 100 mailboxes per storage group.

15MB * 8 = 120MB per storage group

[20MB / 10 hours] * 100 mailboxes/SG = 200MB message traffic in 1 hour

60 minutes * 120MB/200MB = 36 minutes worth of data

In 36 minutes, 120+ logs are generated per storage group on the server.

The other question that may come to mind is whether the transport dumpster comes with time window guarantees. The answer to that is essentially no. If there are no message size limits in your organization (or if you do not follow the recommendations for configuring the MaxDumpsterSizePerStorageGroup for your environment), a single 18 MB message will purge all other messages for given storage group on a given Hub Transport server.

Return to top

Incremental Reseed

Consider the previous scenario: NodeA unexpectedly failed after closing out log generation 4 and starting log generation 5, but before log generation 4 could be replicated to NodeB. NodeB took ownership of the clustered mailbox server, brought the storage group online, and initiated the transport dumpster redelivery process to recover messages that had previously been submitted to the clustered mailbox server during the failure window. In addition, since the storage group is online, it is servicing requests and generating its own generation 4 and generation 5 log files.

Eventually, the administrators repair the problem on NodeA and bring it back online. NodeA will communicate with the cluster and learn that NodeB owns the cluster mailbox server. The Replication service on NodeA will then determine where divergence occurred. The divergence detection logic is relatively simple. The Replication service compares its highest generation log file with the highest generation log file on the source and compares the headers to determine if they are the same log file. If they are the same, then no divergence has occurred. If they are not the same log file, then the Replication service will continue comparing log files until it finds a log file that matches a log file on the source. This is known as the divergence point.

Locating the point of divergence

In our scenario, the Replication service on NodeA will compare generation 5 and find that it does not match NodeB’s generation 5. It will then find the same for generation 4, but it will determine that generation 3 is a match.

Now that the Replication service knows where divergence occurred for this storage group, we need to determine how to correct it. A reseed of the copy on NodeA will correct the divergence. However, this is very expensive for large databases from a network perspective, storage I/O and throughput perspective, and from an SLA perspective. To reduce the need to perform a database reseed, the Exchange team focused on the common case: When a lossy failure occurs, only a few log files per storage group are lost and that the down node will be repaired relatively quickly.

The solution the team came up with in Exchange 2007 for this common case was two-fold.

• The transaction log files were reduced from 5 MB to 1 MB. This size reduction ensures that only a small amount of data is lost per log file.

• LLR was added to ESE.

If the divergence point is greater than or equal to the waypoint (meaning that the diverged log file generation is higher than the log files listed in the “Logs Required” portion of the database header), then the log files can be discarded as the data divergence is only in the log files and not the database. Keep in mind that the logs that are discarded do contain data and that data may be lost in this case. If the divergence point is less than the waypoint (meaning that the diverged log file generation is within the log file generations listed in the “Logs Required” portion of the database header), then there is database divergence and a reseed is required.

If we once again return to our NodeA, NodeB scenario, log generations 1, 2, and 3 have been committed to the database on NodeA. Those logs have also been replicated to NodeB and have been replayed into its database. Then, NodeA failed and log generations 4 and 5 were never replicated to NodeB. NodeB took ownership of the clustered mailbox server, brought the storage group online, and generated its own log generations 4 and 5, in addition to log generation 6. NodeA came back online and its Replication service detected divergence occurred at log generation 4. The Replication service on NodeA consults the database header and determines that log generations 4 and 5 are not needed, and it discards them because they are above the waypoint marker. If we were to examine the database header, we would see the following:

Initiating FILE DUMP mode...

Database: priv1.edb

...

State: Dirty Shutdown

Log Required: 2-3 (0x2-0x3)

Log Committed: 0-5 (0x0-0x5)

...

• Committed is log generation 5 (the last log generation listed in “Log Committed”).

• Checkpoint is log generation 2 (the first log generation listed in “Log Required”).

• Waypoint is log generation 3 (the last log generation listed in “Log Required”).

• Log generations 4 and 5 are beyond the waypoint.

The Replication service then copies the new log files (generations 4 and 5) from NodeB (refer to the Log Shipping and Replay section for more information on the checks that are performed) and brings its database copy up-to-date.

The process where the Replication service detects the divergence point, removes log files that are not necessary for database replay, and copies the needed log files from the active instance is called Incremental Reseed.

Database Reseed Scenarios

Before discussing which situations require database reseeds, note that the following scenarios do not require database reseeds.

• In a scheduled outage scenario, the active log file is closed and all required log files are shipped to the target database copy ensuring that the copy activation is lossless.

• Unscheduled outages where log shipping is healthy and keeping up with the log generation on the active will not diverge the database copies. This can be corrected by incremental reseed as a result of LLR enforcing a database lag so that the updated log stream can just be applied and rolled forward. Ensuring that log shipping is healthy is critical for environments that utilize continuous replication. For more information, see Monitoring Continuous Replication.

• Network outages do not require database reseeds as long as the log streams on the source are not truncated. Log truncation will not happen automatically as continuous replication is in an unhealthy state or otherwise inoperable. For more information, see the "Log Truncation" section.

• Network outages where there are multiple cluster replication host names defined do not require database reseeds, as automatic failover between the defined networks will occur, thus ensuring that the log streams are successfully copied to the passive node.

The following scenarios require reseeds:

• The administrator performed offline maintenance (ESEUTIL defragmentation, ISINTEG, ESEUTIL hard repair) against the production database. Offline maintenance operations like offline defragmentation change the structure of the database and these changes are not logged; therefore, a database reseed is required.

• A log in the log stream is corrupted or accidentally deleted before it was copied to the passive. In this case the only way past that log for the copy is to perform a database reseed.

• A failover occurred that significantly diverged the database copies. For this scenario to occur, the databases have to be brought online either through administrator intervention or by setting the ForcedDatabaseMountAfter value on the clustered mailbox server. Recall that when a lossy failover occurs, the Replication service calculates the loss in terms of log files and compares this to the AutoDatabaseMountDial value. If this loss is greater than the LLR depth, then database divergence has occurred between the nodes and this can only be corrected by performing a database reseed.

• Setup for CCR automatically assigns ownership of the clustered mailbox servers to both the active and the passive node. However, if the passive node has failed or is otherwise unavailable, the passive node can be removed from the clustered mailbox server's list of RedundantMachines to allow for log truncation to occur on the active instance. However, once the passive node is repaired or brought back online and re-added to the RedundantMachines list, the database on the passive node will need to be reseeded as there is a gap in the log sequence. For more information on how to adjust the RedundantMachines list, see How to Configure Clustered Mailbox Server Ownership in a CCR Environment.

• Using the Failover Cluster Management tool, Cluster Administrator, or Cluster.exe to manage storage group resources in a CCR environment could result in database divergence. The Cluster Administrator does not include any logic that the Exchange cmdlets have to prevent and warn administrators concerning database divergence issues during lossy failures. For more information, see the Exchange Team blog entry, Exchange Tools vs. Cluster Tools: What's the Deal?

  Note

  The content of each blog and its URL are subject to change without notice. The content within each blog is provided "AS IS" with no warranties, and confers no rights. Use of included script samples or code is subject to the terms specified in the Microsoft Terms of Use.

• In RTM, if you enabled the page zeroing functionality, this was only performed during an online streaming backup. As a result, page zeroing was not a logged operation and therefore, you would have to perform a database reseed to propagate the changes. This has been addressed in SP1, by allowing page zeroing to be an opt-in online maintenance task. For more information about how to use the functionality in SP1, see Online Maintenance Database Scanning in Exchange 2007 SP1 and SP2.

• When a point-in-time recovery has occurred. In this scenario, an earlier copy of the database and old log files have been restored, taking the active database back to a specific point in time, thereby causing divergence from the passive copy. You would have to reseed the passive copy.

Return to top

Standby Continuous Replication

Standby continuous replication is a disaster recovery feature that employs continuous replication to create and maintain multiple copies of a storage group. It is provided as a feature to facilitate site resiliency. SCR utilizes the same continuous replication technology as LCR and CCR. The following are the few key differences between SCR and LCR or CCR.

• A lossy activation on an SCR target will never invoke the transport dumpster.

• LCR and CCR only support a 1:1 replication model. SCR supports a many:many (1:1, many:1, and 1:many) replication model. (A source storage group can be replicated to multiple SCR targets, though we recommend having no more than four copies.)

  • When planning many:1 replication models, keep in mind that SCR has the same path requirements like CCR or LCR. Each database must be replicated to the same corresponding drive and path. When designing source servers, ensure that each folder path is unique.

• Log copying to the SCR target is continuous, however SCR supports log replay lag and truncation lag.

  • Replay lag is designed to minimize the chance of needing to reseed when the source is also using continuous replication and experiences a lossy failure. By default, replay lag is set to one day.

  • Truncation lag is designed so that the logs on the target can be used to recover from loss on source. By default, truncation lag is disabled.

  • Even though an option for replay lag exists, during recovery/activation, all log files are replayed (unless the log files are removed from the log path manually).

• Initial seeding of the database to the SCR target is not instantaneous. By default, the database will not be seeded until after 50 log files are replicated and the replay lag time has been exceeded. However, this can be overridden by executing Update-StorageGroupCopy.

• There is no Exchange-aware backup mechanism to back up SCR target storage groups. That being said, you can suspend replication and take a flat file backup of the database and log files on the SCR target.

• Log truncation is SCR-aware. For more information on how log truncation occurs, see "Log Truncation" later in this white paper.

Continuous Replication and Backups

One of the benefits of using continuous replication is the ability to offload Volume Shadow Copy Service (VSS)-based backups from the active storage groups to the passive storage groups. Exchange-aware VSS backups are supported for both the active and passive storage groups and databases. The passive copy backup solution we provide is VSS-only, and it is implemented by the Exchange Replica VSS Writer that is part of the Replication service. Streaming backups are only supported from the active storage groups. You cannot use streaming backup APIs to back up the database on the passive side. You also need to use a third-party backup application that supports Exchange VSS, as Windows Server Backup is not Exchange VSS-aware.

When you're making a VSS backup off the passive copy, what happens to the transaction logs?  A common task during Exchange-aware backups is the truncation of transaction log files after the backup has completed successfully. The continuous replication feature guarantees that logs that have not been replicated are not deleted.

The challenges with taking a backup on the passive copy are that backups modify the header of the database. For example, they add information about the time of the last backup of the database. The VSS backup is made possible by the Exchange Replica VSS Writer in the Replication service, and the Replication service has a copy of the data, but it can only get its data and modifications from the store. It can't independently modify its copy of the database; that would produce divergence. Therefore, it can't modify the header of its database copy.

The solution is to have the Replication service coordinate its backups with the store. As soon as you start a backup on the passive copy, the Replication service contacts the store on the active and tells it that a backup is about to start. This is done to prevent the same storage group on both the active and the passive from being backed up simultaneously. Once the backup is finished, the Replication service contacts the store and lets it know that the backup completed.

The database header modifications resulting from the backup are then made by the store on the active. This action generates a log record, which through continuous replication is copied to the passive. When it is replayed, the database header on the passive is then updated.

This is a little more complex than traditional backups, and it has some interesting side effects. For example, if you back up the passive and then immediately after the backup has finished you look at the database on the passive, it will not reflect the backup. The database on the active node will however, reflect it.

So if you are backing up databases in a continuous replication environment, looking at the database on the active is the most accurate way to determine what the last backup is.

Another side effect is that, if the Information Store is not running, you can't backup the passive. Running the Information Store is required so that backups can be coordinated and so the database header can be updated.

Log Truncation

How does log truncation occur, especially when continuous replication is involved? Within Exchange, the Information Store service is responsible for truncating log files on the active database using either the ESE.DLL or the ESEBACK.DLL. Which DLL is used depends on whether continuous replication is enabled and whether circular logging is enabled. If continuous replication is not enabled, then ESE.DLL is responsible for performing log truncation when circular logging is enabled. Otherwise, ESEBACK.DLL is responsible for log truncation.

When continuous replication is enabled, ESE is initialized with the “do not perform circular logging” flag, regardless of the storage group’s circular logging setting. This is done because ESE-based circular logging is unaware of continuous replication. For continuous replication to work, log continuity is required. Therefore, logs cannot be overwritten until they are no longer needed by the replication engine. However, you can combine circular logging with continuous replication. In this configuration, you have a new type of circular logging called continuous replication circular logging (CRCL), which is different from the ESE circular logging. Whereas ESE circular logging is performed and managed by the Microsoft Exchange Information Store service, CRCL is performed and managed by the Replication service.

The Replication service (on the target node if CCR or SCR is deployed) and the Information Store service communicate over RPC regarding which log files can be deleted on the source; ESEBACK.DLL then performs the log truncation. The Replication service is responsible for truncating logs for the database copy.

In an LCR or CCR environment, in order for a log to be truncated, the following criteria must be met.

• The log file has been backed up (ignored if circular logging is enabled).

• The log file generation sequence is below the log file checkpoint for the storage group.

• The passive copy of the storage group is in a state that allows the log file to be truncated (in other words, the passive copy must have replayed the log file).

In addition, if the LCR or CCR environment is a source for an SCR target, the following additional condition must be met.

• All SCR targets have inspected the log file.

In an SCR environment, if the following criteria are met, a log file will be truncated on an SCR target.

• The log file generation sequence is below the log file checkpoint for the storage group.

• The log file is older than ReplayLagTime + TruncationLagTime.

• The log file must have been deleted on the source.

  Note

  There is a known issue where log truncation does not occur on an SCR target if TruncationLagTime is greater than ReplayLagTime (even after the TruncationLagTime has passed). This will affect capacity allocation on the log drives of the SCR target machines. There are only two workarounds here. Either delete the unneeded log files (this is somewhat complex as you have to ensure they are beyond ReplayLagTime and TruncationLagTime), or ensure TruncationLagTime is less than ReplayLagTime (or disable it). To change the values for either parameter you need to disable SCR and then re-enable SCR using the new values. This issue is fixed in Update Rollup 4 for Service Pack 1 and later. To obtain the latest update rollup or service pack, see Microsoft Knowledge Base article 937052, How to obtain the latest service pack or update rollup for Exchange 2007.

Return to top

Comparing Continuous Replication with Other Replication Solutions

Microsoft recommends using continuous replication for mailbox data availability over other replication solutions. Consider the following issues with respect to third-party replication solutions.

• Because third-party replication solutions are not Exchange-aware, they do not know how to validate the data being copied at the application layer to ensure it is usable. In addition, because the solutions are not Exchange-aware, they can require more complex special steps to activate.

• Third-party replication solutions must be synchronous in order to be supported by Exchange. Synchronous solutions require good latency between the active and passive copies.

• Third-party replication solutions may maintain two copies that are not independent from each other. This means that corruption introduced in the many layers executing on the hardware (Exchange, the operating system, storage components, and so on) are replicated to the second copy, resulting in a very high Recovery Point Objective (RPO) because both copies end up with the corruption.

By contrast, continuous replication offers the following solutions.

• Continuous replication is integrated into Exchange Server providing a completely Exchange-aware copy with integrated cmdlets and processes to activate, manage, and monitor the copy. For example, when combining replication with clustering technology, CCR provides integrated server and data redundancy, providing automatic simultaneous server and data failover.

• Uses asynchronous replication to ship closed transaction logs to the second copy, making it very efficient and resilient to latency issues.

• Uses copies that are independent copies, so types of corruption that are replicated by third-party solutions are not replicated by continuous replication. For example, if a -1018 occurs on the active database, the passive copy will not suffer from the same corruption (assuming independent storage solutions).

• The logs for database copies are independently validated before they are replayed into the database copy.

• Activation of a database copy is relatively quick. For example, in a CCR environment, failover to the passive node with database copies is integrated into Exchange, and takes roughly two minutes.

• For the LCR and CCR scenarios, when a lossy failure occurs, the only data that can be lost is data that does not go through a Hub Transport server. This small set of data only includes contacts, tasks, personal appointments, notes, and drafts. Anything that goes through Hub Transport is held in the transport dumpster for a configurable amount of time and resubmitted to the Mailbox server.

Conclusion

Continuous replication was introduced to provide data availability for Exchange 2007 by maintaining a copy of the data through asynchronous replication, or log shipping. The replication is based on ESE logging in which closed log files are replicated, verified, and then replayed into the copy of the database.

Since continuous replication is an asynchronous log replication model, ESE was enhanced in Exchange 2007 to reduce the likelihood of requiring database reseeds in the event of an unscheduled outage. Lost log resilience delays committing changes to the database until after one or more log files are replicated, thereby reducing the need to perform database reseeds if the two copies are diverged.

In addition, continuous replication also employs a transport dumpster for redelivery of messages that were recently submitted to the Mailbox server before the failure event. This allows the Mailbox server to recover e-mail data that may have been recorded but not replicated to the passive copy.

Additional Information

• Extensible Storage Engine Architecture

• Lost Log Resilience and Transaction Log Activity in Exchange 2007

• Understanding Transaction Logging

• Cluster Continuous Replication

• Local Continuous Replication

• Standby Continuous Replication

• Exchange Server 2007 SP1 ESE Changes Part 1

• Exchange Server 2007 SP1 ESE Changes Part 2