mongomirror¶
mongomirror
is intended to be used as a utility for performing
a one-time migration of a dataset into a MongoDB Atlas cluster from a MongoDB
deployment hosted outside of MongoDB Atlas. It can also be used for performing
a one-time migration of a dataset from one Atlas cluster into another Atlas
cluster. Any other use of this utility is not supported.
mongomirror
does not import user/role data or
the config
database.
Download mongomirror
¶
Debian 7 reached
End of Life on
May 31, 2018. mongomirror
0.3.1 removed support for
Debian 7 systems.
Operating System | Download |
---|---|
Amazon Linux 1 | mongomirror-linux-x86_64-enterprise-amzn64-0.11.0.tgz |
Amazon Linux 2 | mongomirror-linux-x86_64-enterprise-amazon2-0.11.0.tgz |
Debian 8 | mongomirror-linux-x86_64-debian81-0.11.0.tgz |
Debian 9 | mongomirror-linux-x86_64-debian92-0.11.0.tgz |
macOS | mongomirror-osx-x86_64-0.11.0.tgz |
RHEL 6.2 | mongomirror-linux-x86_64-rhel62-0.11.0.tgz |
RHEL 7.0 | mongomirror-linux-x86_64-rhel70-0.11.0.tgz |
Suse 12 | mongomirror-linux-x86_64-suse12-0.11.0.tgz |
Ubuntu 14.04 | mongomirror-linux-x86_64-ubuntu1404-0.11.0.tgz |
Ubuntu 16.04 | mongomirror-linux-x86_64-ubuntu1604-0.11.0.tgz |
Ubuntu 18.04 | mongomirror-linux-x86_64-ubuntu1804-0.11.0.tgz |
Windows | mongomirror-win32-x86_64-0.11.0.zip |
Syntax¶
To run mongomirror
, you must specify the source replica
set and the target Atlas replica set. For Atlas, you must
specify a user in the Atlas cluster with
appropriate privileges and the
corresponding password. If the source replica set requires
authentication, you must specify a user with
appropriate privileges.
mongomirror --host <sourceReplSet> \ --destination <atlasCluster> \ --destinationUsername <atlasAdminUser> \ --destinationPassword <atlasPassword> \ [Additional options]
For details on the options, see Options.
Migration Process¶
When you start mongomirror
:
- First,
mongomirror
performs an initial sync, copying collections from the existing MongoDB replica set to the target cluster in Atlas. - After the initial sync,
mongomirror
continuously tails the replica set's oplog for incoming changes and replays them on the target cluster in MongoDB Atlas.
As of version 0.9.0, mongomirror
uses wire compression if it is
enabled with either the source or the target. Use the --compressors
option to specify which compression libraries to allow.
Once started, mongomirror
runs continuously until you
shut down the process.
- If you shut down
mongomirror
during the initial sync stage, either ensure that the target cluster is empty before you restartmongomirror
or runmongomirror
with the--drop
option. - If you shut down
mongomirror
during the oplog tailing stage, you can restartmongomirror
to continue from the last oplog record processed. See the--bookmarkFile
option.
Progress Measurement¶
mongomirror
logs its progress to the standard output in
the terminal:
During the initial sync, mongomirror
logs a progress
bar for each collection it copies. For example:
2016-12-16T17:54:53.638-0800 [#....................] park.events 2179/34184 (6.4%) 2016-12-16T17:54:53.638-0800 [#############........] zoo.animals 29000/49778 (58.3%)
When tailing the oplog, mongomirror
logs the lag time,
in seconds, between the most recent oplog entry on the source and the
last processed oplog entry on the target. For example:
2016-12-12T16:22:17.027-0800 Current lag from source: 6s
A lag time of 6 seconds means that the last oplog entry
mongomirror
processed was 6 seconds behind the most
recent one available on the source.
The amount of time it takes mongomirror
to catch up
completely may be greater or lesser than 6 seconds, depending on
how many entries arrive per second.
A lag time of 0 seconds indicates that mongomirror
is
processing entries that arrived less than one second before the latest
oplog entry.
If there are indexes on the source cluster, mongomirror
builds the
same indexes on the target cluster. The progress log shows the start
time and end time of the index building process, but does not display
the iterative progress of the process. To check on the progress
of the index builds, you can either:
Use the currentOp command on the primary node of the target cluster. The
command
field shows information about the currently running operation. Index building entries look similar to the following:"command" : { "createIndexes" : "perfs", "indexes" : [ { "key" : { "workTitle" : "text" }, "name" : "workTitle_text" } ]... Download the mongodb logs for your target cluster and check recent entries for index-related lines. Log messages related to index building look similar to the following:
2018-11-29T16:58:25.018-0800 I INDEX [conn1] build index on: example.airlines properties: { v: 2, key: { airline: 1.0 }, name: "airline_1", ns: "example.airlines" }
Performance¶
To avoid contention for network and CPU resources, do not run
mongomirror
on the same hosts that provide your replica
set’s mongod instances.
mongomirror
must have network access to the source replica set.mongomirror
must have network access to the target cluster.mongomirror
has approximately the same performance impact on your source replica set as a secondary:- For the initial sync stage, the load scales with the size of your data set.
- Once an initial sync completes, the load scales with oplog gigabytes used per hour.
Considerations¶
Source MongoDB Deployment¶
- The source MongoDB deployment must be a replica set. If the source is
a standalone MongoDB deployment, convert to a replica set first to
use
mongomirror
. See Convert a Standalone to a Replica Set. - The source replica set must run MongoDB version 2.6 or higher.
- The source MongoDB replica set cannot be an
M0
(Free Tier) orM2/M5
shared cluster. - You should not change the
featureCompatibilityVersion
flag whilemongomirror
is running. You should not make any namespace changes during the migration process, such as using the
renameCollection
command or executing an aggregation pipeline that includes the$out
aggregation stage.Contact MongoDB Support if you believe renaming or dropping a collection needs to occur during the migration process. From the Atlas UI, click Support to open a support ticket.
Target Atlas Cluster¶
- The target Atlas cluster must be a replica set, but cannot be an
M0
(Free Tier) orM2/M5
shared cluster. - The target Atlas cluster MongoDB version must be the same as or later than the source cluster MongoDB version.
- For the initial sync stage of
mongomirror
, the target Atlas cluster should be empty or you should runmongomirror
with the--drop
option. The target Atlas cluster must include in its IP access list either
- The public IP address of the server on which
mongomirror
is running, or - If set up for VPC peering, either the peer’s VPC CIDR block (or a subset) or the peer VPC’s Security Group.
- The public IP address of the server on which
mongomirror
automatically connects to MongoDB Atlas over TLS /SSL .- You should not change the
featureCompatibilityVersion
flag whilemongomirror
is running. mongomirror
is not compatible with TTL indexes. Drop any existing TTL indexes and rebuild them when the migration process is complete.
Required Access on Source Replica Set¶
If the source replica set requires authentication, you must include
user credentials when running mongomirror
. You must
specify a database user with, at a minimum, the following privileges:
- Read all databases and collections. The
readAnyDatabase
role on theadmin
database covers this requirement. - Read the oplog.
- Run the
getParameter
command.
If no such user exists, create the user in your source MongoDB
replica set. Different MongoDB server versions have different
built-in roles. Select a built-in role based on your MongoDB
server version and execute the appropriate commands in the
mongo
shell:
For source clusters running MongoDB version 3.4 or later a user must have, at a minimum, both
clusterMonitor
andreadAnyDatabase
roles. For example:use admin db.createUser( { user: "mySourceUser", pwd: "mySourceP@$$word", roles: [ "clusterMonitor", "readAnyDatabase" ] } ) For source clusters running MongoDB version 3.2 a user must have, at a minimum, both
clusterManager
andreadAnyDatabase
roles, as well as read access on thelocal
database. This requires a custom role, which you can create with the following commands:use admin db.createRole( { role: "migrate", privileges: [ { resource: { db: "local", collection: "" }, actions: [ "find" ] } ], roles: ["readAnyDatabase", "clusterManager"] } ) db.createUser( { user: "mySourceUser", pwd: "mySourceP@$$word", roles: [ "migrate" ] } ) For source clusters running MongoDB version 2.6 or 3.0 a user must have, at a minimum, the
readAnyDatabase
role. For example:use admin db.createUser( { user: "mySourceUser", pwd: "mySourceP@$$word", roles: [ "readAnyDatabase" ] } )
Required Access on Destination Cluster¶
You must specify an database user with the
Atlas admin
role to run mongomirror
.
See Add Database Users for documentation on creating a
database user.
Options¶
--host <host>
¶The host information for the source replica set. Specify the replica set name and a seed list of the members, as in the following:
<RSname>/<host1>:<port1>,<host2>:<port2>,<host3>:<port3>
--username <username>
¶If the source replica set requires authentication, the name of a user in the source replica set with privileges to read any database, including the
local
database. A user with thebackup
role provides the appropriate privileges. For details on the specific privileges required, see Required Access on Source Replica Set.
--password <password>
¶Password for the user specified in
--username
.
--authenticationDatabase <authenticationDatabase>
¶The database in the source replica set where the user specified in
--username
was created.
--authenticationMechanism <authenticationMechanism>
¶The authentication mechanism to use to authenticate the user to the source replica set.
Value Description SCRAM-SHA-1 RFC 5802 standard Salted Challenge Response Authentication Mechanism using the SHA1 hash function. MONGODB-CR MongoDB challenge/response authentication. MONGODB-X509 MongoDB TLS/SSL certificate authentication. GSSAPI (Kerberos) External authentication using Kerberos. This mechanism is available only in MongoDB Enterprise. PLAIN (LDAP SASL) External authentication using LDAP. You can also use PLAIN
for authenticating in-database users.PLAIN
transmits passwords in plain text. This mechanism is available only in MongoDB Enterprise.MONGODB-IAM New in version 0.10.0
External authentication with AWS IAM.
To authenticate with AWS IAM credentials, use the following options:
--username
<aws access key id>--password
<secret access key id>--awsSessionToken
<aws session token>
--awsSessionToken
¶New in version 0.10.0
An AWS session token for use with the
MONGODB-IAM
authentication mechanism.
--compressors <snappy,...>
¶New in version 0.9.0
Comma-separated list of compressors to enable. Use 'none' to disable. Default:
snappy,zstd,zlib
--destination <destination>
¶The host information for the target Atlas replica set.
Specify the replica set name and a seed list of the members, as in the following:
<RSname>/<host1>:<port1>,<host2>:<port2>,<host3>:<port3>
--destinationUsername <Atlas user name>
¶Name of a database user in the Atlas cluster with privileges to read, write, and admin any database. A user with the Atlas admin role provides the appropriate privileges. For details on the specific privileges required, see Required Access on Destination Cluster.
--destinationPassword <password>
¶Password of the database user specified in
--destinationUsername
.
--drop
¶Flag that indicates that
mongomirror
should drop all but thesystem.*
collections in the target cluster.
--noIndexRestore
¶New in version 0.10.0
Omit indexes when migrating data.
--includeNamespace <database.collection>
¶Specify a namespace on the source cluster to mirror to the target cluster. May be provided multiple times.
--includeDB <database>
¶Specify a database on the source cluster to mirror to the target cluster. May be provided multiple times.
--ssl
¶Enables TLS/SSL encrypted connections to the source replica set.
--sslPEMKeyFile <file>
¶The .pem file if the source replica set requires clients to present a certificate. The .pem file contains both the TLS/SSL certificate and key. Specify the file using relative or absolute paths.
--sslPEMKeyPassword <value>
¶Password to decrypt the certificate-key file specified in
--sslPEMKeyFile
. Use if the--sslPEMKeyFile
is encrypted.
--sslCAFile <file>
¶The .pem file that contains the root certificate chain from the Certificate Authority(CA) for the source replica set. Specify the file using relative or absolute paths.
--sslCRLFile <filename>
¶The .pem file that contains the Certificate Revocation List for the source replica set. Specify the file using relative or absolute paths.
--sslAllowInvalidHostnames
¶Disables the validation of the hostnames in TLS/SSL certificates presented by the source replica set. Allows
mongomirror
to connect to the source replica set if the hostname in the certificates does not match the specified hostname.
--sslAllowInvalidCertificates
¶Bypasses the validation checks for certificates presented by the source replica set and allows the use of invalid certificates. When using the
--allowInvalidCertificates
setting, MongoDB logs as a warning the use of the invalid certificate.
--gssapiServiceName <name>
¶If the source replica set uses Kerberos authentication, the name of the service using GSSAPI/Kerberos. Only required if the service does not use the default name of
mongodb
.This option is available only in MongoDB Enterprise.
--gssapiHostName <host>
¶If the source replica set uses Kerberos authentication, the hostname of a service using GSSAPI/Kerberos. Only required if the hostname of a machine does not match the hostname resolved by DNS.
This option is available only in MongoDB Enterprise.
--readPreference <read preference>
¶Deprecated since version 0.9.0
mongomirror
always reads from the primary unless the source is a single host without a replica set name, in which case it makes a direct connection only to that host.
--writeConcern <write concern>
¶Deprecated since version 0.2.3.
mongomirror
always uses majority write concern.
--numParallelCollections <num>, -j <num>
¶Default: 4
The number of collections to copy and restore in parallel.
--bypassDocumentValidation
¶Deprecated since version 0.2.3.
mongomirror
always bypasses document validation.
--bookmarkFile <file>
¶Default: mongomirror.timestamp
Name of the oplog timestamp bookmark file.
--forceDump
¶Flag that indicates that
mongomirror
resync all source collections, even if a nonempty bookmark file exists.
--oplogPath <path>
¶New in version 0.5.0
Enables
mongomirror
to buffer the initial sync oplog window to disk. When you specify a value for this option,mongomirror
streams the source oplog entries to the specified directory in a single file:<oplogPath>/oplog-mongomirror.bson.sz
. After the entire oplog file is replayed to the destination cluster,mongomirror
removes the file and starts tailing the source oplog without buffering.By default,
mongomirror
streams oplog entries from the source and applies them to the destination cluster. However, the migration may fail if the source oplog is not large enough to contain the entire initial sync oplog window. To avoid this error, you can either increase the size of the source oplog, or specify this option to ensure that the source oplog will not run out of space during the migration process.ImportantThere must be enough disk space to accommodate all of the source oplog entries that occur during the initial
mongomirror
sync.ExampleIf the source oplog is 10 GB and covers 24 hours of changes, and
mongomirror
's sync is estimated to take 48 hours, there must be at least 20 GB of free disk space in the specified directory.
--oplogBatchSize <num>
¶New in version 0.8.0
Specify the number of oplog entries to send as a batch. Defaults to 10,000.
--httpStatusPort <num>
¶Directs
mongomirror
to start an HTTP server on the specified port. You can retrieve the current status ofmongomirror
by issuing an HTTPGET
request tohttp://localhost:<num>
.When running with
--httpStatusPort
,mongomirror
does not exit when it encounters an error. Instead, it logs the error as normal and reports the error over HTTP to the specified port.mongomirror
returns a document in response to the HTTP request. The following example syntax represents all the possible output fields - the actual response may only return a subset of these fields. See the subsequent table for a description of the fields and when to expect them.{ "stage" : "<stage Name>", "phase" : "<phase Name>", "details" : { "currentTimestamp" : "<BSON timestamp>", "latestTimestamp" : "<BSON timestamp>", "lastWriteOnSourceTimestamp" : "<BSON timestamp>", "<namespace>" : { "complete" : <boolean>, "copiedBytes" : <integer>, "totalBytes" : <integer>, "createIndexes" : <integer> }, ... }, "errorMessage" : "<error message>" } The following table describes each field and its possible values:
Field Description stage
The name of the stage in progress. Possible values are:
initializing
mongomirror
has started but is not yet copying any data.initial sync
mongomirror
is copying documents and indexes that already exist on the source deployment.mongomirror
also tails and applies entries from the oplog.oplog sync
mongomirror
is tailing and applying entries from the oplog.
phase
The name of the phase. Provides more specific details about what part of the stage
is in progress.details
A document providing a detailed description of the progress of the current phase.
During the
initial sync
stage, each subdocument indetails
represents a single collection being copied bymongomirror
.Depending on the
stage
orphase
,mongomirror
may not include this field in the response.details.<namespace>
The full namespace of the collection being copied, displayed as
<database>.<collection>
.Only displays during the
initial sync
phase when copying documents or indexes.details.<namespace>.complete
Displays
true
orfalse
depending on whether or notmongomirror
has copied all documents or indexes from the collection to the target Atlas cluster.Only displays during the
initial sync
phase when copying documents or indexes.details.<namespace>.copiedBytes
The number of bytes copied so far. Note that this is a different measurement from the
mongomirror
logs, which report the current/total number of documents copied.Only displays during the
initial sync
phase when copying non-index data.details.<namespace>.totalBytes
The total size (in bytes) of the collection.
Only displays during the
initial sync
phase when copying non-index data.details.<namespace>.createIndexes
The number of indexes that have been or will be created.
Only displays during the
initial sync
stage when copying indexes.details.currentTimestamp
The BSON timestamp value of the oplog entry most recently processed.
mongomirror
only refreshes this data point every 10 seconds, somongomirror
may be slightly further ahead of the reported time.Only displays during the
initial sync
oroplog sync
stages when tailing or applying oplog entries.details.latestTimestamp
During the
initial sync
stage, this represents the BSON timestamp value of the latest oplog entry available after the initial data was copied during initial sync.During the
oplog sync
stage, this represents the BSON timestamp value of the latest oplog entry available on the source deployment.Only displays during the
initial sync
oroplog sync
stages when tailing or applying oplog entries.details
.lastWriteOnSourceTimestamp
The BSON timestamp value of the most recent oplog entry that is not a no-op. No-op entries are generally system-level operations such as heartbearts that do not write or edit data in the database.
mongomirror
refreshes this value every 10 seconds. Operations which write or edit data in the database may not be reported until the next refresh occurs.The
lastWriteOnSourceTimestamp
field is useful as a confirmation that no new writes are occurring on the source deployment before cutting over during a migration.errorMessage
A string that describes any error encountered by mongomirror
.
Examples¶
Migrate a Replica Set into Atlas¶
The following example migrates from a source replica set that does not require authentication:
mongomirror --host sourceRS/source-host1:27017,source-host2:27017 \ --destination myAtlasRS/atlas-host1:27017,atlas-host2:27017 \ --destinationUsername myAtlasUser \ --destinationPassword myAtlasPwd
To migrate from a source replica set that does not require
authentication, run mongomirror
with the following
options:
--host
<sourceReplSet/seed list of members>--destination
<Atlas Cluster>--destinationUsername
<atlasUser>--destinationPassword
<atlasPassword>
For the destination, specify the replica set name followed by a seed list of members in the following format:
<replicaSetName>/<host1>:<port1>,<host2>:<port2>,<host3>:<port3>,...
The specified Atlas user must have the required privileges for
Atlas. The Atlas admin
role provides the required privileges.
Source Replica Set Uses SCRAM-SHA1 Authentication¶
The following example migrates from a source replica set that uses SCRAM-SHA1 authentication:
mongomirror --host sourceRS/source-host1:27017,source-host2:27017,source-host3:27017 \ --username mySourceUser \ --password mySourcePassword \ --authenticationDatabase admin \ --destination myAtlasRS/atlas-host1:27017,atlas-host2:27017 \ --destinationUsername myAtlasUser \ --destinationPassword atlasPassw0Rd
To migrate from a source replica set that does uses SCRAM-SHA1
authentication, run mongomirror
with the following
options:
--host
<sourceReplSet/seed list of members>--username
<sourceUser>--password
<sourcePassword>--authenticationDatabase
<sourceDatabase>--destination
<Atlas Cluster>--destinationUsername
<atlasUser>--destinationPassword
<atlasPassword>
The source replica set user must have the required access on source
cluster. The backup
role provides the appropriate privileges.
For the destination, specify the replica set name followed by a seed list of members in the following format:
<replicaSetName>/<replicaMember>,<replicaMember>,<replicaMember>,...
The specified Atlas user must have the required access on Atlas. The
Atlas admin
role provides the required privileges.
Save mongomirror
Output to a File¶
It is often helpful to save the output logs from a mongomirror
procedure to a file for later examination and debugging. Use the following
format to save output to a file called mongomirror.log
:
mongomirror <args> 2>&1 | tee -a mongomirror.log
Source Replica Set Requires X509 Client Authentication¶
The following example migrates from a source replica set that uses X.509 authentication:
mongomirror --host sourceRS/source-host1:27017,source-host2:27017,source-host3:27017 \ --username "CN=myName,OU=myOrgUnit,O=myOrg,L=myLocality,ST=myState,C=myCountry" \ --authenticationDatabase '$external' \ --authenticationMechanism MONGODB-X509 \ --ssl \ --sslPEMKeyFile <path-to-my-client-certificate.pem> \ --sslCAFile <path-to-my-certificate-authority-certificate.pem> \ --destination myAtlasRS/atlas-host1:27017,atlas-host2:27017 \ --destinationUsername myAtlasUser \ --destinationPassword atlasPassw0Rd
To migrate from a source replica set that uses X.509 authentication,
run mongomirror
with the following options:
--host
<sourceReplSet/seed list of members>--username
<subject from the client certificate>--authenticationMechanism
MONGODB-X509
--authenticationDatabase
'$external'
--ssl
--sslPEMKeyFile
<path-to-my-client-certificate.pem>--sslCAFile
<path to root CA PEM file>--destination
<Atlas Cluster>--destinationUsername
<atlasUser>--destinationPassword
<atlasPassword>
The source replica set user must have the required access on source
cluster. The backup
role provides the appropriate privileges.
For the destination, specify the replica set name followed by a seed list of members in the following format:
<replicaSetName>/<replicaMember>,<replicaMember>,<replicaMember>,...
The specified Atlas user must have the required access on Atlas. The
Atlas admin
role provides the required privileges.
Source Replica Set Requires Kerberos/GSSAPI Authentication¶
The following example migrates from a source replica set that uses Kerberos authentication:
mongomirror --host sourceRS/source-host1:27017,source-host2:27017,source-host3:27017 \ --username sourceUser/administrator@MYREALM.COM \ --authenticationDatabase '$external' \ --authenticationMechanism GSSAPI \ --destination myAtlasRS/atlas-host1:27017,atlas-host2:27017,atlas-host3:27017 \ --destinationUsername atlasUser \ --destinationPassword atlasPass
To migrate from a source replica set that uses X.509 authentication,
run mongomirror
with the following options:
--host
<sourceReplSet/seed list of members>--username
<Kerberos user principal>--authenticationDatabase
'$external'
--authenticationMechanism
GSSAPI
--destination
<Atlas Cluster>--destinationUsername
<atlasUser>--destinationPassword
<atlasPassword>
The source replica set user must have the required access on source
cluster. The backup
role provides the appropriate privileges.
For the destination, specify the replica set name followed by a seed list of members in the following format:
<replicaSetName>/<replicaMember>,<replicaMember>,<replicaMember>,...
The specified Atlas user must have the required access on Atlas. The
Atlas admin
role provides the required privileges.