Atlas Data Lake Dataset or Online Archive

stores

1
"stores" : [
2
  {
3
    "name" : "<string>",
4
    "provider": "<string>",
5
    "region": "<string>"
6
  }
7
]

stores

Array of objects where each object represents a data store to associate with the federated database instance. The federated database instance store references files in an S3 bucket, documents in an Atlas cluster, files stored at publicly accessible URLs, Atlas Data Lake datasets, or Atlas online archives. Atlas Data Federation can only access data stores defined in the stores object.

stores.[n].name

Name of the federated database instance store. The databases.[n].collections.[n].dataSources.[n].storeName field references this value as part of mapping configuration.

stores.[n].provider

Cloud provider where the snapshot data is stored. Value must be dls:<subtype> for a snapshot. Atlas Data Federation supports the following subtypes:

• aws, for which the value must be dls:aws

stores.[n].region

Region name of your Data Lake or online archive. Each store is associated with a single region, where both the metadata and snapshot data are stored. If you have multiple datasets or online archives in different regions, you must add a store for each region to map data in that region to virtual databases and collection in federated database instance.

To learn more about the supported regions for AWS, see Atlas Data Federation Regions.

databases

1
"databases" : [
2
  {
3
    "name" : "<string>",
4
    "collections" : [
5
      {
6
        "name" : "<string>",
7
        "dataSources" : [
8
          {
9
            "storeName" : "<string>",
10
            "datasetName" : "<string>",
11
            "datasetPrefix": "<string>",
12
            "trimLevel": <int>,
13
            "provenanceFieldName": "<string>",
14
            "maxDatasets": <int>
15
          }
16
        ]
17
      }
18
    ],
19
    "views" : [ 
20
      {
21
        "name" : "<string>", 
22
        "source" : "<string>", 
23
        "pipeline" : "<string>" 
24
      }
25
    ] 
26
  }
27
]

databases

Array of objects that define the mapping between each federated database instance store defined in stores and Atlas Data Lake datasets or online archives. Each object represents a database, its collections, and, optionally, any views on the collections. Each database can have multiple collections and views objects.

databases.[n].name

Name of the database to which Atlas Data Federation maps the data contained in the data store.

databases.[n].collections

Array of objects where each object represents a collection and data sources that map to a stores federated database instance store.

databases.[n].collections.[n].name

Name of the collection to which Atlas Data Federation maps the data contained in each databases.[n].collections.[n].dataSources.[n].storeName. Each object in the array represents the mapping between the collection and an object in the stores array.

You can generate collection names dynamically by specifying * for the collection name. To dynamically generate collection names, you must also specify the following:

• databases.[n].collections.[n].dataSources.[n].datasetPrefix

• databases.[n].collections.[n].dataSources.[n].trimLevel

For wildcard collections, Atlas Data Federation maps a dataset name to a collection name first by splitting the namespace into a list of fields on the $ delimiter, then by trimming a number of fields from the left of the list, and finally by combining the remaining fields using _.

databases.[n].collections.[n].dataSources

Array of objects where each object represents a federated database instance store in the stores array to map with the collection. You can specify multiple dataSources for a wildcard collection only if all the dataSources for the collection map to Atlas Data Lake dataset stores.

databases.[n].collections.[n].dataSources.[n].storeName

Name of a federated database instance store to map to the <collection>. Must match the name of an object in the stores array.

databases.[n].collections.[n].dataSources.[n].datasetName

Name of the Atlas Data Lake dataset or online archive dataset to map with the collection. The Atlas Data Lake datasetName is in the following format:

<version>$<type>$<subtype>$<clusterName>$<dbName>$<collectionName>$<snapshotId>

Example

Consider the following Atlas Data Lake dataset name.

v1$atlas$snapshot$clusterName$dbName$collectionName$snapshotId

Here:

• v1 is the version

• atlas is the type of data source

• snapshot is the subtype

• clusterName is the name of the Atlas cluster

• dbName is the name of the database on the Atlas cluster

• collectionName is the collection in the database

• snapshotId is the unique identifier of the snapshot of the data in the collection

Note

For a non-wildcard collection, you can't specify this option if you specify the databases.[n].collections.[n].dataSources.[n].datasetPrefix option because Atlas Data Federation automatically generates collections for the latest dataset using the name specified through databases.[n].collections.[n].dataSources.[n].datasetPrefix option.

databases.[n].collections.[n].dataSources.[n].datasetPrefix

Required for wildcard collections.

Dataset name prefix to match against the Atlas Data Lake dataset name or online archive dataset name. When you set this for wildcard collections, Atlas Data Federation maps collections to only the Atlas Data Lake dataset names or online archive dataset names whose prefix match the value specified here.

Example

{
  ...,
  "name": "*",
  "dataSources": [
    {
      "storeName": "aws-dl-store",
      "datasetPrefix":
      "v1$atlas$snapshot$MyCluster$MyDB$MyCollection",
      "trimLevel": 3
    }
  ]
}

If you specify this setting for a non-wildcard collection, Atlas Data Federation maps only the latest dataset (for the most recently captured snapshot) to the collection. You can't specify the databases.[n].collections.[n].dataSources.[n].datasetName also if you specify this option for a non-wildcard collection.

Example

{
  ...,
  "name": "myFederatedDbCollection",
  "dataSources": [
    {
      "storeName": "aws-dl-store",
      "datasetPrefix": "v1$atlas$snapshot$MyCluster$MyDB$MyCollection"
    }
  ]
}

databases.[n].collections.[n].dataSources.[n].trimLevel

Unsigned integer that specifies how many fields of the dataset name to trim from the left of the dataset name before mapping the remaining fields to a wildcard collection name. Value must be greater than 0 and less than 7. You can set this setting for wildcard collections only.

Example

Consider the following Atlas Data Lake dataset name:

v1$atlas$snapshot$MyCluster$MyDB$MyCollection$snapshotDate

Atlas Data Federation dynamically generates the following collection names for different trim levels:

Trim Level	Collection Name
5	MyCollection_snapshotDate
4	MyDB_MyCollection_snapshotDate
3	MyCluster_MyDB_MyCollection_snapshotDate
2	snapshot_MyCluster_MyDB_MyCollection_snapshotDate
1	atlas_snapshot_atlas_MyCluster_MyDB_MyCollection_snapshotDate
0	v1_atlas_snapshot_atlas_MyCluster_MyDB_MyCollection_snapshotDate

You can't configure this setting using the Visual Editor in the Atlas UI. Therefore, this setting defaults to trim level 5 for configurations using the Visual Editor.

databases.[n].collections.[n].dataSources.[n].provenanceFieldName

Name for the field that includes the provenance of the documents in the results. If you specify this setting in the storage configuration, Atlas Data Federation returns the following fields for each document in the result:

Field Name	Description
provider	Provider (stores.[n].provider) in the federated database instance storage configuration
clusterName	Name of the Atlas cluster
databaseName	Name of the database on the Atlas cluster
collectionName	Name of the collection
snapshotID	Unique 24-hexadecimal character string that identifies the snapshot
dataSetName	Name of the Data Lake dataset or online archive dataset (databases.[n].collections.[n].dataSources.[n].datasetName)

You can't configure this setting using the Visual Editor in the Atlas UI.

databases.[n].collections.[n].dataSources.[n].maxDatasets

Unsigned integer that specifies the maximum number of datasets from which to dynamically generate collections for the data source. You must provide a value greater than 0. You can set this setting for wildcard collections only. Atlas Data Federation returns datasets in reverse lexicographical order.

Note

You can't configure this setting using the Visual Editor in the Atlas UI. Therefore, Atlas Data Federation configuration doesn't include a limit on the number of datasets for configurations using the Visual Editor.