MongoDB Integration with Edilitics

MongoDB is a NoSQL document database that offers flexibility and scalability. Unlike relational databases, it stores data in JSON-like documents, allowing for diverse data structures within a single collection. This makes MongoDB a popular choice for modern applications that require agile data modeling and rapid development cycles. To learn more about MongoDB, visit the official MongoDB website.

Within the Edilitics platform, MongoDB can be used both as a source and destination for data, making it a versatile tool for data analysis workflows. This enables you to leverage MongoDB's strengths for data storage and retrieval within your Edilitics environment.

This step-by-step guide equips you with the knowledge to configure your MongoDB database instance and integrate it with Edilitics for data analysis.

Before You Begin#

Make sure you have the following:

• Replication is set up for MongoDB OpLog.

• Edilitics has the necessary permissions to read the generic MongoDB databases.

• Edilitics’ IP addresses are enabled on your MongoDB server:

  • 34.93.42.224

  • 34.131.133.218

Setting Up MongoDB for Edilitics Integration#

Step 1: Enable Replication#

Note: Edilitics supports encrypted data ingestion from MongoDB via OpLog.

To set up replication for OpLog, follow these steps:

• Modify the MongoDB Server Configuration#

  • The MongoDB configuration file (mongod.conf) is generally located in the /etc/ directory on a Linux system. Configure the following options:
  
  

  net:

  bindIp: 34.131.133.218, 34.93.42.224 # Edilitics IP addresses

  replication:

  replSetName: "repSet0"

  oplogSizeMB: 2048

  
  

• Configure Replication through MongoDB Shell#

  • Open your Mongo shell on the replication server and execute the following commands:
  
  

  rs.initiate() # Initializes the replica set

  rs.conf() # Displays the current replication configuration

  rs.status() # Shows the status of the replication

  
  

Step 2: Read Permissions for Generic MongoDB Databases#

After setting up replication for OpLog and Change Streams, assign the necessary read privileges to the database user to access the local database and other databases to be replicated. Follow these steps:

• Open the MongoDB shell.

• Connect to your replica set or sharded cluster as an admin user.

• Run the appropriate commands based on your MongoDB version:

  • For MongoDB versions between 2.4 and 2.6:

    
    

    use admin

    db.addUser({

    user: "<username>",

    pwd: "<password>",

    roles: ["readAnyDatabase"]

    })

    
    

  • For MongoDB versions between 3.0 and 3.2:

    
    

    use admin

    db.createUser({

    user: "<username>",

    pwd: "<password>",

    roles: ["readAnyDatabase"]

    })

    
    

  • For MongoDB version 3.2 and later:

    
    

    use admin

    db.createUser({

    user: "<username>",

    pwd: "<password>",

    roles: ["readAnyDatabase", { role: "read", db: "local" }]

    })

    
    

Note: Replace <username> and <password> with your chosen username and password.

Connecting MongoDB to Edilitics#

Step 3: Add the MongoDB Connector in Edilitics#

• In Edilitics, go to the Integrations module and find the New Integration option.

• Choose the MongoDB connector from the list or use the search bar to locate it.

Step 4: Configure the MongoDB Connection#

A screen will appear with connection details. Fill in the following information accurately:

Alternatively, connect using a connection string:

The mongodb:// Connection String#

• <username>: The username for authenticating with the MongoDB server.

• <password>: The password for the specified username.

• <host>: The hostname or IP address of the MongoDB server.

• <port>: The port number on which the MongoDB server is running (default is 27017).

• <database_name>: The name of the database you want to connect to.

• ?options: Additional connection options (optional).

The mongodb+srv:// Connection String#

This format is commonly used for connecting to MongoDB Atlas or any deployment supporting DNS Seedlist.

Include options like:

• retryWrites: Enables retryable writes.

• w: Specifies the write concern.

• authSource: Specifies the database to authenticate against.

Step 5: Test the Connection#

Once you've entered your credentials, click the Test Connection button. Edilitics will attempt to connect to your MongoDB database. You'll see a confirmation message if the connection is successful. If there's a problem, Edilitics will provide details to help you troubleshoot.

Step 6: Verify & Finalize Integration#

Upon successful database connection, you can:

• Preview (Optional): Confirm accuracy by retrieving a sample of tables and data.

• Update (Optional): Modify connection details if the preview deviates from expectations.

• Save: Finalize by encrypting and storing connection details within Edilitics.