Environments

The Environments section allows you to configure various environments for your project. You can add new environments, edit existing ones, or delete environments as needed.

Environment Types

You can configure the following types of environments in DATAMIMIC:

• Database:
• Kafka
• Mongo
• Data Warehouse

Tip

If a required environment type is missing, please get in touch with our team to enable this environment type for your account or instance.

Managing Environments

Once you have added environments, you can manage them from the Environments pane.

Manage Environments

Available Actions

• Add: Add a new environment.
• Scan Metadata: For database environments (except SQLite), scan the connection to create or refresh database metadata.
• Reset Metadata: Remove stored metadata for the selected database environment.
• Edit: Modify the settings of an existing environment.
• Delete: Remove an environment.

Danger

Deleting an environment is a permanent action and cannot be undone.

Scan Metadata (Database Only)

Use the Scan Metadata action (repeat icon) in the table row to create or refresh database metadata for the selected environment. This action is available only for database environments that are not SQLite and prompts for confirmation before starting.

Scan behavior:

• The scan reflects all schemas in the selected database that are accessible for the configured user.
• The environment's configured schema remains the default schema for unqualified names.
• You can scope scans by adding a Custom Attribute include_tables (comma-separated table names, for example public.users,public.orders).
• Schemas without access permissions are skipped and logged as warnings in the task timeline.

Reset Metadata (Database Only)

Use Reset Metadata to clear stale or inconsistent metadata before running a fresh scan.

Recommended sequence after schema changes:

1. Reset Metadata
2. Scan Metadata

Adding an Environment

To add a new environment, click the Add button on the top right of the Environments pane.

Wizard Steps for Adding an Environment

The wizard guides you through several steps to configure your environment:

Step 1: Choose Name And Type

Provide a name for your environment and select one of the environment types.

• Database
• Kafka
• Mongo
• Data Warehouse

Choose Environment Type

Step 2: Configure Connection Settings

Settings in this step depend on the selected environment type.

Database Settings

Configure your database settings. For example, if you select PostgreSQL, you will need to provide details like Host, Port, Database name, Schema, User, and Password.

For Kafka-specific TLS/SASL connection guidance, see Kafka TLS & Security.

Oracle-specific notes:

• Oracle connections support both service name and SID targets.
• If both service-style target and SID are provided, service-style resolution is preferred.
• Legacy Oracle setups that only provide database are still treated as service-name style target resolution.

Test your Database Settings

Kafka TLS & Security (Simplified Flow)

When you configure a Kafka environment with SSL or SASL_SSL, the wizard now adds a dedicated TLS & Security step.
If you use PLAINTEXT, this step is hidden so the flow stays focused.

Kafka TLS step with simplified certificate input (upload flow with staged apply)

Why this flow is simpler:

• No empty security step: TLS/SASL sections appear only when required by the selected security protocol.
• Upload-first UX: Drag-and-drop is the primary input for PEM/key files, and staged changes are applied explicitly.
• Reduced certificate friction: CA Certificate, Client Certificate, and Client Key are grouped in one TLS step; provide client cert/key when broker requires mTLS.
• Consistent storage/runtime behavior: Uploaded binary materials are converted and persisted in a runtime-safe format for health checks and task execution.

Kafka Security Modes (Enterprise Quick Guide)

Protocol	Transport encryption	Authentication	Typical certificate requirement
PLAINTEXT	No	None	No TLS material
SASL_PLAINTEXT	No	SASL user/password	No TLS material
SSL	Yes	TLS certificate-based	CA Certificate always, client cert/key only for mTLS
SASL_SSL	Yes	SASL user/password over TLS	CA Certificate always, client cert/key only if broker enforces mTLS

What Each TLS Field Means

• CA Certificate: trust anchor(s) used by the client to verify broker certificates. Required for SSL and SASL_SSL.
• Client Certificate: public certificate chain presented by the client when broker requires client authentication (mTLS).
• Client Key: private key matching the client certificate.
• Client Key Password: only needed when the private key is encrypted.

Certificate Chain Rules

• Use PEM format for all TLS materials.
• Client Certificate file should contain:
• leaf/client certificate first
• intermediate CA certificates after that
• do not include the private key in this file
• CA Certificate file should contain trusted CA certs (root and, if needed, intermediates).
  Include intermediates when your broker chain is incomplete or enterprise trust policy requires explicit bundles.

mTLS Decision Logic

• If broker-side ssl.client.auth=required, client must provide valid Client Certificate + Client Key.
• If broker-side client auth is not required, you can connect with CA Certificate only (plus SASL credentials for SASL_SSL).
• If Client Key is set without Client Certificate, client-certificate auth cannot work; configure both together.

Recommended Validation Sequence

1. Verify protocol and auth mode with platform/network team (SSL vs SASL_SSL, mTLS required or not).
2. Validate broker certificate trust path using your CA bundle.
3. If mTLS is required, validate client cert/key pair and key password together before rollout.
4. Run a negative test:
5. expected mTLS: connection without client cert should fail.
6. expected non-mTLS: connection should still succeed with CA + protocol credentials.
7. Confirm behavior against effective runtime broker config (not only repository defaults).

Common Misconfiguration Patterns

• Wrong key password appears to have no effect: usually means client key was not used (for example missing client certificate).
• Handshake failures on enterprise clusters: often caused by incomplete CA chain or wrong cert/key pairing.
• Intermittent environment differences: broker-side effective config differs between test/prod clusters.

Step 3: Confirm

Review your settings and confirm the creation of the new environment.

Tip

You cannot find your database system? Get in touch with our team to enable this type for your account or instance.