Bỏ qua

Data Sources

DATAMIMIC supports multiple data source types for reading and writing data.

Data Source Overview

Source Type Read Write Encoding Config Delimiter Config
Project Files ✓ Via UI Auto-detected Auto-detected
Object Storage ✓ ✓ Configurable Configurable
Databases ✓ ✓ N/A N/A
MongoDB ✓ ✓ N/A N/A
Kafka ✓ ✓ Configurable N/A

Project Files

Files within your project (uploaded via UI):

1
<generate name="customers" source="data/customers.ent.csv" target="mem"/>
  • Upload: Any encoding/delimiter (auto-detected)
  • Download: Uses project settings (encoding + delimiter)
  • Supported types: .ent.csv, .wgt.csv, .optl.csv, .xml, .json
  • Note: All columns from the file are already in context. Use <key> only to override/add fields.

Upload Any Format

You can upload CSV files with any encoding or delimiter. The platform automatically detects and converts them.

Object Storage

External files from S3, Azure Blob, or MinIO:

1
2
3
4
5
6
7
8
<object-storage id="s3" />

<generate name="legacy"
          source="s3"
          bucket="imports"
          sourceUri="data/legacy.csv"
          encoding="windows-1252"
          separator=","/>
  • Encoding: Configurable via encoding attribute
  • Delimiter: Configurable via separator attribute
  • Use case: Legacy systems, shared data, large files

Quick map (read vs write)

Scenario Attribute(s) Meaning
Read from object storage source + sourceUri source selects client id; sourceUri is object key/path
Write to object storage storageId + exportUri storageId selects client; exportUri is path/prefix only

Object Storage Attributes

Attribute Description Required
source Object-storage client id (<object-storage id="...">) Yes
bucket / container Storage bucket (MinIO/S3) or container (Azure) Yes
sourceUri Path to file within bucket Yes
encoding Character encoding No (default: UTF-8)
separator CSV delimiter No (default: from runtime config)

Object Storage Writes

For file exporters (CSV/JSON/XML/Template), choose the storage client with storageId and set the output path with exportUri. exportUri is path/prefix only; the filename comes from name or targetEntity, and the extension is defined by the exporter.

1
2
3
4
5
6
7
8
9
<object-storage id="s3" />

<generate name="orders_export"
          target="CSV"
          storageId="s3"
          bucket="exports"
          exportUri="daily/">
  <key name="id" generator="IncrementGenerator"/>
</generate>

Databases

SQL databases (PostgreSQL, Oracle, MySQL, MSSQL):

1
2
3
4
5
6
<database id="source" system="production_db" />

<generate name="customers"
          source="source"
          selector="SELECT * FROM customers">
</generate>
  • Encoding: Handled by database driver
  • Use case: Live data, transactional systems

Database Attributes

Attribute Description Required
source Database client ID Yes
selector SQL query Yes (or use table name)
type Target table for iteration No

Contract note:

  • On top-level <generate>, selector uses datasource/loader read semantics.
  • If you need setup-cached reference data semantics instead, use a database-backed <variable selector="...">.

MongoDB

Document database:

1
2
3
4
5
6
<mongodb id="mongo" system="mongodb" />

<generate name="orders"
          source="mongo"
          selector="find: 'orders', filter: {'status': 'active'}">
</generate>

MongoDB Operations

Operation Selector Format
Find find: 'collection', filter: {...}
Aggregate aggregate: 'collection', pipeline: [...]

Important note:

  • Mongo aggregate is a selector-kind exception and should not be assumed to share the same paging semantics as Mongo find or SQL selectors.

Kafka

Message streaming:

1
2
3
<kafka-importer id="events" system="kafka" />

<generate name="events" source="events"/>
  • Encoding: Configurable via environment settings
  • Use case: Real-time data, event processing

Future Sources

Planned support for:

  • FTP/SFTP
  • HTTP/REST APIs
  • WebDAV

Choosing a Data Source

Scenario Recommended Source
Static reference data Project Files
Large external datasets Object Storage
Live transactional data Database
Document data MongoDB
Real-time events Kafka
Legacy systems with specific encoding Object Storage

Encoding Priority

When reading from external sources, encoding is resolved in this order:

Priority Source Example
1 Element attribute encoding="windows-1252"
2 Setup defaults <setup defaultEncoding="ISO-8859-1">
3 Runtime config (UI) Settings → Runtime Configuration
4 Fallback UTF-8

Delimiter Priority

For CSV sources, delimiter is resolved in this order:

Priority Source Example
1 Element attribute separator=";"
2 Setup defaults <setup defaultSeparator=";"/>
3 Runtime config (UI) Settings → Runtime Configuration
4 Fallback Pipe (\|)