target Attribute¶
The target attribute within the generate and execute element in DATAMIMIC allows users to specify the desired output format for the generated data. This functionality provides flexibility in directing the output to different formats or environments, whether built-in options like CSV or custom configurations like databases.
Exporter Types¶
DATAMIMIC provides two categories of file exporters:
Streaming Exporters (Single File Output)¶
These exporters write all records to one file per generate statement:
| Exporter | Output | Use Case |
|---|---|---|
CSV |
One .csv file |
Tabular data, spreadsheets |
JSON |
One .json file |
API payloads, data interchange |
XML |
One .xml file |
Structured documents |
Template |
One file (any format) | Custom formats, batch messages |
Why single file? Streaming exporters are optimized for high-throughput scenarios. They use chunking, batching, and queue streaming that require a stable filename throughout the generation run.
Single-File Exporters (One File Per Record)¶
These exporters create one file per record:
| Exporter | Output | Use Case |
|---|---|---|
JSONSingle |
name_1.json, name_2.json, ... |
Individual JSON documents |
XMLSingle |
name_1.xml, name_2.xml, ... |
Individual XML documents |
TemplateSingle |
name_1.ext, name_2.ext, ... |
SWIFT messages, EDI, invoices |
Ideal for: Payment messages (MT103, PACS), invoices, individual documents where each record needs its own file.
Built-in Supported Targets¶
DATAMIMIC natively supports the following output targets:
Preview: Default value, data generated will always get output with a limited subset to ourDATAMIMIC UI Preview.LogExporter: Prints the data in theLog View. This exporter it mostly used for debug purpose, so it has a default max limit of 100 items. We recommend to use our other exporters such as CSV, TXT... if you were to handle larger dataset.ConsoleExporter: Prints the generated data directly to the console, useful for quick debugging and verification.mem: Outputs the data to an in-memory store, allowing for quick access and manipulation within the same environment.CSV: Exports the generated data to a CSV file, which is ideal for handling tabular data and can be easily imported into spreadsheet applications. This file can be downloaded from our DATAMIMIC UI Artifact.JSON: Outputs the data in JSON format, suitable for web applications and data interchange between systems. This file can be downloaded from our DATAMIMIC UI Artifact.JSONSingle: Outputs the data in JSON format, but with each record in a single file. This file can be downloaded from our DATAMIMIC UI Artifact.OpenSearchBulk: Exports the data in OpenSearch bulk format, which is useful for bulk indexing data into OpenSearch. This file can be downloaded from our DATAMIMIC UI Artifact.TXT: Saves the data as a plain text file, suitable for simple text-based records or logs. This file can be downloaded from our DATAMIMIC UI Artifact.XML: Exports the data to an XML file, useful for structured data storage and data exchange between different systems. This file can be downloaded from our DATAMIMIC UI Artifact.Template: Exports the data to a file using a template file with variable substitution. Multiple records are written to a single file with a configurable separator. This file can be downloaded from our DATAMIMIC UI Artifact.TemplateSingle: Exports the data to files using a template file with variable substitution. Each record is written to a separate file. These files can be downloaded from our DATAMIMIC UI Artifact.
Notes
The target attribute supports multiple values, allowing you to combine different output formats or environments. This can be particularly useful when you want to direct the same generated data to multiple locations simultaneously.
Example Usage with Built-in Targets¶
Here’s an example of how to use the <generate/> element with a built-in target:
1 2 3 4 5 6 | |
Custom Environment Targets¶
DATAMIMIC also supports custom environments, allowing users to direct output to specific databases, message brokers, or other external systems. For example:
- Databases: Write generated data directly into a database.
- Kafka: Send generated data to a Kafka topic.
- MongoDB: Insert generated data into a MongoDB collection.
To use a custom environment as a target, we must first create the Environment from our DATAMIMIC UI, and then specify it in the target attribute:
1 2 3 4 5 | |
Check out our Database Tutorial for a more detail example usage.
Example Usage and configuration of supported targets¶
CSV¶
1 2 3 4 | |
we also can specify the CSV file configuration such as delimiter, quotechar, quoting, encoding, line_terminator and chunk_size:
1 2 3 4 | |
JSON / JSONSingle¶
We can generate JSON files with multiple records in a single file or each record in a separate file by define the target as JSON or JSONSingle:
JSONSingle is actually a special case of JSON, where each record is saved in a separate file. The records are not saved in an array, but as individual JSON files.
The same is happening when we use the JSON(chunk_size=1) target.
1 2 3 4 | |
We can also specify the JSON file configuration such as use_ndjson, encoding and chunk_size:
1 2 3 4 | |
OpenSearchBulk¶
The OpenSearchBulk target is used to generate data in the OpenSearch bulk format. This format is used to bulk index data into OpenSearch. The generated data is saved in a file with the extension .json or .ndjson and can be imported into OpenSearch using the bulk API.
The default configuration for the OpenSearchBulk target is as follows chunk_size=None, encoding='utf-8', use_ndjson=True: This means no chunking, utf-8 encoding, and using NDJSON format.
1 2 3 4 | |
when we set use_ndjson=False, the generated data will be saved in a JSON format and stored in an array:
1 2 3 4 5 6 | |
To use the OpenSearchBulk target, we can define the target as OpenSearchBulk and the model needs to specify the keys for the index, id, and routing:
$$_action$$: The action to perform on the document, such as index, create, or delete.
$$_index$$: The index to which the document belongs.
$$_id$$: The unique identifier for the document.
$$routing$$: The routing value for the document.
Here is an example of how to use the OpenSearchBulk target:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | |
here is the content of the script/template_xyz.json file:
1 2 3 4 5 6 | |
the output will be saved in a file with the extension .json and can be downloaded from our DATAMIMIC UI Artifact.
Preview of the generated data in the OpenSearchBulk format:
1 2 3 4 5 6 7 8 | |
TXT¶
The TXT target exports the generated data to a plain text file. By default, it includes both the generate name and key names along with their values.
1 2 3 4 | |
We can also specify the TXT file configuration using the value_only parameter:
value_only: A boolean parameter that controls the output format:
- True: Output only the values (without generate name and key names)
- False (default): Output the generate name, key names, and their values
Example with value_only=True:
1 2 3 4 5 | |
Output: Creates 1 file TXTExporter.txt with 10 lines containing only values separated by pipe:
1 2 3 | |
Example with value_only=False (default):
1 2 3 4 5 | |
Output: Creates 1 file TXTExporter.txt with 10 lines containing generate name, key names, and values:
1 2 3 | |
Template / TemplateSingle¶
Template and TemplateSingle are flexible exporters that generate messages/records from template files with variable substitution. They support any format including XML, EDI, SWIFT, JSON, TXT, etc.
Template vs TemplateSingle¶
| Feature | Template | TemplateSingle |
|---|---|---|
| Output | Multiple records into a single file | One file per record |
| Separator | Configurable template_separator |
No separator (each file is independent) |
Template Naming Convention¶
Template files must follow the naming convention: name.ext.template
payment.xml.template→ generatespayment.xmlinvoice.edi.template→ generatesinvoice.edistatement.mt940.template→ generatesstatement.mt940message.txt.template→ generatesmessage.txt
File extension is automatically derived from the template filename.
Basic Syntax¶
Template (Streaming - multiple records into one file):
1 2 3 4 5 6 7 | |
Output: Creates 1 file only_value.txt containing 10 records, each separated by line separator (default \n).
TemplateSingle (One file per record):
1 2 3 4 5 6 7 | |
Output: Creates 2 files: template_1.txt, template_2.txt
Template File Format¶
Template files use placeholder syntax: __variable_name__
Example Template Files:
templates/only_value.txt.template:
1 | |
templates/only_value.json.template:
1 2 3 | |
templates/payment.mt103.template:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | |
templates/invoice.edi.template:
1 2 3 4 5 6 7 8 9 10 11 12 13 | |
Parameters¶
Template Parameters:
| Parameter | Type | Default | Description |
|---|---|---|---|
exported_name |
string | Auto-derived | Output filename (without extension) |
template_separator |
string | \n |
Separator between records/messages |
error_handling |
string | "error" |
How to handle missing variables: "ignore", "warn", "error" |
encoding |
string | "utf-8" |
Character encoding |
exported_prefix |
string | None |
Prefix for filename |
exported_suffix |
string | None |
Suffix for filename |
TemplateSingle Parameters:
| Parameter | Type | Default | Description |
|---|---|---|---|
exported_name |
string | Auto-derived | Base output filename (will append _1, _2, ...) |
error_handling |
string | "error" |
How to handle missing variables: "ignore", "warn", "error" |
encoding |
string | "utf-8" |
Character encoding |
exported_prefix |
string | None |
Prefix for filename |
exported_suffix |
string | None |
Suffix for filename |
Usage Examples¶
Example 1: Template with custom separator
1 2 3 4 5 6 7 | |
Output: 1 file templateSeparatorTest.txt with 5 records separated by ---SEPARATOR---
Example 2: Template with error handling
1 2 3 4 5 6 7 8 9 10 11 | |
Output: 25 separate files: mt103_payments_1.mt103, mt103_payments_2.mt103, ...
Example 3: Auto-derived name from template
1 2 3 4 5 6 | |
Output: File orders_edi.edi (name auto-derived from orders.edi.template)
Example 4: Multiple formats
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | |
Variable Substitution¶
Placeholder Syntax:
__variable_name__- Single variable__var1____var2____var3__- Concatenated variables (joined without spaces)
Example:
Template:
1 | |
XML:
1 2 3 4 | |
Output:
1 | |
Error Handling¶
Modes:
"error"(default): RaisesRuntimeErrorwhen variable is missing"warn": Logs warning and continues (placeholder remains unchanged)"ignore": Silently skips (placeholder remains unchanged)
Example:
1 2 3 4 5 | |
Auto-derivation Features¶
Auto-derived file extension:
Extension is automatically derived from template filename:
payment.xml.template→.xmlinvoice.edi.template→.edistatement.mt940.template→.mt940
Auto-derived exported name:
If exported_name is not specified, name is derived from template:
orders.edi.template→orders_edipayment.mt103.template→payment_mt103message.template→message
Best Practices¶
- Template naming: Always use
name.ext.templateconvention to auto-derive extension - Separator for multi-line messages: Use explicit
template_separatorfor XML/EDI/SWIFT:
1 | |
- Error handling: Use
error_handling='ignore'when template has optional variables - File-per-message: Use
TemplateSinglewhen each message needs a separate file (SWIFT, EDI) - Batch messages: Use
Templatewhen streaming multiple messages into one file
Common Use Cases¶
SWIFT Messages (MT103, MT202, MT940):
1 2 3 4 5 6 7 8 9 10 11 | |
EDI Messages:
1 2 3 4 5 6 7 | |
ISO 20022 XML Messages:
1 2 3 4 5 6 7 8 9 | |
Notes¶
- Template exporter supports any format (XML, EDI, SWIFT, JSON, TXT, etc.)
- Variables can be generated from generators, entities, or constants
- Template files can contain multi-line content
- Separator only applies to
Template, not toTemplateSingle - File extension is auto-derived from template filename
Empty Template Output Behavior¶
When a template resolves to an empty string, the behavior differs between exporter types:
| Exporter Type | Empty Output Behavior |
|---|---|
TemplateSingle |
Creates an empty file (valid artifact) |
JSONSingle |
Creates an empty file (valid artifact) |
XMLSingle |
Creates an empty file (valid artifact) |
Template (streaming) |
No file created |
When to use which:
- Use
TemplateSinglewhen empty outputs are meaningful test data (e.g., optional message sections) - Use
Templatefor high-throughput streaming where zero-byte artifacts are not needed
1 2 3 4 5 6 7 8 9 10 11 12 13 | |
Dynamic targetEntity¶
Single-file exporters support dynamic filenames based on record data.
Why Only Single-File Exporters?¶
Streaming exporters (CSV, JSON, XML, Template) cannot support dynamic filenames because:
- They batch multiple records into one file
- Chunking and flush semantics require a stable filename
- Changing filenames mid-stream would break high-throughput guarantees
Single-file exporters create one file per record, so each file can have a unique name.
Syntax¶
| Syntax | Behavior |
|---|---|
{expr} |
Evaluated per record for targetEntity; use this form for clarity. |
{{ expr }} |
Treated the same as {expr} for targetEntity (no practical difference). |
Note: For targetEntity, {expr} and {{ expr }} resolve to the same filenames because the expression cache is reset
for each record and the value is evaluated once per record. For the general difference between cached ({expr}) and dynamic ({{ expr }}) expressions in other contexts, see Expression Syntax.
Example: Customer-Based Invoice Filenames¶
1 2 3 4 5 6 7 8 9 10 11 | |
Output files: CUST001.json, CUST002.json, CUST003.json, etc.
If the same targetEntity repeats in a single‑process run, a suffix is added on the duplicate:
CUST001_1.json, CUST001_2.json, ...
Example: Payment Reference Filenames¶
1 2 3 4 5 6 7 8 9 10 11 | |
Output files: PAYABC123456.mt103, PAYDEF789012.mt103, etc. (duplicates would get PAYABC123456_1.mt103)
Supported Exporters¶
| Exporter | Dynamic targetEntity | Reason |
|---|---|---|
TemplateSingle |
✓ | One file per record |
JSONSingle |
✓ | One file per record |
XMLSingle |
✓ | One file per record |
CSV |
✗ | Streams to single file |
JSON |
✗ | Streams to single file |
XML |
✗ | Streams to single file |
Template |
✗ | Streams to single file |
Extension Override (TemplateSingle)¶
Override the file extension derived from the template name:
1 2 3 4 5 6 7 8 9 | |
Output: PAY001.dat, PAY002.dat, etc. (instead of .mt103)
Error Handling¶
Using dynamic targetEntity with streaming exporters produces error I322:
1 2 3 4 | |
Error message:
1 2 | |
See Error Codes Reference for more details.
Filename Pattern (Single-File Exporters)¶
Naming depends on whether targetEntity is dynamic and whether multiprocessing is enabled:
- Static
targetEntity(default): always appends a counter:basename_1.json,basename_2.json, ... - Dynamic
targetEntity+ single process: first occurrence isbasename.json; duplicates add a suffix (basename_1.json,basename_2.json, ...). - Multi-process: always uses
basename_<worker_id>.<counter>.json(no cross‑process duplicate detection).