Element <generate>

Purpose: Generate, transform, iterate over, and optionally export a product stream.

Why: Use it for a bounded product stream that generates, transforms, or exports records.

Example

<generate name="customers" count="1"/>

Decision guide

Business value: Turns synthetic or source-derived records into a bounded, optionally exported product stream.

• Use when

  • Creating a fixed synthetic dataset.
  • Transforming a bounded source into one or more targets.
  • Generating a deterministic time-series window.

• Choose another approach when

  • Traversing a source without producing an output target.
  • Mutating an existing dataset through an operation-control workflow.

• Prerequisites

  • Provide a product name and one unambiguous execution basis: count, source, or time window.

• Alternatives

  • Use iterate to communicate source traversal without an output target. (See: <iterate>)
  • Use operate for reviewed mutations of existing data. (See: <operate>)

Complete examples

Generate a synthetic diagnostic product

Use this minimal complete model when no production-derived source data is required and inspect the bounded result in task logs before selecting a persistent target.

<setup>
    <generate name="customers" count="10" target="LogExporter">
        <id name="id" generator="IncrementGenerator"/>
        <key name="email" generator="EmailAddressGenerator"/>
    </generate>
</setup>

Transform an ordered entity CSV source

Use this form when source order must be retained while inspecting a bounded transformation before selecting its persistent target.

id|name
1|Ada
2|Grace
3|Linus

<setup defaultSeparator="|">
    <generate name="ordered_customers"
              source="data/customers.ent.csv"
              count="3"
              distribution="ordered"
              target="LogExporter"/>
</setup>

Reuse a bounded memstore source cyclically

Use cyclic only when deliberate repetition is preferable to stopping at pool exhaustion.

<setup>
    <generate name="seed_customers" count="3" target="mem">
        <id name="id" generator="IncrementGenerator"/>
    </generate>
    <generate name="cycled_customers"
              type="seed_customers"
              source="mem"
              count="8"
              cyclic="true"
              distribution="ordered"
              target="LogExporter"/>
</setup>

Generate a bounded time series

Use start/end/interval when row positions represent deterministic time ticks.

<setup>
    <generate name="events"
              start="2026-01-01T00:00:00+00:00"
              end="2026-01-02T00:00:00+00:00"
              interval="PT1H"
              target="LogExporter">
        <key name="timestamp" script="ts.now.isoformat()"/>
        <key name="step" script="ts.step"/>
    </generate>
</setup>

Rules and invalid combinations

Generation requires a count strategy, source, script, time-series interval, or operation target.

Attributes: count, minCount, maxCount, source, script, interval, target

Why: Without an input or bounded generation strategy, the runtime cannot determine work or cardinality.

<generate name="customers" count="10"/>

<generate name="customers"/>

Source-selection options require source.

Attributes: source, selector, separator, sourceScripted, cyclic, weightColumn, stratifyBy

Why: selector, separator, sourceScripted, cyclic, weightColumn, and stratifyBy only modify a source read.

<generate name="customers" source="data/customers.ent.csv" count="10"/>

<generate name="customers" count="10" selector="select * from customers"/>

A source may specify type or selector, but not both.

Attributes: source, type, selector

Why: selector owns the query shape, while type identifies an unselected source entity.

<generate name="customers" source="customerDb" type="Customer" count="10"/>

<generate name="customers" source="customerDb" type="Customer" selector="select * from customers" count="10"/>

Use count or the randomized minCount/maxCount strategy, never both.

Attributes: count, minCount, maxCount

Why: Both strategies own the number of rows, so combining them would make execution ambiguous.

<generate name="customers" count="10"/>

<generate name="customers" count="10" minCount="2"/>

minCount must be less than or equal to maxCount.

Attributes: minCount, maxCount

Why: The randomized count is drawn from the inclusive interval between the two bounds.

<generate name="customers" minCount="2" maxCount="10"/>

<generate name="customers" minCount="10" maxCount="2"/>

minCount/maxCount cannot be combined with start/end/interval.

Attributes: minCount, maxCount, start, end, interval

Why: Time-series length is derived from its window and interval, not from a randomized row count.

<generate name="events" start="2026-01-01T00:00:00Z" end="2026-01-02T00:00:00Z" interval="PT1H"/>

<generate name="events" minCount="2" start="2026-01-01T00:00:00Z" end="2026-01-02T00:00:00Z" interval="PT1H"/>

offset requires source.

Attributes: offset, source

Why: An offset skips source rows and therefore has no meaning for purely synthetic generation.

<generate name="customers" source="data/customers.ent.csv" offset="2" count="10"/>

<generate name="customers" count="10" offset="2"/>

resume='group' requires resumeGroup.

Attributes: resume, resumeGroup

Why: The runtime needs the shared group name to resolve continuation state.

<generate name="customers" count="10" resume="group" resumeGroup="customer-import"/>

<generate name="customers" count="10" resume="group"/>

resumeGroup requires resume='group'.

Attributes: resume, resumeGroup

Why: The group name has no meaning for statement- or table-scoped continuation.

<generate name="customers" count="10" resume="group" resumeGroup="customer-import"/>

<generate name="customers" count="10" resume="stmt" resumeGroup="customer-import"/>

unique requires a finite source.

Attributes: unique, source

Why: Distinct sampling without replacement needs a source pool.

<generate name="customers" source="data/customers.ent.csv" unique="true" count="10"/>

<generate name="customers" count="10" unique="true"/>

unique cannot be combined with cyclic.

Attributes: unique, cyclic

Why: Unique selection consumes without replacement, while cyclic repeats exhausted rows.

<generate name="customers" source="data/customers.ent.csv" unique="true" count="10"/>

<generate name="customers" source="data/customers.ent.csv" count="10" unique="true" cyclic="true"/>

unique combines only with an explicit distribution='random'.

Attributes: unique, distribution

Why: Ordered or weighted selection would conflict with unique random sampling without replacement.

<generate name="customers" source="data/customers.ent.csv" unique="true" distribution="random" count="10"/>

<generate name="customers" source="data/customers.ent.csv" count="10" unique="true" distribution="ordered"/>

cyclic requires a distribution that supports repetition.

Attributes: cyclic, distribution

Why: Full-pool weighted and stratified draws do not define a stable exhaustion boundary to restart.

<generate name="customers" source="data/customers.ent.csv" cyclic="true" distribution="ordered" count="10"/>

<generate name="customers" source="data/customers.ent.csv" count="10" cyclic="true" distribution="weighted" weightColumn="weight"/>

distribution='weighted' requires weightColumn.

Attributes: distribution, weightColumn

Why: The selector needs a non-negative numeric column to calculate sampling probabilities.

<generate name="customers" source="data/customers.ent.csv" distribution="weighted" weightColumn="weight" count="10"/>

<generate name="customers" source="data/customers.ent.csv" count="10" distribution="weighted"/>

distribution='stratified' requires stratifyBy.

Attributes: distribution, stratifyBy

Why: The selector needs a source column that identifies each stratum.

<generate name="customers" source="data/customers.ent.csv" distribution="stratified" stratifyBy="segment" count="10"/>

<generate name="customers" source="data/customers.ent.csv" count="10" distribution="stratified"/>

A semantic project-file source requires its catalogued metadata attribute.

Attributes: source, weightColumn

Why: Weighted entity rows need weightColumn so selection can consume and then remove the metadata column.

<generate name="customers" source="data/customers.wgt.ent.csv" distribution="weighted" weightColumn="weight" count="10"/>

<generate name="customers" source="data/customers.wgt.ent.csv" count="10"/>

A semantic project-file source requires its catalogued distribution.

Attributes: source, distribution

Why: The file suffix declares selection semantics that an explicit conflicting distribution cannot override.

<generate name="customers" source="data/customers.wgt.ent.csv" distribution="weighted" weightColumn="weight" count="10"/>

<generate name="customers" source="data/customers.wgt.ent.csv" count="10" weightColumn="weight" distribution="ordered"/>

A weighted entity project source requires explicit count.

Attributes: source, count

Why: Weighted selection does not infer a bounded output size from source length.

<generate name="customers" source="data/customers.wgt.ent.csv" distribution="weighted" weightColumn="weight" count="10"/>

<generate name="customers" source="data/customers.wgt.ent.csv" weightColumn="weight"/>

W004 — Deprecated XML Attribute Ignored

<{element}> attribute '{attribute}' in descriptor '{descriptor}' is deprecated and ignored; execution continues. {migration}

Why: The descriptor uses a known legacy attribute that no longer controls execution.

Resolution: Follow the migration hint when updating the model. Removing the attribute is not required to run it.

Full rule

I883 — Time-Series Bad Start

Invalid time-series configuration: {detail}

Why: The 'start' attribute is not a valid ISO 8601 datetime.

Resolution: Set start to an ISO 8601 datetime, e.g. '2026-01-01T00:00:00Z'.

Full rule

I884 — Time-Series Bad End

Invalid time-series configuration: {detail}

Why: The 'end' attribute is not a valid ISO 8601 datetime.

Resolution: Set end to an ISO 8601 datetime, e.g. '2026-01-02T00:00:00Z'.

Full rule

I885 — Time-Series Bad Interval

Invalid time-series configuration: {detail}

Why: The 'interval' attribute is not a valid ISO 8601 duration.

Resolution: Set interval to an ISO 8601 duration, e.g. 'PT1H'.

Full rule

I886 — Time-Series End Not After Start

Invalid time-series configuration: {detail}

Why: The 'end' attribute is not strictly after the 'start' attribute.

Resolution: Set end to a datetime strictly after start.

Full rule

I887 — Time-Series Interval Too Fine

Invalid time-series configuration: {detail}

Why: The 'interval' duration is non-positive, sub-microsecond, or a months/years Duration.

Resolution: Set interval to a positive constant-length duration of at least 1us, e.g. 'PT1S'.

Full rule

I888 — Time-Series Namespace Collision

is not allowed inside a time-series : 'ts' is reserved for the time-iterator namespace (ts.now/ts.step/ts.series). Rename the variable, e.g. 'ts_meta'.

Why: A would shadow the reserved time-iterator namespace.

Resolution: Rename the variable to something other than 'ts', e.g. 'ts_meta'.

Full rule

I889 — Time-Series Incomplete Config

Time-series attributes start/end/interval must be set together; missing: {missing}

Why: Only some of the time-series attributes start/end/interval were provided.

Resolution: Provide all three attributes (start, end, interval) or none of them.

Full rule

I949 — Source ML Model Option Unsupported

ML model source '{source}' does not support {option}={value}. Supported behavior: {supported}.

Why: The requested source-selection option cannot be preserved by generated ml:// model samples.

Resolution: Remove the unsupported option and use bounded ordered ML model generation.

Full rule

I192 — Source Reference Identifier Empty

Explicit source URI for family '{family}' requires a non-empty identifier

Why: A recognized source-family URI was declared without the identifier needed to resolve its source.

Resolution: Add the source identifier after the URI scheme and retry.

Full rule

I194 — Source Reference Client Family Mismatch

Explicit source family '{family}' does not match configured client '{client_id}' of type '{actual_client_type}'

Why: The referenced client exists but does not implement the source family declared by the URI.

Resolution: Use the URI scheme matching the configured client or reference a client of the declared family.

Full rule

I195 — Source Reference Identifier Conflict

Explicit source identifier '{identifier}' conflicts with {attribute}='{configured_identifier}'

Why: Two source attributes select different identities for the same explicit source reference.

Resolution: Remove the legacy override or make it equal to the identifier in source.

Full rule

Allowed parents / Allowed children

Allowed parents: else, else-if, generate, if, iterate, setup, while

Allowed children:

array, assert, condition, echo, generate, id, include, iterate, key, list, mapping, nestedKey, reference, rule, sourceConstraints, targetConstraints, variable, while

Related concepts and use cases

• Dynamic Includes and Fragment Parameters — Explains URI interpolation, scoped include properties, typed fragment params, runtime caller context, nesting, conditions, and cataloged failures for reusable XML composition.
• Expression Evaluation and Lifetime — Explains the different evaluation lifetimes of literal values, cached expressions, and dynamic expressions so descriptor authors do not accidentally move runtime state into setup-time attributes.
• Variable Scoping in Nested Generates — Prevents accidental ancestor reads and shadowing, and routes memstore reads through the public Scripting API.
• Data De-identification — Distinguishes de-identification outcomes from masking, generalization, pseudonymization, and anonymization, then proves the current converter behavior with an executable source project.
• First DATAMIMIC Model — Builds the smallest model and then composes preparation and generation stages.
• Upgrade your models from DATAMIMIC 3.5 to 4.0 — Helps existing 3.5 users identify affected models, understand the reasons for 4.0 changes, and migrate syntax and expected results without confusing retained compatibility with breakage.
• Migrate Benerator Models to DATAMIMIC — Maps Benerator concepts to the current generated grammar without freezing another support matrix.
• Generate Values from Regular Expressions — Shows bounded structural patterns and explains why regex shape is not domain validation.
• Date and Time Generation — Explains deterministic windows, weighted selection, offsets, and epoch output.
• Generate Database-backed Sequences — Explains bounded database sequence reservation and its single-process policy.
• Custom Generators and Converters — Shows the trusted-code boundary and loader-provided base classes for project-local generators and converters without exposing internal import paths.
• Use the Scripting API in a Model — Shows all read-only scripting helpers together at their real expression-evaluation boundary.
• Structured Data and Rule Pipelines — Shows how composite output shapes and the ordered source-filter, mapping, and target-filter pipeline work together without hiding phase boundaries.
• Choose a source and target — Guides users and agents from an input ownership boundary and intended side effect to one bounded source and target contract.
• Choose Output Artifacts and Logs — Separates persistent Platform artifacts from bounded diagnostic logs and documents representative runtime-owned file target options.
• Scripted Source Templates — Explains typed full-value source expressions, embedded string substitution, and the boundary between sourceScripted templates and field or generate scripts.
• Assemble a complex deterministic model — Provides the requirement-to-model sequence for products, field dependencies, relationships, targets, and acceptance evidence.
• Model relationships and correlated data — Explains when to use nested products, memstore lineage, or one correlated reference selection.
• Determinism, parallelism, paging, and distribution — Separates seed, distribution, worker policy, paging, and streaming so topology choices do not become false determinism claims.
• RabbitMQ bounded source and target — Explains bounded RabbitMQ consumption, competing-consumer ordering, broker-owned topology, publisher confirms, and unsupported finite-pool options.

Attributes

Show all 50 attributes

bucket

Bucket name for the external source or target.

optional; string; Default: null.

container

Container name for the external source or target.

optional; string; Default: null.

converter

Converter for element data transformation.

optional; string; Default: null.

count

Number of records to generate.

optional; string; Default: null.

cyclic

Enable or disable cyclic generation.

optional; boolean; Default: null.

device

Computation device to use.

optional; string; Default: null.

distribution

Distribution type for data generation.

optional; string; Default: null; Values: ordered, random, weighted, stratified, round_robin, reservoir, cumulated.

encoding

Override encoding for this generate task.

optional; string; Default: null.

end

Time-series window end (ISO 8601 datetime).

optional; string; Default: null.

exportUri

Explicit URI for exporters that support file paths.

optional; string; Default: null.

fairness

Fairness configuration (JSON).

optional; string; Default: null.

generationBatchSize

Batch size during generation.

optional; integer; Default: null.

imputation

Imputation configuration (JSON).

optional; string; Default: null.

interval

Time-series tick interval (ISO 8601 duration).

optional; string; Default: null.

iterationSelector

Selector evaluated per iteration.

optional; string; Default: null.

maxCount

Maximum count for randomized generate/iterate length (mutually exclusive with 'count').

optional; integer; Default: null.

minCount

Minimum count for randomized generate/iterate length (mutually exclusive with 'count').

optional; integer; Default: null.

mpPlatform

Multiprocessing platform override.

optional; string; Default: null; Values: multiprocessing, fork, spawn, forkserver.

multiprocessing

Deprecated: accepted with a warning and ignored. Use numProcess and mpPlatform instead.

optional; string; Default: null.

name

Name of the generation task.

required; string.

numProcess

Specify the number of processes to use.

optional; integer; Default: null.

offset

Skip the first N rows of the source.

optional; integer; Default: null.

pageSize

Page size for processing data.

optional; integer; Default: null.

page_bytes_cap

Hard cap on page size in bytes.

optional; integer; Default: null.

page_memory_cap_mb

Cap in megabytes for page memory usage.

optional; integer; Default: null.

rareCategoryReplacementMethod

Method for handling rare categories.

optional; string; Default: null; Values: constant, sample.

rebalancing

Class/feature rebalancing configuration (JSON).

optional; string; Default: null.

resume

Optional per-statement resume scope override.

optional; string; Default: null; Values: stmt, table, group.

resumeGroup

Logical resume group key used when resume='group'.

optional; string; Default: null.

samplingTemperature

Sampling temperature used for generation (0-2).

optional; number; Default: null.

samplingTopP

Top-p (nucleus) sampling threshold (0-1).

optional; number; Default: null.

script

Script driving generation logic.

optional; string; Default: null.

selector

Selector for data generation.

optional; string; Default: null.

separator

Separator for generated data.

optional; string; Default: null.

source

Canonical source URI (for example file://data/orders.csv, database://sourceDb, or ml://customer_model); legacy raw source values remain accepted.

optional; string; Default: null.

sourceClient

Override the client used to read when source is ambiguous.

optional; string; Default: null.

sourceEntity

Explicit physical entity to read (table/collection/product). Overrides selector or inferred name.

optional; string; Default: null.

sourceScripted

Enable or disable scripted sources.

optional; boolean; Default: null.

sourceUri

URI backing the source (file/object storage).

optional; string; Default: null.

start

Time-series window start (ISO 8601 datetime).

optional; string; Default: null.

storageId

Object storage client id.

optional; string; Default: null.

stratifyBy

Stratum column for distribution='stratified' source generation.

optional; string; Default: null.

target

Target output or explicit client operation for generated data.

optional; string; Default: null.

targetClient

Override the client used to write/operate when target is ambiguous.

optional; string; Default: null.

targetEntity

Explicit physical entity to write or operate on (table, collection, or product).

optional; string; Default: null.

type

Type of data generation.

optional; string; Default: null.

unique

Emit each source row at most once (distinct selection without replacement).

optional; boolean; Default: null.

variablePrefix

Prefix before field's name for query select data in selector element

optional; string; Default: null.

variableSuffix

Suffix after field's name for query select data in selector element

optional; string; Default: null.

weightColumn

Weight column for distribution='weighted' source generation; required for a .wgt.ent.csv source.

optional; string; Default: null.