Upgrade your models from DATAMIMIC 3.5 to 4.0

You do not need to rewrite every model. Use the checklist below to find the features your projects use, then follow only the relevant migration steps. Unknown attributes and invalid values still fail. The known legacy forms listed below remain executable with a warning: unused settings are ignored, and the old Memstore dataset selector is converted to sourceEntity.

This guide covers XML models and their output. Deployment, sign-in, environment-variable and API changes are separate upgrade tasks. If you are moving from Benerator, use the Benerator migration guide instead.

Why these changes?

4.0 makes model settings more explicit and catches more mistakes before they affect your data. A misspelled option or bogus value still reports an error. A small set of harmless legacy declarations is accepted with a warning so existing models do not need cleanup just to run. The warning directs optional cleanup; it does not reactivate the old setting. Cleanup should clearly say whether it deletes selected records or an entire dataset. Seeded tests should not change just because they run on another day.

These improvements have an upgrade cost: some old settings must change, and some models produce different values. Each section explains when you are affected, what the change helps with, and what to do. If you do not use the feature, skip it.

Find the changes that affect you

If your project uses…	Review before upgrading
Seeds, random expressions, or exact expected-output files	Random values and reproducibility
Hash, saved pseudonyms, or hashed identifiers used in joins	Hashes and linked identifiers
Current dates, relative dates, or date/time weights	Dates and weight expressions
Old attributes or type for entity selection	XML attributes and element names
<memstore> or data reused between generation steps	Memstore: write and read with mem
.wgt.ent.csv files	Weighted entity sources
Kafka input or ML model sources	Bounded sources and supported options
Database deletion or cleanup	Delete, clear, and drop
<operate> template processing	Template processing and export destinations
<ama-generate> or AMA/EDIFACT templates	AMA declarations and references
Cyclic database reads, multiple single-file exports, or whole Company entities	Row counts and output files
Custom Python generators or direct EE imports	Python extensions

Before testing, save your 3.5 project, referenced input files, configuration and expected outputs. Use a separate test destination, especially for database writes and pseudonyms. A successful run is not enough: compare row counts, identifiers, relationships, dates and output locations before accepting new reference files.

Random values and reproducibility

Relevant when: your tests compare exact generated values, you rely on the same seed, or expressions call random.seed(...) or secrets.*.

What changed and why: generators now share one way of managing randomness. Each model step, field and row has its own random sequence, so independent fields do not accidentally repeat the same sequence. The trade-off: the same seed can produce different values in 4.0 than in 3.5. Runs without an explicit seed no longer get a hidden seed from the task ID.

How to migrate:

1. Set rngSeed on <setup> when reproducibility is required. An existing explicit seed can remain, but expected values need review.
2. Remove expression calls to random.seed(...): they still execute but no longer reseed anything. Replace secrets.* expressions with a supported generator or expression from the scripting reference.
3. Compare business rules first, then approve new 4.0 expected-output files. If two fields must be equal, reuse the computed value; do not rely on independently generated fields accidentally matching.
4. Keep the engine version, complete inputs, source snapshot/order and execution settings fixed for replay tests. Renaming or moving statements can change their random streams.

The rule-based replay boundary is:

Same engine version + same complete deterministic inputs + same explicit seed + same execution topology + same deterministic serialization and ordering → byte-identical rule-based artifact, across machines and time.

This is not a 3.5-to-4.0 byte-compatibility promise. Changing worker count needs a separate check for the source and output you use. ML output is not covered by the byte-identical rule-based replay claim.

Hashes and linked identifiers

Relevant when: you use Hash with a third argument or salt=, compare saved hashes, or use hashed values as identifiers across files, fields or tables.

What changed and why: Hash now uses the run seed and the field's location in the model instead of a separate salt argument. This lets the seed control hash values while keeping unrelated fields separate. That separation is not suitable for every use case: if two tables need matching IDs, hashing each field separately may break their links.

How to migrate:

• Replace Hash('sha256', 'hex', 'old-salt') with Hash('sha256', 'hex'); remove named salt= arguments too. Configure rngSeed on <setup> for seeded hashing. Expect different hash values; this does not preserve old pseudonyms.
• Check all joins and persisted identifiers. Equal input values map consistently across rows within the same seeded field context, but the same value and seed in different field paths do not guarantee the same token.
• Where several outputs must share an identifier, compute its pseudonym once and reuse that mapping. Agree the mapping and migration with your project owner before replacing linked datasets.
• Keep old and new pseudonymized datasets separate until their consumers have been migrated together. Do not simply overwrite expected hashes to make a test pass.

Without a seed, Hash uses an unkeyed, value-only hash. Removing the seed is not a migration strategy for preserving the previous protection or token contract. A seed is also not a substitute for secret-key management. See the converter reference for the versioned signature.

Dates and weight expressions

Current and relative dates

Relevant when: your model uses datetime.now(), datetime.utcnow(), date.today(), or generator windows relative to the current time.

What changed and why: runs with a seed use a fixed reference clock so a test does not change simply because it runs tomorrow. The default seeded reference is 2025-01-01 at 12:00:00, not the machine's current time. Unseeded runs use the live clock.

How to migrate: review date-dependent assertions and relative windows. If your scenario needs a specific business date, supply an explicit value or date window rather than assuming that now() means the execution date. For example, a fixed DateTimeGenerator(value='2026-07-30 12:00:00') expresses a scenario date directly. If you genuinely need live dates, evaluate the consequences of an unseeded run for the whole model before removing its seed. See date and time generation.

Date/time weights

Relevant when: hour_weights, minute_weights, month_weights or similar arguments contain Python expressions or list comprehensions.

What changed and why: weights are now read as lists of numbers, not executed as Python code. You can write a list directly, repeat it or join lists. This catches invalid weights without running code just to define how often a value should occur.

How to migrate: replace a string such as '[1 for _ in range(24)]' with '[1]*24'. Keep the required number of entries, non-negative weights and at least one positive weight. Recheck the resulting distribution, not just whether the generator starts.

XML attributes and element names

Relevant when: a 3.5 model contains obsolete attributes, spelling mistakes or uses type to select a dataset.

What changed and why: XML elements reject undeclared attributes by default. A typo still produces an error instead of appearing to configure behavior that the engine ignores. Known legacy attributes remain in the grammar for compatibility: any legacy string on those registered attributes executes with warning W004 and is ignored. Canonical typed fields still validate their values, and documented driver-specific attributes remain exceptions on their client declarations.

How to migrate: use the validation error's element and attribute name, then check the model reference. Common edits are:

Old input	Change for 4.0
multiprocessing on <setup>, <generate> or <iterate>	Any legacy string is accepted with W004 and ignored. Optionally remove it; use numProcess for process count and mpPlatform on <generate> for the multiprocessing start method. Do not treat the old value as a process count.
<setup schemaVersion="…">	Any legacy string is accepted with W004 and ignored. Optionally remove it; the installed engine determines the DSL version.
<operate template-dir="…">	Accepted with W004 and ignored. Optionally remove it; put the correct template reference in each control row.
<nestedKey source="mem" type="orders">	Alias for sourceEntity="orders"; accepted with warning W006, so a misspelled list or dict becomes visible. Rewrite it when convenient. Keep type for list or dict. An explicitly different sourceEntity remains an error, and other source types still validate strictly.
<ama-generate variable_prefix="…" variable_suffix="…">	Alias for variablePrefix/variableSuffix; accepted without a warning. 3.5 ignored the snake_case spelling and used the default __; 4.0 applies the value, so review models that set another prefix or suffix.
Object-storage type="s3"	Use the supported type="aws". The old spelling was not a working S3 client type.

Unknown attributes and bogus values on canonical typed fields remain errors. W004 applies only to the registered legacy attributes above; their legacy string values are accepted and ignored. Any <ml-train mode> string, including persist, is accepted with W004 and ignored; it never controls persistence. Remove it when convenient.

Do not remove supported aliases merely because they look old. For example, type="integer", database postgres/sqlserver, and mpPlatform="multiprocessing" remain accepted. See compatibility retained for forms accepted at runtime but no longer suggested for new models.

Memstore: write and read with mem

Relevant when: your model declares <memstore> or stores generated records for a later step in the same run. If you already use target="mem" and source="mem" without a separate declaration, no change is needed.

What changed and why: the old standalone <memstore> declaration already had no effect on the 3.5 runtime. A legacy <memstore id="…"/> child with a required, valid id remains accepted with warning W005 and ignored, so existing models need no unnecessary edit just to run. The declaration does not create a custom client, source or target ID. Remove it when convenient; the built-in Memstore workflow remains unchanged: generate data once and reuse it in later steps without exporting and reloading a file.

How to migrate: removing the declaration is optional cleanup. Keep the steps that generate and reuse your data. Use mem as the built-in source and target name:

Setting	Meaning
target="mem"	Write generated records to the Memstore.
name="customers"	Store them under customers by default. An explicit targetEntity changes that dataset name.
source="mem"	Read from the Memstore.
sourceEntity="customers"	Select the stored dataset named customers. Use the producer's targetEntity instead if you set one.

The legacy declaration's id is not the product name. Set the product name on the producing targetEntity and consuming sourceEntity when needed. A missing or invalid legacy id is still an error.

mem is the built-in name, not an automatic output destination when target is omitted. Write to it explicitly and place the producing step before the reading step.

This complete example stores two customers, then reads them back and shows them in the task log:

<setup numProcess="1">
  <generate name="customers" count="2" target="mem">
    <key name="customer_id" generator="IncrementGenerator"/>
    <key name="status" constant="active"/>
  </generate>
  <generate name="customer_copy" source="mem" sourceEntity="customers"
            distribution="ordered" target="LogExporter"/>
</setup>

Check the result: customer_copy contains the same two customers, with IDs 1 and 2 and status active. Memstore data belongs to this run; use a file or database export if another run needs it. LogExporter here is only for checking the small example.

Weighted entity sources

Relevant when: a source filename ends in .wgt.ent.csv.

What changed and why: .wgt.ent.csv now consistently means that rows are selected according to their weights. You must name the weight column and say how many rows to select. This prevents a weighted file from being read like an ordinary list, ignoring how often each row should be chosen.

How to migrate:

1. On <generate>, add weightColumn="sample_weight", replacing sample_weight with the actual column name. <variable> and <reference> default to the column weight, as in 3.5; set weightColumn only when your column has another name.
2. Add count on <generate>; it must not depend on the file's row count being inferred. For example, count="100" requests 100 selections.
3. Use distribution="weighted" or omit it to use the weighted default. Remove cyclic="true" and unique="true"; weighted selection allows repeated rows.
4. Check that downstream consumers do not require the weight column as an output field: it is selection metadata and is removed after selection.

If you never intended weighted sampling, rename the file to .ent.csv, update its references and review its columns. Do not add arbitrary weights just to silence validation.

Bounded sources and supported options

Kafka input and ML model sources

Relevant when: a <generate> reads Kafka messages or an ML model source and previously used selectors, cyclic/unique sampling or a non-ordered distribution.

What changed and why: these sources now report an error for options they do not support instead of ignoring them. This prevents a model from appearing to filter or remove duplicates when it does neither. Kafka is a stream, so you must also limit how many messages a generation step reads.

How to migrate:

• For Kafka, set an explicit positive count for the intended batch, for example count="100". This is the simplest migration, not the only supported count expression: the shared count logic also accepts computed counts and minCount/maxCount. A zero count requests no records.
• For ML sources, set count or a bounded maxCount strategy.
• Omit distribution or use distribution="ordered". Remove selector, iterationSelector, cyclic="true" and unique="true". Explicit false does not enable the unsupported feature.
• If filtering or deduplication was your intent, implement it in the upstream data/topic or training-data preparation. Removing an ignored option does not add the behavior it was meant to request.

Kafka order remains partition-local; ordered does not promise global ordering across partitions. RabbitMQ is new in 4.0, so its setup is not a migration of existing 3.5 models.

Kafka and ML configuration values

Relevant when: validation rejects a previously unchecked Kafka or ML option.

What changed and why: supported choices and numeric bounds are checked before starting the operation, so invalid settings do not become late client or training failures.

How to migrate: follow the allowed values in the model reference. For example, Kafka acks="2" is invalid; choose 0, 1 or all according to the acknowledgement behavior you require. ML batchSize="0", negative samplingTopP, and trainValSplit above 1 are invalid. Choose a valid value for your intended training behavior rather than copying a default indiscriminately. Already-valid settings need no change.

Delete, clear, and drop

Relevant when: a model uses a database .delete target, particularly for cleanup of a whole table or collection.

What changed and why: deleting selected records, emptying a table and removing a collection now have distinct operations. This makes destructive steps easier to review. Reading and deleting the same database table page by page is rejected: deletions move rows between pages and can cause records to be skipped.

How to migrate: first decide which operation you actually intend.

Intent	Target and model shape
Delete selected records	Keep .delete and an explicit selection. For MongoDB, use a non-empty filter.
Remove all rows/documents but retain the table/collection	Use .clear with targetEntity.
Remove a MongoDB collection	Use .drop with targetEntity. RDBMS .drop is not supported.

Run .clear and .drop once at the root <setup> level. Remove source, selector, count, child elements and additional targets from that operation. These operations destroy data. Test them only against a disposable target with a recovery plan.

For selective RDBMS deletion, read the keys from a separate staging table or save them separately before deleting. Two client IDs pointing at the same physical source table do not avoid the restriction. Complex selectors whose source table cannot be established can also be rejected; use a simple, explicit staging source.

Some old cleanup forms still run with warning W003, but compatibility is conditional. They must be root-only, have one target, no script or children, and an explicit targetEntity unless a static MongoDB find supplies it. Missing or ambiguous selectors are not permission to clear data. Prefer the explicit form when editing the model.

Template processing and export destinations

Relevant when: your model contains <operate>, especially if output location depended on FORCE_LOCAL_EXPORT or control rows omitted a template.

What changed and why: the model now determines where files go; FORCE_LOCAL_EXPORT no longer redirects them. You can check the destination in the model itself. Missing template references and unsupported formats now report errors, helping you catch incomplete output instead of mistaking it for a successful delivery.

How to migrate:

• Choose export_strategy="local" to write beside the descriptor. This is the new default, so check destinations even if your model never set export_strategy.
• Choose export_strategy="minio" only with exactly one configured MinIO client and a bucket. Remove reliance on FORCE_LOCAL_EXPORT.
• Replace export_strategy="s3" with a supported destination or a separately supported object-storage export workflow. The old value was accepted but did not implement AWS S3 delivery.
• Give every non-empty control row a template reference ending in .xml or .json. A missing template cell or an unsupported suffix is a configuration error even under a warning policy.
• Distinguish that from a named template file that cannot be found: template_not_found_action still controls that case. Verify file paths and review warnings before accepting the output.

AMA declarations and references

Relevant when: your model uses <ama-generate> with profiles, templates, outputs or packages.

What changed and why: profile and template IDs must be unique, and every reference must point to an existing definition. These checks catch missing or conflicting settings before output is created, making mistakes easier to locate.

How to migrate: give profiles and templates unique IDs, resolve every profile/template reference, and select exactly one of type or messageType on each output. Keep at most one package, with type="zip". Remove unknown attributes and compare the produced artifacts with the expected set. Use the model reference for the attributes of each child element.

Template tokens: 3.5 ignored month and year offsets in {Date: offset=…} templates, so +P3M20D or -P30Y produced the current date, and it left {IK: prefix=…} unresolved in the output. 4.0 applies ISO 8601 durations with calendar-month arithmetic and resolves {IK: prefix=NN} to the 2-digit prefix followed by seven digits; another prefix leaves the token visible. An unsupported duration stays visible in the output instead of becoming today's date. Compare dates and IK values in AMA outputs with your expected files.

Row counts and output files

Cyclic database reads

Relevant when: you use cyclic="true" with a database source and request more records than the source contains.

What changed and why: the source snapshot is repeated to fill the requested count rather than silently returning fewer rows. This makes the requested test volume meaningful, but can introduce repetitions your old expected output did not contain.

How to migrate: compare counts and duplicate expectations. Keep cyclic when repetition is intended; otherwise remove it and choose a count supported by the source. Do not reduce the count automatically if the larger test volume was the original requirement.

Whole Company entity output

Relevant when: a model exports or compares the complete Company entity instead of individual fields.

What changed and why: whole-entity output now includes legal_form, which was already available as company.legal_form. The entity reference lists the same fields that the export writes.

How to migrate: add legal_form to expected files and to downstream schemas that read the full entity. Models that select individual fields are unaffected.

Static single-file output collisions

Relevant when: several root generation stages, including included stages, write JSONSingle, XMLSingle or TemplateSingle outputs with the same resolved static name.

What changed and why: a later conflicting stage is rejected before it writes, protecting earlier output from being overwritten.

How to migrate: give independent deliveries distinct names or export prefixes/suffixes/locations. Check the resulting filenames in downstream jobs. This check is not a universal overwrite guarantee: output routed through targetEntity is excluded, so you must still verify those destinations yourself.

Compatibility retained

These forms do not require a rewrite just to execute in 4.0. Prefer the clearer form when editing them:

Accepted runtime form	Preferred form and reason
<variable source="mem" type="orders">	Use sourceEntity="orders". It separates entity selection from scalar type; the old form is no longer suggested by the editor/Authoring contract.
<nestedKey source="mem" type="orders">	Alias for sourceEntity="orders"; accepted with warning W006. Keep type for list or dict. Conflicting explicit sourceEntity values remain errors.
<ml-train mode="...">	Any legacy string is accepted with W004 and ignored. Remove it when convenient; it does not control persistence.
schemaVersion on <setup>	Any legacy string is accepted with W004 and ignored. Remove it when convenient; the installed engine determines the DSL version.
multiprocessing on <setup>, <generate> or <iterate>	Any legacy string is accepted with W004 and ignored. Remove it when convenient; use numProcess and mpPlatform for parallelism.
<operate template-dir="…">	Accepted with W004 and ignored. Remove it when convenient; use the template reference in each control row.
<setup><memstore id="…"/></setup>	Accepted with W005 and ignored when id is valid. Remove it when convenient; keep using target="mem", source="mem" and sourceEntity.
<ama-generate variable_prefix="…" variable_suffix="…">	Prefer variablePrefix/variableSuffix; the snake_case form is an alias for them and is accepted without a warning.
<iterate> with a target	Use <generate> for a stage that creates/exports output. Runtime compatibility remains, while Authoring reserves iterate for targetless source traversal.
Supported legacy cleanup shapes	Follow the W003 warning and migrate to explicit .clear/.drop intent as described above.

Python extensions

Relevant when: you maintain custom Python generators or directly import EE implementation modules. XML-only projects can skip this section.

What changed and why: custom generators now use the same random context as built-in generators. This allows their values to follow the run's seed too. Some old constructor arguments and internal imports have been removed, so extensions that use them need updating.

How to migrate: review removed seed/rng/seeded_mode constructor arguments and imports such as old RNG helpers against the current generator reference. Use the context-aware bind(...)/generate(rng_ctx) interface where your extension needs the run's randomness. A legacy custom generate(self) remains callable; not every extension requires a signature change. Acceptance of a legacy call does not prove deterministic output. Run the extension's own tests with the 4.0 dependency before deploying it.

Verify the migrated project

1. Validate the edited XML in the IDE or Platform and resolve unsupported attributes/options.
2. Run a small dataset against isolated test destinations. Review warnings as well as errors.
3. Check row counts, required fields, identifier uniqueness, cross-table relationships, dates and output paths.
4. For replay tests, repeat with the same complete inputs and seed. Keep ML quality checks separate from exact-output comparisons.
5. Approve new expected outputs only after the business checks pass. Keep the old project and data until dependent consumers have accepted the migration.

If a migration changes shared pseudonyms, destructive operations or a custom extension whose contract is unclear, agree the expected behavior with your DATAMIMIC project contact before rolling it out.