Release notes

2.5.1¶

Fixes¶

🐛 File Loading: Fixed an issue where switching file during edit can cause infinite loading.

2.5.0¶

Breaking changes¶

⬆️ API v2 is now Production Ready (API Users Only):
API v2 is now the default and fully supported version for all API integration.
Note: These changes only affect users who interact with the platform via the API. The web UI and other features are not impacted.
All existing API v1 endpoints are now officially deprecated and will be planned to be removed in our next major release.
The external API documentation has been updated to highlight and prioritize v2 endpoints.
To help API users transition smoothly, we provide a detailed Migration Guide with step-by-step instructions and best practices.

Fixes¶

🐛 User Sign Up: Resolved an issue preventing users from accessing the sign up page after logging out.
🐛 Demo Card Skeleton: Corrected the demo card placeholder width and layout when loading demo projects.
🐛 Project Settings: Fixed an error triggered when filtering users in project settings.
🐛 System Tables: Removed the index column from the system projects and system users tables.
🐛 Account Management: Allowed admins to clear first and last names to empty strings.
🐛 File Downloads: Ensured the latest file version is downloaded immediately after saving.
🐛 Project Deletion: Disabled the delete button while a multi-project deletion is still processing.

Features¶

✨ User Menu: The external API link in the user menu, previously visible only to admins, is now available to all users.
✨ User Projects: Added a loading state indicator when accessing or fetching the user's projects, improving the user experience during data retrieval.

2.4.0¶

Breaking changes¶

⬆️ Constraint syntax refactor:
<targetConstraints> / <sourceConstraints> now use require= for filtering only.
<mapping> now uses set= for transformations.
Legacy <rule> syntax removed.

Fixes¶

🐛 Tabular Data:
Fixed an issue where Tabular Data edited and saved through the UI was incorrectly persisted as a UI-JSON structure inside .csv files (e.g. .wgt.csv, .ent.csv, .optl.csv) instead of proper CSV rows.
- This caused tabular files to appear malformed or unreadable when accessed outside the UI.
- A migration now normalizes affected files into canonical pipe-separated CSV.
- Enable with DM_NORMALIZE_TABULAR_ON_START=true (default: disabled).
- Runs in dry-run mode by default; apply changes with DM_NORMALIZE_APPLY=true.
- Suitable for medium-sized MinIO / project repositories. For larger datasets, we provide a standalone script and support.
- Alternative: open affected files in the UI and re-save; they will be rewritten into correct CSV format and synced back to Project Data.
🐛 Maintenance Mode: Admin users no longer redirected away during maintenance.
🐛 Logging polish: Console + Redis handlers now consistently honor runtime_config.log_level.
🐛 Global project: Fixed an issue where global project file content doesn't display properly for normal user.
🐛 User Login: Fixed an issue where user might not be able to login when password is updated.
🐛 UI Toolbar: Fixed an issue where unnecessary toolbar is displayed on Windows devices.
🐛 User Project: User drop-down now shows and searches all users, not just 30, in project settings.

Upgrades¶

⬆️ Extended Log Handling: More meaningful, less noisy logs across the platform.
- Central policy allowlist for INFO events, sampling/deduplication, and sensitive-data scrubbing; consistent console/JSON formats.
- Honors LOG_LEVEL, LOG_FORMAT, LOG_COLOR; structured event fields with millisecond timestamps.
- Keeps the success path quiet while preserving high‑signal events for observability.
⬆️ Enterprise Database Connectivity:
- Oracle: SID + service_name support.
- MSSQL: instance, encryption, trustServerCertificate support; correct driver auto-selection.
⬆️ Read-only Safeguards:
- MongoDB: compressors normalized, read-only wrapper added.
- RDBMS: non-SELECT blocked, READ ONLY transactions enforced.
⬆️ Cursor & Keyset Execution Model:
- Offset/limit replaced by cursor/keyset iteration.
- Workers lease shards via manifest.
- Scales linearly to billions of records, reduces load on target systems, ensures exact-once sync.
⬆️ Memory & Engine Safety:
- Improved memory handling in loaders.
- Engine kwargs whitelisted for safe defaults.
⬆️ Rust Fast-Path Evaluation Engine:
- Core script evaluation hot paths rewritten in Rust.
- Higher throughput for heavy transformations.
⬆️ DateTimeGenerator: Major upgrade for enterprise use cases — see: DateTimeGenerator
- Modes: fixed value, random window with min/max, or current time.
- Bounds: strict min/max respected down to seconds; edge-day clamping.
- Determinism: private RNG with seed for repeatable sequences.
- Weights (intra‑day): hour_weights, minute_weights, second_weights (length-validated; non‑negative; sum>0).
- Day selection weights (inter‑day): month_weights, weekday_weights, dom_weights with proportional sampling.
- Sugar filters: friendly presets for months/weekdays/dom, hours_preset, minute_granularity, second_granularity (explicit weights override sugar).
- Epoch support: via XML formats (inDateFormat="epoch", outDateFormat="epoch|epoch_millis|epoch_micros|epoch_nanos|%Nf") and Python flags (as_epoch, epoch_unit).
- Performance: Rust fast‑path when no seed and no weights/sugar; buffered for throughput; Python O(1) sampling after per‑window build.
- Validation and error handling for invalid lengths/values, empty day sets, invalid granularities, and unknown presets.

Features¶

✨ Runtime Configuration (new): Tune log levels, paging, Redis/leases, Kafka, preview limits — now configurable for the first time.
✨ Capability-driven Environment Dialogs: Dynamic forms for DB, Kafka, Mongo, Object Storage; backend always in sync with UI.
✨ Enterprise Health checks: Oracle SELECT 1 FROM DUAL, MSSQL encrypted connections, Kafka TLS/SASL, Mongo compressors. Fail fast, realistic probes.
✨ Backup & Restore Enhancements: Backups embed DATAMIMIC version; Postgres backups exclude owner metadata → portable, audit-friendly.
✨ Duplicate Files via Filetree: Quickly duplicate project files with consistent naming (-copy, -copy-1, …).
✨ File Locking System: Multiuser file locking to prevent overwrites; supports intentional lock stealing for collaboration transparency. See: File Locking
✨ Filetree Usability Improvements: Realtime search filter, keyboard shortcuts (rename/delete), and “no results” hints for smoother navigation.
✨ API: Added new admin-only endpoints to access DATAMIMIC's instance Redis and MinIO statistics.
✨ Editor: Switched the in-app code editor to Monaco for improved reliability and consistency, while keeping the UI unchanged for a seamless user experience.
✨ Task Id: Task Id can be easily copied in the TaskView now. For usage in API for example.

Refactors¶

🔥 Registry slimmed to router-only role; parsing moved into loaders.
🔥 Dead code removed; helpers modularized.
🔥 Logging unified with millisecond precision timestamps.

2.3.0¶

Breaking Changes¶

⬆️ Syntax: Backslash-escaped colons (e.g., gc\:CodeList) are no longer supported.
- Please use bracket notation for namespaced keys: root['gc:CodeList'] (recommended).
- Alternatively, use plain colon identifiers (gc:CodeList), which are automatically sanitized.

Features¶

✨ Template System: Introduced a new Template Exporter and Management System for business messages:
- Export templates in free-text format with placeholder variables from the generation context.
- Create templates via the File Wizard, supporting JSON, XML, SWIFT, PACS, EDIFACT, and more.
- Seamless integration with existing data generation workflows.
✨ Variables: Enhanced variable management with the new storage attribute:
- storage="value" – Single static value (default for generators and constants).
- storage="data" – Complete data list stored in memory.
- storage="iterator" – Iterator that cycles or exhausts based on the cyclic setting.
✨ DateTimeGenerator: Extended with new attributes:
- as_epoch: bool = False returns the epoch timestamp as an integer.
- epoch_unit: str = "seconds" specifies epoch precision (seconds, milliseconds).
✨ Generate: Added IterateSelector to nested generate tasks for dynamic data source selection:
- Enables dynamic querying for each parent iteration.
- Supports both static and dynamic data sources.
- Provides greater flexibility in data generation scenarios.
✨ Demos: Added new demos:
- Template Exporter demo, showcasing capabilities and integration.
- Structural operations demo for XML and JSON, highlighting new Operate Element features.
- Extended DateTimeGenerator demo to illustrate new attributes.
✨ Namespace Access: Introduced a global root alias for robust namespaced key addressing (e.g., root['gc:CodeList']).
✨ Maintenance: Improved zip file handling for backup and restore operations, significantly reducing backup file size.

Refactors¶

🔥 Core: Refactored datasource and target handling for better performance and efficiency.
🔥 UI: Optimized API and performance across all UI components.
🔥 Contexts: Extracted a streamlined evaluation pipeline.
🔥 API: Improved error reporting for missing custom properties files in Generate API requests.
🔥 Workers: Worker system now acknowledges messages immediately after processing, preventing duplicate processing.

Fixes¶

🐛 Fixed an issue where zombie processes were not cleaned up properly.
🐛 Fixed truncation of expression error messages with float literals (e.g., 0.05); messages are now accurate and informative.
🐛 Improved handling of TypeError/ValueError in user expressions, providing actionable guidance and name-type details.
🐛 Enhanced error messages:
- Name-type hints (e.g., x=int, y=undefined).
- Specific guidance for common TypeErrors; colon retry path includes the original exception.
🐛 Maintenance Restore: Fixed restore failures for Postgres databases containing task data.
🐛 Maintenance Restore: Corrected confirmation dialog to display the correct filename during restore.
🐛 Editor: Resolved issues with values attribute examples.
🐛 Dashboard: Removed unnecessary refetch interval for the system user API call.
🐛 Error Connection Message: Connection error message now remains visible until the network is restored.
🐛 Notification View: Fixed React rendering error caused by objects being rendered as children.
🐛 Menu: Fixed missing username in the menu bar.

2.2.0¶

Breaking changes¶

⬆️ Project Reset: Removed the ability to reset a project from a specific task ID.

Features¶

✨ Snapshot Management: Enhanced snapshot capabilities for improved task execution control:
Re-run previous tasks from a snapshot, preserving exact project state
Download snapshot files from tasks to capture project state during execution
✨ API Integration: Integrated new API structure across all UI modules (Demo, Template, Job, Log, User, Login, System Management, Access Token)
✨ SQL Formatter: Upgraded from sql-formatter to sqltools-formatter for improved SQL formatting capabilities
✨ Operate Element: Introduced comprehensive batch operation support for template transformations:
Dual format support for XML (XPath) and JSON (JSONPath) templates
Advanced XML operations: replaceChildren, appendChild, upsertNode, mergeAttributes, clearChildren
JSON-specific operations: mergeObject, upsertProperty, replaceArray, appendItem, clearArray
Function operations for custom logic execution without content modification
Dynamic value injection using DATAMIMIC generators
Configurable error handling strategies
CSV-based operation definitions with customizable prefixes
Automatic validation of operation sequences
Seamless integration with business domain generators

Refactors¶

🔥 MongoDB: Improved delete operations for both selector-based and selector-free deletions

Fixes¶

🐛 MongoDB Pagination: Optimized memory consumption through improved pagesize control
🐛 Table Pagination: Hidden pagination controls when no data is present
🐛 CSV Editor: Removed default placeholder test data
🐛 Database Metadata: Resolved cache issues preventing retrieval of latest user metadata
🐛 UI Consistency: Adjusted Advanced Settings button padding for better alignment
🐛 Dark Mode: Corrected color inconsistencies for preview and artifact buttons
🐛 Project Navigation: Fixed sorting issue in "5 Latest Projects" popover display

2.1.0¶

Features¶

✨ API: The Generate API now supports injecting dynamic properties at execution time, enabling more flexible and customizable data generation workflows via API calls.

Refactors¶

🔥 UX: In Cron Generate tab, the Generate button is now disabled when the Cron value is empty, preventing invalid submissions.
🔥 UX: In Task table, the View Log button is now enabled even when the Task status is Queued, allowing users to access logs earlier in the task lifecycle.
🔥 Global Project: For protected Global Project files, the Download button is disabled now.

Fixes¶

🐛 Global Project: Fixed an issue where xml parser not working properly for global project's files.

2.0.0¶

Breaking changes¶

⬆️ multiprocessing: The multiprocessing attribute has been replaced with a unified numProcess and mpPlatform configuration for improved clarity and flexibility in task execution.
⬆️ Condition Nodes: The <if> node has been replaced by a more versatile <condition> node structure, which now includes <if>, <if-else>, and <else> nodes. This change enables more complex conditional logic in data generation tasks.
⬆️ Project Access Token: Introduced a Project Access Token system, providing more secure and flexible API access.
⬆️ Operate: The operate task type has been completely refactored and decoupled from the generate task type. It now supports XPATH and JSON Path operations on XML and JSON structures, enabling advanced data manipulation capabilities.

Fixes¶

🐛 Git provider: Fixed an issue where the selected Git provider was not saved correctly.
🐛 Delete project: Deleting a project now cleans up all related resources.
🐛 System task: Fixed an issue where the system task table could not be searched by task ID.

Features¶

✨ Architecture: The DATAMIMIC platform now supports configuring multiple worker charts. Tasks can now run with multiple workers, improving performance and scalability.
✨ Generate: Added support for additional task types (standard, infinite, and timeout) with multi-worker capability.
✨ UI: The UI has been completely redesigned with a modern look.
✨ Performance: Internal file handling has been improved for enhanced performance and scalability.
✨ Demo store: Introduced tag-based filtering tabs and redesigned demo cards for better usability.
✨ Project view: Enabled filtering by tags and project types via new tabbed navigation.
✨ Notification: Launched a dedicated notification center with improved display logic.
✨ System maintenance: Introduced Backup & Restore functionality, allowing Admins to back up and restore DATAMIMIC instance data.
✨ System banner: Added banner configuration, enabling Admins to display system alerts to all instance users.
✨ Table data: Upgraded all tables with search, sorting, multi-filtering, and column visibility. This aims to improve performance with large data sets.
✨ Git Project: Git projects can now be synchronized with branches other than the default.
✨ System projects: Admin users can now perform bulk project deletions for more efficient management.
✨ Project settings: Added tag management, task type configuration, and log level controls.
✨ Project settings: Added a new button to copy the project ID.
✨ Create project files: Allows uploading multiple files at once.
✨ Template Bulk Upload: Users can now bulk upload JSON or XML templates and add prefixes or suffixes for greater flexibility.
✨ Operate on JSON and XML structures: Users can now perform operations (set, delete, etc.) via XPATH or JSON Path on XML and JSON structures.
✨ API: Extended support for API endpoint usage with Project Access Token.
✨ API: Introduced external_api_docs to provide a more comprehensive and user-friendly way to use the API for standard workflows.
✨ API: Added a new API endpoint to allow downloading all artifacts related to a task as a single zip file.

Refactors¶

🔥 Architecture: All containers have been refactored to be more modular and easier to maintain, reducing Docker container size and improving performance.
🔥 Cleanup system task: Moved the cleanup button into the system task view (Dashboard).
🔥 Reset project: Reset project logic has been separated for clearer context between resetting the entire project or by task.
🔥 UI: The System API button now directs users to external_api_docs by default. The advanced docs remain accessible but are no longer linked from the UI.
🔥 Documentation: Project access token usage has been updated to use the new external_api_docs endpoints and examples.

1.10.0¶

Fixes¶

🐛 SourceConstraints + random distribution (DAT-628): Fixed a defect that caused rows to be dropped when sourceConstraints were combined with distribution="random".
🐛 RemoveNoneOrEmptyElement on root level: When uploading to MongoDB (and other database exporters), the converter now removes root-level fields that resolve to None, empty strings, or empty sub-documents.

1.9.1¶

Fixes¶

🐛 Exporter: Update oracle db driver and refactor OracleDB engine
🐛 Exporter: Update mssql driver to pymssql

Features¶

✨ XML Model: Shorten_variable_script_path (this.var to access keys and variables on the same level)

Refactors¶

🔥 Generate: General context improvements
🔥 Exporter: Refactor KafkaImporterClient: Remove signal handling for timeouts and streamline message polling logic
🔥 Exporter: Extend error handling in ml train task

1.9.0¶

Fixes¶

🐛 Expression Evaluation: Handle colon characters correctly in Python expressions (e.g., for XML namespaces).
🐛 Generate: Improve error messages for undefined items and invalid data structures.
🐛 Generate: Fix handling of recursive and nested dictionaries during evaluation.
🐛 XML Load: Improve type detection and handling for OrderedDict and dict in XML input files.

Features¶

✨ API: Extend standard generate endpoint and artifact to be able to use with project access token.
✨ Generate: Support for dynamic sources in element-level generate tasks.
✨ Generate: Introduce new multiprocessing platform Ray for distributed remote execution.
✨ Generate: Replace multiprocessing attribute with unified numProcess and mpPlatform configuration.
✨ Constraints: Introduce three-layer constraint system with <sourceConstraints>, <mapping>, and <targetConstraints>.
✨ Script Attribute: Add support for directly accessing and transforming nested data structures in script elements.
✨ SQLAlchemy: Support for .env.property-based configuration for SQLAlchemy database connections.
✨ FileCacheRegistry: Introduce caching mechanism for efficient file processing and reuse.
✨ Domain Generators: Extend domain-specific generator library with advanced weighting-based models for Healthcare, Public Sector, and E-commerce datasets. Improves out-of-the-box realism for structured scenarios.

Preview¶

🚧 Operate Task: Add task type to perform post-processing directly on datasources.
🚧 ML-Train Task: Enable advanced enrichment of data from source datasets using a fully integrated ML pipeline. The <ml-train> task supports training tabular generators based on data samples (e.g. first 1,000 rows), leveraging Tabular ARGN to capture realistic distributions and inter-column relationships. Trained models are persisted and can be reused across <generate> tasks for statistically grounded synthetic data creation.
🚧 ML-Generator Reusability & Scalable Execution: Persisted models are referenced via source="..." in <generate> tasks. Batch-based generation using pageSize enables memory-efficient processing. <key> blocks allow real-time post-processing and formatting. Constraint rules (column, row, cross-record) are enforced during generation.

Refactors¶

🔥 Generate: Improve multiprocessing logic and simplify configuration by deprecating multiprocessing in favor of numProcess.
🔥 Data Pipeline: Refactor evaluation context for better reliability and performance during data generation.

Docs¶

📝 Add Advanced Guide to use Project Access Tokens programmatically (EN/DE)
📝 Add Domain Generator Guide covering configuration and tuning of Healthcare, Public Sector, and E-commerce weighting models

1.8.5¶

Fixes¶

🐛 Simple mode: Improve selection of target properties
🐛 Editor: Fix an issue where downloaded JSON was incomplete

1.8.4¶

Fixes¶

🐛 Simple mode: Fix proper selecting and handling of adding properties
🐛 Simple mode: Fix an issue where the user cannot add a new property in the simple mode when no value field exists.
🐛 Autocomplete: Improve autocompletion for condition and generator elements.
🐛 Editor: Small ui fix with duplicated arrow for collapse/expand in the editor.

Refactors¶

🔥 Simple Mode: Improve the Simple Mode rendering.
🔥 Data Table: Improve handling and sorting of data tables.

1.8.3¶

Fixes¶

🐛 Job API: Fix an issue where job cannot be filter correctly for advance search support.
🐛 Git: Fix an issue where git push button is not visible in Task View.

Features¶

✨ System Task: Add ability to copy task id and perform search for Admin in System Task Table.

Docs¶

📝 Add initial docs in german language.
📝 Rework and split reference model data definition

1.8.2¶

Fixes¶

🐛 Global Projects: Fix global projects and environments not working on generate with infinite mode.

1.8.1¶

Fixes¶

🐛 Kafka-Exporter: Fix handling of acknowledgment (ack) attribute by properly processing special characters and edge cases in configurations.
🐛 Autocomplete: Enhance Autocompletion by adding comprehensive attribute listings for Kafka exporters and importers

1.8.0¶

Fixes¶

🐛 Logging: Wrong log level for the generate, condition, and nestedKey elements
🐛 Logview: Fix flickering when log updates occur
🐛 Preview: Fix incorrect preview limitation handling and set an expiration time for previews (4 days)
🐛 NestedKey Input Source: Fix dictionary data generation returning key lists instead of maintaining dictionary structure (issue not present with distribution="ordered")
🐛 Table Editor: Fix reloading of table when changing between browser tabs and different windows

Features¶

✨ Helm Charts: Add instance management scripts (rdd-miniomgr.py and rdd-redismgr.py) for worker pods
✨ Helm Charts: Add OpenShift compatibility with emptyDir volume for ephemeral storage
✨ Helm Charts: Enhance worker health checks
✨ Logging: Add project log level configuration
✨ Logging: Add multiprocess logging handler for sub worker process logs
✨ DB Source: Add chunking support to improve performance and reduce memory usage

Refactors¶

🔥 Helm Charts: Remove deprecated WORKERS_PER_CORE and WORKERS variables from worker configmap
🔥 Artifacts: Rework artifact handling for better performance and memory usage
🔥 File Wizard: Improve usability and user experience
🔥 UI: Enhance handling of non-existent projects
🔥 Logging: Refactor Loghandler for multiprocess support and efficiency

Docs¶

📝 Source Scripting: Extend documentation for source scripting features
📝 Variable Substitution: Add comprehensive guide on variable substitution
📝 MongoDB Operations: Extend Document MongoDB-specific features and operations

1.7.2¶

Fixes¶

Kafka SSL: Fix an issue where the Kafka SSL configuration was not working correctly.
Logging: Fix an issue where debug logs were set to INFO level. *

Refactors¶

🔥 Logging: Improve logging for Cascaded Generate Tasks.

1.7.1¶

Fixes¶

Exporter System: Clean up behavior in cascaded generate tasks and naming logic for single processing.

1.7.0¶

Fixes¶

Editor: Fix an issue where files are not decoded properly when the project is reset from the Editor.
Editor: Fix an issue where JSON related scripts cannot be displayed properly with the new base64 encoding protocol.
Job: Fix an issue where the Job cannot be run.
Global Projects: Fix an issue where global project files are not being handled correctly.

Features¶

✨ Database View: Introducing a new Database view that allows customization of data generation and modeling based on SQL databases. Read More
✨ API: Allow the Admin user to always have access to any task log.
✨ API: The log is now base64 encoded when sent from the API.
✨ UI Log: Add an option to enable auto-refresh of the log every 5 seconds.
✨ MongoDB: Extend the authentication mechanism options to include PLAIN, which is also the default value when setting up.
✨ Project Setting UI: Introducing a new design for the project settings view for a more comprehensive workflow.
✨ Exporter: Add a new exporter OpenSearchBulk for exporting entities into OpenSearch bulk format. Read More
✨ Generate: The default log level for generate operations is now set to INFO. An option has been added to run generate in Debug mode for more detailed logging and troubleshooting.

Refactors¶

🔥 Email: Reduce overhead for our email template service.
🔥 Exporter-System: Refactor the exporter system to be more flexible, allowing users to configure the exporters, making them more efficient in multiprocessing, and enabling chunking. Read More
🔥 UI: Move system configuration settings to the new User Dashboard.
🔥 Job UI: Only display Last Run time in Job View when the job has been executed at least once.
🔥 Editor UI: Refactor the Create files/models flow to be more synchronized with the environment creation flow.
🔥 Editor Advanced Mode: Combine the Format Files and Format Variables buttons into a single button.
🔥 Editor: Newly created files will now be selected by default.
🔥 Editor: Update the file tree to be more user-friendly.
🔥 Editor: Move the Navigation bar to the top of the editor to enhance the user experience and maximize screen space.
🔥 Artifact: Update the artifact download button's color background.

1.6.0¶

Fixes¶

Editor: Fix Error when project time stamp is updated even when there are no changes to the files.
condition element: Fix error generate element inside of generate|condition not consuming.
UI Change behavior where local time stamp is show with 24:00 instead of 00:00AM.
Editor Fix an error where user cannot rename model file in editor.
Global Projects: Fix priority of duplicate entity files when generating with global projects.
Editor: Truncate long file name in File Tree, allow user to re size file tree horizontally.
Autocomplete & Linting: Added missing element Null Quota
UI: Fixed an issue with not supported Filetypes in the File Creation Wizard.
UI: Fixed an issue with the File Creation Wizard creating JSON templates
Script: Fixed and issue with evaluating property files correctly into the context.

Features¶

nestedKey element: script attribute can be combined with with cyclic.
IncrementGenerator: add a configuration option is step, flexibility in terms of controlling how much to increment the value each time.
StringGenerator: generates strings based on a character set, prefix, suffix, and length, while ensuring all generated strings are unique.
Admin Users Projects: Enable Admin users to perform bulk deletion of projects directly from the Project Systems View table. This includes enhanced search and filtering by user’s email to manage multiple projects efficiently..
API Allow Admin user to always have access to any projects and files.

Refactors¶

🔥 Environment configuration: Refactor the environment configuration (especially kafka) to be more user-friendly.
🔥 Simple Mode: Refactor the Simple Mode to be more user-friendly and intuitive.
🔥 UI colors: Refactor the UI colors to have more contrast and be more user-friendly.

1.5.2¶

Fixes¶

Rename Files: Fix the issue where renaming files in the editor was not working correctly.

1.5.1¶

Fixes¶

Artifact Exporter: Fix the path for the .tempfiles before they get pushed to the object-store.
Demo: Fixed the naming of the sql script file in the selector demo.

1.5.0¶

Fixes¶

Artifact Download: Fix an issue where the artifact download was not working correctly and very slow with large files.
Autocomplete: Fix an issue where the autocomplete was not working correctly for script attributes.
Datetime precision: Fix default precision for datetime generation. The default precision %f is now set to milliseconds or %6f, when not other specified.
File Type: Remove not supported file types from the file creation Wizard.
Demo: Fix a small issue related to condition demo.

Features¶

✨ Model Autocompletion: Extended DATAMIMIC model autocompletion and linting for NestedKey and array nodes. Enriched the documentation and added more examples showcasing JSON and XML usage.
✨ JSON Header exporter: Implemented a new JSONHeader exporter for exporting entities into separate JSON files with a header, defined by a key with the name $$header$$.
✨ Converter: Extended a RemoveNoneOrEmpty converter to remove None or empty values from the data. This converter is applicable in generate, nestedKey and list nodes.
✨ JSON Pretty: We added a pretty formatting to our JSON exporter to make them easier readable.
✨ Global environments: Extend global project ability, use can now have the global project's system environment include when add global project.

Refactors¶

🔥 Default Value: Refactored the defaultValue function to be consistent in key and nested key nodes.
🔥 UI Enhancements: Move system configuration to the new configuration view.

Docs¶

📝 Configuration View: Updated the documentation for the new configuration view, explaining all available options.
📝 Global Project Section: Updated documentation for global project functionality with recent changes.
📝 Target: Extended the documentation for the JSONHeader exporter.

1.4.0¶

Fixes¶

File Creation and Update: Resolved an issue where file creation and update via drop zone was broken.
JSON Exporter: Fixed an issue where the JSON Exporter produced incorrect output when using 10 pages and 0 entities with a specific configuration.
Autocompletion: Addressed a missing pattern in the autocompletion model.
Project Menu: Fixed the issue where the project quick access menu was static.

Features¶

✨ Model Autocompletion: Extended DATAMIMIC model autocompletion and linting for NestedKey and array nodes. Enriched the documentation and added more examples showcasing JSON and XML usage.
✨ Configurable Signup Options: Implemented configurable signup options for DATAMIMIC, allowing admins to configure initial user roles and disable self-signups.
✨ Standalone Library: Integrated the DATAMIMIC library as a standalone module for realistic data generation in platform tests.
✨ String Attribute for complex Strings: Added a new string attribute to the variable and key nodes. This attribute allows users to define complex strings where variables are dynamically evaluated and inserted using configurable delimiters (e.g., __variable__)
✨ Global Project: Add ability that allow admin user to switch a global project back to standard project.

Refactors¶

🔥 Project View: Refactored the project view to enhance usability and design.
🔥 UI Enhancements: Improved the UI for the editor sidebar tools to streamline the user experience.

Docs¶

📝 Configuration View: Updated the documentation for the new configuration view, explaining all available options.
📝 Global Project Section: Updated documentation for global project functionality with recent changes.
📝 Model configuration and definition: Extended the documentation for model configuration data-definition to include more example use cases and examples.

1.3.0¶

Breaking Changes¶

⬆️ Condition Nodes: The condition nodes have been significantly refactored to support more complex logic. The previous if node has been replaced by a more versatile <condition> node structure, which includes <if>, <if-else>, and <else> nodes. The <if> and <if-else> nodes define the primary conditions, while the <else> node specifies the alternative path.

Fixes¶

Resolved an issue with the lazy iterator for variables, which caused cyclic references.
Fixed an error when reading CSV files from givennamegenerator and familynamegenerator.
Addressed a problem with CSV export when using Faker data.
Fixed an issue where accessing data with numeric keys in base properties caused errors.

Features¶

✨ Global Project: Introduced the ability to create a global project that can be added as a dependency to user projects, allowing for shared data, such as Python libraries, custom components, and data mappings.
✨ Improved Dynamic Selectors: Enhanced the efficiency of using variables in all types of selectors, including selector and iterationSelector.
✨ Condition Nodes: Redesigned condition nodes to support more complex conditional logic.
✨ Default Value: Added the defaultValue attribute to keys, allowing a default value to be set when the key's value is null or not specified.
✨ Complex Conditions: Enabled more advanced conditional logic by introducing <if-else> and <else> nodes, which can be used to wrap keys.
✨ More Demos: Added two new demos, Condition Demo and Selector Demo, showcasing the new condition handling in keys and nodes, as well as dynamic selectors.
✨ Datetime Epoch: Added support for handling epoch transformations at the level of seconds, milliseconds, microseconds, and nanoseconds using outDateFormat and inDateFormat.
✨ Datetime Precision: Introduced the ability to set the precision of datetime transformations to seconds, milliseconds, microseconds, and nanoseconds via outDateFormat and inDateFormat.
✨ Data Distribution: Added new options for data distribution from a source, allowing for ordered or random distribution through the distribution attribute.
✨ Simple Mode: Reworked and improved the UI for Simple Mode, making it more user-friendly.
✨ Data Access: Added the ability to access data from sibling <generate> nodes by loading them into a variable using the source attribute.

Refactors¶

🔥 Project View: Refactored the project view to enhance usability and design.
🔥 Demo Store: Improved the demo store’s usability and design through a comprehensive refactor.
🔥 Code Cleanup: Performed general code cleanup, including refactoring redundant code and improving code readability.

Docs¶

📝 Updated the documentation to include details on updating the person entity information in generators.
📝 Extended documentation for the <target> element.
📝 Enhanced documentation for UI project general settings.
📝 Expanded documentation on data distribution methods.
📝 Added comprehensive documentation covering the new features and changes introduced in this release.

1.2.1¶

Documentation¶

Project Settings: Update documentation for general project settings.

Refactors¶

Preview UI: Update color for Tables, Json, Log View in Dark mode to keep it consistent with background color.

Fixes¶

Environment file parsing: Fixed an issue where the environment file was not parsed correctly.
Kafka SSL: Add additional Kafka SSL configuration to change the SSL Truststore root certificate alias.
MongoDB: Add additional MongoDB configuration to configure the authentication mechanism and the authentication source.
MongoDB: Fixed an issue where the MongoDB queries where not evaluated dynamically.

1.2.0¶

Redux Management: Remove Redux packages.
Enhancement: Cleanup temporary task directory in deployed environments.
Task/System Task View: Show Started time as user local date time.
MongoDB username, password: Handle special character in MongodDB username and password.
Playwright: Add User Interface Test to ensure users having best experience in DATAMIMIC.

Fixes¶

Error Handling: Enhanced error handling across all Rust components for improved stability.
Redis Retrieval: Resolved issues with retrieving results from Redis for media previews.
MongoDB Authentication: Fixed MongoDB authentication issues related to special characters in usernames.
Task Directory Cleanup: Implemented logic to clean up task directories when not running locally.
KeyTool: Added missing KeyTool to worker containers to support proper authentication handling for Kafka environments.
Project Download: Fixed missing project name in the download file archive.
Codemirror Linter: Optimized linter to check wrong child for echo cases.
Preview View: Optimized the height of table.
Simple Mode: Optimized the dragging and dropping feature causes element removal or strange element combination.
Simple Mode: Optimized wrong child elements in Simple Mode which are imported from Advanced Mode.

Upgrades¶

State Management: Transitioned from Redux to React Query for more efficient data fetching and state management.
Demo Enhancements: Improved demo icons and error handling mechanisms by implementing a default icon for missing demo icons and enhancing error handling for faulty Demo TOML files.
React 18 Migration: Migrate React from version 17 to 18.

Features¶

✨ Object-Store: Implemented support for object storage environments, enabling reading/importing and writing/exporting data from/to object storages like S3, Azure Blob, and Minio.
✨ Exporter: Implement JSONSingle Exporter for exporting entities into separate JSON files.
✨ Pattern: Improved error handling for patterns and enhanced migration guidance.
✨ Custom Components: Added the ability to incorporate custom Python components such as generators and converters.
✨ Kafka: Implemented support for more advanced Kafka configurations.
✨ Generator: Developed a TableSequenceGenerator for generating unique sequences in tables, useful when other services access the same table or when tables have unique constraints with pre-existing data.
✨ Demo: Created a new demo custom component for Python to showcase features and provide a starting point for users.
✨ Demo: Introduced a new demo for complex database mapping scenarios using custom converters and database system connections.
✨ Simple Mode: Styled search input as sticky element in dialogs' combobox.
✨ Simple Mode: Add multi select feature when adding or editing target property.

Refactors¶

🔥 Logging: Refactored console logging to use console.debug for better log verbosity control.
🔥 Code Cleanup: Performed general code cleanup, including refactoring redundant code and improving code readability.
Playwright: Add User Interface Test to ensure users having best experience in DATAMIMIC.

Docs¶

📝 Migration Guide: Extended guidance on migration warnings to assist users during transitions.
📝 Feature Guides: Added new feature guides for Object-Store, Custom Components, and Kafka configurations.
📝 Login: Removed recovery account troubleshooting

1.1.0¶

Fixes¶

Demo Store: Added a process to ensure the demo store and template are always up to date during the startup process.
Project: Displayed the correct local date and time when creating a project.

Upgrades¶

Access Token: Increased the maximum token expiry from 30 days to 365 days.
Simple Mode: Retrieve data from IndexedDB instead of directly from the API.

Features¶

✨ Processing: Enhanced processing with the introduction of pageSize and defaultPageSize, and improved process logging.
✨ Notification: Improved handling for independently removing notification items.
✨ Simple Mode: Added Enable/Disable Element and Drag-and-Drop features.
✨ Kafka: Implemented SASL_SSL properties.
✨ Project View: Add new UI for project base on card design.
✨ Get Project List API: Add update and create time for project read.
✨ Demo Store: Redesign demo store cards to keep the style consistent.

1.0.1¶

Fixes¶

Environment SQLite: Changed path of temporary SQLite during task execution to support OpenShift constraints
Demo Store: Template update options are no longer shown to users.
UI: Fix color of Reset Project Dialog in Dark Mode.
Reset Project From Task: Reset project from Task won't show the correct files state.

Features¶

✨ Core: Enhanced handling of task memory limits.
✨ System Projects: Users can now be assigned to non-owned projects.

1.0.0¶

Features¶

✨ Initial release: DATAMIMIC: AI-Driven, Model-Based Data Generation with Advanced JSON/XML Handling for Efficient, Compliant Development.
✨ Introduce new auto model creation for JSON, enabling users to generate complex, nested JSON structures effortlessly.
✨ Introduce new auto model creation for XML, simplifying the creation of intricate XML data models.
✨ Embrace a Model-Driven Approach for simplified maintenance and abstract data processing, enhancing adaptability and consistency in test data generation.
✨ Implement Abstract Data Transformation to transform various inputs into flexible entities for diverse outputs like SQL, XML, JSON, CSV, and more.
✨ Advanced JSON Handling for expert manipulation of nested JSON structures, supporting hierarchical data modeling and deep nesting capabilities.
✨ Introduce Data Anonymization and Pseudonymization features for field-level obfuscation, ensuring GDPR compliance and data privacy.
✨ AI-Enhanced Data Generation employing advanced algorithms for realistic and diverse data sets.
✨ Achieve High Performance with optimizations for handling large data volumes, suitable for processing millions of transactions in real time.
✨ Provide an Intuitive User Experience with a user-friendly interface and clear documentation.
✨ Offer Flexible and Scalable solutions for projects of various sizes and complexities.
✨ Extend Functionality with an Extensive Function Library for realistic data creation, supporting custom code extensions.
✨ Ensure Seamless Integration with development, testing environments, and CI/CD pipelines.

Docs¶

📝 Add initial docs in English language, covering the growing spectrum of DATAMIMIC features and user guidance.

Translations¶

🌐 Add German translation for docs/de/docs/index.md, docs/de/docs/features.md.
🌐 Add Vietnamese translation for docs/de/docs/index.md, docs/de/docs/features.md.

Template for release note sections¶

### Breaking changes

* ⬆️ n/a.

### Fixes

* n/a.

## Upgrades

* n/a.

### Features

* ✨ Introduce new auto model creation for JSON.
* ✨ Introduce new auto model creation for XML.

### Refactors

* 🔥 n/a.

### Docs

* 📝 Add initial docs in english language.

### Translations

* 🌐 Add german translation for `docs/de/docs/features.md`.