Release notes

Latest Changes¶

Features¶

✨ API: Extend standard generate endpoint and artifact to be able to use with project access token.

Docs¶

📝 Add Advanced Guide to use Project Access Tokens programmatically (EN/DE)

1.8.5¶

Fixes¶

🐛 Simple mode: Improve selection of target properties
🐛 Editor: Fix an issue where downloaded JSON was incomplete

1.8.4¶

Fixes¶

🐛 Simple mode: Fix proper selecting and handling of adding properties
🐛 Simple mode: Fix an issue where the user cannot add a new property in the simple mode when no value field exists.
🐛 Autocomplete: Improve autocompletion for condition and generator elements.
🐛 Editor: Small ui fix with duplicated arrow for collapse/expand in the editor.

Refactors¶

🔥 Simple Mode: Improve the Simple Mode rendering.
🔥 Data Table: Improve handling and sorting of data tables.

1.8.3¶

Fixes¶

🐛 Job API: Fix an issue where job cannot be filter correctly for advance search support.
🐛 Git: Fix an issue where git push button is not visible in Task View.

Features¶

✨ System Task: Add ability to copy task id and perform search for Admin in System Task Table.

Docs¶

📝 Add initial docs in german language.
📝 Rework and split reference model data definition

1.8.2¶

Fixes¶

🐛 Global Projects: Fix global projects and environments not working on generate with infinite mode.

1.8.1¶

Fixes¶

🐛 Kafka-Exporter: Fix handling of acknowledgment (ack) attribute by properly processing special characters and edge cases in configurations.
🐛 Autocomplete: Enhance Autocompletion by adding comprehensive attribute listings for Kafka exporters and importers

1.8.0¶

Fixes¶

🐛 Logging: Wrong log level for the generate, condition, and nestedKey elements
🐛 Logview: Fix flickering when log updates occur
🐛 Preview: Fix incorrect preview limitation handling and set an expiration time for previews (4 days)
🐛 NestedKey Input Source: Fix dictionary data generation returning key lists instead of maintaining dictionary structure (issue not present with distribution="ordered")
🐛 Table Editor: Fix reloading of table when changing between browser tabs and different windows

Features¶

✨ Helm Charts: Add instance management scripts (rdd-miniomgr.py and rdd-redismgr.py) for worker pods
✨ Helm Charts: Add OpenShift compatibility with emptyDir volume for ephemeral storage
✨ Helm Charts: Enhance worker health checks
✨ Logging: Add project log level configuration
✨ Logging: Add multiprocess logging handler for sub worker process logs
✨ DB Source: Add chunking support to improve performance and reduce memory usage

Refactors¶

🔥 Helm Charts: Remove deprecated WORKERS_PER_CORE and WORKERS variables from worker configmap
🔥 Artifacts: Rework artifact handling for better performance and memory usage
🔥 File Wizard: Improve usability and user experience
🔥 UI: Enhance handling of non-existent projects
🔥 Logging: Refactor Loghandler for multiprocess support and efficiency

Docs¶

📝 Source Scripting: Extend documentation for source scripting features
📝 Variable Substitution: Add comprehensive guide on variable substitution
📝 MongoDB Operations: Extend Document MongoDB-specific features and operations

1.7.2¶

Fixes¶

Kafka SSL: Fix an issue where the Kafka SSL configuration was not working correctly.
Logging: Fix an issue where debug logs were set to INFO level. *

Refactors¶

🔥 Logging: Improve logging for Cascaded Generate Tasks.

1.7.1¶

Fixes¶

Exporter System: Clean up behavior in cascaded generate tasks and naming logic for single processing.

1.7.0¶

Fixes¶

Editor: Fix an issue where files are not decoded properly when the project is reset from the Editor.
Editor: Fix an issue where JSON related scripts cannot be displayed properly with the new base64 encoding protocol.
Job: Fix an issue where the Job cannot be run.
Global Projects: Fix an issue where global project files are not being handled correctly.

Features¶

✨ Database View: Introducing a new Database view that allows customization of data generation and modeling based on SQL databases. Read More
✨ API: Allow the Admin user to always have access to any task log.
✨ API: The log is now base64 encoded when sent from the API.
✨ UI Log: Add an option to enable auto-refresh of the log every 5 seconds.
✨ MongoDB: Extend the authentication mechanism options to include PLAIN, which is also the default value when setting up.
✨ Project Setting UI: Introducing a new design for the project settings view for a more comprehensive workflow.
✨ Exporter: Add a new exporter OpenSearchBulk for exporting entities into OpenSearch bulk format. Read More
✨ Generate: The default log level for generate operations is now set to INFO. An option has been added to run generate in Debug mode for more detailed logging and troubleshooting.

Refactors¶

🔥 Email: Reduce overhead for our email template service.
🔥 Exporter-System: Refactor the exporter system to be more flexible, allowing users to configure the exporters, making them more efficient in multiprocessing, and enabling chunking. Read More
🔥 UI: Move system configuration settings to the new User Dashboard.
🔥 Job UI: Only display Last Run time in Job View when the job has been executed at least once.
🔥 Editor UI: Refactor the Create files/models flow to be more synchronized with the environment creation flow.
🔥 Editor Advanced Mode: Combine the Format Files and Format Variables buttons into a single button.
🔥 Editor: Newly created files will now be selected by default.
🔥 Editor: Update the file tree to be more user-friendly.
🔥 Editor: Move the Navigation bar to the top of the editor to enhance the user experience and maximize screen space.
🔥 Artifact: Update the artifact download button's color background.

1.6.0¶

Fixes¶

Editor: Fix Error when project time stamp is updated even when there are no changes to the files.
condition element: Fix error generate element inside of generate|condition not consuming.
UI Change behavior where local time stamp is show with 24:00 instead of 00:00AM.
Editor Fix an error where user cannot rename model file in editor.
Global Projects: Fix priority of duplicate entity files when generating with global projects.
Editor: Truncate long file name in File Tree, allow user to re size file tree horizontally.
Autocomplete & Linting: Added missing element Null Quota
UI: Fixed an issue with not supported Filetypes in the File Creation Wizard.
UI: Fixed an issue with the File Creation Wizard creating JSON templates
Script: Fixed and issue with evaluating property files correctly into the context.

Features¶

nestedKey element: script attribute can be combined with with cyclic.
IncrementGenerator: add a configuration option is step, flexibility in terms of controlling how much to increment the value each time.
StringGenerator: generates strings based on a character set, prefix, suffix, and length, while ensuring all generated strings are unique.
Admin Users Projects: Enable Admin users to perform bulk deletion of projects directly from the Project Systems View table. This includes enhanced search and filtering by user’s email to manage multiple projects efficiently..
API Allow Admin user to always have access to any projects and files.

Refactors¶

🔥 Environment configuration: Refactor the environment configuration (especially kafka) to be more user-friendly.
🔥 Simple Mode: Refactor the Simple Mode to be more user-friendly and intuitive.
🔥 UI colors: Refactor the UI colors to have more contrast and be more user-friendly.

1.5.2¶

Fixes¶

Rename Files: Fix the issue where renaming files in the editor was not working correctly.

1.5.1¶

Fixes¶

Artifact Exporter: Fix the path for the .tempfiles before they get pushed to the object-store.
Demo: Fixed the naming of the sql script file in the selector demo.

1.5.0¶

Fixes¶

Artifact Download: Fix an issue where the artifact download was not working correctly and very slow with large files.
Autocomplete: Fix an issue where the autocomplete was not working correctly for script attributes.
Datetime precision: Fix default precision for datetime generation. The default precision %f is now set to milliseconds or %6f, when not other specified.
File Type: Remove not supported file types from the file creation Wizard.
Demo: Fix a small issue related to condition demo.

Features¶

✨ Model Autocompletion: Extended DATAMIMIC model autocompletion and linting for NestedKey and array nodes. Enriched the documentation and added more examples showcasing JSON and XML usage.
✨ JSON Header exporter: Implemented a new JSONHeader exporter for exporting entities into separate JSON files with a header, defined by a key with the name $$header$$.
✨ Converter: Extended a RemoveNoneOrEmpty converter to remove None or empty values from the data. This converter is applicable in generate, nestedKey and list nodes.
✨ JSON Pretty: We added a pretty formatting to our JSON exporter to make them easier readable.
✨ Global environments: Extend global project ability, use can now have the global project's system environment include when add global project.

Refactors¶

🔥 Default Value: Refactored the defaultValue function to be consistent in key and nested key nodes.
🔥 UI Enhancements: Move system configuration to the new configuration view.

Docs¶

📝 Configuration View: Updated the documentation for the new configuration view, explaining all available options.
📝 Global Project Section: Updated documentation for global project functionality with recent changes.
📝 Target: Extended the documentation for the JSONHeader exporter.

1.4.0¶

Fixes¶

File Creation and Update: Resolved an issue where file creation and update via drop zone was broken.
JSON Exporter: Fixed an issue where the JSON Exporter produced incorrect output when using 10 pages and 0 entities with a specific configuration.
Autocompletion: Addressed a missing pattern in the autocompletion model.
Project Menu: Fixed the issue where the project quick access menu was static.

Features¶

✨ Model Autocompletion: Extended DATAMIMIC model autocompletion and linting for NestedKey and array nodes. Enriched the documentation and added more examples showcasing JSON and XML usage.
✨ Configurable Signup Options: Implemented configurable signup options for DATAMIMIC, allowing admins to configure initial user roles and disable self-signups.
✨ Standalone Library: Integrated the DATAMIMIC library as a standalone module for realistic data generation in platform tests.
✨ String Attribute for complex Strings: Added a new string attribute to the variable and key nodes. This attribute allows users to define complex strings where variables are dynamically evaluated and inserted using configurable delimiters (e.g., __variable__)
✨ Global Project: Add ability that allow admin user to switch a global project back to standard project.

Refactors¶

🔥 Project View: Refactored the project view to enhance usability and design.
🔥 UI Enhancements: Improved the UI for the editor sidebar tools to streamline the user experience.

Docs¶

📝 Configuration View: Updated the documentation for the new configuration view, explaining all available options.
📝 Global Project Section: Updated documentation for global project functionality with recent changes.
📝 Model configuration and definition: Extended the documentation for model configuration data-definition to include more example use cases and examples.

1.3.0¶

Breaking Changes¶

⬆️ Condition Nodes: The condition nodes have been significantly refactored to support more complex logic. The previous if node has been replaced by a more versatile <condition> node structure, which includes <if>, <if-else>, and <else> nodes. The <if> and <if-else> nodes define the primary conditions, while the <else> node specifies the alternative path.

Fixes¶

Resolved an issue with the lazy iterator for variables, which caused cyclic references.
Fixed an error when reading CSV files from givennamegenerator and familynamegenerator.
Addressed a problem with CSV export when using Faker data.
Fixed an issue where accessing data with numeric keys in base properties caused errors.

Features¶

✨ Global Project: Introduced the ability to create a global project that can be added as a dependency to user projects, allowing for shared data, such as Python libraries, custom components, and data mappings.
✨ Improved Dynamic Selectors: Enhanced the efficiency of using variables in all types of selectors, including selector and iterationSelector.
✨ Condition Nodes: Redesigned condition nodes to support more complex conditional logic.
✨ Default Value: Added the defaultValue attribute to keys, allowing a default value to be set when the key's value is null or not specified.
✨ Complex Conditions: Enabled more advanced conditional logic by introducing <if-else> and <else> nodes, which can be used to wrap keys.
✨ More Demos: Added two new demos, Condition Demo and Selector Demo, showcasing the new condition handling in keys and nodes, as well as dynamic selectors.
✨ Datetime Epoch: Added support for handling epoch transformations at the level of seconds, milliseconds, microseconds, and nanoseconds using outDateFormat and inDateFormat.
✨ Datetime Precision: Introduced the ability to set the precision of datetime transformations to seconds, milliseconds, microseconds, and nanoseconds via outDateFormat and inDateFormat.
✨ Data Distribution: Added new options for data distribution from a source, allowing for ordered or random distribution through the distribution attribute.
✨ Simple Mode: Reworked and improved the UI for Simple Mode, making it more user-friendly.
✨ Data Access: Added the ability to access data from sibling <generate> nodes by loading them into a variable using the source attribute.

Refactors¶

🔥 Project View: Refactored the project view to enhance usability and design.
🔥 Demo Store: Improved the demo store’s usability and design through a comprehensive refactor.
🔥 Code Cleanup: Performed general code cleanup, including refactoring redundant code and improving code readability.

Docs¶

📝 Updated the documentation to include details on updating the person entity information in generators.
📝 Extended documentation for the <target> element.
📝 Enhanced documentation for UI project general settings.
📝 Expanded documentation on data distribution methods.
📝 Added comprehensive documentation covering the new features and changes introduced in this release.

1.2.1¶

Documentation¶

Project Settings: Update documentation for general project settings.

Refactors¶

Preview UI: Update color for Tables, Json, Log View in Dark mode to keep it consistent with background color.

Fixes¶

Environment file parsing: Fixed an issue where the environment file was not parsed correctly.
Kafka SSL: Add additional Kafka SSL configuration to change the SSL Truststore root certificate alias.
MongoDB: Add additional MongoDB configuration to configure the authentication mechanism and the authentication source.
MongoDB: Fixed an issue where the MongoDB queries where not evaluated dynamically.

1.2.0¶

Redux Management: Remove Redux packages.
Enhancement: Cleanup temporary task directory in deployed environments.
Task/System Task View: Show Started time as user local date time.
MongoDB username, password: Handle special character in MongodDB username and password.
Playwright: Add User Interface Test to ensure users having best experience in DATAMIMIC.

Fixes¶

Error Handling: Enhanced error handling across all Rust components for improved stability.
Redis Retrieval: Resolved issues with retrieving results from Redis for media previews.
MongoDB Authentication: Fixed MongoDB authentication issues related to special characters in usernames.
Task Directory Cleanup: Implemented logic to clean up task directories when not running locally.
KeyTool: Added missing KeyTool to worker containers to support proper authentication handling for Kafka environments.
Project Download: Fixed missing project name in the download file archive.
Codemirror Linter: Optimized linter to check wrong child for echo cases.
Preview View: Optimized the height of table.
Simple Mode: Optimized the dragging and dropping feature causes element removal or strange element combination.
Simple Mode: Optimized wrong child elements in Simple Mode which are imported from Advanced Mode.

Upgrades¶

State Management: Transitioned from Redux to React Query for more efficient data fetching and state management.
Demo Enhancements: Improved demo icons and error handling mechanisms by implementing a default icon for missing demo icons and enhancing error handling for faulty Demo TOML files.
React 18 Migration: Migrate React from version 17 to 18.

Features¶

✨ Object-Store: Implemented support for object storage environments, enabling reading/importing and writing/exporting data from/to object storages like S3, Azure Blob, and Minio.
✨ Exporter: Implement JSONSingle Exporter for exporting entities into separate JSON files.
✨ Pattern: Improved error handling for patterns and enhanced migration guidance.
✨ Custom Components: Added the ability to incorporate custom Python components such as generators and converters.
✨ Kafka: Implemented support for more advanced Kafka configurations.
✨ Generator: Developed a TableSequenceGenerator for generating unique sequences in tables, useful when other services access the same table or when tables have unique constraints with pre-existing data.
✨ Demo: Created a new demo custom component for Python to showcase features and provide a starting point for users.
✨ Demo: Introduced a new demo for complex database mapping scenarios using custom converters and database system connections.
✨ Simple Mode: Styled search input as sticky element in dialogs' combobox.
✨ Simple Mode: Add multi select feature when adding or editing target property.

Refactors¶

🔥 Logging: Refactored console logging to use console.debug for better log verbosity control.
🔥 Code Cleanup: Performed general code cleanup, including refactoring redundant code and improving code readability.
Playwright: Add User Interface Test to ensure users having best experience in DATAMIMIC.

Docs¶

📝 Migration Guide: Extended guidance on migration warnings to assist users during transitions.
📝 Feature Guides: Added new feature guides for Object-Store, Custom Components, and Kafka configurations.
📝 Login: Removed recovery account troubleshooting

1.1.0¶

Fixes¶

Demo Store: Added a process to ensure the demo store and template are always up to date during the startup process.
Project: Displayed the correct local date and time when creating a project.

Upgrades¶

Access Token: Increased the maximum token expiry from 30 days to 365 days.
Simple Mode: Retrieve data from IndexedDB instead of directly from the API.

Features¶

✨ Processing: Enhanced processing with the introduction of pageSize and defaultPageSize, and improved process logging.
✨ Notification: Improved handling for independently removing notification items.
✨ Simple Mode: Added Enable/Disable Element and Drag-and-Drop features.
✨ Kafka: Implemented SASL_SSL properties.
✨ Project View: Add new UI for project base on card design.
✨ Get Project List API: Add update and create time for project read.
✨ Demo Store: Redesign demo store cards to keep the style consistent.

1.0.1¶

Fixes¶

Environment SQLite: Changed path of temporary SQLite during task execution to support OpenShift constraints
Demo Store: Template update options are no longer shown to users.
UI: Fix color of Reset Project Dialog in Dark Mode.
Reset Project From Task: Reset project from Task won't show the correct files state.

Features¶

✨ Core: Enhanced handling of task memory limits.
✨ System Projects: Users can now be assigned to non-owned projects.

1.0.0¶

Features¶

✨ Initial release: DATAMIMIC: AI-Driven, Model-Based Data Generation with Advanced JSON/XML Handling for Efficient, Compliant Development.
✨ Introduce new auto model creation for JSON, enabling users to generate complex, nested JSON structures effortlessly.
✨ Introduce new auto model creation for XML, simplifying the creation of intricate XML data models.
✨ Embrace a Model-Driven Approach for simplified maintenance and abstract data processing, enhancing adaptability and consistency in test data generation.
✨ Implement Abstract Data Transformation to transform various inputs into flexible entities for diverse outputs like SQL, XML, JSON, CSV, and more.
✨ Advanced JSON Handling for expert manipulation of nested JSON structures, supporting hierarchical data modeling and deep nesting capabilities.
✨ Introduce Data Anonymization and Pseudonymization features for field-level obfuscation, ensuring GDPR compliance and data privacy.
✨ AI-Enhanced Data Generation employing advanced algorithms for realistic and diverse data sets.
✨ Achieve High Performance with optimizations for handling large data volumes, suitable for processing millions of transactions in real time.
✨ Provide an Intuitive User Experience with a user-friendly interface and clear documentation.
✨ Offer Flexible and Scalable solutions for projects of various sizes and complexities.
✨ Extend Functionality with an Extensive Function Library for realistic data creation, supporting custom code extensions.
✨ Ensure Seamless Integration with development, testing environments, and CI/CD pipelines.

Docs¶

📝 Add initial docs in English language, covering the growing spectrum of DATAMIMIC features and user guidance.

Translations¶

🌐 Add German translation for docs/de/docs/index.md, docs/de/docs/features.md.
🌐 Add Vietnamese translation for docs/de/docs/index.md, docs/de/docs/features.md.

Template for release note sections¶

### Breaking changes

* ⬆️ n/a.

### Fixes

* n/a.

## Upgrades

* n/a.

### Features

* ✨ Introduce new auto model creation for JSON.
* ✨ Introduce new auto model creation for XML.

### Refactors

* 🔥 n/a.

### Docs

* 📝 Add initial docs in english language.

### Translations

* 🌐 Add german translation for `docs/de/docs/features.md`.