Release notes

3.2.3

Improvements

• ✨ DB Metadata Scan Robustness for Non-JSON Scalars: Metadata scan and analysis now handle non-JSON-native database driver values (for example bytes-like values, timedelta, recursive/custom scalar payloads) through a unified scalar-coercion path, reducing scan aborts on edge cases.
• ✨ DB Scan Diagnostics Transparency: Scan task logs and run summaries now include explicit scalar-normalization counters and bounded fallback examples, plus analysis/table-processing resilience diagnostics for faster root-cause analysis.
• ✨ Audit Log Signal Quality: Request/auth logs now carry auditable actor context (actor_type, actor email for authenticated users) while preserving privacy controls and avoiding dedup of critical request/auth lifecycle events.
• ✨ Notification Center Visual Refresh: Notification menu and list items were restyled for stronger readability/contrast and clearer action affordances in dark/light contexts.

Fixes

• 🐛 DB Generate Fails Less on Shared/Noisy Schemas: DB generation/export no longer fails as easily when unrelated schema objects are broken or missing. This improves stability in shared Oracle/PostgreSQL environments.
• 🐛 Clearer DB Error Context for Users: When a DB target table error occurs, user-facing logs now better separate the configured target table from an additional failing dependency object, making troubleshooting faster and less confusing.
• 🐛 DB Metadata Scan Fail-Open Behavior: Table-level extraction/analysis anomalies are now handled per-table, so one malformed table no longer fails the full metadata scan run.
• 🐛 Unknown Scalar Type Handling Across DB Dialects: Model generation from metadata now safely normalizes previously problematic scalar edge types across Oracle/PostgreSQL/MySQL/MSSQL paths.
• 🐛 Server Monitor Shutdown Stability: Lifespan shutdown now signals monitor stop events defensively and joins non-daemon monitor threads safely, reducing shutdown race/error risk.
• 🐛 Notification Toast Simplification: Toast notifications no longer render a duplicate “Open in notification center” deep link, reducing visual noise in transient alerts.
• 🐛 Database View Filtered Bulk Selection: In Zone 1 table scope, Select all now applies to the currently visible filtered table list (instead of all tables), and Deselect all clears only the currently visible filtered tables. Bulk actions are disabled when filters match no tables.

3.2.2

Improvements

• ✨ Oracle Metadata Scan Scope Expanded (Backend): Database metadata scans now include SYSTEM tables and schemas, giving a more complete view of your Oracle database structure.
• ✨ External API Documentation Clarity: OpenAPI documentation for external file/artifact/task/git endpoints is now more explicit (streaming media types, path usage, and error behavior), making API usage in Swagger and CI pipelines clearer.

Fixes

• 🐛 Simple Mode UI: Fixed multiple visual representation issues, including disappearing expand icons and improved navigation flow.
• 🐛 Simple Mode Keyboard Flow: Fixed an issue where attribute deletion could be triggered incorrectly during keyboard-based interaction.
• 🐛 Simple Mode Extend-Node Flow: Fixed an issue where elements could no longer be extended in Simple Mode.
• 🐛 Git Project Update Robustness: Updating projects from external Git sources no longer hard-fails when invalid project files are present. Invalid files are now skipped and recorded in task logs.
• 🐛 Artifact Filename Validation Semantics: For artifact metadata/download routes, invalid entity_name values containing path separators now return explicit 400 INVALID_FILE_PATH validation errors instead of ambiguous not-found behavior.

3.2.1

Improvements

• ✨ Oracle Metadata Scan Scope Expanded (Backend): Database metadata scans are now even less restrictive for Oracle enterprise setups. The scan now unions schema discovery from inspector metadata and ALL_TABLES owners (best-effort), keeps system schemas excluded, and continues with a safe fallback when ALL_TABLES is not accessible.

3.2.0

Breaking Changes

• ⬆️ Simple Mode Disabled Elements Require Marker Comments: Disabled elements in Simple Mode are now recognized only when the XML comment contains the @dm-disabled:v1 marker. Legacy commented-out elements without this marker are treated as normal comments. If you want to manage those legacy elements in Simple Mode, you must migrate them manually to the marker format. No automatic migration is provided.

Features

• ✨ Simple Mode Keyboard-First Creation & Editing Flow: Simple Mode now supports end-to-end keyboard operation for the main creation/editing flow. How to use: from the File Tree, press ArrowRight (or Tab) to move focus into Simple Mode; navigate with ArrowUp / ArrowDown (siblings) and ArrowRight / ArrowLeft (child/parent, with top-level ArrowLeft returning to File Tree); on the active node, press C to add a child or A to add an attribute; use Tab to move through attributes and Enter to edit; confirm dialog steps with Enter. For remove flow, press Delete / Backspace on the active non-root node to open confirmation. New child creation keeps focus in the editor and moves selection to the created node. If legacy disabled comments without @dm-disabled:v1 are found, Simple Mode now shows a warning notification so manual migration is explicit.
• ✨ Database View Weighting File Generation: Database View now supports creating weighting artifacts directly from selected metadata as project files. Use Create File → Weighting from Database Metadata to generate .wgt.csv (single selected column) or .wgt.ent.csv (multiple selected columns). Sampling defaults to 1000 rows and supports deterministic or fresh sampling modes plus optional NULL-class inclusion. The stored weighting factor is normalized (count / sampled_rows) for stable distribution behavior across runs.
• ✨ Database Metadata Scan Scope Expanded: Scan Metadata now inspects all accessible schemas of the selected database connection (not only one schema). Inaccessible schemas are skipped with task-log warnings instead of failing the full scan. This increases metadata coverage for cross-schema relationship discovery and subset planning.
• ✨ Project Creation from Submenu: You can now create a new project directly from the project submenu for a faster top-level workflow.
• ✨ Database View Flow Redesign (Zone-Based): Database View now uses a clearer workflow structure with Zone 1: Selection Scope and explicit step progression for metadata-driven generation.
• ✨ Column-Level PII Persistence in Database View: PII-relevant selections can now be persisted at column level to keep anonymization-related intent stable across review and follow-up actions.
• ✨ Database Workbench Guided Flow: The workbench now follows an explicit sequence: Scope Selection (table selection) → Plan Subset → handle violations/issues → choose execution mode (synthetic, anonymization, or ml_train).
• ✨ Schema History and Schema Drift Visibility: Database View now introduces schema history and schema drift visibility so structural changes in the reflected DB model are easier to detect and review.
• ✨ Reference Source Mode Policy SPOT: Validation for reference_source_mode vs builder_mode is centralized in shared defaults and enforced consistently across API schema, server service contract checks, and worker generation contracts.
• ✨ Reference Node Composite + Source Mode Guidance: The database-generation reference contract now documents one unified <reference><field .../></reference> mapping model for single and composite keys and clarifies how to choose reference_source_mode (database vs weighting_files). See Data Definition Model - Advanced Elements, Database View, and Auto-Generate Model from Database.
• ✨ Scope-Aware Generate Name Policy: DATAMIMIC default schema-qualified <generate name="schema.table"> behavior is now scope-aware for execution safety in cross-DB target flows, while preserving schema-qualified defaults for source-only generation.

Improvements

• ✨ Demo Navigation Simplification: The standalone Demo navigation link was removed; Demo Store navigation is now anchored on the DATAMIMIC logo/icon.
• ✨ Kafka TLS Dialog Simplification: The Kafka environment wizard now hides empty security steps for PLAINTEXT, keeps certificate handling in an upload-first flow with explicit apply semantics, and groups TLS certificate/key inputs in one step. This reduces configuration friction while keeping TLS material handling consistent across persistence, health checks, and runtime execution. See Environments.
• ✨ Weighting Workflow Separation: Weighting file creation is now surfaced in a dedicated Database View tab to reduce flow mixing and improve task clarity. See Database View and Auto-Generate Model from Database.
• ✨ ML Generator View Redesign: The ML Generator view was redesigned to improve overview and usability, making generator intent and behavior easier to understand at a glance. See ML Generator View and ML Generator from Database Metadata.
• ✨ Task Logs: User-Friendly Run Summaries: DB metadata scans and model-generation tasks (DB/JSON/XML/CSV, including anonymize flows) now emit a readable run-summary line with key metrics (table/column totals, action mix, PII candidate counts, and top generators) to make downloaded logs easier to review.
• ✨ PII Contract Hardening Across Scan → UI → Anonymize: Metadata columns now use a strict PII contract (pii_candidate, pii_confidence, pii_source) as the single decision path. UI no longer infers PII from reason-string parsing, reducing drift between displayed and executed behavior.

Fixes

• 🐛 Cross-DB Infer Execution: Fixed runtime export failures caused by schema-qualified source names being used against incompatible target namespaces in cross-database generation scenarios.
• 🐛 Database View Wizard Defaults: Fixed reference_source_mode default resolution in File Wizard V2 metadata descriptors to align with backend contracts and avoid implicit weighting-files mode drift.
• 🐛 Single-Column CSV Upload Robustness: Fixed archive/project upload processing for CSV files that contain only one column (no delimiter present). Valid single-column inputs (for example .ent.csv) no longer fail with delimiter-detection errors.
• 🐛 Anonymize/UI PII Mismatch: Fixed a contract mismatch where columns not marked as PII in metadata UI could still be masked during DB anonymize auto-detect.
• 🐛 Database View Metadata Empty-State UX: Opening Database View without a metadata snapshot now returns a dedicated "initial scan required" flow instead of generic error handling.

3.1.0

Breaking Changes

• ⬆️ Database Model Generation Flow Moved: Generating DATAMIMIC models from relational database metadata is now handled in Database View. The legacy Editor file flow for this use case is removed. See Database View and Auto-Generate Model from Database.

Features

• ✨ Database View (Metadata Workspace): Expanded database metadata workflow with clearer table/column transparency, schema-aware table identity, and direct model generation from selected metadata.
• ✨ Oracle + PostgreSQL Metadata Support: Improved schema handling and metadata interpretation for Oracle and PostgreSQL generation scenarios.
• ✨ Client Subset Workflow: Improved subset selection UX in Database View, including fast selection reset patterns and bulk operations.
• ✨ Plan Subset (Relationship Closure): Added relationship-aware subset planning that expands selected tables with required foreign-key dependencies and gates inferred model generation until closure is reviewed.
• ✨ Pre-Select PII in Database Flow: PII candidate pre-selection is integrated into the metadata-driven generation flow for faster secure-by-default model setup.
• ✨ Create SQL Script from Metadata (New): Database View can now emit a SQL schema artifact from scanned metadata in the same flow as model generation.
• ✨ Duplicate Folder in FileTree: Added folder duplication including contained files for faster project structuring. See Business Mode File Tree.
• ✨ Object Storage Health Check Task: Added dedicated object storage health check support (CHK-OBJ) in the health-check task family.

Improvements

• ✨ Project Service Architecture: Project service domain was flattened and split by responsibility for better maintainability, testability, and extension.
• ✨ Database Metadata Reliability: Hardened metadata merge and schema fallback behavior to reduce ambiguity and improve deterministic generation contracts.
• ✨ Database View UX: Improved interaction stability for selection and preflight flows to reduce friction on large schemas.
• ✨ Core Recommendation Contract Integration: Platform now consumes core recommendation actions built from structural constraints and physical DB facts more consistently, improving inferred model quality.
• ✨ Core Oracle Connectivity Integration: Improved Oracle service-name/SID handling and script parsing integration for more deterministic Oracle generation and execution flows.
• ✨ Core Object Storage Integration: Upgraded object-storage read paths with stronger MinIO/AWS-S3 behavior and streaming-friendly execution characteristics for larger payloads.
• ✨ Core Diagnostics Integration: Expanded core Issue/diagnostics mapping yields clearer task-level error signatures for DB and export flows.

Fixes

• 🐛 Project View Sorting Refresh: Fixed stale ordering behavior so Last Edited-based project sorting refreshes correctly without manual view/filter toggles.
• 🐛 FileTree Stability: Fixed ordering and files_uri contract issues that could cause visible jumps/reordering in larger trees.
• 🐛 Oracle Recommendation Contract: Fixed Oracle NUMBER/constraint metadata propagation so recommendations are based on correct structural facts.
• 🐛 Oracle SQL Artifact Execution: Hardened SQL artifact generation and execution behavior for Oracle-oriented flows.
• 🐛 SQL Script Artifact Guardrails: SQL-script generation now enforces same-DBMS source/target pairs to prevent non-executable cross-dialect artifacts.
• 🐛 Project Archive Upload Robustness: Unsupported files are filtered out during upload instead of failing the full import process.
• 🐛 Project Upload/Download File Validation: Centralized project-file validation for project-menu flows to prevent invalid files from entering project context.
• 🐛 Git Project Sync Data-Loss Hardening: Improved git sync/apply flow safeguards to prevent accidental project-data overwrite scenarios during error paths.
• 🐛 Add User Dialog UX: Reduced disruptive reload behavior in the "Add User to Project" selection flow.
• 🐛 Generate Runtime Property Overrides: Fixed /api/v2/projects/{project_id}/generate so custom_property_file overrides are forwarded to worker tasks, applied from project-root-relative paths, and consistently affect runtime generation behavior.

3.0.0

Breaking Changes

• ⬆️ TemplateSingle Filename Counter: The filename counter format for TemplateSingle changed from hyphen (file-1.ext, file-2.ext) to underscore (file_1.ext, file_2.ext) to align with JSONSingle and XMLSingle exporters. Update any scripts or workflows that depend on the previous hyphen-based naming convention.
• ⬆️ Default CSV Separator Changed to Semicolon: With the introduction of platform-wide encoding and delimiter configuration, the default CSV separator for downloaded and exported files is now ; (semicolon). To use a different separator, configure it in Project Settings or specify the separator attribute directly in your model. See Data Sources for model-level configuration.
• ⬆️ Project File Encoding Standardized: Local project files under the descriptor directory are now always read as UTF-8 with | (pipe) as the CSV delimiter. Files using other encodings must be converted to UTF-8 before use.
• ⬆️ Project Path Policy: Local project files must reside under the descriptor directory. Symlinked descriptor directories and non-local file paths are rejected. Review your project structure if you use symlinks or reference files outside the descriptor directory.
• ⬆️ Dynamic Expressions Restricted: Double-brace {{ expr }} (dynamic) expressions are no longer allowed for attributes that resolve at setup time: source, uri, target, sourceUri, and exportUri. Replace with single-brace {expr} (cached) or literal values.
• ⬆️ Dynamic targetEntity Limited to Single-File Exporters: Dynamic targetEntity is now only supported with TemplateSingle, JSONSingle, and XMLSingle. Other exporter types will reject dynamic target values.
• ⬆️ exportUri Requires Object Storage: exportUri is now only supported when used with a configured object-storage client (storageId). For local exports, use target instead.

Features

• ✨ Encoding & Delimiter Configuration: Configure default encoding and CSV delimiter in the UI. Upload files with any format — the platform converts automatically. Downloaded files use your configured settings. See Runtime Configuration.
• ✨ Dynamic targetEntity: Single-file exporters (TemplateSingle, JSONSingle, XMLSingle) support dynamic filenames using {expr} syntax for per-record basenames. See Dynamic targetEntity.
• ✨ Scripting API: New read-only Python API (datamimic_ee.scripting) for accessing memstore data, variables, and context information from custom functions. See Scripting API.
• ✨ Folder Hierarchy: Create nested folder structures to organize project files in the File Tree. See File Tree.
• ✨ TemplateSingle Extension Override: Override output file extension with the extension parameter.

Improvements

• ✨ Data Sources Documentation: New comprehensive guide for all data source types (project files, object storage, databases, MongoDB, Kafka). See Data Sources.
• ✨ Unified Error Codes: Consistent Issue Codes with actionable guidance (I322, I867, I868, etc.). See Error Codes Reference.
• ✨ External Source Encoding: Full encoding/delimiter control for object storage and external sources via encoding and separator attributes.
• ✨ Cleaner Logs: Reduced log noise and more consistent error messages for easier debugging.

Fixes

• 🐛 Nested Generate Count: Fixed count resolution for nested generate flows that could produce incorrect record counts.
• 🐛 Exporter Streaming: Fixed streaming issues across JSON, XML, and Template exporters.
• 🐛 Database Error Reporting: Improved error messages for MongoDB and RDBMS data sources with specific Issue Codes.
• 🐛 BOM Handling: Fixed duplicate byte-order-mark issue in exported files.

---

2.7.1

Breaking changes

• ⬆️ Scheduler Persistence: Scheduling now uses Postgres (backend.schedule) with explicit scope and DB constraints. Redis import runs automatically on scheduler startup; legacy Redis-only schedules are no longer used.
• ⬆️ Schedule Uniqueness: System and backup infrastructure schedules are singletons per task_type; project schedules allow multiple rows.

Features

• ✨ Postgres Scheduler: New scope-aware PostgresScheduler with schema verification, auto-disable for invalid schedules, and Redis import on startup.
• ✨ TaskScope Model: Unified TaskScope/TaskPolicy governs storage, scheduling, and log bucket selection across server/worker/UI.
• ✨ Admin Views: Admin Tasks shows project + system scopes; backup/restore tasks remain in the maintenance view.
• ✨ Generated TaskType SPOT: UI task labels/routing keys are generated from backend TaskType to prevent drift.

Fixes

• 🐛 GEN-SCH Log Archiving: Scheduled generation logs now archive to MinIO like other generate tasks.
• 🐛 Log Bucket Setup: Log bucket is created during prestart, ensuring archived logs are always persisted.
• 🐛 Search Validation: Invalid schedule_id/project_id filters now return 400 for fast feedback.
• 🐛 Routing Key Filters: Task search routing_keys require canonical datamimic.* values.

2.7.0

Fixes

• 🐛 Project Deletion: Deleting projects now removes related artifacts, logs, tasks, and snapshots cleanly.
• 🐛 Log Viewer: System restore logs download correctly; log display and scroll issues are resolved.
• 🐛 Tables: Hide-column now hides rows; pagination and per-page handling are consistent across server-backed tables.
• 🐛 File Naming: Template/file name sanitization now strips invalid characters to prevent bad filenames.
• 🐛 UI Stability: Fixed Vite chunk mismatch after rebuild and Linux-specific styling glitches.
• 🐛 XML Exports: Nested list handling now renders correctly in XML/XMLSingle outputs.

Features

• ✨ Housekeeping (New): Run or schedule housekeeping with retention windows; tasks can be preserved to exclude them from cleanup, with full task history and logs.
• ✨ Scheduled System Backups: Schedule recurring backups with configurable scope and clearer review before execution.
• ✨ Search & Filters: Faster, consistent search and filtering for tasks, schedules, users, and projects with stable pagination and sorting.
• ✨ SQL Exporter (Streamed): Streamed SQL output supports large exports and smoother defaults out of the box.
• ✨ Modeling & XML Fidelity: Improved model generation quality with better XML mirror accuracy and richer validation hints.
• ✨ Clearer Configuration Errors: Missing or invalid assets now surface actionable errors that point to the right config file.
• ✨ Log Downloads: Task and system log views support direct log downloads.

Upgrades

• ⬆️ Fail-Fast Healthchecks: Default healthchecks may now be effectively “on” in more environments; misconfigured clients fail earlier by design.
• ⬆️ Validation Warnings: Unknown generators, converters, or entities may now emit warnings to help catch typos or unsupported identifiers.

2.6.1

Fixes

• 🐛 Runtime config stability: The server now fetches runtime capabilities via the worker before validating project settings, eliminating the hard dependency on datamimic_ee and returning friendly 503 messages when the worker is unavailable. The UI no longer redirects to maintenance mode for non-maintenance 503s.

2.6.0

Breaking changes

• ⬆️ API v1 Removed:
  The legacy API v1 surface (routers, schemas, and the old LSP package) has been fully removed.
  All integrations must now use API v2, which provides typed schemas, clearer authentication, and unified file/task workflows.

• ⬆️ Template Exporter Renamed:
  The previous TemplateExporter is now split into:

  • Template — multiple records → one file
  • TemplateSingle — one record → one file
    Automation referencing the old name must switch to the correct variant.

Fixes

• 🐛 ML Assets: Improved deletion of trained ML models; all associated MinIO artifacts are now removed correctly.
• 🐛 Task Logs & Restore: Redis restores no longer drop task logs; backup dialogs now display the selected filename before execution; log views keep filters and pagination synchronized.
• 🐛 Access Controls: Project access tokens can no longer be created with expired dates; admin search accepts exact user emails again.
• 🐛 File Management: File Wizard V2 no longer hangs during file switching; blank/not-blank filters in weighting tables work correctly; empty filenames are prevented.
• 🐛 Platform Stability: API/WebSocket handlers keep DB sessions alive for their full lifecycle; pagination tables reset filters when rows disappear; checksum uploads work even without multipart form data.
• 🐛 EDIFACT Migration: EDIFACT migrations now run through the same worker/service stack as DB flows, producing consistent logs, generators, and QA evidence.
• 🐛 Logging & Auditability: All worker tasks now emit structured logs (start/progress/complete), ensuring full audit trails across migrations.

Features

• ✨ Backup & Restore Reliability: Backups now run as isolated backup infrastructure tasks with streaming uploads, strict checksum validation, and automatic log archival to MinIO for predictable runtimes and durable logs.
• ✨ ML Generator Explorer (Beta): New dashboard and API v2 endpoints to browse ML generators, inspect versions, and review QA summaries directly in the UI.
• ✨ Unified Generation Pipeline: CSV/JSON/XML/DB flows now share a single drift-aware pipeline that preserves overrides, handles composite foreign keys, and emits unified audit metadata.
• ✨ EDIFACT Template Editor Refresh: Updated Core UI styling, dynamic spec loader, and support for custom segment elements for smoother EDIFACT editing.
• ✨ Async Git Operations: Git push/update actions now run on Celery workers with clearer progress logs, keeping the UI responsive during large syncs.
• ✨ Recommendation Engine Integration: The datamimic_ee recommendation engine now powers generator selection across all migration paths (CSV, JSON, XML, EDIFACT, DB).
• ✨ Runtime Capabilities API: A dedicated router exposes a cached, canonical capabilities descriptor so the UI always reflects current engine and sampling options.
• ✨ Deterministic Generator Defaults: Domain-scoped defaults and deterministic RNG seeding make large-scale runs reproducible while allowing tenant-level format adjustments.
• ✨ Filename Handling: Uppercase filenames are preserved end-to-end to avoid accidental renames, especially in regulated message formats.
• ✨ File Wizard V2: “Create New File” now opens the V2 registry (street → truck → container), centralizing file creation, validation, and future folder structures.
• ✨ Template / TemplateSingle Enhancements:
• Clear separation between single-file and per-record exports
• Auto-derives output extension (e.g., invoice.xml.template → invoice.xml)
  Ideal for SWIFT, EDI, PACS and other message formats.
• ✨ TXT Exporter Improvements:
• value_only=False → names + keys + values
• value_only=True → values only
  Useful for strict text-based pipelines.

2.5.1

Fixes

• 🐛 File Loading: Fixed an issue where switching file during edit can cause infinite loading.

2.5.0

Breaking changes

• ⬆️ API v2 is now Production Ready (API Users Only):
  API v2 is now the default and fully supported version for all API integration.
• Note: These changes only affect users who interact with the platform via the API. The web UI and other features are not impacted.
  • All existing API v1 endpoints are now officially deprecated and will be planned to be removed in our next major release.
  • The external API documentation has been updated to highlight and prioritize v2 endpoints.
  • To help API users transition smoothly, we provide a detailed Migration Guide with step-by-step instructions and best practices.

Fixes

• 🐛 User Sign Up: Resolved an issue preventing users from accessing the sign up page after logging out.
• 🐛 Demo Card Skeleton: Corrected the demo card placeholder width and layout when loading demo projects.
• 🐛 Project Settings: Fixed an error triggered when filtering users in project settings.
• 🐛 System Tables: Removed the index column from the system projects and system users tables.
• 🐛 Account Management: Allowed admins to clear first and last names to empty strings.
• 🐛 File Downloads: Ensured the latest file version is downloaded immediately after saving.
• 🐛 Project Deletion: Disabled the delete button while a multi-project deletion is still processing.

Features

• ✨ User Menu: The external API link in the user menu, previously visible only to admins, is now available to all users.
• ✨ User Projects: Added a loading state indicator when accessing or fetching the user's projects, improving the user experience during data retrieval.

2.4.0

Breaking changes

• ⬆️ Constraint syntax refactor:

• <targetConstraints> / <sourceConstraints> now use require= for filtering only.

• <mapping> now uses set= for transformations.
• Legacy <rule> syntax removed.

Fixes

• 🐛 Tabular Data:
  Fixed an issue where Tabular Data edited and saved through the UI was incorrectly persisted as a UI-JSON structure inside .csv files (e.g. .wgt.csv, .ent.csv, .optl.csv) instead of proper CSV rows.
  • This caused tabular files to appear malformed or unreadable when accessed outside the UI.
  • A migration now normalizes affected files into canonical pipe-separated CSV.
  • Enable with DM_NORMALIZE_TABULAR_ON_START=true (default: disabled).
  • Runs in dry-run mode by default; apply changes with DM_NORMALIZE_APPLY=true.
  • Suitable for medium-sized MinIO / project repositories. For larger datasets, we provide a standalone script and support.
  • Alternative: open affected files in the UI and re-save; they will be rewritten into correct CSV format and synced back to Project Data.
• 🐛 Maintenance Mode: Admin users no longer redirected away during maintenance.
• 🐛 Logging polish: Console + Redis handlers now consistently honor runtime_config.log_level.
• 🐛 Global project: Fixed an issue where global project file content doesn't display properly for normal user.
• 🐛 User Login: Fixed an issue where user might not be able to login when password is updated.
• 🐛 UI Toolbar: Fixed an issue where unnecessary toolbar is displayed on Windows devices.
• 🐛 User Project: User drop-down now shows and searches all users, not just 30, in project settings.

Upgrades

• ⬆️ Extended Log Handling: More meaningful, less noisy logs across the platform.

  • Central policy allowlist for INFO events, sampling/deduplication, and sensitive-data scrubbing; consistent console/JSON formats.
  • Honors LOG_LEVEL, LOG_FORMAT, LOG_COLOR; structured event fields with millisecond timestamps.
  • Keeps the success path quiet while preserving high‑signal events for observability.

• ⬆️ Enterprise Database Connectivity:

  • Oracle: SID + service_name support.
  • MSSQL: instance, encryption, trustServerCertificate support; correct driver auto-selection.

• ⬆️ Read-only Safeguards:

  • MongoDB: compressors normalized, read-only wrapper added.
  • RDBMS: non-SELECT blocked, READ ONLY transactions enforced.

• ⬆️ Cursor & Keyset Execution Model:

  • Offset/limit replaced by cursor/keyset iteration.
  • Workers lease shards via manifest.
  • Scales linearly to billions of records, reduces load on target systems, ensures exact-once sync.

• ⬆️ Memory & Engine Safety:

  • Improved memory handling in loaders.
  • Engine kwargs whitelisted for safe defaults.

• ⬆️ Rust Fast-Path Evaluation Engine:

  • Core script evaluation hot paths rewritten in Rust.
  • Higher throughput for heavy transformations.

• ⬆️ DateTimeGenerator: Major upgrade for enterprise use cases — see: DateTimeGenerator

  • Modes: fixed value, random window with min/max, or current time.
  • Bounds: strict min/max respected down to seconds; edge-day clamping.
  • Determinism: private RNG with seed for repeatable sequences.
  • Weights (intra‑day): hour_weights, minute_weights, second_weights (length-validated; non‑negative; sum>0).
  • Day selection weights (inter‑day): month_weights, weekday_weights, dom_weights with proportional sampling.
  • Sugar filters: friendly presets for months/weekdays/dom, hours_preset, minute_granularity, second_granularity (explicit weights override sugar).
  • Epoch support: via XML formats (inDateFormat="epoch", outDateFormat="epoch|epoch_millis|epoch_micros|epoch_nanos|%Nf") and Python flags (as_epoch, epoch_unit).
  • Performance: Rust fast‑path when no seed and no weights/sugar; buffered for throughput; Python O(1) sampling after per‑window build.
  • Validation and error handling for invalid lengths/values, empty day sets, invalid granularities, and unknown presets.

Features

• ✨ Runtime Configuration (new): Tune log levels, paging, Redis/leases, Kafka, preview limits — now configurable for the first time.
• ✨ Capability-driven Environment Dialogs: Dynamic forms for DB, Kafka, Mongo, Object Storage; backend always in sync with UI.
• ✨ Enterprise Health checks: Oracle SELECT 1 FROM DUAL, MSSQL encrypted connections, Kafka TLS/SASL, Mongo compressors. Fail fast, realistic probes.
• ✨ Backup & Restore Enhancements: Backups embed DATAMIMIC version; Postgres backups exclude owner metadata → portable, audit-friendly.
• ✨ Duplicate Files via Filetree: Quickly duplicate project files with consistent naming (-copy, -copy-1, …).
• ✨ File Locking System: Multiuser file locking to prevent overwrites; supports intentional lock stealing for collaboration transparency. See: File Locking
• ✨ Filetree Usability Improvements: Realtime search filter, keyboard shortcuts (rename/delete), and “no results” hints for smoother navigation.
• ✨ API: Added new admin-only endpoints to access DATAMIMIC's instance Redis and MinIO statistics.
• ✨ Editor: Switched the in-app code editor to Monaco for improved reliability and consistency, while keeping the UI unchanged for a seamless user experience.
• ✨ Task Id: Task Id can be easily copied in the TaskView now. For usage in API for example.

Refactors

• 🔥 Registry slimmed to router-only role; parsing moved into loaders.
• 🔥 Dead code removed; helpers modularized.
• 🔥 Logging unified with millisecond precision timestamps.

---

2.3.0

Breaking Changes

• ⬆️ Syntax: Backslash-escaped colons (e.g., gc\:CodeList) are no longer supported.
  • Please use bracket notation for namespaced keys: root['gc:CodeList'] (recommended).
  • Alternatively, use plain colon identifiers (gc:CodeList), which are automatically sanitized.

Features

• ✨ Template System: Introduced a new Template Exporter and Management System for business messages:
  • Export templates in free-text format with placeholder variables from the generation context.
  • Create templates via the File Wizard, supporting JSON, XML, SWIFT, PACS, EDIFACT, and more.
  • Seamless integration with existing data generation workflows.
• ✨ Variables: Enhanced variable management with the new storage attribute:
  • storage="value" – Single static value (default for generators and constants).
  • storage="data" – Complete data list stored in memory.
  • storage="iterator" – Iterator that cycles or exhausts based on the cyclic setting.
• ✨ DateTimeGenerator: Extended with new attributes:
  • as_epoch: bool = False returns the epoch timestamp as an integer.
  • epoch_unit: str = "seconds" specifies epoch precision (seconds, milliseconds).
• ✨ Generate: Added IterateSelector to nested generate tasks for dynamic data source selection:
  • Enables dynamic querying for each parent iteration.
  • Supports both static and dynamic data sources.
  • Provides greater flexibility in data generation scenarios.
• ✨ Demos: Added new demos:
  • Template Exporter demo, showcasing capabilities and integration.
  • Structural operations demo for XML and JSON, highlighting new Operate Element features.
  • Extended DateTimeGenerator demo to illustrate new attributes.
• ✨ Namespace Access: Introduced a global root alias for robust namespaced key addressing (e.g., root['gc:CodeList']).
• ✨ Maintenance: Improved zip file handling for backup and restore operations, significantly reducing backup file size.

Refactors

• 🔥 Core: Refactored datasource and target handling for better performance and efficiency.
• 🔥 UI: Optimized API and performance across all UI components.
• 🔥 Contexts: Extracted a streamlined evaluation pipeline.
• 🔥 API: Improved error reporting for missing custom properties files in Generate API requests.
• 🔥 Workers: Worker system now acknowledges messages immediately after processing, preventing duplicate processing.

Fixes

• 🐛 Fixed an issue where zombie processes were not cleaned up properly.
• 🐛 Fixed truncation of expression error messages with float literals (e.g., 0.05); messages are now accurate and informative.
• 🐛 Improved handling of TypeError/ValueError in user expressions, providing actionable guidance and name-type details.
• 🐛 Enhanced error messages:
  • Name-type hints (e.g., x=int, y=undefined).
  • Specific guidance for common TypeErrors; colon retry path includes the original exception.
• 🐛 Maintenance Restore: Fixed restore failures for Postgres databases containing task data.
• 🐛 Maintenance Restore: Corrected confirmation dialog to display the correct filename during restore.
• 🐛 Editor: Resolved issues with values attribute examples.
• 🐛 Dashboard: Removed unnecessary refetch interval for the system user API call.
• 🐛 Error Connection Message: Connection error message now remains visible until the network is restored.
• 🐛 Notification View: Fixed React rendering error caused by objects being rendered as children.
• 🐛 Menu: Fixed missing username in the menu bar.

2.2.0

Breaking changes

• ⬆️ Project Reset: Removed the ability to reset a project from a specific task ID.

Features

• ✨ Snapshot Management: Enhanced snapshot capabilities for improved task execution control:
• Re-run previous tasks from a snapshot, preserving exact project state
• Download snapshot files from tasks to capture project state during execution
• ✨ API Integration: Integrated new API structure across all UI modules (Demo, Template, Schedule, Log, User, Login, System Management, Access Token)
• ✨ SQL Formatter: Upgraded from sql-formatter to sqltools-formatter for improved SQL formatting capabilities
• ✨ Operate Element: Introduced comprehensive batch operation support for template transformations:
• Dual format support for XML (XPath) and JSON (JSONPath) templates
• Advanced XML operations: replaceChildren, appendChild, upsertNode, mergeAttributes, clearChildren
• JSON-specific operations: mergeObject, upsertProperty, replaceArray, appendItem, clearArray
• Function operations for custom logic execution without content modification
• Dynamic value injection using DATAMIMIC generators
• Configurable error handling strategies
• CSV-based operation definitions with customizable prefixes
• Automatic validation of operation sequences
• Seamless integration with business domain generators

Refactors

• 🔥 MongoDB: Improved delete operations for both selector-based and selector-free deletions

Fixes

• 🐛 MongoDB Pagination: Optimized memory consumption through improved pagesize control
• 🐛 Table Pagination: Hidden pagination controls when no data is present
• 🐛 CSV Editor: Removed default placeholder test data
• 🐛 Database Metadata: Resolved cache issues preventing retrieval of latest user metadata
• 🐛 UI Consistency: Adjusted Advanced Settings button padding for better alignment
• 🐛 Dark Mode: Corrected color inconsistencies for preview and artifact buttons
• 🐛 Project Navigation: Fixed sorting issue in "5 Latest Projects" popover display

2.1.0

Features

• ✨ API: The Generate API now supports injecting dynamic properties at execution time, enabling more flexible and customizable data generation workflows via API calls.

Refactors

• 🔥 UX: In Cron Generate tab, the Generate button is now disabled when the Cron value is empty, preventing invalid submissions.
• 🔥 UX: In Task table, the View Log button is now enabled even when the Task status is Queued, allowing users to access logs earlier in the task lifecycle.
• 🔥 Global Project: For protected Global Project files, the Download button is disabled now.

Fixes

• 🐛 Global Project: Fixed an issue where xml parser not working properly for global project's files.

2.0.0

Breaking changes

• ⬆️ multiprocessing: The multiprocessing attribute has been replaced with a unified numProcess and mpPlatform configuration for improved clarity and flexibility in task execution.
• ⬆️ Condition Nodes: The <if> node has been replaced by a more versatile <condition> node structure, which now includes <if>, <if-else>, and <else> nodes. This change enables more complex conditional logic in data generation tasks.
• ⬆️ Project Access Token: Introduced a Project Access Token system, providing more secure and flexible API access.
• ⬆️ Operate: The operate task type has been completely refactored and decoupled from the generate task type. It now supports XPATH and JSON Path operations on XML and JSON structures, enabling advanced data manipulation capabilities.

Fixes

• 🐛 Git provider: Fixed an issue where the selected Git provider was not saved correctly.
• 🐛 Delete project: Deleting a project now cleans up all related resources.
• 🐛 Admin task: Fixed an issue where the admin task table could not be searched by task ID.

Features

• ✨ Architecture: The DATAMIMIC platform now supports configuring multiple worker charts. Tasks can now run with multiple workers, improving performance and scalability.
• ✨ Generate: Added support for additional task types (standard, infinite, and timeout) with multi-worker capability.
• ✨ UI: The UI has been completely redesigned with a modern look.
• ✨ Performance: Internal file handling has been improved for enhanced performance and scalability.
• ✨ Demo store: Introduced tag-based filtering tabs and redesigned demo cards for better usability.
• ✨ Project view: Enabled filtering by tags and project types via new tabbed navigation.
• ✨ Notification: Launched a dedicated notification center with improved display logic.
• ✨ System maintenance: Introduced Backup & Restore functionality, allowing Admins to back up and restore DATAMIMIC instance data.
• ✨ System banner: Added banner configuration, enabling Admins to display system alerts to all instance users.
• ✨ Table data: Upgraded all tables with search, sorting, multi-filtering, and column visibility. This aims to improve performance with large data sets.
• ✨ Git Project: Git projects can now be synchronized with branches other than the default.
• ✨ System projects: Admin users can now perform bulk project deletions for more efficient management.
• ✨ Project settings: Added tag management, task type configuration, and log level controls.
• ✨ Project settings: Added a new button to copy the project ID.
• ✨ Create project files: Allows uploading multiple files at once.
• ✨ Template Bulk Upload: Users can now bulk upload JSON or XML templates and add prefixes or suffixes for greater flexibility.
• ✨ Operate on JSON and XML structures: Users can now perform operations (set, delete, etc.) via XPATH or JSON Path on XML and JSON structures.
• ✨ API: Extended support for API endpoint usage with Project Access Token.
• ✨ API: Introduced external_api_docs to provide a more comprehensive and user-friendly way to use the API for standard workflows.
• ✨ API: Added a new API endpoint to allow downloading all artifacts related to a task as a single zip file.

Refactors

• 🔥 Architecture: All containers have been refactored to be more modular and easier to maintain, reducing Docker container size and improving performance.
• 🔥 Cleanup admin task: Moved the cleanup button into the admin task view (Dashboard).
• 🔥 Reset project: Reset project logic has been separated for clearer context between resetting the entire project or by task.
• 🔥 UI: The System API button now directs users to external_api_docs by default. The advanced docs remain accessible but are no longer linked from the UI.
• 🔥 Documentation: Project access token usage has been updated to use the new external_api_docs endpoints and examples.

1.10.0

Fixes

• 🐛 SourceConstraints + random distribution (DAT-628): Fixed a defect that caused rows to be dropped when sourceConstraints were combined with distribution="random".
• 🐛 RemoveNoneOrEmptyElement on root level: When uploading to MongoDB (and other database exporters), the converter now removes root-level fields that resolve to None, empty strings, or empty sub-documents.

1.9.1

Fixes

• 🐛 Exporter: Update oracle db driver and refactor OracleDB engine
• 🐛 Exporter: Update mssql driver to pymssql

Features

• ✨ XML Model: Shorten_variable_script_path (this.var to access keys and variables on the same level)

Refactors

• 🔥 Generate: General context improvements
• 🔥 Exporter: Refactor KafkaImporterClient: Remove signal handling for timeouts and streamline message polling logic
• 🔥 Exporter: Extend error handling in ml train task

1.9.0

Fixes

• 🐛 Expression Evaluation: Handle colon characters correctly in Python expressions (e.g., for XML namespaces).
• 🐛 Generate: Improve error messages for undefined items and invalid data structures.
• 🐛 Generate: Fix handling of recursive and nested dictionaries during evaluation.
• 🐛 XML Load: Improve type detection and handling for OrderedDict and dict in XML input files.

Features

• ✨ API: Extend standard generate endpoint and artifact to be able to use with project access token.
• ✨ Generate: Support for dynamic sources in element-level generate tasks.
• ✨ Generate: Introduce new multiprocessing platform Ray for distributed remote execution.
• ✨ Generate: Replace multiprocessing attribute with unified numProcess and mpPlatform configuration.
• ✨ Constraints: Introduce three-layer constraint system with <sourceConstraints>, <mapping>, and <targetConstraints>.
• ✨ Script Attribute: Add support for directly accessing and transforming nested data structures in script elements.
• ✨ SQLAlchemy: Support for .env.property-based configuration for SQLAlchemy database connections.
• ✨ FileCacheRegistry: Introduce caching mechanism for efficient file processing and reuse.
• ✨ Domain Generators: Extend domain-specific generator library with advanced weighting-based models for Healthcare, Public Sector, and E-commerce datasets. Improves out-of-the-box realism for structured scenarios.

Preview

• 🚧 Operate Task: Add task type to perform post-processing directly on datasources.
• 🚧 ML-Train Task: Enable advanced enrichment of data from source datasets using a fully integrated ML pipeline. The <ml-train> task supports training tabular generators based on data samples (e.g. first 1,000 rows), leveraging Tabular ARGN to capture realistic distributions and inter-column relationships. Trained models are persisted and can be reused across <generate> tasks for statistically grounded synthetic data creation.
• 🚧 ML-Generator Reusability & Scalable Execution: Persisted models are referenced via source="..." in <generate> tasks. Batch-based generation using pageSize enables memory-efficient processing. <key> blocks allow real-time post-processing and formatting. Constraint rules (column, row, cross-record) are enforced during generation.

Refactors

• 🔥 Generate: Improve multiprocessing logic and simplify configuration by deprecating multiprocessing in favor of numProcess.
• 🔥 Data Pipeline: Refactor evaluation context for better reliability and performance during data generation.

Docs

• 📝 Add Advanced Guide to use Project Access Tokens programmatically (EN/DE)
• 📝 Add Domain Generator Guide covering configuration and tuning of Healthcare, Public Sector, and E-commerce weighting models

1.8.5

Fixes

• 🐛 Simple mode: Improve selection of target properties
• 🐛 Editor: Fix an issue where downloaded JSON was incomplete

1.8.4

Fixes

• 🐛 Simple mode: Fix proper selecting and handling of adding properties
• 🐛 Simple mode: Fix an issue where the user cannot add a new property in the simple mode when no value field exists.
• 🐛 Autocomplete: Improve autocompletion for condition and generator elements.
• 🐛 Editor: Small ui fix with duplicated arrow for collapse/expand in the editor.

Refactors

• 🔥 Simple Mode: Improve the Simple Mode rendering.
• 🔥 Data Table: Improve handling and sorting of data tables.

1.8.3

Fixes

• 🐛 Schedule API: Fix an issue where schedule cannot be filtered correctly for advance search support.
• 🐛 Git: Fix an issue where git push button is not visible in Task View.

Features

• ✨ Admin Task: Add ability to copy task id and perform search for Admin in Admin Task Table.

Docs

• 📝 Add initial docs in german language.
• 📝 Rework and split reference model data definition

1.8.2

Fixes

• 🐛 Global Projects: Fix global projects and environments not working on generate with infinite mode.

1.8.1

Fixes

• 🐛 Kafka-Exporter: Fix handling of acknowledgment (ack) attribute by properly processing special characters and edge cases in configurations.
• 🐛 Autocomplete: Enhance Autocompletion by adding comprehensive attribute listings for Kafka exporters and importers

1.8.0

Fixes

• 🐛 Logging: Wrong log level for the generate, condition, and nestedKey elements
• 🐛 Logview: Fix flickering when log updates occur
• 🐛 Preview: Fix incorrect preview limitation handling and set an expiration time for previews (4 days)
• 🐛 NestedKey Input Source: Fix dictionary data generation returning key lists instead of maintaining dictionary structure (issue not present with distribution="ordered")
• 🐛 Table Editor: Fix reloading of table when changing between browser tabs and different windows

Features

• ✨ Helm Charts: Add instance management scripts (rdd-miniomgr.py and rdd-redismgr.py) for worker pods
• ✨ Helm Charts: Add OpenShift compatibility with emptyDir volume for ephemeral storage
• ✨ Helm Charts: Enhance worker health checks
• ✨ Logging: Add project log level configuration
• ✨ Logging: Add multiprocess logging handler for sub worker process logs
• ✨ DB Source: Add chunking support to improve performance and reduce memory usage

Refactors

• 🔥 Helm Charts: Remove deprecated WORKERS_PER_CORE and WORKERS variables from worker configmap
• 🔥 Artifacts: Rework artifact handling for better performance and memory usage
• 🔥 File Wizard: Improve usability and user experience
• 🔥 UI: Enhance handling of non-existent projects
• 🔥 Logging: Refactor Loghandler for multiprocess support and efficiency

Docs

• 📝 Source Scripting: Extend documentation for source scripting features
• 📝 Variable Substitution: Add comprehensive guide on variable substitution
• 📝 MongoDB Operations: Extend Document MongoDB-specific features and operations

1.7.2

Fixes

• Kafka SSL: Fix an issue where the Kafka SSL configuration was not working correctly.
• Logging: Fix an issue where debug logs were set to INFO level. *

Refactors

• 🔥 Logging: Improve logging for Cascaded Generate Tasks.

1.7.1

Fixes

• Exporter System: Clean up behavior in cascaded generate tasks and naming logic for single processing.

1.7.0

Fixes

• Editor: Fix an issue where files are not decoded properly when the project is reset from the Editor.
• Editor: Fix an issue where JSON related scripts cannot be displayed properly with the new base64 encoding protocol.
• Schedule: Fix an issue where the Schedule cannot be run.
• Global Projects: Fix an issue where global project files are not being handled correctly.

Features

• ✨ Database View: Introducing a new Database view that allows customization of data generation and modeling based on SQL databases. Read More
• ✨ API: Allow the Admin user to always have access to any task log.
• ✨ API: The log is now base64 encoded when sent from the API.
• ✨ UI Log: Add an option to enable auto-refresh of the log every 5 seconds.
• ✨ MongoDB: Extend the authentication mechanism options to include PLAIN, which is also the default value when setting up.
• ✨ Project Setting UI: Introducing a new design for the project settings view for a more comprehensive workflow.
• ✨ Exporter: Add a new exporter OpenSearchBulk for exporting entities into OpenSearch bulk format. Read More
• ✨ Generate: The default log level for generate operations is now set to INFO. An option has been added to run generate in Debug mode for more detailed logging and troubleshooting.

Refactors

• 🔥 Email: Reduce overhead for our email template service.
• 🔥 Exporter-System: Refactor the exporter system to be more flexible, allowing users to configure the exporters, making them more efficient in multiprocessing, and enabling chunking. Read More
• 🔥 UI: Move system configuration settings to the new User Dashboard.
• 🔥 Schedule UI: Only display Last Run time in Schedule View when the schedule has been executed at least once.
• 🔥 Editor UI: Refactor the Create files/models flow to be more synchronized with the environment creation flow.
• 🔥 Editor Advanced Mode: Combine the Format Files and Format Variables buttons into a single button.
• 🔥 Editor: Newly created files will now be selected by default.
• 🔥 Editor: Update the file tree to be more user-friendly.
• 🔥 Editor: Move the Navigation bar to the top of the editor to enhance the user experience and maximize screen space.
• 🔥 Artifact: Update the artifact download button's color background.

1.6.0

Fixes

• Editor: Fix Error when project time stamp is updated even when there are no changes to the files.
• condition element: Fix error generate element inside of generate|condition not consuming.
• UI Change behavior where local time stamp is show with 24:00 instead of 00:00AM.
• Editor Fix an error where user cannot rename model file in editor.
• Global Projects: Fix priority of duplicate entity files when generating with global projects.
• Editor: Truncate long file name in File Tree, allow user to re size file tree horizontally.
• Autocomplete & Linting: Added missing element Null Quota
• UI: Fixed an issue with not supported Filetypes in the File Creation Wizard.
• UI: Fixed an issue with the File Creation Wizard creating JSON templates
• Script: Fixed and issue with evaluating property files correctly into the context.

Features

• nestedKey element: script attribute can be combined with with cyclic.
• IncrementGenerator: add a configuration option is step, flexibility in terms of controlling how much to increment the value each time.
• StringGenerator: generates strings based on a character set, prefix, suffix, and length, while ensuring all generated strings are unique.
• Admin Users Projects: Enable Admin users to perform bulk deletion of projects directly from the Project Systems View table. This includes enhanced search and filtering by user’s email to manage multiple projects efficiently..
• API Allow Admin user to always have access to any projects and files.

Refactors

• 🔥 Environment configuration: Refactor the environment configuration (especially kafka) to be more user-friendly.
• 🔥 Simple Mode: Refactor the Simple Mode to be more user-friendly and intuitive.
• 🔥 UI colors: Refactor the UI colors to have more contrast and be more user-friendly.

1.5.2

Fixes

• Rename Files: Fix the issue where renaming files in the editor was not working correctly.

1.5.1

Fixes

• Artifact Exporter: Fix the path for the .tempfiles before they get pushed to the object-store.
• Demo: Fixed the naming of the sql script file in the selector demo.

1.5.0

Fixes

• Artifact Download: Fix an issue where the artifact download was not working correctly and very slow with large files.
• Autocomplete: Fix an issue where the autocomplete was not working correctly for script attributes.
• Datetime precision: Fix default precision for datetime generation. The default precision %f is now set to milliseconds or %6f, when not other specified.
• File Type: Remove not supported file types from the file creation Wizard.
• Demo: Fix a small issue related to condition demo.

Features

• ✨ Model Autocompletion: Extended DATAMIMIC model autocompletion and linting for NestedKey and array nodes. Enriched the documentation and added more examples showcasing JSON and XML usage.
• ✨ JSON Header exporter: Implemented a new JSONHeader exporter for exporting entities into separate JSON files with a header, defined by a key with the name $$header$$.
• ✨ Converter: Extended a RemoveNoneOrEmpty converter to remove None or empty values from the data. This converter is applicable in generate, nestedKey and list nodes.
• ✨ JSON Pretty: We added a pretty formatting to our JSON exporter to make them easier readable.
• ✨ Global environments: Extend global project ability, use can now have the global project's system environment include when add global project.

Refactors

• 🔥 Default Value: Refactored the defaultValue function to be consistent in key and nested key nodes.
• 🔥 UI Enhancements: Move system configuration to the new configuration view.

Docs

• 📝 Configuration View: Updated the documentation for the new configuration view, explaining all available options.
• 📝 Global Project Section: Updated documentation for global project functionality with recent changes.
• 📝 Target: Extended the documentation for the JSONHeader exporter.

1.4.0

Fixes

• File Creation and Update: Resolved an issue where file creation and update via drop zone was broken.
• JSON Exporter: Fixed an issue where the JSON Exporter produced incorrect output when using 10 pages and 0 entities with a specific configuration.
• Autocompletion: Addressed a missing pattern in the autocompletion model.
• Project Menu: Fixed the issue where the project quick access menu was static.

Features

• ✨ Model Autocompletion: Extended DATAMIMIC model autocompletion and linting for NestedKey and array nodes. Enriched the documentation and added more examples showcasing JSON and XML usage.
• ✨ Configurable Signup Options: Implemented configurable signup options for DATAMIMIC, allowing admins to configure initial user roles and disable self-signups.
• ✨ Standalone Library: Integrated the DATAMIMIC library as a standalone module for realistic data generation in platform tests.
• ✨ String Attribute for complex Strings: Added a new string attribute to the variable and key nodes. This attribute allows users to define complex strings where variables are dynamically evaluated and inserted using configurable delimiters (e.g., __variable__)
• ✨ Global Project: Add ability that allow admin user to switch a global project back to standard project.

Refactors

• 🔥 Project View: Refactored the project view to enhance usability and design.
• 🔥 UI Enhancements: Improved the UI for the editor sidebar tools to streamline the user experience.

Docs

• 📝 Configuration View: Updated the documentation for the new configuration view, explaining all available options.
• 📝 Global Project Section: Updated documentation for global project functionality with recent changes.
• 📝 Model configuration and definition: Extended the documentation for model configuration data-definition to include more example use cases and examples.

1.3.0

Breaking Changes

• ⬆️ Condition Nodes: The condition nodes have been significantly refactored to support more complex logic. The previous if node has been replaced by a more versatile <condition> node structure, which includes <if>, <if-else>, and <else> nodes. The <if> and <if-else> nodes define the primary conditions, while the <else> node specifies the alternative path.

Fixes

• Resolved an issue with the lazy iterator for variables, which caused cyclic references.
• Fixed an error when reading CSV files from givennamegenerator and familynamegenerator.
• Addressed a problem with CSV export when using Faker data.
• Fixed an issue where accessing data with numeric keys in base properties caused errors.

Features

• ✨ Global Project: Introduced the ability to create a global project that can be added as a dependency to user projects, allowing for shared data, such as Python libraries, custom components, and data mappings.
• ✨ Improved Dynamic Selectors: Enhanced the efficiency of using variables in all types of selectors, including selector and iterationSelector.
• ✨ Condition Nodes: Redesigned condition nodes to support more complex conditional logic.
• ✨ Default Value: Added the defaultValue attribute to keys, allowing a default value to be set when the key's value is null or not specified.
• ✨ Complex Conditions: Enabled more advanced conditional logic by introducing <if-else> and <else> nodes, which can be used to wrap keys.
• ✨ More Demos: Added two new demos, Condition Demo and Selector Demo, showcasing the new condition handling in keys and nodes, as well as dynamic selectors.
• ✨ Datetime Epoch: Added support for handling epoch transformations at the level of seconds, milliseconds, microseconds, and nanoseconds using outDateFormat and inDateFormat.
• ✨ Datetime Precision: Introduced the ability to set the precision of datetime transformations to seconds, milliseconds, microseconds, and nanoseconds via outDateFormat and inDateFormat.
• ✨ Data Distribution: Added new options for data distribution from a source, allowing for ordered or random distribution through the distribution attribute.
• ✨ Simple Mode: Reworked and improved the UI for Simple Mode, making it more user-friendly.
• ✨ Data Access: Added the ability to access data from sibling <generate> nodes by loading them into a variable using the source attribute.

Refactors

• 🔥 Project View: Refactored the project view to enhance usability and design.
• 🔥 Demo Store: Improved the demo store’s usability and design through a comprehensive refactor.
• 🔥 Code Cleanup: Performed general code cleanup, including refactoring redundant code and improving code readability.

Docs

• 📝 Updated the documentation to include details on updating the person entity information in generators.
• 📝 Extended documentation for the <target> element.
• 📝 Enhanced documentation for UI project general settings.
• 📝 Expanded documentation on data distribution methods.
• 📝 Added comprehensive documentation covering the new features and changes introduced in this release.

1.2.1

Documentation

• Project Settings: Update documentation for general project settings.

Refactors

• Preview UI: Update color for Tables, Json, Log View in Dark mode to keep it consistent with background color.

Fixes

• Environment file parsing: Fixed an issue where the environment file was not parsed correctly.
• Kafka SSL: Add additional Kafka SSL configuration to change the SSL Truststore root certificate alias.
• MongoDB: Add additional MongoDB configuration to configure the authentication mechanism and the authentication source.
• MongoDB: Fixed an issue where the MongoDB queries where not evaluated dynamically.

1.2.0

• Redux Management: Remove Redux packages.
• Enhancement: Cleanup temporary task directory in deployed environments.
• Task/Admin Task View: Show Started time as user local date time.
• MongoDB username, password: Handle special character in MongodDB username and password.
• Playwright: Add User Interface Test to ensure users having best experience in DATAMIMIC.

Fixes

• Error Handling: Enhanced error handling across all Rust components for improved stability.
• Redis Retrieval: Resolved issues with retrieving results from Redis for media previews.
• MongoDB Authentication: Fixed MongoDB authentication issues related to special characters in usernames.
• Task Directory Cleanup: Implemented logic to clean up task directories when not running locally.
• KeyTool: Added missing KeyTool to worker containers to support proper authentication handling for Kafka environments.
• Project Download: Fixed missing project name in the download file archive.
• Codemirror Linter: Optimized linter to check wrong child for echo cases.
• Preview View: Optimized the height of table.
• Simple Mode: Optimized the dragging and dropping feature causes element removal or strange element combination.
• Simple Mode: Optimized wrong child elements in Simple Mode which are imported from Advanced Mode.

Upgrades

• State Management: Transitioned from Redux to React Query for more efficient data fetching and state management.
• Demo Enhancements: Improved demo icons and error handling mechanisms by implementing a default icon for missing demo icons and enhancing error handling for faulty Demo TOML files.
• React 18 Migration: Migrate React from version 17 to 18.

Features

• ✨ Object-Store: Implemented support for object storage environments, enabling reading/importing and writing/exporting data from/to object storages like S3, Azure Blob, and Minio.
• ✨ Exporter: Implement JSONSingle Exporter for exporting entities into separate JSON files.
• ✨ Pattern: Improved error handling for patterns and enhanced migration guidance.
• ✨ Custom Components: Added the ability to incorporate custom Python components such as generators and converters.
• ✨ Kafka: Implemented support for more advanced Kafka configurations.
• ✨ Generator: Developed a TableSequenceGenerator for generating unique sequences in tables, useful when other services access the same table or when tables have unique constraints with pre-existing data.
• ✨ Demo: Created a new demo custom component for Python to showcase features and provide a starting point for users.
• ✨ Demo: Introduced a new demo for complex database mapping scenarios using custom converters and database system connections.
• ✨ Simple Mode: Styled search input as sticky element in dialogs' combobox.
• ✨ Simple Mode: Add multi select feature when adding or editing target property.

Refactors

• 🔥 Logging: Refactored console logging to use console.debug for better log verbosity control.
• 🔥 Code Cleanup: Performed general code cleanup, including refactoring redundant code and improving code readability.
• Playwright: Add User Interface Test to ensure users having best experience in DATAMIMIC.

Docs

• 📝 Migration Guide: Extended guidance on migration warnings to assist users during transitions.
• 📝 Feature Guides: Added new feature guides for Object-Store, Custom Components, and Kafka configurations.
• 📝 Login: Removed recovery account troubleshooting

1.1.0

Fixes

• Demo Store: Added a process to ensure the demo store and template are always up to date during the startup process.
• Project: Displayed the correct local date and time when creating a project.

Upgrades

• Access Token: Increased the maximum token expiry from 30 days to 365 days.
• Simple Mode: Retrieve data from IndexedDB instead of directly from the API.

Features

• ✨ Processing: Enhanced processing with the introduction of pageSize and defaultPageSize, and improved process logging.
• ✨ Notification: Improved handling for independently removing notification items.
• ✨ Simple Mode: Added Enable/Disable Element and Drag-and-Drop features.
• ✨ Kafka: Implemented SASL_SSL properties.
• ✨ Project View: Add new UI for project base on card design.
• ✨ Get Project List API: Add update and create time for project read.
• ✨ Demo Store: Redesign demo store cards to keep the style consistent.

1.0.1

Fixes

• Environment SQLite: Changed path of temporary SQLite during task execution to support OpenShift constraints
• Demo Store: Template update options are no longer shown to users.
• UI: Fix color of Reset Project Dialog in Dark Mode.
• Reset Project From Task: Reset project from Task won't show the correct files state.

Features

• ✨ Core: Enhanced handling of task memory limits.
• ✨ System Projects: Users can now be assigned to non-owned projects.

1.0.0

Features

• ✨ Initial release: DATAMIMIC: AI-Driven, Model-Based Data Generation with Advanced JSON/XML Handling for Efficient, Compliant Development.
• ✨ Introduce new auto model creation for JSON, enabling users to generate complex, nested JSON structures effortlessly.
• ✨ Introduce new auto model creation for XML, simplifying the creation of intricate XML data models.
• ✨ Embrace a Model-Driven Approach for simplified maintenance and abstract data processing, enhancing adaptability and consistency in test data generation.
• ✨ Implement Abstract Data Transformation to transform various inputs into flexible entities for diverse outputs like SQL, XML, JSON, CSV, and more.
• ✨ Advanced JSON Handling for expert manipulation of nested JSON structures, supporting hierarchical data modeling and deep nesting capabilities.
• ✨ Introduce Data Anonymization and Pseudonymization features for field-level obfuscation, ensuring GDPR compliance and data privacy.
• ✨ AI-Enhanced Data Generation employing advanced algorithms for realistic and diverse data sets.
• ✨ Achieve High Performance with optimizations for handling large data volumes, suitable for processing millions of transactions in real time.
• ✨ Provide an Intuitive User Experience with a user-friendly interface and clear documentation.
• ✨ Offer Flexible and Scalable solutions for projects of various sizes and complexities.
• ✨ Extend Functionality with an Extensive Function Library for realistic data creation, supporting custom code extensions.
• ✨ Ensure Seamless Integration with development, testing environments, and CI/CD pipelines.

Docs

• 📝 Add initial docs in English language, covering the growing spectrum of DATAMIMIC features and user guidance.

Translations

• 🌐 Add German translation for docs/de/docs/index.md, docs/de/docs/features.md.
• 🌐 Add Vietnamese translation for docs/de/docs/index.md, docs/de/docs/features.md.

Template for release note sections

### Breaking changes

* ⬆️ n/a.

### Fixes

* n/a.

## Upgrades

* n/a.


### Features

* ✨ Introduce new auto model creation for JSON.
* ✨ Introduce new auto model creation for XML.

### Refactors

* 🔥 n/a.

### Docs

* 📝 Add initial docs in english language.

### Translations

* 🌐 Add german translation for `docs/de/docs/features.md`.