DATAMIMIC

DATAMIMIC: AI-Driven, Model-Based Data Generation and Privacy
with Advanced JSON/XML Handling for Efficient, GDPR Compliant Development.

Documentation: https://docs.datamimic.io

Community Project: https://github.com/rapiddweller/datamimic

Enterprise Edition: https://datamimic.io

DATAMIMIC is the AI-driven, model-based toolkit for efficient and compliant test data generation, with advanced JSON/XML handling capabilities.

The key features of DATAMIMIC are:

Model-Driven Approach: A structured, model-based framework for precision in test data generation.
Versatile Data Transformation: In the DATAMIMIC model, all input data are transformed into abstract entities, enabling flexible provisioning into various formats like SQL databases, XML, JSON, CSV, EDIFACT and many more.
Abstract Data Transformation: DATAMIMIC transforms various inputs into flexible entities for diverse outputs like SQL, XML, JSON, CSV, and more.
Advanced JSON/XML Handling: Expert handling of complex data structures.
Data Anonymization and Pseudonymization: Effortlessly configure field-level obfuscation ensuring ease of maintenance and adaptability.
GDPR Compliance: In-built features ensuring data privacy and protection.
AI-Enhanced Data Generation: Advanced algorithms and data analysis features to create realistic and diverse data sets.
High Performance: Utilizing the latest advancements in Python and Rust in DATAMIMIC's high-scalable and high-speed processing core, optimized for handling large data volumes efficiently and ensuring unmatched efficiency and robustness.
Streamlined Agile and TDD: Elevate your Agile and Test Driven Development processes with tailored, on-demand test data generation, supporting rapid development cycles without compromising data quality or security.
Intuitive User Experience: User-friendly interface and clear documentation.
Flexible and Scalable: Suitable for various project sizes and complexities.
Extensive Function Library: Wide range of in-built generators and functions for realistic data creation.
Seamless Integration: Easy integration with existing development, testing environments and CI/CD.

Opinions¶

"We've been using DATAMIMIC for our test data needs, and it has exceeded our expectations. The model-driven approach makes it easy to adapt to various data structures. Its advanced JSON and XML handling capabilities have been invaluable for our NoSQL projects. Plus, the user-friendly interface ensures our team can hit the ground running."

Tim Lafferenz - All4Labels

Requirements¶

CE: Python 3.11
EE: Modern web browser - DATAMIMIC UI is available as SaaS or can be deployed on-premise (Docker, Podman, Kubernetes, OpenShift, Helm)

Example UI test data generation (EE)¶

Click Clone in the tile of 'Basic Script' to create your first project from Demo Store.

Generate it¶

Start your first DATAMIMIC task by hitting the button 'GENERATE'.
The status window informs you about the processing progress.

Check it¶

Click 'Previews' to see a preview of the data being created.
Or click 'Logs' to get detailed insights about the task, its processing speed, throughput's and more.
Or navigate to 'Tasks' to get an overview of all task executions and its status of your project.

Easily switch between the main views (Editor, Jobs, Tasks, Settings) in the project bar.

Download it¶

Navigate to 'Tasks'.
Click the 'Artifact' Icon and select the generated file(s) on the Artifact View for download.

Example DATAMIMIC model¶

In DATAMIMIC, every project begins with an XML-based main model. These models can be auto-generated from connected databases, JSON, XML, or other file types. In this example, we'll start with a basic script example to illustrate how DATAMIMIC's model works.

<setup>
    <generate name="datamimic_user_list" count="100" target="CSV,JSON,XML">
        <variable name="person" entity="Person(min_age=18, max_age=90, female_quota=0.5)"/>
        <key name="id" generator="IncrementGenerator"/>
        <key name="first_name" script="person.given_name"/>
        <key name="last_name" script="person.family_name"/>
        <key name="gender" script="person.gender"/>
        <key name="birthDate" script="person.birthdate" converter="DateFormat('%d.%m.%Y')"/>
        <key name="email" script="person.family_name + '@' + person.given_name + '.de'"/>
        <key name="ce_user" values="True, False"/>
        <key name="ee_user" values="True, False"/>
        <key name="datamimic_lover" constant="DEFINITELY"/>
    </generate>
</setup>

The above is a DATAMIMIC model for generating a dataset named "datamimic_user_list." This dataset will contain 100 records, and its target formats include CSV, JSON, and XML. Let's break down the key components of this script:

<setup>: This node encloses each DATAMIMIC model and can be used for more advanced configurations.
<generate>: This node defines the dataset to be generated, specifying its name, record count, and target formats.
<variable>: Here, we define a variable named "person" and associate it with the "Person" entity. This entity is configured to generate data for individuals with age between 18 and 90, with a 50% female quota.
<key>: Multiple key elements represent the attributes of each record. For example:
- "id" is generated using the IncrementGenerator.
- "first_name" is generated using the value given_name from the Person object stored in the <variable\>.
- "last_name" is generated using the value family_name from the Person object stored in the <variable\>.
- "gender" is generated using the value gender from the Person object stored in the <variable\>.
- "birthDate" is generated using the value birthdate from the Person object stored in the <variable\> and formatted with the "DateFormat" converter.
- "email" is generated using a script that combines "family_name" and "given_name" to create an email address.
- "ce_user" and "ee_user" are generated with predefined values "True" and "False" that are randomly distributed.
- "datamimic_lover" is set as a constant value "DEFINITELY".

This very simple example demonstrates how DATAMIMIC allows you to define and customize data generation models with precision and flexibility, making it a powerful tool for generating test data for various purposes.

For a more complete example including further <nodes\> and features such as database connectivity, complex JSON modelling and obfuscation scenarios, see the Tutorial - User Guide.

Further Support¶

Ready to Take the Next Step?

Whether you're exploring DATAMIMIC for the first time or you're already enhancing your data management with our tool, we're here to support your journey.

For New Customers:

If you're considering DATAMIMIC for your organization, discover how our tool can revolutionize your development, testing, and training processes. Book a demo to see DATAMIMIC's features in action, tailored to your unique needs.

Book a Demo

For Existing Customers:

Need further assistance or want to deep dive into more advanced features? Our support team is ready to help you navigate through any challenges or queries you might have. Create a support ticket or book a dedicated support session to get personalized assistance.

Explore More:

Visit www.datamimic.io for additional services, detailed use cases, and to make your data generation project a success. Our premium features and resources are designed to ease your start into complex scenarios and enhance your data management capabilities.

For any questions or need for further clarification on any aspect of DATAMIMIC, our team is here to ensure your experience is as smooth and beneficial as possible.

Contact the DATAMIMIC Development Team

Remember, the journey to robust, efficient, and compliant data management begins with a single step. Let rapiddweller be your partner in data generation, obfuscation, and migration, every step of the way.