Skip to content

Inject Dynamic Properties When Generate

When integrating DATAMIMIC into CI/CD pipelines, you might encounter a challenge: running the same project in parallel with different parameters. While you could modify project files directly, this approach creates race conditions and persistence issues in parallel execution environments.

The Dynamic Properties Injection feature solves this problem by allowing you to inject custom property values during the generation process without modifying the project files permanently.

The Problem

Consider these scenarios:

  • Parallel Testing: Running the same test suite across multiple branches or environments simultaneously
  • Load Testing: Executing the same data generation model with different volume parameters
  • Feature Flags: Testing different feature configurations without creating separate projects
  • Environment-specific Values: Using different database connections or API endpoints per environment

Without dynamic properties injection, you'd need to:

  1. Create separate projects for each configuration (maintenance overhead)
  2. Modify files via API before each run (race conditions in parallel executions)
  3. Use complex templating mechanisms (increased complexity)

The Solution

Dynamic Properties Injection allows you to:

  1. Keep projects unchanged: The original project files remain untouched
  2. Inject values per execution: Each API call can provide different property values
  3. Maintain parallel safety: No race conditions between concurrent executions
  4. Preserve execution isolation: Each run gets its own temporary property values

How It Works

The process follows these steps:

  1. Setup Phase: Create a properties file in your project (e.g., runtime.properties)
  2. Configuration Phase: Include this properties file in your main model with highest priority
  3. Execution Phase: Call the generate API with custom property values
  4. Injection Phase: DATAMIMIC creates a temporary copy of your project and overwrites the properties file with your provided values
  5. Processing Phase: The generation runs with your custom values
  6. Cleanup Phase: Temporary files are cleaned up, original project remains unchanged

Project Setup

Step 1: Create Properties File

Create a properties file in your project's conf folder. The file name can be anything but must end with .properties.

conf/runtime.properties
1
2
3
4
5
# Default values for runtime parameters
count=1000
user_segment=basic
feature_premium=false
database_timeout=30

Step 2: Configure Your Main Model

Include the properties file in your main DATAMIMIC model (datamimic.xml) as the last include to ensure it takes the highest priority:

datamimic.xml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
<setup>
    <!-- Include base configuration first -->
    <include uri="conf/base.properties"/>

    <!-- Include other configurations -->
    <include uri="conf/database.properties"/>

    <!-- Include runtime properties LAST for highest priority -->
    <include uri="conf/runtime.properties"/>

    <!-- Your data generation models -->
    <include uri="models/generate_users.xml"/>
    <include uri="models/generate_orders.xml"/>
</setup>

Include Order Matters

Properties files included later override values from earlier includes. Always include your runtime properties file as the last configuration include.

Step 3: Use Properties in Your Models

Reference the properties in your data generation models:

models/generate_users.xml
1
2
3
4
5
6
7
8
<setup>
    <generate name="users" count="{count}">
        <key name="user_id" generator="IncrementGenerator"/>
        <key name="user_segment" values="'{user_segment}'"/>
        <key name="is_premium" values="'{feature_premium}'"/>
        <key name="created_at" generator="DateTimeGenerator"/>
    </generate>
</setup>

API Usage

Basic Example

Once you have set up your DATAMIMIC project in the UI, you can use our API to trigger the data generation process and supply custom property values dynamically.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
curl -X POST "https://your-datamimic-instance/api/v1/{project-id}/generate" \
  -H "Authorization: Bearer {your-access-token}" \
  -H "Content-Type: application/json" \
  -d '{
    "task_type": "standard",
    "custom_property_file": {
      "name": "runtime.properties",
      "data": {
        "count": 5000,
        "user_segment": "premium",
        "feature_premium": true,
        "database_timeout": 60
      }
    }
  }'

After submitting your request, you will receive a task ID that you can use to track the status of your data generation job and retrieve the resulting artifacts. Read more about how to use the task ID to get the result in Project Access Token guide.

With this approach, you can run the same project with different configurations in parallel, ensuring each run has its own isolated property values. For every request, you can provide a different set of properties to generate different data.

Use different model file in each request

You can easily use different model file in each request by providing the model file name in the request. Let's say you have two model files in your project: models/load_performance.xml and models/integration_test.xml.

You want to include different model file depends on the request. For example, you want to use models/load_performance.xml for load performance test and models/integration_test.xml for integration test.

You can use the following approach to achieve this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
<!-- datamimic.xml -->
<setup>
    <!-- Include the dynamic model file -->
    <include uri={dynamic_model_path}/>
    <!-- Your data generation models -->
    <include uri="models/generate_users.xml"/>
    <include uri="models/generate_orders.xml"/>

    <include uri="conf/dynamic_model_path.properties"/>
</setup>

Your conf/dynamic_model_path.properties file should look like this:

1
dynamic_model_path="models/load_performance.xml"

Then you can use the following API request to generate data with the dynamic model file:

For your load performance test, you can use the following API request:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
curl -X POST "https://your-datamimic-instance.com/api/v1/projects/your-project-id/generate" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "task_type": "standard",
    "custom_property_file": {
      "name": "runtime.properties",
      "data": {
        "dynamic_model_path": "models/load_performance.xml"
      }
    }
  }'

For your integration test, you can use the following API request:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
curl -X POST "https://your-datamimic-instance.com/api/v1/projects/your-project-id/generate" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "task_type": "standard",
    "custom_property_file": {
      "name": "runtime.properties",
      "data": {
        "dynamic_model_path": "models/integration_test.xml"
      }
    }
  }'

These request can be executed in parallel, and you will get different data generated for each request.

Troubleshooting

Common Issues

1. Properties File Not Found

Error: FileNotFoundError: runtime.properties not existed

Solution: The properties file must exist in your project before using dynamic injection.

1
2
3
4
5
6
7
8
9
# Ensure the file exists in your project
curl -X POST "https://your-datamimic-instance.com/api/v1/projects/your-project-id/files" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "runtime.properties",
    "folder": "conf",
    "content": "count=1000\nuser_segment=basic\n"
  }'

2. Property Values Not Applied

Problem: Generated data doesn't reflect the injected property values.

Solutions:

  1. Check include order: Ensure runtime properties are included last
  2. Verify property names: Ensure property names match exactly between the file and your models
  3. Check syntax: Ensure property references use correct syntax: {property_name}

3. Invalid Property File Name

Error: ValidationError: The file name must end with .properties

Solution: Ensure the file name ends with .properties:

1
2
3
4
5
6
{
  "custom_property_file": {
    "name": "runtime.properties",  // ✅ Correct
    "data": { ... }
  }
}

Debugging Tips

  1. Check task logs: Use the task ID from the API response to check detailed logs
  2. Test with minimal properties: Start with a single property to isolate issues
  3. Validate JSON: Ensure your API request JSON is valid
  4. Check file permissions: Ensure the properties file is readable in your project

Conclusion

Dynamic Properties Injection is a powerful feature that enables safe, parallel execution of DATAMIMIC projects with varying configurations. By following the patterns and best practices outlined in this guide, you can:

  • Eliminate race conditions in parallel CI/CD environments
  • Maintain clean, reusable project configurations
  • Implement flexible testing strategies
  • Scale data generation across multiple scenarios simultaneously

The key to success is proper project setup with strategic property file inclusion and careful API integration with proper error handling and monitoring.

For questions or advanced use cases, please contact our support team.