Skip to content

operate Element

The <operate> element performs batch operations on templates for data transformation. It supports both XML (using XPath) and JSON (using JSONPath) templates.

Note

For JSON, create-missing operations use a strict object-path subset of JSONPath, not the full JSONPath query language.

Overview

The operate element reads operation definitions from a CSV file and applies them to templates, enabling: - Batch data transformations - Template modifications before data generation - Dynamic value injection using generators - Complex structural operations

Attributes

Attribute Required Description Default
source Yes CSV file path containing operations and models -
operation_prefix No Prefix for operation column names in CSV op
template_not_found_action No Action when template is missing: warn, error, ignore warn
operation_not_matched_action No Action when operation is invalid: warn, error, ignore warn

CSV File Format

The CSV file must contain: - id: Unique identifier for each artifact - template: Path to the template file - Operation columns: Named with the operation_prefix (e.g., op1, op2, op3)

Operation Types

Basic Operations

set

Updates or creates a value at the specified path.

XML Example: 

1
2
id|template|op1|op2
user001|user.xml|set(//user/name,John Doe)|set(//user/age,30)

JSON Example: 

1
2
id|template|op1|op2
user001|user.json|set($.user.name,John Doe)|set($.user.age,30)

delete

Removes a node/property at the specified path.

XML Example: 

1
2
id|template|op1
user001|user.xml|delete(//user/email)

JSON Example: 

1
2
id|template|op1
user001|user.json|delete($.user.email)

Advanced Structural Operations

XML-Specific Operations

  • replaceChildren: Replace all children of a node 

    1
2
    op1
replaceChildren(//root/data,<item>New Item</item>)

  • appendChild: Add a child to a node 

    1
2
    op1
appendChild(//root/metadata,<updated>2025-07-17</updated>)

  • upsertNode: Update or insert a node 

    1
2
    op1
upsertNode(//root/status,active)

  • mergeAttributes: Merge attributes into a node 

    1
2
    op1
mergeAttributes(//root,id="test-123" enhanced="true")

  • clearChildren: Remove all children from a node 

    1
2
    op1
clearChildren(//root/temp)

  • upsertAttribute: Update or insert an attribute 

    1
2
    op1
upsertAttribute(//root,version="2.0")

JSON-Specific Operations

  • mergeObject: Merge objects 

    1
2
    op1
mergeObject($.config,{"timeout": 30, "ssl": true})

  • upsertProperty: Update or insert a property 

    1
2
    op1
upsertProperty($.metadata.version,2.0)

  • replaceArray: Replace an entire array 

    1
2
    op1
replaceArray($.items,[{"id": 3, "name": "New Item"}])

  • appendItem: Add item to array 

    1
2
    op1
appendItem($.tags,new_tag)

  • clearArray: Empty an array 

    1
2
    op1
clearArray($.temp)

Dynamic Value Injection

Use generators and variables within operations:

1
2
id|template|op1|op2|op3
gen001|user.xml|set(//user/name,{GivenNameGenerator})|set(//user/id,{IncrementGenerator})|set(//user/email,{EmailAddressGenerator})

Complex Examples

Example 1: Basic XML Operations

Setup (datamimic.xml): 

1
2
3
<setup defaultSeparator="|">
    <operate source="data/operations.opctl.csv" operation_prefix="op"/>
</setup>

CSV (operations.opctl.csv): 

1
2
3
id|template|op1|op2|op3
user001|templates/user.xml|set(//user/name,John Doe)|set(//user/age,30)|delete(//user/temp)
company001|templates/company.xml|set(//company/name,TechCorp)|set(//company/employees,150)|upsertAttribute(//company,status="active")

Example 2: JSON with Complex Paths

1
2
3
id|template|op1|op2|op3
json001|config.json|set($.users[0].name,Alice)|set($.users[*].active,true)|delete($.deprecated)
json002|config.json|set($..id,GLOBAL-ID)|set($.users[?(@.active)].verified,true)|mergeObject($.settings,{"theme":"dark"})

Example 3: Healthcare Business Domain

Setup: 

1
2
3
4
5
6
7
8
<setup>
    <execute uri="scripts/healthcare_functions.scr.py"/>
    <variable name="patient" entity="Person(dataset='US', min_age=0, max_age=95)"/>
    <variable name="patientId" script="generate_patient_id()"/>

    <operate source="data/patient_records.opctl.csv" 
            operation-prefix="op"/>
</setup>

CSV: 

1
2
id|template|op1|op2|op3|op4
patient_001|patient_record.xml|set(//patient/id,{patientId})|set(//patient/name,{patient.given_name} {patient.family_name})|set(//patient/bloodType,{bloodType})|set(//diagnosis/code,{diagnosis.code})

Example 4: Error Handling

1
2
3
4
5
6
<setup>
    <operate source="data/operations.csv" 
             operation_prefix="op"
             template_not_found_action="error"
             operation_not_matched_action="warn"/>
</setup>

Best Practices

  1. Operation Order: Plan operations carefully as they are applied sequentially
  2. Avoid modifying nodes after deletion

  3. Create parent nodes before children

  4. Path Validation: Ensure paths exist before operations

  5. Use upsert operations for uncertain paths

  6. Handle missing paths gracefully with error actions

  7. Performance: For large datasets

  8. Group related operations together
  9. Use batch operations when possible

  10. Consider template complexity

  11. Error Handling:

  12. Set appropriate template_not_found_action based on requirements
  13. Use warn for development, error for production

  14. Monitor logs for operation warnings

  15. Maintainability:

  16. Use descriptive IDs in CSV files
  17. Document complex operations
  18. Keep templates organized in logical directories

Logical Validation

The operate system detects and reports invalid operation sequences: - Operations on deleted nodes - Conflicting modifications - Invalid paths after structural changes

Integration with Generators

Operate integrates with DATAMIMIC generators: - Use generator expressions in set operations - Combine with variables for dynamic content - Support for all DATAMIMIC generator types

Examples by Use Case

E-commerce Product Catalog

1
2
id|template|op1|op2|op3
prod001|product.xml|set(//product/name,{ProductNameGenerator})|set(//product/price,{RandomDoubleGenerator(10,1000)})|set(//product/category,Electronics)

Financial Services

1
2
id|template|op1|op2|op3
loan001|loan_app.xml|set(//loan/applicant,{PersonGenerator})|set(//loan/amount,{RandomIntegerGenerator(1000,100000)})|set(//loan/status,pending)

Government Permits

1
2
id|template|op1|op2
permit001|permit.xml|set(//permit/id,PERMIT-{IncrementGenerator})|set(//permit/issueDate,{DateTimeGenerator})