Skip to content

Data Definition Model - Advanced Elements

This document covers advanced features and elements in DATAMIMIC's data definition models. Make sure you're familiar with the Core Data Definition Elements before diving into these advanced features.

Complex Data Structures

<nestedKey>

The <nestedKey> element defines nested key fields and their generation methods within a data generation task. It allows you to structure complex data in a hierarchical format, such as dictionaries (dict) or lists (list), and control its content dynamically.

See the detailed reference.

<reference>

The <reference> element allows referencing other generated data.

Attributes

  • name: Specifies the name of the reference.
  • source: Specifies the source of the reference.
  • sourceType: Specifies the type of the source.
  • sourceKey: Specifies the key of the source.
  • unique: Ensures the reference is unique.

<list>

The <list> element defines a collection of data items, where each item can contain its own attributes, keys, and arrays. Lists are useful for representing structured data, such as rows in a table, or collections of objects with shared attributes. The <list> can contain multiple <item> elements that represent individual entries.

See the detailed list element reference.

<item>

See the detailed item element reference.

<array>

The <array> element defines arrays of data items that can be either statically defined or generated dynamically using scripts. Arrays are essential when you need to generate multiple items of the same data type, such as lists of values, and can be combined with other elements like <list>, <key>, and <nestedKey> to create complex data structures.

See the detailed array element reference.

Control Elements

<condition>

The <condition> element is used to execute a set of child tags (<if>, <else-if>, and <else>) based on specific logical conditions. It provides a way to control the data generation process by applying conditions that determine which elements will be included in the output.

Structure

  • The <if> tag is always the first child of a <condition> element and defines the primary condition.
  • Zero or more <else-if> tags can follow the <if>, each specifying additional conditions to evaluate if the previous conditions are not met.
  • The <else> tag is optional and provides a fallback action if none of the preceding conditions are met.

Rules

  • A <condition> element must have one <if> tag.
  • There can be zero or more <else-if> tags.
  • Only one <else> tag is allowed, and it must appear as the last child of the <condition> element.

Children

  • if: Defines the primary condition to evaluate.
  • else-if: Defines additional conditions to check if the previous conditions are false.
  • else: Fallback action if none of the conditions are met.

Attributes

  • condition: A Python-like expression that evaluates to True or False. The result determines whether the content inside the corresponding tag will be executed.

Example 1: Simple Condition with If-Else Logic

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
<condition>
    <if condition="category_id == 'DRNK/ALCO'">
        <echo>Category: DRNK/ALCO</echo>
    </if>
    <else-if condition="category_id == 'FOOD/CONF'">
        <echo>Category: FOOD/CONF</echo>
    </else-if>
    <else>
        <echo>Category not found</echo>
    </else>
</condition>

In this example, based on the value of category_id, the appropriate message is printed.

  • If category_id is 'DRNK/ALCO', the first <if> block is executed.
  • If category_id is 'FOOD/CONF', the <else-if> block is executed.
  • If neither condition is met, the <else> block is executed.

Example 2: Complex Condition with Nested Keys and Default Values

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
<setup>
    <generate name="group_name_not_override" count="10">
        <variable name="ifVar" generator="BooleanGenerator"/>
        <variable name="elseIfVar" generator="BooleanGenerator"/>

        <condition>
            <if condition="ifVar">
                <key name="if_true" constant="true"/> <!-- Generated if ifVar = True -->
            </if>
            <else-if condition="elseIfVar">
                <key name="else_if_true" constant="true"/> <!-- Generated if ifVar = False and elseIfVar = True -->
            </else-if>
            <else>
                <key name="else_true" constant="true"/> <!-- Generated if ifVar = False and elseIfVar = False -->
            </else>
        </condition>
    </generate>
</setup>

In this example, two variables (ifVar and elseIfVar) are generated using a Boolean generator, and depending on their values, one of the <key> elements (if_true, else_if_true, or else_true) will be generated:

  • If ifVar is True, if_true is generated.
  • If ifVar is False and elseIfVar is True, else_if_true is generated.
  • If both conditions are False, else_true is generated.

Example 3: Conditions with Nested Structures

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<setup>
    <generate name="bike" count="10">
        <key name="id" type="int" generator="IncrementGenerator"/>
        <key name="year" type="int" values="1970, 2023"/>

        <!-- Conditional nested key generation -->
        <nestedKey name="condition_true" type="dict" condition="True">
            <key name="serial" type="int" condition="id % 2 == 1"/>
            <key name="count" type="int" generator="IncrementGenerator"/>
        </nestedKey>

        <condition>
            <if condition="True">
                <nestedKey name="if_true" type="dict">
                    <key name="id" type="int"/>
                </nestedKey>
            </if>
            <else-if condition="False">
                <nestedKey name="if_false" type="dict">
                    <key name="id" type="int"/>
                </nestedKey>
            </else-if>
        </condition>
    </generate>
</setup>

In this example:

  • The condition_true nested key block will always be executed because its condition is True.
  • The conditional block within <condition> executes the <if> block because its condition is True, while the <else-if> is ignored as it evaluates to False.

Example 4: Conditions with Default Values

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<setup>
    <generate name="condition" count="10">
        <key name="id" generator="IncrementGenerator"/>

        <condition>
            <if condition="id == 1">
                <echo>Condition met: id is 1</echo>
                <key name="if_true" constant="1"/>
            </if>
            <else-if condition="id == 3">
                <echo>Condition met: id is 3</echo>
                <key name="else_if_3_true" constant="3"/>
            </else-if>
            <else-if condition="id == 4">
                <echo>Condition met: id is 4</echo>
                <key name="else_if_4_true" constant="4"/>
            </else-if>
            <else>
                <echo>Condition not met, proceeding with default values</echo>
                <key name="else_true" constant="else_value"/>
            </else>
        </condition>
    </generate>
</setup>

Here, the conditional logic checks the value of id:

  • If id is 1, the if_true key is generated.
  • If id is 3 or 4, the appropriate else_if block is executed.
  • If no conditions match, the else_true key is generated with a fallback value.

Example 5: Conditional Removal of Elements

1
2
3
4
5
<setup>
    <generate name="group_name_not_override" count="10">
        <key name="removeElement" script="" defaultValue="{}" condition="False"/>
    </generate>
</setup>

In this case, the removeElement key will not be generated because the condition is set to False.

Example 6: Using Default Values in Conditional NestedKeys

1
2
3
4
5
6
7
<setup>
    <generate name="bike" count="10">
        <nestedKey name="condition_false" type="dict" condition="False" defaultValue="None">
            <key name="serial" type="int" condition="id % 2 == 1"/>
        </nestedKey>
    </generate>
</setup>
  • Since the condition is False, the condition_false nested key will take the defaultValue of None.

Best Practices for Using <condition>

  1. Use Conditions to Control Output: Conditions are an excellent way to control the generation of data elements dynamically based on the current state of variables.

  2. Fallbacks with Default Values: Use default values when you want to provide a fallback if a condition evaluates to False.

  3. Combine with Nested Structures: You can use conditions with nested keys, lists, and arrays to build complex logic-driven data models.

  4. Use else-if for Multiple Conditions: To handle multiple possible states, use a combination of <if>, <else-if>, and <else> to cover all scenarios.

<echo>

The <echo> element outputs text, variables, or expressions for logging, debugging, or monitoring purposes. This can be helpful in tracking the progress of data generation or inspecting the values of variables during runtime. It accepts dynamic content, including variables and expressions enclosed in {}.

Attributes

  • text: The static or dynamic content to be printed. Dynamic values are wrapped in {}, allowing you to output variables, expressions, or results of functions during execution.

Usage

  • Use <echo> to print out the values of variables or track the flow of the setup process.
  • It can output text to the console or log files, depending on the target defined in the setup.

Example 1: Basic Usage for Debugging

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
<setup>
    <variable name="user" source="dbPostgres" cyclic="False"
              selector="SELECT id, text FROM public.db_postgres_test_query_setup_context_variable"
              distribution="ordered"/>

    <variable name="all_users" source="dbPostgres" cyclic="False"
              iterationSelector="SELECT id, text FROM public.db_postgres_test_query_setup_context_variable"/>

    <echo>user is a DotableDict: {user}</echo>
    <echo>all_users is a list of dict: {all_users}</echo>
</setup>

In this example:

  • The <echo> tag prints the current value of the user variable, which is a DotableDict.
  • It also prints the all_users variable, which is a list of dictionaries retrieved from the database.

Example 2: Using Echo for Debugging Scripted Variables

1
2
3
4
5
6
7
<setup>
    <variable name="random_number" generator="RandomNumberGenerator(min=1, max=100)"/>
    <echo>Generated random number: {random_number}</echo>

    <key name="status" script="random_number > 50 ? 'High' : 'Low'"/>
    <echo>The status based on random_number is: {status}</echo>
</setup>

In this example:

  • A random number is generated and echoed for debugging.
  • The status is then calculated based on the random number, and the result is printed using <echo>.

Best Practices for Using <echo>

  1. Debug Complex Logic: Use <echo> to debug variables and complex expressions, especially when using scripts or database sources to generate data dynamically.
  2. Monitor Data Generation: Track the progress of your data generation by echoing values at key points in your setup.
  3. Combine with Variables: You can use variables and expressions within the text of <echo> to output dynamic content during the generation process.

<generator>

The <generator> element specifies custom generators for data generation.

Attributes

  • name: Specifies the name of the generator.
  • generator: Specifies your custom generator.

Example

1
2
3
4
5
6
7
8
<setup>
    <generator name="my_custom_date_gen" generator="DateTimeGenerator(min='2010-08-01', max='2020-08-31', input_format='%Y-%m-%d')"/>
    <generate name="product">
        <key name="product_name" type="string"/>
        <key name="import_date" generator="my_custom_date_gen"/>
        <key name="export_date" generator="my_custom_date_gen"/>
    </generate>
</setup>

<element>

The <element> element specifies child elements within an XML node, making it useful for generating nested XML structures. The name of the <element> tag becomes an XML attribute, and the generated content becomes the attribute's value.

Attributes

  • name: Specifies the name of the child XML element or attribute. This is mandatory.
  • script: Specifies a script to dynamically generate the value of the element.
  • constant: A constant value for the element, if no dynamic generation is required.
  • values: A list of possible values to randomly select from, if applicable.

Example 1: Simple XML Generation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
<setup>
    <generate name="generate_xml" count="2" target="XML">
        <variable name="person" entity="Person"/>
        <key name="author" script="None">
            <element name="name" script="person.name"/>
            <element name="gender" script="person.gender"/>
            <element name="birthdate" script="person.birthdate"/>
        </key>
    </generate>
</setup>

In this example:

  • The author key creates an XML node, and each <element> within defines attributes (such as name, gender, birthdate) populated with the values generated from the person entity.

Example 2: NestedKey with Elements and Attributes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
<setup>
    <generate name="part_list" count="1" target="XML">
        <nestedKey name="book" type="list" count="4">
            <element name="title" values="'Book 1', 'Book 2', 'Book 3', 'Book 4'"/>
            <element name="language" values="'de', 'en'"/>
            <element name="pages" generator="IntegerGenerator(min=100,max=800)"/>
            <element name="release_date" generator="DateTimeGenerator(min='2020-01-01', max='2023-12-31', input_format='%Y-%m-%d')"/>
        </nestedKey>
    </generate>
</setup>

Here:

  • The book list generates multiple XML elements (title, language, pages, release_date), each containing attributes for various books.
  • The content for each element is dynamically generated based on values or generators.

Example 3: Generating XML with Nested Lists and Dictionaries

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
<setup>
    <generate name="part_dict" count="1" target="XML">
        <nestedKey name="book" type="dict">
            <key name="title" values="'Book 1', 'Book 2'">
                <element name="language" values="'de', 'en'"/>
            </key>
            <key name="pages" generator="IntegerGenerator(min=100,max=800)"/>
            <key name="release_date" generator="DateTimeGenerator(min='2020-01-01', max='2023-12-31', input_format='%Y-%m-%d')"/>
        </nestedKey>
        <nestedKey name="magazine" type="dict">
            <key name="title" values="'Magazine #1', 'Magazine #2'"/>
            <key name="language" values="'de', 'en'"/>
            <key name="pages" generator="IntegerGenerator(min=30,max=70)"/>
        </nestedKey>
    </generate>
</setup>

In this case:

  • Both books and magazines are generated as XML elements with dynamically generated attributes such as language, pages, and release_date.

Example 4: Generating XML with Arrays and Lists

1
2
3
4
5
6
<setup>
    <generate name="array_xml" count="2" target="XML">
        <array name="random_string" type="string" count="3"/>
        <array name="random_number" type="int" count="3"/>
    </generate>
</setup>

In this example:

  • Two arrays (random_string and random_number) are generated as XML elements, with each array containing 3 items.
  • This demonstrates how arrays can be incorporated into the XML generation process.

Example 5: Complex XML with Lists and NestedKeys

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
<setup>
    <generate name="list_xml" count="1" target="XML">
        <list name="detail">
            <item>
                <key name="number" type="int" constant="2"/>
            </item>
            <item>
                <key name="text" type="string"/>
            </item>
            <item>
                <nestedKey name="employees" type="list" count="2">
                    <key name="code" type="string"/>
                    <key name="age" values="25, 30, 28, 45"/>
                </nestedKey>
            </item>
        </list>
    </generate>
</setup>

This example shows how to generate a list of items, where each item can contain additional nested keys and sub-elements, demonstrating a hierarchical structure in XML.

Example 6: Generating XML from a Template

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
<setup>
    <generate name="product" source="data/user.template.xml" target="ConsoleExporter, XML">
        <nestedKey name="xga:dgu.gewerbemeldung.0230">
            <element name="new_key" constant="new_key_value"/>
            <nestedKey name="bn-g2g:nachrichtenkopf.g2g">
                <element name="new_key2" constant="new_key_value2"/>
                <key name="bn-g2g:identifikation.nachricht" constant="abc"/>
            </nestedKey>
        </nestedKey>
        <key name="additionalTag" constant="extra">
            <element name="new_key" constant="new_key_value"/>
            <element name="new_key1" constant="new_key_value1"/>
        </key>
        <list name="array">
            <item>
                <key name="additionalTag" constant="extra">
                    <element name="new_key" constant="new_key_value"/>
                </key>
            </item>
        </list>
    </generate>
</setup>

In this case:

  • The XML generation is based on a predefined template (data/user.template.xml), and the structure is dynamically extended using nestedKey, key, and list elements.
  • This demonstrates how to augment existing XML structures with new elements and attributes.

Best Practices for Using <element>

  1. Use <element> to Create Structured XML: The <element> tag is a flexible way to build structured XML documents where each element and its attributes can be dynamically generated or statically defined.
  2. Combine <element> with Other Elements: Use the <element> tag in combination with <list>, <array>, and <nestedKey> to create complex XML structures.
  3. Leverage Dynamic Scripts: Take advantage of the script attribute to dynamically generate values for the elements based on complex logic or external variables.
  4. Use constant for Fixed Values: For cases where an element's value should not change, use the constant attribute.

Data control elements

<sourceConstraints>

The <sourceConstraints> element validates and filters input source data based on specified conditions. It is applied at the beginning of the data generation process to ensure that only data meeting the required criteria is processed from the source file. This helps maintain data quality and prevents invalid or unwanted data from entering the generation pipeline.

Syntax

1
<sourceConstraints if="condition" require="expression"/>

Rules

  • <sourceConstraints> ONLY supports as direct child tag of <generate> or <nestedKey>.
  • Multiple <sourceConstraints> elements can be used within the same parent element.
  • It's a good practice to place <sourceConstraints> element as first subtag of <generate>/<nestedKey>.

Attributes

  • if: A Python-like expression that evaluates to True or False. The result determines whether the source constraint applies.
  • require: Expression that must evaluate to True to keep the source data. If False, the source data is filtered out.

Example 1: Simple sourceConstraints for generate element

1
2
3
4
5
6
7
8
9
<setup>
    <generate name="synthetic_customers" count="10000" pageSize="1000"
              source="script/person_data.json" cyclic="True">
        <sourceConstraints if="credit_score &lt; 600" require="risk_profile == 'High'"/>
        <sourceConstraints if="credit_score &gt;= 600 and credit_score &lt; 750" require="risk_profile == 'Medium'"/>
        <sourceConstraints if="credit_score &gt;= 750" require="risk_profile == 'Low'"/>
        <key name="id" generator="IncrementGenerator"/>
    </generate>
</setup>

Structure of person_data file's data:

1
2
3
4
5
6
7
8
{
  "firstname": "Charlie",
  "lastname": "Brown",
  "age": 61,
  "city": "New York",
  "credit_score": 707,
  "risk_profile": "Low"
}

In this example:

  • The XML gets data from source "script/person_data.json" to generate data.
  • <sourceConstraints> will filter that source data before generation.
  • The first <sourceConstraints> applies when credit_score field in source data is less than 600. It keeps data only if risk_profile is "High".
  • The second <sourceConstraints> applies when credit_score is between 600 and 750. It keeps data only if risk_profile is "Medium".
  • The third <sourceConstraints> applies when credit_score is 750 or higher. It keeps data only if risk_profile is "Low".
  • In the end, the input data from source will satisfy all the constraints that are placed in <sourceConstraints>.

Example 2: Constraints in cascade structure

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
<setup>
    <generate name="container" count="1">
        <generate name="synthetic_customers" count="10000" pageSize="1000" source="script/person_data.json" cyclic="True">
            <sourceConstraints if="credit_score &lt; 600" require="risk_profile == 'High'"/>
            <sourceConstraints if="credit_score &gt;= 600 and credit_score &lt; 750" require="risk_profile == 'Medium'"/>
            <sourceConstraints if="credit_score &gt;= 750" require="risk_profile == 'Low'"/>
            <key name="id" generator="IncrementGenerator"/>
        </generate>
    </generate>
</setup>

In this case:

  • The <sourceConstraints> affects only its direct parent element: <generate> name "synthetic_customers"

Example 3: Constraints for nestedKey element

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
<setup>
    <generate name="container" count="1">
        <nestedKey name="cyclic_true" source="script/person_data.json"
                   type="list" count="1000" cyclic="True">
            <sourceConstraints if="credit_score &lt; 600" require="risk_profile == 'High'"/>
            <sourceConstraints if="credit_score &gt;= 600 and credit_score &lt; 750" require="risk_profile == 'Medium'"/>
            <sourceConstraints if="credit_score &gt;= 750" require="risk_profile == 'Low'"/>
            <key name="id" generator="IncrementGenerator"/>
        </nestedKey>
    </generate>
</setup>

In this case:

  • The <sourceConstraints> affects only its direct parent element: <nestedKey> name "cyclic_true"
  • This structure is useful for filtering data within deeply nested lists or structures.

Best Practices for Using <sourceConstraints>

  1. Filter Unwanted Data Early: Use <sourceConstraints> to ensure only relevant or valid data is used from the source file. This helps reduce errors and improves data quality during generation.

  2. Use Multiple Constraints: You can use multiple <sourceConstraints> elements to apply different filtering conditions based on various criteria.

  3. Place Constraints First: It's recommended to place <sourceConstraints> as the first child element of <generate> or <nestedKey> for optimal performance.

<targetConstraints>

The <targetConstraints> element filters out records that don't meet the required conditions. Applied at the end to validate and filter output data.

Syntax

1
<targetConstraints if="condition" require="expression"/>

Parameters

  • if: Boolean condition that determines when the target constraint applies
  • require: Expression that must evaluate to True to keep the generated record. If False, the record is filtered out from the final output

Usage Examples

Example 1: Basic Filtering

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
<!-- Filter out customers with poor credit and high loan amounts -->
<targetConstraints if="credit_category == 'Poor' and max_loan > 50000" require="False" />

<!-- Filter out customers with extreme debt-to-income ratio -->
<targetConstraints if="monthly_payment > income * 0.28" require="False" />

<!-- Apply final eligibility check -->
<targetConstraints if="credit_score < 500" require="False" />

<!-- Only keep customers with sufficient income -->
<targetConstraints if="income >= 40000" require="True" />
<targetConstraints if="income < 40000" require="False" />

Example 2: Complex Filtering Conditions

1
2
3
4
5
<!-- Keep only approved customers -->
<targetConstraints if="credit_limit >= 25000 and interest_rate <= 0.08" require="approval_status == 'Approved'" />

<!-- Filter out denied customers -->
<targetConstraints if="credit_limit <= 5000 and interest_rate >= 0.12" require="approval_status != 'Denied'" />

Example 3: Simple Boolean Check

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
<setup>
    <generate name="customers_name" count="1000" target="JSON">
        <key name="id" generator="IncrementGenerator" />
        <key name="name" type="string" values="'HARRY', 'MARK'" />
        <key name="targetConstraints" type="bool" constant="False"/>

        <mapping if="name == 'HARRY'" set="targetConstraints = True"/>
        <targetConstraints if="id <= 50" require="targetConstraints == True"/>
    </generate>
</setup>

In this case:

  • Records with id <= 50 are only kept if targetConstraints == True
  • Since targetConstraints is set to True only when name == 'HARRY', only Harry records with id <= 50 are kept
  • Mark records with id <= 50 are filtered out because targetConstraints remains False

<mapping>

The <mapping> element transforms data values based on rule set. Applied during processing to transform selected records.

Syntax

1
<mapping if="condition" set="expression"/>

Parameters

  • if: Boolean condition that determines when the mapping rule applies
  • set: Expression that can be:
  • Assignment expression (key = value)
  • Boolean expression (True/False)
  • Any valid Python expression

Usage Examples

Example 1: Basic Value Assignment

1
2
3
<mapping if="risk_profile == 'High'" set="interest_rate = 0.15" />
<mapping if="risk_profile == 'Medium'" set="interest_rate = 0.10" />
<mapping if="risk_profile == 'Low'" set="interest_rate = 0.05" />

Example 2: Conditional Calculations

1
2
3
4
<mapping if="income <= 30000" set="credit_limit = 5000" />
<mapping if="income > 30000" set="credit_limit = 10000" />
<mapping if="income > 50000 and years_employed > 2" set="credit_limit = 25000" />
<mapping if="income > 100000 and years_employed > 5" set="credit_limit = 50000" />

Example 3: Mathematical Expressions

1
2
<mapping if="True" set="max_loan = income * 0.4" />
<mapping if="max_loan > 0" set="monthly_payment = max_loan * interest_rate / 12" />

Example 4: Sequential Mapping (Dependencies)

1
2
<mapping if="True" set="mapped_value = 42" />
<mapping if="mapped_value == 42" set="target_value = 'success'" />

Example 5: Default Value Assignment

1
<mapping if="True" set="approval_status = 'Pending'" />

Example 6: Filtering with Boolean Results

1
<mapping if="income < 20000" set="False" />

Complete Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
<setup>
    <generate name="container" count="1">
        <generate name="synthetic_customers" count="1000" pageSize="100"
            source="script/person_data.json" cyclic="True">
            <!-- Mapping: Transform attributes based on source constraints -->
            <mapping if="risk_profile == 'High'" set="interest_rate = 0.15"/>
            <mapping if="risk_profile == 'Medium'" set="interest_rate = 0.10"/>
            <mapping if="risk_profile == 'Low'" set="interest_rate = 0.05"/>
            <!-- Additional rules for credit limits -->
            <mapping if="income > 100000" set="credit_limit = 50000" />
            <mapping if="income > 50000 and income <= 100000" set="credit_limit = 25000" />
            <mapping if="income > 30000 and income <= 50000" set="credit_limit = 10000" />
            <mapping if="income <= 30000" set="credit_limit = 5000" />
        </generate>
    </generate>
</setup>

In this case:

  • When risk_profile is High, <mapping> adds new attribute interest_rate with value 0.15
  • When income is greater than 100000, <mapping> sets credit_limit to 50000
  • Sequential mapping allows later rules to reference values set by earlier rules

Execution Order

The constraint system executes in the following order:

  1. Source Constraints - Validate and filter input source data before processing begins
  2. Mapping - Transform data values if conditions are met during generation
  3. Target Constraints - Validate and filter generated output data before final export

Each layer operates on different stages of the data pipeline and can build upon the results of previous layers.

Key Features

Expression Support

All three constraint elements support: - Boolean expressions (True, False, comparisons) - Assignment expressions (key = value) - Mathematical calculations (income * 0.4) - String operations and concatenation - Complex conditional logic with and, or, not - Function calls and method invocations

Sequential Processing

  • Source constraints rules are processed sequentially
  • Mapping rules are processed sequentially
  • Later rules can reference values set by earlier rules
  • Target constraints can reference all previously set values

Common Behavior

  • All three elements check the if condition first
  • Only when if condition is True, the action (set/require) is executed
  • Source constraints validate and filter input source data before processing
  • Mapping transforms data values during the generation process
  • Target constraints validate and filter generated output data before final export

Error Handling

  • Invalid expressions are logged as warnings by default
  • Set ERROR_CONFIG.fail_fast = True to raise exceptions on errors
  • Type conversion is automatically handled for numeric strings

Performance Considerations

  • Expressions are evaluated using Python's eval() function with restricted globals
  • Complex expressions may impact performance on large datasets
  • Consider using simpler conditions for better performance

Best Practices

  1. Use descriptive variable names for better readability
  2. Order mapping rules logically - simple conditions first, complex ones later
  3. Test expressions thoroughly before deploying to production
  4. Use target constraints sparingly for final validation only
  5. Document complex business rules with comments in XML
  6. Validate data types when performing mathematical operations
  7. Consider performance impact of complex expressions on large datasets