Skip to content

Benerator to DATAMIMIC Migration Guide

This guide provides detailed instructions and examples to help you migrate from Benerator to DATAMIMIC. It includes a comprehensive comparison of supported elements, attributes, generators, and converters, along with suggestions for manual migration and Python scripting alternatives.

Benerator and DATAMIMIC Comparison

Below is a detailed analysis and comparison of the Benerator model with the DATAMIMIC model.

DATAMIMIC Supported Elements and Attributes

Removed Elements:

  • <bean/>
  • <setting/> -> replace with variable
  • <property/> -> replace with variable
  • <pre-parse-generate/>
  • <domtree/>
  • <run-task/>
  • <defaultComponents/>
  • <comment/> -> replace with echo
  • <error/>
  • <beep/>
  • <dbsanity/>
  • <iterate/> -> replace with generate
  • <attribute/> -> replace with key
  • <part/> -> replace with nestedKey
  • <id/> -> replace with key
  • <consumer/>
  • <evaluate/>
  • <import/>
  • <execute/>

Removed Attributes:

  • consumer="Preview" - this happens automatically
  • consumer="Result"
  • consumer="NoConsumer" - just remove target and it will not be exported
  • consumer -> replace with target
  • type in generate and iterate -> replace with name
  • subSelector -> can be replaced with selector
  • acceptUnknownTypes
  • defaultScript
  • defaultErrorHandler
  • defaultImports
  • generatorFactory
  • maxCount
  • defaultNull
  • defaultTimeZone -> no immediate need
  • defaultPageSize
  • defaultOneToOne
  • defaultSourceScripted

Custom Jars:

  • Not supported yet. We are working on a prototype with GraalPy and GraalVM to make this possible in the future.

JDBC-Metadata Analysis:

  • Removed, we are using SQL Alchemy for database connections. Additionally, we will provide more database-related comfort features in our DATAMIMIC UI.

Script Support

  • JS / BEN script support removed.
  • dynamicSource -> Source (only) and there is a boolean attribute sourceScripted can be used like this in variable, key, and nestedKey: 
    1
    <variable name="people" source="data/people.wgt.ent.csv" sourceScripted="True" cyclic="True"/>
    Scripted data needs to be surrounded by curly braces like this 1,Steve,{48+11},15.
  • Final state of <variable/> in global context -> storing dynamic values needs to be handled by database lookups/file caches/etc.

Accessing Components and Variables

  • Root Level Access: When accessing keys or variables at the root level within a script attribute, the first generate node does not need to be mentioned. For instance, to access a variable or key directly in the root level: 

    1
    <generate script="variableOrKeyInRootLvl"/>

  • Nested Level Access: If the variable or key is within a nested structure, such as another generate node or a nestedKey node at level 2, the name of the nested node must be prefixed. For example, to access a variable or key in the second level: 

    1
    <generate script="nestedKeyOrGenerateName.variableOrKeyInNestedLvl2"/>

Examples Using Provided Model

Given the following model:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
<setup>
    <generate name="persons" count="5">
        <key name="name" type="string" values="'Alice','Bob','Charlie'"/>
        <nestedKey name="notes" type="list" count="1">
            <key name="pet" type="string" constant="Dog"/>
            <key name="hobby" type="string" values="'Skiing','Sailing'"/>
            <key name="friend" type="string" constant="Nemo"/>
            <key name="work" type="string" values="'Google','Microsoft','Tesla'"/>
        </nestedKey>
        <nestedKey name="cars" type="list" minCount="0" maxCount="3">
            <key name="model" type="string" values="'Audi Q4 e-tron','Tesla','VW Passat','BMW iX'" />
            <key name="color" type="string" generator="DataFakerGenerator('color_name')" />
            <nestedKey name="accidents" type="list" source="data/accidents.ent.csv" maxCount="3">
              <key name="repair_required" source="data/active.wgt.csv"/>
            </nestedKey>
        </nestedKey>
    </generate>
</setup>

And Entity and Weighting Data as:

data/accidents.ent.csv
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
date;description;damage_cost
03-12;Pedestrian impact;2500
03-13;Animal collision;800
03-14;Weather-related incident;600
03-15;Hit and run;2000
03-16;Pedestrian impact;2500
03-17;Rear-end collision;1500
03-18;Head-on collision;3000
03-19;Rear-end collision;1500
03-20;Weather-related incident;600
03-21;Parking lot scrape;300
data/active.wgt.csv
1
2
3
True;80
False;10
;10

  • Root Level Access Example: To access the name key in the root level: 

    1
    <generate script="name"/>

  • Nested Level Access Examples:

  • To access the car key within the notes nestedKey: 
    1
    <generate script="notes.pet"/>
  • To access the color key within the cars nestedKey: 
    1
    <generate script="cars.color"/>
  • To access the repair_required key within the accidents nestedKey inside cars: 
    1
    <generate script="cars.accidents.repair_required"/>
  • To access the name key within the family nestedKey: 
    1
    <generate script="family.name"/>

Advantages Compared to Benerator

  • Clear Component Access: The DATAMIMIC model structure provides a clear and consistent way to access components and variables. In Benerator, accessing components, especially with cascading generate and iterate nodes, can be confusing and error-prone. DATAMIMIC simplifies this by using a straightforward naming convention for nested levels, improving readability and maintainability.

  • Structured Navigation: With DATAMIMIC, the structure is more intuitive. Root-level components are accessed directly, while nested components are accessed by prefixing with their parent node names. This eliminates the ambiguity found in Benerator when dealing with complex nested structures.

  • Enhanced Readability: The explicit naming of nested levels in DATAMIMIC scripts makes the code easier to read and understand. This reduces the cognitive load on developers and minimizes the likelihood of mistakes when accessing deeply nested variables.

  • Consistency: DATAMIMIC enforces a consistent method for accessing variables and keys, whether they are at the root level or nested within multiple levels. Benerator’s approach can lead to inconsistencies, especially in large and complex models.

  • Improved Maintainability: The clear and structured access method in DATAMIMIC makes it easier to maintain and update the models. Developers can quickly identify and modify the necessary components without sifting through unclear and tangled access patterns.

Differences and Migration Guide

Elements and Attributes Comparison Matrix

Element Benerator Attributes DATAMIMIC Attributes Comments and Changes
<setup> acceptUnknownSimpleTypes, defaultScript, ... Not supported Most setup attributes are not used in DATAMIMIC.
<database> id Supported, id Similar functionality.
<kafka-importer> id, system Supported, id, system Similar functionality.
<kafka-exporter> id, system Supported, id, system Similar functionality.
<mongodb> id Supported, id Similar functionality.
<generate> name, type, scope, converter, ... name, scope, converter, ... Minor changes, type replaced with name.
<iterate> name, type, scope, converter, ... name, scope, converter, ... Replaced with <generate>.
<part> name, container, source, ... Replaced with nestedKey Consolidation.
<attribute> name, type, scope, converter, ... Replaced with key Consolidation.
<reference> name, type, scope, converter, ... Supported Similar functionality.
<variable> name, type, scope, converter, ... Supported Similar functionality.
<id> name, type, scope, converter, ... Replaced with key Consolidation.
<include> uri Supported Similar functionality.
<import> domains, platforms Not supported Removed.
<echo> type Supported Similar functionality.
<memstore> id Supported Similar functionality.
<consumer> class Not supported Removed.
<setting> name, value, default, ref, source Replaced with variable Consolidation.
<execute> uri, encoding, type, separator, optimize, ... uri, target Works with SQL Scripts and Python
<if> test condition Similar functionality.
<else>, <then> None Not supported Removed.

Generators

Generator Name Benerator DATAMIMIC Comments and Changes
IncrementGenerator Supported Supported Similar functionality.
StringGenerator Supported Supported Similar functionality.
RegexStringGenerator Supported Not Supported Use custom Python script for regex-based generation.
MessageGenerator Supported Not Supported Use custom Python script for message generation.
LuhnGenerator Supported Not Supported Use custom Python script for Luhn algorithm.
CityGenerator Supported Not Supported Use DataFakerGenerator for city generation.
CountryGenerator Supported Not Supported Use DataFakerGenerator for country generation.
CurrentDateGenerator Supported Not Supported Use DateTimeGenerator for current date generation.
CurrentDateTimeGenerator Supported Not Supported Use DateTimeGenerator for current date-time generation.
CurrentMilliTimeGenerator Supported Not Supported Use custom Python script for millisecond time generation.
CurrentNanoTimeGenerator Supported Not Supported Use custom Python script for nanosecond time generation.
CurrentTimeGenerator Supported Not Supported Use DateTimeGenerator for current time generation.
DateGenerator Supported Not Supported Use DateTimeGenerator for date generation.
DayGenerator Supported Not Supported Use DateTimeGenerator for day generation.
DateTimeGenerator Supported Supported Similar functionality.
StateGenerator Supported Not Supported Use DataFakerGenerator for state generation.
StateTransitionGenerator Supported Not Supported Use custom Python script for state transitions.
SeedWordGenerator Supported Not Supported Use DataFakerGenerator for word generation.
SeedSentenceGenerator Supported Not Supported Use DataFakerGenerator for sentence generation.
PersonGenerator Supported Not Supported Use DataFakerGenerator for person generation.
AddressGenerator Supported Not Supported Use DataFakerGenerator for address generation.
StreetNameGenerator Supported Supported Similar functionality.
PhoneNumberGenerator Supported Supported Similar functionality.
MobilePhoneNumberGenerator Supported Not Supported Use PhoneNumberGenerator for mobile numbers.
DomainGenerator Supported Supported Similar functionality.
CompanyNameGenerator Supported Supported Similar functionality.
DepartmentNameGenerator Supported Supported Similar functionality.
BankAccountGenerator Supported Not Supported Use custom Python script for bank account generation.
CreditCardNumberGenerator Supported Not Supported Use custom Python script for credit card generation.
EAN8Generator Supported Not Supported Use custom Python script for EAN-8 generation.
EAN13Generator Supported Not Supported Use custom Python script for EAN-13 generation.
EANGenerator Supported Supported Similar functionality.
CPNJGenerator Supported Supported Similar functionality.
CPFGenerator Supported Supported Similar functionality.
SSNGenerator Supported Supported Similar functionality.

Converters

Converter Name Benerator DATAMIMIC Comments and Changes
Base64ToByteArrayConverter Supported Not Supported Use custom implementation in Python if needed.
ByteBufferToByteArrayConverter Supported Not Supported Use custom implementation in Python if needed.
ByteArrayToBase64Converter Supported Not Supported Use custom implementation in Python if needed.
CaseConverter Supported Supported Similar functionality.
ToLowerCaseConverter Supported Supported Similar functionality.
ToUpperCaseConverter Supported Supported Similar functionality.
ToHashCodeConverter Supported Not Supported Use custom implementation in Python if needed.
LiteralParser Supported Not Supported Use custom implementation in Python if needed.
MessageConverter Supported Not Supported Use custom implementation in Python if needed.
PropertyResourceBundleConverter Supported Not Supported Use custom implementation in Python if needed.
ToArrayConverter Supported Not Supported Use custom implementation in Python if needed.
ToStringConverter Supported Not Supported Use custom implementation in Python if needed.
ToCollectionConverter Supported Not Supported Use custom implementation in Python if needed.
UniqueStringConverter Supported Not Supported Use custom implementation in Python if needed.
URLEncodeConverter Supported Not Supported Use custom implementation in Python if needed.
URLDecodeConverter Supported Not Supported Use custom implementation in Python if needed.
PrintfConverter Supported Not Supported Use custom implementation in Python if needed.
RegexReplacer Supported Not Supported Use custom implementation in Python if needed.
SubArrayExtractor Supported Not Supported Use custom implementation in Python if needed.
SubstringExtractor Supported Not Supported Use custom implementation in Python if needed.
EscapingConverter Supported Supported Similar functionality.
Char2StringConverter Supported Not Supported Use custom implementation in Python if needed.
Number2CharConverter Supported Not Supported Use custom implementation in Python if needed.
Number2BigDecimalConverter Supported Not Supported Use custom implementation in Python if needed.
Number2BigIntegerConverter Supported Not Supported Use custom implementation in Python if needed.
Number2BooleanConverter Supported Not Supported Use custom implementation in Python if needed.
Number2ByteConverter Supported Not Supported Use custom implementation in Python if needed.
Number2CharConverter Supported Not Supported Use custom implementation in Python if needed.
Number2DoubleConverter Supported Not Supported Use custom implementation in Python if needed.
Number2FloatConverter Supported Not Supported Use custom implementation in Python if needed.
Number2IntegerConverter Supported Not Supported Use custom implementation in Python if needed.
Number2LongConverter Supported Not Supported Use custom implementation in Python if needed.
Number2ShortConverter Supported Not Supported Use custom implementation in Python if needed.
Number2StringConverter Supported Not Supported Use custom implementation in Python if needed.
NumberQuantizer Supported Not Supported Use custom implementation in Python if needed.
DelocalizingConverter Supported Not Supported Use custom implementation in Python if needed.
SHA256Hash Supported Not Supported Use HashConverter in DATAMIMIC.
SHA1HashBase64 Supported Not Supported Use HashConverter in DATAMIMIC.
MD5Hash Supported Not Supported Use HashConverter in DATAMIMIC.
SHA1Hash Supported Not Supported Use HashConverter in DATAMIMIC.
SHA1HashBase64 Supported Not Supported Use HashConverter in DATAMIMIC.
JavaHash Supported Supported Similar functionality.
SHA1Hash Supported Not Supported Use HashConverter in DATAMIMIC.
new MiddleMask(2,3) Supported Supported Similar functionality.
Mask Supported Supported Similar functionality.
new CutLength(3) Supported Supported Similar functionality.
NameNormalizer Supported Not Supported Use custom implementation in Python if needed.
NullToEmptyStringConverter Supported Not Supported Use custom implementation in Python if needed.
NormalizeSpaceConverter Supported Not Supported Use custom implementation in Python if needed.
ToHexConverter Supported Not Supported Use custom implementation in Python if needed.
FloatingPointQuantizer Supported Not Supported Use custom implementation in Python if needed.
IntegralQuantizer Supported Not Supported Use custom implementation in Python if needed.
NoiseInducer Supported Not Supported Use custom implementation in Python if needed.
String2ByteArrayConverter Supported Not Supported Use custom implementation in Python if needed.
String2EnumConverter Supported Not Supported Use custom implementation in Python if needed.
String2NumberConverter Supported Not Supported Use custom implementation in Python if needed.
String2NumberFormatConverter Supported Not Supported Use custom implementation in Python if needed.
String2PatternConverter Supported Not Supported Use custom implementation in Python if needed.
PercentageFormatter Supported Not Supported Use custom implementation in Python if needed.

Improved Time Transformation Handling in DATAMIMIC

DATAMIMIC offers an improved and flexible way of handling date and time transformations through the use of inDateFormat and outDateFormat attributes. These attributes enable seamless conversion of date and time formats, simplifying the process of generating and manipulating date and time values.

Date and Time Conversion Attributes

  • inDateFormat: Specifies the format of the input date and time value.
  • outDateFormat: Specifies the format of the output date and time value.

These attributes can be applied to any key to convert date and time values as needed. This provides a powerful alternative to the various specific date and time converters used in Benerator.

Explanation

  • datetime_with_in: Converts a constant date and time value from the format "%Y-%m-%d %H:%M:%S.%f".
  • datetime_with_out: Generates a date and time value and outputs it in the format "%d.%m.%Y %H:%M:%S.%f".
  • date_with_in_out: Converts a constant date value from the format "%Y-%m-%d" to the format "%d.%m.%Y".
  • day_out: Outputs the current day in the format "%d".
  • month_out: Outputs the current month in the format "%m".
  • year_out: Outputs the current year in the format "%Y".
  • time_out: Outputs the current time in the format "%H:%M:%S".

Replacing Missing Converters

In DATAMIMIC, the flexible inDateFormat and outDateFormat attributes can replace several specific converters that were used in Benerator. Here's how you can achieve the same functionality with DATAMIMIC:

Converter Benerator Support DATAMIMIC Support Replacement Approach
TimestampFormatter Supported Not Supported Use outDateFormat to format timestamps.
String2LocalDateConverter Supported Not Supported Use inDateFormat to parse local dates.
String2DateConverter Supported Not Supported Use inDateFormat to parse dates.
String2DateFormatConverter Supported Not Supported Use inDateFormat and outDateFormat for conversion.
String2TimeConverter Supported Not Supported Use inDateFormat to parse time values.
String2TimestampConverter Supported Not Supported Use inDateFormat to parse timestamps.
String2YearMonthConverter Supported Not Supported Use inDateFormat to parse year-month values.
String2ZonedDateTimeConverter Supported Not Supported Use inDateFormat to parse zoned date-time values.
Time2StringConverter Supported Not Supported Use outDateFormat to format time values.
Timestamp2DateConverter Supported Supported Similar functionality using outDateFormat.
String2GregorianCalendarConverter Supported Not Supported Use inDateFormat to parse Gregorian calendar dates.
String2LocalDateConverter Supported Not Supported Use inDateFormat to parse local dates.
String2LocalDateTimeConverter Supported Not Supported Use inDateFormat to parse local date-time values.

These improvements allow for a more streamlined approach to handling date and time transformations in DATAMIMIC, reducing the need for multiple specific converters and providing a more flexible and powerful solution. For more detailed documentation on date and time handling, please refer to the DATAMIMIC documentation at Date and Time Generators

Custom Generator and Converter Example in Python

To further support users in creating custom generators and converters.

Custom Generator Example

1
2
3
4
5
6
7
8
class CustomStringGenerator(Generator):
    def __init__(self, prefix: str = ""):
        self.prefix = prefix
        self.counter = 0

    def generate(self) -> str:
        self.counter += 1
        return f"{self.prefix}{self.counter}"

Custom Converter Example

1
2
3
class CustomConverter(Converter):
    def convert(self, value):
        return f"Converted: {value}"

For more information on creating custom generators and converters in Python, please check Custom Generators.

Suggestions for Migration

Automatic Migration

For the start and for more simple Benerator configurations, you can use the DATAMIMIC UI to import and convert your existing Benerator scripts.

  1. Import Benerator Script: Use the DATAMIMIC UI to create a DATAMIMIC project from Benerator project Project menu.

  2. Review and Validate: Review the imported script and validate the data generation. DATAMIMIC migrates and translates the Benerator model automatically into its new model syntax. Incompatible nodes, elements or attributes are commented and marked for manual migration along with additional recommendations.

  3. Run Data Generation: Execute the data generation task to ensure compatibility and functionality.

Manual Migration

  1. Map Elements and Attributes: Use the comparison matrix to map Benerator elements and attributes to their DATAMIMIC equivalents. This helps in understanding what needs to be changed or replaced.

  2. Remove Unsupported Elements and Attributes: Identify elements and attributes not supported in DATAMIMIC and remove them. Use suggested replacements where applicable.

  3. Update Scripts and Variables: For elements like <setting> and <attribute>, update scripts to use the new variable or key elements in DATAMIMIC.

  4. Review Custom Logic: Review any custom jars or scripts used in Benerator and replace them with DATAMIMIC supported functionalities.

  5. Final Review and Testing: Thoroughly test the migrated data generation scripts to ensure they work as expected in the DATAMIMIC environment.