Element <reference>

Purpose: Sample correlated fields from one source row and map them into the current product.

Why: Use it when several output fields must come from the same sampled source row.

Example

<reference name="customer_id" source="data/customers.wgt.ent.csv" sourceKey="id" weightColumn="weight"/>

Decision guide

Business value: Preserves correlations when several output fields must come from the same selected source row.

• Use when

  • Copying multiple related columns from one source record into the current product.

• Choose another approach when

  • Each field may be selected independently or only one direct value is needed.

• Prerequisites

  • Provide a supported source and declare the target/source field mappings.

• Alternatives

  • Use one source-bound variable when later scripts need the full row. (See: <variable>)

Complete examples

Copy correlated fields from one weighted customer row

Use one reference with field children when several foreign-key values must always come from the same selected source record.

id|country|weight
1|DE|1
2|FR|1

<setup numProcess="1">
    <generate name="orders" count="2" target="LogExporter">
        <id name="order_id" generator="IncrementGenerator"/>
        <reference name="customer"
                   source="data/customers.wgt.ent.csv"
                   weightColumn="weight">
            <field target="customer_id" sourceKey="id"/>
            <field target="customer_country" sourceKey="country"/>
        </reference>
    </generate>
</setup>

Rules and invalid combinations

unique cannot be combined with cyclic or ordered selection.

Attributes: unique, cyclic, distribution

Why: Unique selection samples without replacement, while cyclic and ordered modes intentionally walk or repeat source rows.

<reference name="customer" source="customerDb" sourceType="customers" unique="true"/>

<reference name="customer" source="customerDb" sourceType="customers" unique="true" cyclic="true" distribution="ordered"/>

cyclic requires distribution='ordered'.

Attributes: cyclic, distribution

Why: Only an ordered walk has a defined end that cyclic can wrap back to the first row.

<reference name="customer" source="customerDb" sourceType="customers" distribution="ordered" cyclic="true"/>

<reference name="customer" source="customerDb" sourceType="customers" cyclic="true"/>

sourceType is required unless source is a weighted reference file.

Attributes: source, sourceType

Why: Database-style sources need an entity name; .wgt.csv and .wgt.ent.csv encode their reference shape in the file.

<reference name="customer" source="data/customers.wgt.ent.csv"/>

<reference name="customer" source="data/customers.ent.csv"/>

I843 — Task Reference Unsupported Source

Reference '{name}' requires a configured relational or MongoDB source, or a .wgt.csv/.wgt.ent.csv project file

Why: The reference source is neither a supported configured client nor a weighted project file.

Resolution: Use a relational or MongoDB source id, or provide a .wgt.csv/.wgt.ent.csv project file.

Full rule

I844 — Task Reference No Data

No data found for reference '{name}'

Why: An error occurred during task configuration.

Resolution: Review the configuration and retry.

Full rule

I845 — Task Reference Unique Too Few

Cannot generate {requested} unique values - only {available} available for reference '{name}'

Why: The system could not generate {requested} unique values - only {available} available for reference '{name}'.

Resolution: Review the configuration and retry.

Full rule

I846 — Task Reference Iterator Not Initialized

Iterator was not properly initialized for reference '{name}'

Why: An error occurred during task configuration.

Resolution: Review the configuration and retry.

Full rule

I874 — Task Reference Query Failed

Failed to resolve reference '{name}' from source '{source_type}' and sourceKey '{source_key}': {error_type}{error_code_hint}{error_category_hint}: {error}

Why: Database query failed while resolving values for a task.

Resolution: Review sourceType/sourceKey, SQL identifier validity, and database permissions before retrying.

Full rule

I875 — Task Reference Single Weighting Multi Field

Reference '{name}' uses single-column weighting source '{source}' with {target_count} mapped fields. Use exactly one field for '.wgt.csv' or switch to '.wgt.ent.csv' for tuple references.

Why: Single-column weighting files cannot represent multi-column tuple references.

Resolution: Map exactly one target field for '.wgt.csv' or use '.wgt.ent.csv' with mappings.

Full rule

I876 — Task Reference Unique Duplicates

Reference '{name}' requires business-unique values but source has {duplicates} duplicate rows ({unique_available} unique values from {total_rows} rows). Set platform_json 'reference_unique_duplicate_policy' to warning/skip/ignore if deduplication is acceptable.

Why: The unique reference source contains duplicate business tuples while strict duplicate policy is enabled.

Resolution: Deduplicate the source values, or change reference_unique_duplicate_policy to warning/skip/ignore when duplicates are expected.

Full rule

I896 — Task Reference Ordered Exhausted

Ordered reference '{name}' exhausted after {count} rows; add cyclic="true" to wrap, or reduce the generated count.

Why: A non-cyclic distribution="ordered" was walked past its last candidate.

Resolution: Add cyclic="true" to wrap, or reduce the generated count.

Full rule

Allowed parents / Allowed children

Allowed parents: else, else-if, generate, if, iterate, while

Allowed children:

field

Related concepts and use cases

• Model relationships and correlated data — Explains when to use nested products, memstore lineage, or one correlated reference selection.

Attributes

Show all 8 attributes

cyclic

Wrap the source back to the start when exhausted (typically with ordered).

optional; boolean; Default: null.

distribution

How candidates are picked per record: random (default, with replacement) or ordered (source order). cyclic wraps ordered once exhausted.

optional; string; Default: null; Values: ordered, random.

name

Name of the reference field.

required; string.

source

Source of data for the reference field.

required; string.

sourceKey

Key field in the source for the reference (legacy single-field form).

optional; string; Default: null.

sourceType

Source type for the reference field.

optional; string; Default: null.

unique

Uniqueness constraint for the reference field.

optional; boolean; Default: null.

weightColumn

Optional numeric weight column used for weighted sampling from sourceType.

optional; string; Default: null.