> For the complete documentation index, see [llms.txt](https://docs.brigge.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.brigge.ai/documentation/brigge-for-transformers/pipelines/pipeline-nodes/merge-node.md). # Merge node ## Introduction The *Merge* node receives incoming datasets, performs the configured operation, and outputs a single dataset. Six merge methods are supported, each designed for a different use case depending on how you want to handle the incoming data, and the shape of output data needed:
Inner join
Outer join
Left join
Union
Combine
Insert at
For all but two merge types ([combine](#combine) and [insert at](#insert-at)), selecting a merge method enables two input stubs (labelled `A` and `B`) on the *Merge* node, because two incoming datasets are expected:
These labels reflect the processing order: dataset `A` is matched against dataset `B`, and when output is assembled, dataset `B` is merged into dataset `A`. So, dataset `A` is the primary dataset. ## How is data matched for merging? In order to merge records received from two datasets, we need a mechanism to determine how records in dataset `A` map to records in dataset `B`. For most merge methods that *join* data from two inputs ([inner join](#inner-join), [outer join](#outer-join), [left join](#left-join)), an `input key` is defined in the *Merge* node configuration. This is a field that's common to both sets of data, and uniquely identifies each record in the dataset. For example: {% columns %} {% column %} **Dataset A (employees)** {% code overflow="wrap" %} ```json { "employees": [ {"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}, {"id": 3, "name": "Charlie"} ] } ``` {% endcode %} {% endcolumn %} {% column %} **Dataset B (departments)** {% code overflow="wrap" %} ```json { "departments": [ {"id": 1, "dept": "Product"}, {"id": 3, "dept": "Finance"}, {"id": 4, "dept": "Sales"} ] } ``` {% endcode %} {% endcolumn %} {% endcolumns %} Here, the `id` field is unique to each record and common to both datasets. So (depending on which merge method is selected in *Merge* node settings), we can define `id` as the `input key` and match/merge records with the same `id` number. {% hint style="info" %} The [union](#union) merge method also joins data from two inputs but no matching is needed. {% endhint %} Other merge methods ([combine](#combine) and [insert at](#insert-at)) do not match data. Instead, data is merged based on the incoming data structure. ## Sending data into a merge node Different merge methods have different requirements in terms of how incoming data is structured. When considering the shape of data sent into a *Merge* node, it can be helpful to think in terms of updating rows in a database. For example, suppose we are joining names and departments from two datasets, where incoming payloads contain multiple items (think multiple 'rows' or 'records'):
Our *Merge* node must target each individual record, which means it must either: * Have a mechanism to target specific records in each payload. * Receive single-item payloads from each data stream, so a match is inherent. As noted [previously](#how-is-data-matched-for-merging), explicit *join* methods ([inner join](#inner-join), [outer join](#outer-join), [left join](#left-join)) have the facility to match two incoming datasets on a unique key, so it's no issue if the *Merge* node receives payloads containing multiple records. For other methods, it's advisable to use a [*Split*](/documentation/brigge-for-transformers/pipelines/pipeline-nodes/split-node.md) node upstream to batch the data into single-item payloads, before it reaches the *Merge* node. ## Merge methods
Inner join
You provide `Input A key` and `Input B key` fields, which are used to determine which incoming records match. {% hint style="info" %} For more information about field matching, see [How is data matched for merging?](#how-is-data-matched-for-merging) {% endhint %} Where a value in the `Input A key` matches a value in the `Input B key`, the two records are joined and output. The full `Input A` record forms the top level of the output, and the full `Input B` record (including the field used for matching) is nested under the specified `insertion key`. Records that exist in only one data source are discarded entirely. No partial records or empty fields are output. #### Example The illustration below shows two incoming data sources (*Employees* and *Teams*) where `id` is the shared (`input`) key. The output is the result for an inner join:
The Brigge pipeline setup for this scenario is shown below:
Pipeline
Incoming dataset A
{
  "staff": [
    {
      "id": 1,
      "name": "Alice James"
    },
    {
      "id": 2,
      "name": "Bob Smith"
    },
    {
      "id": 3,
      "name": "Esme Fox"
    }
  ]
}
Incoming dataset B
{
  "teams": [
    {
      "id": 1,
      "team": "Jupiter"
    },
    {
      "id": 3,
      "team": "Saturn"
    },
    {
      "id": 4,
      "team": "Pluto"
    }
  ]
}
Merge node config
Success output
{
  "staff": [
    {
      "id": 1,
      "name": "Alice James",
      "work_team": {
        "id": 1,
        "team": "Jupiter"
      }
    },
    {
      "id": 3,
      "name": "Esme Fox",
      "work_team": {
        "id": 3,
        "team": "Saturn"
      }
    }
  ]
}
Failed records
[
  {
    "id": 2,
    "name": "Bob Smith"
  },
  {
    "id": 4,
    "team": "Pluto"
  }
]
#### Configuration
Configuration options are summarised below:
OptionSummary
Input A / B schemasChoose a schema associated with the expected structure of payloads for both incoming datasets. Schemas are optional, but they do make defining the subsequent input keys easier.
Input A / B keysSelect or enter the field in both incoming data streams that should be used for matching records to be joined. This field must be unique to each record in the payload - for more information, see How is data matched for merging?

If you have applied schemas, you can navigate the payload structure and select the required field - otherwise enter a dot notation path manually.
Insertion keySets the name of the merged object from dataset B.
Outer join
You provide `Input A key` and `Input B key` fields, which are used to determine which incoming records match. {% hint style="info" %} For more information about field matching, see [How is data matched for merging?](#how-is-data-matched-for-merging) {% endhint %} Where a value in the `Input A key` matches a value in the `Input B key`, the two records are joined and output. The full `Input A` record forms the top level of the output, and the full `Input B` record (including the field used for matching) is nested under the specified `insertion key`. Additionally: * If dataset `A` (the left input) has no match in dataset `B`, it is output on its own. * If dataset `B` (the right input) has no match in dataset `A`, it is output on its own. #### Example The illustration below shows two incoming data sources (*Employees* and *Departments*) where `id` is the shared (`input`) key:
The Brigge pipeline setup for this scenario is shown below:
Pipeline
Incoming dataset A
{
  "staff": [
    {
      "id": 1,
      "name": "Alice James"
    },
    {
      "id": 2,
      "name": "Bob Smith"
    },
    {
      "id": 3,
      "name": "Esme Fox"
    }
  ]
}
Incoming dataset B
{
  "teams": [
    {
      "id": 1,
      "team": "Jupiter"
    },
    {
      "id": 3,
      "team": "Saturn"
    },
    {
      "id": 4,
      "team": "Pluto"
    }
  ]
}
Merge node config
Success output
{
  "staff": [
    {
      "id": 1,
      "name": "Alice James",
      "work_team": {
        "id": 1,
        "team": "Jupiter"
      }
    },
    {
      "id": 2,
      "name": "Bob Smith",
      "work_team": {
        "id": 2
      }
    },
    {
      "id": 3,
      "name": "Esme Fox",
      "work_team": {
        "id": 3,
        "team": "Saturn"
      }
    },
    {
      "id": 4,
      "work_team": {
        "id": 4,
        "team": "Pluto"
      }
    }
  ]
}
Failed recordsNone - all records output successfully.
#### Configuration
Configuration options are summarised below:
OptionSummary
Input A / B schemasChoose a schema associated with the expected structure of payloads for both incoming datasets. Schemas are optional, but they do make defining the subsequent input keys easier.
Input A / B keysSelect or enter the field in both incoming data streams that should be used for matching records to be joined. This field must be unique to each record in the payload - for more information, see How is data matched for merging?

If you have applied schemas, you can navigate the payload structure and select the required field - otherwise enter a dot notation path manually.
Insertion keySets the name of the merged object from dataset B.
Left join
You provide an `Input A key` and `Input B key` fields, which are used to determine which incoming records match. {% hint style="info" %} For more information about field matching, see [How is data matched for merging?](#how-is-data-matched-for-merging) {% endhint %} If a matching `input key` is found in both data sources, the data is joined and output. Also: * If dataset `A` (the left input) has no match in dataset `B`, it is output on its own. * If dataset `B` (the right input) has no match in dataset `A`, it is discarded. In short, dataset `A` is always output, irrespective of whether a join can be made. #### Example The illustration below shows two incoming data sources (*Employees* and *Departments*) where `id` is the shared (`input`) key:
The Brigge pipeline setup for this scenario is shown below:
Pipeline
Incoming dataset A
{
  "staff": [
    {
      "id": 1,
      "name": "Alice James"
    },
    {
      "id": 2,
      "name": "Bob Smith"
    },
    {
      "id": 3,
      "name": "Esme Fox"
    }
  ]
}
Incoming dataset B
{
  "teams": [
    {
      "id": 1,
      "team": "Jupiter"
    },
    {
      "id": 3,
      "team": "Saturn"
    },
    {
      "id": 4,
      "team": "Pluto"
    }
  ]
}
Merge node config
Success output
{
  "staff": [
    {
      "id": 1,
      "name": "Alice James",
      "team": {
        "id": 1,
        "team": "Jupiter"
      }
    },
    {
      "id": 2,
      "name": "Bob Smith"
    },
    {
      "id": 3,
      "name": "Esme Fox",
      "team": {
        "id": 3,
        "team": "Saturn"
      }
    }
  ]
}
Failed records
[
  {
    "id": 4,
    "team": "Pluto"
  }
]
#### Configuration
Configuration options are summarised below:
OptionSummary
Input A / B schemasChoose a schema associated with the expected structure of payloads for both incoming datasets. Schemas are optional, but they do make defining the subsequent input keys easier.
Input A / B keysSelect or enter the field in both incoming data streams that should be used for matching records to be joined. This field must be unique to each record in the payload - for more information, see How is data matched for merging?

If you have applied schemas, you can navigate the payload structure and select the required field - otherwise enter a dot notation path manually.
Insertion keySets the name of the merged object from dataset B.
Union
Stacks incoming (array) datasets into a single flat output array within a named JSON object - dataset `A` followed by `B`. There is no matching, pairing, or key comparison. #### Example
Pipeline
Incoming dataset A
{
  "staff": [
    {
      "id": 1,
      "name": "Alice James"
    },
    {
      "id": 2,
      "name": "Bob Smith"
    },
    {
      "id": 3,
      "name": "Esme Fox"
    }
  ]
}
Incoming dataset B
{
  "staff": [
    {
      "id": 1,
      "name": "Polly Lowe"
    },
    {
      "id": 2,
      "name": "Mia Fey"
    },
    {
      "id": 3,
      "name": "Paul Binns"
    }
  ]
}
Merge node config
Success output
{
  "forenames": [
    {
      "id": 1,
      "name": "Alice James"
    },
    {
      "id": 2,
      "name": "Bob Smith"
    },
    {
      "id": 3,
      "name": "Esme Fox"
    },
    {
      "id": 1,
      "name": "Polly Lowe"
    },
    {
      "id": 2,
      "name": "Mia Fey"
    },
    {
      "id": 3,
      "name": "Paul Binns"
    }
  ]
}
Failed recordsNone - all records output successfully.
#### Configuration
Configuration options are summarised below:
OptionSummary
Input A / B schemasChoose a schema associated with the expected structure of payloads for both incoming datasets. Schemas are optional, but they do make defining the subsequent input keys easier.
Input A / B keysSelect or enter the (array) field name from the incoming payload whose values you want to output. If you have applied a schema, you can navigate the payload structure and select the required field - otherwise enter a dot notation path manually.
Insertion keySets the name of the output object.
Combine
Collects the values for a specified field from each incoming payload and outputs them in a single array. There is no matching, pairing, or key comparison. #### Configuration
Configuration options are summarised below:
OptionSummary
Input A schemaChoose a schema associated with the expected structure of incoming payloads. This setting is optional - it just makes defining the subsequent input A key easier.
Input A keySelect or enter the field name from the incoming payload whose values you want collected into the output array. If you have applied a schema, you can navigate the payload structure and select the required field - otherwise enter a dot notation path manually.
Insertion keySets the name of the output array.
#### Example Suppose a *Merge* node receives multiple, single-item payloads for employee details: {% code overflow="wrap" %} ```json {"id":1,"first":"Alice","last":"Johnson","hours_worked":38} ``` {% endcode %} {% code overflow="wrap" %} ```json {"id":2,"first":"Bob","last":"Smith","hours_worked":42} ``` {% endcode %} {% code overflow="wrap" %} ```json {"id":3,"first":"Charlie","last":"Davis","hours_worked":40} ``` {% endcode %} Using the `combine` option, we can output all received `hours_worked` values into an array, within a named JSON object. For example: {% code overflow="wrap" %} ```json {"to_invoice":[38,42,40,35,45]} ``` {% endcode %} The Brigge pipeline setup for this scenario is shown below:
Pipeline
Incoming dataset A
{
  "hours": [
    {"id": 1, "first": "Alice", "last": "Johnson", "hours_worked": 38},
    {"id": 2, "first": "Bob", "last": "Smith", "hours_worked": 42},
    {"id": 3, "first": "Charlie", "last": "Davis", "hours_worked": 40},
    {"id": 4, "first": "Diana", "last": "Moore", "hours_worked": 35},
    {"id": 5, "first": "Jack", "last": "Brown", "hours_worked": 45}
  ]
}
Split node
Merge node config
Success output
{"hours_to_invoice":[38,42,40,35,45]}
Failed recordsNone - all records output successfully.
Insert at
Injects a field value from dataset `B` into dataset `A` at a specified position. There is no matching - dataset `B` exists purely as a source to extract a value from. {% hint style="info" %} When inserting data, it's advisable to use a [*Split*](/documentation/brigge-for-transformers/pipelines/pipeline-nodes/split-node.md) node upstream to batch the data into single-item payloads, before it reaches the *Merge* node. For more information, see [Sending data into a merge node](#sending-data-into-a-merge-node). {% endhint %} #### Configuration
Configuration options are summarised below:
OptionSummary
Input A / B schemasChoose a schema associated with the expected structure of payloads for both incoming datasets. Schemas are optional, but they do make defining the subsequent insert and input keys easier.
Insert atDetermines the insert position within dataset A. The input from dataset B is inserted immediately after the field named here.
Input B keyDetermines which field to collect from dataset B; any other fields in dataset B are ignored.
Insertion keySets the name of the output object.
#### Example Suppose dataset `A` includes employee names and countries: {% code overflow="wrap" %} ```json {"name": "Alice Jones", "country": "UK"} ``` {% endcode %} ...and dataset `B` includes billing details for employees: {% code overflow="wrap" %} ```json {"currency1": "GBP","currency2": "USD","currency3": "EUR"} ``` {% endcode %} We can use the `insert at` method to inject the relevant currency field from dataset `B` into dataset A: {% code overflow="wrap" %} ```json {"name": "Alice Jones", "country": "UK", "currency": "GBP"} ``` {% endcode %} The Brigge pipeline setup for this scenario is shown below:
Pipeline
Incoming dataset A
{
  "employees": [
    {"name": "Alice Jones","country": "UK" },
    {"name": "Bob Smith","country": "USA"},
    {"name": "Charlie Pink","country": "EU" }
  ]
}
Incoming dataset B
{
"currencies": \[
{ "currency": "GBP"},
{ "currency": "USD"},
{ "currency": "EUR" }
]
}
Split node (dataset A)
Split node (dataset B)
Merge node config
Success output
{"name":"Alice Jones","country":"UK","currency":"GBP"} 
{"name":"Bob Smith","country":"USA","currency":"USD"} 
{"name":"Charlie Pink","country":"EU","currency":"EUR"}
Failed recordsNone - all records output successfully.
{% hint style="info" %} Incoming datasets (each containing one record) arrive in the same order. Each currency record in dataset `B` corresponds positionally to the employee record in dataset `A`. {% endhint %}
## Error handling When using a merge method that joins records from two incoming datasets, there are scenarios where some records can be merged but others can't. For example, the [inner join](#inner-join) method will only merge records where a match is found in *both* incoming datasets. If a record is present in dataset `A` that is not matched in `B`, the `A` record is discarded (and vice versa). With this in mind, enabling manual error handling for *Merge* nodes is recommended. To do this, edit the *Merge* node and click the `advanced settings` (cog) icon in the title bar for *Advanced Settings*, then toggle `handle errors manually` to `on`:
This activates 'success' and 'fail' stubs from the *Merge* node, so you can define different paths for successful outcomes and failed outcomes. For example:
This means: * Non-merged records do not cause the pipeline to fail. * Non-merged records are output as `failed` payloads in [run logs](/documentation/brigge-for-operators/history-and-retries.md).