> For the complete documentation index, see [llms.txt](https://docs.brigge.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.brigge.ai/documentation/brigge-for-transformers/pipelines/data-transformations.md).
# Data transformations
## Introduction
Transformations are used to change the value, format, and/or node of incoming data for onward processing. Brigge includes a transformation library that can be applied in different contexts, including:
* Mapping rules, via the [map](/documentation/brigge-for-transformers/pipelines/pipeline-nodes/map-node.md) node
* Metadata values, via the [set meta](/documentation/brigge-for-transformers/pipelines/pipeline-nodes/set-meta-node.md) node
* System authentication and endpoint parameters
{% hint style="info" %}
Sometimes, the context determines specific configuration rules for a transform function. For example, configuring a [system variable value](#system-variable-value) transform in the map node has different syntax requirements than configuring the same transform for a [system endpoint](/documentation/brigge-for-engineers/systems.md).
{% endhint %}
## Available transformations
Adjust date
Adjust date by a given second, minute, hour, day, week, month, or year.
### **Example**
#### **Input**
```json
{"start_date":"2026-05-18"}
```
#### **Transform configuration**
#### **Output**
{% code overflow="wrap" %}
```json
{"start_date":"2026-06-18"}
```
{% endcode %}
Array math
Perform a mathematical operation on items in an incoming array and output the result. Available methods are:
* add
* average
* max
* min
* median
* mode
### **Example**
#### **Input**
```json
{"hours": [8.0, 7.5, 8.0, 9.0, 6.5, 4.0, 0.0]}
```
#### **Transform configuration**
#### **Output**
{% code overflow="wrap" %}
```json
{"hours":[43]}
```
{% endcode %}
Array unique
Take an array of objects and remove duplicates based on one or more fields you specify as `unique keys`. The output is (typically) a deduplicated array, in which only the first occurrence of each unique value (or value combination) is kept.
If you specify one key (for example, `employee_id`) it deduplicates on that field alone. If you specify two keys (e.g. `employee_id` and `submitted_date`) it deduplicates on the combination of both.
### **Examples**
The following examples all use the same input payload, below:
{% code lineNumbers="true" %}
```json
[
{"employee_id": "EMP-001", "submitted_date": "2025-12-14"},
{"employee_id": "EMP-001", "submitted_date": "2025-12-14"},
{"employee_id": "EMP-002", "submitted_date": "2025-12-14"},
{"employee_id": "EMP-003", "submitted_date": "2025-12-21"},
{"employee_id": "EMP-003", "submitted_date": "2025-12-07"}
]
```
{% endcode %}
Here:
* Rows 2 & 3: duplicate on both fields
* Rows 2, 3 & 4: duplicate on `submitted_date` only
* Rows 5 & 6: duplicate on `employee_id` only
#### **Example 1 (one unique key)**
Here, we want to output array objects but exclude any where the `employee_id` value is duplicated, irrespective of `submitted_date` values.
#### **Transform configuration**
#### **Output**
If an `employee_id` is duplicated, only the *first* occurrence is output. Row 2 is duplicated with row 3, so row 3 is dropped; row 5 is duplicated with row 6, so row 6 is dropped:
{% code overflow="wrap" %}
```json
{
"root": [
{
"employee_id": "EMP-001",
"submitted_date": "2025-12-14"
},
{
"employee_id": "EMP-002",
"submitted_date": "2025-12-14"
},
{
"employee_id": "EMP-003",
"submitted_date": "2025-12-21"
}
]
}
```
{% endcode %}
#### **Example 2 (two unique keys)**
Here, we want to output array objects but exclude any where the `employee_id` AND `submitted_date` values are duplicated.
#### **Transform configuration**
#### **Output**
Rows 2 & 3 are exact duplicates based on both fields, so row 3 is dropped. Notice that `EMP-003` is included twice; this is because these objects have different dates, so there's no duplication based on both fields.
{% code overflow="wrap" %}
```json
{
"root": [
{
"employee_id": "EMP-001",
"submitted_date": "2025-12-14"
},
{
"employee_id": "EMP-002",
"submitted_date": "2025-12-14"
},
{
"employee_id": "EMP-003",
"submitted_date": "2025-12-21"
},
{
"employee_id": "EMP-003",
"submitted_date": "2025-12-07"
}
]
}
```
{% endcode %}
Base64Decode
Converts a [Base64](https://en.wikipedia.org/wiki/Base64)-encoded string back into its original readable value. This is useful when a source system sends encoded data that must be decoded before being mapped or processed.
### **Example**
#### **Input**
```json
{
"employee_id": "EMP-001",
"notes": "T3ZlcnRpbWUgYXBwcm92ZWQgYnkgbWFuYWdlcg=="
}
```
#### **Transform configuration**
#### **Output**
{% code overflow="wrap" %}
```json
{
"employee_id": "EMP-001",
"notes": "Overtime approved by manager"
}
```
{% endcode %}
Base64Encode
Converts a plain text value into a [Base64](https://en.wikipedia.org/wiki/Base64)-encoded string. This is useful when a target system expects encoded data, or when sending data over a channel that can't reliably handle raw text.
### **Example**
#### **Input**
```json
{
"employee_id": "EMP-001",
"notes": "Overtime approved by manager"
}
```
#### **Transform configuration**
#### **Output**
{% code overflow="wrap" %}
```json
{
"employee_id": "EMP-001",
"notes": "T3ZlcnRpbWUgYXBwcm92ZWQgYnkgbWFuYWdlcg=="
}
```
{% endcode %}
Capitalize
Converts the value to title case (i.e. the first letter of each word is capitalised).
{% hint style="info" %}
If an incoming value is entirely uppercase, it is considered to be an abbreviation and ignored.
{% endhint %}
### **Example**
#### **Input**
```json
[
{"department": "human resources"},
{"department": "HUMAN RESOURCES"},
{"department": "HUMAN Resources"},
{"department": "HUMAN resources"},
{"department": "Human resources"},
{"department": "HR"},
{"department": "FINANCE"},
{"department": "It Support"},
{"department": "PAYROLL"},
{"department": "operations"}
]
```
#### **Transform configuration**
#### **Output**
{% code overflow="wrap" %}
```json
[
{"department":"Human Resources"},
{"department":"HUMAN RESOURCES"},
{"department":"HUMAN Resources"},
{"department":"HUMAN Resources"},
{"department":"Human Resources"},
{"department":"HR"},
{"department":"FINANCE"},
{"department":"It Support"},
{"department":"PAYROLL"},
{"department":"Operations"}
]
```
{% endcode %}
Concat
Joins two or more values into a single string, separated by a given delimiter. For example, a source system might output two values for an employe name:
```json
{"first_name": "Jane", "last_name": "Smith"}
```
But the target system requires one value:
```json
{"full_name": "Jane Smith"}
```
In this case, a mapping row would map `first_name` and `last_name` to `name`. Then a `concat` transform is added to join `first_name` and `last_name` values (separated with a space) before sending it in to the target `full_name` field.
### **Example**
#### **Input**
```json
{"first_name": "Jane", "last_name": "Smith"}
```
#### **Transform configuration**
First, we map input `first_name` and `last_name` input fields to the output `full_name` fields
With the mapping row selected, we [add a concat transformation](/documentation/brigge-for-transformers/pipelines/pipeline-nodes/map-node.md#adding-transformations) and configure settings as below:
Fields to note are summarised below:
#
Field
Summary
1
Value to transform
Provided you selected the full mapping row before choosing to add the transformation, the default value will be the last input field in the mapping. You can choose an alternative input from this dropdown list.
2
Delimiter
Enter any ASCII character to separate each joined value. A space can also be used.
3
Inputs
Add all subsequent fields to be joined. The dropdown list will include all input fields for the selected mapping - add these in join sequence.
#### **Output**
{% code overflow="wrap" %}
```json
{"full_name":"Jane Smith"}
```
{% endcode %}
Custom value
Replaces or sets a field value with a fixed, hardcoded value.
### Example 1 (override existing source values)
A target system requires a `pay_currency` field value set to YEN; incoming payloads include this field but with a range of values. You need to override any values received with YEN.
In this case, a mapping row would map `pay_currency` to `pay_currency` as normal. Then, a `custom value` transform is added for this row, with YEN as the value.
#### **Output**
{% code overflow="wrap" %}
```json
{"pay_currency": "YEN"}
```
{% endcode %}
### Example 2 (add source values)
A target system requires a `pay_currency` field value; incoming payloads do not include this field, so you need to add it with a fixed value of YEN.
In this case, the mapping row would not have a source field. Instead, select the target field and then add a `custom value` transform with YEN as the value.
#### **Input**
No input value to transform.
#### **Transform configuration**
#### **Output**
{% code overflow="wrap" %}
```json
{"pay_currency": "YEN"}
```
{% endcode %}
Data warehouse lookup
Outputs stored values from a specified [data warehouse](broken://pages/4HCG40qS88ubOtAji8OK) for onward processing.
### Example
Suppose we are syncing employee data to a target system which requires multiple fields, including `department`. But, our source data has no `department` values - incoming payloads are in the form:
{% code overflow="wrap" lineNumbers="true" %}
```json
{
"id": "EMP-001",
"first_name": "James",
"last_name": "Okafor",
"department": "Unknown"
}
```
{% endcode %}
Fortunately, employee `department` details are regularly pushed to a data warehouse with the associated employee id. This is stored in the form:
{% code overflow="wrap" lineNumbers="true" %}
```json
{
"employee_id": "EMP-001",
"department": "Engineering"
}
```
{% endcode %}
Using the `data warehouse lookup` transformation, we can populate incoming `department` field values (currently set to `Unknown`) with stored data warehouse values, matching against incoming `id` and stored `employee_id` values.
#### The config
First, we map input `id` fields to output `department` fields:
In this case, we know that input `id` and output `department` fields are not a direct mapping. That's OK - we are just defining which input fields to use for a lookup query in the data warehouse.
This makes more sense when we go on to add our `data warehouse lookup` transformation for this mapping row. For example:
Here we configure: which data warehouse to query, a lookup query that matches incoming values to warehouse records, and which warehouse field to return when a match is found. The important fields to configure are:
#
Field
Summary
1
Data warehouse
Select the data warehouse that contains values you want to output.
2
Field to select
Select which field in the data warehouse you want to output upon a successful lookup match.
3
Lookup field(s)
Choose which data warehouse field should be used to match records with input data.
4
-
Choose a match operator for the lookup. In this case, we have selected 'equals'.
5
-
Choose which input field provides the value to be matched against the data warehouse field. Any input fields selected for this mapping row are available for selection here.
Date format
Converts the incoming date format to a selected format.
{% hint style="info" %}
This transformation is exactly the same as [format date](#format-date).
{% endhint %}
### **Example**
#### **Input**
```json
{
"start_date":"2025-01-15T00:00:00Z"
}
```
#### **Transform configuration**
#### **Output**
{% code overflow="wrap" %}
```json
{
"started":"2025-01-15"
}
```
{% endcode %}
End of week
Takes an input date and shifts it forward to the configured end of week day. If the input date falls on the end of week day itself, no shift occurs. If it falls on any other day, the date moves forward to the next occurrence of that day.
The new date is output in ISO 8601 and includes a time set to one second before midnight, UTC.
### **Example**
#### **Input**
#### **Output**
{% code overflow="wrap" lineNumbers="true" %}
```json
{"eow":"2025-05-25T23:59:59+00:00"},
{"eow":"2025-05-25T23:59:59+00:00"},
{"eow":"2025-05-25T23:59:59+00:00"},
{"eow":"2025-05-25T23:59:59+00:00"},
{"eow":"2025-05-25T23:59:59+00:00"},
{"eow":"2025-05-25T23:59:59+00:00"},
{"eow":"2025-05-25T23:59:59+00:00"}
```
{% endcode %}
Environment variable value
Outputs the value of a specified environment variable. You pass the variable name as an input value and the transformation outputs the value assigned to that variable in the pipeline's runtime environment.
### **Example**
Our `production` [environment](/documentation/brigge-for-transformers/environments.md) is configured with a variable named `base_url`:
If the `base_url` variable is used in a pipeline and that pipeline runs in the `production` environment, the resolved value will be `https://prod.brigge.com`.
#### **Input**
{% code lineNumbers="true" %}
```json
{"gateway":"base_url"}
```
{% endcode %}
#### **Mapping**
#### **Transform configuration**
#### **Output**
{% code overflow="wrap" lineNumbers="true" %}
```json
{"gateway":"https://prod.brigge.com"}
```
{% endcode %}
Fallback value
Define a value to be output for a field if the incoming value is empty, or set to a given value.
### **Example**
Incoming data includes staff records where the `department` field value could be empty, `unknown`, or a valid value. We need to replace empty or `unknown` values with a fallback value of `General Staff`.
#### **Input**
{% code lineNumbers="true" %}
```json
{
"records": [
{
"id": "EMP-001",
"first_name": "James",
"last_name": "Okafor",
"department": ""
},
{
"id": "EMP-002",
"first_name": "Sarah",
"last_name": "Mitchell",
"department": "Unknown"
},
{
"id": "EMP-003",
"first_name": "David",
"last_name": "Reyes",
"department": "UNKNOWN"
},
{
"id": "EMP-004",
"first_name": "Priya",
"last_name": "Sharma",
"department": "Product"
},
{
"id": "EMP-005",
"first_name": "Laura",
"last_name": "Chen",
"department": "Finance"
}
]
}
```
{% endcode %}
#### **Mapping**
#### **Transform configuration**
{% hint style="info" %}
Values to replace are case sensitive. If you want to apply the fallback for empty values, press the spacebar, or enter ``.
{% endhint %}
#### **Output**
{% code overflow="wrap" lineNumbers="true" %}
```json
{
"records": [
{
"id": "EMP-001",
"first_name": "James",
"last_name": "Okafor",
"department": "General Staff"
},
{
"id": "EMP-002",
"first_name": "Sarah",
"last_name": "Mitchell",
"department": "General Staff"
},
{
"id": "EMP-003",
"first_name": "David",
"last_name": "Reyes",
"department": "General Staff"
},
{
"id": "EMP-004",
"first_name": "Priya",
"last_name": "Sharma",
"department": "Product"
},
{
"id": "EMP-005",
"first_name": "Laura",
"last_name": "Chen",
"department": "Finance"
}
]
}
```
{% endcode %}
Flatten
Takes a nested array structure and outputs a flat array.
### **Example**
Incoming data includes skills matrix data, where the `tags` array includes objects with an `id` and a `name`. We want to output just the tag names as a list.
#### **Input**
{% code lineNumbers="true" %}
```json
{
"article": {
"title": "Skills Matrix",
"tags": [
{"id": 1, "name": "Python"},
{"id": 2, "name": "PHP"},
{"id": 3, "name": "Javascript"}
]
}
}
```
{% endcode %}
#### **Mapping**
#### **Transform configuration**
#### **Output**
{% code overflow="wrap" lineNumbers="true" %}
```json
{
"article": {
"title": "Skills Matrix",
"tags": [
"Python",
"PHP",
"Javascript"
]
}
}
```
{% endcode %}
Format date
Converts the incoming date format to a selected format.
{% hint style="info" %}
This transformation is exactly the same as [date format](#date-format).
{% endhint %}
### **Example**
#### **Input**
```json
{
"start_date":"2025-01-15T00:00:00Z"
}
```
#### **Transform configuration**
#### **Output**
{% code overflow="wrap" %}
```json
{
"started":"2025-01-15"
}
```
{% endcode %}
Join
Collapses an array of values into a single string, joined with a delimiter between each element.
{% hint style="info" %}
More join options are available via the [*Merge* node](/documentation/brigge-for-transformers/pipelines/pipeline-nodes/merge-node.md).
{% endhint %}
### **Example**
Incoming data includes skills matrix data, where the `tags` array includes a list of skills. We want to output this list as a single string, delimited with a comma.
#### **Input**
{% code lineNumbers="true" %}
```json
{
"article": {
"title": "Skills Matrix",
"tags": ["Python", "PHP", "Javascript"]
}
}
```
{% endcode %}
#### **Mapping**
#### **Transform configuration**
#### **Output**
{% code overflow="wrap" lineNumbers="true" %}
```json
{
"article": {
"title": "Skills Matrix",
"tags": "Python,PHP,JavaScript"
}
}
```
{% endcode %}
Gunzip
Decompresses a [gzip-compressed value](#gzip) back into its original form. Incoming (compressed) data is decompressed and then mapped.
Gzip
Compresses incoming data. for the given field. Incoming data is compressed and then mapped.
Length
Returns the length of a string or array as a number. For a string, the length is the character count. For an array, it's the number of elements.
Lower
Convert incoming values to lowercase.
Pad
Pads a string to a specified total length by adding a character to the start or end.
Math
Performs a basic arithmetic operation on a numeric field using a second (defined) value and outputs the calculated value.
Metadata value
Retrieves a metadata value (defined via a [set metadata node](/documentation/brigge-for-transformers/pipelines/pipeline-nodes/set-meta-node.md) or a [custom script](/documentation/brigge-for-transformers/pipelines/pipeline-nodes/custom-script-node.md)) and outputs it to the target field. Enter the required metadata key name as the value to transform.
{% hint style="info" %}
If you are outputting a [meta variable](/documentation/brigge-for-transformers/pipelines/pipeline-variables.md#meta-variables) via the [map node](/documentation/brigge-for-transformers/pipelines/pipeline-nodes/map-node.md), do not use standard `{{meta.variable}}` syntax (`{{meta.variable_name}`}. Instead, enter the variable name with no template syntax - for example: `batch_id`.
{% endhint %}
Pgp decrypt
Reverses a [PGP encrypt](#pgp-encrypt). Takes an encrypted PGP string and decrypts it back to plain text using a PGP private key. To configure this transform you will need your PGP private key and password.
Pgp encrypt
Takes a plain text field value and encrypts it using a PGP public key, producing an encrypted string that only the holder of the corresponding private key can decrypt. To configure this transform you will need your PGP public key and (if the key is protected) a passphrase.
Pluck by key
Extracts the value of one named field from each object in an array and returns those values as a flat array.
### **Example**
Incoming data includes a `customers` array with `id` and `email` values for each customer. We want to output all `email` values as a single `emails` array.
#### **Input**
{% code lineNumbers="true" %}
```json
{
"customers": [
{ "id": "CUST-1001", "email": "priya.n@example.com" },
{ "id": "CUST-1002", "email": "marco.d@example.com" },
{ "id": "CUST-1003", "email": "amara.o@example.com" }
]
}
```
{% endcode %}
#### **Mapping**
#### **Transform configuration**
Field
Summary
Value to transform
Set to the array containing the objects you want to pluck from - for example; root.customers.*. This should be the array itself, not a single element inside it.
Key
Enter the field name to extract from each object - for example, email. This must match the actual key name in the source objects (not a path to your output field).
#### **Output**
{% code overflow="wrap" lineNumbers="true" %}
```json
{
"emails": [
"priya.n@example.com",
"marco.d@example.com",
"amara.o@example.com"
]
}
```
{% endcode %}
Pluck where
Filters an array of objects by a condition, then extracts a specific field from the matching records.
### **Example**
Incoming data includes an `items` array with `id` and `email` values for each record. We want to find all records where `id` is `1`, and output the `email` value from each matching record as a single array.
#### **Input**
{% code lineNumbers="true" %}
```json
{
"items": [
{ "email": "sophie@example.com", "id": 1 },
{ "email": "bob@example.com", "id": 2 },
{ "email": "fred@example.com", "id": 3 }
]
}
```
{% endcode %}
#### **Mapping**
#### **Transform configuration**
Field
Summary
Value to transform
Set to the array containing the objects to search - for example: root.items. This should be the array itself, not a single element inside it.
Where
The field name to match against, relative to each object in the array - for example: id.
Value
The value to match for the where field - for example: 1.
Pluck
The field name to extract from each matching object.
#### **Output**
{% code overflow="wrap" lineNumbers="true" %}
```json
{"items":"sophie@example.com"}
```
{% endcode %}
Render template
Renders a template string by resolving variable placeholders against the current pipeline context at runtime.You can combine variables to form composite values and, if needed, include static template trext.
The following variable types are supported: `{{payload.key}}`, `{{meta.key}}`, `{{env.key}}`, and {`{system.key}}`.
### **Example**
The pipeline includes a set meta node which applies a `source` value at runtime. The environment (in which this pipeline runs) includes a `source_prefix` variable with a default value set to `TEST`. We want to combine both into a single formatted string and write it to the `source` field, with some static text first.
#### **Input**
Reverse flatten
Takes a flat array and re-introduces nesting (the opposite of [flatten](#flatten)). The required element name is specified as the `target element`.
Replace
Finds a specified string within the input field value and replaces it with another string. You specify the string to find and the replacement string.
Select array index
Retrieves a single element from an array by its position. The `index` field takes a zero-based number, so `0` returns the first element, `1` the second, and so on.
Sign JWT
Takes a payload value and signs it as a JSON Web Token (JWT) using a private key. This produces a signed token string that can be verified by a target system using the corresponding public key.
Sign XML
Applies a digital signature to an incoming XML value, producing a signed XML document.
Split
Takes a delimited string and produces an array (the reverse of [join](#join)).
Start of week
Takes an input date and shifts it back to the configured start of week day. If the input date falls on the start of week day itself, no shift occurs. If it falls on any other day, it moves back to the most recent occurrence of that day.
The new date is output in ISO 8601 and includes a time set to one second before midnight, UTC.
### **Example**
#### **Input**
#### **Output**
{% code overflow="wrap" lineNumbers="true" %}
```json
{"sow":"2025-05-18T00:00:00+00:00"},
{"sow":"2025-05-18T00:00:00+00:00"},
{"sow":"2025-05-18T00:00:00+00:00"},
{"sow":"2025-05-18T00:00:00+00:00"},
{"sow":"2025-05-18T00:00:00+00:00"},
{"sow":"2025-05-18T00:00:00+00:00"},
{"sow":"2025-05-18T00:00:00+00:00"}
```
{% endcode %}
Substring
Extracts a sub-section of a string, starting at a given position.
System variable value
Output a system variable value (for example, pipeline run id, pipeline name, environment name).
{% hint style="info" %}
If you are outputting a system variable via the map node, do not use `{{system.variable}}` templates (e.g. `{{system.pipeline_run_id}`}. Instead, choose to enter the variable name manually and specify the template content (e.g. `pipeline_run_id`).
{% endhint %}
To boolean
Converts a field value to a boolean (`true` or `false`). Use `true values` and `false values` lists to define what counts as `true` and false, respectively (common variations are defined - edit, remove or add values as needed).
To float
Converts the incoming field value to a floating point number (a number with a decimal point).
To integer
Converts the incoming field value to a whole number (any decimal portion is removed).
To string
Converts a numeric or boolean value to a string.
Trim
Removes specified characters from the start and end of a string. To trim whitespace, enter a space in the `characters` field.
Upper
Convert incoming values to uppercase.
Url Param
Extracts a specific query parameter value from a URL string. Use the `key` field to specify the parameter name to extract.
