Transforms

Transforms are the basic pipeline components. There are a fair number of them, but they can be grouped roughly into families what have similar functionality.

Families:

• Input/Output
  • Add: Insert constant fields
  • Read: Import record data from an iterable using a layout
  • Sequence: Generate an integer sequence field
  • Text: Import lines as records from a text file
  • Write: Export records to a writable object using a layout
• Filtering
  • Limit: Filter rows using the row position
  • Match: Filter rows using a regular expression
  • Select: Filter rows using a callable predicate
• Splitting
  • Capture: Split a string into capture groups
  • Split: Split a field using a separator
  • Unnest: Parse a string as a record using a layout
• Merging
  • Merge: Combine multiple fields into one
  • Nest: Merge fields into a string record using a layout
• Reformatting
  • Cast: Change the data type of a field
  • Fill: Fill in missing values by copying down
  • Format: Replace a value with a computed value
  • Lookup: Replace a value with one from a lookup table
  • Lag: Shift values by a number of rows
  • Lift: Fill in missing values by copying up
  • Replace: Rewrite a string using capture groups
• Reshaping
  • Copy: Duplicate fields
  • Divide: Separate one field into two using a predicate
  • Drop: Remove unneeded fields
  • Fold: Rotate many fields into one field
  • Iterate: Expand ragged records into multiple rows
  • Project: Compute a function value as a new field
  • Unfold: Rotate one field to many
• Internals