Skip to content

Lakehouse Engine Documentation

Overview

Initializing search

lakehouse-engine

Lakehouse Engine
How to use the Lakehouse Engine?
API Documentation

Lakehouse Engine Documentation

lakehouse-engine

Lakehouse Engine
How to use the Lakehouse Engine?
How to use the Lakehouse Engine?
- Overview
- Algorithms
  Algorithms
  - Data Loader
    Data Loader
    
    Overview
    
    Scenarios
    Scenarios
    
    Append Load from JDBC with PERMISSIVE mode (default)
    
    Append Load with FAILFAST
    
    Batch Delta Load Init, Delta and Backfill with Merge
    
    Custom Transformer
    
    Custom Transformer (SQL)
    
    Extract from SAP B4 ADSOs
    
    Extract from SAP BW DSOs
    
    Extract from SFTP
    
    Extract using JDBC connection
    
    Filtered Full Load
    
    Filtered Full Load with Selective Replace
    
    Flatten Schema and Explode Columns
    
    Full Load
    
    Read from Dataframe
    
    Streaming Append Load with DROPMALFORMED
    
    Streaming Append Load with Optimize Dataset Terminator
    
    Streaming Delta Load with Group and Rank Condensation
    
    Streaming Delta Load with Late Arriving and Out of Order Events (with and without watermarking)
    
    Write and Read Dataframe
    
    Write to Console
    
    Write to REST API
  - Data Quality
    Data Quality
    
    Overview
    
    Scenarios
    Scenarios
    
    Custom Expectations
    
    Data Quality Validator
    
    Minimal Example
    
    Prisma
    
    Result Sink
    
    Row Tagging
    
    Validations Failing
  - Reconciliator
    Reconciliator
    
    Overview
  - Sensor
    Sensor
    
    Overview
    
    Scenarios
    Scenarios
    
    Delta Table
    
    Sensor from other Sensor Delta Table
    
    Sensor from Files
    
    Sensor from JDBC
    
    Sensor from Kafka
    
    Sensor from SAP
    
    Update Sensor control Delta Table after processing the data
  - GAB
    GAB
    
    Overview
    
    Step-by-Step
API Documentation
API Documentation
- Packages
  Packages
  - Engine
  - Algorithms
    
    Algorithms
    
    Algorithm
    
    Data loader
    
    Dq validator
    
    Exceptions
    
    Gab
    
    Reconciliator
    
    Sensor
  - Configs
  - Core
    
    Core
    
    Dbfs file manager
    
    Definitions
    
    Exec env
    
    Executable
    
    File manager
    
    Gab manager
    
    Gab sql generator
    
    S3 file manager
    
    Sensor manager
    
    Table manager
  - Dq processors
    
    Dq processors
    
    Dq factory
    
    Exceptions
    
    Validator
    
    Custom expectations
    
    Custom expectations
    
    Expect column pair a to be not equal to b
    
    Expect column pair a to be smaller or equal than b
    
    Expect column pair date a to be greater than or equal to date b
    
    Expect column values to be date not older than
    
    Expect column values to not be null or empty string
    
    Expect multicolumn column a must equal b or c
    
    Expect queried column agg value to be
  - Io
    
    Io
    
    Exceptions
    
    Reader
    
    Reader factory
    
    Writer
    
    Writer factory
    
    Readers
    
    Readers
    
    Dataframe reader
    
    File reader
    
    Jdbc reader
    
    Kafka reader
    
    Query reader
    
    Sap b4 reader
    
    Sap bw reader
    
    Sftp reader
    
    Table reader
    
    Writers
    
    Writers
    
    Console writer
    
    Dataframe writer
    
    Delta merge writer
    
    File writer
    
    Jdbc writer
    
    Kafka writer
    
    Rest api writer
    
    Sharepoint writer
    
    Table writer
  - Terminators
    
    Terminators
    
    Cdf processor
    
    Dataset optimizer
    
    Notifier
    
    Notifier factory
    
    Sensor terminator
    
    Spark terminator
    
    Terminator factory
    
    Notifiers
    
    Notifiers
    
    Email notifier
    
    Notification templates
  - Transformers
    
    Transformers
    
    Aggregators
    
    Column creators
    
    Column reshapers
    
    Condensers
    
    Custom transformers
    
    Data maskers
    
    Date transformers
    
    Exceptions
    
    Filters
    
    Joiners
    
    Null handlers
    
    Optimizers
    
    Regex transformers
    
    Repartitioners
    
    Transformer factory
    
    Unions
    
    Watermarker
  - Utils
    
    Utils
    
    Acon utils
    
    Databricks utils
    
    Dq utils
    
    Engine usage stats
    
    Expectations utils
    
    File utils
    
    Gab utils
    
    Logging handler
    
    Rest api
    
    Schema utils
    
    Sharepoint utils
    
    Sql parser utils
    
    Configs
    
    Configs
    
    Config utils
    
    Extraction
    
    Extraction
    
    Jdbc extraction utils
    
    Sap b4 extraction utils
    
    Sap bw extraction utils
    
    Sftp extraction utils
    
    Storage
    
    Storage
    
    Dbfs storage
    
    File storage
    
    File storage functions
    
    Local fs storage
    
    S3 storage

How to use the Lakehouse Engine?¶

Lakehouse engine usage examples for all the algorithms and other core functionalities.

Data Loader
Data Quality
Reconciliator
Sensor
GAB

© 2025 adidas

Made with Material for MkDocs