Skip to content
Lakehouse Engine Documentation
Custom expectations
Initializing search
lakehouse-engine
Lakehouse Engine
How to use the Lakehouse Engine?
API Documentation
Lakehouse Engine Documentation
lakehouse-engine
Lakehouse Engine
How to use the Lakehouse Engine?
How to use the Lakehouse Engine?
Overview
Algorithms
Algorithms
Data Loader
Data Loader
Overview
Scenarios
Scenarios
Append Load from JDBC with PERMISSIVE mode (default)
Append Load with FAILFAST
Batch Delta Load Init, Delta and Backfill with Merge
Custom Transformer
Custom Transformer (SQL)
Extract from SAP B4 ADSOs
Extract from SAP BW DSOs
Extract from SFTP
Extract using JDBC connection
Filtered Full Load
Filtered Full Load with Selective Replace
Flatten Schema and Explode Columns
Full Load
Read from Dataframe
Streaming Append Load with DROPMALFORMED
Streaming Append Load with Optimize Dataset Terminator
Streaming Delta Load with Group and Rank Condensation
Streaming Delta Load with Late Arriving and Out of Order Events (with and without watermarking)
Write and Read Dataframe
Write to Console
Write to REST API
Data Quality
Data Quality
Overview
Scenarios
Scenarios
Custom Expectations
Data Quality Validator
Minimal Example
Prisma
Result Sink
Row Tagging
Validations Failing
Reconciliator
Reconciliator
Overview
Sensor
Sensor
Overview
Scenarios
Scenarios
Delta Table
Sensor from other Sensor Delta Table
Sensor from Files
Sensor from JDBC
Sensor from Kafka
Sensor from SAP
Update Sensor control Delta Table after processing the data
GAB
GAB
Overview
Step-by-Step
API Documentation
API Documentation
Packages
Packages
Engine
Algorithms
Algorithms
Algorithm
Data loader
Dq validator
Exceptions
Gab
Reconciliator
Sensor
Configs
Core
Core
Dbfs file manager
Definitions
Exec env
Executable
File manager
Gab manager
Gab sql generator
S3 file manager
Sensor manager
Table manager
Dq processors
Dq processors
Dq factory
Exceptions
Validator
Custom expectations
Custom expectations
Expect column pair a to be not equal to b
Expect column pair a to be smaller or equal than b
Expect column pair date a to be greater than or equal to date b
Expect column values to be date not older than
Expect column values to not be null or empty string
Expect multicolumn column a must equal b or c
Expect queried column agg value to be
Io
Io
Exceptions
Reader
Reader factory
Writer
Writer factory
Readers
Readers
Dataframe reader
File reader
Jdbc reader
Kafka reader
Query reader
Sap b4 reader
Sap bw reader
Sftp reader
Table reader
Writers
Writers
Console writer
Dataframe writer
Delta merge writer
File writer
Jdbc writer
Kafka writer
Rest api writer
Sharepoint writer
Table writer
Terminators
Terminators
Cdf processor
Dataset optimizer
Notifier
Notifier factory
Sensor terminator
Spark terminator
Terminator factory
Notifiers
Notifiers
Email notifier
Notification templates
Transformers
Transformers
Aggregators
Column creators
Column reshapers
Condensers
Custom transformers
Data maskers
Date transformers
Exceptions
Filters
Joiners
Null handlers
Optimizers
Regex transformers
Repartitioners
Transformer factory
Unions
Watermarker
Utils
Utils
Acon utils
Databricks utils
Dq utils
Engine usage stats
Expectations utils
File utils
Gab utils
Logging handler
Rest api
Schema utils
Sharepoint utils
Sql parser utils
Configs
Configs
Config utils
Extraction
Extraction
Jdbc extraction utils
Sap b4 extraction utils
Sap bw extraction utils
Sftp extraction utils
Storage
Storage
Dbfs storage
File storage
File storage functions
Local fs storage
S3 storage
Custom expectations
Package containing custom DQ expectations available in the lakehouse engine.
Back to top