Skip to content

Kafka reader

Module to define behaviour to read from Kafka.

KafkaReader

Bases: Reader

Class to read from Kafka.

Source code in mkdocs/lakehouse_engine/packages/io/readers/kafka_reader.py 
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
class KafkaReader(Reader):
    """Class to read from Kafka."""

    def __init__(self, input_spec: InputSpec):
        """Construct KafkaReader instances.

        Args:
            input_spec: input specification.
        """
        super().__init__(input_spec)

    def read(self) -> DataFrame:
        """Read Kafka data.

        Returns:
            A dataframe containing the data from Kafka.
        """
        df = ExecEnv.SESSION.readStream.load(
            format=InputFormat.KAFKA.value,
            **self._input_spec.options if self._input_spec.options else {},
        )

        return df

__init__(input_spec)

Construct KafkaReader instances.

Parameters:

Name Type Description Default
input_spec InputSpec

input specification.

 required
Source code in mkdocs/lakehouse_engine/packages/io/readers/kafka_reader.py 
13
14
15
16
17
18
19
def __init__(self, input_spec: InputSpec):
    """Construct KafkaReader instances.

    Args:
        input_spec: input specification.
    """
    super().__init__(input_spec)

read()

Read Kafka data.

Returns:

Type Description
DataFrame

A dataframe containing the data from Kafka.
Source code in mkdocs/lakehouse_engine/packages/io/readers/kafka_reader.py 
21
22
23
24
25
26
27
28
29
30
31
32
def read(self) -> DataFrame:
    """Read Kafka data.

    Returns:
        A dataframe containing the data from Kafka.
    """
    df = ExecEnv.SESSION.readStream.load(
        format=InputFormat.KAFKA.value,
        **self._input_spec.options if self._input_spec.options else {},
    )

    return df