Background-thread Parquet compaction sink fed by a lock-free ring buffer. More...

#include <streaming_sink.hpp>

Classes
struct	Options
	Configuration options for StreamingSink::create(). More...

Public Member Functions
	StreamingSink (StreamingSink &&)=default

StreamingSink &	operator= (StreamingSink &&)=default

	StreamingSink (const StreamingSink &)=delete

StreamingSink &	operator= (const StreamingSink &)=delete

	~StreamingSink ()

expected< void >	submit (StreamRecord rec)
	Submit a fully-constructed StreamRecord to the ring buffer.

expected< void >	submit (int64_t timestamp_ns, uint32_t type_id, const uint8_t *data, size_t size)
	Submit a record from raw bytes.

expected< void >	submit (int64_t timestamp_ns, uint32_t type_id, std::string_view sv)
	Submit a record from a string_view payload.

void	start ()
	Start the background compaction thread (no-op if already running).

expected< void >	stop ()
	Stop the background thread, drain remaining records, and close open files.

void	stop_nowait ()
	Signal the background thread to stop without waiting for it to finish.

expected< void >	flush ()
	Drain the ring buffer and write all pending records, then close the current Parquet file so it has a valid footer and is immediately readable.

uint64_t	records_submitted () const
	Total number of records successfully submitted to the ring buffer.

int64_t	records_written () const
	Total number of records written to Parquet files.

uint64_t	records_dropped () const
	Total number of records dropped due to ring buffer overflow.

int64_t	files_written () const
	Number of completed Parquet output files.

int64_t	bytes_written () const
	Approximate total bytes written to the current output file.

std::vector< std::string >	output_files () const
	List of completed Parquet output file paths (thread-safe snapshot).

Static Public Member Functions
static expected< StreamingSink >	create (Options opts)
	Create a StreamingSink with the given options.

Detailed Description

Background-thread Parquet compaction sink fed by a lock-free ring buffer.

Producers submit StreamRecords via submit(). A background thread drains the ring buffer at the configured flush_interval, writing records into Parquet files with automatic row-group flushing and file rolling.

Movable via unique_ptr<Impl>; all non-movable state (mutex, thread, ring) lives inside the Impl struct. Use StreamingSink::create() to construct.

See also: StreamRecord, HybridReader

Definition at line 405 of file streaming_sink.hpp.

Constructor & Destructor Documentation

◆ StreamingSink() [1/2]

signet::forge::StreamingSink::StreamingSink ( StreamingSink && )

default

◆ StreamingSink() [2/2]

signet::forge::StreamingSink::StreamingSink ( const StreamingSink & )

delete

◆ ~StreamingSink()

signet::forge::StreamingSink::~StreamingSink ( )

inline

Definition at line 477 of file streaming_sink.hpp.

Member Function Documentation

◆ bytes_written()

int64_t signet::forge::StreamingSink::bytes_written ( ) const

inline

Approximate total bytes written to the current output file.

Definition at line 558 of file streaming_sink.hpp.

◆ create()

static expected< StreamingSink > signet::forge::StreamingSink::create ( Options opts )

inlinestatic

Create a StreamingSink with the given options.

Creates the output directory if it does not exist. Validates that ring_buffer_capacity is a power of two. If auto_start is true (default), the background compaction thread is started immediately.

Parameters

opts	Configuration options.

Returns: StreamingSink on success, Error on validation or I/O failure.

Definition at line 434 of file streaming_sink.hpp.

◆ files_written()

int64_t signet::forge::StreamingSink::files_written ( ) const

inline

Number of completed Parquet output files.

Definition at line 556 of file streaming_sink.hpp.

◆ flush()

expected< void > signet::forge::StreamingSink::flush ( )

inline

Drain the ring buffer and write all pending records, then close the current Parquet file so it has a valid footer and is immediately readable.

Blocks until complete.

Returns: Error on I/O failure.

Definition at line 543 of file streaming_sink.hpp.

◆ operator=() [1/2]

StreamingSink & signet::forge::StreamingSink::operator= ( const StreamingSink & )

delete

◆ operator=() [2/2]

StreamingSink & signet::forge::StreamingSink::operator= ( StreamingSink && )

default

◆ output_files()

std::vector< std::string > signet::forge::StreamingSink::output_files ( ) const

inline

List of completed Parquet output file paths (thread-safe snapshot).

Definition at line 565 of file streaming_sink.hpp.

◆ records_dropped()

uint64_t signet::forge::StreamingSink::records_dropped ( ) const

inline

Total number of records dropped due to ring buffer overflow.

Definition at line 554 of file streaming_sink.hpp.

◆ records_submitted()

uint64_t signet::forge::StreamingSink::records_submitted ( ) const

inline

Total number of records successfully submitted to the ring buffer.

Definition at line 550 of file streaming_sink.hpp.

◆ records_written()

int64_t signet::forge::StreamingSink::records_written ( ) const

inline

Total number of records written to Parquet files.

Definition at line 552 of file streaming_sink.hpp.

◆ start()

void signet::forge::StreamingSink::start ( )

inline

Start the background compaction thread (no-op if already running).

Definition at line 527 of file streaming_sink.hpp.

◆ stop()

expected< void > signet::forge::StreamingSink::stop ( )

inline

Stop the background thread, drain remaining records, and close open files.

Blocks until the thread has joined.

Returns: Error on I/O failure during final drain.

Definition at line 531 of file streaming_sink.hpp.

◆ stop_nowait()

void signet::forge::StreamingSink::stop_nowait ( )

inline

Signal the background thread to stop without waiting for it to finish.

Definition at line 533 of file streaming_sink.hpp.

◆ submit() [1/3]

expected< void > signet::forge::StreamingSink::submit	(	int64_t	timestamp_ns,
		uint32_t	type_id,
		const uint8_t *	data,
		size_t	size
	)

inline

Submit a record from raw bytes.

Parameters

timestamp_ns	Wall-clock timestamp in nanoseconds.
type_id	User-defined record type tag.
data	Raw payload bytes.
size	Number of bytes.

Returns: Error if the ring buffer is full.

Definition at line 499 of file streaming_sink.hpp.

◆ submit() [2/3]

expected< void > signet::forge::StreamingSink::submit	(	int64_t	timestamp_ns,
		uint32_t	type_id,
		std::string_view	sv
	)

inline

Submit a record from a string_view payload.

Parameters

timestamp_ns	Wall-clock timestamp in nanoseconds.
type_id	User-defined record type tag.
sv	Payload as string_view.

Returns: Error if the ring buffer is full.

Definition at line 513 of file streaming_sink.hpp.

◆ submit() [3/3]

expected< void > signet::forge::StreamingSink::submit ( StreamRecord rec )

inline

Submit a fully-constructed StreamRecord to the ring buffer.

Thread-safe (serialized via internal mutex).

Parameters

rec	Record to submit.

Returns: Error if the ring buffer is full (record dropped).

Definition at line 489 of file streaming_sink.hpp.

The documentation for this class was generated from the following file:

include/signet/ai/streaming_sink.hpp

Classes

Public Member Functions

Static Public Member Functions

Detailed Description

Constructor & Destructor Documentation

◆ StreamingSink() [1/2]

◆ StreamingSink() [2/2]

◆ ~StreamingSink()

Member Function Documentation

◆ bytes_written()

◆ create()

◆ files_written()

◆ flush()

◆ operator=() [1/2]

◆ operator=() [2/2]

◆ output_files()

◆ records_dropped()

◆ records_submitted()

◆ records_written()

◆ start()

◆ stop()

◆ stop_nowait()

◆ submit() [1/3]

◆ submit() [2/3]

◆ submit() [3/3]