Stages

Pronghorn

AbstractAppendablePayloadResponseStage

Abstraction for response payloads using UTF-8 or other text encoding. See ByteArrayPayloadResponseStage to implement a custom REST responder that responds only using bytes. Use this to define custom HTTP REST behavior.

InputPipe<HTTPRequestSchema> inputs: HTTP schema request inputs

OutputPipe<ServerResponseSchema> outputs: Response schema outputs


InputPipe<HTTPRequestSchema> inputs: Input request

Pipe<?> otherInputs: Other inputs

OutputPipe<ServerResponseSchema> outputs: Output response


BlockingSupportStage

Stage that allows for blocking calls, e.g. to make a call to a database and then wait until a response is received.

InputPipe<T> input: Input that will be released until ready

OutputPipe<P> output: Pipe onto which the input will be released on


ByteArrayEqualsStage

Takes an array and an input pipe. If the bytes match the raw bytes, wasEqual will be true. For testing RawDataSchema.

InputPipe<RawDataSchema> input: Pipe with RawDataSchema that will compares to the byte array

OutputNo output pipes

ByteArrayPayloadResponseStage

Extend this class to write your own REST responders. Implement payload() to return a byte array. See AbstractAppendablePayloadResponseStage for implementing a UTF-8 REST responder.

InputPipe<HTTPRequestSchema> inputs: Input pipes with HTTPRequestSchema

OutputPipe<ServerResponseSchema> outputs: Output pipes as several ServerResponseSchema


InputPipe<HTTPRequestSchema> inputs: Input pipes with HTTPRequestSchema

Pipe otherInputs: Multiple other input pipes to respond to

OutputPipe<ServerResponseSchema> outputs: Output pipes as several ServerResponseSchema


ByteArrayProducerStage

Takes an array of bytes and writes them to the output pipe once. Useful for testing RawDataSchema

InputNo input pipes
OutputPipe<RawDataSchema> output: The pipe onto which the rawData byte array will be written to


ClientSocketReaderStage

Client-side stage that reads sockets using a ClientCoordinator based on a release acknowledgment. Accepts only expected calls (unlike ServerSocketReaderStage), since it is a client.

InputPipe<ReleaseSchema> parseAck: The release acknowledgment input pipes.

OutputPipe output: The read payload from the socket.


ClientSocketWriterStage

Write to a socket using a client coordinator.

InputPipe input: Payload that will be written to socket.

OutputNo output pipes

ConsoleJSONDumpStage

For some Schema T encode this data in JSON and write it to the target appendable. Can be set to assume that bytes are UTF8. The default output is System.out

InputPipe<T> input: Pipe to be dumped

OutputNo output pipes

ConsoleSummaryStage

For some Schema T keeps running totals of each message type. Periodically reports the Number of each message type to target appendable. Default target is system.out.

InputPipe<T> input: Any schema input pipe that should be summarized.

OutputNo output pipes

DotModuleStage

Rest service stage which responds with the dot file needed to display the telemetry. This dot file is a snapshot of the system representing its state within the last 40 ms.

InputPipe<HTTPRequestSchema> inputs: Pipe containing request for generated .dot file.

OutputPipe<ServerResponseSchema> outputs: Pipe that will contain HTTP response containing newly generated .dot file.


DummyRestStage

Dummy REST stage that simply returns 501. Use this to quickly build yourself a web server without worrying about implementation, or to test concurrent behavior.

InputPipe<HTTPRequestSchema> inputPipes: Input pipes containing the HTTP request

OutputPipe<ServerResponseSchema> outputs: No output except 501 error.


FileBlobReadStage

Reads files from disk as blobs based on an input path.

InputNo input pipes
OutputPipe<RawDataSchema> output: Pipe that file from inputPathString will be written to.


FileBlobWriteStage

Writes data to file on disk (in blobs).

InputPipe<RawDataSchema> input: RawDataSchema that will be written to file.

OutputNo output pipes

FileReadModuleStage

Minimal memory usage and leverages SSD for file reading module. Provides HTTP file transfer, i.e. browser requests file and this stage returns it.

InputPipe<HTTPRequestSchema> inputs: Array of requests for file(s).

OutputPipe<ServerResponseSchema> outputs: Array of responses with the requested file(s).


HTTP1xResponseParserStage

Parses HTTP1.x responses from the server and sends an acknowledgment to an output pipe to be sent back to a request stage.

InputPipe<NetPayloadSchema> input: Pipe containing the HTTP payload.

OutputPipe<NetResponseSchema> output: Net response.

Pipe<ReleaseSchema> ackStop: Acknowledgment for forwarding.


HTTP1xRouterStage

Main HTTP router. Quickly redirects any incoming traffic to corresponding routes.

InputPipe<NetPayloadSchema> input: The payload that will be routed.

OutputPipe<HTTPRequestSchema> outputs: The HTTP request parsed from the NetPayloadSchema.

Pipe<HTTPLogRequestSchema> log: Logging output.

Pipe<ReleaseSchema> ackStop: Acknowledgment for ReleaseSchema.


HTTPClientRequestStage

Takes a HTTP client request and responds with a net payload using a TrieParserReader.

InputPipe<ClientHTTPRequestSchema> input: Multiple HTTP client requests

OutputPipe output: Multiple net payload responses


HTTPLogUnificationStage

Takes multiple HTTP log requests and responses and turns them into a RawDataSchema for easier output.

InputPipe<HTTPLogRequestSchema> requestInputs: All HTTP request logs.

Pipe<HTTPLogResponseSchema> responseInputs: All HTTP response logs.

OutputPipe<RawDataSchema> output: All the request and response logs combined onto the output pipe as a RawDataSchema.


HTTPRequestJSONExtractionStage

Using a JSONExtractor, takes a HTTP request with JSON and turns it into a ServerResponseSchema onto a ServerResponseSchema pipe.

InputPipe<HTTPRequestSchema> input: The HTTP request containing JSON.

OutputPipe<HTTPRequestSchema> output: The HTTP response.

Pipe<ServerResponseSchema> err: Contains ServerResponseSchema if error occurred


NetResponseDumpStage

Dumps a NetResponseSchema onto an Appendable target.

InputPipe<NetResponseSchema> input: The net response input pipe.

OutputNo output pipes

NetResponseJSONStage

Parses a JSON response using a JSONStreamVisitor.

InputPipe<NetResponseSchema> input: The NetResponseSchema containing the JSON to be parsed.

OutputNo output pipes

InputPipe<NetResponseSchema> input: The NetResponseSchema containing the JSON to be parsed.

OutputPipe<M> output: Put the parsed JSON onto this output pipe.

Pipe otherOutputs: Put the parsed JSON onto multiple output pipes.


OrderSupervisorStage

Consumes the sequence number in order and holds a pool entry for this connection. Sends the data in order to the right pool entry for encryption to be applied down stream.

InputPipe<ServerResponseSchema> inputPipes: The server response which will be supervised.

OutputPipe<HTTPLogResponseSchema> log: The log output pipe.

Pipe<NetPayloadSchema> outgoingPipes: The net payload after order is enforced.


PipeCleanerStage

For some schema T consumes all the data on the input pipes. The data is not reported anyway.

InputPipe<T> input: Any schema input pipe

OutputNo output pipes

InputPipe<T> input: An array of any input pipe

OutputNo output pipes

PipeMonitorStage

Stage which monitors pipes in real time. This data is passed along for the telemetry.

InputNo input pipes
OutputPipe<?> observedRingBuffer: observation pipes

Pipe notifyRingBuffer: notify pipes


PipeNoOp

Empty stage which does no work. It takes an output pipe and writes nothing.

InputNo input pipes
OutputPipe<T> output: Does nothing.


ReplicatorStage

Given n ring buffers with the same FROM/Schema Does not require schema knowledge for copy but does ensure targets and source have the same FROM.

InputPipe<T> source: Any input pipe that will be replicated.

OutputPipe<T> a: Target pipe; will be joined with b.

Pipe<T> b: Target pipe; will be joined with a.


InputPipe<T> source: Any input pipe that will be replicated.

OutputPipe<T> targets: Multiple targets to which the source pipe is replicated.


ResourceModuleStage

Fetches resources as HTTP responses based on request.

InputPipe<HTTPRequestSchema> inputs: Multiple HTTPRequest that are requesting resource(s).

OutputPipe<ServerResponseSchema> outputs: Responds with the resource(s) if it/they exists.


RoundRobinRouteStage

For some schema T distributes the records across N output streams of the same

InputPipe<T> input: Schema T input pipes that will be distributed

OutputPipe<T> outputs: Pipes on which the input pipe will be distributed on


SSLEngineUnWrapStage

Unwraps encrypted content for HTTPS/SSL.

InputPipe<NetPayloadSchema> encryptedContent: Encrypted content to be unencrypted.

OutputPipe<NetPayloadSchema> outgoingPipeLines: Unencrypted content.

Pipe<ReleaseSchema> relesePipe: Acknowledgment for release.

Pipe<NetPayloadSchema> handshakePipe: Responds with a handshake.


SSLEngineWrapStage

Wraps plain content into encrypted content for HTTPS/SSL.

InputPipe<NetPayloadSchema> plainContent: Plain content payload to be encrypted.

OutputPipe encryptedContent: Encrypted payload.


SequentialFileReadWriteStage

Sequentially read and write to file.

InputPipe<SequentialCtlSchema> control: Schemas defining sequential control.

Pipe<RawDataSchema> input: Data being read.

OutputPipe<SequentialRespSchema> response: Schemas defining sequential response.

Pipe<RawDataSchema> output: Data being written.


SequentialReplayerStage

Replays a sequential read/write based on store requests and load responses.


ServerNewConnectionStage

General base class for server construction. Server should minimize garbage but unlike client may not be possible to remove it. No protocol specifics are found in this class, only socket usage logic.

InputNo input pipes
OutputPipe<ServerConnectionSchema> newClientConnections: The ServerConnectionSchema containing the newest client information.


ServerSocketReaderStage

Server-side stage that reads from the socket. Useful for building a server. Accepts unexpected calls (unlike ClientSocketReaderStage).

InputPipe<ReleaseSchema> ack: The release acknowledgment.

OutputPipe<NetPayloadSchema> output: The read payload from the socket.


ServerSocketWriterStage

Server-side stage that writes back to the socket. Useful for building a server.

InputPipe dataToSend: The data to be written to the socket.

OutputNo output pipes

TapeReadStage

Reads a Pronghorn “tape” from disk and writes it back onto a pipe. Tape is a file format that mimics Pronghorn pipe data format, useful for very structured formats.

InputNo input pipes
OutputPipe<RawDataSchema> output: Writes the read tape directly back onto a RawDataSchema pipe.


TapeWriteStage

Write data from a pipe directly to disk. This records both the slab and blob data without any schema concerns. This is the simplest way to save data in a pipe to disk.

InputPipe<T> source: The input pipe that will be written as a tape to disk.

OutputNo output pipes

ToOutputStreamStage

Takes the RawDataSchema on the input pipe and writes it to an output stream.

InputPipe<RawDataSchema> inputRing: The RawDataSchema pipe that will be written to the output stream.

OutputNo output pipes

UpgradeToWebSocketStage

Go from HTTP-based server directly to web sockets for massive speed improvements.

@author Nathan Tippy

@see Pronghorn

InputPipe<HTTPRequestSchema> inputPipes: HTTP request schema to be upgraded.

OutputPipe<ServerResponseSchema> outputs: Resulting ServerResponseSchema (multiple) after upgrade.


JPG-Raster

BMPDumperStage

Dumps the BMP raster of a JPG Schema into a file.

InputPipe<JPGSchema> input: Input pipe to be dumped

OutputNo output pipes

BMPScannerStage

Scans a BMP and puts it onto a JPG Schema pipe.

InputNo input pipes
OutputPipe<JPGSchema> output: Valid JPG schema from BMP will be put onto this pipe


ForwardDCTStage

Updated Forward DCT Algorithm is based on the Guetzli JPEG encoder’s DCT implementation. This code can be found here: DCT_Double.css

InputPipe<JPGSchema> input: Input JPG schema that Forward DCT algorithm will be applied to.

OutputPipe<JPGSchema> output: Output JPG schema with DCT algorithm applied.


HuffmanEncoderStage

Using Huffman encoding, this creates the necessary JPG headers and generates MCUs. Huffman coding

InputPipe<JPGSchema> input: The JPG schema on which Huffman encoding will be applied to

OutputNo output pipes

InverseDCTStage

Updated Inverse DCT Algorithm is based on the Guetzli JPEG encoder’s DCT implementation. This code can be found here: DCT_Double.cc

InputPipe<JPGSchema> input: Defined JPG schema

OutputPipe<JPGSchema> output: Outputs the JPG schema onto the pipe


InverseQuantizerStage

Performs inverse quantization on a JPG schema pipe and returns the results back onto a JPG schema pipe. Examples can be found here: Inverse Quantization

InputPipe<JPGSchema> input: Input JPG schema

OutputPipe<JPGSchema> output: Output JPG schema


JPGScannerStage

Scans a JPG using a Huffman decoder. This creates a useful JPG schema.

InputNo input pipes
OutputPipe<JPGSchema> output: Output JPG schema


QuantizerStage

Performs quantization on a JPG schema pipe and returns the results back onto a JPG schema pipe. Examples can be found here: Quantization

InputPipe<JPGSchema> input: The JPG schema on which quantization will be applied to

OutputPipe<JPGSchema> output: JPG schema with applied quantization


RGBToYCbCrStage

Converts RGB JPG schema to YCbCr JPG schema

InputPipe<JPGSchema> input: The JPG schema that is currently in RGB

OutputPipe<JPGSchema> output: JPG schema that is now in YCbCr color format


YCbCrToRGBStage

Converts YCbCr to RGB of the passed in JPG

InputPipe<JPGSchema> input: Input JPG Schema

OutputPipe<JPGSchema> output: Outputted JPG Schema