Nbuff

Smart buffer. Usage scenario: You are reading bulk newline delimited lines (or any other way structured data) from TCP socket and process it as soon as possible. Every time you received something like:

'line1\nline2\nli' <- note incomplete last line.

from network to your socket buffer you can process 'line1' and 'line2' and then you have keep whole buffer (or copy and save it incomplete part 'li') just because you have some incomplete data.

This leads to unnecessary allocations and data movement (if you choose to free old buffer and save incomplete part) or memory wasting (if you choose to preallocate very large buffer and keep processed and incomplete data).

Nbuff solve this problem by using memory pool and smart pointers - it take memory chunks from pool for reading from network(file, etc...), and authomatically return buffer to pool as soon as you processed all data in it and moved 'processed' pointer forward.

So Nbuff looks like some "window" on the list of buffers, filled with network data, and as soon as buffer moves out of this window and dereferenced it will be automatically returned to memory pool. Please note - there is no GC allocations, everything is done usin malloc.

Here is sample of buffer lifecycle (see code or docs for exact function signatures): Nbuff nbuff; - initialize nbuff structure. buffer = Nbuff.get(bsize) - gives you non-copyable mutable buffer of size >= bsize socket.read(buffer.data) - fill buffer with network data. nbuff.append(buffer) - convert mutable non-copyable buffer to immutable shareable buffer and append it to nbuff valuable_data = nbuff.data(0, 100); - get immutable view to first 100 bytes of nbuff (they can be non-continous) nbuff.pop(100) - release fist 100 bytes of nbuff, marking them as "processed". If there are any previously appended buffers which become unreferenced at this point, then they will be returned to the pool. When nbuff goes out of scope all its buffers will be returned to pool.

struct Nbuff {

this(this);

this(string s);

this(immutable(ubyte)[] s);

void opAssign(T other);

static auto get(size_t size);

bool empty();

auto length();

void append(string s);

void append(UniquePtr!(MutableMemoryChunk) c, size_t l);

void append(NbuffChunk source);

void append(NbuffChunk source, size_t pos, size_t len);

auto frontChunk();

void popChunk();

void pop(long n);

NbuffChunk data(size_t beg, size_t end);

NbuffChunk data();

Nbuff opSlice(size_t start, size_t end);

auto opDollar();

bool opEquals(string other);

bool opEquals(ubyte[] other);

bool opEquals(R other);

auto opIndex(size_t i);

Nbuff[3] findSplitOn(string s, size_t start_from);

Nbuff[3] findSplitOn(immutable(ubyte)[] b, size_t start_from);

bool beginsWith(ubyte[] b);

bool beginsWith(string b);

int countUntil(const(ubyte)[] b, size_t start_from);

import core.sys.posix.sys.uio : iovec;;

version(Posix)

int toIoVec(iovec* v, int n);

}

Constructors

this this(string s): Undocumented in source.
this this(immutable(ubyte)[] s): Undocumented in source.

Destructor

~this ~this(): dispose all allocated pages and dec refs for any data

Postblit

this(this) this(this): copy references to RC-data

Members

Functions

append void append(string s): Undocumented in source. Be warned that the author may not have intended to support it.
append void append(UniquePtr!(MutableMemoryChunk) c, size_t l): Undocumented in source. Be warned that the author may not have intended to support it.
append void append(NbuffChunk source): Undocumented in source. Be warned that the author may not have intended to support it.
append void append(NbuffChunk source, size_t pos, size_t len): append some part of other nbuff
beginsWith bool beginsWith(string b): Undocumented in source. Be warned that the author may not have intended to support it.
beginsWith bool beginsWith(ubyte[] b): Undocumented in source. Be warned that the author may not have intended to support it.
clear void clear(): Undocumented in source. Be warned that the author may not have intended to support it.
countUntil int countUntil(const(ubyte)[] b, size_t start_from): Undocumented in source. Be warned that the author may not have intended to support it.
data NbuffChunk data(size_t beg, size_t end): return immutable continuous view to [beg, end] of this nbuff. Data copy: We can avoid data copy if [beg, end] lies within single stored immutable chunk. Then we just return ref to this chunk We can't avoid data copy if [beg, end] span several chunks. Then we have to join them.
data NbuffChunk data(): return immutable continuous view to this whole nbuff. Data copy: We can avoid data copy if this nbuff store single immutable chunk. Then we just return ref to this chunk We can't avoid data copy if nbuff span several chunks. Then we have to join them.
dump string dump(): Undocumented in source. Be warned that the author may not have intended to support it.
empty bool empty(): Undocumented in source. Be warned that the author may not have intended to support it.
findSplitOn Nbuff[3] findSplitOn(string s, size_t start_from): Undocumented in source. Be warned that the author may not have intended to support it.
findSplitOn Nbuff[3] findSplitOn(immutable(ubyte)[] b, size_t start_from): Undocumented in source. Be warned that the author may not have intended to support it.
frontChunk auto frontChunk(): return first stored chunk
length auto length(): Undocumented in source. Be warned that the author may not have intended to support it.
opAssign void opAssign(T other): copy references to RC-data
opDollar auto opDollar(): Undocumented in source. Be warned that the author may not have intended to support it.
opEquals bool opEquals(string other): Undocumented in source. Be warned that the author may not have intended to support it.
opEquals bool opEquals(ubyte[] other): Undocumented in source. Be warned that the author may not have intended to support it.
opEquals bool opEquals(R other): Undocumented in source. Be warned that the author may not have intended to support it.
opIndex auto opIndex(size_t i): Undocumented in source. Be warned that the author may not have intended to support it.
opSlice Nbuff opSlice(size_t start, size_t end): return non-contiguous vie to slice of this nbuff. Data copy: none
pop void pop(long n): pop n bytes from nbuff
popChunk void popChunk(): throw away first chunk
toIoVec int toIoVec(iovec* v, int n): build iovec from this nbuff Paramenter v - ref to array of iovec structs n - size of array (must be >0) Return number of actually filled entries;
toString string toString(): Undocumented in source. Be warned that the author may not have intended to support it.

Static functions

get auto get(size_t size): Undocumented in source. Be warned that the author may not have intended to support it.