Skip to main content
benchmarkedInspired by Nacre (mother-of-pearl)

Nacre Array

Counterpart to Dynamic Array / Vec (alternative)

Research in progress

The data and performance claims on this page reflect ongoing research, implementation, and experimentation. They may change as benchmarks are rerun and formal papers are finalized.

Key Properties

  • Segmented storage with per-segment summaries and thermal state machine (Hot/Warm/Cold/Compressed)
  • Segment-bounded insert/remove with trailing cumulative-index maintenance
  • Selective compression of cold segments via LZ4/Zstd codecs
  • Fracture planes at segment boundaries for payload-copy-free structural splitting
  • Self-describing elements with metadata headers (ElementHeader)

Operation Complexity

Side-by-side comparison with the classical counterpart.

Operation complexity comparison
OperationNacre ArrayDynamic Array / Vec (alternative)
pushO(1) amortizedO(1) amortized
getO(log S) / O(1) cachedO(1)
insertO(s + S) current implO(n)
removeO(s + S) current implO(n)
splitO(S) current implO(n)
iterO(n)O(n)
tick (maintenance)O(segments)N/A

Interface Preview

nacre_array.rs
rust
1pub struct NacreArray<T: OrganicElement> { /* ... */ }
2
3impl<T: OrganicElement> NacreArray<T> {
4    pub fn new(segment_capacity: usize) -> Self;
5    pub fn with_capacity(total: usize, seg_cap: usize) -> Self;
6    pub fn with_cooldown(self, config: CooldownConfig) -> Self;
7    pub fn with_segment_cache(self, enabled: bool) -> Self;
8
9    // Core
10    pub fn len(&self) -> usize;
11    pub fn is_empty(&self) -> bool;
12    pub fn segment_count(&self) -> usize;
13    pub fn segment_capacity(&self) -> usize;
14
15    // Access & Mutation
16    pub fn push(&mut self, value: T);
17    pub fn get(&self, index: usize) -> Option<&T>;
18    pub fn read(&mut self, index: usize) -> Option<T>;
19    pub fn get_mut(&mut self, index: usize) -> Option<&mut T>;
20    pub fn set(&mut self, index: usize, value: T) -> Option<T>;
21    pub fn insert(&mut self, index: usize, value: T);
22    pub fn remove(&mut self, index: usize) -> Option<T>;
23
24    // Iteration
25    pub fn iter(&self) -> NacreIter<T>;
26    pub fn iter_mut(&mut self) -> NacreIterMut<T>;
27
28    // Fracture
29    pub fn split_at(&mut self, index: usize) -> NacreArray<T>;
30    pub fn fracture_planes(&self) -> Vec<usize>;
31    pub fn nearest_fracture(&self, index: usize) -> usize;
32
33    // Scanning
34    pub fn scan_headers(&self, pred: impl Fn(&ElementHeader) -> bool) -> Vec<&T>;
35    pub fn scan_segments(&self, pred: impl Fn(&SegmentSummary) -> bool) -> Vec<usize>;
36
37    // Segment access
38    pub fn segment(&self, index: usize) -> Option<&Segment<T>>;
39    pub fn segment_mut(&mut self, index: usize) -> Option<&mut Segment<T>>;
40    pub fn par_segments(&self) -> &[Segment<T>];
41
42    // Maintenance
43    pub fn tick(&mut self) -> MaintenanceReport;
44    pub fn compression_stats(&self) -> CompressionStats;
45}
46
nacre_array.ts
typescript
1class NacreArray {
2  constructor(options?: NacreArrayOptions);
3  async init(): Promise<this>;
4  static withCapacity(total: number, segCap?: number): Promise<NacreArray>;
5
6  // Core accessors
7  readonly length: number;
8  readonly isEmpty: boolean;
9  readonly segmentCapacity: number;
10  readonly segmentCount: number;
11  readonly currentTick: number;
12
13  // Access & Mutation (values are f64 via WASM)
14  get(index: number): number | undefined;
15  set(index: number, value: number): number | undefined;
16  push(value: number): void;
17  insert(index: number, value: number): void;
18  remove(index: number): number | undefined;
19
20  // Iteration
21  toFloat64Array(): Float64Array;
22  [Symbol.iterator](): Iterator<number>;
23
24  // Fracture
25  splitAt(index: number): NacreArray;
26  fracturePlanes(): number[];
27  nearestFracture(index: number): number;
28
29  // Maintenance
30  tick(): MaintenanceReport;
31  compressionStats(): CompressionStats;
32}
33

Where This Matters

Apache Kafka

Partition Log

Improvement comparison
TodayWith Nacre ArrayHow
Separate .index file per segmentEliminatedElement headers carry offset metadata in mortar layer
Can't split partitions without rebalancingPayload-copy-free split at fracture planesSacrificial segmentation at tile boundaries
All-or-nothing log compactionPer-segment hot/warm/coldSelective compression based on tile age
Uniform storage cost for old dataLower memory cost for cold regionsCold tiles can compress in place, with explicit reheating on Rust owned reads and transparent reheating through the WASM/TypeScript `get()` path; a reproducible mixed-workload memory benchmark is still pending

Interactive Simulation

Discussion

Questions, ideas, or experiences with Nacre Array? Join the discussion.