source

flyology_object_storage_sqlite

0.1.0-devDevelopment only

SQLite catalog backend for Flyology Object Storage

GitHub All crate versions Download JSON

0.1.0-dev

Development
Authors
Yurii Rashkovskii
Maintainers
Yurii Rashkovskii <yrashk@gmail.com>
Maintainer profiles
@yrashk
Licenses
MIT OR Apache-2.0
Website
https://github.com/flyology-ada/flyology-object-storage
Repository
github.com/flyology-ada/flyology-object-storage
Source revision
5eaf79cf1235
Source directory
sqlite
Tags
ada, aws, object-storage, s3, sqlite
Project files
flyology_object_storage_sqlite.gpr
Manifest
index/fl/flyology_object_storage_sqlite/flyology_object_storage_sqlite-0.1.0-dev.toml

Source documentation

README

View source

Flyology Object Storage

Flyology Object Storage is an in-development S3 client and pluggable S3 server library for Ada. It builds on flyology_http and preserves ordinary synchronous Ada calls for native and Flyology lightweight tasks.

This repository currently establishes the storage boundary and its first conforming backends. It does not yet claim complete S3 compatibility or production qualification. Supported wire behavior will be listed only in the executable compatibility matrix.

Deterministic protocol and state logic is developed under the SPARK assurance boundary; I/O and C interfaces stay in narrow contracted adapters.

Architecture

high-level transfers -> S3 operation client -> signing/wire -> Flyology HTTP

Flyology HTTP server -> S3 wire/auth/policy -> operation service -> backend
                                                              |-> memory
                                                              |-> files
                                                              `-> SQLite add-on

The backend interface deals in buckets, object metadata and streaming byte sources/sinks. It does not see HTTP exchanges or AWS wire DTOs.

The core crate includes:

  • a bounded concurrent memory backend with committed, staged, and in-progress payload reservations (see memory backend capacity);
  • a pure-files backend with opaque key paths and atomic object publication for one process owning an exclusive storage root;
  • shared storage and transfer vocabulary;
  • a generated, pinned, SPARK-proved descriptor for every operation, shape, member location, enumeration, XML trait, checksum trait, and authentication trait in the 116-operation botocore S3 model;
  • AWS-vector-tested SigV4 header signing and bounded, entity-safe REST/XML;
  • a backend-neutral streaming checksum foundation for all ten pinned S3 algorithms, with strict canonical Base64, AWS composite policy, full-object CRC linearization, and a reproducible 320-vector differential corpus (see S3 checksum foundation);
  • complete typed, bounded HeadBucket, GetBucketLocation, ListBuckets, ListObjects v1/v2, ListObjectVersions, and multipart initiation, completion, ListParts, and ListMultipartUploads REST/XML codecs, plus the complete GetObjectAttributes request/output model, with high-level paginated ListBuckets, ListObjects v1, ListObjectsV2, and paired-cursor ListObjectVersions, HeadBucket, CreateBucket, GetBucketLocation, DeleteBucket, and bucket-tag convenience APIs (see bucket convenience client);
  • high-level GetObjectAttributes and non-replaying DeleteObject clients that preserve selected metadata, version, delete-marker, requester-charged, and structured error outcomes (see object convenience client and DeleteObject qualification);
  • signed, bounded CreateBucket, GetBucketLocation, HeadBucket, PutBucketTagging, GetBucketTagging, DeleteBucketTagging, GetObject, HeadObject, PutObject, ListBuckets, ListObjects, ListObjectsV2, ListObjectVersions, DeleteBucket, PutBucketCors, GetBucketCors, DeleteBucketCors, GetBucketLifecycleConfiguration, PutBucketLifecycleConfiguration, GetBucketNotificationConfiguration, PutBucketNotificationConfiguration, GetBucketReplication, GetBucketMetricsConfiguration, GetBucketAnalyticsConfiguration, GetBucketIntelligentTieringConfiguration, GetBucketInventoryConfiguration, GetBucketLogging, GetBucketWebsite, GetBucketMetadataConfiguration, ListBucketMetricsConfigurations, ListBucketAnalyticsConfigurations, ListBucketIntelligentTieringConfigurations, ListBucketInventoryConfigurations, and PutBucketReplication (see read qualification and write qualification), all thirteen remaining bodyless bucket-configuration deletes, GetBucketAbac, GetBucketAccelerateConfiguration, GetBucketPolicy, GetBucketPolicyStatus, GetBucketRequestPayment, GetPublicAccessBlock, PutBucketAbac, PutBucketAccelerateConfiguration, PutBucketPolicy, PutBucketRequestPayment, PutPublicAccessBlock, DeleteObject, DeleteObjects REST/XML, and multipart initiate/upload/complete/abort/ListParts/ListMultipartUploads low-level operations over caller-owned flyology_http clients, plus a synchronous one-shot UploadPart API whose bodies are borrowed from forward-only streaming sources and whose post-admission exceptions require ListParts reconciliation;
  • completion-set-aware conditional Put, bounded whole Get, generation-bound single-range Get, bodyless Head, non-replaying Delete and CopyObject, multipart initiation, one-shot UploadPart, and one-shot multipart completion operations, plus service-level ListBuckets, CreateBucket, DeleteBucket, HeadBucket, bounded GetBucketLocation, bounded GetBucketPolicy, GetBucketPolicyStatus, GetBucketRequestPayment, GetBucketAbac, and GetBucketAccelerateConfiguration, bounded GetBucketAcl and GetObjectAcl, bounded GetBucketMetadataTableConfiguration, and bounded GetBucketLifecycleConfiguration, non-replaying PutBucketLifecycleConfiguration, and bounded GetBucketNotificationConfiguration, non-replaying PutBucketNotificationConfiguration, and bounded GetBucketReplication, GetBucketMetricsConfiguration, GetBucketAnalyticsConfiguration, GetBucketIntelligentTieringConfiguration, GetBucketInventoryConfiguration, GetBucketLogging, GetBucketWebsite, GetBucketMetadataConfiguration, caller-driven paginated ListBucketMetricsConfigurations and ListBucketAnalyticsConfigurations and ListBucketIntelligentTieringConfigurations and ListBucketInventoryConfigurations, non-replaying PutBucketReplication, and non-replaying PutBucketAbac, PutBucketAccelerateConfiguration, PutBucketRequestPayment, Put/DeleteBucketPolicy, DeleteBucketLifecycle, DeleteBucketReplication, DeleteBucketAnalyticsConfiguration, DeleteBucketMetricsConfiguration, DeleteBucketIntelligentTieringConfiguration, DeleteBucketMetadataConfiguration, DeleteBucketMetadataTableConfiguration, DeleteBucketWebsite, DeleteBucketInventoryConfiguration, and Put/GetBucketVersioning, with typed synchronous overloads that wait on the same owner-driven state machines and never create a per-operation helper task;
  • a bounded ordered DeleteObjects backend batch, with process-atomic memory, transactional SQLite, and explicitly scoped per-file durability semantics (see DeleteObjects qualification);
  • atomic conditional complete-object publication for memory, files, and SQLite, with If-None-Match: * create-if-absent and opaque If-Match compare-and-swap semantics (see conditional PutObject qualification);
  • complete backend and client PutObject semantics for bounded metadata, tags, ten direct checksums, owner policy, and one-shot no-replay publication, with the server's explicit unsupported controls retained as a partial profile (see PutObject qualification);
  • exhaustive request projection and a raw streaming-response execution boundary for all 116 pinned operations, while operation-specific typed codecs and interoperability gates remain tracked as incomplete;
  • an authenticated path-style S3 server application for the initial Create/GetLocation/Head/DeleteBucket, Put/Get/DeleteBucketTagging, GetBucketAcl, ListObjects v1/v2, DeleteObjects, Put/Copy/Get/Head/GetAttributes/DeleteObject, GetObjectAcl, Put/Get/DeleteObjectTagging, and core multipart/ListParts/ ListMultipartUploads slice, with current/null/exact generation routing, SigV4 admission before body acceptance, streamed payload-hash verification before commit, atomic suffix-range resolution, typed S3 error responses, and owner-only ACL reads derived from the existing private single-tenant profile without persisted ACL state;
  • the namespace for the low-level and convenience clients.

The optional flyology_object_storage_sqlite subcrate vendors a pinned SQLite amalgamation and supplies a conforming backend. SQLite owns the transactional namespace and metadata; immutable large payloads remain external files because SQLite BLOB limits are smaller than S3 objects. Its publication order flushes the payload and containing directory before the synchronous=FULL catalog commit, and startup reconciles interrupted writes. One process owns a configured SQLite root, enforced with a system-wide lock.

The separate flyology_object_storage_server executable crate selects memory, files, or SQLite at startup and runs the S3 application under Flyology supervision. Its first slice includes bounded connection handling, conservative loopback binding, graceful SIGTERM drain, and the same independent s5cmd qualification on all three backends. It now creates an owner-only, slow-hashed administrator credential and displays the random bootstrap password exactly once. A second supervised, loopback-only listener serves the authenticated management API and a psqlbench-inspired browser workbench. It reports the actual bound S3 endpoint, selected backend and dependency-ordered service tree; its external HTML, CSS and JavaScript are SHA-256-pinned by the binary so a modified asset set stops startup. Authenticated administrators can create buckets, delete empty buckets through an inline confirmation, and browse byte-safe, opaque-token-paginated object metadata. Management pages preserve 64-bit sizes and timestamps exactly; object mutation remains on the signed S3 endpoint.

The files backend defaults to an fsync-backed Power_Loss_Durable commit policy on qualified POSIX hosts. It synchronizes staged records before rename, both namespace sides of publication, deletions, and multipart lifecycle changes. A separately labeled Process_Crash_Atomic policy omits persistence barriers for controlled comparisons. Barrier-by-barrier device-error injection and 88 actual abrupt-process terminations immediately before and after each barrier require every reopened namespace to be an intact old-or-new state. Windows directory-metadata durability and cross-process root sharing remain unsupported.

The server application remains partial: GET, HEAD and range responses use exact stored lengths without chunked fallback. Flyology HTTP also provides 64-bit request accounting, a 50 TB streaming ceiling, and per-route Body_Size limits.

The SQLite backend’s exact commit, recovery, and ownership model is described in docs/architecture/sqlite-backend.md.

Alire

Configure the Flyology index, then build:

alr index --reset-community
alr index --add=git+https://github.com/flyology-ada/alire-index.git \
  --name=flyology --before=community
alr build
./tests/scripts/test.sh

The composable client resolves through the Flyology index as flyology_http=0.1.3-dev, which depends exactly on flyology_quic=0.1.3-dev. Both indexed crates resolve to reviewed source commit eb09a80a7e06274e93289861c2cae1ca7e8cb1af; Object Storage carries no external git pin for either dependency. Committed machine-local path pins remain prohibited. The indexed client provides protected bounded round-robin HTTP/2 pump handoff and keeps synchronous settlement on the same owner-driven composable pump, with no helper task, extra completion slot, or second protocol engine. It also provides one guarded stale pooled-HTTP/1 recovery for safe bounded-buffer GET/HEAD exchanges only when no source or response bytes exist; mutation and source-bearing calls are never replayed. Ordinary clients retain zero settlement grace.

The independent S3 interoperability matrix uses digest-pinned RustFS and SeaweedFS servers as its permissively licensed primary targets, MinIO as an additional compatibility target, and MIT-licensed s5cmd as a separate byte oracle. The same s5cmd server slice also runs against the memory, files, and SQLite Flyology servers, so backend substitution is checked at the public S3 boundary. Every server is ephemeral and exposed only on a random loopback port:

./tests/scripts/test-s3-matrix.sh

The matrix repeats every server lane three times by default. This is a required qualification gate for the operations it exercises, but it is kept separate from the offline default test action because it needs Docker and registry access. Exact source commits, image digests and reviewed licenses are checked by tools/verify-corpora-lock.sh.

An additional SDK-level lane uses the pinned MIT-licensed s3t Go port of Ceph s3-tests. Build revision 51506ac904f6e35424b3ec9d38716985023beba6, then provide its binary to the same matrix:

FLYOLOGY_S3T_BIN=/path/to/s3t ./tests/scripts/test-s3-matrix.sh

The runner rejects a different or dirty revision and executes the checked 113-node allowlist with four workers, retaining JSON reports under obj/s3t. The memory, files, and SQLite Flyology backends pass 113/113. Pinned RustFS passes 112 with one exact expected failure (InvalidArgument instead of NoSuchUpload), while pinned SeaweedFS passes 107 with six exact expected failures: four bucket/listing divergences and two UploadPartCopy source-range validation divergences. Pinned supplemental MinIO passes 106 with seven exact expected failures: its six multipart/listing divergences plus InvalidArgument in place of InvalidRange for an out-of-bounds UploadPartCopy source range. Unexpected failures and unexpected passes both fail every lane; MinIO is not used as a permissively licensed performance oracle.

The server performance comparison is specified in docs/qualification/performance.md. It uses the same client and host for RustFS, SeaweedFS, and each Flyology backend; durability modes and raw latency/throughput samples remain separate. The checked implementation matrix requires both permissive references and the memory, files, and SQLite backend series; each Flyology series is reported as an absolute result and as a same-run ratio to both references.

With the exact reviewed HTTP dependency resolved, run the correctness-checked aggregate smoke comparison with:

./benchmarks/run-matrix.sh

The launcher refuses any other HTTP dependency so GET framing is comparable. Dedicated-host campaigns use FLYOLOGY_BENCH_PROFILE=full and must record the host, power, and CPU policies described in the performance guide.

The checked CI policy, local equivalents, toolchain pins, oracle split, and artifact retention are documented in continuous integration qualification.

Client shape

The client will expose the complete model-driven S3 operation surface and a smaller handwritten convenience layer. The latter owns multipart and multi-object policy while Flyology performs the actual task-aware waiting:

package Transfers renames Flyology.Object_Storage.Client.Transfers;

Uploaded := Transfers.Upload_File
  (Client, Origin, "bucket", "key", "archive.tar", Identity);
Checksummed := Transfers.Upload_File
  (Client, Origin, "bucket", "verified", "archive.tar", Identity,
   Checksum =>
     (Enabled => True,
      Algorithm => Transfers.Checksum_Policy.Core.SHA256,
      Kind => Transfers.Checksum_Policy.Composite));
Downloaded := Transfers.Download_File
  (Client, Origin, "bucket", "key", "archive.tar", Identity);
Copied := Transfers.Copy_Object
  (Client, Origin, "source-bucket", "source key", "bucket", "copy key",
   Identity);
Head := Transfers.Head_Object
  (Client, Origin, "bucket", "key", Identity);
Transfers.Transfer_Many
  (Client, Origin, Items, Results, Identity, Options => Options);

Upload_File switches to multipart at 64 MiB by default, streams 16 MiB parts from one open descriptor, and exposes both thresholds as trailing policy parameters. Its optional checksum policy supports direct full-object values and multipart full-object/composite values; a composite selection forces a nonempty small file through multipart and a successful outcome retains the verified checksum/type. Copy_Object owns copy-source URI encoding and signing, including special-character keys, and returns compact copy metadata or the structured S3 error. Head_Object is the bodyless reconciliation primitive: its common fields expose 64-bit size, entity tag, modification/content/version metadata, while Details preserves every modeled response member. Trailing named arguments cover all modeled conditions, range, response overrides, SSE-C, request-payer, part, owner, and checksum controls. A structured rejection preserves request IDs even when the HEAD response has no XML body. Multipart parts for one file are deliberately sequential; batch parallelism is across independent subjects. Transfer_Many remains a synchronous structured scope and bounds concurrent objects, HTTP requests and in-flight bytes independently. Flyology's runtime owns multiplexing, backpressure and waiting, so the convenience layer does not create one task per chunk or retain a massive object.

The completion-set-aware overloads are colocated with their synchronous providers: object operations in Client.Objects, bucket operations in Client.Buckets, and multipart and copy operations in Client.Transfers. They cover conditional Put, whole and exact-range Get, Head, Delete, CreateMultipartUpload, UploadPart, CompleteMultipartUpload, AbortMultipartUpload, bounded ListParts and ListMultipartUploads, and CopyObject, together with service-level ListBuckets and non-replaying CreateBucket, bounded CreateSession, DeleteBucket, DeleteBucketLifecycle, bounded GetBucketLifecycleConfiguration and non-replaying PutBucketLifecycleConfiguration, bounded GetBucketNotificationConfiguration and non-replaying PutBucketNotificationConfiguration, bounded GetBucketReplication, GetBucketMetricsConfiguration, GetBucketAnalyticsConfiguration, GetBucketIntelligentTieringConfiguration, GetBucketInventoryConfiguration, GetBucketLogging, GetBucketWebsite, GetBucketMetadataConfiguration, caller-driven paginated ListBucketMetricsConfigurations and ListBucketAnalyticsConfigurations and ListBucketIntelligentTieringConfigurations and ListBucketInventoryConfigurations, non-replaying PutBucketReplication, DeleteBucketReplication, DeleteBucketAnalyticsConfiguration, DeleteBucketMetricsConfiguration, DeleteBucketIntelligentTieringConfiguration, DeleteBucketMetadataConfiguration, DeleteBucketMetadataTableConfiguration, DeleteBucketWebsite, DeleteBucketInventoryConfiguration, and CreateBucketMetadataTableConfiguration, paired with bounded GetBucketMetadataTableConfiguration reconciliation. The typed synchronous overloads wait on those same owner-driven state machines. Conditional DeleteObjectAnnotation is likewise exposed as provider-owned Client.Objects.Delete_Annotation constructor, operation-last restart, typed Finish, and synchronous wait forms. It never replays the mutation, and an unknown post-admission result requires generation-bound read-only reconciliation before any caller-selected retry. Multipart initiation and abort use one-shot empty sources, UploadPart moves one owned bounded buffer, and completion owns the exact serialized XML behind a one-shot source. Each preserves HTTP admission and mutation certainty: after possible admission, a lost or invalid response is unknown and must be reconciled before any retry.

A lost response to CompleteMultipartUpload is inherently ambiguous: the server may have committed the object. Best-effort abort is cleanup, not rollback. The typed result therefore reports unknown completion after possible admission, including an error embedded in HTTP 200, and applications reconcile the destination object plus the exact upload before choosing retry or abort. Abort is likewise one-shot: only a complete validated 204 proves acceptance. Every service rejection or failure after possible admission remains unknown, and applications reconcile the exact upload read-only before any later action. Composable ListParts is that bounded read-only exact-upload primitive: its successful page must echo bucket, key, upload ID, marker, and maximum exactly; separate pages remain independent service snapshots. Composable ListMultipartUploads applies the same bounded owner-driven contract to bucket-level discovery, binding the paired cursor and every echoed scope field to one prepared request while leaving pages as independent snapshots. Composable CopyObject owns its encoded raw source and complete options record, uses one non-rewindable empty request source and one XML-limit-bounded response sink, and never replays the mutation. Only a complete validated success proves publication. An admitted transport failure, invalid response, or malformed embedded HTTP-200 error is Outcome_Unknown; callers reconcile the destination with a generation-bound whole Get before deciding whether to retry. The detailed policy is in client transfers.

Operation automation

The human-reviewed S3 operation registry inventories all 116 pinned model operations, including exact coverage, implementation/test provenance, and executable evidence. Its uv tool audits complete reachable model shapes, refuses unresolved semantic decisions, drives focused qualification, and generates coverage counts, test registration, and the complete documentation list. The workflow and generated-versus-human ownership boundary are documented in S3 operation automation.

Secret erasure boundary

Secure erasure does not inherently require C. This crate uses one tiny C11 volatile-store shim because it provides a stable optimizer-resistant boundary across the supported Ada compilers. It wipes owned access keys, secret keys, session tokens and derived SigV4 keys during finalization; it cannot erase copies made by callers, runtimes, kernels, crash dumps or swap.

This is memory-secret hygiene, not a promise to physically erase deleted objects. Filesystems with copy-on-write, snapshots, journaling, SSD wear leveling, and SQLite/WAL history make physical object erasure a deployment and storage-device policy. DeleteObject removes the live logical object according to backend semantics.

License

The Ada sources are available under MIT OR Apache-2.0. Vendored SQLite is in the public domain; its exact provenance is recorded in sqlite/vendor/sqlite/README.md.

Agent setup

This repository uses APM 0.28.0 to provision one locked graph of shared and repository-specific instructions and skills for Codex and Claude. In a new clone or worktree, run:

curl -sSL https://aka.ms/apm-unix | sh -s -- @v0.28.0
apm --version

apm install --frozen
apm compile --target codex

The compiled AGENTS.md is committed so Codex receives the instructions without setup. APM generates Claude's native rules and the Codex and Claude native skill trees locally from the same apm.lock.yaml graph; those deployment outputs are intentionally ignored. Start a fresh client session after installation so it discovers the generated skills. Repository-specific instructions live under agent-packages/, and shared Ada/workflow resources come from the flyology-ada/agents main update channel at the exact revision recorded in apm.lock.yaml.

To review an intentional shared-resource upgrade, run:

apm outdated
apm update flyology-ada/agents
apm compile --target codex
apm audit --ci
git diff --check

Review the resulting lockfile, generated AGENTS.md, and behavior before committing them. Frozen installation and CI never update the selected shared revision.