lehnert.cloud/ad-ds-simple-file-server
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

even more performance benefits

Browse Source
This commit is contained in:
Ludwig Lehnert
2026-08-12 13:26:31 +00:00
parent 0ea17c6401
commit 29340d778d
5 changed files with 353 additions and 92 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+4 -3
View File
@@ -54,6 +54,7 @@ The database contains:
- `shares`: AD group-to-folder lifecycle and ACL reconciliation state;
- `audit_events`: normalized read, write, move, and delete events;
- `audit_sources`: Samba log inode/offset checkpoints;
- `audit_read_dedup`: bounded, persistent fingerprints for restart-safe read deduplication;
- `audit_daily_totals`, `audit_daily_counts`, and `audit_daily_facets`: compact materialized metadata for fast activity counts and filters;
- `audit_paths`, `audit_paths_fts`, and `audit_path_events`: deduplicated trigram path search with an incrementally maintained event mapping;
- `audit_rollup_state`: bounded legacy-event backfill progress;
@@ -180,7 +181,7 @@ The E2E suite verifies:
- SMB allow/deny behavior and real file operations;
- Data, Private, and FSLogix usage aggregation;
- high-level `full_audit` ingestion for all four actions, service-account exclusion, filters, facets, and pagination;
- shared SQLite schema, integrity, indexes, legacy-log removal, and ordered read deduplication;
- shared SQLite schema, integrity, indexes, legacy-log removal, and read deduplication across interleaved events and collector polls;
- real rsync transfer progress, completed backup status, log output, remote snapshot marker, and per-group encrypted non-solid 7z archives;
- anonymous action rejection plus authenticated manual backup and reconciliation actions, terminal progress, and live reconciliation output;
- overview and system-health aggregation;
@@ -399,7 +400,7 @@ Samba emits selected high-level `full_audit` operations for Private, Data, and F
- each event records a UTC timestamp, user, client address/name, share, result, path, and one of `read`, `write`, `move`, or `delete`;
- directory listings, sessions, metadata access, file-open/create noise, and all other VFS operations are discarded; users ending in a configured `AUDIT_SKIP_USER_SUFFIXES` value are also discarded;
- FSLogix profile-container events are retained and queried through their own partial indexes, API route, and UI log instead of appearing in the main activity stream;
- immediately consecutive identical reads within the same UTC second are collapsed into one event, including across collector polling cycles; a different event interrupts the sequence and preserves later reads;
- identical reads within the same UTC second are collapsed into one event regardless of intervening events, log source, or collector polling cycle; the persistent fingerprint cache covers the latest 48 hours and can be tuned with `AUDIT_READ_DEDUP_WINDOW_SECONDS`;
- activity pages read only `limit + 1` indexed rows and use a stable time/id cursor;
- exact counts for date, user, share, action, and result filters and all facet lists come from daily rollups;
- selective substring path searches use the deduplicated trigram index and skip an expensive exact count while more pages exist;
@@ -407,7 +408,7 @@ Samba emits selected high-level `full_audit` operations for Private, Data, and F
- collector inserts and rollup updates are batched in one transaction;
- no activity retention deletion is performed.
Collection starts even when the web UI is disabled. Existing databases remain queryable while the collector backfills rollups, path IDs, and path-event mappings in bounded chunks after prioritizing each live append. On the first collector start after the legacy archive migration, recognized daily `.jsonl`/`.jsonl.gz` files and the old collector state file under `/state/audit` are deleted without import. Existing raw Samba log content is then processed using the current action, suffix, and deduplication policy.
Collection starts even when the web UI is disabled. Existing databases remain queryable while the collector backfills rollups, path IDs, path-event mappings, and recent read deduplication in bounded chunks after prioritizing each live append. On the first collector start after the legacy archive migration, recognized daily `.jsonl`/`.jsonl.gz` files and the old collector state file under `/state/audit` are deleted without import. Existing raw Samba log content is then processed using the current action, suffix, and deduplication policy.
## Backups