Disk Storage System

We use the disk instead of a database to reduce complexity. We leave the hard optimization to the filesystem layer.

API

`collection.create(T)` - creates an object of type T, saving it to the disk
`collection.get(id) : T` - gets an object of type T based on the id field configured
`collection.update(T)` - updates an object of type T, saving it to the disk
`collection.delete(T)` - removes an object from the system and the disk
`collection.all([options])` - iterate over all objects
`collection.find(criteria[, options])` - find all objects that match the criteria *that have an index*
`collection.on(event, callback)` - set a callback for the given event (create,update,get,delete,write,index,all,find)
`collection.off(event, callback)` - remove a callback for the given event

Example

import * as fsdb from '@andyburke/fsdb';
import { FSDB_INDEXER_SYMLINKS } from '@andyburke/fsdb/indexers';
import { by_character, by_email, by_lurid, by_phone } from '@andyburke/fsdb/organizers';

type USER = {
	id: string;
	email: string;
	phone: string;
	quote: string;
};

const item_collection: fsdb.FSDB_COLLECTION<USER> = new fsdb.FSDB_COLLECTION<USER>({
	name: 'users',
	indexers: {
		email: new FSDB_INDEXER_SYMLINKS<USER>({
			name: 'email',
			field: 'email',
			organize: by_email
		}),
		phone: new FSDB_INDEXER_SYMLINKS<USER>({
			name: 'phone',
			field: 'phone',
			organize: by_phone
		}),
		by_character_test: new FSDB_INDEXER_SYMLINKS<USER>({
			name: 'quote_keywords',
			organize: by_character,
			get_values_to_index: (user: USER) => user.quote.split(/\W/).filter((word) => word.length > 3).map((word) => word.toLowerCase().trim()),
			to_many: true
		})
	}
});

Would create a folder structure of users that looked something like:

 .fsdb/
   users/
     .fsdb.collection.json <-- collection info
     able-fish-door/
       able-fish-door-your-deal-fire-unit/
         able-fish-door-your-deal-fire-unit-trip-give-they/
           able-fish-door-your-deal-fire-unit-trip-give-they.json <-- the user
           .index.symlink.quote_keywords.chicken <-- points back to an index symlink for quote_keywords
           .index.symlink.quote_keywords.contrary
           .index.symlink.quote_keywords.first
           .index.symlink.quote_keywords.pineapple
           .index.symlink.quote_keywords.sensible
           .index.symlink.quote_keywords.that
           .index.symlink.quote_keywords.unfortunately
           .index.symlink.quote_keywords.wrong
           .index.symlink.email.took-case-path@grow-hold-such.edu <-- email index reverse symlink
           .index.symlink.phone.01.072-902.2106 <-- phone number index reverse symlink
     .indexes/
       email/
         edu/
           grow-hold-such.edu/
             took-case-path@grow-hold-such.edu/
               took-case-path@grow-hold-such.edu.json <-- symlink to the user above
       phone/
         01/
           072/
             902/
               072-902-2106/
                 072-902-2106.json <-- symlink to the user above
       quote_keywords/
         c/
           ch/
             chi/
               chicken/
                 able-fish-door-your-deal-fire-unit-trip-give-they.json <-- symlink to the user above, which has used this keyword in their quote
           co/
             con/
               contrary/
                 able-fish-door-your-deal-fire-unit-trip-give-they.json <-- symlink to the user above, which has used this keyword in their quote
           ...

CLI

[you@machine:~/] fsdb users create '{
    "id": "able-fish-door-with-star-snow-idea-edge-salt-many",
    "email": "commandlinetestuser@domain.com",
    "phone": "213-555-1234",
    "value": "By the way a horse is a lemon from the right perspective."
}'
created: {
    "id": "able-fish-door-with-star-snow-idea-edge-salt-many",
    "email": "commandlinetestuser@domain.com",
    "phone": "213-555-1234",
    "value": "By the way a horse is a lemon from the right perspective."
}

[you@machine:~/] fsdb users update '{
    "id": "able-fish-door-with-star-snow-idea-edge-salt-many",
    "email": "commandlinetestuser@domain.com",
    "phone": "213-555-1234",
    "value": "By the way a horse is a lemon from the right angle."
}'
updated: {
    "id": "able-fish-door-with-star-snow-idea-edge-salt-many",
    "email": "commandlinetestuser@domain.com",
    "phone": "213-555-1234",
    "value": "By the way a horse is a lemon from the right angle."
}

[you@machine:~/] fsdb users get able-fish-door-with-star-snow-idea-edge-salt-many
{
    "id": "able-fish-door-with-star-snow-idea-edge-salt-many",
    "email": "commandlinetestuser@domain.com",
    "phone": "213-555-1234",
    "value": "By the way a horse is a lemon from the right angle."
}

[you@machine:~/] fsdb users delete '{
    "id": "able-fish-door-with-star-snow-idea-edge-salt-many",
    "email": "commandlinetestuser@domain.com",
    "phone": "213-555-1234",
    "value": "By the way a horse is a lemon from the right angle."
}'
deleted: {
    "id": "able-fish-door-with-star-snow-idea-edge-salt-many",
    "email": "commandlinetestuser@domain.com",
    "phone": "213-555-1234",
    "value": "By the way a horse is a lemon from the right angle."
}

Indexers

Symlinks

We create symlinks on the disk to help find objects more quickly and to allow for browsing the data as a human.

SQLite

TODO: index everything into a sqlite setup as well? would give a way to run SQL against data still stored on disk in a nicely human browsable format.

Environment Variables

variable	description
FSDB_ROOT	controls the root directory, default: ./.fsdb
FSDB_PERF	set to true for performance tracking
FSDB_LOG_EVENTS	set to true to log the events system

TODO