UpEnd

Ʌ: Projects

UpEnd

https://gitlab.com/tmladek/upend 🌐

Intro

UpEnd is a project born out of frustration with several long-standing limitations in personal computing, as well as the recently reinvigorated interest in personal information management, hypertext and augmented knowledge work.

The core issues / concepts it intends to address are:

  1. limitations of the hierarchical structure as present in nearly all of software
  2. the neglect of (unrealized potential of) of development of base OS abstractions and features

In short, UpEnd is an attempt to build a new ubiquitous storage layer for the personal computer - kind of like “the filesystem” is now, but with more advanced semantics that better reflect the inherent interconnectedness of the data, as well as its inner “meaning”, which is nowadays mostly locked within so-called application silos. Namely, it should allow for more than trivial hierarchies, building on the work done on tag-based systems and transhierarchical systems, in that all data objects (which can be files but also arbitrary structures) can be meaningfully interrelated (e.g. multiple audio tracks being renditions of the same symphony; books as well as paintings being related to the same author/genre…), arbitrarily annotated (à la ID3 tags) and traversed according to their connections - not locations; while not doing away with the benefits of hierarchies (and files) altogether.

Its conception as well as the user interface is not entirely finalized yet, but I am currently experimenting with a prototype, which I intend to publish soon. (Summer 2021?)

It aims to draw on the following approaches - the virtues of all of which have sufficiently been described elsewhere, so for now, I’ll just mention the stuff that is relevant to UpEnd.

Triplestores (a.k.a. Entity/Attribute/Value databases)

The triplestore (as in Datalog 🌐/Datomic 🌐) is able to provide a means of encoding free-form metadata, while avoiding becoming a completely undefferiated mess of JSONs as in object stores.

It’s based on the idea that all data is recorded as a triplet:

  1. Entity it describes
    • most often an abstract meaningless ID, but it can also be a (file) content hash in UpEnd
    • e.g. “123”
  2. Attribute
    • the Key of the Key/Value pair; for now, a simple string
    • e.g. “rating”, “date”, “alternate version of”
  3. Value
    • the Value of the Key/Value pair
    • e.g. “85%”, “2021-04-13”, “id:123”

An “entity” (JSON dictionary, RDBMS row, OOP object) doesn’t therefore have to specify (all) possible attributes in advance, meaning it’s easy to extend them during normal use (like adding a tag), and not just in “maintenance phase” (like ‘ALTER TABLE’).

This also allows to only supply a few nouns that interact with a lot of verbs, a characteristic that I think is crucial for making composable systems.

Semantic Web and Self-describing Information

It’s not by chance that the semantic web 🌐, or rather, RDF 🌐 directly corresponds to triplestore structure. For one, it allows for making logical and structural inferences - because the metadata lives in the same space as the factual data, achieving a kind of lispy homoiconicity 🌐, in that metadata can also be described by metadata.1

This also means that not only the system doesn’t have to specify all possible “shapes” of data in advance, it doesn’t have to rigidly specify the kinds and functions of their relations either! It can just as easily support a genealogical database (the ur-example for logic programming languages), a personal contact book, a photo gallery, and everything in between.

“Trait” systems in OOP

This property also enables a kind of a type-based schema, for which I draw inspiration mostly from Rust.2 A “type” in UpEnd is just a collection of relations the entity must possess in order to be of that type - e.g. a trait “Photo” means the entity has to supply “Size”, “Date”, “Focal length”, etc. - but the image can also be marked with the trait “Reference for a 3D modeling project”, supplying “what object it is”, “point of view”, etc. It can also be tagged with “apple”, where a tag is simply an entity that supplies the “Tag” trait and has a “target”, “name”, but can also have “description”, “color”, “priority”…

Transclusion and Transhierarchies

All of this together can supply the base for a system of organizing data along the lines of Xanadu and Roam. Since all links are two-way, and all items can be connected, one can build a densely interconnected web of hetereogeneous data. The links and relations of different hypermedia objects that we really have no representation for can be codified and interacted with.

There’s already exciting work done on creating referential “vaults” in the form of roam books - mechanized text that can be traversed according to the reader, without having to supply a predetermined path through.

The idea is that UpEnd can serve as a personal database, just as a media vault in which connections and annotations can be put together and serve as a reference for its study.

End goals, perspective

Outcomes

File Browser+

The UpEnd project should provide an application that would in its base state serve as a better alternative to the built-in system file browser.

The principal requirement is that the files that the user already has, and the folder structures they are organized by, should not be any harder to reach than the user is used to; however, the file browser should allow for more than just that, and also visualize the relationships of the data, and provide capabilities for simple ad-hoc organization that goes beyond that of hierarchies and folder structures.

This file browser should also allow for plugins, so that one can add a graph view (as in Obsidian), or a tabular view, or an outliner view, or a slideshow view, or…

API

Additionally, UpEnd should be a suitable component for other systems as well. It doesn’t aim to cover all possible use-cases of data, that much is impossible. But if one wishes to develop, say, a new music player, a reference management system, or a note-taking tool, UpEnd should provide a good-enough base to start building upon. The “File Browser” can still be a fallback as an inspection tool, or even (given the plugin architecture) a way to provide an alternative viewpoint.

How realistic is this?

I realize this is no small project, but I do not think it’s unfeasible. I think that the composability of the system, if captured correctly, means that only a robust “core” is really required, so that the rest of the functionality can just become a “possibility” within the system.

Spreadsheets are at their core rather simple, but allow for anything from organizing a book club to running your taxes or anything else. But if someone, in a world where spreadsheets didn’t exist, started to describe a “universal database that is just built on tables”, it would sound similarly outlandish.

Ultimately, this is akin to Engelbart’s Bootstrapping 🌐 concept, where with each step, the next step becomes a bit easier.

However, even if the project isn’t successful in its full scope, I will be perfectly happy if it serves just as an example of an approach that I feel is sorely underrepresented, and as a way of challenging certain preconceptions, so that the next project can perhaps be successful. We need more visions!

Use-Cases

  • Personal Information Management
  • Multimedial work - organization, “reference management”
  • Making novel connections throughout media libraries
  • Publishing of completed vaults - exegeses
  • apps living in information-space 🌐

Key phrases

  • alternative approach to filesystems
  • file storage as database; database as file storage
  • fs ⇋ db ⟹ almighty library
  • ahierarchical // polyhierarcal approach…

Resources // Inspiration

Artistic / extradisciplinary

Aby Warburg

https://garage.sdbs.cz/people.AbyWarburg.md 🌐

Burroughs’ cut-up technique

https://en.wikipedia.org/wiki/Cut-up_technique 🌐

Cut word lines Cut music lines Smash the control images Smash the control machine Burn the books Kill the priests Kill! Kill! Kill!

  • William S. Burroughs - The Soft Machine (1961)

Deleuze and Guattari

Particularly the idea of a rhizome and of the multiplicity.

I realize it ultimately may be futile to attempt to develop software based on post-structuralist ideas, so I realize I’m very much at risk of only paying lip-service to it, but their work is still very good at provoking perspectives that do away with hierarchies completely, and of handling systems without predetermined structures - such as those of media libraries and of human thought!d

Criticisms and filesystems

Programmer’s critique of missing structure of operating systems 🌐

There is only one OS, and it’s been obsolete for decades 🌐

Files, Formats and Byte Arrays 🌐

Already existing systems

Design docs

https://www.nayuki.io/page/designing-better-file-organization-around-tags-not-hierarchies 🌐

Similar projects

Perkeep 🌐

greglook/vault 🌐

greglook/blocks 🌐

greglook/merkledag-core 🌐

Components

BeOS/Haiku OS’s filesystem/database 🌐

Learn Datalog Today! 🌐


deprecated and old and czech notes

🚧🚧🚧 TRANSLATE 🚧🚧🚧

zakladni myslenka

v podstate moje idea je :

Base structure

  • databaze kde mas dva druhy objektu: data, a meta
    • jedinej rozdil je, ze data actually odkazujou hashem na nejakej file, a meta neodkazujou nikam, jenom slouzej jako “anchor”
  • a ke kazdymu objektu muzes mit arbitrarni keys a values
    • struktura filesystemu samotna je taky key a value: FILE_shaiusdhuijhngsoyuhsdf BELONGS_TO_DIRECTORY shaoidsuhjaoijoiasjdioj
    • (kazdej key/value je taky objekt na kterej se da odkazovat, kdyby sis chtel anotovat anotace)

Schemas

  • objekty maj kategorie, ktery znamenaj ze se muzes spolehnout ze objekt co je otagovanej “radio” ma: cas pridani, slozku, heaviness, etc.; co je otagovanej “public” ma: “projekt”, “filetype”, “rok”, etc.
    • to je taky E/A/V triplet: Entita je ten objekt, Attribut je neco jako IS_A a Value je odkaz na kategorii - ktera je taky objekt s E/A/V tripletama, znacici co musi objekt splnovat

Input

  • pridavas soubory bud pres web rovnou do databaze, nebo syncthingoidne do filetree, a pak zarazujes

Output

  • output je bud ten, ze je ven exposly jakysi APIcko, ktery ti dava jenom pristup do jedny z moznejch hierarchii, a s tim pracuje treba radio, nebo photo uploader, nebo OCRko
    • nebo ten, ze to proste pouzivas misto file browseru, ale kdykoli mas moznost se vratit zpatky na baseline filesystem kdyz najdes, co hledas

Geekmiscellanea

extremne zmateny notes z prubehu konceptualizace

(clarification - neni to teda jenom key/value, zjistil jsem ze se tomu rika entity/attribute/value, kdy ten key/value je jenom ta posledni dvojice, a “entity” je nejakej abstraktni anchor ke kterymu se pojej ruzny key/values; moje pointa je, ze entity nemusi bejt enom IDcko na ktery vazes, ale i primo data, ktery muzou existovat tim padem na vic mistech / jako vic filu zaroven (nebo taky zadny), a i ty samotny vztahy mezi entitama, cimz ti vznika moznost i nejak znackovat co vlastne o tech datech rikas)

Goals

  • democratic user interface model a la Syncthing
  • Content-Addressability
  • K/V metadata store
  • schemas / constraints on metadata
    • (auto-admin?)
  • Links between objects
    • 1-way = Tags
    • 2-way = Relations
    • 3-way = ???
  • 1st order tags
  • 1st order links (?)
  • kompozice souboru
    • skrz schemata!
  • Modes
    • R/O only
    • Hybrid
    • Full
  • FS revisions / metadata only backups
  • staging area, queues…

Information enters, is processed

(via https://twitter.com/geoffreylitt/status/1387777988205465600?s=20 🌐)

Random (and outdated) points

  • content-addressability
  • hidden structure within filetypes
  • breaking apart from a hirearchy
  • key/value store protocol // backend
  • VFS compat layer
  • config keys as files
  • FS as a rich API
  • syncthing model, sidecar files
  • S3 API ?
  1. JSON, through JSON schema technically allows the same thing - but the atom of JSON is a document, not a claim, so it doesn’t compose as neatly, as its primitives are very dumb. 

  2. For no other reason than because I first encountered them through Rust - I’m sure Rust didn’t invent them.