Files and file-systems
This is the project I’m currently trying to push forward the most.
UpEnd is a project born out of frustration with several long-standing limitations in personal computing, as well as the recently reinvigorated interest in personal information management, hypertext and augmented knowledge work.
The core issues / concepts it intends to address are:
- limitations of the hierarchical structure as present in nearly all of software
- the neglect of (unrealized potential of) of development of base OS abstractions and features
In short, UpEnd is an attempt to build a new ubiquitous storage layer for the personal computer - kind of like “the filesystem” is now, but with more advanced semantics that better reflect the inherent interconnectedness of the data, as well as its inner “meaning”, which is nowadays mostly locked within so-called application silos. Namely, it should allow for more than trivial hierarchies, building on the work done on tag-based systems and transhierarchical systems, in that all data objects (which can be files but also arbitrary structures) can be meaningfully interrelated (e.g. multiple audio tracks being renditions of the same symphony; books as well as paintings being related to the same author/genre…), arbitrarily annotated (à la ID3 tags) and traversed according to their connections - not locations; while not doing away with the benefits of hierarchies (and files) altogether.
Its conception as well as the user interface is not entirely finalized yet, but I am currently experimenting with a prototype, which I intend to publish soon. (Summer 2021?)
It aims to draw on the following approaches - the virtues of all of which have sufficiently been described elsewhere, so for now, I’ll just mention the stuff that is relevant to UpEnd.
Life-cycle of files
- A way to denote less or more accessed files?
- MP3s not played in years, code read a long time ago…
- Where did a file come from?
- Both in a “Downlodaded from the internet”/”Created via Photoshop” meaning, as well as the “where was this file before I moved it here”?
Generic Out-of-Band Markup editor (/format?)
(opens the door to many other projects!)
If anyone wants to mark up a document it’s easy to just fire up any text editor. On the other hand, there’s no equivalent for out-of-band markup.
To be fair, there isn’t even any common format for OOB markup - maybe start there? How to simply represent text + attached information?
Overlaps with Multi-Annotator?
(non-essential, but scratches an itch I have annoyingly often)
Making video as easy to navigate as text!
Concepts to utilize:
- Pseudo-“movie barcode” (as on https://media.ccc.de 🌐)
- Have recursive seek-bars - coarse and finer
- above and below the video?
- (Automatic) bookmarks/meaningful history
- To allow for digressions (see thoughts.interfaces)
- Video timestamps ⇆ text; do not change state unexpectedly
- Markers based on scene detection (adjustable sensitivity)
Itch spotted elsewhere: https://twitter.com/rsnous/status/1357532520812748801?s=20 🌐
(exists in some forms already but none that would really fit me god dammit)
Social annotation tools are really nice, but so is reading from paper - why is the workflow to merge the two so complicated? Imagine an app:
- Take a picture of a book 1.5. Select the appropriate text excerpt by “highlighting it”
- It is OCR’ed and stored
- You can attach information about the book, page number etc.
Machinical a/v cut-ups // Decontextualizer
Priority: Solid B?
- Take a set of videos that are subtitled
- Sort subtitle segments by length (text, time)
- Filter or even modulate chosen segment length
- Randomly pick N segments
- Join them together along with their videos
There could be an intermediate format (i.e. step
4.5) - a text file that is then fed to a final video editor (ffmpeg script); and hence editable using standard tools (grep, vim, etc.).
(The simplest version that’s already useful should not be too difficult to implement.)
A program that extracts slides as bitmaps based on image (dis)similarity when given a video of the lecture (and possibly coordinates where the slides are in the video?)
Additionally: Just output the timestamps of slide changes, e.g. for annotating an audio track so that each segment can afterwards be paired with the slide…
RIP, another Google-gutted feature with great potential. Let’s do better.
(Stuff like popcorn.js 🌐 would have been sufficient, but died - for a reason? Additionally, a lot of the work there is in UX polish and doing Google’s work.)
Essentially comments with a time-stamp + spatial info (being able to annotate a certain area of the video).
Would the W3C Web Annotation data model suffice?
- It would. Implementation details / integration payoff uncertain.
From existing comments
ALREADY DONE - https://github.com/ris58h/youtube-timestamps 🌐
Go through current page of comments, check for timestamps, insert them on the timeline - kind of like SoundCloud used to?
(Big project, lots of essential complexity. Perhaps UpEnd will help this along?)
Like https://docdrop.org 🌐 except with native understanding of the formats
What’s the underlying structure of annotations?
- check out w3c standards, hypothes.is…
- target + body = annotation
- in-document targetting - url fragments insufficient to mark, e.g. a sentence
- W3C handles it by just providing a ton of options
- in-document targetting - url fragments insufficient to mark, e.g. a sentence
(Too interesting not to attempt.)
A collaborative spatial canvas that you can place any sort of info in (files, images, links, text), but in which every participant has a tangible presence.
A real “gathering place” for projects, planning, etc.
There’s projects are trying to solve the awkwardness of video calls, but they approach the spatialization either in a completely abstract way (Wonder 🌐, or a bit too “skeuomorphic” for my tastes (Kumospace 🌐.
Why not join the natural metaphor of spatialized presence with an actual (hyper-)pinboard system - kind of like Muse 🌐 does?
You could, at the same time, build a 2D multimedia mind-map collage of whatever you’re talking about, while actually being casually “present” in that document.
Open-source Identity Toolkit
(There’s people who can do it much better and faster than I can.)
A ready-to-use software package that contains:
- Webcam-controlled 3D face model that can be parametrized
- A Virtual Webcam
- A voice changer
(Think VRChat, but without the VR.)
- There’s a ton of legitimate reasons not to show one’s face/voice online (in fact, some would argue that internet best be used anonymously), but that means no facial expressions to communicate with.
- Personal experiments with what kind of identities can one assume; identity “self-forming”. k
Various Text-Messaging Experiments
(Not really all that important, but would be fun)
A chat where various rules can be imposed (including several rules at once), just to see what it does to communication:
- r9k 🌐 (i.e. forbid repetition)
- “no e”, a.k.a. lipograms 🌐, or other forms of constrained writing
- ALL UPPER CASE FOR ALL USERS / all lowercase for all users
- Time-limit, character budget…
- No nicknames
messaging app where everyone has a unique speech bubble style, like in comics (@tophtucker 🌐). every time you say something, it slightly but permanently perturbs your speech bubble's design
IRC thread detector
Usually, people in IRC reply back an forth to each other, with each message prefixed with the nick of the recipient. Would it be possible to automatically detect an on-going conversation and highlight/hide it, if one wants to (not) follow that one?
Modular Social Network / Discussion Board
A modular “mix-and-match” system for building a social network.
The underlying structure of most (if not all?) discussion systems can be expressed using a differently shaped tree of messages.
- Global linear list (IRC-like)
- List of linear lists (4chan, phpBB-like, StackOverflow)
- A selection from user-specific (semi-)linear lists (Twitter, Instagram, Facebook…)
- List of trees (Reddit)
This is not some grand insight, but since the base is the same, the rest can be thought of as different “modifiers”. For example:
- Optional (4chan)
- Media type:
- None (4chan)
- Voting scales
- Single scale (Reddit)
- Multiple scales (Slashdot, Facebook)
- Voting options (only adding or also subtracting score?)
- Voting consequence
- Approval necessary? (Metafilter)
- Expiration date of posts, r9k-like constraints…
Obviously, it’s a tough job to write software for just one type of a social network, let alone all of them. But for experiment’s sake, this doesn’t really need to scale to millions of users, and the UX doesn’t need to be perfect either.
It would be interesting enough just to try out what a Twitter with optional anonymity would look like, image-only Reddit, real-time phpBB bulletin board, time-limited IRC with voting and forced anonymity (etc.), and I don’t think it’s impossible to create a UI that is abstract/general enough to allow even for weird use cases like these.
Hardware // Interfaces
For note-taking on the go or other input-disadvantaged situations etc.
- Most probably chorded 🌐
- but are there alternatives?
Shape could be 3D printed to fit a hand’s grip kind of like a Nunchuk 🌐?
- How to have feedback for what’s typed?
After I got into computers by 1962… damnit I wanted a machine that would allow me to type with one hand wherever I went! And we still don’t have it.
Ted Nelson - Intertwingularity: When Ideas Collide (9:37) 🌐
Haptic feedback mouse
- (I think this has been done before, but only in a handful of models)
- why the heck is there no haptic feedback on a mouse?
- Even just a vibration motor strapped on a regular mouse, buzzing on a button mouse-over or on window edges seems like a huge improvement
- (not to mention the magic Apple does nowadays with simulating depth, etc.).
Shouldn’t be too hard to prototype on the hardware side?
Personal Information Management
Multi-Calendar // Timeline
- Allow preliminary dates / times (see Uncertain Interfaces)
- De-duplication of events
- Be intelligent about multiple calendars having the same event (same time, fuzzy match titles) and merge them
- Broad / long-term events, boundaries
- No Earlier Than, etc.
Personal “everything log”
- e.g. allow for importing Garmin Connect data, Daylio data, Loop Habit tracker data, etc…
- partly implemented in:
Mental fatigue tracking
Priority: Literally my bachelor thesis topic
The cursor entropy part, anyway; I’d like to experiment with the rest later.
- Blinking tracking
- Cursor entropy tracking
1st phase: Just see if it works and corresponds to subjective ratings? 2nd phase: Adjust UIs, color temperature based on fatigue levels?
Sleep video movement tracker
Point a cheap IR camera at my bed, quantify pixel differences, see if it matches up with wristwatch sleep tracking?
(Cons: Might get really creepy, but worth trying!)
mp4museum with a web interface
- On first run, start its own AP
- Allowing you to connect to a web interface, allowing to potentially set a SSID/password to connect to
- Build image from https://www.raspberrypi.org/software/operating-systems/#raspberry-pi-os-32-bit 🌐
- Is there something like Docker but for Raspberry Pi images?
Perspective Transformator // Mapper
Mostly a workflow-optimizer rather than a novel approach, but touches on interesting topics (i.e. real-time rendering on the raspi)
Universal video-mapping solution for galleries, etc.
Take HDMI in, transform according to the web interface, output over HDMI.
(Hardware for HDMI input: https://github.com/pikvm/pikvm 🌐)
https://explainshell.com/ 🌐 but in reverse?
- Rewrite what’s feasible (
- Models for parsing the output of known utilities?
- Could also include typing of input - i.e. smart autocomplete based on what’s expected at any given position…
- (What language to use?)
Priority: B https://gitlab.com/tmladek/overseer 🌐 except better
- Add quote sources
- From twitter channels, etc.
MQTT-based process SIGSTOPper
VS Code Outliner Extension
Priority: Why me?
There’s currently no extension that would provide an outliner editing experience but without the whole shebang (e.g. Foam 🌐, I think?).
I just want to be able to travel in lists and collapse them easily.
- A way to correlate data in human-time?
- NLP data -> language conversion?
Other lists of ± project ideas that I found particularly inspiring: