Skip to content

ETL to QE, Update 46, A Vision that Never Solidifies

Date: 2024-11-14

So here we are a year into these update posts, and coming up on a year and a half since the first commit in the original Question Engine, Jan 22 2023, and over 4 years since my the first commit of keybase binding, Sat Oct 17 2020.

The original vision of Keybase Binding was to be able to sync up ones keybase data with a daemon that they can interact with rather than the keybase client. I should go take a look at my keybase exports and see the first mention of Keybase Binding. Well I do have this video where I described what became of the keybase binding #dentropyDaemon 001 A Vision for applications on the internet that was uploaded, May 29 2021. It would be maximum cringe to watch that right now, so I shall.

Oh I even found the script here.

Here are some thoughts on that old video,

The one thing I said back then that may be useful is, "If people owned their data they might treat one another differently". But before we analyze that let's sort out some other issues first

Rant about Homelab Software

The code I presented still exists here though I only run and use like two of the apps I talked about back then.

There were a series of issues with that self hosted stuff such as,

  • Authentication was far too complex
    • Every app required a separate username and password, though it was possible to setup LDAP as a centralized server for applications that only worked on a single domain/server and did not federate.
  • Lack of proper federation
    • Federation is app specific, the only apps with federation are ActivityPub apps like Misskey and Nextcloud
    • ActivityPub federation is really glitchy with many implementations not talking nicely to one another. You also can't migrate your old messages one Misskey/ActivityPub server to another.
  • Lack of high availability
    • Servers go down all the time therefore having a backup is very important. High availability was/is too complex to setup, let alone maintain.
    • For example maintaining a multi node PostgreSQL SQL cluster really messes with setups especially when some services want a specific PostgreSQL server version, specific PostgreSQL extensions, or a whiny and require something like MariaDB
    • Some apps only run on SQLite which effectively run over a network file share like NFS, samba, or ceph. Though there are backup streaming utilities Like the one talked about here. This makes deployment, backup, and restore much more complex. Plus then there are sqlite extensions making things even more complicated.
  • Lack of easy backups
    • Backups usually require shutting down an application for up to 10 minutes which can be really annoying.
    • Restoring form a backup is very complicated and not automated. And even if it was automated it would probably break most the time unless the system is designed correctly the first time.
  • Complexity of data sachems
    • One part of the Dentropy Daemon vision I never articulated well back then was taking all those disparate self hosted application then creating a unified schema for the different apps to talk to one another.
    • The idea was that all the data across all self hosted apps could be queried via a single interface.
    • Then centralized apps such as Telegram, Instagram, and Reddit could be bridges to specific apps so their data can be queried via the single interface as well.
    • For example, a contact in Monica CRM would connect to a user in wikijs which could connect to a user on Misskey. Or a project in Open Project could connect seamlessly to Trilium Notes and wikijs.
    • This task was far too complicated especially without any
  • Self Hosted Stuff has lots of bugs
    • I spent a couple hours trying to deploy Misskey and get it to federate about a year ago. After troubleshooting for like 4 hours I said fuck it, I don't want to maintain an ActivityPub server any more. If I can't troubleshoot my own services then the people running my Homelab Server OS aren't going to have much luck.
    • Making these self hosted apps highly available makes them harder to troubleshoot.
  • Plus there are tones of existing projects that do exactly what I was aiming for such as

What We Learned and What Comes Next

  • All the apps I use break down into some core functionality
    • RBACed Lists of Nodes with Tags and Edges
      • All Social Media
      • Blog
      • Todo Lists
      • Obsidian
      • Bookmarks (Raindrop.io)
    • Event Streams
      • Activity Watch
      • Browsing History
      • SystemD Logs for Self Hosted Apps
    • Graph Annotations
      • Annotate Ebooks
      • Annotate Websites
  • It looks like we are going to have to commit the sin of the Standards XKCD and create a new standard... except indexing what already exists.