Skip to content

ETL to QE, Update 52, Scrape What Now?

Date: 2024-12-16 and 2024-12-20

In my Nostr Tutorials I have documented how to scrape nostr events and put them in a easy to query format. But what is the point of having all this data? What was the point of the using the Discord Binding to generating answers for all those Questions for Discord Data?

As a reminder the goals of the Nostr Tutorials is to allow one to,

  • Publish and update a Obsidian(Markdown) folder to Nostr using NIP54
  • Develop Bots that can be tagged to interact with threads
  • Develop bots that can be interacted over encrypted direct messages
  • Scrape (export) nostr data and transform and load it into a easy to query format (ETL)
    • This includes replaying events to other Nostr relays

The last item in that list is something I am having a hard time developing a proper user story for. What does one do once they have scraped a million nostr events? What does sense making look like?

Sense requires a goal, an Animated vision of the future. I have a couple documents in here I should be able to use as a compass for meaning and direction,

Those are too many directions, we need way to organize these similar to - My Grift - Organizing Principal - Personal Legend

Well there is that quote from Pantheon 2021,

Greatness is Other People

I need to stop thinking of people as abstract data points, but as individuals I can authentically connect with.

The problem I have with authentically connecting with people I find online is I become overwhelmed and massively over think, then go back to lurking because that is more comfortable.

To effectively understand a problem you need to be interacting interactively with the world

What would it take to get people I know to care about NOSTR?

Everyone I know seems to spend their time in their specific Meme Sphere. Once we are at QE we are supposed to have a map of Meme Spheres

The problem that I am Nerd Snipped by is How to produce a contextualized infotainment feed?

I want to map how people grew in popularity over time and how they interact with other people.

Then once I patters and playbooks for how people interact with one another I can act the same way rather than getting overwhelmed.

I want to Engage Consciously

I want to describe, map out, the narratives people are participating in. For example, when someone posts a nature photo, what are their expectations for how people are going to react?

Do we want to craft a Schizo persona that engages in a very unique fashion? Or what about being the Smokey Mirror, a reflection of ones persona making them see themselves similar to a Jester or Internet Comment Etiquette with Erik, remember Fold awareness upon itself

Okay here is the root of my problem, I want to know everything, immediately, quickly. Since this is not humanly possible, I end up doing nothing... or in my case writing into the void.

What ever happened to, What audiences am I trying to write for with my blog?

My goal is to find my online community, make friends, explore ideas, and inspire one another towards greatness.

The idea that keeps poping up, is I want my own Nostr NIP for labeling data within the context of Knowledge Graph's. For example, Schizo posters in one category, AI people in another, Nostr Developers in another, Shitposters in another, Blockchain people in another... you get the idea

There is a quote I wrote in my journal today, Master the basics or you are fucked. Here I am trying to solve a undefined problem without understanding the basics.

The basics of what?

What is the message of the medium of Social Media?

Now that is a blog post I should write, do we want to be famous, no, well maybe, we want to find the most interesting conversations on the internet as they are happening. Remember Reddit was described as The Front Page of the Internet.

This CGFS, QE, Dentropy Daemon medium is supposed for force people to Engage Consciously. But what does Engage Consciously even mean?

When was the last time I Engaged Consciously?

I believe next steps for this Scrape What problem is to come up with a bunch of Search Phrases so when I throw all my data into paradedb. I can use my Raindrop.io tags and that list of Compas Meaning Directions to come up with search queries. Maybe some LLM magic might come in handy.

What is that in clear language, so clear that you can hand the task off to someone else?

We are tacking the problem of How to make sense from a fire hose of social media data?. We want to avoid useless analytics queries unless they enable one to Engage Consciously. We want to focus on finding the most interesting conversations happening on the protocol. Articulate what you are most interested in and what you want to talk about. Then brainstorm a list of queries and prompts.