vanderwal.net

My Automations Over the Ages and Today’s View

For a few decades I’ve converted regular manual tasks into automated tasks for many things. I use short code keyboard conversions with TextExpander (TextExpander), Alfred (Alfred), Keyboard Maestro (Keyboard Maestro), Hazel (Hazel by Noodlesoft), Apple Shortcuts, and a couple other small apps and services. But, I also have quite a few scripts in python and Ruby, but a few pure bash scripts. I have a handful of apps that use Claude Cowork (Claude Cowork | Claude by Anthropic) on a schedule and used where the data is not always standard, so it aggregates and structures the data where it often later gets processed by python.

I started building and running automation scripts in the 1990s to handle tedious tasks with multiple steps, or would likely be victims for typos, and which I was handling daily. A task that would take 3 or more minutes (often longer) and had structured stable data I could search made it viable to become a script and automated. I had them to filter email to folders, filter downloads to common folders / directories, and one to turn off audio for the evening which monitored Twitter every 30 minutes for my nightly signoff (“off to read and sleep - zzzz”) and it would mute audio. I’ve had a few for audio that would capture podcasts I was on for a local copy during recording that took just one button click to set Audio Hijack Pro properly and it just worked.

Moving to the Current Adaptive Flows

The last six months (or a little more) I’ve been building aggregators to capture what I’ve done the day before (and the week prior for a manual review). I’ve also automated my morning Obsidian (Obsidian) setup, so that my daily note (or Daily Dump for my notes, in my vocabulary) templates are created for the week ahead with a python script, rather than creating a new note and use a TextExpander snippet to build the structure each morning. But, doing this I realized that I could take my daily task list and the list of potential tasks I keep at the top of my Daily Dump and script the moving uncompleted items to that day’s Daily Dump from the day prior, but also quickly run a calendar scan for the day with Claude and the BusyCal MCP to have my known items at the top of the note written directly in the note. The completed items and checked off were left on the prior day’s Daily Dump in their sections.

In order to run the scheduled python scripts I need my laptop to be awake in the background (but it still the screen remain locked) so to run early morning tasks. I found PMSET as the key to handle the wake-up for the first run and have a python script handle the remaining wake-up processes for that day.

I now have a mix of Claude Cowork Skills (which if I sort out the data is stable enough I convert them to python for consistency, speed, and sustainability) and python scripts that pull my prior day’s files created and worked on list, my items in a feed of the prior day’s additions to Instapaper, Pinboard, and a list of new items in DevonThink as PDF (with links to the PDF as well as a working URL to the source of the item), a list of Readwise articles and video transcripts, and videos I liked and sent to my personal instance of Slack. These lists are in markdown and placed into the directory Obsidian watches over. Most of them a little later in the morning are transcluded into my Daily Dump note for the day where I read through them.

Morning Process

My morning process is to read through what was done the day before and edit them to add links to other resources in Obsidian and write notes. Often I have some entries that will turn into that day’s list of things to do or one of the “potentials” list. I will take items that need to be their own item note or be included into a concept note. This keeps the items fresh on my brain and connecting things to where I can get back to them more easily.

I write this today, as this morning I found my morning automation hadn’t fully run properly, because my Sunday evening script to create the week’s Daily Dump notes didn’t run as it was a cronjob and I was cooking with the laptop asleep so the job didn’t kick off. This morning I converted the cronjob to a launchd task instead and to run on Sunday during another automated wake open window.

Weekly Aggregation

Sunday’s I also run a weekly wrap-up, that pulls everything done for the week. I’m finding this is a bit too much information for a quick review and a manual assessment for the week where I aim to map out things for the upcoming week. It is a bit too much information overload. I don’t know what I’m going to do about it. Now that I have things automated and aggregated I need to rework my human side of things.

The Value of the Automation

The goal of my automations is to give me time back and remove the manual and grunt work so I can do the thinking work. I’ve done this with personal life and work life (and large systems and projects). The current round of automation and aggregation surfaced that I am doing more than I thought I was doing. Another goal is to give more time to do the human thinking and the projects and work needed. I do this so I can do the creative human work and the human thinking work (I find large errors in summarization in nearly all generative AI and creative work so I keep that at bey and use it for grunt work that I check). The scripted automations don’t need the hand holding and I have some occasional test run to check on them to see if they need attention and honing.

Automating the Daily Dump Note

In work and personal life I’ve long been an automator. It started in the ’90s, but actually started when I was a kid and talking with my dad about systems, making them more efficient, and removing steps, and automations. Any manual rote step I look at and think about making it a click or “just run”. I’ve used Obsidian since June 2020 and have a lot of things in my markdown notes trigged by TextExpander.

Reducing Steps Setting my My Daily Dump Note

Little by little I’ve been building out some automations for my note taking that sits under Obsidian’s watchful eye, but may start elsewhere or seeds start elsewhere. In the morning I run some automations that sweep up videos of interest I forwarded to myself from the prior evening or that morning that are in a personal instance of Slack. I have one that pulls in notes in Drafts with a specific tag then archives the note. One last piece I’d really like to add is to have my known items from across my calendars dropped in at the top of that Daily Dump note.

Getting the calendar dump is something I’d really like to have before I get to my laptop in the morning. But, each of these automations could and should be automated and dropped in the note before I fully get to it as well.

This takes one thing… having the note.

Creating the note with the series of headers and subheadings is something I usually do in a new markdown note with TextExpander each morning. But, I can’t automate with TextExpander (or I haven’t sorted out how to trigger that from a far).

Automating but Not with Apple Shortcuts

I started a couple days ago thinking that since I had Actions for Obsidian I could easily create a new Daily Dump with the correct future date as a title and have it trigger the Templater template for the note with the correct date. I quickly got the file created properly, but in over 4 hours that included a couple hours the following day it was still not working properly. This also included a few different apps and pieces and if any of them change it breaks.

Since I not only wanted to create one note with a template for the next day, but to have a few days ahead created (say a week at a time). I gave up on Shortcuts and in under 20 minutes I had a python script doing exactly what I wanted. This included setting up a cronjob to run it on Sunday evening and have a week of notes set.

Using the Output

This morning I opened Obsidian and one of the future day’s notes I had setup as tests was a click away and I was capturing an idea. This is a nice change and one I wished I had done long ago.

The next step is automating bringing in the aggregation automations notes and dropping that into my Daily Dump.

My Writing Process with Obsidian

A few times lately my writing process has come up as it relies heavily on markdown for portability, longevity access, flexibility, and using the Shift Happened – Part 2: Small Apps Loosely Joined – Personal InfoCloud. The Small Apps Loosely Joined concept is something I use heavily for many things, but for writing the use of markdown I use the concept quite heavily.

Recently in a IndieWeb gathering, James who runs and writes James’ Coffee Blog shared his process and workflow for writing through to posting on his site, which was similar to my own process, but uses different tools along the way. He had something in his process he was looking to improve upon so I walked through my process.

My Process has Morphed Over the Years

My current writing process is an extension and evolution from my initial processes that trace back to college. But, it was in the late 1990s and early 2000s that my current process formed iterated upon. My early writing for blog posts and articles all start in markdown, which years back was well structured text and sometimes HTML for the structure.

My notes prior to formal blogging (started at the tail end of 2000) that I posted to my site were all hand coded HTML (or raw coding). If a note was going to turn into a blog post it quickly was marked up, and often as I was making the note or post.

Automate Early

One of the core elements I learned in the 1990s was to automate early for anything you can automate, use the tools you have at hand to help yourself be more productive. Most of my writing up through around 2010 was text and then quickly turned into HTML markup, as it is simple to do. But, the tough part is connecting related content, which is why I created my personal content management system (CMS) to run this vanderwal.net blog in 2000 and fully put it to use in 2001 (I used Blogger for a bit early on). Taking rote patterns and automating them was a great addition to help my process in 2000 to now.

In 2010 I shifted all of my note taking to markdown as moving across apps and devices made using other note making methods difficult to access and use. This shift to full markdown for notes, helped my writing move from notes to posts and articles much easier.

Markdown Workflow Process

Today and the last 5 years my notes start most often in Obsidian (and on mobile in Drafts, which is great for the good practice of get it out of your head and then sort out what to do with it). I have many notes flagged as “blogfodder” and track those through the writing process and moving them from a stub of an idea, to draft, not posted but ready, and posted (which includes a link to where it was posted). I’ll have another post about my new blogfodder process, which I’m really liking.

The writing starts as a note and gets fleshed out, if it is needed. If it is a short item I may stay in Obsidian and then grab and drag the markdown file in the Finder to the next step (I will get to this shortly).

Quite often I will leave Obsidian and from it click to open the Finder with the markdown file highlighted and then open that file in iA Writer - iA Writer, which is a nice focussed writing app for markdown, with additional capabilities. I often use the old journalism marker for needs attention (tk!) or “to correct” for things that need links, fact checks, or reference notes. Once I’m happy with the writing and structure in iA Writer (or Obsidian) I move to the next step.

Move to Prep for Posting

In years past I would take the markdown and quickly take the markdown structure and convert it to HTML markup by hand. Around 2013 or so I started scripting this transition, but the script was fussy. I’m not sure when, but it wasn’t long after this, I ran across Marked 2 - Marked 2. I think I started using Marked to convert markdown to PDFs and Word documents (for things that need to be sent out for formal article transformation in publications from Word). But, I realized Marked 2 had really good markdown to HTML conversion that was as good as my script, but not fussy. As I moved to Obsidian with properties in front matter and a footer with blogfodder tracking, it can remove all of that with ease, translate external links very nicely, and remove all backlink notation.

It is often in Marked 2 that I find markdown problems or the tk! marks. Marked 2 also includes some light grammar checking, which I appreciate and I’ll work through those suggestions. Then I flip to the HTML markup view and make edits, if needed, there.

Open My Blog Entry Form

Once I’m happy enough with the post I open my blog entry for for vanderwal.net or Personal InfoCloud and paste the HMTL into the form for the body of the post. On vanderwal.net I add in the title, location, type of post (these days everything is a weblog, but in the past it was more diverse), then select the related categories and submit it. Then I just to look at the post and review it again. If it needs an edit (up until July 5th) I would go into the database and make edits to the post there, but now have an edit process in my CMS (after 25 years, I figured it was about time). Once that is done, I go click to generate the RSS feed for the site, and send out alerts to services that share out links and summaries farther.

Wrap-up of the Workflow

This workflow is now done, except for seeing spelling errors or things not right and needing tweaks.

The process that starts in a markdown note, then progresses through to a more formal writing process and flow. I replaced a lot of manual steps that I didn’t think were difficult nor took a lot of time and automated the steps that do exactly what I had been doing with the same level of care, but saving time and reducing errors.

I don’t use AI in any of this writing process. I run across too much AI written content that is lifeless and doesn’t sound like the writer any longer. I’ve stopped reading many colleagues who used to have great ideas and a great personal voice, but are now just bland slop through the use of AI that tilts at, but doesn’t achieve mediocrity.

I have my own quirks and writing patterns, which I am fine with. I don’t write to impress, but to get honest ideas and understandings out. I blog and write to find connection with others of like or similar minds.