vanderwal.net

Updated Categories with Sparklines and Search is Now in Production

I made a couple of updates I have long wanted to make to this site. I’ve been wanting to see frequency of categories used on my blog for more than 15 years (or pretty much since I’ve had category list pages). I have also wanted to have blog search and the utter mess that Google Search has become in recent years, where my site isn’t showing at all at times has driven this. These additions will likely iterate and adapt a bit going forward.

Updates to Category Lists with Sparklines

I have basically had two category list pages for years: Category List (which is alphabetical sort) and Category List by Use. I have kept these two and added sparklines to them (Sparkline - Wikipedia). Each line now has a small line chart that covers the 25+ years and what periods had used the category and some sense of the volume of use over time. One category list view I wanted and was missing was one to show a view with the focus of most recently used categories, so there is now a Category Recently Used List that not only groups by most recently used (and in the same entry keeps the alphabetical sort) but also shows the date of the last use in the list. Personally, I have been finding this recently used list view the most helpful and interesting. Skimming through the list I know I have more recent posts that have covered or touched on a subject, but it didn’t include the category, and that becomes a quick task to fix that gap.

Sparklines?

I have been a big fan of sparklines to give quick understanding of data’s distributions at a glance, which I learned about in “recommended reading” of Edward Tufte’s book The Visual Display of Quantitative Information | Edward Tufte in grad school in social / policy quant classes. (There are are many Tufte essays and book annotations on sparklines at Edward Tufte Notebooks & Sketches | Art, Science and Sculpture).

Creating the Sparklines

In creating the sparklines for my category lists I looked initially (and have long looked at them) creating static images from the data and bringing the images in (this would mean updating the images and replacing the old with the new ones, which is relatively straight forward programmatically and something I’ve done in the past, but not optimal) and I also looked at JavaScript but it was a bit slow. I poked at using creating SVGs (which work well when printing or zooming in) and often are much quicker and less strain on a browser than JavaScript. I’ve had a few goes at SVG in the past and I get get my mind around simple shapes, but I would need a little help with sparklines. A couple years back on a sparkline spelunking I found Easy SVG sparklines | Alex Plescan which showed the how. But, I have SVGs somewhat in the same category as regex, which is I do it rarely and I’ll just use Claude Code | Anthropic’s agentic coding system \ Anthropic to assist with the creation.

Chunking the Data for a Sparkline

The other part of sparklines is they are intended to be small glimpses and I have 25+ years of posts and a monthly temporal segmentation would make for a long graphic. I played around with breaking things down to quarters, but in the end I went with two segments per year and roughly 50 data points to map out on a line chart. Running a test with the two data points a year was a reasonable enough glimpse to sort out if the category was used recently or what the variation of use was over time.

One of the interesting discoveries with the first lab run of the categories of sparklines was the rather “U shaped” distribution of the use of categories, which pretty much calls out the lull I had in blogging. This softening of blog post rhythm is something I call, “when Twitter ate my blog” (where the interesting things I would discover and want to share and interact around ended up on Twitter rather than my blog(s)). Other patterns that surfaced were limited use a category in a period when I was rather sure I had posted on the subject, some of this was I was not using the term in that way or I didn’t have the category in my system yet. One of the things that helped sort this out was using my blog search.

Search is Now Out of the Lab

One of the things I have been working on and using my my Lab at vanderwal.net is blog search. But, the modifications I made to the Category pages I found I was leaning on my blog search a fair amount to investigate things. But, the categories and blog search are both in the blog section of this site, so making the change from the lab to the production side made sense. One of the things holding back moving search over, was I had an SVG of a magnifying class in the menu bar with “Blog Search”, but no matter how small I made the image it still was messing with the vertical layout of the menubar. In removing the magnifying glass and just using text things kept to the same layout.

Bringing in Search

The search in the menu in the pages in the blog section with “Off the Top” or “random” in the URL which is where there are currently menubar links to Blog Search. I have the menu bar link to a search page to search from rather than a JavaScript drop down or other menu bar convention (again layout of the menu bar was part of the considerations).

When I was working on search in the Lab section I found I needed to make some modifications to the database to have quicker search and I needed to modify the database engine so I could have search include 3 letter terms as a minimum rather than 4 letter words. In working on search I found many of my early posts didn’t (and still don’t) have titles and I was using the title as the link. I initially thought I would just add titles, but there are around 300 posts that don’t have titles (I’m adding some as I touch the posts for other clean-up issues), but I ended up coding the search results to have the results just fill in “Blog Post #…” as a proxy for a proper title.

The initial 170 or so posts are not in the database and are therefore not in the search.

Bringing Search and Categories In

As I went to move the category list pages out of the lab and into the production side I needed to modify a few other templates and pages to add the updated links. In doing this I realized I could also easily update the menu bar to include “Blog Search”. So, I took a little bit of time and made both changes at the same time.

Not all of the links are in yet. If you see something a little off with category lists or missing blog search links let me know.

Joy Filled Email about Poem1 Shipping

This weekend started with an email about something shipping that I’ve anticipated for nearly 2.5 years. In January 2024 Matt Webb posted about a hardware clock what shows AI created poems for all the times of a day on an e-ink screen. This was something that caught my interest, less for the AI and more for the whimsey. But, the other part of it was hardware and the adventure that brings.

Not Matt’s First Hardware Experience

Matt had been part of Webb and Schulze, which turned into BERG London that was highly innovative playing with ideas at the edges. One of their creations was little printer. Part of watching that journey was some of the creation and manufacturing of the hardware that Berg shared out.

With this poem clock that Matt named “Poem1”, he turned it into a kickstarter project, and that brought along with it a look into the creation of a hardware product as it really is for things not at the scale of tens of thousands, if not orders of magnitude higher. Hardware is hard for electronics with getting all the parts approved, and then lined up to run through production it is a painstaking process and incredibly long.

So Excited by the Email

When I saw the email I was excited, like one is for people finishing an around the globe sailing adventure or some other long arduous feat. Getting the device will bring enjoyment, but watching from the blog about the inception of the idea, through turning it into a Kickstarter project and the emails about progress and the long waits between emails when approval waits were happening (or looking for plan B or Q for dealing with a part change needed).

I am so happy for Matt and having his device out into the world in people’s hands shortly.

Automating the Daily Dump Note

In work and personal life I’ve long been an automator. It started in the ’90s, but actually started when I was a kid and talking with my dad about systems, making them more efficient, and removing steps, and automations. Any manual rote step I look at and think about making it a click or “just run”. I’ve used Obsidian since June 2020 and have a lot of things in my markdown notes trigged by TextExpander.

Reducing Steps Setting my My Daily Dump Note

Little by little I’ve been building out some automations for my note taking that sits under Obsidian’s watchful eye, but may start elsewhere or seeds start elsewhere. In the morning I run some automations that sweep up videos of interest I forwarded to myself from the prior evening or that morning that are in a personal instance of Slack. I have one that pulls in notes in Drafts with a specific tag then archives the note. One last piece I’d really like to add is to have my known items from across my calendars dropped in at the top of that Daily Dump note.

Getting the calendar dump is something I’d really like to have before I get to my laptop in the morning. But, each of these automations could and should be automated and dropped in the note before I fully get to it as well.

This takes one thing… having the note.

Creating the note with the series of headers and subheadings is something I usually do in a new markdown note with TextExpander each morning. But, I can’t automate with TextExpander (or I haven’t sorted out how to trigger that from a far).

Automating but Not with Apple Shortcuts

I started a couple days ago thinking that since I had Actions for Obsidian I could easily create a new Daily Dump with the correct future date as a title and have it trigger the Templater template for the note with the correct date. I quickly got the file created properly, but in over 4 hours that included a couple hours the following day it was still not working properly. This also included a few different apps and pieces and if any of them change it breaks.

Since I not only wanted to create one note with a template for the next day, but to have a few days ahead created (say a week at a time). I gave up on Shortcuts and in under 20 minutes I had a python script doing exactly what I wanted. This included setting up a cronjob to run it on Sunday evening and have a week of notes set.

Using the Output

This morning I opened Obsidian and one of the future day’s notes I had setup as tests was a click away and I was capturing an idea. This is a nice change and one I wished I had done long ago.

The next step is automating bringing in the aggregation automations notes and dropping that into my Daily Dump.