The Gilpatrick

Weather In My Life

Scott Franz — Sat, 19 Oct 2024 04:00:00 GMT

The inspiration for this blog post comes from Volodymyr Agafonkin’s old Observable Notebook where he looked back at how snowy his childhood winters were in Ukraine. I used NOAA’s Climate Data Online to fetch the weather data of the three places I have lived so far in my life. Here are the highlights, but feel free to look through the heatmap yourself.

🌡️ Hottest day of my life: , — °F in
🥶 Coldest day of my life: , — °F in
❄️ Snowiest day of my life: , — in. of snow in
🌧️ Wettest day of my life: , — in. of rain in

One of my most memorable days of childhood was the one and only snow day in Bakersfield, CA on , it was inches of snow and we were able to make a snowman and have a neighborhood snowball fight. I also remember the first snowfall we had in Ann Arbor on , , it turns out (to my surprise) to be the biggest we have had since we moved here almost 10 years ago!

Code

Salvador Dahlia

Scott Franz — Mon, 16 Oct 2023 04:00:00 GMT

Introduction

I was inspired by Shruti Mukhtyar’s Observable notebook a very long time ago. Originally, I was planning on taking pictures of Ann Arbor’s infamous Peony Garden at the University of Michigan’s Nichols Arboretum. The peonies bloomed in early June. My family and I definitely enjoyed them, but I forgot to take pictures. So instead, this blog post was left on my long list of ideas.

Back in August, I went on a long walk and ended up at a community garden by my house. This time I remembered to take pictures, and so below are the pictures I took of a beautiful dahlia garden. Project Grow is an awesome organization in Ann Arbor. They just had their 20th Annual Tomato tasting. We had a plot when we first moved to Ann Arbor. It was so much fun. My family and I always try to go to their plant sales in the early spring.

Dahlia Garden

I imported Pierre Ripoll’s image input and used it to navigate the pictures of flowers I took on my phone. If you click on or hover over a photo with your mouse the flower image below will change along with the color palette. Try it out.

Code

Google Sheets in Observable

Scott Franz — Fri, 07 Jul 2023 15:03:14 GMT

Introduction

Google Sheets are a great way to stream data into your Quarto workflow with Observable. D3 has a handy way of fetching different data files if you provide the URL. This is Observable’s recommended way of going about it. I find Google Sheets great for small datasets that are updated frequently.

Read a Google Sheet

Here is an example that is taken from Observable’s Google Sheets Starter Notebook.

getCsvUrl = url => {
  url = new URL(url);
  const id = url.pathname.split("/")[3]
  const gid = new URLSearchParams(url.hash.slice(1)).get("gid") || 0;
  return `https://docs.google.com/spreadsheets/d/${id}/export?format=csv&gid=${gid}`
}

// Above converts a Google Sheets website URL to its CSV URL.
// You can also go to “File → Publish to web”, select the “Comma-separated values (.csv)” 
// type, select the sheet with your data, and use that CSV URL directly with `d3.csv` above.
// If you need data to update faster, try the Google Sheets 
// API: https://stackoverflow.com/questions/30082277/accessing-a-new-style-public-google-sheet-as-json/44479726#44479726

data = d3.csv(getCsvUrl("https://docs.google.com/spreadsheets/d/1wZhPLMCHKJvwOkP4juclhjFgqIY8fQFMemwKL2c64vk/edit#gid=0"), d3.autoType)

I replaced the URL with the Data is Plural Google Sheet.¹ I hooked it up to a search input and a table where you can select a row to see the details and dataset links.

Code

Recipe Web Scraping in Observable

Scott Franz — Mon, 19 Jun 2023 04:00:00 GMT

Introduction

This web scraping blog post is heavily inspired by the Paprika App which I just started using. It saves the recipe information and gets rid of all of the annoying ads. My biggest pet peeve is scrolling through 5 million pop-up ads to get to a recipe. This blog post is really just to see if I could web scrape in ObservableJS. I plan on creating an actual recipe web app eventually where I can display my recipe information from my own database. But for now this is my proof of concept.

How it Works

I stumbled upon Ben Awad’s blog post on scraping recipe websites. It turns out that most websites have metadata for search engines. Nicolas Lambert’s Observable Notebook shows how you can use both Axios and Cheerio in ObservableJS to web scrape data.

Tip

One problem with this method is that if a recipe website uses purely JavaScript you might need to do more work (look into Selenium or Puppeteer).

axios = require('https://cdn.jsdelivr.net/npm/axios/dist/axios.min.js')

result = axios({
  method: "get",
  url: `https://corsproxy.io/?${input}`
}).then((result) => result.data)

Most of the time recipe websites will not have CORS enabled so to get the request you will need to set up a CORS proxy server. So the above code gets the HTML request via the Axios library. Here is what the raw HTML looks like.

Code

UK Pet Cats

Scott Franz — Tue, 31 Jan 2023 05:00:00 GMT

Introduction

I wanted to do a Tidy Tuesday blog post for a while now and I finally carved out some time to make one. This week has tracking data from a study on pet cats, looks like only the UK test site was included. Checkout the Github Readme for more info.

Data Processing

I decided to use Observable JS this week and imported the Arquero library in the spirit of being tidy. Arquero is basically dplyr for JavaScript, so if you were too annoyed or intimidated by JavaScript data wrangling methods before hopefully this gives you another excuse to try it out. I didn’t do much processing this week, just combined the cats_uk.csv with the cats_uk_reference.csv. I left in all of the outliers and didn’t compute any summary statistics. You can download my combined dataset by clicking on the button below.

Code

R TidyCensus with Observable

Scott Franz — Fri, 27 Jan 2023 05:00:00 GMT

Introduction

As you may already know I am a big fan of ObservableJS. I use it to create dashboards at work and for fun side projects like this blog. I think a big advantage to ObservableJS is that it is an extension of JavaScript with a reactive runtime. JavaScript is the language of interactivity on the web. All of it can be run from your browser, you don’t even need a server like you do for Shiny.

Sure there are R packages that create interactive graphs and maps, but for the most part they are built on top of JavaScript. There are also limitations to the browser’s engine, you have to be wise with the amount of data you are using (or be wise about how you use it). But the degree to which you can customize your visualizations without knowing too much about web development is pretty unparalleled. Sure you could learn a JavaScript framework like React or Svelte and spend hours learning all of the related languages of the web.

But I find that ObservableJS allows you to live at the sweet spot of knowing just enough to do whatever you envision without being a full blown full stack web developer. Observable Plot just came out with some new mapping capabilities. There is a great tutorial on how to use these new features of Plot. But today I am going to show you how you can pair ObservableJS with the R tidycensus package using Quarto.

TidyCensus examples

The tidycensus package is a nice and tidy way to interact with the Census API. Kyle Walker created the tidycensus R package and has a free book called Analyzing US Census Data that covers the package extensively. I highly recommend perusing it if this is the first time you have heard of the tidycensus package. Below are two examples straight from his book.

When you first use tidycensus you will want to run the census_api_key() function. You can get a free API key from the Census. The second argument allows R to save your API key for future R sessions. So you don’t have to use this function everytime you use tidycensus. I wont go over the R code line by line but I commented in what each chunk is doing. Quarto enables us to use R for wrangling and pass the data into Observable.

library(tidycensus)

Warning: package 'tidycensus' was built under R version 4.1.3

library(tidyverse)

-- Attaching packages --------------------------------------- tidyverse 1.3.1 --

v ggplot2 3.3.5     v purrr   0.3.4
v tibble  3.1.6     v dplyr   1.0.7
v tidyr   1.1.4     v stringr 1.4.0
v readr   2.1.1     v forcats 0.5.1

-- Conflicts ------------------------------------------ tidyverse_conflicts() --
x dplyr::filter() masks stats::filter()
x dplyr::lag()    masks stats::lag()

# census_api_key("YOUR KEY GOES HERE", install = TRUE)

# gets median age of each state in 2010 census
age10 <- get_decennial(geography = "state", 
                       variables = "P013001", 
                       year = 2010)

Getting data from the 2010 decennial Census

Using Census Summary File 1

# function to make the data usable in Observable JS
ojs_define(example1 = age10)

State Median Ages

ObservableJS has a bunch of inputs built in. Inputs.table() is an easy way to see your data in a table. It isn’t an input in this example because it isn’t connected to anything. It is just a table to see our data columns and rows visually. But if you hover on the left side of the table you can select certain rows. This could be used as an input to filter rows in a dataset. You can also sort the rows in either ascending or descending order if you click on the column header.

Inputs.table(transpose(example1))

My only complaint (and it is a small one) is that you need to transpose the data when it is passed into an ObservableJS environment from R. Which either means you always have to put your data name in a transpose() function or have another step to name your dataset so it looks prettier. But renaming your data all the time gets old fast. So I tend to just transpose inline.

Observable Plot

Below I recreated Kyle’s basic ggplot2 example for comparison’s sake. Except I added a tooltip that enlarges the circle and shows the median age when you hover over it. This is obviously achievable in many R interactive graphing packages.

Code

Hello, Observable!

Scott Franz — Sat, 21 May 2022 04:00:00 GMT

Introduction

Observable is a new environment for JavaScript with a couple of tweaks (also supports HTML, CSS, Markdown, SQL, and TeX). It is the brainchild of Mike Bostock (D3.js creator). What sets Observable apart from previous programming notebook environments is its reactive runtime. In previous notebook iterations like Jupyter and R Markdown each cell could only reference previous cells in a certain order (from top to bottom). Observable is more like a spreadsheet where any cell/value referenced can update automatically (in any order). This makes it fun to use when composing interactive visualizations.

Why use Observable when there are so many interactive data visualization libraries in both R and Python? Well my take is that every interactive data visualization library in R and Python probably has JavaScript running under the hood. I feel like the only reasons we use these libraries are:

We don’t want to learn another language (especially JavaScript).
JavaScript is geared towards web developers who understand all of the other pieces needed (the DOM, HTML, CSS, SVG, Canvas, SSR, Frameworks, etc).
Data wrangling in R and Python is a lot easier and more powerful than in JavaScript.

But since JavaScript is the language that manages interactivity on the web, it only makes sense to use it for interactive data visualizations. Observable helps us use JavaScript without the trouble of not knowing what we are doing as non-web developers.

Tutorials

So I know notebooks are not for everyone, but I think they are a great resource for learning (which for me is half the fun of programming). I started using Observable, about two years ago because of this Vega-Lite tutorial (pictured below) by Jeffrey Heer at the University of Washington.

This was the start of the adventure for me. After learning the basics I was hooked. I learned more about what Observable itself is capable of doing. They have a great YouTube channel with plenty of live-streams and tutorials. They also have notebook tutorials on D3, their new Observable Plot, and how to analyze time series data. There are plenty of other contributors too. These two tutorials on D3: Tyler Wolf’s 25 Days of D3 and NYU Visualization’s Guides/Examples, were both very helpful. I find the community encouraging. The ability to interact with the code and fork other user’s notebooks has been a huge bonus. I regularly find interesting notebooks where I import a part to use for my own work. Mike Freeman, Philippe Rivière, Ian Johnson, and of course Mike Bostock have been integral to my learning.

Imports

In Observable, you can import anything. Whether it is a chart, function, data, input, etc. I will show you a few examples here and give you further reading if you want to get into the details.

import {chart as streamgraph} from "@d3/streamgraph-transitions"

The cell above imports a chart from a notebook in the D3 Gallery. It was created by Mike Bostock as an example of what you can do with D3 transitions. Here I am using it as an example of how easy it is to import things. In the cell below all I had to do was type streamgraph and voila.

streamgraph

If we wanted to import a map and a drop down menu from another notebook. We could do that too. This drop down menu (input) selects different map projections. Check it out! It is also interactive. Use your mouse to click and drag to explore.

import {chart as map, viewof projection} from "@d3/versor-dragging"

viewof projection

map

Below I imported an example dataset and some functions to add my own twist to the chart. Mike Freeman created some add-on functions for Observable Plot that enable you to both customize the tooltips and animation. So I over engineered this faceted scatter plot to animate the dots and when you hover over the dot it shows both a tooltip and enlarges the dot.

import {barley} from "@observablehq/plot-facets"
import {addTooltips} from "@mkfreeman/plot-tooltip"
import {addAnimation} from "@mkfreeman/plot-animation"

visibility().then(() =>
addAnimation(addTooltips(Plot.plot({ // I just added these wrappers to the beginning and
  marginTop: 50,
  marginLeft: 110,
  height: 800,
  grid: true,
  x: {
    nice: true
  },
  y: {
    domain: d3.groupSort(barley, g => -d3.median(g, d => d.yield), d => d.variety),
    inset: 5
  },
  fy: {
    domain: d3.groupSort(barley, g => -d3.median(g, d => d.yield), d => d.site)
  },
  color: {
    type: "categorical"
  },
  facet: {
    data: barley,
    y: "site",
    marginRight: 90
  },
  marks: [
    Plot.frame(),
    Plot.dot(barley, {x: "yield", y: "variety", r: 0, stroke: "year", title: (d) => 
      `Yield: ${parseFloat((d.yield).toFixed(1))}  \n Variety: ${d.variety} \n Year: ${d.year} \n Site:${d.site}`}) // this line for the tooltip titles and
  ] // this line on the end to add animation and enlarge the dots on hover
}), {r:15}), {type: "circle", attribute: "r", endValue: 3, delay: 100} ))

But wait that is not all that you can do! You can also modify your imported charts with your own or somebody else’s data. Here is an example of using a Bar Chart Race with the data from the original notebook.

import {chart as barChartRace, viewof replay} from "@d3/bar-chart-race"

viewof replay

barChartRace

So above is what it would look like if I didn’t change the data. Below is what it looks like after I import different data. This data comes from Emil Hvitfeldt’s notebook where he uploaded a csv file of R package downloads over time. I could have also uploaded my own csv file, it just has to match the format of the data for the imported chart.

import {data as rPackages} from "@emilhvitfeldt/race-for-most-downloaded-r-package"
import {chart as rRace, viewof replay as replay2} with {rPackages as data} from "@d3/bar-chart-race"

viewof replay2

rRace

A couple of things to note before we move on. You may have noticed that most of the charts were named charts in their original notebooks. In order to use more than one imported chart in this notebook I had to change the name. Otherwise, Observable would get confused if I referred to multiple cells with the same name. So in order to do that I use the as newName syntax. Additionally, it is easy to reference whichever Observable notebook you want by using the end of the URL. For example, I referenced most of the D3 charts by using from "@D3/nameofnotebook". There is a lot more to learn about imports and how useful they can be. Here is an introduction and here is an explainer of a new feature where you can “lock” your imports so that if a dependency changes it doesn’t ruin your import.

Inputs

Observable has some built-in libraries to help with user inputs. It leverages views which is a clever and way less confusing way to handle interactions and their outputs on the web.

In the cell below I gave a range of values 0 to 25, I gave a starting value value: 10 and I gave an interval to step between values in our range step: 1 and then I just named it r so we can reference it later. In Observable notebooks, viewof creates reactive variables of inputs so if I just do the following below.

viewof r = Inputs.range([0, 25], {value: 10, step: 1})

It updates when we move the slider. This works both in JavaScript and in Markdown here is the number: . This r variable can be plugged into anything we want. Lets add some color and data to make a chart.

import {penguins} from "@enjalot/palmer-penguins"

viewof color = Inputs.radio(["red", "green", "blue"], {value: "red"})

I reused the addTootips function I imported earlier and added the two reactive input variables we just created r and color. These two variables control the attributes when you hover over the dots in this visualization. Go ahead and change the fill color and size of the dots with the inputs above.

addTooltips(
Plot.plot({
  grid: true,
  facet: {
    data: penguins,
    x: "sex",
    y: "species",
    marginTop: 50,
    marginRight: 80
  },
  marks : [
    Plot.frame(),
    Plot.dot(penguins, {x: "flipper_length_mm", y: "body_mass_g", stroke: "island", title: (d) =>
        `${d.species} \n flipper length: ${d.flipper_length_mm} mm \n body mass: ${d.body_mass_g} g`})
  ]}), {r: r, fill: color}) //This is where the reactivity is happening

This also makes interacting with data very easy. Here are some NBA team names with their location.

teams = [
  {name: "Lakers", location: "Los Angeles, California"},
  {name: "Warriors", location: "San Francisco, California"},
  {name: "Celtics", location: "Boston, Massachusetts"},
  {name: "Nets", location: "New York City, New York"},
  {name: "Raptors", location: "Toronto, Ontario"},
]

viewof favorite = Inputs.radio(teams, {value: teams[0], label: "Favorite team", format: x => x.name})

So the cell above creates our input. It takes the teams dataset and uses the first variable in the array as its starting value value: teams[0]. The format option specifies how you want to present the value to the reader.

favorite.location

The name of the reactive input variable is favorite and we can access whichever variable in the dataset by putting a . and then the name of that variable. In this case it is location. To show the location of the team you can use the ${favorite.location} in the markdown. Try clicking on the radio buttons to see the different locations: . This way of writing in markdown is valuable when you want to create data driven documents. No need to edit your values manually in word documents anymore.

Below is an example of filtering a dataset (I am reusing the penguins dataset here). Click on the options below and see how the data changes when you select different options.

viewof checkbox = Inputs.checkbox(
  d3.group(penguins, (d) => d.island),
  { key: ["Dream", "Torgersen"]}
)

filtered = checkbox.flat()

Plot.dot(filtered, {x: "flipper_length_mm", y: "body_mass_g", stroke: "island"}).plot()

The chart above uses the filtered dataset to create the scatterplot. As you can imagine there are endless capabilities of pairing these reactive input variables with whatever you are trying to create. I won’t try to make something super complex right now, but explore Observable and you will get a sense of the possibilities.

Data Wrangling (and Analysis)

Originally, when I first started using Observable I would wrangle my data in R and then save a CSV file and then upload it to Observable. Since then I found Arquero which has been really awesome! There is actually a Tidy Data in JavaScript using Arquero that follows Hadley Wickham’s Tidy Data (Chapter 12, R for Data Science). Also this Illustrated Guide to Arquero Verbs is a good cheat sheet.

Data manipulation in vanilla JavaScript is still wonky to me. It makes very little sense in my head. A concern I had in the beginning was doing all of my data work in the browser. Saving my work online with a relatively new service gave me pause. For now Observable allows you to create unlimited amounts of notebooks for free. I have made over one hundred notebooks, and have not had any problems so far. Another concern I had (kinda still have) was Observable’s reliance on the browser’s engine. It is not yet equipped for processing big data files. File sizes are limited to 50 MBs per notebook and 1 GB over a 28-day period. There are alternatives like connecting to databases through Observable or using web APIs, but even then it is smart to subset your data if it is over a certain number of MBs for performance.

Luckily, I found a promising new product that will probably be my go to work environment very soon. It is called Quarto. It is a project sponsored by Rstudio that integrates Observable JS, Julia, R, and Python. It does a lot of cool things that I won’t even touch on but the coolest to me is the ability to write in both R and Observable within the same notebook. So if you need to do something with a large dataset it is possible to preprocess it in R and then do what you want in Observable. There is an option to freeze the execution of R if you are doing some heavy processing and you don’t want it to run every time someone loads your site. Additionally, it looks like you can pair Observable with Shiny Reactives. Leveraging a Shiny server with an Observable front end seems super convenient for any cases where Observable alone can’t handle the workload. I haven’t used it enough to give my full thoughts on it yet, but so far it has been extremely promising development for my workflow.

NBA Stats Exploration

Scott Franz — Thu, 03 Mar 2022 05:00:00 GMT

Introduction

Hello! This first chart explores all of the seasons in this dataset that I found on Kaggle. I organized the stats by team and you can pick whichever statistic and season(s) you would like to compare. If you mouseover the dots you can see the player’s name. If you go back far enough you can even see some old teams.

Code

Your Life in Weeks

Scott Franz — Tue, 22 Feb 2022 05:00:00 GMT

Introduction

Here is a little existential angst for your . Each red box is a week of your life. Each gray box in addition is a week left in an average individual’s lifetime in the US.

Code

The Gilpatrick

Weather In My Life

Salvador Dahlia

Introduction

Dahlia Garden

Google Sheets in Observable

Introduction

Read a Google Sheet

Recipe Web Scraping in Observable

Introduction

How it Works

UK Pet Cats

Introduction

Data Processing

R TidyCensus with Observable

Introduction

TidyCensus examples

State Median Ages

Observable Plot

Hello, Observable!

Introduction

Tutorials

Imports

Inputs

Data Wrangling (and Analysis)

Sharing Your Work

NBA Stats Exploration

Introduction

Your Life in Weeks

Introduction