Colin's Blog

### Welcome!

Hi, and welcome to my blog & newsletter.

I’m a **Senior Software Engineer at Google** in Mountain View, since September 2017. I manage the Real-time Communications Abuse team within Counter-Abuse Technology. 

Beyond that, in my free time, I write quite a bit of TypeScript and React in a handful of side projects. 

In my newsletter, I write about my experiences at Google; working with cutting-edge JS frameworks; and more. 

About Me

*Papyrus is a blazing-fast, privacy-first, no-frills-attached blogging & newsletter platform. [Sign up](https://papyrus.so?utm_source=papyrus&utm_medium=colin_blogpost&utm_campaign=huginn_blogpost) and start writing posts just like this one.*


---

Productivity & automation tools can be powerful. I was delighted to come across a recent  [thread on Hacker News](https://news.ycombinator.com/item?id=21772610) discussing a supercharged automation tool: [Huginn](https://github.com/huginn/huginn). This open-source software performs automated tasks by using 'agents' to watch for 'events', and triggering 'actions' based on these events.

For example, if there's a sudden spike in discussion on Twitter with the terms "San Francisco Earthquake", Huginn can send a text to my phone. Or, if a time-sensitive flight deal is posted on one of the many deal-finding websites out there, Huginn can send me an email with the price and a link to Google Flights.

 ![](https://storage.googleapis.com/papyrus_images/beb6d3ce2c1e7917517258ffcb127122)Compared to other popular automation tools (IFTTT, Zapier), Huginn has the following benefits:

* Self-hosted & completely private
* Powerful data processing: write your own JS or use shell scripts
* Liquid templating

I wanted to go beyond just automation and introduce some organization - I wanted all notifications to be cataloged & delivered in a centralized way. A personal Slack workspace seemed like the perfect solution for this - I can have a `#flights` channel for flight deals, or a `#trending` channel for the, er, pending San Francisco emergencies.

I also wanted all of this to be free. Huginn has pretty lax runtime resource requirements (even able to run on a [Raspberry Pi](https://github.com/huginn/huginn/wiki/Running-Huginn-on-minimal-systems-with-low-RAM-&-CPU-e.g.-Raspberry-Pi), with some tweaking), so a free GCP micro tier instance was perfect for this.

## Automation Goals

Let's formalize what I specifically wanted to accomplish with Huginn. Note that this is a small subset of the things possible with Huginn - check out the project's [Github](https://github.com/huginn/huginn#here-are-some-of-the-things-that-you-can-do-with-huginn) for more inspiration.

* **Twitter notifications**: whenever keywords of interest are tweeted (such as my projects or blog), I want to get notified immediately. Whenever a spike occurs for other keywords ("San Francisco Emergency"), notify me.
* **Hacker news notifications**: whenever an article hits the frontpage discussing something I'm interested in, notify me.
* **Flight deals**: if a flight deal is posted online to one of the many websites I follow (Secret Flying, ThePointsGuy, FlyerTalk), and the flight originates from a nearby airport, notify me.
* **Product deals**: if a product I'm interested in is posted on Slickdeals, notify me.
* **Amazon price drops**: if a product I'm interested in drops below some predefined price threshold, notify me.

I want all notifications to be sent to me via a personal Slack workspace, on different channels.

## Deploying Huginn

### Prerequisites

The easiest way to [install Huginn](https://github.com/huginn/huginn/blob/master/doc/docker/install.md) is via Docker. Luckily, Google Compute Engine supports deploying Docker containers natively on a lean [container-optimized OS](https://cloud.google.com/container-optimized-os/docs).

There are a few key things we need to do in order to have a successful Huginn deploy on the f1-micro (free tier) instances.

**Enable and create a swap file**.

f1-micro instances have 614MB of memory. This is not enough to run Huginn out of the box - doing so will cause Docker to encounter `Error Code 137` [(out of memory)](https://success.docker.com/article/what-causes-a-container-to-exit-with-code-137) errors. To solve this, we need to create a swap file in the VM. Note that a swapfile will decrease the performance of Huginn - if you're interested in better performance for a price, consider deploying on a better VM.

Disk-based swap is [disabled](https://stackoverflow.com/questions/58210222/how-to-enable-swap-swapfile-on-google-container-optimized-os-on-gce) by default in container-optimized OS. To enable and set the swap file every time the VM is booted, we can use a custom startup script (shown below).

**Mount the Huginn MySQL database to a directory on the host**.

By default, Huginn creates a MySQL database inside the container. This is problematic, as the container now relies on *state*, and your database will [get deleted every Huginn upgrade](https://github.com/huginn/huginn/blob/master/docker/multi-process/README.md). We can use a volume mount to mount the database in the container to a directory in the host. Alternatively, you can mount a persistent disk and write the database to it.

### Deployment on GCP using Docker

Head over to GCP, create a new project, and create a new instance.

On the instance creation page, use the following settings:

* `f1-micro` machine type
* Check 'Deploy a container image to this VM instance'
* Container image URL is `docker.io/huginn/huginn`
* Add a Directory volume mount. The mount and host paths should be `/var/lib/mysql`

We also need to add in a statup script. This script lets us 1) enable and turn on a swap file, and 2) change permissions of the volume mount on the host. The latter is required or MySQL won't be able to start.

```bash
#! /bin/bash
sysctl vm.disk_based_swap=1
fallocate -l 2G /var/swapfile
chmod 600 /var/swapfile
mkswap /var/swapfile
swapon /var/swapfile
chmod 777 /var/lib/mysql
```

After the VM is created, head to your VM's external URL (port 3000) and you should be greeted with the default Huginn login page!

 ![](https://storage.googleapis.com/papyrus_images/1fa9a70e48ff805867a428ff2cd3d881)I suggest [reserving a static external IP](https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address) for this VM, so the IP doesn't change. You can even take it a step further and associate this to a domain name - like [automation.colinarms.com](http://automation.colinarms.com) - for ease of access.

Now, let's dive into Huginn.

## How Huginn Works: Agents & Events

At a high level, Huginn relies on two key things: agents and events. Agents are things that monitor for you and create events (possibly if some criteria is met). An example agent is an `RssAgent`, which monitors an RSS feed for new articles. Events created by the `RssAgent` can be passed to a `TriggerAgent`, which uses some regex filter to only listen to keywords of interest; and finally it emits a formatted message, perhaps to a `SlackAgent`, that finally sends a message to a Slack channel.

You can imagine how this works in practice. For the flight deals usecase, for example: we can create an `RssAgent` for the [Secret Flying RSS feed](https://www.secretflying.com/feed/). The `TriggerAgent` can listen to these events, filter for "San Francisco Airport", and the `SlackAgent` can message my `#flights` channel when this happens

Multiple agents for a single usecase can be grouped into a `Scenario` - in the above example, a `Flight Deal Scenario` would make sense.

Let's walk through this example. If you want to get started immediately, you can [download my agents](https://gist.github.com/seeARMS/103b3399f3f925fb6c366600f0bad3c6) and import them into Huginn directly.

## Creating your First Agent

### 1) Monitoring the RSS feed

On Huginn, create a new RSS Agent. Configure the following params:

* **Name** your agent something descriptive. I used "Secret Flying RSS Agent".
* **Schedule** your agent for however frequently you'd like it to check for updates. I used 30 mins.
* **Keep events** for some period of time. I used 7 days.

Use the following JSON in the options:

```
{
  "expected_update_period_in_days": "2",
  "clean": "true",
  "url": "https://www.secretflying.com/feed/"
}
```

Expected update period is the period at which Huginn should expect the agent to be updated - if it doesn't happen, the agent is considered not working.

Save your agent, and give it a manual run - you should see events populate from the underlying feed.

### 2) Filtering for nearby airports

Now, create a `TriggerAgent`. We want to filter newly posted articles for only nearby airports - in my case, San Francisco or San Jose airport.

Fill it out similar to the first agent. But, this time select your RSS Agent as this agent's `source`. Events from the RSS Agent will be fed into this.

Use the following JSON in the options:

```
{
  "expected_receive_period_in_days": "2",
  "keep_event": "false",
  "rules": [
    {
      "type": "regex",
      "value": "(sfo|san francisco|sjc|san jose)",
      "path": "title"
    }
  ],
  "message": "[Secret Flying] <a href=\"{{url}}\">{{title}}<\/a> {{description}}"
}
```

We're filtering the RSS feed's title to contain my nearby airports, and emitting a `message` with the URL, title and a description.

### 3) Notifying on Slack

Lastly, create a `SlackAgent`. After [registering a new Slack workspace](https://slack.com/get-started#/) and [creating a new Slack webhook](https://my.slack.com/services/new/incoming-webhook), set the `source` to the Trigger Agent you just created.

Put the following JSON in the options:

```
{
  "webhook_url": "https://hooks.slack.com/services/your-webhook-url",
  "channel": "#flights",
  "username": "Huginn",
  "message": "{{message}}",
  "icon": ""
}
```

Save your agent, and eventually you should begin receiving flight deals!

 ![](https://storage.googleapis.com/papyrus_images/ee7b868109ff0eb7bb8dc89ed2920083)

## Wrapping it up

This example describes a single usecase for what's possible with Huginn. If you're interested in the other usecases I described above, you can [download and import them](https://gist.github.com/seeARMS/103b3399f3f925fb6c366600f0bad3c6) into your own installation.

This post just scratches the surface of what's possible. With Huginn, let your `Agents` monitor on your behalf and free up your time for more important things.

Personal automation with Huginn using Slack, Docker and GCP

*Papyrus is a blazing-fast, privacy-first, no-frills-attached blogging & newsletter platform. [Sign up](https://papyrus.so?utm_source=papyrus&utm_medium=colin_blogpost&utm_campaign=nextjs_mismatch_blog) and start writing posts just like this one.*


---

NextJS is powerful: it brings together the best of server-side to the best of client-side. You can build powerful applications written in React, and produce a static website which has all the speed of a regular ol’ CDN-backed website.

Despite the benefits, you may sometimes hit some challenges working with both server-side rendering (SSR) and client-side hydration.

One of the more common errors is when the server-side and client-side DOM doesn’t match. NextJS might complain:

```javascript
Expected server HTML to contain a matching <div> in <div>
```

```javascript
Warning: Text content did not match. Server: "Log in" Client: "Continue to dashboard"
```

Additionally, the HTML (in particular, the DOM elements) may not render correctly when viewing the webpage.

# Why is this happening?

NextJS uses both server-side and client-side rendering together. The server serves static HTML whenever possible, and the client hydrates it to give full interactivity.

If you manipulate the DOM at any time between when the server-side HTML is served and the client-side is rendered, there will be a mismatch and React will not be able to hydrate the DOM successfully. Thus, **the first render of any page must match the initial render of the server**. This is explained more in-depth in [this Github issue.](https://github.com/vercel/next.js/discussions/17443)

I encountered this issue when relying on cookies. I was doing something like this:

```javascript
const user = Cookies.get('jwt')

if (user?.id) {
   return <span>User logged in!</span>
}

return <span>User not logged in!</span>
```

Cookies are only accessible in the browser, so when the server provides the static HTML the user would not be logged in, but when the client is rendered, the user would be logged in! Hence, there is a mismatch and NextJS will complain.

# How do I fix this?

Put all browser-only code inside `useEffect`. This ensures that the client is rendered before the DOM is updated. This means the client-side and server-side DOM will match.

> JS Tip: `useEffect` is a [React hook](https://reactjs.org/docs/hooks-effect.html) that lets you issue side effects - such as updating the DOM - after the page or component is mounted.

The above example can be refactored:

```javascript
const [isUserLoggedIn, setIsUserLoggedIn] = useState(false)
const user = Cookies.get('jwt')

useEffect(() => {
  setIsUserLoggedIn(!!user?.id)
}, [user])

if (isUserLoggedIn) {
  return <span>User logged in!</span>
}

return <span>User not logged in!</span>
```

The error goes away!

NextJS: server-side and client-side mismatch

I came across an interesting webdev bug today.

For some reason, on my site, my cursor was not working correctly. When hovering over links or buttons or elements that set `cursor: pointer`, my cursor did not turn into a pointer hand; instead, it stayed as a regular ol’ cursor.

This was odd. I didn’t change any styling that could affect this. I took to Google, and the first link gave me the right answer:

 ![](https://storage.googleapis.com/papyrus_images/0712ba4d3f727f87811f576071ed1d5c)

I saw that I was indeed running Photoshop at the time. I closed it, and my cursor began acting normally.

How strange!

If anyone can shed light as to why Photshop and other programs do this, please let me know!o

Photoshop Hijacks Your Cursor

*Papyrus is a blazing-fast, privacy-first, no-frills-attached blogging platform. [Sign up](https://papyrus.dev?utm_source=papyrus&utm_medium=blogpost&utm_campaign=unused_imports_vim_ale) and start writing posts just like this one.*


***

[ESLint](https://github.com/eslint/eslint) provides checks (and auto-fixes!) for a large list of Javascript and Typescript violations. Paired with a Vim plugin called [ALE](https://github.com/dense-analysis/ale), these fixes can run on every save.

This blogpost details how to set up ESLint and ALE in Vim, and uses a single usecase - removing unused imports & variables - as an illustration of how powerful these tools are.

### Unused Imports & Variables

At the start of a a new React project, I create many different components, and I bootstrap them by copying & pasting component files. This can quickly lead to a large mess of unused imports & variable declarations:

 ![](https://storage.googleapis.com/papyrus_images/d4746c5999e28bd839764a46e87ec096)Let’s see how we can remove them - automatically - using the power of ESLint and Vim.

### Configuring ESLint

The core solution to this problem is a handy ESLint plugin called [eslint-plugin-unused-imports](https://www.npmjs.com/package/eslint-plugin-unused-imports). (Other solutions I found online rely on TSList, which is deprecated, so an ESLint-based solution is preferred).

Let’s install ESLint, as well as the `eslint-plugin-unused-imports` plugin:

```javascript
yarn add -D eslint
yarn add -D eslint-plugin-unused-imports
```

Now, modify your `.eslintrc` to enable the plugin:

```javascript
// .eslintrc

{
	"plugins": ["unused-imports"]
}
```

Configure your ESLint rules to error on unused imports or unused variables:

```javascript
// .eslintrc

{
	"rules": {
		"no-unused-vars": "off", // or "@typescript-eslint/no-unused-vars": "off",
		"unused-imports/no-unused-imports": "error",
		"unused-imports/no-unused-vars": [
			"warn",
			{ "vars": "all", "varsIgnorePattern": "^_", "args": "after-used", "argsIgnorePattern": "^_" }
		]
	}
}
```

Now, we should see `unused-imports/no-unused-imports` detecting the unused imports (instead of no-unused-vars, as in the screenshot above):

 ![](https://storage.googleapis.com/papyrus_images/b927ecacf802db7ad5d5c9d2f8bbc057)And running `eslint --fix` should automatically remove these imports!

We can do better, though - let’s configure Vim to auto-fix on every save.

### Setting up ALE to auto-fix on save

[ALE](https://github.com/dense-analysis/ale) is a plugin for Vim and NeoVim that lints as you type, and auto-fixes syntax and semantic errors. It uses ESLint (and many other extensible tools) underneath to perform the actual fixing logic.

Install ALE by [following the instructions](https://github.com/dense-analysis/ale#3-installation) and using Pathogen, Vundle, Plug, or one of the other installation options.

After installing ALE, edit your `.vimrc` file set two options:

1. `let g:ale_fixers` -  Run ESLint as a fixer in your relevant filetypes, which lets ALE tap into ESLint for fixing files
2. `let g:ale_fix_on_save` - Automatically run all fixers on save

It’ll look something like this:

```javascript
// .vimrc

let g:ale_fix_on_save = 1
let g:ale_fixers = {
\   'javascript': ['eslint'],
\   'typescript': ['eslint'],
\   'typescriptreact': ['eslint'],
\}
```

Now, every time you save a `js`, `ts` or `tsx` file, `eslint --fix` will automatically run and clean up your imports & unused variables!