Exploration in LangChain and Agentic Tools

i’m tinkering around with trying to understand agentic AI and tool-use.

What I would like to be able to do is have the AI read a Google Sheet and use Tools to build multiple emails for me.

Hours 1-2

I read that LangChain is a framework for creating Agents. I presume it’s free? I followed these instructions to get a local LLM installed on my laptop. These instructions said to use Ollama so that’s the path I’m heading down. https://js.langchain.com/docs/tutorials/local_rag

My laptop was able to read the example document. That’s a win. I have a Macbook Air M3 with 16GB of RAM that I bought in September 2024 for about $1,400 USD. In other words — a firmly middle-of-the-road new Mac. I presume it’s going to run slow as molasses?

Hours 3

Back to tools. The Ollama Blog announced tool support in July 2024 which led me to this code example. Now I think we’re in business!

https://github.com/ollama/ollama-python/blob/main/examples/tools/main.py

Reading the docs for Llama 3.2B, it appears this new version is OK for tools and designed to run on less powerful machines. I don’t know what 1B or 3B means. I guess 3B words indexed?

Ollama blog notes were very helpful in explaining the steps necessary to index my documents so that AI will be able to do something with the text. It involves making embeddings by using an indexer and a vector store. I don’t know what that means. Hopefully I don’t need to know? https://ollama.com/blog/llms-in-obsidian

Hour 4

So much information my mind is spinning right now.

This agent tutorial is exactly what I’d like to figure out. https://python.langchain.com/docs/tutorials/agents

LangChain has a lot of community-contributed tools. Most interesting tools I’m seeing right now are these tools:

  • Twilio – Can I get the Agent to create a message for me? Limited to SMS and WhatsApp messages so not quite what I’m looking for.
  • Playwright Browser Toolkit – maybe too advanced for a newbie. Requests may be better.
  • Requests interesting for scraping sites. I wonder if it could help me categorize blog content.
  • RSS Loader – could it help me curate stories?
  • Obsidian – for on-my-computer notes

Hour 5

https://python.langchain.com/docs/how_to/tool_calling

lost in the ether of wrestling with getting this to call a feed and find relevant articles. Realizing now that reading content has certainly been done-to-death. How can I make this do something more interesting for me?

Hour 6

struggled with having Ollama read and summarize my Flipboard magazine. this has been a waste of an hour

Hour 7

Back to the original idea of why did I want agent AI in the first place?

Goal: I want to increase sales. General one-message-for-all approach is not working. I want to split people up into groups/clusters of interests.

I also want to find a pattern of behavior that occurs before people buy.

It has been suggested to me that a Graph Neural Network (GNN) is a good way to identify this behavior. GNN is a whole different rabbit hole to look into. My friend told me to look at JanusGraph

At this point, I’m not quite sure how AI and Graphs fits into the picture here. The insight will come eventually.

Google Sheet to Figma Plugin

At Calm, we rely heavily on Figma to generate marketing copy proofs for copywriters and other marketing folks to see their text visually in an email mockup.

Typically the process might go as follows:

  1. Using an existing template, our designer mocks up a series of emails containing rough draft.
  2. 3-email series x4 population segments = [12 emails x 15 minutes per mockup = 3 hours]
  3. He shares with the copywriters.
  4. Copywriters eyeball the design and write new copy to fit the theme of the year. They don’t have Figma seats, so they write their revisions as Figma comments.
  5. The Designer now transfers the text edits from the Figma comments into the Figma text nodes. [5 minutes per email = 1 hour]
  6. Designer shares the Figma file back to copywriters. Every round of back-and-forth is 1 hour.
  7. Now we have to transfer the text from the Figma into email templates! [15 minutes per email = 3 hours]

The total effort is about 7 hours for a 12-email campaign or 35 minutes per email.

Our work process is overly reliant on having access to a Figma license. The process served us well for several years but we came to a crossroads where there are too many back-and-forths between design and copywriting and it was taking up a lot of time.

In effort to streamline or workload, I created a Figma plugin to pull text from a Google sheet

In Figma,

The Figma mockups have a specific # of text layers. I give each text layer a unique name:

Section1_Header
Section1_Body
Section1_Link

In Google Sheet,

Map a Google sheet row-for-row to each text layer.

Figma Layer NameGoogle Sheet AGoogle Sheet B
###.Email1Email1Email2
Section1_HeaderMy first newsletterMy second newsletter
Section1_BodyI was vacationing in Anaheim with family…I was on my way to Albuquerque on business…
Section1_CTAREAD MOREREAD MORE
Section2_HeaderEmployee of the monthEmployee of the month
Section2_BodyEdward Heinz wins for achievement in…Eliza Doolittle wins for achievement in…

Key difference from other plugins

I setup a Google App Sheets API client so that I could share any Google sheet with a programatically-generated email address (Google provides the email address). I can share any private Sheets document with the special email address rather than making the Sheet publicly viewable. I do not like the idea of a public Sheet.

How we use Google Sheets to manage email production

Generated with Speechify Studio


Hello Reforge team. A quick note: I’m playing around with an AI voice generator to make narrating this video faster!

This is how Calm uses Google Sheets to build and track emails.

I’ve created a simplified version of our sheet specifically for Reforge. In practice, Calm’s sheets have 70 or 80 rows instead of the 28 rows in this example.

At right you can see a basic newsletter. We break up each piece of the newsletter into chunks that we map to blocks of rows on the Google Sheet.

We replicate this google sheet column multiple times to represent different versions. In this example I’ve layed out six email newsletters. It could be 6 months or 6 weeks. whatever your newsletter cadence.

We also use the spreadsheets to manage newsletter translations for different languages.

Each section of the spreadsheet maps to a section of the email. Starting with the hero section on the email mockup, we have a header, a paragraph body,  and a call to action link. Each of those elements has a corresponding line on the spreadsheet.

We also have three sections that look the same. Therefore we have  three corresponding sections on the spreadsheet.

Finally, we finish with the email section on the bottom.

Now that the copy has a defined structure and pattern, we can interpret the email easily programmatically.

I connected the spreadsheet to a Google Apps Script that shows a basic preview of the newsletter to make the email easier to review and check for errors.

Notes on updating User Profile via Iterable webhooks

Iterable journeys have two webhook nodes to choose from so I was trying to figure out which node will make an outbound call to an API, receive data, and then write data back into the User Profile.

The “Journey webhook” node will not write data to the user profile.

  • This is an outbound-only HTTP request.
  • We can use the webhook to tell external certain things about the user that Iterable knows. But the webhook itself is fire-and-forget.

We are forced to use the “User Profile update” node

  • Iterable automatically stuffs a ?email=<email address> parameter on every request (security prob?)
  • This node allows for an Authorization header.
    • Screenshot shows Authorization: abcdefg
    • We can do Authorization: Bearer <token> to do basic auth instead of authorization token?

Example request header from User Profile update via webhook

GET /?email=exampleemail@foobar.com HTTP/1.1
Host: <HOST>
User-Agent: AHC/2.1
Accept: */*
Authorization: <STRING>
X-Forwarded-For: <IP ADDRESS>
X-Forwarded-Host: <HOST>
X-Forwarded-Proto: https
Accept-Encoding: gzip

A thoughtful UX Teardown on my work

Calm was the subject of a email UX teardown recently which was well-written and had several interesting insights. The obvious first reaction to this post was “oh no, we’re horrible” but the reality is she articulates several weaknesses that the team has been aware of.


We missed an opportunity for personalization

Complaint – She was a happy user churning out of the product. When she churned we hit her with a generic urgent messaging that others recognized not to be soothing. Why are we talking to her like a beginner?

What is the root of the complaint? We don’t distinguish “happy churn” from other types of churn. (Note to self: what are the other types of churn?) Our churned-user experience is certainly under-appreciated.

This user has multiple email addresses: a paid subscription email and a trial churned email. Presumably she used the trial churn email to conduct her research.

The paid subscription email address was unsubscribed from marketing emails. She received transactional emails only. This underlines that we should sneak in some more light marketing into our transactional emails.

What technology do we need to solve this? Lots of people have multiple email addresses. It’s not a technical issue. However we could benefit from having consistent time-based email cadence (use as a default) augmented by an event-based cadence when users are well engaged.

Is there a barrier to fixing it? No barriers. We have the capability to look at the user’s engagement within the last several months or weeks to separate out an engaged user from unengaged users — at all times!


Inconsistent email cadence

Complaint – she experienced an email dead zone where she didn’t see anything messaging from us for weeks (months?)

What is the root of the complaint? There was a dead zone for several weeks after churn from trial. She eventually got enrolled into a monthly email but monthly may not be enough frequency to maintain engagement.

What technology do we need to solve this? Increase production

Is there a barrier to fixing it? Emails are too difficult to specify right now.


Off-brand copywriting

Complaint – The author and several LinkedIn commenters perceived the emails to be aggressive and ironically not “calming”

What is the root of the complaint? She received several sale-oriented headlines during a large revenue-producing campaign.

What technology do we need to solve this? No new technology necessary to write copy. Slow down and work cross-functionally to capture critique from marketing & content teams to get on the same page for user-facing communication.

Is there a barrier to fixing it? We are not trained copywriters. However, we can learn to write in an on-brand voice. That is a marketable skill that can be augmented with AI.

Referring back to the personalization problem, we could come up with alternate subject lines for well-engaged users who do not need urgency-oriented messaging.

Big-time copy-writing failure!

I had to add a disclaimer block to some messaging so that users don’t mistakenly look for coupons on Apple or Android.

I suspect this line of text caused some of the drop.

“This discount is valid only for subscriptions purchased through [domain name]. Offer not valid for subscriptions purchased in the Apple Store or Google Play Store.”

Notes from Iterable Activate 2022 in San Francisco

I attended Activate in San Francisco held at the Marriott Marquis. It was a fine time. I gravitated toward high-level strategic talks in-person. I take a look

Most insightful talk: Women Language + Power and Standard Communication by Susannah Baldwin

When I walked into the room vaguely labeled “professional development” session I was late so I hurriedly took a seat at the side of the room. After I got settled and composed I realized I was one of two men in a room of one hundred women. Ooops! I felt way out of place but I stayed in the room and happy I’m happy I did.

The speaker’s point was that women are taught at an early age to speak in a soft or indirect manner. To reach the upper echelons of But that’s not

Many Useful takeaways : Find Your Target: Reaching Different Personality Types Without Flooding Your Marketing Channels by Tim Hemingway, Havas Media

Broadening my horizons: Presenting with Impact by Susannah Baldwin

Reforge notes: Retention & Engagement

Rough notes while I go through the Reforge Summer 2022 program on Retention & Engagement.

4 Strategies to increase ENGAGEMENT

  1. Add Use Cases
    1. Uber
      1. start with going to airport
      2. add going out with friends
      3. add commuting to work
    2. JIRA
      1. start with bug tracking
      2. add product releases
      3. add project management
    3. Calm
      1. listen to sleep stories
      2. add journaling
      3. add coaching
  2. Increase usage frequency
    1. Metric is weekly active user (all are weekly actives)
      1. 1x/week
      2. 4x/week
      3. multiple times/day?
  3. Increase feature usage
    1. Expand breadth of features does individual use.
      1. User 1: Sleep stories & music
      2. User 2: music & Daily Calm & Journal
  4. Increase intensity of usage
    1. amount of time per use? (listen to Calm music 5hrs/session)
    2. $ spent
    3. # actions per use (?)

Add Use Cases

In what ways can we expand the number of uses for subscription apps that I am trying to improve?

Calm use cases

  1. Bed time
  2. Focus during work/study
  3. Leisure during commute
  4. Consistent daily mindful practice (advanced)
  5. (expand to) measuring personal growth?
  6. (expand to) handling anxiety in-the-moment

TTS Inc. use cases

  1. Leisure reading fiction
  2. Reading textbooks
  3. Reading work documents
  4. Proof-reading, editing, review
  5. (expand to) studying, taking notes on subjects.
  6. (expand to)

Black Friday 2021 notes

I identified a profitable segment combining app usage and sales data from the previous year

I found a particularly profitable segment of users by looking cross-referencing app usage and purchase behavior. For the Black Friday 2020 campaign we sold far more high ticket items to people who had used the app for 12 consecutive months (the size of the circles represent # plans purchased). It helped us send our most active users our best deal. (High price but high discount)

Bigger circle = more sales. X-axis is consecutive months of activity.

Spreadsheet command center

To keep track of the 60 campaigns I built out a spreadsheet to track the different offers and email template modules to include in the campaign. Different segments received personalized offers (pictured) based on analysis from our Data Science team that found the optimal offer according revenue per email sent.

Each cohort, the day of the campaign, and the sales modules to be offered to the recipients

Landing page switch led to 40% conversion improvement

Toward the end of the campaign I created an A-B test for the email click’s landing page.

You should notice that the original page is a very simple credit card page. Because we were advertising a Black Friday time-limited sale our logic was to collect payment as quickly as possible.

You will notice that Treatment version has a lot more information. It has a big hero image, some science facts, a product comparison checklist in the middle, and testimonials on the bottom. Our product & engineering team had refined and tested this landing page for peak performance earlier in the year but we had never tested these two variants against each other within the context of a holiday sale.

The A-B test showed 50%+ sales conversion improvement over the 3-day observation period for the campaign. I wished I had tested these two landing pages earlier in the campaign!

I believe that part of the reason the page performed so much better is that we only sent this page to users who never subscribed before. These people barely know what Calm is so I believe that having the extra information on the page properly educated the returning inactive users and clearly did a better job at explaining the value of the Black Friday deal that we were offering them.

Next year I’ll be sure to compare the two variants across broader audience (churned, current paid, etc.)

Original
Treatment

Treatment landing page (green line) outperformed for duration of the sale
52% improvement & statistically significant