How I manage my Iterable Catalogs using Google Sheets

Calm’s dynamic emails leverage the Iterable Catalog feature extensively to keep campaigns evergreen and easy to maintain. I use it for a variety of use cases:

  • Email localization strings. Calm serves content in English, French, Spanish, Japanese, Korean, Portuguese, and German.
  • Content calendars. Calm has a variety of content that releases on a regular cadence including the Daily Calm and the Daily Trip.
  • Follow-up lessons. For special programs we send messages that remind listeners of key takeaways, lessons, or review material.
  • Top N lists. The most recent Catalog I added allows others to curate lists of top/new/hot/trending content.

I have been most tickled with my most recent effort to store Top-N lists (Top 10 newest music, Top 10 most popular, Top 5 fill-in-the-blank) in our Catalog because it streamlines our process for creating ad-hoc special announcement emails. Creating this did not require a lot of code and makes me wish I had thought of doing this earlier.

Google Sheet as Content Management System (CMS)

Using a Google Sheet shared with my colleagues allows us to have a mix of curated and automated feeds controlled by different:

Automated feeds:

  • the newest Sleep Stories for each language
  • the newest Music releases for each language
  • the newest Meditations for each language
  • 3 content types * 7 languages = 21 individual feeds

Curated feeds

  • featured English new releases
  • English-language blog content
  • special blocks for other departments to provide content (b2b, marketing, product, engineering)
The Sheet Tab name “konewSleep” becomes the name of the data feed in the Catalog. The sheet columns headers are key names of each object.

The Catalog

See how the Google Sheet columns map to JSON key names.

Retrieve and Render Iterable Catalog Data

Using a known key, such as ennewSleep I am able to retrieve the appropriate content for Korean language new Sleep Stories.

{{!-- look up new Sleep Stories in English (en) --}}
{{#catalog "Datafeeds" "ennewSleep" as |datafeed|}}
{{#lookup datafeed "programs" as |prog|}}
<h1>{{{ prog.title }}}</h1>
<p>{{{ prog.description }}}</p>
<hr />
{{/lookup}}
{{/catalog}}

Basic HTML result would be

<h1>Sleep story title 1</h1>
<p>Description of story 1</p>
<hr />
<h1>Sleep story title 2</h1>
<p>Description of story 2</p>
<hr />

Addendum: Roadblocks along the way

My initial intent for this project was to lean on Iterable Datafeeds and Google App Scripts to provide my dynamic data. I created a Google Apps Script to create a web app that would grab some API data, package it up into a more email-friendly rendering payload, and return JSON. I ran into 2 major problems while trying to set this up which eventually led to my Catalog solution.

First, I found out that I cannot change the user-agent HTTP header when using the UrlFetchApp library. UrlFetchApp sends something like Google/app-script-client when doing its HTTP fetch. I needed control of the user-agent header to pull the different language content from the API. My solution to this was to set up a free Netlify account that would proxy a request Google -> Netlify -> Calm. I created the proxy as a serverless function. This only took a couple of hours thanks to Netlify for its very developer-friendly deployment tools.

My second problem is that Google App Script only serves a few requests per second (expected). I was expecting to lean on the caching option in the Iterable datafeed fetcher. My theory was Iterable would hit my endpoint once and cache the response for 1 hour as described in their documentation. Despite several attempts at configuring the templates and datafeeds I could not get Iterable to cache my feeds properly. During campaign send time Iterable would hit my Google App Script hundreds of times per second and fail out due to rate limiting problems.

Figuring that the caching roadblock would be unsolvable on my own, I decided to use the Iterable Catalog solution instead. The Catalogs are useful because I don’t have to worry about maintaining uptime on the services but not as nimble as a API solution because I have to constantly sync my Google Sheet to the Catalog(s).

Do Gmail promotion cards improve email performance?

Long story short: gmail promotion cards did not have statistically significant positive impact on our Black Friday email campaign.

My first year being on the retailer’s side of Black Friday was 2019. It was a harrowing experience due to lack of experience and being new to my job. We hectically threw together our Black Friday campaigns around 3 weeks before Thanksgiving: copy writing, design, QA, pre-testing, and identifying segments for the biggest revenue week of the year all done as quickly as possible. This is not an experience I recommend. In 2019 I knew about Promotion Cards but since we had a very short timeline to work on the emails and I didn’t have time to test the Promotion Cards. In 2020, we started planning in early October. Unfortunately I forgot about Promotion Cards until the Black Friday week. Remembering that I wanted to test the cards in 2019 I scrambled to incorporate them into a handful of our early campaigns sent on Tuesday of Thanksgiving week to see if the cards would be able to measurably increase sales conversion rates for our biggest email sends running from Black Friday to Cyber Monday.

I tested the promo cards on 4 different emails with volume around 4 million people. I looked primarily at 2 metrics: open rates and send-to-purchase conversion. I know open rate is a vanity metric — call me vain! Promotion Card’s best performance garnered +5% better open rates (stat sig) but send-to-purchase conversions were -1.2% worse than control (not stat sig). The Promotion Cards lagged in both metrics in other campaigns as well and could not achieve statistical significance for purchase conversion in any test. Since I couldn’t get stat sig on the sales I decided to take the easy road: no Promotion Cards for the remainder of Black Friday 2020 campaigns.


What are Promotion Cards?

Several years ago Gmail came up with the “Promotions” tab to automatically separate out marketing emails from people’s inboxes and into a separate folder. Marketers freaked out because the Promotions tab, essetially, is a Spam-lite inbox. All the marketing riffraff ends up lost in the outer limits email equivalent of the axe-murderer-creepy cluttered basement or the spider-filled attic. The Promotions tab on Mobile devices is especially challenging because the tab lies within a menu drawer.

I find out about new GMail features from the Product Managers that work at Google. Both at Flipboard and Calm, the Gmail account management team has reached out to test new features: I was able to use promotion cards and AMP for email as part of the Gmail beta programs. Is it an exclusive club? I don’t know. But it does feel kind of cool to test new features in the world’s most popular email service before most people.

In this case I think the kindly folks at Gmail came up with Promotion Cards annotation to give emails extra visibility on mobile clients to make up for banishing marketing emails into the Promotions tab. You can see a Promotion Card in action below.

As you can see above, Gmail added a bold “Promotions” section at the top of the Primary Inbox. That space contains a couple of teasers for the Promotions Tab (buybuy Baby, Carter’s, etc.).

Observations: increased open rate in 3 tests, decreased in 1. The case where promo code loss may be due to poor subject line choice.

Observation: less clicks

Observation: sales inconclusive.

Result: slight edge to promo cards but it will not change your business. Take it or leave it.

How to set up your email domains and SPF records

What is SPF (Sender Policy Framework)?

SPF is a DNS TXT entry for your domains and subdomains that list the services and individual IP addresses that you use to send emails.

v=spf1 include:_spf.google.com include:_something.amazonaws.com ~all

Hypothetical SPF record. This says that I use Google and AWS to send emails

SPF helps prevent email address spoofing because ISPs can look up the TXT record and compare with the email headers to verify that I acknowledge that I want Google or AWS to send emails for me.

Here is a larger SPF record. Note that in the block above I have 4 include: lines. That means I have 4 SPF records for my domain.

# infoentropy.com
v=spf1
include:_spf.google.com         # 1
include:u123456.wl.sendgrid.net # 2
include:other.service.com       # 3
include:other.service2.com      # 4
-all

Now, a problem: according to specificiations, each domain can have a maximum of 10 SPF records in each DNS TXT record. If you have more than 10 SPF records the ISP could ignore the entire DNS record or only look at the first 10 while ignoring the rest.

If I used my root domain for all email services, I would reach the 10-record limit quickly. Look at the table below — a medium-sized company could end up using many different SaaS platforms, each of which vies for space on the root domain.

Software serviceDepartmentwhat do they send?
GmailEveryoneall emails
MailchimpMarketingEmail blasts
Blog platform (WordPress, Squarespace, etc)MarketingBlog announcements
CRM (Salesforce, Hubspot, etc.)SalesCorrespondence with customers
e-commerce (Squarespace, Magento, etc.)SalesPurchase receipts
Customer Support (Zendesk, etc.)SupportHelp tickets
Surveys (SurveyMonkey, etc)MiscellaneousProduct survey, customer satisfaction survey, etc.
Transactional emails (Mailgun, Sendgrid, Sparkpost, AWS SES)EngineeringThings related to your apps
Engineering Internal Services (Jenkins, datadog, github)EngineeringStuff engineers look at
Legal things (DocuSign, etc)Legal
Finance (ADP, Carta, etc)Finance
Recruiting emailsHR
More stuff! JIRA, ASANA, Slack, BOX, DropboxEveryone
Various outbound emails & vendors that might use your domain name

This is 13 SPF records trying that would need SPF records. However, not all of these services necessarily need to be on the root infoentropy.com domain. I can have more than 10 SPF records by using multiple subdomains. Each subdomain can have 10 records.

Use multiple email subdomain names to stay under the SPF 10-record limit.

My proposed solution is to, loosely, divide my email domains according to business function: sales, customer support, marketing/bulk email and product, and internal.

Doing so will (1) give recipients have some clue of who is sending the email and (2) allow you to identify vendor deliverability issues. For shared SaaS services you might include that sender in more than one subdomain SPF record. SurveyMonkey, for example, might be used for more than one business function.

  • Sales: biz.infoentropy.com
    • reports@biz.infoentropy.com
    • Sales
      • hubspot
      • salesforce
      • squarespace
      • SurveyMonkey*
  • Help: help.infoentropy.com
    • support@help.infoentropy.com
    • Support
      • zendesk
    • Transactional (password reset, billing?)
      • AWS SES*
  • Marketing: email.infoentropy.com
    • deals@email.infoentropy.com
    • High volume bulk email
      • Iterable
      • Mailchimp
      • Pardot
      • Marketo
    • Surveys
      • SurveyMonkey*
  • Product: app.infoentropy.com
    • notifications@app.infoentropy.com
    • Transactional (you did something in the app!)
      • Sendgrid/Mailgun/AWS SES*
    • Notifications (X sent you a message)
      • Sendgrid/Mailgun/AWS SES*
  • Internal/corporate: corp.infoentropy.com
    • jeff@corp.infoentropy.com
    • Gmail/Outlook 365?

*Single service used on multiple subdomains.

In this situation I need 5 separate SPF TXT records, 1 for each domain. Each domain lists the specific services that I use to send emails.

# help.infoentropy.com
v=spf1
include: _spf.zendesk.com        # 1 Zendesk
include: _spf.amazonaws.com      # 2 AWS
-all

In this case I only use 2 services to send something@help.infoentropy.com emails. If I tried to send with the address jeff@help.infoentropy.com via Mailchimp, the verification would fail DMARC and the email would end up in the SPAM folder.

How to make your logo visible in the inbox with a BIMI record

I just heard of BIMI for the first time a few weeks ago. Most of the resources I can find are trying to sell DMARC management services. While these services are useful and valuable I was trying to figure out some rules of thumb to self manage my own DNS with regard to DMARC/DKIM/SPF/BIMI compliance.

First thing to note is that in order to get your email logo on the inbox, you need to comply with BIMI standards. That means you need to:

  • Create a BIMI record
  • Configure domain names
  • Configure DKIM & SPF
  • Create your DMARC rules
  • Monitor your DMARC
  • Lock down your DMARC

Set up your BIMI record

I have this step early in the blog because you can set it’s the simplest step. You won’t get the snazzy BIMI logo until you complete all of the steps.

Create an SVG version of your logo and upload it somewhere on your CDN or wherever you serve your images. You can probably put it in the same directory as the ubiquitous favicon.ico file.

Add a TXT record to your DNS that points to the newly uploaded SVG file.

default._bimi.acme.com

v=BIMI1; l=https://cdn.acme.com/_email/logos/acme-icon.svg

Configure your domain names

If you’re putting together your email system from scratch, (i.e. you work for a 1-10 person startup and you happen to be the devops guy) see other post for guidance on how to Setup your email domains and SPF records. This is a hairy process that IT people need to be involved with because it requires mucking with DNS and planning out how you will send emails in the future. Do it right early!

Configure DKIM & SPF for your domains.

In order to pass DMARC both your SPF and DKIM need to validate.

The SPF record means that you added the 3rd party services to your DNS, as described above.

The DKIM is a signature key that you share with your email sending service(s). These services will add the DKIM signature to the email headers of every message they send for you so that recipient ISPs can verify that emails that have your domain on it are coming from you.

In order to qualify for BIMI, you need to make sure the SPF and DKIM are “aligned“. For example, f you use Sendgrid, you should have sengrid in your SPF and sendgrid in the DKIM signature.

Create your DMARC rules

In order for your logo to show up on emails, your DMARC must be set to “quarantine” or “reject”.

Now, monitor the results of your DMARC (SPF+DKIM) configuration

Using a service such as Valimail, you can get reports of which domains and IP addresses are attempting to send mail using your domain names. If you recognize IP addresses or domains that are “failing” that should not be, then you need to check configuration of those services. In keeping with the Sendgrid example, if you are using them as your provider and you see a sendgrid.net in your “Mostly failing IPs” box then something is wrong with your SPF/DKIM DNS configuration.

Definitely a phish/spam/spoofer

Once DMARC you have quarantine on, and your SPF and DKIM are “aligned” your BIMI logo might start showing up!

I say “might” because not all email providers suppport the the BIMI standard. Anecdotally, as of 2020, I do believe Hotmail, Yahoo mail have logos in emails. Gmail has its own brand of JSON-LD syntax to jam logos into the inbox

Iterable recipe: inject data from a custom event into userProfile for email template

An email I’ve been working on needs to refer back to information from a previous event.

... TEMPLATE ...

Hi {{firstName}}, 

Thank you for watching {{title of movie I watched}}!

... RENDERED ...

Hi Jeff, 

Thank you for watching STRANGER THINGS!

The problem here is that the I wanted to reference the title of a video that was watched in the past within a blast email. This is not possible without saving the information somewhere in Iterable. In my case I decided to store the data on the user’s profile.

To save data from an event onto the user profile I use a workflow and the Change Contact Field node.

The event data might look like this:

{
    "eventName": "Video : Watched",
    "eventType": "customEvent",
    "email": "<EMAIL_ADDRESS>",
    "dataFields": {
        "video_title": "Stranger Things",
    }
}
  1. Create workflow
  2. Add node “change contact field”
  3. In the CHANGE CONTACT FIELD node define the data I want to save in JSON format.
{"video_title":"{{video_title}}"}
## (You should use better field names than this) ##

Now when the event comes through Iterable will replace the template variable with the new value on the user profile and thus making it accessible on any email template.

Here I can put the saved_program_title into the subject line or email template body using the Handlebars.

Limit number of php-cgi processes spawned by nginx

Here’s how to Mine was around here somewhere.

File was here

sudo nano /usr/bin/php-fastcgi

Original file looked something like this.

#!/bin/bash
/usr/bin/spawn-fcgi -a 127.0.0.1 -p 10005 -u www-data -g www-data -f /usr/bin/php-cgi

Added -C argument to enforce a limit.

#!/bin/bash
/usr/bin/spawn-fcgi -C 2 -a 127.0.0.1 -p 10005 -u www-data -g www-data -f /usr/bin/php-cgi

http://forum.slicehost.com/index.php?p=/discussion/3671/limit-number-of-php-cgi-processes-nginx/p1

I just got a notice from my VPS host that my box was using too much swap and therefore impacting other users on the machine. As such, my provider did a hard reset on my instance which summarily stopped my web app (I don’t start the app on startup. I should.)

I just upgraded my server a few versions up the Ubuntu chain because of the heartbleed SSL bug so obviously it was something related to the upgrade. I’m not super savvy with figuring out which components changed so I figured if I could just cut back on the memory usage things wouldn’t go to swap. My server is running an app and a couple of old Drupal blogs that don’t get much traffic but I like to keep around for nostalgia. So, I figured I could sacrifice substantial performance on the blogs.

Things I changed to hopefully save memory:
/etc/mysql/my.cnf
* reduce max_connections to 50 from Not specified.

/etc/php/cgi/php.ini
* reduce memory_limit from 128M to 64M

/usr/bin/php5-cgi
* add “-C” flag to constrain number of processes to 3. (default was 5)

The php5-cgi change by far had the largest affect because each php5-cgi process was using 25MB per thread (~125MB, or 50% of my 256MB VPS).

HTML Email: Image height is 1px high for Gmail in IE10

Lately I have had to redo some of the HTML emails. We had an outside contractor do most of the work and he did a fantastic job. However, I was noticing that under a very specific condition, the email images were not rendering properly in desktop Gmail on Internet Explorer 10. I have no idea how many people are using GMail+IE10 but since this is our first real contact with the user, I though it would be important to ensure the best user experience possible. Broken images are not a good experience.

Here is the problem:
Email image poorly rendered

Old HTML

<img class="imageScale" 
style="display: block; width: 550px; height:auto;" 
width="550" height="auto" src="{{ img_url }}" 
border="0" />

New HTML

<img class="imageScale" 
style="display: block; width: 550px;" src="{{ img_url }}" 
border="0" />

Final outcome:
email-good

Why does it fix it?

I came across a couple of quirks at play in this porblem.

First, I learned that GMail automatically converts CSS height attribute to min-height with reckless abandon.

Second, the original HTML IMG tags have height:auto. With GMail’s reckles height conversion it becomes min-height:auto which essentially means 0px or 1px.

To solve, I removed height attributes on the img tags. It turns out that all of the browsers will automatically just render the image at full size of the parent container. In this case we have a series of nested tables that set maximum width of the parent to about 550 and the minimum width is 100% of screen width.

Working on a real iOS app.

As a personal endeavor I’m trying to make an iOS app. Making a native app for iOS has been on my to-do list for at least 2-3 years but I could never figure out the Objective C language and I haven’t done a lot of object-oriented programming. With newfound determination I have been trying to learn this stuff when I have downtime at work. I’ve had a lot of downtime lately so my learning progress has been good!

Things I’ve been skimming through:

Stanford undergraduate course on iOS through iTunes U (Not Recommended)

I started off trying to learn iOS by following Stanford University courseware via iTunes U. I thought “hey, Stanford is a great school. This should inspire me.” Big mistake — I despised computer science lectures when I was in college 10 years ago and evidently I still hate them. The first few lectures were long-winded and impractical for the purposes of building a simple app.

Big Nerd Ranch Guide (2012)

Big Nerd Ranch guide proved useful for understanding some fundamentals of Objective C. It’s densely written and some of the chapters are difficult to parse in my head. This is slightly easier to deal with than a boring iTunes U lecture but it has heavy reliance on extending code in preceding chapters so you cannot jump around from chapter-to-chapter to pick what you need. I would prefer a just-in-time piecemeal approach.

Beginning iOS 6 Development (Apress, Jan 2013)

This book is written more simply than Big Nerd Ranch. I ran through the first few chapters and finally understood the workflow of using XCode to create the UI by linking buttons to actions and code. Understanding XCode was a major breakthrough for me. The first handful of chapters are useful because the examples are step-by-step and do not depend heavily on using code from previous examples.

Apple’s documentation (horrible)

Large swaths of Apple’s iOS documentation and example code is outdated. However, I slogged through some of the examples to help understand some of the components that are not discussed in the other 2 books.

Various web tutorials

As you would expect this day in age, there are tons of video tutorials on the web, YouTube and other content authors who blog about learning iOS on their own. These are often helpful. The problem with these tutorials is that many of them are for older versions of XCode and iOS. Working with iOS has apparently changed quite substantially over the years because the code samples can often look completely different from modern stuff. I’m sure the recent release of iOS 7 makes this truer than ever.

Stack Overflow

Whenever I run into  issues Stack Overflow has the answer 97% of the time. Long live SO!

Make a media player – http://www.codigator.com/tutorials/how-to-make-a-custom-ios-music-player/

Understanding the design of the Angular JS framework

AngularJS Logo

 

I’ve been trying to figure out Angular JS for a little while but I don’t have a deep understanding of software engineering patterns or practices. My theory is that I haven’t been able to figure out the framework because I don’t understand the theory behind Angular’s design. The documentation on the website is thorough yet written like an engineering manual. It uses a lot of CS jargon that will explode your brain into a million pieces.

Here are some resources that helped me understand the “why” of Angular.

Angular JS Directives

  • AngularJS directives and the computer science of JavaScript [Adobe]
  • a directive is a way to  provide some custom behavior HTML. Commonly at the element or attribute level
    • <mydirectivelement> <span mydirectivename="">.
  • Directives are like jQuery plugins – they are reusable UI components/widgets. Examples
    • autocomplete search box
    • dropdown menu
    • header toolbars
  • Directives can be used as building blocks to create a UI. Use directives liberally.
  • Separate your view code out using templateUrl instead of template inline if your HTML is going to be more than just a single line.
  • When creating an reusable component use an ‘isolate‘ scope so that the component will not read or modify data of the parent scope.
  • ‘isolate’ scope will isolate (verb) the component from others of the same type. Thus they will not share the same instance variables.
  • question… Do ‘isolate’ and ‘transclude’ contradict each other?
    • Answer: No. Isolate and Transclude are unrelated. Transclude allows you to have HTML within your directive that will be preserved rather than being blown away when Angular replaces the directive’s placeholder with the actual rendered template code.
    • Angular JS Transclusion Basics video tutorial [egghead.io]

The Angular  documentation for directives is incredibly hard to understand. I do have a CS degree and I don’t understand half the terminology in it. Probably because I did poorly in school but that’s a story for another day.

Angular Service vs Factory vs Provider?

  • Angular service or factory? [iffycan]
    • service and factory are convenience functions that result in a provider
  • Angular JS service vs factory vs provider [Stack Overflow]
    • service returns a function that you can use later
    • factory returns a value
    • provider is the base class of the above 2 convenience functions.
  • provider is configurable.
    • In jQuery terms it’s similar to how $.get() and $.post() are convenience functions to $.ajax(). $.ajax() can be configured to your exact specifications while the 2 subclasses are simpler to use.

Read Example Code

Angular Seed [github] is a very good starting point for your app. The Angular JS documentation website links to the source on github but the documentation does not push using the seed quite enough. I did not notice the Angular Seed for a long time while I was trying to understand how to use the framework .

Angular JS FoodMe [github] App by Igor Minar is a fully functioning Angular JS using best practices. This project is built off the Angular Seed. When you look at FoodMe and Angular Seed you will notice some commonalities. Studying FoodMe and Angular Seed together made things more salient for me.

Angular JS ToDo MVC [todomvc/github] has four variations of how Angular JS can be used to create a basic To-Do list application. I suggest looking at either the Architecture example or the Optimized example.