Black Friday 2021 notes

I identified a profitable segment combining app usage and sales data from the previous year

I found a particularly profitable segment of users by looking cross-referencing app usage and purchase behavior. For the Black Friday 2020 campaign we sold far more high ticket items to people who had used the app for 12 consecutive months (the size of the circles represent # plans purchased). It helped us send our most active users our best deal. (High price but high discount)

Bigger circle = more sales. X-axis is consecutive months of activity.

Spreadsheet command center

To keep track of the 60 campaigns I built out a spreadsheet to track the different offers and email template modules to include in the campaign. Different segments received personalized offers (pictured) based on analysis from our Data Science team that found the optimal offer according revenue per email sent.

Each cohort, the day of the campaign, and the sales modules to be offered to the recipients

Landing page switch led to 40% conversion improvement

Toward the end of the campaign I created an A-B test for the email click’s landing page.

You should notice that the original page is a very simple credit card page. Because we were advertising a Black Friday time-limited sale our logic was to collect payment as quickly as possible.

You will notice that Treatment version has a lot more information. It has a big hero image, some science facts, a product comparison checklist in the middle, and testimonials on the bottom. Our product & engineering team had refined and tested this landing page for peak performance earlier in the year but we had never tested these two variants against each other within the context of a holiday sale.

The A-B test showed 50%+ sales conversion improvement over the 3-day observation period for the campaign. I wished I had tested these two landing pages earlier in the campaign!

I believe that part of the reason the page performed so much better is that we only sent this page to users who never subscribed before. These people barely know what Calm is so I believe that having the extra information on the page properly educated the returning inactive users and clearly did a better job at explaining the value of the Black Friday deal that we were offering them.

Next year I’ll be sure to compare the two variants across broader audience (churned, current paid, etc.)

Original
Treatment

Treatment landing page (green line) outperformed for duration of the sale
52% improvement & statistically significant

Patterns in engineering job hunting

Newer job seekers and people not may not be familiar with the common job hunting/candidate recruitment process. I’ll try to explain the commonly used process that I’ve noticed to be used by many companies in the Bay Area.

Phase 1: Finding potential jobs

Roughly in order of effectiveness:

Internal Employee Referral – warm

If you happen to know somebody who works at a nice company and you’d like to join them there reach out! Take them out to coffee or get lunch or just IM chat with them. It really depends on how well you know the individual. It’s probably useful to be upfront that you’re open to new opportunities early. If your friend or co-worker is on a team that is actively hiring, you’ve got a warm employee referral on your hands! Going to interviews on a warm employee referral is so much better because I’ve gone to an onsite interview — completely blown half of the questions — and still gotten the job because I knew people that worked at the company who were able to vouch for my real work abilities that aren’t always apparent in a marathon of 1-hour interviews. (Plus jangled nerves)

Internal Employee Referral – cold

You contact your friend at Company X but her group isn’t hiring right now. It’s still OK to go into the pool of general applicants. This isn’t as effective because it could be a while before you hear back from anyone. Effectively you become the same as any other non-referral applicant but at least you’re in the system and maybe down the road your friend will get a referral bonus because you got hired through her efforts to get you into the system.

Internal recruiter cold-email reach out

A recruiter sends you an email or a LinkedIn InMail. Respond to it and you’ll at least get to an informational phone call. That is useful because at least you can reach out to the recruiter later to confirm that you were seen and acknowledged as a human being. You do exist and your resume had something worthy of attention!

LinkedIn network status update

People say they are hiring all the time these days. DM your LinkedIn connection and ask them to refer you into their recruiting system. They can leave notes about you and it will put your resume toward the top of the list. It doesn’t mean you will get called back but it increases the likelihood that your resume will have been seen and reviewed at least for 4 or 5 seconds. Which is better than the methods below.

Direct application on Careers section of website

Surprisingly, I find myself sending people to the careers page of places I’ve worked many times. At startups, the CVs do get reviewed by people internally. Larger companies are less effective

3rd party recruiter cold-email reach out or cold call

Very often these recruiters have very poor information on the job. GuruJobs or Robert Hat. It does not appear that they have a direct connection with the hiring manager. Unclear to me how effective a job hunt will be through these sources.

Big Job Board job application (LinkedIn, Indeed, Monster, HotJobs, Dice, Craigslist)

By far job boards are probably the worst way to find a job. I don’t care what all the commercials say. Don’t waste your time crafting a personal statement or cover letter on these job boards it’s basically pointless. Go directly to a company’s website and submit your resume there instead.

Phase 2: figure out which company is the most interesting

If you can get past the resume screen, companies will start with an informational 30-minute call with a sourcer/recruiter responsible for filling the position. Recruiters are friendly professionals that specialize in talking to other people. They are usually not subject matter experts in engineering, marketing, or whatever your field of work may be.

They are experts at identifying sifting out legitimate experience and from bullshit. In the informational call the recruiter’s job is to describe to you the responsibilities of the role and gauge your interest in the position. Then they sniff out whether you correctly speak the jargon and match the profile of the type of candidate that the hiring manager wants (years of experience, projects worked, interests).

Recruiters will tell you about the company’s mission and culture. They won’t know the company’s five-year product roadmap. They won’t typically know the specifics of the day-to-day of your role. They will be able to tell you is the manager is looking for a specialist or a jack-of-all-trades? How big is the team? Do they prefer Ivy League graduates? You still need to do homework about the company so that you don’t ask blatantly un-researched questions about the company.

Phase 3: Interviewing

After you have talked with the recruiter and sufficiently proven that you are interested in the role, the company, and the department within the company you are ready for some interviews. This is my knowledge of Silicon Valley-style tech interview process.

First, you will have the phone screen interview. Recently interviews have moved off of the phone and into Zoom with screen shares and social coding platforms. The phone screen should last 30-60 minutes and administered either on phone or Zoom where you’ll talk to a fellow engineer who may be a member of the team that wants to hire you. Usually the interviewer has pulled a pre-determined problem pulled from a bank of questions that the team has agreed to use. You will be asked to solve the problem within that 30-60 minute period. You do need to be able to finish the interview within the allotted time because the interviewer probably has another meeting right after your call. Phone interviews are supposed to be “easier.” After many years of dealing with this interview pattern, I know that if I find myself barely getting through the phone interview I know that the on-site is definitely going to be a waste of time for everyone involved.

I think I’ve been asked these “basic” items than a few times on a first-round phone interview:

  • What is a closure?
  • How do you reverse a list?
  • Code a basic HTML/CSS page that can do X,Y,Z

The “on-site” interview (should change the name now that Zoom has become prevalent) is where you line up a day’s worth of interviews and teams grill with 4+ different exercises in varying subjects. For example you might get the abstract puzzle interview, followed by SQL exercise, followed by an algorithm test, followed by a system design discussion. Having failed at literally dozens of these interviews it’s hard to believe I actually ever was able to survive in Silicon Valley for 15 years as an engineer. I do know why I never made it to FAANG because I am pretty horrible at all of these types of interviews and really can only pass the hands-on “let’s build something practical together” type of interview. I don’t have any tips on how to pass these interviews. However, expect to see something like

  • Traverse a graph to do something
  • System design something for millions of people. Web servers and load balancers and calculations of throughput will likely be involved. (I wouldn’t know because I don’t pass these tests)
  • Do something recursive, probably with sorting. It definitely needs to run better than O(n^2) or O(n!)

Phase 4: Results

Now this is the shitty part. Sometimes you might do pretty good and pass all the technical aspects of your interviews. Even if you pass all the tests, you may only have 5% of landing the job because you are competing against 20 other candidates just like you who also passed the tests. This is especially true at a hot startup or a FAANG company.

After your interview all the people you met with will convene in a meeting to decide your fate. The more people you interviewed with during the process, the more chances there are for one of those interviewers to say “no.” That’s not a death sentence but it does hurt you a lot. You would never know it but that’s what goes on after the interviews are over. I have been on hiring panels of 6 people. Sometimes 5 people say “medium yes” and 1 says “strong no” you’re toast. But if you get 5 “strong yes” and 1 “medium no” you still have a chance — that “no” can be swayed.

This is just me, but receiving a “Thank you” note from a candidate does nothing to sway me in any direction for or against you. Experts suggest you to do it but, honestly, the second you walked out of the door or hung up the Zoom I already know if I’m going to vote for you or against you. An after-interview email is meaningless and chances are I probably won’t see it at all because I don’t read email. All of my work in JIRA and Slack.

At the end of the day, if you get rejected from a job application know that it’s not necessarily something about you. There are infinite factors during an interview that are outside of your control that have nothing to do with you. Just be as authentic as you can be. Answer questions as honestly as you can. Be “right” or “correct” on factual questions as much as you can be. Then be prepared to do at least 10+ full interviews (phases 1,2,3,4) before you receive an offer.

How I manage my Iterable Catalogs using Google Sheets

Calm’s dynamic emails leverage the Iterable Catalog feature extensively to keep campaigns evergreen and easy to maintain. I use it for a variety of use cases:

  • Email localization strings. Calm serves content in English, French, Spanish, Japanese, Korean, Portuguese, and German.
  • Content calendars. Calm has a variety of content that releases on a regular cadence including the Daily Calm and the Daily Trip.
  • Follow-up lessons. For special programs we send messages that remind listeners of key takeaways, lessons, or review material.
  • Top N lists. The most recent Catalog I added allows others to curate lists of top/new/hot/trending content.

I have been most tickled with my most recent effort to store Top-N lists (Top 10 newest music, Top 10 most popular, Top 5 fill-in-the-blank) in our Catalog because it streamlines our process for creating ad-hoc special announcement emails. Creating this did not require a lot of code and makes me wish I had thought of doing this earlier.

Google Sheet as Content Management System (CMS)

Using a Google Sheet shared with my colleagues allows us to have a mix of curated and automated feeds controlled by different:

Automated feeds:

  • the newest Sleep Stories for each language
  • the newest Music releases for each language
  • the newest Meditations for each language
  • 3 content types * 7 languages = 21 individual feeds

Curated feeds

  • featured English new releases
  • English-language blog content
  • special blocks for other departments to provide content (b2b, marketing, product, engineering)
The Sheet Tab name “konewSleep” becomes the name of the data feed in the Catalog. The sheet columns headers are key names of each object.

The Catalog

See how the Google Sheet columns map to JSON key names.

Retrieve and Render Iterable Catalog Data

Using a known key, such as ennewSleep I am able to retrieve the appropriate content for Korean language new Sleep Stories.

{{!-- look up new Sleep Stories in English (en) --}}
{{#catalog "Datafeeds" "ennewSleep" as |datafeed|}}
{{#lookup datafeed "programs" as |prog|}}
<h1>{{{ prog.title }}}</h1>
<p>{{{ prog.description }}}</p>
<hr />
{{/lookup}}
{{/catalog}}

Basic HTML result would be

<h1>Sleep story title 1</h1>
<p>Description of story 1</p>
<hr />
<h1>Sleep story title 2</h1>
<p>Description of story 2</p>
<hr />

Addendum: Roadblocks along the way

My initial intent for this project was to lean on Iterable Datafeeds and Google App Scripts to provide my dynamic data. I created a Google Apps Script to create a web app that would grab some API data, package it up into a more email-friendly rendering payload, and return JSON. I ran into 2 major problems while trying to set this up which eventually led to my Catalog solution.

First, I found out that I cannot change the user-agent HTTP header when using the UrlFetchApp library. UrlFetchApp sends something like Google/app-script-client when doing its HTTP fetch. I needed control of the user-agent header to pull the different language content from the API. My solution to this was to set up a free Netlify account that would proxy a request Google -> Netlify -> Calm. I created the proxy as a serverless function. This only took a couple of hours thanks to Netlify for its very developer-friendly deployment tools.

My second problem is that Google App Script only serves a few requests per second (expected). I was expecting to lean on the caching option in the Iterable datafeed fetcher. My theory was Iterable would hit my endpoint once and cache the response for 1 hour as described in their documentation. Despite several attempts at configuring the templates and datafeeds I could not get Iterable to cache my feeds properly. During campaign send time Iterable would hit my Google App Script hundreds of times per second and fail out due to rate limiting problems.

Figuring that the caching roadblock would be unsolvable on my own, I decided to use the Iterable Catalog solution instead. The Catalogs are useful because I don’t have to worry about maintaining uptime on the services but not as nimble as a API solution because I have to constantly sync my Google Sheet to the Catalog(s).