We’ve all been there — there comes a time when we must update our Python version to meet a different Python version yet, be it at work or when working on personal projects.

Just a quick search about “update python version” and we will be bombarded with suggestions to run python --version followed by brew upgrade python3 or sudo apt-get update.

Okay cool. Problem solved right?

Not really. Probably 9/10 times an upgrade won’t cut it — enter another project. And guess what? It wants a different Python version, maybe an older one just to make our life a little bit more miserable.

So, here we are, stuck between a rock and a hard place, asking ourselves, "Do I downgrade Python now? But what if I need to juggle both projects? I didn't sign up for this symlink or PATH variable wrestling match!”

Let's use pyenv?

We know it’s not uncommon either to find ourselves in another project needing a different Python version yet.

Now, with some quick search, you can tell most Python folks swear by pyenv (docs) for managing Python versions.

Don’t get me wrong, it works but it’s just not for me.

Well, given that I work with a lot of CLI tools like go, node, terraform, git, etc., I prefer the simplicity of using a single tool – asdf.

In other words, I very much prefer to use asdf to manage all my programming language or CLI tool versions, rather than dealing with the likes of gvm, nvm, and pyenv separately.

asdf Python Quick Guide

Beyond Python, asdf supports various plugins. But, let's focus on Python without delving into exhaustive details.

Installation

Easy, just follow asdf-vm.com/guide/getting-started.html based on your system specifications:

1. OS (e.g. linux, macOS)
2. Package manager (e.g. brew, apt, pacman, etc.)
3. Shell (e.g. zsh, bash, etc.)

For instance, on macOS with Homebrew and ZSH:

# Using Homebrew on macOS
brew install asdf
echo -e "\\n. $(brew --prefix asdf)/libexec/asdf.sh" >> ${ZDOTDIR:-~}/.zshrc

Setup

Add the Python plugin:

# asdf plugin add <name>: Adds a plugin for managing a specific runtime
# e.g. <name>: python, nodejs, golang
asdf plugin add python

Basic Usage

Let’s install our first Python version:

# asdf install <name> <version>: Installs a specific version of a runtime
asdf install python 3.12.1

What if most of your projects rely on Python 3.12.1? Well, let’s set Python 3.12.1 as our global/default Python version:

# asdf global <name> <version>: Sets a global (default) version of a runtime
asdf global python 3.12.1
cd && python --version # Python 3.12.1

Next, let’s install more Python versions!

asdf install python 3.8.13
asdf install python 3.9.16
asdf install python 3.10.9
asdf install python 3.11.3

Wait, I lost track, how many different Python versions have I installed…?

asdf list python
#  3.10.9
#  3.11.3
# *3.12.1
#  3.8.13
#  3.9.16

"*3.12.1" in the asdf list python output indicates that 3.12.1 is the currently active (local) Python version in the current directory.

Now, you can easily switch between different Python versions:

# Go to my project
cd ~/github.com/ngshiheng/burplist

# Project current Python version
python --version # Python 3.12.1

# But, I need Python 3.8.13
asdf local python 3.8.13

# Yay!
python --version # Python 3.8.13

That’s it! You'll likely find yourself using this set of commands about 80% of the time.

Cheatsheet

Here's a quick refresher:

# "How to install a specific Python version?"
asdf install python 3.12.1

# "What versions have I installed?"
asdf list python

# Set version on global level
asdf global python 3.12.1

# Set version on project level
asdf local python 3.12.1

# "What is my current Python version in this dir?"
python --version

.tools-version

Now you may notice that your project directory may contain a file called .tool-versions. It's used to remember which versions of these tools each project needs (reference).

Should I commit this file to Git?

If having the .tool-versions file in your project helps everyone on your team use the same versions of tools, then it's a good idea to include it in your source control. It keeps things consistent for everyone.

Not Just Python

This approach isn't limited to Python; it works for managing versions of other tools like Node.js, Go, etc. All you need to do is to replace "python" with the respective tool/plugin name:

# Same examples, but in golang:
asdf plugin add golang
asdf install golang 1.21.6
asdf list golang
asdf global golang 1.21.6
asdf local golang 1.21.6
go version

# Same examples, but in nodejs:
asdf plugin add nodejs
asdf install nodejs 21.6.1
asdf list nodejs
asdf global nodejs 21.6.1
asdf local nodejs 21.6.1
node --version

Closing Thought

These days, when considering adopting a new tool, I've adopted a systematic approach:

1. Firstly, I check asdf to see if there's plugin support available (asdf plugin list all). Vet the plugin first!
2. If not, I explore whether the specific CLI tool has its own version manager like gvm, nvm, rubyenv, pyenv, tfenv, etc.
3. If neither option is viable, then only I resort to installing the tool from the source via my package manager like brew or apt

Following this decision-making chain has significantly simplified version management for all my tools, saving me considerable time and pain.

References

• https://github.com/asdf-vm/asdf
• https://asdf-vm.com/
• https://github.com/pyenv/pyenv

Making Internet money is kinda cool. As I wrap up 2023, I decided to jot down the various Internet revenue streams that I have made throughout the year. However, little did I anticipate the nuances involved.

Ironically, I usually pride myself on being a well-organized person. But honestly, I never anticipated that these inconsequential ventures would bring in any money. So, here I am, realizing I never bother to consolidate everything into a single place. Oops.

So, I ended up finding myself jumping from one platform to another, navigating through a maze of dashboards. It felt like a digital treasure hunt just to nail down the right numbers.

Anyway, let’s see…

TL;DR

In the year 2023, I made a total profit of $920.26 (USD):

• Income: $1027.66
• Expenses: -$107.40

Revenue

Total: $1,027.66

Medium Partner Program

Income: $229.70

All of my latest articles find their first home on jerrynsh.com. At the same time, they are automatically cross-posted to Medium.com using Zapier:

So, here's the revenue breakdown by month in 2024 (cutoff on 28 December 2023):

As you can see, most months didn't bring in much, except for the final month when a post about my adventures in automating stuff gained some traction on Medium.

Oh, It's worth noting that I am paying a whopping 30% withholding tax! Ouch.

As I was documenting everything, I started to wonder how this compare to last year. Turns out, I’m down by 64.5%. Yeah, a substantial drop from last year. Oh well.

Overall, I think Medium is not a bad distribution channel for people who already write.

Google AdSense

Income: $117.73 (~$155.94 SGD)

If you've read some of the posts on jerrynsh.com, you may have come across some ads (unless you're using an ad-blocker, of course).

Obviously, the earnings numbers don't mean a thing on their own.

Looking at the views, I'm hitting the 11k mark every month recently. It's kind of wild to think people would want to hang out here on my little corner of the Internet.

Beyond that, you guys are spending an average of 1 minute and 45 seconds reading stuff here on this blog.

About 70% of this traffic comes from organic search, which I think is great.

Now, full disclosure, I'm not a huge fan of most ads, but they pull their weight by helping cover my costs for my tiny projects.

Having said that, I'm eyeing a switch to EthicalAds or Carbon Ads next year. The plan is to bring in more relevant and less intrusive ads.

Affiliate/Referrals Rewards

Income: $635.00

This year, somehow, by some dumb luck, I've managed to pull in some decent cash through affiliate/referral links that were scattered in my blog posts a year or two ago:

1. Nium — $100.00
2. ScraperAPI — $535.00

I reached out to Nium around 2-3 years ago, but I haven't been writing much about them since the partnership started. Most of them came from a single blog post that I wrote back in Jan 2020.

Nonetheless, I genuinely like their service for money transfers compared to the traditional banking hassle and fees for foreign transactions. Although, I'm not sure how long this revenue stream will keep flowing.

On the other hand, the money brought in by ScraperAPI is quite decent this year. Though, most of my earnings from them come from one loyal user (talk about putting all your eggs in one basket).

I'll be honest, the sustainability of this income source seems a bit iffy. We'll see how it goes.

Tournacat

Income: $40.23

Let me introduce you to Tournacat, my little brainchild from this year. It's a simple Google Calendar workspace add-on that syncs upcoming Esports matches/tournaments right to your Google Calendar.

Seriously, if you're into Esports, you should totally give it a try — you won't be disappointed!

Tournacat is free to use from the get-go. But, if users feel a bit fancy and opt for the Pro plan for just $2.50/month (which comes with a purchasing power parity discount too!), they can unlock some pretty neat features.

Oh, sorry for the sales pitch — here’s the breakdown of how much it made:

For now, I'm letting it grow organically. As I mentioned in my previous blog post, as long as the cost of growth is covered, I'm okay with giving away free stuff.

Donations

Income: $5.00

After years of setting up Ko-fi, I finally got my first donation. I actually did end up buying myself a cup of coffee that day.

Cost/Expenses

Total: -$107.40

Domains

Cost: -$29.64

So far, I've only got 3 domains in my collection: jerrynsh.com, tournacat.com, and burplist.com. Each of them set me back $9.88, summing up to $29.64 for the year.

I've hitched my domain registration and DNS wagon to Cloudflare. Their all-in-one service suits me well, making management a bit easier.

Hosting

Cost: -$77.76

After making the shift from Heroku, my projects now call three different Platform-as-a-Service (PaaS) homes. The good news? Most of them still don't cost me anything yet. The hassle is well worth it I suppose.

However, there's one exception — a Digital Ocean droplet that's currently setting me back $6.48/month. Do the math for a year, and we're looking at $77.76 (taxes included).

Looking down the road, I do expect to start paying for Cloudflare Workers because of Tournacat. I really like them, but yeah, we'll see how that unfolds.

What are you doing with these profits?

The responsible thing would be to invest in the stock market or something. But nahhhhhh...

Well, buying coffee seems like a solid plan, right? Isn't that the point of Ko-fi? Jokes aside. Honestly, no concrete plans. Maybe snag a few more domains for some ideas at the back of my head.

On a side note, I did manage to snatch a couple of games that run well on the Steam Deck. I'm really excited to play them!

Looking into 2024

So, the big question: Do I expect profit/revenue to shoot up? Probably not.

Why? Well, most (if not all) of the revenue sources are inconsistent — take those affiliate links, for example. Without those, the profit numbers would have been down from last year.

Another chunk of the revenue is tied to this blog post (plus cross-posting on Medium). Growing blog traffic is a challenge for me because I don't bother much with SEO or self-promotion on social sites like Twitter, LinkedIn or Facebook.

Closing Thoughts

This blog was largely inspired by a read at “Xe's blog made $2564.42 in profit last year”.

If my younger self were to ask myself about starting a blog solely for the extra cash, I'd say think twice. As you can see, it's not the most profitable idea for the most part. The return on investment (time and effort) is very tough to justify, financially.

I think what kept me going was the fun I got out of writing stuff and the opportunity to talk to random people on the Internet.

Learnings

I'm no hero, but this year taught me that I'm genuinely happy when people use or read the stuff I made — whether it's a blog, software, or projects. Even if no money is rolling in. So please keep sending your random DMs/requests my way.

Earlier in my career, I always thought everything I invested time and effort in must somehow turn into some form of financial gain; otherwise, I was just wasting my time. I'm glad and grateful that I don't hold on to that anymore.

Anyway, thanks for reading! Happy New Year!

A little bit more than 7 years into my career, I thought it would be fun to pen down a summary post about my adventure in automating various tiny aspects of my life. Most if not all of the stuff here sprouted from my own problems and itches I needed to scratch.

This post will probably read more like a personal diary of the minor nuances that I encountered and the sweet minutes/hours that I managed to snatch back through automation.

On to the first one –

Six Percent: Automating ASNB Purchases

2018 – 2022

For context, ASNB a unit trust management company offers a fixed-price fund (i.e. it can never go up or down!) that promised a sweet 6% p.a. dividend back then. It’s virtually risk-free.

Well, there was a catch. The limited funds pool meant that it was selling like hotcakes — you had to keep retrying to snag those units. So, what did I do? I wrote this.

Cost

As this turns out to be a tiny Windows executable, it didn’t really cost me any money to host or anything.

Free lunch, literally

Beyond saving me countless mindless hours clicking like a headless chicken, I got a free thank-you meal out of it.

Fast forward to today, and I'm no longer actively using or maintaining the project. Over the years, there have been a few minor bug fixes here and there, but it's essentially retired. I can't guarantee that it still works, but man, it saved me so many hours.

What did I learn

Today, I’ve become quite comfortable with any form of browser automation. If I can interact with it on the web, I can automate it. The script opened doors for tackling repetitive/mundane tasks, aiding in integration testing, etc.

That aside, I did also learn other cool tricks like how to solve a CAPTCHA using OCR and how to package a Python app using PyInstaller!

Todoleet: Daily Leetcode Questions in Todoist

2021 – Present

Back in early 2021, I started to solve the LeetCode Daily challenge as part of my morning routine. Quite frankly, it wasn’t exactly fun for me; it was necessary.

How it works

Even though I've ditched my morning LeetCode routine, Todoleet is still up and running today.

Under the hood, it’s merely a simple JS script that talks to the undocumented LeetCode API and then creates a new to-do task using the Todoist API.

If you're looking for the nitty-gritty, I've spilled the beans on how I sync the Daily Leetcode Challenge to my Todoist.

Cost

$0. The free Cloudflare Worker tier has got me covered! This solution costs 1 request per day and I didn’t need to store anything.

Time saved

I did manage to save a few clicks (seconds) every day. They all add up, I guess?

Any learnings?

This was my introduction to Cloudflare Worker. It paved the road for all the other projects I've tinkered with in my free time!

Future plans

I did consider taking this to another level and listing it as a Todoist integration. But, to be honest, I didn't care enough to make it happen.

Burplist: Sipping on Craft Beer Savings

2021 – Present

Craft beers are delicious. There was just one hiccup — the price. Then I figured, wouldn’t it be great if I could have current and historical prices of all craft beers in Singapore, all in a place?

Now, instead of wrestling with 10+ websites for the best deals, a quick search from my database would do the trick. This saves me time and sanity.

How it works

Burplist is essentially a web scraper built using Scrapy. It scours over 10 local online stores and e-commerce sites every morning (Singapore time), fetching craft beer prices and storing them in a Postgres database. As a result, I've amassed two years of historical craft beer price data in Singapore.

Cost

Other than the ~$10/year for the domain name, running Burplist has always been free. After Heroku phased out of its free tier plan:

1. The scraper Cron job was moved to Northflank
2. The Postgres database is hosted on Railway
3. The website was moved to Koyeb

Free beer for that Christmas

So, initially, I gave it a shot to make some money out of this. All I did was put it up for sale on Gumroad, but it didn't really catch on.

I also tried reaching out to a few companies through cold emails to see if they'd be interested in partnering up with some affiliate links, but only one replied — well, it didn’t work out either.

But hey, no big deal! It's all good because something awesome still came out of it! Thirsty, this local craft beer company, actually surprised me with this amazing package of craft beer for Christmas that year! I was over the moon.

Any lesson learned?

This was quite a big one. I’ve had so much fun and learned so much from making this project. Burplist is the fanciest web scraper that I’ve built thus far. I’ve written down some of the learnings in a blog post.

Looking ahead

Future plans? Maybe migrate from Postgres to SQLite. Then, update the daily job to push the SQLite file directly to GitHub and present the data through something like this nifty method. Expected result? One less webserver to babysit. If this ever happens, I’ll probably write about it somewhere.

Wraith: Automating Ghost Blog Backup

2022 – Present

At the start of my blogging journey, I didn’t really think too much about what would happen if this $6/month Droplet crashed. I mean, I had all my blog entries in Notion, so I figured "Meh, it wouldn’t hurt to copy-paste ~10 of them back if anything bad happens".

As time went on, the realization hit — losing all my data and whatnot now would really suck. So, the solution? Automate the backup. With the blog serving around 10k views monthly, any downtime or a prolonged 404 or 500 is something I'd rather avoid.

What’s underneath

It’s a pretty simple Bash script that does three things:

1. ghost backup
2. mysqldump
3. rclone to a remote drive (e.g. Dropbox, Google Drive, etc.)

This Bash script gets a weekly run in a Cron job, and it's as easy as that.

Cost

$0. I didn’t have to pay for Dropbox; the free tier fits my needs just fine.

Time saved

I mean, the alternative would be manually SSH-ing into my Droplet, running backup steps one by one — a process taking roughly about 5 minutes or less. So, that's the weekly saving of ~5 minutes.

What did I learn

Picked up a few best practices for writing Bash script along the way. Besides that, I did learn about new tools like expect, rclone, and pass (the Linux password manager).

Future plans

Not a whole lot to be honest. Perhaps a backup restore script could be handy. Thought about moving passwords from plain text to using pass. But, that means users dealing with gpg + pass CLI setup — an extra hurdle.

Tournacat: Sync Esports Schedules to Google Calendar

Started 2023

I got tired of missing highly anticipated Dota 2 Esports matches, dealing with wonky time zones, and the mental gymnastics of remembering it all. Since Google Calendar is practically my second brain, I figured, why not sync upcoming matches straight into it?

Turning it into a micro SaaS

I started by sharing the Dota 2 calendar with friends. Then, a lightbulb moment — if it's handy for them, maybe others would pay for it. And so, D2GCal was born. Pay, and get a public link to the Google Calendar. Simple.

But wait — people are picky about their calendars. Some find it “noisy”, and some want a customized experience. Enter Tournacat, supporting 10+ Esports titles (not just limited to Dota 2!), giving users the power to own and customize their calendars.

Cost

~$10/year for now for the domain name.

Firstly, the website (tournacat.com) operates as a static site built using Hugo. It's currently hosted using Cloudflare Pages which is free.

The Google Workspace add-on is built and runs on Google Apps Script (GAS) for free.

Lastly, Tournacat has an API server running on Cloudflare Worker. It fetches the upcoming Esports events from a data source. Running on Cloudflare Worker is currently still within the free tier limit but it won't be so for long.

The neat part? Tournacat doesn't store any user info. No names, no emails — nothing at all. This means that no database is needed.

Revenue

Today, Tournacat has about 90 users. In terms of paid subscriptions, it's made a tiny profit of $40.23 in 11 months.

Honestly, I never expected to rake in big bucks anyway. The goal was to cover growing worker usage costs. Any surplus? Well, that's coffee money.

Lessons learned

The journey with Tournacat has been a blast. The realization that a handful of people want and use what you built yourself is an indescribable feeling.

This tiny venture has taught me many valuable lessons:

1. Developing on the GAS platform
2. Publishing of a Google Calendar add-on to Workspace
3. Working with payments and subscriptions
4. Crafting a micro SaaS

Besides, the journey also introduced me to the mundane world of social media marketing, which, isn't my cup of tea. On the flip side, I had an amazing time speaking to people about their feedback (feel free to ask me anything)!

Overall, I feel like the Esports landscape is shaped by a generation accustomed to abundant free entertainment and tools. Convincing people to spend money can be a tough sell.

Any future plans?

There are plenty of tasks/tickets on the project board, but I'm taking it slow. The plan? I’ll just be working on minor improvements here and there unless specific user requests pop up.

Being a solo developer has its perks — the turnaround time for new requests is pretty quick. I've been able to incorporate feedback almost immediately (as long as it's somewhat reasonable).

Overall, the product feels pretty complete as it is.

SGS Issuance Calendar: Automated T-Bill Tracking

Started 2023

Feeling the manual-checking fatigue for the Singapore MAS T-bill issuance calendar, I decided, "Why not automate this?" So, I crafted a schedule to run every month using Google Apps Script (GAS) and detailed the process in this blog post.

How it Works

Under the hood, it's a small GAS project written in TypeScript. The script gets triggered every month; pings the MAS API, and then creates important dates (e.g. announcement/auction date) as Google Calendar events. Simple as that!

Cost

$0. The best part about using GAS is that I don’t need to be bothered by the underlying infrastructure hassle. No fretting over scaling, upgrades, backups, availability — none of that. It just does its thing.

Some lessons learned

This project brought some learnings to the table. I am now able to build GAS projects in TypeScript and delved into the world of writing unit tests for GAS. This newfound skillset later found a home in Tournacat's add-on UI codebase, which is also written in GAS.

What's next?

I've added support for SGS and SSB calendars as well! For now, no concrete plans unless user feedback or GitHub issues come knocking. Two months in, and it's been serving me well.

Closing Thoughts

The journey has been nothing short of fun. My approach to automating/building usually unfolds like this:

1. Identify my own pain point or problem (note it down immediately!); however small it may be. Look for quick wins!
2. Prefer leveraging what's already out there. Check if there's something in the market for it, like Zapier or IFTTT
3. If not, then roll up my sleeves and code/build
4. See if I can generalize the solution or approach
5. Write down the process somewhere
6. Rinse and repeat

“Aren’t some of these side projects?"

Well, I guess. I just think the term feels so worn out at this point. It's like, they're more about automating some parts of my life. Sure, one of them is making coffee money monthly, a bit laughable. The whole idea of “hustling” just doesn't quite vibe with me. I've realized I just want to do things just for fun. No, really.

Thanks for sticking around! 🍻 Here's to more automation to come!

I finally had enough of manually checking the Monetary Authority of Singapore (MAS) T-Bill calendar from the official site. Just miss the buying window? Great! now I'll have to wait for another week or so.

Sometimes life gets in the way, then you forget, then you miss again. All that cash just sitting there doing nothing. I mean, there must be a better way to handle this, right?

The Calendar

Here's the finished product:

Here are the links to the individual calendars:

Idea

Existing Solution

Before diving into any project, I always like to see if there are existing solutions that might fit the bill. In my search, I stumbled upon ilovessb.com, which supports email notifications.

That's a step in the right direction, but there's still the possibility of missing those email notifications. Plus, I don’t really feel like giving away my email.

Google Calendar

Google Calendar notifications on the other hand are pretty neat. It supports device notifications, emails, or both. So yeah, it’s a real winner!

Anyway, I generally start my week by looking at my Google Calendar to see what's ahead for the week. Naturally, it occurred to me that having the MAS T-bill issuance calendar sync to my Google Calendar would be a neat solution. Why not automate this process?

The idea is pretty straightforward:

1. Get the necessary data (bonds/bills issuance information and dates)
2. Generate calendars and events using that data
3. Run this on a regular schedule (monthly/yearly)

This should be an easy one. After all, I've already dabbled in a similar project before.

Goal

The ultimate objective here is to create a calendar that I can easily share with anyone by simply providing them a link. The idea is to offer people the benefits of using this calendar without requiring them to give up their email or name.

The only potential downside to this is that if I accidentally mess things up and delete these calendars, users might have to subscribe to a new calendar with a new link. But as long as I don't touch it, it should be smooth sailing (famous last words I guess). Anyway…

Data Source

The first step involves figuring out how to obtain the data we need to create events for the calendar. There are 2 approaches:

1. Inspecting the network requests made when visiting the issuance calendar site
2. Scraping the site’s HTML

Option 1 is always my preferred method due to its more structured and reliable nature.

MAS API

After some digging around the network tabs, I was able to identify the API calls made, which opened the door for us to fetch the data.

Then, I created a simple class that wraps around the respective MAS API requests. This file contains a class that wraps around the API requests. This makes it possible for us to interact with MAS API and gather the data required for our calendars later on.

If we want to fetch other data from other endpoints in the future, we can easily add support for those endpoints. No sweat!

Google Apps Script (GAS)

The core logic of Cron jobs (i.e. time-driven triggers), creating calendars, and events is written in TypeScript and deployed as a GAS project.

You can check out the code here.

Why Google Apps Script?

Well, there are a few reasons:

1. It's free to run. This is an important point, especially since I plan to maintain this calendar for public use. Ensuring it's free is essential for its sustainability.
2. Its Google Calendar API is super easy to use. No need to handle authentication like you would with its other languages SDKs. Dealing with that kind of stuff is an extra annoyance.
3. It can be written in JavaScript/TypeScript, which is a language I'm already familiar with.

Also, since I haven't written any GAS project using TypeScript, I saw this as an opportunity to experiment with it. (The developer experience turned out great!)

Challenges

Oddly enough, I find the most challenging part of this project is to write tests. Well, more specifically when it comes to writing tests for GAS projects.

Wait, why bother writing tests for a pet project?

Well, the number of “live” tiny projects that I currently maintain is slowly getting out of hand. I’d imagine coming back to this project a year later, trying to refactor, add a small feature, or fix a bug, and thinking, "Will I break this..?".

For me, writing tests reduces that mental toil. It provides a certain “safety net”, assuring me that the changes won't break existing functionality. It's an oddly comforting feeling.

That being said, I always try to strike a balance between testing and maintenance. I believe in writing enough tests to check for regressions, but I also think that testing can become a maintenance burden if taken to an extreme. I don’t want to end up in a scenario where I spend more time wrestling with these darn unit tests than actually fixing the bug or adding a feature!

Why is it hard?

Testing a GAS project can be tricky. When you run code locally (e.g. using Jest) with Node.js, it uses the Node.js runtime. However, when you deploy that code as a GAS, it runs in a V8 runtime. This difference can cause issues when it comes to testing.

Let’s look at an example. First, make sure you have jest and ts-jest installed. If not, you can install them using npm or yarn:

npm install --save-dev jest ts-jest @types/jest

Now, imagine you've written a function called createMonthlyTrigger, which, as the name suggests, is used to create a monthly time-driven using the GAS built-in ScriptApp library.

Here's the code example:

export function createMonthlyTrigger(): GoogleAppsScript.Script.Trigger {
    const triggers = ScriptApp.getProjectTriggers();
    for (const trigger of triggers) {
        const triggerExist = trigger.getHandlerFunction() === main.name;
        if (triggerExist) {
            Logger.log(`Trigger "${main.name}" already exists`);
            return trigger;
        }
    }

    Logger.log(`Creating a new monthly trigger`);
    return ScriptApp.newTrigger(main.name).timeBased().onMonthDay(1).atHour(1).create();
}

The catch here is that you can't simply run jest and expect it to work smoothly.

Instead, we need to mock the ScriptApp module since it's a GAS-specific library and not available in a typical Node.js environment. To achieve this, you can use a library like ts-jest with jest to mock ScriptApp.

Without mock, we will run into errors like ReferenceError: Logger is not defined.

Here's an example of how to do it:

import { createMonthlyTrigger } from "../src/index";

// Create a mock for ScriptApp
const mockScriptApp = {
    getProjectTriggers: jest.fn(),
    newTrigger: jest.fn().mockReturnThis(),
    timeBased: jest.fn().mockReturnThis(),
    onMonthDay: jest.fn().mockReturnThis(),
    atHour: jest.fn().mockReturnThis(),
    create: jest.fn(),
};

// Mock the Logger
const mockLogger = {
    log: jest.fn(),
};

// Mock the global objects
(global as any)["Logger"] = mockLogger;
(global as any)["ScriptApp"] = mockScriptApp;

describe("createMonthlyTrigger", () => {
    afterEach(() => {
        // Clear the mock calls after each test
        jest.clearAllMocks();
    });

    it("should return an existing trigger if one exists", () => {
        // Mock an existing trigger
        mockScriptApp.getProjectTriggers.mockReturnValue([
            {
                getHandlerFunction: jest.fn().mockReturnValue("main"),
            },
        ]);

        createMonthlyTrigger();
        expect(mockScriptApp.newTrigger).not.toHaveBeenCalled();
    });

    it("should create a new trigger if none exists", () => {
        // Mock no existing triggers
        mockScriptApp.getProjectTriggers.mockReturnValue([]);

        createMonthlyTrigger();
        expect(mockScriptApp.newTrigger).toHaveBeenCalled();
    });
});

DevOps

These are the stuff that I always aim to automate as much as possible.

Continuous Integration (CI)

Setting up CI for your GAS project is pretty much the same as you'd do for any other Node.js project. There's nothing special or unconventional about it.

Take a look at the ci.yaml file.

Manual Deployment

I highly recommend using clasp (GitHub) when working on your GAS project. It's a game-changer, making your life a whole lot easier. It streamlines the development and deployment process, making it a more enjoyable experience.

I've been thinking about adding a Continuous Deployment (CD) part in the future, which would automate the deployment process. Though, I'm not sure how often I'll be deploying updates anytime soon. This project feels quite "completed" for now, but who knows?

Release

I'm using semantic-release to automate my releases. However, in this project, I've disabled the part where it automatically publishes the codebase as an NPM package (I mean it’s not a library where you would import).

You can take a look at the release.yml and package.json if you're curious.

Dependency Update

Just like many of my other projects, I've set up automatic dependency updates using Renovate. It's great for keeping the project's dependencies fresh and up-to-date with minimal effort on my part.

The best part? With CI setup and tests, it adds another layer of confidence to make sure that any automated updates don't pose a big risk to the project's stability.

Closing Thoughts

So, there you have it! This project is short and sweet, but it has been on my to-do list for a while. It's one of those small, nagging issues that bother me a bit every day, and I'm happy to finally find a solution.

Thanks for reading!

The other day, I was casually browsing the web and stumbled upon the need for an API for my new little project. Instead of starting from scratch, I thought it would be great if I could find an existing API to build upon. That's when I came across this repository consisting of public APIs.

As I was browsing through the list of public APIs, I couldn't help but think how cool it would be to create a tiny website using some of these APIs. It just seemed like such a fun project that could be quickly put together and shown to others.

But guess what's even more exciting? Building this website using the available APIs with HTMX instead of our typical Single-Page Application (SPA) frameworks!

Why

Well, hype aside, I find this exciting because of the challenge it presents. It’s no secret that HTMX can work with both hypermedia and JSON responses. Although, HTMX is primarily designed for hypermedia. However, I want to prove that it's completely possible (and not even that difficult) to use HTMX with JSON API as well.

JSON is the most popular data format for APIs

Also, this seems to be a common real-life scenario that many encounter.

Just imagine someone saying, "Hey, I want to migrate to HTMX but I already have a bunch of JSON APIs. Do I have to change my entire server-side implementation? What a bummer!".

So, today, I’m going to write about how I came to create a very simple website powered by HTMX, Nunjucks, and Spaceflight News API (SNAPI). But really you can replicate this with any JSON API of your choice.

Static Sites

I love static websites. They are fast, cheap, and super easy to host. The best part? They hardly require any maintenance!

I really like the idea of serving dynamic content from a static site with just a single HTML file. When you combine it with HTMX, it takes away all the hassle of dealing with messy JavaScript code in your HTML files.

Prerequisites

• Basic knowledge of HTML and JavaScript
• A code editor with a live server for testing (e.g., Visual Studio Code with the Live Server extension) or any code playground like JSFiddle

Goals

By the end of this, you will have built a Spaceflight News blog page using HTMX with the following features:

1. Displaying blog posts with titles, authors, publication dates, and summaries
2. Pagination with a "Load More" button
3. Search functionality

The final outcome would somewhat be akin to this ’Click to Load’ example, but built with an actual JSON API (refer to SNAPI documentation).

Implementation

Step 1: Basic HTML Structure

Let's get started with building our web page. The first step is to set up the basic HTML structure. It's super easy!

Just copy the provided HTML code into a new HTML file. You can name it something like index.html. Oh, and don't forget to include the necessary script in the <head> section:

<!DOCTYPE html>
<html lang="en">
    <head>
        <title>Spaceflight News Blogs</title>
        <meta charset="utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />
        <script src="https://unpkg.com/htmx.org"></script>
        <script src="https://unpkg.com/htmx.org/dist/ext/client-side-templates.js"></script>
        <script src="https://cdn.jsdelivr.net/npm/nunjucks@3.2.4/browser/nunjucks.min.js"></script>
        <link
            rel="stylesheet"
            href="https://cdn.jsdelivr.net/gh/kognise/water.css@latest/dist/dark.css"
        />
    </head>
    <body>
        <h1>Spaceflight News Blog</h1>
        <p>
            Demo of how the <b>load more</b> UX pattern works on htmx with JSON
            data APIs! You can find the source code on
            <a
                href="https://github.com/ngshiheng/spaceflight-news/blob/main/blogs.html"
                >GitHub</a
            >. Go take a look!
        </p>
    </body>
</html>

Now, before we move on, I want to share some libraries that we'll be using:

• HTMX for AJAX interactions
• Nunjucks for client-side templates
• Optional: water.css classless CSS for basic styling

API Schema

Here, we’ll be using the /v4/blogs endpoint. According to the SNAPI docs, we've got an example response that we can work with:

{
  "count": 123,
  "next": "http://api.example.org/accounts/?offset=400&limit=100",
  "previous": "http://api.example.org/accounts/?offset=200&limit=100",
  "results": [
    {
      "id": 0,
      "title": "string",
      "url": "string",
      "image_url": "string",
      "news_site": "string",
      "summary": "string",
      "published_at": "2023-09-12T23:51:31.675Z",
      "updated_at": "2023-09-12T23:51:31.675Z",
      "featured": true,
      "launches": [
        {
          "launch_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
          "provider": "string"
        }
      ],
      "events": [
        {
          "event_id": 2147483647,
          "provider": "string"
        }
      ]
    }
  ]
}

If we take a look at the JSON response, it looks like we can make good use of the title, url, summary, and published_at fields to create our blog page.

The first 10 characters of an ISO 8601 date string format are in the format YYYY-MM-DD. So we’ll use the first 10 characters of published_at as our published date later.

The next and previous keys will come in handy for pagination.

Step 2: Client Side Templates

Next, we'll delve into actually calling the API and displaying the available blog posts on SNAPI.

<!DOCTYPE html>
<html lang="en">
    <head>
        <!-- omitted for brevity -->
    </head>
    <body>
        <h1>Spaceflight News Blog</h1>
        <!-- omitted for brevity -->
        <div hx-ext="client-side-templates">
            <div
                hx-get="https://api.spaceflightnewsapi.net/v4/blogs/"
                hx-trigger="load"
                nunjucks-template="blogs-template"
            >
                <template id="blogs-template">
                    
                <!-- count -->
                {% if not previous %}
                <i> Found {{ count }} articles.</i>
                {% endif %}

                <!-- results -->
                {% for blog in results %}
                <div>
                    <hr />
                    <h2>
                        <a href="{{ blog.url }}" target="_blank"
                            >{{ blog.title }}</a
                        >
                    </h2>
                    <h4>
                        Published by {{ blog.news_site }}, {{
                        blog.published_at | truncate(10, true, "")}}
                    </h4>
                    <p>{{ blog.summary }}</p>
                </div>
                {% endfor %}
                    
                </template>
              <div id="result"></div>
            </div>
        </div>
    </body>
</html>

Now, let me break it down a bit:

1. We use the hx-ext="client-side-templates" attribute to activate the client-side templates extension for HTMX. This little beauty allows us to employ a templating engine (e.g. Nunjucks) to transform the JSON response from the API into HTML before it shows up on the page.
2. The nunjucks-template attribute specifies the name of the Nunjucks template (blogs-template) we'll be using to convert the JSON response from the API into HTML. This template comprises the following elements:
3. A count of the total number of results
4. A list of the results, each featuring a title, news site, published date, and summary
5. When the hx-get event is triggered on load, the https://api.spaceflightnewsapi.net/v4/blogs/ endpoint is called and the response is parsed as JSON. The Nunjucks template is then used to render the JSON response into HTML. The rendered HTML is then swapped into the DOM, replacing the div element with the ID blogs-template.

Now, let's take a peek at all this in action on our browser... Huh, it should have worked smoothly, right? Why is the blog section of our page completely blank?! Where are the blog posts?!?!

Debugging: CORS Workaround

Let’s check this out with some good old browser inspection. Checking out the Console tab, we see an error message stating:

Access to XMLHttpRequest at 'https://api.spaceflightnewsapi.net/v4/blogs/' from origin 'https://fiddle.jshell.net' has been blocked by CORS policy: Request header field hx-request is not allowed by Access-Control-Allow-Headers in preflight response.

Ugh! It seems to be a CORS error. A bit of Googling led me to a helpful GitHub issue. After a bit of reading, I found out that there's an event listener for htmx:configRequest that we could use.

We need to work around this issue because the Spaceflight News API doesn't allow cross-origin requests by default.

Here's the script to add to the <head> section to implement the CORS workaround when making requests to external APIs with HTMX. It essentially just removes any custom headers:

<!DOCTYPE html>
<html lang="en">
    <head>
        <!-- omitted for brevity -->
        <script>
            // CORS workaround
            document.addEventListener("htmx:configRequest", (evt) => {
                evt.detail.headers = [];
            });
        </script>
    </head>
    <body>
       <!-- omitted for brevity -->
    </body>
</html>

We can now see a list of blog posts being rendered on our HTML page! Yay!

However, we are only seeing the results from the first page. Our next step is to work on the pagination part: add a "Load More" button at the bottom of the page.

Step 3: Pagination

The "Load More" button is implemented as a <button> element:

<!DOCTYPE html>
<html lang="en">
    <head>
        <!-- omitted for brevity -->
    </head>
    <body>
        <h1>Spaceflight News Blog</h1>
        <!-- omitted for brevity -->
        <div hx-ext="client-side-templates">
            <div
                hx-get="https://api.spaceflightnewsapi.net/v4/blogs/"
                hx-trigger="load"
                nunjucks-template="blogs-template"
            >
                <template id="blogs-template">
                    
                  <!-- omitted count for brevity -->
  
                  <!-- omitted results for brevity -->
  
                  <!-- load more -->
                  {% if next %}
                  <button
                      hx-get="{{ next }}"
                      hx-swap="outerHTML"
                      hx-trigger="click"
                      nunjucks-template="blogs-template"
                  >
                      Load More...
                  </button>
                  {% endif %}
          
                </template>
                <div id="result"></div>
            </div>
        </div>
    </body>
</html>

When the “Load More” button is clicked, here’s what happens:

1. The hx-get event is triggered. The next variable is used to specify the URL of the next page of blogs. The HTML that gets swapped into the DOM is the same as the one for the blogs-template element but with some fancy new blogs added to the end.
2. The hx-swap attribute basically tells us how we should swap the new HTML into the DOM. In our case, we’re using the outerHTML swap mode. This means that the new HTML will replace the entire existing HTML for the blogs-template element.
3. Lastly, the hx-trigger attribute tells us which event will trigger the hx-get event. Here, we're using the good ol' click event. So, whenever you click on that "Load More" button, hx-get event gets triggered and our pagination game begins!

That’s it! We’ve now got a “Load More” pagination implemented for our blog.

We're not done yet! We need to find the blog post about the Chandrayaan-3 lunar exploration mission. So, let's keep clicking that "Load More" button...and more...and more...

Step 4: Search Input

Just kidding! Who would want to keep clicking more to look for specific stuff? Thankfully, the /v4/blog of SNAPI supports search queries out of the box. So, we can easily search for the article about the Chandrayaan-3 lunar exploration mission.

Let's work on modifying the original div to a search input. So, you can easily type in your search keywords and find the articles you're interested in:

<!DOCTYPE html>
<html lang="en">

  <head>
    <!-- omitted for brevity -->
  </head>

  <body>
    <h1>Spaceflight News Blog</h1>
    <!-- omitted for brevity -->
    <div hx-ext="client-side-templates">
      <!-- search -->
       <input
             id="search-input"
             autofocus
             hx-get="https://api.spaceflightnewsapi.net/v4/blogs/"
             hx-indicator=".htmx-indicator"
             hx-target="#result"
             hx-trigger="load delay:100ms, keyup changed delay:500ms, search"
             name="search"
             nunjucks-template="blogs-template"
             placeholder="Search blogs..."
             type="search"
      />


      <template id="blogs-template">

        <!-- omitted for brevity -->

      </template>
      <div id="result"></div>

    </div>
  </body>

</html>

So, the <input> element acts as a search bar that uses the hx-get event to load a list of blogs from the API when you type in a search query.

The hx-get event is triggered by the load, keyup and search events (reference). The search results are then rendered in the #result element using the blogs-template Nunjucks template.

Finally, I can search for “Chandrayaan” related blog posts. No more endless clicking ‘Load More’!

Conclusion

We have successfully created a Spaceflight News blog page with live search functionality, dynamic content loading, and client-side templates using HTMX and JSON API.

So, what’s next? You could give the blog page a makeover by incorporating infinite scrolling, just like this. Or you could add a next-previous button instead.

For a better search experience, you could try to update the search input to sync with the search query of the URL so that you can do something like this.

Thanks for reading!

References

Making this wouldn’t be possible without these awesome resources:

• https://htmx.org/docs/
• https://mozilla.github.io/nunjucks/
• https://api.spaceflightnewsapi.net/v4/docs/

When it comes to raising exceptions and exception handling in Python, I've often found myself pondering, "Should I re-raise this exception? Or maybe I should raise it from another exception?"

You see, the thing is, there are a bunch of ways to handle exceptions in Python. We’d often just wing it without really grasping the why and when to use these patterns.

In this little exploration, we’re going to uncover the differences behind these different exception-handling patterns with code examples. Today, we'll be unraveling:

• When to catch and re-raise an exception?
• When to raise a new exception?
• When to chain exception?
• When to avoid using each of the above?

Enough talk, let’s dive right into these exception-raising dilemmas and turn them into informed decisions. As we go through the examples, feel free to copy and paste the code snippet to try it out yourself on something like ipython!

Pattern 1: Catch and Re-Raise Exception

def divide(x=1, y=0):
    try:
        return x / y

    except ZeroDivisionError as e:
        raise e # NOTE: this is almost equivalent to bare `raise`

In this pattern, if an exception occurs during the division operation (e.g., division by zero), the original exception will be re-raised with its traceback.

As a result, the original exception is propagated all the way up the call stack with its original traceback.

Use Case

Generally speaking, this is often a good default pattern for its clarity and preservation of original traceback.

Why? Preserving traceback information is often considered a good practice as it helps in diagnosing errors effectively and understanding the sequence of events that led to the exception.

In short, this pattern is used when you want to catch a specific exception (ZeroDivisionError in this case), do something, and then re-raise the exception. It allows you to perform some specific actions before propagating the exception further.

Effectively the same as not having the try block at all

def divide(x=1, y=0):
    return x / y

Wait what?! Then why do we need the try block then?

Knowing this, the try block in our previous example might suddenly seem redundant. However, you can imagine scenarios where additional logging or error-handling logic would make the try block very useful:

def divide(x=1, y=0):
    try:
        return x / y

    except ZeroDivisionError as e:
        print("An error occurred while performing the division.")
        # Log the error, send notifications, retries etc.
        raise e

Having that said, if you do not intend to do any additional stuff, feel free to omit the try block.

Avoid

While this is generally a useful pattern, there can be scenarios where it might not be the best choice.

For instance, you may want to avoid using this pattern when you want to hide sensitive information or when the traceback contains sensitive data that you don't want to expose. Here’s an example:

try:
    user = login(email="foo@example.com",password="12345")
    # Perform some sensitive operations

except InvalidPasswordFormatError as e:
    raise e

In this example, if an exception occurs during the sensitive operation, the original exception is re-raised with its traceback. This pattern may expose sensitive information, such as login failure information in the traceback.

This is often referred to as "leaking" or "revealing" information. Exposing detailed login failure information is bad because it aids attackers. They can exploit specifics to guess usernames and launch targeted attacks.

To prevent this, it's better to handle the exception without re-raising it or to raise a new exception with a more generalized error message (see Pattern 3).

Pattern 2: Raise New Exception

def divide(x=1, y=0):
    try:
        return x / y

    except ZeroDivisionError:
        raise ValueError("Pattern 2 error.")

In this example, a new exception is raised with a custom message, while preserving the original exception's traceback. If a ZeroDivisionError occurs, a new ValueError is raised with a custom message.

The traceback will include both the ZeroDivisionError and the ValueError that was raised.

Use Case

This pattern is useful when you want to raise a different (more meaningful) type of exception to indicate a specific error condition. This still allows us to preserve the original exception’s traceback.

Avoid

Avoid using this when you need to preserve the original exception.

Pattern 3: Raise New Exception from None

def divide(x=1, y=0):
    try:
        return x / y

    except ZeroDivisionError:
        raise ValueError("Pattern 3 error.") from None

This pattern is similar to Pattern 2. But, using from None suppresses the original ZeroDivisionError exception.

Here, the traceback will not include the original ZeroDivisionError, only the ValueError exception and the custom error message raised.

Use Case

Similar to Pattern 2, you would want to use this pattern when you want to raise a new exception with a custom message.

The difference here is that this will not include the traceback of the original exception. It is useful when you want to hide the details of the original exception from the user.

More Examples

If you are wrapping a library that throws internal exceptions and you want to present transformed external exceptions to your application users.

In this scenario, using this pattern is a suitable approach. Here’s a simple example:

try:
    # Some library code that might raise an internal exception
    result = library_function(data)

except InternalException as e:
    raise ExternalException("An error has occurred") from None

In this example, wrapping internal exceptions with external exceptions helps isolate your application code from the specifics of the internal library's implementation. This can come in handy when the users of your code don't need to understand or handle the internal exceptions thrown by the library.

Avoid

Avoid using this approach when you (or your users) need to understand the full context of where the original exception occurred and how it led to the new exception.

Suppressing exceptions can make it more difficult to track down the root cause of an error, so it should only be done when necessary.

Pattern 4: Chaining Exception

def divide(x=1, y=0):
    try:
        return x / y

    except ZeroDivisionError as e:
        raise ValueError("Pattern 4 error.") from e

Again, the ZeroDivisionError exception is caught and a new ValueError exception is raised with a custom message.

Though, the from e clause tells Python to pass the original ZeroDivisionError exception as an argument to the new ValueError exception. As a result:

• This allows the caller of the divide() function to know what the original error was
• The traceback of the original exception (e) will be included in the printed traceback of the newly raised exception

Use Case

This pattern is commonly used when you want to raise a new exception with a custom message and include the traceback of the original exception as its cause. It is useful when you want to provide both the specific error message and the context of the original exception.

In terms of best practices – it is generally recommended to use the from e syntax when raising a new exception from an inner except block. This allows us to preserve the stack trace of the original error, which (again) can be helpful for debugging.

Pattern 2 vs. Pattern 4

"What's the difference between Pattern 2 vs. Pattern 4 then? They seem awfully similar!"

In Pattern 2, the ZeroDivisionError exception is simply raised (from the line return x / y) without being handled. This means that the caller of the divide() function will not be aware of the error.

In comparison, Pattern 4 is more informative and therefore the better choice in most cases. However, Pattern 2 may be used if the caller of the divide() function does not need to know about the original error.

More Example

You're building a file-processing application that uses an external library for reading and processing files. If the library raises an internal FileNotFoundError, you want to raise your own custom exception to provide more context and information to the user.

class FileProcessingError(Exception):
    def __init__(self, message):
        super().__init__(message)

def process_file(file_path):
    try:
        content = read_file(file_path)
        # Process content...

    except FileNotFoundError as e:
        raise FileProcessingError("Unable to process file.") from e

def read_file(file_path):
    # Simulate a file not found error
    raise FileNotFoundError(f"File not found: {file_path}")

try:
    process_file("example.txt")
except FileProcessingError as e:
    print(f"Error: {e}") # Error: Unable to process file.
    print("Original Exception:", e.__cause__) # Original Exception: File not found: example.txt

In this example, the FileProcessingError is raised with the context of the original FileNotFoundError. This provides more information to the user and helps in debugging by maintaining the traceback chain.

Avoid

Avoid using this pattern when you want to hide the details of the original exception or when the original traceback is not needed (see Pattern 3) to understand the higher-level error.

In some cases, preserving both tracebacks can be confusing if not handled carefully.

Summary

Exception handling in Python is about dealing with errors in your code. The best way to handle exceptions often depends on what you want to achieve.

Anyway, here’s a TL;DR of what we went through:

• Pattern 1 (good default): Re-raises the same exception with its original traceback.
• Pattern 2 (situational): Re-raises a new exception, does not lose original traceback.
• Pattern 3 (situational): Re-raises a new exception with a chained exception relationship (from None), but loses the original traceback.
• Pattern 4 (best): Re-raises a new exception with a chained exception relationship (from e), including both the new and the original traceback.

Best Practices

1. Use the from e syntax when raising a new exception from an inner except block. This allows us to preserve the stack trace of the original error.
2. Do not suppress exceptions unless it is absolutely necessary. Suppressing exceptions can make it more difficult to track down the root cause of an error.
3. Use meaningful error messages. The error message should be clear and concise, and it should provide enough information to help the user understand what went wrong.
4. Handle all (possible) errors. It is important to handle all possible errors that your code can throw. This will help to prevent your code from crashing unexpectedly.

Remember, the way you handle exceptions should make your code easy to understand and debug. Always think about what helps you and others know what went wrong and why.

Besides learning the right way to handle exceptions, it's just as important to stop using exceptions like this in Python!

Recently, I found myself facing the need to convert my personal PDF files to text. While there were many existing PDF to text converters available for this task, I couldn't shake the feeling of unease about uploading my private documents to unknown servers. Who knows what could happen to them or how secure they truly are?

I decided to build my own PDF to text converter (demo). Right from the get-go, I knew this was possible by using only Python.

Getting Started

Before that, I must confess, I'm not particularly skilled in frontend development. So, I opted to use PyWebIO, sparing me the hassle of dealing with HTML and CSS.

Main Dependencies

To begin, I set up a new project directory named pypdf2txt and initialized it with Git and Poetry (my go-to virtual environment and package manager for Python).

I included the necessary dependencies — pdfminer.six for working with PDFs and pywebio for building the web application.

Lastly, I created a main.py file to hold the core functionality of the converter.

mkdir pypdf2txt
cd pypdf2txt
git init
poetry init
poetry add pdfminer.six pywebio
touch main.py

Optional: Dev Dependencies

I added autopep8 and ruff (instead of flake8) as development dependencies using Poetry. These will help with automatic code formatting and linting:

poetry add -D autopep8 ruff

Converting PDF to Text

NOTE: Throughout this entire walkthrough, we only have one main.py to work with.

Adding Site Description

Let’s start with something a little bit more simple, shall we? To render any output to the browser, we can simply use the pywebio.output module (reference).

# main.py

from functools import partial
from io import BytesIO, StringIO
from pathlib import Path

from pdfminer.high_level import extract_text_to_fp
from pywebio import config, session, start_server
from pywebio.input import file_upload
from pywebio.output import clear, put_buttons, put_code, put_markdown, toast, use_scope
from pywebio.session import download, run_js

def render_description():
    description = """
    # Pypdf2: Convert PDF to Text

    A simple Python web service that allows you to convert your PDF documents to text.
    Extract text from PDF files without compromising **privacy**, **security**, and **ownership**.

    ## Features

    -   Converts PDF documents to text
    -   Simple and easy-to-use web interface
    -   Fast and efficient text extraction
    """

    put_markdown(description)

In the render_description function, I define the app's description using a markdown-formatted string.

Then, I use put_markdown from the pywebio.output module to display the description in the web interface.

Setting Up the Web App

This is the main function that brings everything together:

# ... (rest of the code)

def main():
    session.run_js(
        'WebIO._state.CurrentSession.on_session_close(()=>{setTimeout(()=>location.reload(), 4000})',
    )

    render_description()
    # process_pdf() #  TODO: create & uncomment later 

if __name__ == "__main__":
    start_server(main, port=8080)

It starts the PyWebIO server, runs the JavaScript code for auto-reloading the page on session close, renders the initial description of the app, and invokes the process_pdf function to handle the PDF to text conversion.

Now, when you run the app, it will render the description at the start:

poetry run python3 main.py

File Upload

Next, I needed to upload the PDF file through the web interface. Let’s create a function name process_pdf utilizing the file_upload API. I found a handy example of file upload in the PyWebIO documentation, and with a bit of copy-pasting, I was well on my way:

# ... (rest of the code)

def process_pdf():
    put_markdown(
        """
        ## Convert PDF To Text
        """,
    )

    while True:
        pdf_file = file_upload(
            "Select PDF",
            accept="application/pdf",
            max_size="10M",
            multiple=False,
            help_text="sample.pdf",
        )

        text_output = extract_text_from_pdf(pdf_file) # TODO: work on this later
        text_filename = f"{Path(pdf_file['filename']).stem}.txt"

# ... (rest of the code)

First, I added a "Convert PDF To Text” heading to the UI, making it clear what the app was designed to do.

The file_upload function handled the file selection, ensuring that only PDFs were accepted and imposing a reasonable file size limit of 10MB — for now.

In the future, we could configure this based on the deployment server's available RAM.

Extracting Text from PDF

Once the user uploaded the PDF file, the extract_text_from_pdf function swung into action. Here's the code snippet for extracting text from the uploaded PDF:

# ... (rest of the code)

from pdfminer.high_level import extract_text_to_fp

def extract_text_from_pdf(pdf_file):
    pdf_buffer = BytesIO(pdf_file['content'])
    text_buffer = StringIO()
    extract_text_to_fp(pdf_buffer, text_buffer)
    return text_buffer.getvalue()

# ... (rest of the code)

Here's where the magic happened. To extract text from the PDF, I utilized the extract_text_to_fp API from pdfminer to do the job. The concise PDFMiner documentation guided me through the process.

Keeping Output Area Clean

To maintain a clean user experience, I encapsulated the text output and download button in a "scope" using PyWebIO's use_scope function (reference).

This allowed me to clear the specified output area every time a new file was uploaded, ensuring that each conversion was separate and didn't clutter the interface.

# ... (rest of the code)

def process_pdf():
    put_markdown(
        """
        ## Convert PDF To Text
        """,
    )

    while True:
        pdf_file = file_upload(
            "Select PDF",
            accept="application/pdf",
            max_size="10M",
            multiple=False,
            help_text="sample.pdf",
        )
        clear('text-output-area') # NOTE: to clear previous text output

        text_output = extract_text_from_pdf(pdf_file)
        text_filename = f"{Path(pdf_file['filename']).stem}.txt"

        with use_scope('text-output-area'):
            put_markdown(
                """
                ### Text Output
                """,
            )

            put_code(text_output, rows=10)
            put_buttons(
                [
                    "Copy to Clipboard",
                    'Click to Download',
                ],
                onclick=[
                    partial(
												copy_to_clipboard, # TODO: create later
												text=text_output
										),
                    partial(
                        click_to_download, # TODO: create later
                        filename=text_filename,
                        text=text_output.encode(),
                    ),
                ],
            )

# ... (rest of the code)

Download Text Output

Implementing a “Click to Download” feature for our app is super straightforward. In fact, you can choose between using put_file (reference) or download (reference).

I chose the latter because I wanted it to be a button instead of a link:

# ... (rest of the code)

def click_to_download(filename: str, text: bytes):
    download(
        filename,
        text,
    )
    toast('Text file downloaded')

# ... (rest of the code)

With this, users can download the output text with a simple click of a button.

Copy to Clipboard

Lastly, let’s work on our “Copy to Clipboard” feature. This is where things are a little bit more tricky.

I learned that PyWebIO doesn't natively support this copy-to-clipboard function. Yet, the demo above showcased its presence, sparking my determination to uncover its implementation.

Despite my thorough search through the official documentation, the feature remained elusive. However, my experience has led me to Sourcegraph, where I unearthed the secrets behind its clever implementation. Oh, the joy of discovery!

After taking inspiration from PyWebIO's demo example, here’s what I arrived at:

# ... (rest of the code)

def copy_to_clipboard(text: str):
    clipboard_setup = """
    window.writeText = function(text) {
        const input = document.createElement('textarea');
        input.style.opacity  = 0;
        input.style.position = 'absolute';
        input.style.left = '-100000px';
        document.body.appendChild(input);

        input.value = text;
        input.select();
        input.setSelectionRange(0, text.length);
        document.execCommand('copy');
        document.body.removeChild(input);
        return true;
    }
    """
    run_js(clipboard_setup)
    run_js("writeText(text)", text=text)
    toast('Text copied to the clipboard')

# ... (rest of the code)

When the user clicks the "Copy to Clipboard" button, the JavaScript code within copy_to_clipboard sets up a temporary textarea, copies the text to it, selects the content, and executes the copy command.

As a result, the extracted text is now ready for pasting elsewhere, and a toast notification confirms the successful copy operation. Yay!

Full Code

You may find the full code example here:

GitHub - ngshiheng/pypdf2txt at v0.1.0

A simple Python web service that allows you to convert your PDF documents to text. - GitHub - ngshiheng/pypdf2txt at v0.1.0

GitHubngshiheng

Closing Thoughts

Congratulations! You now have the power of converting PDF to text at your fingertips, without sacrificing privacy, security, and ownership.

I remember the first time I stumbled upon PyWebIO, and I was immediately intrigued by its promise of creating web apps with just a few lines of Python code. While it might not suit complex projects, for simple tools, it's amazing.

If I were to build a GUI application, I would have gone for something like PyQt. Today, I’d say no more GUI frameworks; just one browser app is all you need.

Happy building!

At Grab, we rely heavily on a large Go monorepo for backend development, which offers benefits like code reusability and discoverability. However, as we continue to grow, managing a large monorepo brings about its own set of unique challenges.

As an example, using Go commands such as go get and go list can be incredibly slow when fetching Go modules residing in a large multi-module repository. This sluggishness takes a toll on developer productivity, burdens our Continuous Integration (CI) systems, and strains our Version Control System host (VCS), GitLab.

In this blog post, we look at how Athens, a Go module proxy, helps to improve the overall developer experience of engineers working with a large Go monorepo at Grab.

Key highlights

• We reduced the time of executing the go get command from ~18 minutes to ~12 seconds when fetching monorepo Go modules.
• We scaled in and scaled down our entire Athens cluster by 70% by utilising the fallback network mode in Athens along with Golang’s GOVCS mode, resulting in cost savings and enhanced efficiency.

Problem statements and solutions

1. Painfully slow performance of Go commands

Problem summary: Running the go get command in our monorepo takes a considerable amount of time and can lead to performance degradation in our VCS.

When working with the Go programming language, go get is one of the most common commands that you’ll use every day. Besides developers, this command is also used by CI systems.

What does go get do?

The go get command is used to download and install packages and their dependencies in Go. Note that it operates differently depending on whether it is run in legacy GOPATH mode or module-aware mode. In Grab, we’re using the module-aware mode in a multi-module repository setup.

Every time go get is run, it uses Git commands, like git ls-remote, git tag, git fetch, etc, to search and download the entire worktree. The excessive use of these Git commands on our monorepo contributes to the long processing time and can be strenuous to our VCS.

How big is our monorepo?

To fully grasp the challenges faced by our engineering teams, it’s crucial to understand the vast scale of the monorepo that we work with daily. For this, we use git-sizer to analyse our monorepo.

Here’s what we found:

• Overall repository size: The monorepo has a total uncompressed size of 69.3 GiB, a fairly substantial figure. To put things into perspective, the Linux kernel repository, known for its vastness, currently stands at 55.8 GiB.
• Trees: The total number of trees is 3.21M and tree entries are 99.8M, which consume 3.65 GiB. This may cause performance issues during some Git operations.
• References: Totalling 10.7k references.
• Biggest checkouts: There are 64.7k directories in our monorepo. This affects operations like git status and git checkout. Moreover, our monorepo has a maximum path depth of 20. This contributes to a slow processing time on Git and negatively impacts developer experience. The number of files (354k) and the total size of files (5.08 GiB) are also concerns due to their potential impact on the repository’s performance.

To draw a comparison, refer to the git-sizer output of the Linux repository.

How slow is “slow”?

To illustrate the issue further, we will compare the time taken for various Go commands to fetch a single module in our monorepo at a 10 MBps download speed.

This is an example of how a module is structured in our monorepo:

gitlab.company.com/monorepo/go
  |-- go.mod
  |-- commons/util/gk
        |-- go.mod

In this example, using go get to fetch a module took over 18 minutes to complete. If we needed to retrieve more than one module in our monorepo, it can be incredibly time-consuming.

Why is it slow in a monorepo?

In a large Go monorepo, go get commands can be slow due to several factors:

1. Large number of files and directories: When running go get, the command needs to search and download the entire worktree. In a large multi-module monorepo, the vast number of files and directories make this search process very expensive and time-consuming.
2. Number of refs: A large number of refs (branches or tags) in our monorepo can affect performance. Ref advertisements (git ls-remote), which contain every ref in our monorepo, are the first phase in any remote Git operation, such as git clone or git fetch. With a large number of refs, performance takes a hit when performing these operations.
3. Commit history traversal: Operations that need to traverse a repository’s commit history and consider each ref will be slow in a monorepo. The larger the monorepo, the more time-consuming these operations become.

The consequences: Stifled productivity and strained systems

Developers and CI

When Go command operations like go get are slow, they contribute to significant delays and inefficiencies in software development workflows. This leads to reduced productivity and demotivated developers.

Optimising Go command operations’ speed is crucial to ensure efficient software development workflows and high-quality software products.

Version Control System

It’s also worth noting that overusing go get commands can also lead to performance issues for VCS. When Go packages are frequently downloaded using go get, we saw that it caused a bottleneck in our VCS cluster, which can lead to performance degradation or even cause rate-limiting queue issues.

This negatively impacts the performance of our VCS infrastructure, causing delays or sometimes unavailability for some users and CI.

Solution: Athens + fallback Network Mode + GOVCS + Custom Cache Refresh Solution

Problem summary: Speed up go get command by not fetching from our VCS

We addressed the speed issue by using Athens, a proxy server for Go modules (read more about the GOPROXY protocol).

How does Athens work?

The following sequence diagram describes the default flow of go get command with Athens.

Athens uses a storage system for Go module packages, which can also be configured to use various storage systems such as Amazon S3, and Google Cloud Storage, among others.

By caching these module packages in storage, Athens can serve the packages directly from storage rather than requesting them from an upstream VCS while serving Go commands such as go mod download and certain go build modes. However, just using a Go module proxy didn’t fully resolve our issue since the go get and go list commands still hit our VCS through the proxy.

With this in mind, we thought “what if we could just serve the Go modules directly from Athens’ storage for go get?” This question led us to discover Athens network mode.

What is Athens network mode?

Athens NetworkMode configures how Athens will return the results of the Go commands. It can be assembled from both its own storage and the upstream VCS. As of Athens v0.12.1, it currently supports these 3 modes:

1. strict: merge VCS versions with storage versions, but fail if either of them fails.
2. offline: only get storage versions, never reach out to VCS.
3. fallback: only return storage versions, if VCS fails. Fallback mode does the best effort of giving you what’s available at the time of requesting versions.

Our Athens clusters were initially set to use strict network mode, but this was not ideal for us. So we explored the other network modes.

Exploring offline mode

We initially sought to explore the idea of putting Athens in offline network mode, which would allow Athens to serve Go requests only from its storage. This concept aligned with our aim of reducing VCS hits while also leading to significant performance improvement in Go workflows.

However in practice, it’s not an ideal approach. The default Athens setup (strict mode) automatically updates the module version when a user requests a new module version. Nevertheless, switching Athens to offline mode would disable the automatic updates as it wouldn’t connect to the VCS.

Custom cache refresh solution

To solve this, we implemented a CI pipeline that refreshes Athens’ module cache whenever a new module is released in our monorepo. Employing this with offline mode made Athens effective for the monorepo but it resulted in the loss of automatic updates for other repositories

Restoring this feature requires applying our custom cache refresh solution to all other Go repositories. However, implementing this workaround can be quite cumbersome and significant additional time and effort. We decided to look for another solution that would be easier to maintain in the long run.

A balanced approach: fallback Mode and GOVCS

This approach builds upon our aforementioned custom cache refresh which is specifically designed for the monorepo.

We came across the GOVCS environment variable, which we use in combination with the fallback network mode to effectively put only the monorepo in “offline” mode.

When GOVCS is set to gitlab.company.com/monorepo/go:off, Athens encounters an error whenever it tries to fetch modules from VCS:

gitlab.company.com/monorepo/go/commons/util/gk@v1.1.44: unrecognized import path "gitlab.company.com/monorepo/go/commons/util/gk": GOVCS disallows using git for private gitlab.company.com/monorepo/go; see 'go help vcs'

If Athens network mode is set to strict, Athens returns 404 errors to the user. By switching to fallback mode, Athens tries to retrieve the module from its storage if a GOVCS failure occurs.

Here’s the updated Athens configuration (example default config):

GoBinaryEnvVars = ["GOPROXY=direct",
"GOPRIVATE=gitlab.company.com",
"GOVCS=gitlab.company.com/monorepo/go:off"]

NetworkMode = "fallback"

With the custom cache refresh solution coupled with this approach, we not only accelerate the retrieval of Go modules within the monorepo but also allow for automatic updates for non-monorepo Go modules.

Final results

This solution resulted in a significant improvement in the performance of Go commands for our developers. With Athens, the same command is completed in just ~12 seconds (down from ~18 minutes), which is impressively fast.

In addition, this change to our Athens cluster also leads to substantial reduction in average cluster CPU and memory utilisation. This also enabled us to scale in and scale down our entire Athens cluster by 70%, resulting in cost savings and enhanced efficiency. On top of that, we were also able to effectively eliminate VCS’s rate-limiting issues while making the monorepo’s command operation considerably faster.

2. Go modules in GitLab subgroups

Problem summary: Go modules are unable to work natively with private or internal repositories under GitLab subgroups.

When it comes to managing code repositories and packages, GitLab subgroups and Go modules have become an integral part of the development process at Grab. Go modules help to organise and manage dependencies, and GitLab subgroups provide an additional layer of structure to group related repositories together.

However, a common issue when using Go modules is that they do not work natively with private or internal repositories under a GitLab subgroup (see this GitHub issue).

For example, using go get to retrieve a module from gitlab.company.com/gitlab-org/subgroup/repo will result in a failure. This problem is not specific to Go modules, all repositories under the subgroup will face the same issue.

A cumbersome workaround

To overcome this issue, we had to use workarounds. One workaround is to authenticate the HTTPS calls to GitLab by adding authentication details to the .netrc file on your machine.

The following lines can be added to the .netrc file:

machine gitlab.company.com
    login user@company.com
    password <personal-access-token>

In our case, we are using a Personal Access Token (PAT) since we have 2FA enabled. If 2FA is not enabled, the GitLab password can be used instead. However, this approach would mean configuring the .netrc file in the CI environments as well as on the machine of every Go developer.

Solution: Athens + .netrc

A feasible solution is to set up the .netrc file in the Go proxy server. This method eliminates the need for N number of developers to configure their own .netrc files. Instead, the responsibility for this task is delegated to the Go proxy server.

3. Sharing common libraries

Problem summary: Distributing internal common libraries within a monorepo without granting direct repository access can be challenging.

At Grab, we work with various cross-functional teams, and some could have distinct network access like different VPNs. This adds complexity to sharing our monorepo’s internal common libraries with them. To maintain the security and integrity of our monorepo, we use a Go proxy for controlled access to necessary libraries.

The key difference between granting direct access to the monorepo via VCS and using a Go proxy is that the former allows users to read everything in the repository, while the latter enables us to grant access only to the specific libraries users need within the monorepo. This approach ensures secure and efficient collaboration across diverse network configurations.

Without Go module proxy

Without Athens, we would need to create a separate repository to store the code we want to share and then use a build system to automatically mirror the code from the monorepo to the public repository.

This process can be cumbersome and lead to inconsistencies in code versions between the two repositories, ultimately making it challenging to maintain the shared libraries.

Furthermore, copying code can lead to errors and increase the risk of security breaches by exposing confidential or sensitive information.

Solution: Athens + Download Mode File

To tackle this problem statement, we utilise Athens’ download mode file feature using an allowlist approach to specify which repositories can be downloaded by users.

Here’s an example of the Athens download mode config file:

downloadURL = "https://proxy.golang.org"

mode = "sync"

download "gitlab.company.com/repo/a" {
    mode = "sync"
}

download "gitlab.company.com/repo/b" {
    mode = "sync"
}

download "gitlab.company.com/*" {
    mode = "none"
}

In the configuration file, we specify allowlist entries for each desired repo, including their respective download modes. For example, in the snippet above, repo/a and repo/b are allowed (mode = “sync”), while everything else is blocked using mode = “none”.

Final results

By using Athens’ download mode feature in this case, the benefits are clear. Athens provides a secure, centralised place to store Go modules. This approach not only provides consistency but also improves maintainability, as all code versions are managed in one single location.

Additional benefits of Go proxy

As we’ve touched upon the impressive results achieved by implementing Athens Go proxy at Grab, it’s crucial to explore the supplementary advantages that accompany this powerful solution.

These unsung benefits, though possibly overlooked, play a vital role in enriching the overall developer experience at Grab and promoting more robust software development practices:

1. Module immutability: As the software world continues to face issues around changing or disappearing libraries, Athens serves as a useful tool in mitigating build disruptions by providing immutable storage for copied VCS code. The use of a Go proxy also ensures that builds remain deterministic, improving consistency across our software.
2. Uninterrupted development: Athens allows users to fetch dependencies even when VCS is down, ensuring continuous and seamless development workflows.
3. Enhanced security: Athens offers access control by enabling the blocking of specific packages within Grab. This added layer of security protects our work against potential risks from malicious third-party packages.
4. Vendor directory removal: Athens prepares us for the eventual removal of the vendor directory, fostering faster workflows in the future.

What’s next?

Since adopting Athens as a Go module proxy, we have observed considerable benefits, such as:

1. Accelerated Go command operations
2. Reduced infrastructure costs
3. Mitigated VCS load issues

Moreover, its lesser-known advantages like module immutability, uninterrupted development, enhanced security, and vendor directory transition have also contributed to improved development practices and an enriched developer experience for Grab engineers.

Today, the straightforward process of exporting three environment variables has greatly influenced our developers’ experience at Grab.

export GOPROXY="goproxy.company.com|proxy.golang.org,direct"
export GONOSUMDB="gitlab.company.com"
export GONOPROXY="none"

At Grab, we are always looking for ways to improve and optimise the way we work, so we contribute to open-sourced projects like Athens, where we help with bug fixes. If you are interested in setting up a Go module proxy, do give Athens (github.com/gomods/athens) a try!

So, a few weeks back, I stumbled upon this awesome talk by Brandon Rhodes. I was so hooked that I couldn't resist the temptation to dive right into the example he shared and jot down my learnings.

One key takeaway I've learned is the importance of separating I/O operations (i.e. network requests, database calls, etc.) from the core logic of our code. By doing so, we can make our code more modular and testable.

I won't be delving into the intricacies of Clean Architecture or Clean Code in this blog post. While those concepts hold their own value, I want to focus on something that you can put into use immediately.

Let's dive right into an example that will serve as our starting point. This example is taken directly from Brandon Rhodes' slides, which you can find here.

Exploring the find_definition Function

Imagine we have a Python function called find_definition that performs data processing and involves making HTTP requests to an external API.

import requests                      # Listing 1
from urllib.parse import urlencode

def find_definition(word):
    q = 'define ' + word
    url = 'http://api.duckduckgo.com/?'
    url += urlencode({'q': q, 'format': 'json'})
    response = requests.get(url)     # I/O
    data = response.json()           # I/O
    definition = data[u'Definition']
    if definition == u'':
        raise ValueError('that is not a word')
    return definition

Writing our first test

To write a unit test for the find_definition function, we can utilize Python's built-in unittest module. Here's an example of how we can approach it:

import unittest
from unittest.mock import patch

class TestFindDefinition(unittest.TestCase):
    @patch('requests.get')
    def test_find_definition(self, mock_get):
        mock_response = {u'Definition': 'Visit tournacat.com'}
        mock_get.return_value.json.return_value = mock_response
        
        expected_definition = 'Visit tournacat.com'
        definition = find_definition('tournacat')
        
        self.assertEqual(definition, expected_definition)
        mock_get.assert_called_with('http://api.duckduckgo.com/?q=define+tournacat&format=json')

To isolate the I/O operations, we use the patch decorator from the unittest.mock module. It allows us to mock the behavior of the requests.get function.

By doing so, we can control the response that our function receives during testing. This way, we can test the find_definition function in isolation without actually making real HTTP requests.

Testing difficulties and tight coupling

By using the patch decorator to mock the behavior of the requests.get function, we tightly couple the tests to the internal workings of the function. This makes the tests more susceptible to breaking if there are changes in the implementation or dependencies.

If the implementation of find_definition changes, such as:

1. Using a different HTTP library
2. Modifying the structure of the API response
3. Changes in the API endpoint

The tests may need to be updated accordingly. In the case of find_definition, writing and maintaining unit tests becomes a cumbersome task.

Hiding I/O: A Common Mistake

Typically, when working with functions like find_definition that involve I/O operations, I’d often refactor the code to extract the I/O operations into a separate function, such as call_json_api, as shown in the updated code below (again, borrowed from Brandon’s slides):

def find_definition(word):           # Listing 2
    q = 'define ' + word
    url = 'http://api.duckduckgo.com/?'
    url += urlencode({'q': q, 'format': 'json'})
    data = call_json_api(url)
    definition = data[u'Definition']
    if definition == u'':
        raise ValueError('that is not a word')
    return definition

def call_json_api(url):
    response = requests.get(url)     # I/O
    data = response.json()           # I/O
    return data

By extracting the I/O operations into a separate function, we achieve a level of abstraction and encapsulation.

The find_definition function now delegates the responsibility of making the HTTP request and parsing the JSON response to the call_json_api function.

Updating the test

Again, we utilize the patch decorator from the unittest.mock module to mock the behavior of the call_json_api function (instead of requests.get). By doing so, we can control the response that find_definition receives during testing.

import unittest
from unittest.mock import patch

class TestFindDefinition(unittest.TestCase):
    @patch('call_json_api')
    def test_find_definition(self, mock_call_json_api):
        mock_response = {u'Definition': 'Visit tournacat.com'}
        mock_call_json_api.return_value = mock_response
        
        expected_definition = 'Visit tournacat.com'
        definition = find_definition('tournacat')
        
        self.assertEqual(definition, expected_definition)
        mock_call_json_api.assert_called_with('http://api.duckduckgo.com/?q=define+tournacat&format=json')

”We have hidden I/O, but have we really decoupled it?”

However, it's important to note that although we have hidden the I/O operations behind the call_json_api function, we haven't completely decoupled them.

The find_definition function still depends on the implementation details  call_json_api and assumes it will handle the I/O operations correctly.

Dependency Injection: Decoupling

To achieve a more decoupled design, we could further separate the I/O concerns by using dependency injection.

Here's an updated version of the find_definition:

import requests

def find_definition(word, api_client=requests):  # Dependency injection
    q = 'define ' + word
    url = 'http://api.duckduckgo.com/?'
    url += urlencode({'q': q, 'format': 'json'})
    response = api_client.get(url)               # I/O 
    data = response.json()                       # I/O
    definition = data[u'Definition']
    if definition == u'':
        raise ValueError('that is not a word')
    return definition

The api_client parameter is introduced, which represents the dependency responsible for making the API calls. By default, it is set to requests, allowing us to use the requests library for the I/O operations.

Unit testing with dependency injection

Using dependency injection allows for better control and predictability in unit testing. Here's an example of how we can write unit tests for the find_definition function with dependency injection:

import unittest
from unittest.mock import MagicMock

class TestFindDefinition(unittest.TestCase):
    def test_find_definition(self):
        mock_response = {u'Definition': u'How to add Esports schedules to Google Calendar?'}
        mock_api_client = MagicMock()
        mock_api_client.get.return_value.json.return_value = mock_response

        word = 'example'
        expected_definition = 'How to add Esports schedules to Google Calendar?'

        definition = find_definition(word, api_client=mock_api_client)

        self.assertEqual(definition, expected_definition)
        mock_api_client.get.assert_called_once_with('http://api.duckduckgo.com/?q=define+example&format=json')

In the updated unit test example, we create a mock API client using the MagicMock class from the unittest.mock module. The mock API client is configured to return a predefined response i.e. mock_response when its get method is called.

Yay! In the case where we want to use a different HTTP library, we're now in a much better spot.

Problems with dependency injection

While dependency injection offers several benefits, it can also introduce some challenges. As highlighted by Brandon, there are a few potential problems to be aware of:

1. Mock vs. Real Library: The mock objects used for testing might not fully replicate the behavior of the real dependencies. This could lead to discrepancies between test results and actual runtime behavior.
2. Complex Dependencies: Functions or components with multiple dependencies, such as a combination of database, filesystem, and external services, can require significant injection setup and management, making the codebase more complex.

This brings us to the next point.

Separating I/O Operations from Core Logic

In the pursuit of a flexible and testable code, we can adopt a different approach without relying on explicit dependency injection

We can achieve a clear separation of concerns by placing the I/O operations at the outermost layer of our code. Here's an example that demonstrates this concept:

def find_definition(word):           # Listing 3
    url = build_url(word)
    data = requests.get(url).json()  # I/O
    return pluck_definition(data)

Here, the find_definition function focuses solely on the core logic of extracting the definition from the received data. The I/O operations, such as making the HTTP request and retrieving the JSON response, are performed at the outer layer.

In addition, the find_definition function also relies on two separate functions:

1. build_url function constructs the URL for the API request
2. pluck_definition function extracts the definition from the API response.

Here are the corresponding code snippets:

def build_url(word):
    q = 'define ' + word
    url = 'http://api.duckduckgo.com/?'
    url += urlencode({'q': q, 'format': 'json'})
    return url

def pluck_definition(data):
    definition = data[u'Definition']
    if definition == u'':
        raise ValueError('that is not a word')
    return definition

By putting I/O at the outermost layer, code becomes more flexible. We successfully created functions that can be individually tested and replaced as needed.

For example, you can easily switch to a different API endpoint by modifying the build_url function, or handle alternative error scenarios in the pluck_definition function.

This separation of concerns enables modifications to the I/O layer without impacting the core functionality of find_definition, enhancing the overall maintainability and adaptability of the codebase.

Updating unit tests (again)

To demonstrate the improved flexibility and control offered by the modular design, let's update our unit tests for the find_definition function.

Here's the updated code snippet:

import unittest
from unittest.mock import patch

class TestFindDefinition(unittest.TestCase):
    @patch('requests.get')
    def test_find_definition(self, mock_get):
        mock_response = {'Definition': 'Visit tournacat.com'}
        mock_get.return_value.json.return_value = mock_response
        word = 'example'
        expected_definition = 'Visit tournacat.com'
        
        definition = find_definition(word)
        
        self.assertEqual(definition, expected_definition)
        mock_get.assert_called_once_with(build_url(word))
    
    def test_build_url(self):
        word = 'example'
        expected_url = 'http://api.duckduckgo.com/?q=define+example&format=json'
        
        url = build_url(word)
        self.assertEqual(url, expected_url)
    
    def test_pluck_definition(self):
        mock_response = {'Definition': 'What does tournacat.com do?'}
        expected_definition = 'What does tournacat.com do?'
        
        definition = pluck_definition(mock_response)
        self.assertEqual(definition, expected_definition)

if __name__ == '__main__':
    unittest.main()

In the updated unit tests, we now have separate test methods for each of the modular components:

1. test_find_definition remains largely unchanged from the previous example before dependency injection was introduced, verifying the correct behavior of the find_definition function. However, it now asserts that the requests.get function is called with the URL generated by the build_url function, demonstrating the updated interaction between the modular components.
2. test_build_url verifies that the build_url function correctly constructs the URL based on the given word.
3. test_pluck_definition ensures that the pluck_definition function correctly extracts the definition from the provided data.

By updating our unit tests, we can now test each component independently, ensuring that they function correctly in isolation.

Summary

In short, we’ve explored different approaches to refactoring to address tight coupling and achieve loose coupling between components. On top of that, we witnessed how unit testing can be enhanced by mocking I/O operations and controlling the behavior of external dependencies.

By placing I/O operations at the outermost layer of our code, we achieve a clear separation of concerns, enhancing the modularity and maintainability of our codebase.

Finally, if you're interested in reading more from me, check out my other articles about Python.

I made Tournacat, an add-on that syncs upcoming Esports schedules of tournaments to Google Calendar. Two months later, Tournacat has garnered 4 paid users and is slowly growing.

In this post, I will share my experience building Tournacat using various technologies including Cloudflare Worker, Hugo, and Google Apps Script.

Along the way, I encountered some interesting challenges and learned valuable lessons. Specifically, I want to share my anecdotal experience with issues I encountered while developing the Tournacat add-on using Google Apps Script.

Why Did I Make Tournacat

Tournacat started as an idea from D2GCal, a link to a Google Calendar featuring upcoming Dota 2 tournaments available for purchase on Gumroad.

I built Tournacat out of the frustration of missing out on big Dota 2 matches. I was tired of constantly scouring the internet to find out when my favorite matches were happening.

As an avid Esports fan, I knew there had to be a better way (cliché I know). So, I started building Tournacat for my own use.

Today, Tournacat has grown to support major Esports titles including Counter-Strike, Dota 2, LoL, Valorant, Overwatch, and more.

Why Google Calendar

I use Google Calendar extensively in my daily life, from scheduling meetings to keeping track of personal events. So, having an Esports schedule alongside everything in one place is a huge plus.

With Tournacat, you can just install the add-on to your Google Calendar and let it work its magic. It's a seamless solution that integrates perfectly with Google Calendar.

Existing Solutions (Calendars)

I've been on the lookout for a solution like Tournacat for years. I've tried all sorts of public Esports calendars in the past, but they always fell short in one way or another.

Either the author of the calendar stopped updating it, or it was just an unreliable source that was prone to human error. It got to the point where I gave up hope of ever finding a calendar that actually worked.

Tech Stack

Tournacat is built using a variety of tools and technologies. The tech stack includes:

• Frontend — Hugo
• Backend — Cloudflare Worker
• Add-on — Google Apps Script

Hugo

Hugo is a static site generator that I used to create the Tournacat website. I decided to use Hugo for a couple of reasons.

First, I'm not particularly skilled in front-end development, so I wanted to use a static site generator that would allow me to use pre-built themes without having to write much front-end code myself. Hugo fit the bill perfectly in this regard.

Second, Hugo is known for its speed and ease of use. I wanted to be able to iterate quickly and not have to spend a lot of time fussing with my site's performance or configuration. Hugo's documentation and tooling made it easy to get up and running quickly. With that, I was able to focus on building out the content for Tournacat’s website rather than worrying about technical details.

I then hosted the website on Cloudflare Pages. Using Cloudflare Pages allowed me to keep costs down, as I didn't need to pay for expensive hosting services.

Cloudflare Worker

For the backend, I decided to go with Cloudflare Worker. Cloudflare Worker is a serverless computing platform that allows you to run JavaScript code on the edge.

To make my life easier, I decided to build the backend server Tournacat on top of a framework — Worktop, a lightweight Cloudflare Worker web framework that simplifies the development of APIs using TypeScript.

GitHub - lukeed/worktop: The next generation web framework for Cloudflare Workers

The next generation web framework for Cloudflare Workers - GitHub - lukeed/worktop: The next generation web framework for Cloudflare Workers

GitHublukeed

Given the opportunity to start over, I would consider using Hono instead, as it appears to be more actively maintained on GitHub.

My decision to choose Cloudflare Worker was based on my comfort level with it, having done several previous projects with it. The developer experience was nothing short of amazing, and deployment was easy. Furthermore, it was affordable with a generous free tier.

While the performance was impressive, it was not the main reason why I picked it.

Google Apps Script

Google Apps Script is the core technology behind the Tournacat add-on. It is a scripting language based on JavaScript that allows you to extend and automate Google Workspace products like Google Calendar, Sheets, and Drive.

While building Google Workspace add-ons is possible with other coding languages (runtimes), I decided to go for Google Apps Script anyway due to convenience. Frankly, I never thought that I’d be impressed at how simple it is to build a workspace add-on!

Do keep in mind that it does come with some limitations and quotas.

Navigating the Bumps

Creating a Google Workspace add-on is relatively straightforward. The add-on tutorial provided by Google Workspace is pretty easy to follow.

In short, It took me a day to figure out most things and about a week to get the add-on approved and published on Google Workspace Marketplace.

Advantages

1. One of the pros of using Google Apps Script is that it literally cost nothing to run.
2. Since it uses JavaScript, it's easy for most developers to pick up and start working with.

Challenges

1. Although Google claims that Apps Script is now supported by the V8 runtime, it doesn't actually support Promises or async-await (Issue Tracker). Consequently, you may need to resort to using workarounds like triggers (Stackoverflow references: 1, 2).
2. Pushing changes to the Workspace add-on is rather unorthodox. I noticed this when one of my users couldn’t get their time-based triggers to work properly after a new deployment. Turns out the “correct” way to update a workspace add-on is to edit a versioned deployment instead of creating a new one (Stackoverflow reference).

3. It has to respect the limits set by Google. However, it wasn't documented if the limits were shared across all add-on users cumulatively (hint: it is not).

4. Lastly, I came across an authentication-related issue (Stackoverflow references: 1, 2) when attempting to use Google Apps Script as a webhook to allow my users to automatically activate their subscriptions after payment. Some answers suggested temporarily disabling the V8 runtime to fix this.

Despite these hiccups, Google Apps Script is a great platform to work with. It gets 90% of the job done very effectively and the last 10% is the challenges that I mentioned.

Some Lessons Learned

1. Use what makes you the most productive. Don’t worry about picking the perfect tech stack that handles scale.
2. Users couldn’t care less about your code. Nobody except you cares about the fancy frameworks, design patterns, or architecture used.
3. Keep your deployment simple. This allows for fast iterations. While I could have gone all-in with AWS and set up a sophisticated Terraform setup, I realized that I would have taken more time to ship if I were to go this route.

Closing Thoughts

I have always had the itch to build a micro SaaS. So, I decided to build one realizing that it wouldn’t make me rich or change the world. I simply enjoy building stuff that people want to use and money is the ultimate validation for that.

However, building a SaaS was not exactly what I thought it would be — keeping my head down and coding away. I needed to understand my users and their needs and learn how to market my product effectively.

I must admit, marketing is not something I truly enjoy. But, I realized that it's a necessary job that comes with building a product.

Overall, building Tournacat was a rewarding experience. Seeing Tournacat slowly grow and gain traction has been an exciting journey. I'm eager to continue on this journey and I look forward to what the future holds.

Can you use else in a for loop in Python? The answer is a resounding yes! I know it sounds crazy. The use of else after for and while loops in Python is not a widely known idiom. The use of the for-else construct is often misunderstood and overlooked.

For those who are coming from other programming languages, the syntax may be jarring. The fault is not on them though, the else clause is mostly associated only with the if statement.

But hold on, don’t jump to conclusions just yet! The else clause in loops may seem strange at first, but it can be a useful tool in your Python toolkit.

TL;DR

• Think of else in a for-else construct as “no break”.
• Consider using else with for or while loops instead of a flag variable to handle function break out of the loop.
• An else clause is only useful with a preceding break in a loop. In other words, it’s pointless to use the for-else construct without a break statement.

What is for else in Python

Even today, the use of the for-else construct in Python remains largely unpopular. It often confuses even seasoned Python programmers.

Here’s a trick to remember — the else in a for-else construct can be remembered as “no break”.

It basically handles cases where no break statement is being executed within a loop. Easy, right?

But, when would you use the for-else construct in Python?

When and how to use else in a for loop

A common use case is to go through a loop until something is found. When we find what we want, we would then break out of the loop and handle that case accordingly.

In short, we need to determine which case has happened:

1. We found the target (break out early)
2. We finished the loop without finding the target

One common method is to create a flag variable that will let us do a check later to see how the loop was exited.

Now, let’s take a look at an example. Say you want to find the index of the first occurrence of a target in a sequence. You can do this using a flag variable:

def find(seq, target):
    """Find the index of the first occurrence of `target` in `seq`."""

    found = False  # a flag variable
    for i, value in enumerate(seq):
        if value == target:
            found = True
            break

    if not found:
        return -1

    return i

Although this example is a little bit simplistic, code like this sometimes intermesh with other more complex code and there’s no shortcut out (i.e. can’t return the result immediately).

However, using the for-else construct, we can instead have:

def find(seq, target):
    """Find the index of the first occurrence of `target` in `seq`."""

    for i, value in enumerate(seq):
        if value == target:
            break

    else:  # no break
        return -1
    return i

Perhaps looking more elegant, this can simplify your code and improve readability instead of using a flag variable with an additional if check.

In short, the else clause in a for loop can be used to execute a block of code when the loop has finished running without encountering a break statement.

For fun: benchmark

As I was down this rabbit hole, I got a bit curious about how these 4 different methods would perform. With some semantical differences, these functions can be used to find the index of the first occurrence of a character within a string.

Here’s what I ran:

import cProfile

# finds the index of a target in a sequence with a flag variable
cProfile.run("""
def flag_var_find(seq, target):    
    found = False
    for i, value in enumerate(seq):
        if value == target:
            found = True
            break
    
    if not found:
        return -1
    return i

flag_var_find('a' * 100_000_000, 'b')
""")

# finds the index of a target in a sequence using a for-else construct
cProfile.run("""
def else_find(seq, target):
    for i, value in enumerate(seq):
        if value == target:
            break

    else:
        return -1
    return i
else_find('a' * 100_000_000, 'b')
""")

# finds the index of a target in a sequence using a generator expression
cProfile.run("""
next((i for i, char in enumerate('a' * 100_000_000) if char == 'b'), -1)
""")

# find the index of the first occurrence of a substring within a string
cProfile.run("""
('a' * 100_000_000).find('b')
""")

Here are the results:

      4 function calls in 6.462 seconds

Ordered by: standard name

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      1    0.019    0.019    6.462    6.462 <string>:1(<module>)
      1    6.443    6.443    6.443    6.443 <string>:2(flag_var_find)
      1    0.000    0.000    6.462    6.462 {built-in method builtins.exec}
      1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

      4 function calls in 5.973 seconds

Ordered by: standard name

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      1    0.008    0.008    5.973    5.973 <string>:1(<module>)
      1    5.965    5.965    5.965    5.965 <string>:2(else_find)
      1    0.000    0.000    5.973    5.973 {built-in method builtins.exec}
      1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

      5 function calls in 5.763 seconds

Ordered by: standard name

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      1    0.007    0.007    5.763    5.763 <string>:1(<module>)
      1    5.756    5.756    5.756    5.756 <string>:2(<genexpr>)
      1    0.000    0.000    5.763    5.763 {built-in method builtins.exec}
      1    0.000    0.000    5.756    5.756 {built-in method builtins.next}
      1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

      4 function calls in 0.012 seconds

Ordered by: standard name

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      1    0.007    0.007    0.012    0.012 <string>:1(<module>)
      1    0.000    0.000    0.012    0.012 {built-in method builtins.exec}
      1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
      1    0.004    0.004    0.004    0.004 {method 'find' of 'str' objects}

Subtly, we can see that using the for-else construct is just a tiny bit faster than the flag variable way.

On the other hand, using Python’s built-in string find method is significantly faster for this specific question. However, it will not work in the cases of other Sequence Types like Lists or Tuples.

Using else with while loops

Here’s a version of the same find function using a while loop instead of a for loop:

def find(seq, target):
    """Find the index of the first occurrence of `target` in `seq`."""

    i = 0
    while i < len(seq):
        if seq[i] == target:
            return i
        i += 1
    else:
        return -1

When not to use the else clause in a loop

Simple. It’s pointless to use a for-else construct without a preceding break statement in a loop.

How to use else with a try-except block

Loops aside, the else clause can also be used with a try-except block to run code when no exceptions were raised in the try block.

Similar to the for-else construct, the try-else construct is useful to distinguish between:

1. The code ran successfully without any exception
2. The code ran into an exception

try:
	book = find_book(id=1)

except BookNotFoundError:
    # handle the exception
	print("Book not found! :(")

else:
    # run this block if no exception was raised
	send_slack_notification(book)

finally: 
	# this block is always executed
	close_db_connection()

In this case, the code in the else block will only run if the code in the try block was executed successfully, without any exceptions.

In summary, consider using the try-else construct to separate the normal execution path from the error handling code.

Do not confuse else with finally in try blocks

• try-else — executes if there’s NO exception
• try-finally — always gets executed regardless of exception

What’s Next

To summarize, the else clause in Python provides a way to execute code after a loop has completed execution without encountering a break statement.

I owe my thanks to this great talk by Raymond Hettinger, titled “Transforming Code into Beautiful, Idiomatic Python”. In the talk, he briefly spoke about the history of the for-else construct. I highly recommend watching it — I’ve been using Python for 5+ years and I still learn many great tidbits in such a short time from him.

I personally would use the for-else construct whenever it makes sense. Having that said, I would recommend trying to get your team to be on the same page before adopting this widely unpopular construct.

Lastly, if you’re interested to dive deeper into this rabbit hole, check out [Python-ideas] Summary of for...else threads.

I hopped on the DuckDuckGo (DDG) hype train some years ago. It lasted for about 5 months. Unfortunately, I can’t help to be constantly reminded how good Google site search is. A story probably for another day.

While DDG did not become my default search engine, there’s one feature that changes how I look at browsers and search engines’ user experience (UX) forever — DuckDuckGo Bang.

Trust me, people can go on quite a bit about the Bang (!) syntax in DDG. The address bar has become my main entry point to the Internet these days.

TL;DR

Replicate DDG's bang syntax in the Chrome address bar at chrome://settings/searchEngines and add new search engines under “Site Search”.

Bangs

In DDG, Bang is nice as a quick shortcut that takes you to search results of other sites.

For example, if I enter !r m1 macbook review on DDG’s search, it takes me directly to Reddit search at https://www.reddit.com/search?q=m1%20macbook%20review.

It's a great way to cut through the noise and find the information that we need.

Google was my first stop

Whenever I had to research anything, I’d just simply Google it without any fancy search operators. Anything from gadget reviews to technical searches like “how to remove duplicates in a list in python”.

But as of late, I find myself increasingly appending site:reddit.com, site:github.com/user/repo, site:stackoverflow.com, etc. to my search query to get more refined and meaningful results.

Doing so provides a relatively unbiased set of results that isn’t tied to my browsing habit. Getting to reviews that are hopefully written by actual users.

How to Google With a Bang

If you use Chrome, you can craft a similar UX as DDG Bang syntax. There’s no need to install any extensions. It’s available natively on your Chrome browser.

Here's an example:

Here’s how:

1. Go to chrome://settings/searchEngines and add new search engines under “Site Search”.
2. Edit the search engine accordingly.

• The “Search engine” field is a way to name your “Shortcut”, it can be anything.
• Note that the example above uses the Google search engine.
• Alternatively, if you want to use the site’s search engine, change the “URL with %s in place of query” to the site’s search query. E.g. https://www.reddit.com/search?q=%s to search using Reddit’s search instead.
• You can even prepend site:reddit.com/r/<subreddit> for a more refined search.

Example for appending “site:reddit.com” when prefixing “!r” shortcut to your search term

3. Repeat.

Some ideas

• !r for reddit.com — product reviews and community-specific information.
• !so for stackoverflow.com — software engineering, programming-related topics.
• !p for phind.com – Search engine optimized for developers. Add https://phind.com/search?q=%s and you're set!
• !gh for github.com — libraries or framework-specific GitHub issues (e.g. bugs) and workarounds. This has been incredibly useful to me.
• !hn for hackernews.com —  I use this to look for discussions or anecdotes on specific topics.
• !yt for youtube.com — quicker (1-step) video search on YouTube.

The possibilities are endless. Refer to duckduckgo.com/bangs for more ideas.

Useful resources

• How to search on Google
• Refined web searches

Closing Thoughts

Over the years, the quality of the top search results from Google has deteriorated:

• The first page is often filled with basic information that barely scratches the surface.
• When it comes to software engineering or programming-related topics, it’s the same problem. Outside of Stack Overflow, most articles are written to appeal to a broad audience.
• SEO (page ranking) has become increasingly gamified. It probably killed all kinds of real reviews. That’s why people have been gradually appending site:reddit.com to their Google searches because Reddit (and HN) is one of the few places online you can read actual human thoughts.

I know I am not the only one that feels this way. A “normal” Google search now does not yield satisfactory results anymore.

Having said that, Google search is still very much superior in my opinion — mostly because of Google Maps.

It's funny how we often take for granted the power of search engines, and how we drown in the sea of information that's available to us today.

So, you have heard of the Global Interpreter Lock, or GIL for short? What exactly is GIL? Well, in short GIL is a mechanism that is built into the CPython interpreter. It is designed to prevent multiple native threads from executing Python bytecodes at once.

“Blah-blah-blah, stop trying to sound smart! What the hell are CPython and bytecode?!”. Hopefully, I haven’t lost you yet at this point.

Before we start talking about GIL, let me briefly introduce to you what CPython and bytecode are to understand the typical answers about GIL that you read on the Internet.

Don’t worry, it’s not as scary as it sounds! Here's my GIL ELI5 take.

What is CPython

Just like how there are many different brands of soda (Coke, Pepsi, Mountain Dew, etc.), there are also many different versions of Python that you can use to write and run your programs.

CPython is like the "original" or "standard" version of the Python programming language. You’re probably using it every day!

Reference implementation

Written in C, CPython is called the "reference implementation" of Python, which means that it's the version that other versions of Python are compared to and tested against.

It's a bit like how Coke is the original soda that other sodas are based on or try to imitate.

Not actually compiling to C

To be clear, CPython does not translate your Python code to C. The Cython project on the other hand lets you compile your code to C.

Alternatives

Several alternative implementations of Python are built to address different needs or to improve the performance of CPython in certain areas.

A non-exhaustive list of alternative implementations of Python includes:

1. PyPy: a fast, compliant implementation of Python that is built using a just-in-time compiler. It is often faster than CPython, particularly for larger programs.
2. Jython: an implementation of Python that is written in Java and can be used to run Python code on the Java Virtual Machine (JVM). This allows Python programs to integrate with Java programs and libraries.
3. IronPython: an implementation of Python built using the .NET framework and designed to be used with other .NET languages such as C#.

What is bytecode

Are you ready to learn about the mysterious world of Python bytecodes? Buckle up!

When you write a Python program and run it, the Python interpreter first compiles the source code into bytecodes and then executes the bytecodes to produce the desired output.

Soda analogy

Imagine that you have a recipe for making the perfect soda. The recipe is written in a language that you can understand, like English.

But when you go to make the soda, you don't actually follow the recipe in English. You translate it into a series of instructions that your soda machine can understand – these instructions are the "bytecodes".

Came across .pyc?

In Python, bytecodes are a lower-level representation of the source code of a Python program.

They are typically stored in files with an .pyc extension, which stands for "Python compiled code". These files are created automatically by the Python interpreter when it encounters a .py file that it needs to execute.

They are used to speed up the execution of Python programs by avoiding the need to recompile the source code each time the program is run.

So that's bytecode in a nutshell! It's a set of instructions that tells your computer what to do, kind of like a recipe for soda.

Just don't try to drink your bytecode — that would be a bit weird.

Why is GIL necessary

Well, you see, CPython wasn’t designed to be thread-safe. In other words, multiple threads can potentially interfere with one another. This is bad because it can lead to something called “race condition”.

The GIL makes our life easier as Python developers to write multi-threaded programs without worrying about race conditions.

How do I use GIL?

In case you’re wondering — you don't need to do anything special to use the GIL in your Python code. It is implemented automatically in the CPython interpreter and is active by default.

However, you can use the threading module (docs) to create and manage threads in your Python code:

import threading


def make_soda():
    print("Making a delicious soda!")

if __name__ == "__main__":
    t1 = threading.Thread(target=make_soda)
    t2 = threading.Thread(target=make_soda)
    t3 = threading.Thread(target=make_soda)

    t1.start()
    t2.start()
    t3.start()

Race condition

Imagine that you and your friends are making a batch of soda together. You've got all the ingredients ready to go — sugar, water, caramel coloring, and some secret spices.

But you all try to add the ingredients to the same pot at the same time. This can lead to unexpected results because you don't know which ingredients will be mixed in first. This is similar to a race condition in a program, where two or more threads try to modify the same shared resource at the same time!

But because the GIL is in place, the other thread will have to wait for its turn before it can start making some fizzy goodness.

You may have already noticed — GIL may seem like a bottleneck because of this wait. But, it’s actually just there to keep things running smoothly.

Input/Output (I/O) Bound

It is important to note that GIL does not affect programs that rely on I/O operations as these operations typically release the GIL while waiting for data to be transferred.

What exactly do you mean by I/O bound?

At its most basic, I/O bound means that a program is waiting for I/O operations to complete. This can include things like:

1. Reading from or writing to a file
2. Sending or receiving data over a network
3. Interacting with a user through a graphical user interface (GUI), etc.

Essentially, any time a program has to wait for something to happen outside of itself, it's probably doing I/O-bound work.

The Debate

A TL;DR about all the fuss about the Global Interpreter Lock, or GIL.

Supporters

“The GIL is a great feature of Python! It makes it easy for us to write multi-threaded programs without having to worry about race conditions!” – says the supporters.

They argue that most Python programs aren’t CPU-intensive, to begin with. Most of the time, the programs are doing I/O-bound work which GIL doesn’t really affect. So, why all the fuss?

Critics

“The GIL is holding us back!” – say the critics with an opposite view.

With GIL, only one thread can execute Python bytecodes at a time.

As a result, this can limit the performance of CPU-bound Python programs on systems with multiple CPU cores. Because of this, GIL is known as a major limitation of Python.

Although most Python programs are I/O-bound, it doesn’t mean that we shouldn’t be able to write high-performance CPU-bound programs in Python.

Conclusion

So who's right? Well, as with most debates, there's no easy answer.

The GIL may be a necessary compromise that makes it easy for Python programmers to write multi-threaded programs. Nevertheless, it can also limit the performance of CPU-bound programs on systems with multiple CPU cores.

The debate lives on, for now, I guess. Lastly, check out this HN post and comments if you're interested to dive a little bit deeper into the wonderful world of GIL.

Decorators are incredibly powerful in Python. They are useful for creating modular and reusable code. Plus, they're pretty cool once you get the hang of them!

In this blog post, we'll take a look at what decorators are, why you should use them, and when you should and shouldn't use them.

As always, I’ll try to explain everything in layman's terms. My goal is to get you comfortable with decorators.

TL;DR

• Think of Python decorators as little "wrappers" that you can place around a function (or another object) to modify its behavior in some way.
• You would typically use decorators for logging, authentication, or to make a function run faster by caching its result.

What are Decorators

Let’s start with a simple explanation of what decorators are in Python.

A decorator is really just a function that takes in a function as an argument. It then returns a modified version of that function.

Here’s a classic example:

def my_decorator(func):
    def wrapper():
        print("Before the function is called.")
        func()
        print("After the function is called.")
    return wrapper

• In this example, the my_decorator function takes a function as an argument (which we'll call func) and defines a new inner function called wrapper.
• The wrapper function prints a message before and after calling the original func function.
• The my_decorator function then returns the wrapper function.

To use a decorator, we would apply it to a function using the @ symbol, e.g. @my_decorator. Here’s an example:

def my_decorator(func):
    ...

@my_decorator
def greet_ben():
    print("Hello, Ben!")

If we were to call greet_ben, the output would look like this:

Before the function is called.
Hello, Ben!
After the function is called.

See how we effectively modified greet_ben to print additional messages without changing the code of the original function itself?

Syntactic sugar

You see, the decorator syntax @my_decorator is merely syntactic sugar. The 2 following function definitions are semantically equivalent:

def my_decorator(func):
    ...

# Function definition 1
def greet_ben():
    ...
greet_ben = my_decorator(greet_ben)

# Function definition 2
@my_decorator
def greet_ben():
    ...

Interesting, right?

Python decorator template

Now, hold your thought for a second, that was not how we would typically write a decorator function in Python — there are better ways.

Instead, the following template can be used as a starting point for creating custom decorators:

from functools import wraps

def decorator_function(func):
    """
    A generic decorator function template.
    
    :param func: The function to be decorated.
    :return: The decorated function.
    """
    @wraps(func)
    def wrapper(*args, **kwargs):
        """
        The wrapper function that will be executed when the decorated function is called.
        
        :param *args: Positional arguments passed to the decorated function.
        :param **kwargs: Keyword arguments passed to the decorated function.
        :return: The result of the decorated function.
        """
        # Add any pre-function execution code here.
        # For example, logging, timing, or input validation.

        result = func(*args, **kwargs)  # Call the decorated function.

        # Add any post-function execution code here.
        # For example, logging, timing, or output validation.

        return result

    return wrapper

You can modify the pre-function and post-function execution code sections to implement the desired behavior for your specific use case.

Why do we need functools.wraps?

Using from functools import wraps and applying the @wraps(func) decorator to the wrapper function is not strictly necessary, but it provides some important benefits when creating decorators.

When you create a decorator without using wraps, the decorated function's metadata, such as its name, docstring, and module, are replaced by the metadata of the wrapper function. This can lead to confusion and make debugging more difficult.

Here's an example to illustrate the issue:

def decorator_function(func):
    def wrapper():
        func()
    return wrapper

@decorator_function
def my_function():
    """This is my_function's docstring."""
    pass

print(my_function.__name__)  # Output: 'wrapper'
print(my_function.__doc__)   # Output: None

As you can see, the name and docstring of my_function are lost when it's decorated without using wraps.

Now, let's use wraps:

from functools import wraps

def decorator_function(func):
    @wraps(func)
    def wrapper():
        func()
    return wrapper

@decorator_function
def my_function():
    """This is my_function's docstring."""
    pass

print(my_function.__name__)  # Output: 'my_function'
print(my_function.__doc__)   # Output: "This is my_function's docstring."

By using @wraps(func), the decorated function retains its original metadata, making it easier to understand and debug.

Overall, using functools.wraps is highly recommended when creating decorators in Python as it helps to create more reliable, maintainable, and compatible decorators that work well with other tools and libraries.

Why Use Decorators

Understanding the “why” is kind of like adding the icing on the cake — it just makes the whole thing a lot sweeter. So why use decorators at all?

Well, decorators provide a flexible and convenient way to add extra functionality to existing functions, making them more reusable and modular.

This is useful when you want to add a feature to a function that is used in multiple places in your code, but you don't want to modify the original function itself.

Avoid cluttering

For example, you might want to add logging or error handling to a function. But, you don't want to clutter the original function with that extra code.

By using a decorator, you can add extra functionality without changing the original function, making your code cleaner and more organized.

When to Use Decorators

Now that we know why we should use decorators, let's talk about when to use them.

Generally speaking, Python decorators are most useful when you want to add extra functionality to a function without modifying its code. This might include things like:

1. Logging
2. Error handling
3. Caching
4. Authentication (see example in Flask)
5. Timing a function execution (see an advanced example)
6. Backoff and retry

A caching example

Here's an example of how you might implement a simple cache for a Fibonacci function:

cache = {}

def fib(n):
    if n in cache:
        return cache[n]

    if n <= 1:
        return n

    result = fib(n - 1) + fib(n - 2)
    cache[n] = result
    return result

print(fib(100)) # 354224848179261915075

Notice that the caching code lives inside the fib function itself.

However, with decorators, we can simply define a caching decorator and use it to add caching functionality to our Fibonacci function:

cache = {}

def cache_results(func):
    def wrapper(n):
        if n in cache:
            return cache[n]

        result = func(n)
        cache[n] = result
        return result
    return wrapper

@cache_results
def fib(n):
    if n <= 1:
        return n
    return fib(n - 1) + fib(n - 2)

print(fib(100)) # 354224848179261915075

• Here, we've implemented our own simple cache using a dictionary.
• We've also defined our own decorator function cache_results that takes a function as an argument and returns a new function that wraps the original function.
• This inner function checks the cache to see if a result has already been calculated, and if so, it returns the cached result.
• Otherwise, it calculates the result and adds it to the cache before returning it.
• This way, we can apply the cache_results decorator to any function we want to add caching for.

While both examples work, the latter allows us to easily add caching to other functions without having to write the caching code inside each function.

This makes the caching code in our example more reusable and modular!

A better example with built-in modules!

Instead of reinventing the wheel, we can use the lru_cache decorator from the functools module to add caching to your fib function:

from functools import lru_cache

@lru_cache(maxsize=None)
def fib(n):
    if n <= 1:
        return n
    return fib(n - 1) + fib(n - 2)

print(fib(100)) # 354224848179261915075

• Using lru_cache decorator is really easy – all you have to do is add the decorator to your function and it automatically handles the caching for you.
• So, using lru_cache can be a great way to optimize your code without a lot of extra effort.

Again, this is just one example of how you can use decorators to make your code more efficient and elegant.

When NOT to Use Decorators

So, when should you avoid using decorators?

Well, decorators are great for keeping your code clean and easy to read, but they can make your code harder to understand if overused or used incorrectly.

In general, it's best to avoid using decorators if they make your code more complex than it needs to be, or if you're not sure how they work.

For instance, if you're working on a very small project and you don't need to keep track of how many times your functions are called, then you probably don't need to use a decorator.

In some cases, you may find that it might be more appropriate to modify the code of the original function directly, rather than using a decorator.

It's always a good idea to keep things simple and easy to understand!

Summary

In short, Python decorators are like little "wrappers" that you can place around a function (or another object) to modify its behavior in some way.

Decorators are useful for creating modular and reusable code in Python.

You can define a decorator once and then apply it to any number of functions. This allows you to easily add the same extra functionality to multiple functions without repeating yourself.

28 November 2022 was a sad day for developers. If you haven't heard, Salesforce (Heroku’s parent organization) has phased out its free tier plan on this date.

For many years, Heroku has been the de facto standard platform as a service (PaaS). So many students and developers deployed their first web application on Heroku. Anecdotally, Heroku has been pivotal for my career.

TL;DR, if you’re looking for Heroku free tier alternatives, check out free-for.dev/#/?id=paas. I migrated my Cron jobs to Northflank and Heroku Dynos to Koyeb.

A lot has happened

For context, a lot has happened in the year 2022 for Heroku. The two most notable events are:

1. On April 2022, Heroku had a security breach (incident) where CI and Review App secrets were compromised. GitHub Actions integrations on Heroku were down for a couple of months. Comms from Heroku were objectively bad. I experienced this firsthand when I had to resort to using a GitHub CI Action for my app deployments.
2. On August 2022, Heroku announced the removal of their free product plans.

If you have been keeping an eye on the Internet, you would see people expressing concerns about how Heroku is losing its magic. Ever since the security breach incident, I see more talks about Heroku alternatives taking place.

Though things started to look bad, whatever Heroku had to offer was still great and I decided to stick to it. Until now, that is.

Heroku Alternatives

For the past couple of weeks, I have been moving my demos and tiny projects out of Heroku.

If you’re still looking for Heroku free tier alternatives, check out free-for.dev (GitHub).

While there are a bunch of blog posts and recommendations scattered on the Internet now, free-for.dev provides the best coverage. The list comes with a brief description of what each Heroku alternative does.

What I was looking for

Do note that my use case is mostly for small-scale personal projects. So, whatever I have written may not be ideal for your use case.

1. Equivalent free tiers in terms of CPU and RAM. (spoiler: none of them were as good)
2. Support for Cron jobs out of the box. Ideally, no weird workaround is required
3. Custom domain
4. GitHub integration
5. Support for basic logging and monitoring (i.e. CPU, memory, disk, network metrics)

The bandwidth cost and supported region of each PaaS were not my primary concerns.

My picks

Heroku Postgres — since May 2022, I have migrated from Heroku Postgres to Railway. So far, I have no complaints about it. Another alternative that I was looking at was PlanetScale. However, I didn’t go for it because it isn’t Postgres-compatible.

Heroku Dynos apps — Burplist was migrated to Koyeb. Having tried out other notable Heroku alternatives like Fly.io, Northflank, and Railway, I can safely say that the migration from Heroku to Koyeb required the least amount of effort and kinks. It just works in my case.

Heroku Scheduler — for my Cron jobs, I opted for Northflank. Their support for Cron jobs is by far the best. Pricing aside, the developer experience is much better than Heroku Scheduler.

Heroku Add-ons — not applicable to my case. I was relying on Heroku Add-ons for monitoring and logging. Thankfully most of the modern PaaS today support that out of the box.

Some thoughts

Woefully, most Heroku alternatives’ free plans aren’t as good as Heroku. For example, most of the free compute instances provided are 1 shared CPU and 256 MB RAM (Heroku Free Dynos started at 512 MB RAM). Not considering the limited number of apps allowed yet.

To my surprise, most of these PaaS still don’t support running Cron jobs natively. Shoutout to Northflank again for providing such capability with great developer experience.

For Koyeb, the supported regions for free tiers are quite limited at the time of writing this.

Lastly, render.com is another popular alternative out there in the market. You may want to check them out.

Cost Comparisons

Updated as of 1 Dec 2022.

Free tiers

Pricing

Free tiers aside, most of these Heroku alternatives bill you for the resources you use by the second/minute/hour.

One thing that I did not cover here is bandwidth cost (inbound + outbound). This is something you might want to seriously consider for bigger projects.

They feel… different

Perhaps I am biased. There aren’t any obvious drop-in replacements for Heroku free tiers (at least not on par with Heroku’s generosity). Throwing free plans aside, most of them don’t offer the same kind of developer experience as Heroku.

Heroku free tiers are one of the best things that have ever happened to software engineering in my opinion.

I really appreciated Heroku’s free dyno hours. I didn’t mind the fact that dynos goes to sleep and only gets woken up when it’s needed — this means I could host multiple demos at once and only consume my free limits on demand.

Need to show something to someone quickly? Simply send them your URL. It’ll be up in a couple of seconds and I am totally cool with the cold start. This developer experience is something all the existing platforms cannot give, at least not with their free plans.

While I’m a happy paying customer, it stings to pay $5/month for something that I do not use 99% of the time.

Final Thoughts

Sadly, it is how it is. The Internet is a gnarly place. If you are offering free service on the Internet, people will find a way to abuse it. I remember hosting my first URL shortener. On my first day of making it public, the service immediately got spammed. I had to implement CAPTCHA and expiring links to mitigate abuse.

Today, whenever I see something available for “truly free” (if you know what I mean) created by individuals, I can’t help but wonder how are they going to keep the lights on sustainably out of goodwill. I just hate to see someone’s good intentions fail. Maybe we should start paying for Internet stuff.

Though I use cloud services like AWS almost every day, PaaS like Heroku always holds a special place in my heart.

Heroku was amazing. I think it did a great job of raising the bar in the PaaS field besides indirectly advocating free education (in one way or another).

Hats off to you Heroku.