devnotes on Roy Tang

DevNotes: Python's yield

hello@roytang.net (Roy Tang) — Thu, 07 Nov 2019 00:00:00 +0000

I’ve been using Python for well over 10 years, and I still don’t have an intuitive mastery of one of its keywords: yield. Everytime I see it in someone’s code I need to stop and mentally remind myself what it does. I figured I’d write a devnote to help improve my recall.

Typically, yield is used in a function with a loop, like so:

def some_func(lim):
    for i in range(0, lim):
        yield i

yield means the function returns a “generator” that can be used as an iterable in a loop:

for val in some_func(5):
    print(val)

You can also straight up convert the generator into a list via something like list(some_func(5)).

The equivalent function, without using yield would be something like:

def some_func(lim):
    result = []
    for i in range(0, lim):
        result.append(i)
    return result

This is definitely the kind of function I’ve written often! Now that I’ve actually written it down, yield is rather straightforward, and hopefully it can help me shorten some of the python code I’ll be writing in the future!

Devnotes: Python Pathlib

hello@roytang.net (Roy Tang) — Tue, 27 Aug 2019 00:00:00 +0000

Ever since I started learning Python back in 2008ish, I’ve been using it as my primary scripting language for various tasks such as processing log files, organizing my own file system, processing stuff on this blog, and so on. A lot of it is basically moving files around. In the days of Python 2, that involved a lot of imports of different libraries like os, shutil and glob. It can become a bit messy with so many imports, and I often can’t remember which import I need for a particular case and end up having to search for the documentation (or stackoverflow, let’s not kid ourselves here).

With Python 3, a new cleaner option is available to replace all of the above libraries: Pathlib, introduced in Python 3.4, provides an object-oriented way of doing file operations, replacing many of the most common uses I had for the libraries above. Some sample usage:

from pathlib import path

# Declaring a path object is just passing the path string
p = Path("/home/roytang/stuff")

# Other paths can be derived using the / operator
subdir = p / "favorites"

# p.glob replaces stuff like os.walk and glob.glob
for mdfile in p.glob("**/*.md"):
    print(str(mdfile))

    # path objects have properties for individual path elements
    filename = mdfile.name
    stem = mdfile.stem # filename without extension

    # path.exists() replaces os.exists()
    newdir = subdir / stem 
    if not newdir.exists():
        # path.mkdir replaces os.makedirs
        newdir.mkdir(parents=True)

    newfile = newdir / newfile # path objects can be either files or dirs
    # for file copying, you still need shutil!
    import shutil
    shutil.copy(str(mdfile), str(newfile))

For me it’s much cleaner to use than the old methods and require less imports (most of the time). I eventually hope to migrate all my older scripts to use Pathlib (the same way I migrated them away from Python 2 a while back), and moving forward I plan to use it primarily for filesystem operations.

Devnotes: Migrating Mercurial to Git

hello@roytang.net (Roy Tang) — Sat, 24 Aug 2019 00:00:00 +0000

Big news in online repositories this week is that Bitbucket is sunsetting support for Mercurial! This might be the death knell for Mercurial, although Git was already the super popular choice before. Back when I started using online source control for my personal coding projects I started out with Bitbucket over Github because they offered unlimited private repos and Mercurial (which I had already tried out before at work, so at first I preferred it over git). Now that Gitlab and Github both offer unlimited private repos, there’s no reason to stick with Bitbucket either. I had already migrated most of my private Git repos to Gitlab before, but hadn’t realized until now that I also had a couple of Mercurial repos there that needed to be migrated as well. I hadn’t touched those repos in so long that I didn’t even have Mercurial installed locally anymore! (Although I’m still using the code locally!)

Luckily, converting a mercurial repo to git turned out to be straightforward thanks to a project called fast-export. Modified instructions from a convenient stackoverflow post:

brew install hg # install hg on my mac first
cd ~
hg clone https://user@bitbucket.org/user/reponame
git clone git://repo.or.cz/fast-export.git
git init git_repo
cd git_repo
~/fast-export/hg-fast-export.sh -r ~/reponame
git checkout HEAD

The process was fairly quick, and the commit history from the Mercurial side was even retained, I would have been happy just retaining HEAD! After that, I just create a new private repo on Gitlab, set that as the remote, then push to Gitlab as usual.

It’s a good thing I remembered to check my Bitbucket account, as apparently they will be deleting the old Mercurial repos when time is up!

Devnotes: TT Miniproject (Django Rest Framework, Unit Testing, VueJS, Geocoding, Nightwatch e2e Testing)

hello@roytang.net (Roy Tang) — Tue, 20 Aug 2019 00:00:00 +0000

I recently found myself doing a really small project as sort of a proof of concept/demo for a potential client. It often seems that it might be a waste of time to do something like this since you don’t know if the project will actually push through or maybe the client will want something else. To kind of hedge my bets a bit, I decided to take the opportunity to try out some new technologies so that no matter what I at least learned something from all of this. (What is life if not learning?)

Django Rest Framework. I’ve been on projects using DRF before, but it’s my first time building a full CRUD-style API from scratch using it. There is a little bit of a learning curve when it comes to views and serializers, but for the most part it’s straightforward. I did enjoy how the framework has out of the box support for the most common REST API scenarios, and there are available backends for different authentication methods. I used JWT Authentication, which isn’t part of the library by default, but already had a pretty good implementation available elsewhere.

Python Unit Testing. I have dabbled a little bit with Django’s Unit Test framework before, but not having worked with any company that does TDD I’ve never tried doing it the “test first” way. For reference I used an online book: Test Driven Development with DJango. I actually enjoyed the exercise, perhaps a little too much. My impression of TDD has always been that the value lies in how much courage it can give the developer - the courage to try new things without worrying about whether you have broken something that was working before. This support came in handy while I was trying to figure out some issues with DRF especially around permissions etc. Really helpful for backend APIs like this.

Python Social Auth. The proof-of-concept needed some social media logins. I used Python Social Auth for this, really straightforward library with good documentation and supports a lot of backends. I ended up using Twitter and Github OAuth as those were the simplest to implement.

VueJS. The proof-of-concept needed to be a single-page app, so that meant a bunch of Javascript was needed. I willingly admit I’m still behind with the latest web frontend frameworks and trends. In my head I’m still living in a jQuery world. I dabbled a bit with React when trying out React Native, but I wasn’t fond of it. For this project I decided to use VueJS since I had skimmed through their documentation before and things seemed cool. I used it mostly for the dynamic parts and the data binding. I didn’t get too far into using components, my VueJS app was a huge monolith, which is apparently is frowned upon when using these frameworks lol. (Someone familiar with VueJS who looked at the code said it was “very strange”). I have barely scratched the surface of this framework, I will probably need to build something a bit more involved with it to get a better grasp. I will say that I enjoyed the VueJS syntax/structure better than React.

Google Maps API and Geocoding. The miniproject had some components that required me to show stuff on a map and do some geocoding, so I had to set up the API for that. Google’s documentation is fairly good here. One thing I didn’t expect is that you need a Google Cloud account with a billing method in order to access the API. My understanding is I wouldn’t be charged for anything since I was just using the free tier, I hope that was accurate.

Nightwatch e2e Testing. I’ve had some projects with end to end testing before, mostly via scripts or Selenium, but it’s my first time trying out one of these end to end testing frameworks by writing the scripts myself. I chose to go with Nightwatch, it was reasonably strightforward and worked on Mac and Windows. One thing I will note that initially gave me problems during set up is that using the ChromeDriver can be a bit finnicky on Windows. You have to specify the actual location of the executable in your nightwatch.json, like so: "server_path": "node_modules/chromedriver/lib/chromedriver/chromedriver.exe", instead of just the node_modules/.bin/chromedriver the documentation implies.

All in all the mini-project didn’t take much work - maybe 2-3 mandays of effort at most, and I did learn a lot from it, so no matter the outcome it was already a win.

Devnotes: PostgreSQL on the command line

hello@roytang.net (Roy Tang) — Thu, 08 Aug 2019 00:00:00 +0000

I decided to start doing small “devnotes” on developer stuff I’m doing so I can refer to them later (and also because I feel like I could use more technical content on this blog)

Today is about PostgreSQL. I haven’t used it much beyond standard ANSI sql stuff. You won’t always have a graphical interface to access your database, sometimes you need to ssh to prod and query the database from the shell.

The command line for PostgreSQL is psql. You can do:

psql [database-user-name] -d [database-name] and it should prompt you for your password.

But when I tried this today, I got:

psql: FATAL: Peer authentication failed for user "[database-user-name]"

The problem had something to do with the permissions available to my ssh user. The workaround was to tell psql I was doing this from the local machine:

psql: FATAL: Peer authentication failed for user "[database-user-name]" -h 127.0.0.1

Worked fine.

After connecting, you get a prompt that looks like this:

[database-name]=#

Now, when I connected today, I had to do some queries, but I wasn’t superfamiliar with the project’s schema, and would have to muck around to find table names and field names. The psql shell provides some handy shortcuts for that:

\dt - outputs a list of tables

\d [table-name] - describes a specific table, showing field names

You can also just directly run an SQL query, just make sure to end it with a semicolon.

If the command outputs a lot of records, it will show them to you in a page-able vim-like format. You can recognize it when it stops with a : after each page and you can hit spacebar to proceed, and at the end it shows (END). And like vim, you might not know how to get out of this view. I had to google it myself, you just have to hit q to go back to the psql shell.