software-development on Roy Tang

Breaking Prod

hello@roytang.net (Roy Tang) — Wed, 23 Jun 2021 13:18:54 +0000

HBO Max caused a bit of a stir among its subscribers last week when they accidentally sent out an email titled “Integration Test Email #1” to some large number of their subscribers, which spawned a lot of clever commentary and snarky remarks on Twitter. A few days later HBO Max sent out an apology blaming an unnamed intern for the boo-boo, which spawned a lot of “Dear Intern” tweets with many commiserating with the unnamed intern by sharing their own stories of mishaps from their younger days.

There was also a thread of other developers and their stories of how they broke production too. One of my favorite stories has to be from Patrick McKenzie about his worst production bug while he was the one-man engineering team for his startup.

There’s a bit of an adage among programmers on the internet: There are two kinds of developers. Those who have broken production and those who are about to.

In my long career, I’ve either been lucky enough to have never caused any big production issues (either that or I’ve blocked out their memory somehow.) Part of it was probably because when I was a junior developer they almost never gave me access to production (like a sensible company!) Not to mention there’s always been several layers of reviews, testing and QA between me and production. And by the time I was senior enough to be handling production myself, I knew enough to be super careful with everything. Not that I didn’t work with production problems of course, I was just lucky enough to never be the direct cause of them myself. I think. But if the adage above is in any way true, should I be worried that eventually the law of averages will catch up to me, and maybe I should quite while I’m ahead?

Thoughts on Vue.js

hello@roytang.net (Roy Tang) — Sun, 23 May 2021 23:27:00 +0000

I’m not big on modern frontend JavaScript frameworks (mostly because I think web pages should use as little JS as possible), but when I do find the need to use one, my weapon of choice is Vue.js. I dislike React, but mostly because (a) my first experience with React was with mobile development using React Native for mobile app development, which I generally don’t like; and (b) I don’t like Facebook, which backs React. I have no opinion on Angular.

Despite being my frontend framework of choice, I don’t actually have that much experience with Vue beyond a few small projects each with a minimal number of screens / pages. That being said, I do have some thoughts, coming mostly from the POV of a developer more used to the old-school ways of Javascript and web development in general.

The entry point of course, is the getting started guide on the Vue website itself, which walks you through most of the framework features. I have no concern about most of these; the features are explained well and I find most of the documentation quite good.

My issue is with how the site recommends to beginners to set up their Vue projects: via a script include from a CDN. When I first saw this guide, I assumed this was the standard, recommended way to build a Vue project, so this was how I did it for my first small project, a small demo site I needed to produce in a few days. Due to the short timeframe, I had decided to learn as I go. The component registration section of the guide also gives no guidance or recommendation on how to structure your components or projects, so I ended up with most of the components set up in a single JS file. (Admittedly, as a person with more than a decade of web dev experience, I should have at least split it up into several files, but time was short.) A technical person reviewing the code commented that I had implemented the frontend “in a very strange manner”.

It gets worse when you try to install third-party components, which is a big feature of the ecosystem of these frontend frameworks. In my experience, most third-party components will only give instructions for installation via npm and a JS module system, neither of which are covered nor recommended by the beginner sections of the guide.

A few projects later, and I now know that I vastly prefer developing Vue projects as Node projects built using the scaffolding provided by Vue CLI, with the components implemented as standalone .vue files. The usage of these tools and methods are covered in the Installation section of the guide (which the “Get Started” link skips over) and the Single File Components section under the Tooling section (which seemed like an advanced section a beginner is not likely to get into.) This method also makes installing third-party components much more straightforward. It’s not even necessary to cover what tools like Webpack or Babel are doing, since those are largely abstracted away by the scaffolding.

Basically my main complaint is that the guide doesn’t do a good job of transitioning from the introducing the beginner to the language specifics to introducing a recommended approach and tools for organizing more intermediate or complex projects. The method of using a script include seems suitable for little more than a single-page todo list webapp. I suppose the perceived shortcoming might be attributed to my background at least partially - I still tend to think like a developer using jQuery, and I don’t default to thinking about Node applications when doing webapps.

That being said, I am overall happy with Vue.js and it’s still my choice of frontend framework moving forward (should a frontend framework even be needed!) Although, maybe I should at least consider trying out / evaluating Angular so I can pretend to make a fair comparison?

New note

hello@roytang.net (Roy Tang) — Wed, 21 Apr 2021 14:26:20 +0000

Me: Been using Python since 2008

Also me: Need to lookup how to use the filter function every time

(Note for future me: list = filter(func, list)

Programming is hard, but anyone can be a programmer

hello@roytang.net (Roy Tang) — Tue, 13 Apr 2021 05:25:58 +0000

dorinlazar.ro talks about programming being hard:

Most beginners in programming eventually end up with the same ingratiating message: „Programming is easy, everyone can do it”, with some threatening message that people doing the gatekeeping should stop doing that. I’m here to tell you that that is not true. Programming is hard, programming is not for everyone, and for the time being everyone might be able to do it, but most definitely most should not. Like most of the empty, shallow, positive messages coming with an automatic defense against refute, in this case talking about this gatekeeping thing that simply doesn’t happen.

I think gatekeeping is definitely a thing that happens. I should know, I used to hold the opinion that not everybody can be a programmer, because there are non-trivial skill sets that are necessary, such as attention to detail, ability to analyze and reason out possible outcomes, the ability to imagine abstractions in your head, and so on. Back when I did training for new hires, I could easily tell who are the people who wouldn’t pass, by simply noting who had trouble with things like SQL joins. (This was back in the day when we expected our devs to be able to handle everything from backend database stuff to web frontends with HTML/CSS/JS. I assume modern day frontend devs wouldn’t have much use for SQL usage…)

That being said, it’s not a hard gate, where someone can tell you it’s impossible to become a programmer. Anyone can be a programmer, in the same way that anyone can become an accountant or a lawyer or such: it’s going to take a nontrivial amount of effort and hard work (I’m not claiming that being a programmer is necessarily as difficult as those jobs!). Some people may make it look easy, because they may have a bit more aptitude than others (due to background or whatever), in the same way that some people find math easy and others find it nearly impossible. And a lot of the time it’s impossible to detect whether that aptitude exists until you’re trying it out.

There’s also a huge difference between learning to code just to write a few scripts in Python or some macros in Excel vs working professionally as a programmer, possibly in a team, with serious deadlines and crunch and all that. Not everyone is suited to that life.

I’m all for inclusivity - we definitely need more people working in tech, and for people who are struggling, learning to code may be a way for them to live better lives. But we shouldn’t sugarcoat the challenges involved or the effort required. “Anyone can learn to code”, given enough time and effort, but that doesn’t mean programming is easy.

HTML Tags Memory Test

hello@roytang.net (Roy Tang) — Sun, 20 Dec 2020 16:34:25 +0000

Shared via pocket: HTML Tags Memory Test

How Not to Mentor Software Developers

hello@roytang.net (Roy Tang) — Tue, 15 Dec 2020 00:00:00 +0000

I’ve talked a bit about mentoring in software development before, and how early on I used to get feedback about me being “intimidating”. I never got any concrete feedback regarding that, so I don’t know what problems I had specifically. Though I was recently reminded of one particular incident that was a bit cringey for me personally.

This was a bit more than 10 years ago I think. I was usually assigned to mentor new hires, which means fresh graduates who usually needed guidance with a Java (which was our primary programming language at that time) and with our in-house web framework, which could be a bit challenging to work with. That means a lot of my days are spent answering questions or guiding newbies through more complicated tasks.

The incident I remember was with one particular newbie. After they completed a particular task, they asked me to check their work and everything was fine. I don’t have the actual wording of what I said but recall making an offhand in-jest remark congratulating them on getting an entire thing done with zero questions and no actual issues. I remember thinking almost immediately afterwards that it was a terrible thing to say since it implies that asking questions might be frowned upon. It’s kind of like the difference between just saying “Good job” and “Good job, you didn’t screw this one up!”

I might have been overthinking it a bit back then as I don’t remember any actual negative reaction to that particular remark. That’s not really a good metric though, especially when you’re dealing with newbies on their first job. There’s power dynamics to consider, so they may not think it appropriate to complain even if something makes them feel bad or uncomfortable.

As a mentor, it’s important to actually encourage the mentees to ask questions when things are unclear or to raise conceerns when there are problems. It’s an essential communication skill that all developers (maybe arguably all people) should have. And it’s not enough to say “Hey, you can ask me questions any time”, but you also have to reflect that openness in your actions, your feedback, and your comments.

Server Setup Notes: Ubuntu, Nginx, Django, MySQL

hello@roytang.net (Roy Tang) — Thu, 05 Nov 2020 06:59:41 +0000

I recently did a server migration since I moved to new hosting, The move was from managed/shared hosting to a VPS, these are some notes I took during the process, which I figure might be helpful if I ever tried to do this again. (And maybe someone else finds it helpful too). Links and references to helpeful resources are included.

Setting up a webserver and WSGI container

I already knew I wanted to use Nginx (managed hosting on the old server always used Apache), that meant needing to choose a WSGI container for the Django apps. The choices were either gunicorn or uwsgi. Readings suggest minimal performance difference between the two. I did try setting up both, and found uwsgi’s setup much more straightforward and with more helpful step-by-step documentation: https://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html

Nginx itself was pretty straightforward to set up. This “common pitfalls” article was helpful in resolving issues: https://origin-www.wp.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/.

One thing I will note that’s not in the article above is that for static files to be served by Nginx, it was easier to just have everything be under /var/www, for permission purposes. I initially had static files in a different folder in my user root, but really there’s no reason for that since you’re going to serve them statically anyway.

Setting up PHP on Nginx

I needed PHP since I had a few minor scripts/pages and a Wordpress install that needed PHP. For the setup on Nginx, I found this guide from askubuntu super helpful: https://askubuntu.com/questions/134666/what-is-the-easiest-way-to-enable-php-on-nginx

This guide from DigitalOcean was also helpful: https://www.digitalocean.com/community/tutorials/how-to-install-linux-nginx-mysql-php-lemp-stack-in-ubuntu-16-04

I was encountering some issues with fastcgi not mapping requests to the correct PHP script if the Nginx location block was aliased. This was with me using the default snippets/fastcgi-php.conf that was provided with the fastcgi install. I resolved the issue by adding an additional fastcgi_param to the php location block like so:

    location /sample-path {
        alias /var/www/sample-path;
        # autoindex on;

        location ~ \.php$ {
                include snippets/fastcgi-php.conf;
                fastcgi_param  SCRIPT_FILENAME    $request_filename;
                fastcgi_pass unix:/run/php/php7.4-fpm.sock;
        }
    }

The solution for this came from the Nginx common pitfalls article listed above.

Wordpress-specific

(Added 1/21/2021)

Wordpress worked well enough for the most part. Except permalinks didn’t work! I had to add a directive to the nginx config file for unknown urls to be redirected to index.php:

try_files $uri $uri/ /index.php?$args;

Source: https://nginxlibrary.com/wordpress-permalinks/. It’s also mentioned in the official Wordpress docs , but buried in a bunch of other stuff so less helpful.

Choosing a database

I initially wanted to use PostgreSQL (since that’s what I was using locally) and set that up first. Then I realized I had a Wordpress install I wanted to carry over, and apparently Wordpress needs MySQL, so oops. I didn’t want to have two databases running, so that was out. Plan out your database usage boys.

Setting up MySQL for Django

I created the initial MySQL database with character set utf8, which apparently had the dumb implementation of only supporting 3 bytes for multibyte characters. This led to errors when I tried importing records into Django that had emojis and the like in text. Apparently the “correct” character set to use is utf8mb4 (mb4 -> “multibyte 4” I guess?). It’s kinda dumb that they had to invent a “more utf8 character set” because of their initial dumb implementation. It’s also dumb that I didn’t realize this when apparently the MySQL databases on the old server were already using utf8mb4! On the Django side, you need to specify the charset in the database settings. Helpful for resolving this issue was donturner’s answer on this SO question: https://stackoverflow.com/questions/2108824/

After some initial testing, I also encountered a “MySQL has gone away” every so often. Turns out Django was holding on to connections for too long, so a timeout had to be set in the database settings as well. SO reference: https://stackoverflow.com/questions/26958592/

Edit: I just encountered this a few hours after I wrote this post, when I tried to make the site display all times in Manila local time. Apparently, even if I set my Django app to be timezone, aware, additional setup needs to be done on the MySQL side so it knows what the actual timezones are (feels like they should really do this as part of the install!). Basically you need to run mysql_tzinfo_to_sql to load the OS timezone info into the database. This is in the Django documentation and the MySQL documentation.

Misc stuff

I used cron to set up most of the batch jobs I usually run (like the Twitter bots, etc). Initially I had problems where it seemed like cron wasn’t running the jobs, even though syslog showed me the cron daemon was actually running. I don’t have notes on how I resolved this, but mentioning it here so that if this happens again in the future, I know to actuall take notes!

Setting up log files for my cron jobs: I did it this way: 0 * * * * /path-to-batch/hourly.sh >> /var/log/cron/my-log-file.log 2>&1. The “2>&1” at the end is necessary so that stderr is redirected to the same file (otherwise you’d only get stdout).

Log rotation: these is the kind of thing you don’t realize you need to do when you’re managing your own server. Anyway, I found this guide helpful: https://linoxide.com/linux-how-to/setup-log-rotation-logrotate-ubuntu/

I needed to set up AWStats on the new server (and migrate data from the old server). This guide was helpful: https://larsee.com/blog/2020/06/awstats-on-ubuntu-20-04-with-nginx/

When migrating the old AWStats data files, the filename format from the oldserver was slightly different (new install needed to end with *.roytang.net.txt), so I learned a bit of bash scripting to do the batch rename:

for i in /path-to-tmp/awstats/*.txt ; do cp "$i" "/path-to-tmp/awstats/out/$(basename "$i" .txt).roytang.net.txt" ; done

Setting up Let’s Encrypt SSL on Nginx: https://www.nginx.com/blog/using-free-ssltls-certificates-from-lets-encrypt-with-nginx/

Lessons learned

One thing I didn’t expect from the migration from shared hosting to a VPS is how much memory I’d need. My old hosting plan was only on 512MB ram, and the lowest DigitalOcean droplet had 1GB RAM, so I figured that was fine! But MySQL occasionally had out of memory errors on startup even at 1GB ram, so I bumped up to the next level (2GB RAM). I guess the database RAM usage didn’t count against my limits on the shared hosting! The next tier isn’t much more expensive and is still manageable for my casual/hobby hosting purposes, but I will probably try to look into more detail into the memory situation at some point to see what I can optimize.

On Mobile App Development

hello@roytang.net (Roy Tang) — Thu, 03 Sep 2020 00:00:00 +0000

This post is just quite a few thoughts on mobile apps and mobile app development, all mishmashed together. I don’t claim to be a mobile app specialist, at best I’ve dabbled in them, but enough to form some opinions I guess?

A Bit of History

My first exposure to mobile app development when I got pulled to help my then-company’s then-fledgling mobile team with cleaning up the codebase for their iOS app. This was back in maybe 2011? It was one of those projects where some devs built a quick proof-of-concept demo using new technology, then management liked it and asked for new features and they kept adding new features haphazardly until there was just this huge unmaintainable mess of code. This isn’t a knock against Objective-C or iOS development specifically or the dev team; the fact is they were new to the stack and were learning as they went while still trying to meet deadlines. I also spent some time reviewing the corresponding Android team’s code as well. This was the bulk of my exposure to native mobile dev on iOS/Android. (This was before Swift, so I never got to experience that. Language looks neat though.)

Coming from a web dev background, I’ve always found the fragmentation of platforms annoying. Basically your effort gets multiplied by the number of platforms you have to support, and what would work well on iOS may not work well on Android and vice versa.

Later on, this concern was alleviated a bit with the rise of common frameworks that could publish to both iOS and Android. The first one I tried of these was React Native. I tried it for a small mobile project I did around 3-ish years ago. I wasn’t tremendously happy with the experience. My mistake was probably diving head-first into React Native, when this was my first experience with React and front-end frameworks in general (something new to someone with my jQuery background!), so a lot of concepts were new to me.

To make matters worse, the React Native docs at that time seemed to assume some familiarity already with React, or at least similar frameworks like Angular. I had a bit of confusion with whether I should be using Expo or not, and how to eject, and it resulted in having to restart the project a couple of times. It was also my first encounter with NPM’s dependency management, and that annoyed me a lot. A lot of the third-party dependencies I used back then were still immature too, so I often had problems with things like accessing the camera roll and so on.

Nevertheless, it was experience, and the ability to publish cross-platform to iOS and Android was invaluable. I went on to use React Native as my go-to mobile framework I would recommend for other projects afterwards.

Recently, I had to do a UI revamp for that small mobile app I wrote three years ago, and I hadn’t touched it for a while so I had some trouble with NPM trying to get it to run. In frustration, I gave up on it and decided to rewrite it in Flutter, of which I’ve heard good things. Flutter turned out to be great, and it took me just a little over two weeks, part-time to rewrite the app, probably slightly less effort than it took me to write it in the first place. I don’t have enough experience yet to have a detailed review of Flutter, but I was generally much happier with the experience compared to React. Part of that may be I’m much more familiar with the mobile landscape now and have a better idea of what I’m doing, but I really did encounter much fewer points of friction and Flutter seems to have much better documentation. I believe it will be my first choice of frameworks moving forward.

Not Everything Needs to Be a Native App

One of the things that annoys me about mobile apps is that people/companies unnecessarily want things to be native mobile apps when they would work perfectly fine as web apps. I can understand applications such as Uber may need numerous native functionality (GPS, timely notifications, offline functionality), so that’s fine (although many of those things are available using modern browser APIs as well!). Things like online newspapers, forums like Reddit, and so on, are perfectly fine as normal web apps, and should stop prompting me to install their native apps when they detect I am on mobile. Granted that you may get a much better UX experience with a native app, but you also have to install it locally and grant it more access than what the browser’s sandbox provides, and that feels like a tradeoff decision that the user needs to make, and not something to nag the user about constantly.

I guess part of it is that I grew up with the open web and have a significant bias towards it. In general, if I can just access an app via web, I will prefer that, unless the native app experience is vastly superior or otherwise necessary. And this focus on native apps can be bad for the web. This is apparent in apps like Instagram, which don’t allow you to use their primary function - uploading images - on a desktop browser. I also had a similar discussion with a freelance client a few years ago, when they wanted to make a mobile app, but not a corresponding web platform, when all of the app’s functionality was simple stuff that web could do easily.

Different Platforms

The differences in iOS and Android are particularly annoying to me as a developer, especially since I’m used to web development. With web, everything is open and standards-based, so you generally write everything using a single stack, and any reasonably standards-compliant web browser will work with the web app in a reasonably consistent way. (Granted, there will be browser-specific issues, but nowadays those aren’t too common.) With iOS and Android, you have to test on both platforms extensively, and the build processes and distribution methods are different for each. Frameworks like React Native and Flutter lighten this load considerably, but you still really have to devote more time and resources to testing on the different platforms.

Backward Compatibility

Okay, this one may be an unfounded concern of mine since I don’t have much experience in the matter, but I have this worry that native mobile apps tend to rot more easily, and by rot I mean, get worse with time, as the mobile OSes upgrade themselves very rapidly. Unlike, say, Windows, which gets a new major version every few years, iOS and Android are constantly getting new versions and more importantly, deprecating APIs and adding new ones! Hence my worry that unmaintained, a perfectly functional app may become unworkable a few years down the line and no longer work on the latest mobile OSes. It’s my totally unfounded belief that this is a significant reason why more than 50% of my lifetime App Store purchases are no longer listed on the App Store. (And also why I’ve pretty much disavowed making mobile purchases again.) Say what you will about Windows, but at least they have a pretty good record of backward compatibility when software I’ve written for Windows in the early 2000s is still very likely to run on Windows today.

Closed Platforms

Another issue I have with native apps is that I have been used to working on open platforms. Windows, for all its faults, is still an open platform, and anyone can write software for it and distribute it using whatever method they desire. And the web is even more open and standards-based (for now at least). For mobile, Android is mostly open, but iOS is mostly a closed platform.

Apple’s tight control over the iOS platform has been in the tech news recently. First was the kerfuffle with hey.com a few months ago, and more recently, their removal of Fortnite and battle with Epic due to Epic’s objection to Apple taking a cut of all purchases on iOS.

As a mobile developer, the most annoying things about Apple’s tight control over iOS apps are:

that $100 annual fee for an Apple Developer account. If you’re a 1st world developer, that cost isn’t much, but if you’re a young hobbyist developer in a poorer country who just wants to try making an app, it can be a significant hurdle. Google also charges you some token amount for Play Store access, but at least it’s just a one-time fee.
the fact that you need to use a Mac to publish apps to the App Store. This is another significant hurdle for hobbyist developers; Macs are typically very expensive computers. Granted Android development requires a beefy setup too, but at least you get a choice of platform for Android development.
App store review can be unpredictable; sometimes things that were perfectly fine during a previous release are now flagged in the next review. It feels like you need to get lucky with the reviewer you get. And if the reviewer is unfair to your app, you may be out of luck unless you’re a big company like Epic or Wordpress that can raise a stink about it.

Many of the issues that could be levied against Apple for their monopoly over the App Store could be largely addressed if the platform allowed users to install apps via alternative methods, the same way Android does. I fully understand that Apple’s tight controls enable a safer environment for naive end users (presumably, assuming they don’t screw up), and I actually approve some of their controls like their iOS14 plans to restrict targetted advertising. But more advanced users should be given the option to use alternative methods if they so choose. They can still flag apps installed via alternative methods as “potentially unsafe” and add hurdles to prevent less capable users from being easily tricked by malicious actors, (Android does this, I believe).

In conclusion

I don’t really have a point here. Maybe I just want everything to be open? I like open platforms lol.

On Mozilla and Firefox

hello@roytang.net (Roy Tang) — Tue, 18 Aug 2020 00:00:00 +0000

Mozilla made the tech news recently for laying off a whole lot of people. (Official statement). People were alarmed and worried about the future of what is the last major independent browser and the open web, bit it looks like it isn’t that bleak. Most of the layoffs were to teams other than those working on Firefox, things like the experimental browser engine Servo, devtools, and MDN. The core Gecko team seems to be unaffected.

Not that these things aren’t important. MDN, if you’re not familiar, is a set of documentation of web standards and browser support, available online, that is one of the best resources for web developers working on cross-platform things. Mozilla no longer funding it’s maintenance is a big deal, it means it’s less likely to be kept up-to-date, even though community work can still continue. Peter-Paul Koch, writer of quirksmode.org, which was the original resource for cross-browser compatibility back in the heyday of the browser wars, wrote an article about Mozilla’s problems titled “The Cult of the Free” where he argues that it’s time to step out of this mentality that all these resources must be maintained for free by “the community”, but that we as consumers must get used to the idea of paying or donating to support these resources. Expecting “the community” to step in and maintain these resources also devalues the work of technical writers. Trying to get more people to support these “free” resources is a good idea, but I worry it goes against the grain of years of internet culture which has conditioned people to expect online content for free.

Speaking of contributing, a while back I had considered looking for open source projects I could contribute to, and preferred projects that I actively use. Given that I’d been using Firefox since before version 1.0, one of the projects I considered was Firefox. I went through the whole rigmarole of their instructions to set up my machine for the Windows build, and just the setup process, which involved setting up and checking out code took more than an hour. That was fine for initial setup, but actually building and running Firefox itself took an additional hour or so. Might have been more than that even, it was a while back so I don’t clearly remember, but I do remember thinking that I didn’t have time for these hour-long build times. The whole set up also took up a nontrivial amount of my SSD disk space (imagine how much worse the build time would have been if I had used my non-SSD drive!), which I am always dangerously low on. These 2 factors led me to shelve the idea for the time being; perhaps I could revisit the idea of contributing to Firefox when I upgraded my computer or had more SSD space available. (TBF, I kind of gave up real quick after the first build, maybe the builds get faster on succeeding runs?). So that’s my personal “contributing-to-Firefox” story, for now.

The sheer size of the codebase and the long build times just drive home the point of how much modern browsers are these large complex beasts that are doing so much. They are pretty much an OS of their own since they host a ton of features and web applications and JIT JS and all of that. This means it’s really difficult for independent browser efforts to hit feature parity with modern web browsers - even just working with JS and CSS standards, there are so many to implement.

Drew DeVault laments that Web browsers need to stop with all these new functionalities and APIs that add scope and complexity, and maybe focus more on performance, efficiency, etc. I do agree - if I was contributing to Firefox, I would want to find a way to reduce the massive memory footprint it always has. I don’t even use that many tabs, and I already feel burdened by Firefox’s memory footprint. (Pretty sure Chrome is pretty bad in this regard as well.)

Why care about Mozilla/Firefox at all? Why not just use Chrome, it’s pretty good right?

First of all, no, Chrome is not necessarily “pretty good”, given how it props up a surveillance capitalsm engine.

Secondly, even if you didn’t dislike Google, having a browser (engine) monoculture is not a good thing. I got my start in web development at around the height of Internet Explorer 6’s dominance, and I’d rather not find my way back there. Granted, one of the issues back then was that IE6 was a stagnant platform that upstarts Firefox and eventually Chrome left behind. No such problem exists now, since Chrome is still in active development and still pushing new web standards forward. And the fact that Chrome is an open-source platform means in theory someone could just fork it if ever Google goes off the rails or such. (And people have already built indie browsers on top of Chrome of course!) But it’s still dangerous for a single company and/or codebase to be the sole major implementer of web standards. A single company dictating the future of the web isn’t a good thing, and is the very opposite of the idea of an “open” web.

Side note: Maybe I should try building Chromium too, just for comparison.

Anyway, that’s all my ranting for today. In conclusion: reduce your surveillance capitalism footprint and support the open web, use Firefox.

Using Hugo to Automatically Update Your Github Profile

hello@roytang.net (Roy Tang) — Sat, 11 Jul 2020 00:00:00 +0000

So recently Github secretly rolled out a new feature where you can create a file named README.md in a repo named github.com/<your github username>/<your github username>, and that markdown file would be rendered on your Github profile page. It means, you can now put basically anything you want on your Github profile!

I just read this post from Simon Willson about using this new feature + Github actions and a Python script to automatically generate and update his Github profile. That seemed neat so I thought about how I would do it.

My blog is already setup to publish using Hugo and Travis-CI whenever I push a commit, so I figured I could just ride on that framework. You can already see this in action on my Github profile page! Basically, I publish the README.md using my regular Hugo build, then my Travis script will deploy it to the appropriate repo.

If you’re new to Hugo, the trickiest part of this might be using a custom output format to generate the README.md. This is basically just a bunch of settings in your Hugo config file:

Define a new Markdown media type under mediaTypes in your config file. For a TOML config file, that looks like this:
```
    [mediaTypes]
        [mediaTypes."text/markdown"]
            suffixes = ["md"]
```
Define a “readme” output format (or whatever you want to name it) under outputFormats in your config file. For a TOML config file, that looks like this:
```
[outputFormats]
  [outputFormats.readme]
    baseName = "README"
    mediaType = "text/markdown"    
```
The baseName here controls the output file name, while the extension is controlled by the suffixes in step 1. The mediaType you use here should match the one in step
Next is to specify which pages should have this custom output format. I used my home page for this, so under outputs in the config file I added the name of the output format (“readme”) in my case under “home”:
```
[outputs]
  home = ["HTML", "RSS", "jsonfeed", "hfeed", "readme"]
```
Last step, in your theme or your layouts folder, create a file named index.README.md. This will be the template used to generate the README file. Remember that this is a template to generate markdown. You can use all the normal Hugo template functions, but your output should be markdown. Here’s part of what I put in my template, which generates a list of my most recent blog posts:
```
Latest blog posts:
{{ $pages := first 5 (where site.RegularPages "Type" "post") }}    
{{ range $pages }}
- [{{ .Title }}]({{ .Permalink }}){{ end }}

[View all posts]({{ .Permalink }}blog)
```

If you’re not yet too familiar with Hugo templates, you can maybe just use the above example directly first to see what it outputs. (You might need to replace post with posts or whatever your main post type is).

Once you have these setup, just run your usual hugo build and check the output folder to see if the README.md file was generated correctly. It should be in the root of the output folder, same place as the index.html for your home page. One problem I encountered with Hugo is that it insists on lowercasing all the output files, so the output file in my case is always readme.md and not README.md. I just solved this by renaming it during the Travis build.

I won’t discuss the travis deployment process anymore here; if you’ve done CI with Travis and Github integration before, it’s relatively straightforward to add a step to copy the readme file from the Hugo build output to the appropriate repository.

After setting this up, my Github profile page now automatically updates with my latest blog posts/notes/shared links! (I didn’t include photos to keep things simple). The update happens every time my site itself gets a new build, so that’s pretty neat. I’ve attached an image of the new Github profile page for reference!

Image Lightboxes using HTML/CSS

hello@roytang.net (Roy Tang) — Wed, 01 Jul 2020 00:00:00 +0000

In several places on this site (like if you click Photos in the menu up top), I have a grid-like view of a list of photos/images:

(Click to view full-size)

I used to just have each thumbnail open the post permalink on click, with the anchor set to the image itself. The image would be shown in full size inline of the post. This was a bit clunky and not so modern, so I decided to implement it so that the image lists instead will show a Lightbox-style overlay with the full image and some details and an option to click through to the post. That looks a little something like this:

(Click to view full-size)

Now, in the olden days doing this sort of thing required JavaScript (as described in the wikipedia link above to Lightbox). But for the modern web, this can be achieved using HTML and CSS only. The method is described in this blog post by jlelse, specifically for use in Hugo themes. The CSS technique is relatively straightforward: you generate two images, a thumbnail, and a hidden full-size one in an overlay. The thumbnail is wrapped in an anchor pointing to the id of the hidden overlay, and the overlay gets displayed via the :target CSS pseudo-selector.

Here’s how that looks in this site’s current Hugo templates:

( .context is the original image resource, and $thumb is the generated thumbnail)

<a href="#{{ .context.RelPermalink | md5 }}-lightbox">
    <figure>
      <img src="{{ $thumb.RelPermalink }}" alt="{{ $figcaption }}" 
        title="{{ $figcaption }}" class="tn" />
    </figure>
</a>
<div  class="lightbox" id="{{ .context.RelPermalink | md5 }}-lightbox">
  <a href="#_">
    <img src="{{ .context.RelPermalink | safeURL }}" />
  </a>
  <div class="lightbox_overlay">
    <p>{{ $figcaption }}</p>
    <time class="dt-published" datetime="{{ .postDate }}">{{ .postDate }}</time>
    <a href="{{ .mainPermalink }}">View post</a>
    <a href="#_">Close</a>
  </div>
</div>

And the relevant CSS (I don’t use SASS):

.lightbox {
	display: none;
	position: fixed;
	top: 0;
	left: 0;
	right: 0;
	bottom: 0;
	z-index: 999;
	width: 100vw;
	height: 100vh;
    background: rgba(0,0,0,0.8);
}
.lightbox_overlay {
    position: fixed;
    z-index: 1000;
    bottom: 0;
    width: 100%;
    background: rgba(0,0,0,0.6);
    padding-bottom: 1rem;
}
.lightbox_overlay * {
    padding: 1rem;
}
.lightbox img {
    position: absolute;
    top: 0;
    left: 0;
    right: 0;
    bottom: 0;
    margin: auto;
    max-width: 100%;
    max-height: 100%;
}
.lightbox:target {
    outline: none;
    display: block;
}

My only changes on top of the jlelse’s blog post linked above was to add a second overlay for a caption and date to be shown, and a link to the original post, and a close button. Clicking the post itself also dismisses the overlay (by replacing the URL fragment), but I thought it was more usable to also add an obvious Close button. I also initially wanted to put “Previous” and “Next” buttons so that you can easily cycle through the images in a post, but this was a bit more effort. I didn’t have a way to “look ahead” in the loop to get the “next” image, if any, so I think I would have needed to loop through the images twice? And since I use this image list in a number of places on the site, that was a bit troublesome, so maybe it’s something I’ll attempt later on?

In any case, I think the new lightboxes are neat!

Scraping Facebook

hello@roytang.net (Roy Tang) — Wed, 24 Jun 2020 00:00:00 +0000

I had been meaning to quit Facebook for more than a year maybe, but I kept putting it off. The main reason being that I like having backups of my own digital data (still very much a pack rat), and Facebook’s social media export is less than ideal, for me at least.

Less than ideal why? It doesn’t include a lot of content I would like backed up, including:

comments on my posts (there have been some good conversations with friends over the years I would prefer to preserve)
things I’ve reposted from other people
content of certain groups I’m members of (again, mostly for some interesting discussions over the years)
pictures posted by other people that I’ve been tagged in

For a while I looked around for scraper programs/scripts to this for me, but none really did what I wanted. Early this month I finally wrote my own scraper scripts to pull that data out of Facebook, as a supplement to the actual FB export. It seems unlikely that these will be helpful to anyone else, especially since web scraping is an inherently brittle activity; any slight change in the FB HTML generation will break the scripts. In spite of that, I thought I’d share them publicly anyway, via Github gists.

These are basically the same file, with minor changes between configuration and some scraping details. I wasn’t expecting to need these scripts again in the future so I didn’t bother with any kind of reusability.

Some other notes about these scripts:

made with Python! Using my favorite HTML parsing library, BeautifulSoup
the scripts run against the mobile web version m.facebook.com. The main web version is troublesome because it executes JS to do its infinite scroll and I didn’t want to waste time reverse engineering that
there are some variables near the start that you should probably set. Most of it is output folders and group details, but the main one is the FB login cookie - basically you need to login to FB, then use dev tools or whatever to copy your cookie string
the main output file is a gigantic JSON. There is no specified schema, basically I made things up as I went along. I’ll figure out how to process this at a later date lol.
As a secondary output, there is also a cache folder that should contain the raw HTML parsed for each page. These aren’t necessary to retain, as all the information they have should also be in the JSON. Images from posts are also downloaded into a dls folder. (The JSON will include the download path for the images from each post.)
running the timeline export on my machine took the better part of 2-3 hours, for roughly 13 years worth of FB content. The output JSON was around 22 MB.
running this might be considered spammy behavior by Facebook (understandably, but it’s their fault they won’t me do this via API), but the only consequence I’ve seen was that I wasn’t able to view other people’s profiles for some minutes after running the script

As for what I’m going to do with this data, that’s still to be determined. The nonpublic content can be imported into the archive on this blog (if it isn’t there yet), but that’s a lot of manual work. As for the scripts, hopefully I won’t have to use them again in the future, but maybe they can be useful to someone else!

Technical Interview Notes

hello@roytang.net (Roy Tang) — Tue, 28 Apr 2020 00:00:00 +0000

I’ve had the good fortune to be on the interviewer side of technical interviews much more often than I’ve been the interviewee. I’ve been doing a few more of these over the past couple of years and made some notes, so I thought I’d talk about technical interviews for a bit. Caveat: these are largely based on my own experiences, in the local environment here in the PH.

Technical Exam / Screening

Many companies will ask applicants to undertake a technical exam before letting them advance to further stages of the recruitment process. This kind of screening is most useful if your applicants are mostly fresh graduates or junior programmers. The primary purpose of such a screening is to make sure that the candidate is actually a programmer, and not just someone who graduated CS while letting all his groupmates handle the programming work. This is especially useful for companies that get a lot of applicants who are fresh graduates - it lets them filter the candidate pool without using up the valuable time of their senior technical staff. I talked about these kinds of technical screenings in a previous post about my own experiences as an interviewee.

As a sample statistic, I used to handle checking of technical exams for a company that had a few dozen applicants per week. The screening was a straightforward ten-question written exam, none of the questions required writing a lot of code. The questions were along the lines of “where is the problem in the loop?”, or “what is the output of this program?”, essentially confirming the applicant’s ability to understand how code is written. Even for this simple screening, roughly 8-9 out of 10 applicants would regularly fail the screening. (I am not sure if this speaks to the general low quality of graduates in my area instead of a general trend.) There is a reason FizzBuzz has become known as screening tool.

The screening can take different forms. It may be written, requiring you to come into their office, sit at a table and answer some number of programming-related questions on paper (as described in the paragraph above). This is the most common I have encountered, especially in larger companies here.

Some may be done remotely/online, via services such as Codility. This may be the best option, as it allows the candidate to take the exam on their own time, in the comfort of their own environment. Such environments will also automatically check the results, which means the effort for the recruiting company is at a minimum and the candidate gets feedback quickly. The downside is that not all companies may be able to afford such services.

The worst option is when they do a phone screening - someone asking you technical questions over the phone. Phone screening is the worst because there are no real technical questions that can be of any practical use in evaluating a candidate over the phone. Most good technical screening questions will require a lot of text (better on paper than verbally over the phone) or may otherwise have nuance in the possible answers. For the latter case, it’s probably fine if they have a senior developer doing the phone screenings, but few companies can afford that, so most likely it will be some HR person who was provided with a fixed set of answers where if you don’t have the exact answer they have on their cheat sheet, you won’t pass the screening. As much as possible I would suggest to avoid any companies that use phone screening, as it is an organizational smell.

Technical Interview

The technical interview is different from a technical screening in that it usually involves face-to-face time with one or more interviewer(s). You will be usually given one or more technical problems, usually of a higher difficulty than would be present in a technical exam/screening. Usually, you will be asked to provide a solution on paper or on a whiteboard, and would be expected to walk the interviewer(s) through your solution. The purpose of this stage is for the interviewer to see personally how you think through the problem and how you devise a solution and (to a lesser extent) how well you can present it.

I used to do a lot of these styles of interviews back in the day. At my old company, we had a pool of challenging technical interview questions, many of them requiring some level of algorithm knowledge in order to provide efficient answers. Many would require things like graph traversal, binary trees, that kind of thing. Some level of optimization is expected.

There are several problems with this approach, the main one being it often doesn’t really test for how well a person would perform as a software engineer. For one thing, such an environment is not reflective of actual programmer work environments. If you’re interviewing for a web developer position, such complicated problems would come up in your day-to-day work maybe 5-10% of the time at best. (The numbers go up the lower level you go, such questions would be a lot more useful for systems programming work for example.) And unless you are hiring someone specifically for a high pressure position, this type of exam is biased against people with anxiety or who otherwise have trouble working with what is akin to a gun to their head. It is also effort-intensive on the part of the interviewer(s), since often you will need to sit through the candidate explaining their solution and maybe pointing out flaws or inefficiencies in the approach. So I’ve kind of soured on this approach and haven’t done it in a while, but there are specific situations where it would be useful.

These days, what I like to do in technical interviews varies depending on the level of the role I’m interviewing for:

for fresh graduates or junior developer roles, I will generally ask 2-3 comparatively simple programming questions. Something only a bit more difficult than FizzBuzz, maybe along the lines of a fibonacci function or some array processing or such. This is especially useful if I’m doing this for a small company that doesn’t have the technical screening step, or to verify the results of the technical screening step. Afterwards, I will tend to do a short Q&A, asking some questions about projects they have worked on (maybe at school), what difficulties they have encountered, how they approach a problem etc.
for more senior positions, I tend to skip programming questions altogether (I assume that they haven’t spent 5 or so years BSing their way through programming tasks) and prefer a more detailed Q&A, usually tailored towards the expectations for the role and their specific background. Typically I will ask questions that will require the candidate to expound on technical issues such as:
- what was the biggest technical challenge you had to overcome?
- what are the advantages of X over Y and vice versa? (For example, for senior frontend devs, you can ask for the differences between React/Angular/Vue, if they have that experience). If you require specific experience in a particular technology, the interviewer will need to have sufficient experience as well, enough to be able to ask deeper questions while being able to filter out BS
- for this [past project], why did you choose this tech stack instead of [more common stack]? What are the pros and cons?
- can you describe a time when you had to convince upper management to use a risky technical approach?
- what was your worst technical mistake/fuck-up? (Follow-up: what did you learn, how did you recover, etc)
- have you ever had a problematic junior developer working with you? How did you handle it? (Such types of questions are good for technical lead roles)
- have you ever had to dive into a project with an unfamiliar tech stack without time for learning? How did you cope? What were the challenges? etc.
- what is your preferred approach for estimation? what if you are under schedule pressure?
- how do you prefer to onboard a new developer to a large project?
My goal for such questions would be to look for absence of several attributes I consider important for senior developers:
- able to communicate technical issues well (useful both when managing upwards and downwards)
- able to adjust to unfamiliar situations/technologies easily
- able to admit to mistakes on his part

Effectiveness of Technical Interviews

Ultimately, just doing the technical questions or the Q&A can’t guarantee the candidate will be good for the role. Evaluating the performance of individual developers is a notoriously hard problem, even when doing annual performance reviews, what more when limited to a one hour interview? For me, the purpose of a technical interview is to act as a negative screening tool - to filter out candidates who are obviously unfit for the position: candidates who can’t program, who can’t communicate well, who don’t have experience in the specific role or tools you need, and so on. The actual evaluation of effectiveness and fit should happen during a short probationary work period with the company.

How to Prepare for Technical Interviews?

For candidates, how do you prepare for technical interviews?

The most important question is how confident are you in your own programming skills? If you’ve done any nontrivial amount of programming work, generally you won’t have an issue. If you’re not confident, better to practice until you are, start with simpler questions and work your way up.

For the Q&A part, in general you just need to be honest. But you also need to be able to communicate well, which is something not many people will be able to do. Good communication skill is an essential skill as a developer though, you may want to look for ways to practice and gain confidence in it. I recommend posting on a blog (like this one) to sharpen your communication skills! (Verbal is harder to practice, you need to find someone willing to listen and give you feedback.)

I would also recommend becoming familiar with the requirements for the position you are interviewing for, understanding which parts of your background are most relevant, and highlighting them during any Q&A. In fact, I would prepare a short pitch describing why you think you are a good fit for the role. Often during Q&A I like to start off with asking the candidate to give a short overview of their background, and very few actually give a good response (a lot don’t even have a short response), so being prepared in this manner would be very impressive.

It would also help to ask around about the specific recruitment process for the company you’re interviewing for, if you have any contacts, or sometimes there’s information on the internet. (Good internet searching is an essential developer skill as well!) Just knowing what to expect can boost your confidence.

I’ve also given some other advice in a Q&A last year on reddit. Good luck!

Everything I Know About Software Development

hello@roytang.net (Roy Tang) — Sun, 26 Apr 2020 13:27:12 +0800

War stories
Getting a job as a software developer / hiring software developers
Self-improvement
Working with other people
Actually coding
Software career perils
Web Development
Project-level technical concerns
- Version Issues
- Integrating Open Source Libraries
Project management
Things I dislike
- Lies and marketing
Tools
- Source control
  - Git vs CVS
  - Devnotes: Migrating Mercurial to Git
- Editors
  - My history in text editors
- Internet
  - StackOverflow
Devnotes
- HTML
  - Pure HTML Toggles
- CSS
  - Pure CSS Spoilers
- Python
- SQL
  - Is SQL a dying art?
  - Devnotes: PostgreSQL on the command line
- React Native
  - Upgrading a React Native Project
- C++
  - Revisiting C++ after a decade
- JavaScript
  - Javascript: References to out-of-scope variables.
- Others
  - MarkLogic NoSQL
Mini-Projects

Misc:

Things I’ve posted to Stack Overflow:

Unsorted:

Working with people from other cultures

hello@roytang.net (Roy Tang) — Tue, 21 Apr 2020 00:00:00 +0000

Unless you’re working at a small shop that only serves local clients, software development these days is often an international endeavor. That means the aspiring software developer needs to be able to work with and get along with people of different cultures.

In the company I worked with, most projects back in the day we would have a person from the foreign office in charge. Their roles were either as project manager (PM) or system analyst (SA). They were the ones who would be interacting directly with the clients so they get to decide which things need to be done and will often also be setting the schedule. The PM/SA of my first project often communicated directly (via IM) only with his local counterpart, our team leader (TL), so at first our contact with him was minimal. (I found out later that he wasn’t too comfortable speaking to newbies as he’s afraid of being misinterpreted.) But as is wont in software development, often schedules will not be sufficient and deadlines will not be met and so there will be pressure to keep up from the PM/SA. When you’re starting out, there can be a tendency to be off put by these faceless individuals who communicate with us only email, often sending new problems our way.

The PM/SA often had a vastly different culture from us Filipinos who like to joke around and be informal when discussing even work-related matters. And in our case, their first language was Chinese and not English. While they did learn English in school, they use it more formally instead of conversationally. So often their messages or emails will be terse, and when you’re not used to dealing with them you might even view them as bit rude or cold and uninviting.

As I mentioned, the PM of our first project usually often spoke directly only with our TL, but eventually he warmed up to me and one of the newbie testers who was my batchmate. We often had to discuss or clarify program requirements with him so I guess he got used to us after a while. There was another newbie tester on our team however who he never seemed to warm up to. I remember him asking our TL “Who is <tester’s username>?” even though he had previously replied to some emails of the same tester. There was also one time I remember when that tester sent a detailed email to the PM to clarify some items, and the PM replied but addressed his reply to our TL instead! We would often joke around with that tester that the PM probably still had no idea who he was.

Because they often speak in a formal manner, we are often surprised when a glint of humanity shines through. A remember when one other junior developer was talking on the phone with the PM about a particular problem in production and he came back all worried. When we asked him what happened he said “ said some swear words!” Wow, we must have really been in deep trouble for him to lose his cool!

It turns out not all PM/SAs were like that though. Later a couple of younger SAs joined our team and they were much more agreeable to talk to, willing to joke around a bit and talk about their day and stuff. I think it’s because they were fresh from University so maybe they were still used to English. It’s usually the older PMs who are much more stoic.

Later on, I got sent to the foreign office to help with some UAT preparation on-site. The stoic PM met me at the office and took me to dinner and drove me around to some tourist spots on my first night, so we got a bit closer. The next day was a Saturday, and we had to go to the UAT site to check the setup and preparations. I met him at the office before going to the site and I was surprised to find that he was going to go to the UAT site in a T-shirt and shorts. After that, I never took him too seriously!

A significant number of years later, I also got to work with some staff from a European country. It was another culture shift, but this time it was a bit easier. Unlike our Chinese friends, the Europeans were friendlier and more small-talkish. Meetings (even over Skype) would often start off with “Hello, how are you guys doing?” and more pleasant things, in those precious few minutes while waiting for everyone to get set up. It’s not all good though, as I distinctly remember a meeting where one of the callers was coughing and wheezing all the time (learn to use mute!), although that was probably a person-specific thing and not a cultural thing.

Even work-wise, it was a vastly different way of working with the Europeans. With the Chinese people we worked with on our projects before, everything was very formal and well-documented and there would be action items and detailed program specs to follow and any changes had to go through a rigid process and schedules would need to be adjusted and such. With the Europeans, we would often have meetings and many of the items would not be resolved and would be moved to the next meeting. It would not be unusual for discussions to go round and round a few times before finally managing to come to a decision. And we’d never get clear specifications from them, only some general items or ideas of what they wanted, and we really had to iterate and show them how things would look before they would sign off (and even if they did sign off, they’re very likely to change their minds later at a moment’s notice!) If you’ve ever seen the BBC show W1A, it’s a pretty accurate portrayal of how it was!

Working remotely and communication via channels like email, IM or video calls is already a challenge even when working with other locals. The additional dimension of a foreign culture means we have to be much more willing to adjust to other people as needed. This means being more open to how other people work with, giving them the benefit of the doubt, clarifying where things seems unclear or where there may be miscommunication, and so on.

Pure HTML Toggles

hello@roytang.net (Roy Tang) — Fri, 17 Apr 2020 00:00:00 +0000

Just last month, I wrote a method of implementing element toggles using a pure CSS approach. While that post was educational for me, it turns out there was an even simpler way of doing things. I found out about it when I read this post by Jamie Tanna. Apparently the details and summary tags already support HTML toggles, so we can do this with neither CSS or JS!

I’ve updated the spoiler tags on this site to use this new method. I also used this method for the Table of Contents on certain posts (currently only the Covid19 diary). Sample here:

(Spoilers)

This is some text!

The post I linked above covers most of the technical details. My only issue with this is that’s a relatively new feature. Even though browser support is near universal (minus the much-maligned and quite ancient Internet Explorer), I’m not sure if there will be issues or how it renders when viewed in an older browsers or in other contexts such as in a feed reader or a screen reader. I don’t want to accidentally show spoilers! I suppose that’s a minor concern at this point, and maybe I should just be confident that people who make those tools have taken the visibility toggle into account.

(I’ve also edited the CSS post to point to this one so anyone who sees it has better information!)

Web Application Security

hello@roytang.net (Roy Tang) — Thu, 16 Apr 2020 00:00:00 +0000

A while back one of the biggest leaks of personal information in history was made by hackers accessing the Commission on Elections database here in the Philippines. More than 50 million voter registration records, including information such as full names, date of birth, address, among others. A small percentage of the data leaked also included email addresses and even passport numbers. The hack exposed more than half of the country’s population to the possibility of social engineering and other exploits. It was certainly a national embarrassment, especially for those of us working in software development. But the truth of the matter is that securing publicly accessible sites against determined bad actors is difficult, very difficult. It’s something that needs to be taken into account on all levels of the system.

I don’t consider myself any kind of security specialist, but I think it’s something that software developers of all levels need to at least have a passing knowledge of, especially in this modern age where many tech companies store personal/private data for large numbers of users. I tend to classify the security practices and vulnerabilities under several levels:

Physical / hardware layer - this means controlling the physical access of unauthorized personnel to hardware that contains private information. It would take serious levels of screw-ups for a system to be compromised in this manner, or perhaps a well-placed bribe. It is however, comparatively easy to detect any breaches in this layer. In any case, this is not really a concern for the software engineer.
Server OS and local network level - this means that controlling the access to the actual OS running on the machine and the corresponding local network the server is on. Practically as bad (sometimes worse) than compromising the hardware layer, a compromise on this layer means the bad actor can access everything, possibly remotely, making it much more difficult to detect. Securing your system on this layer involves among other things making sure known vulnerabilities on your OS are patched or worked around and network access is restricted as needed. This is often the concern of people like network engineers or devops people who are setting up and securing your servers.
Application container level - this doesn’t necessarily mean your application itself, but typically websites and web applications will be running inside a container software such as a webserver like Apache, Tomcat, ISS, etc. Securing this layer is often just a matter of making sure the latest security patches are applied and known vulnerabilities are closed and access is restricted as needed. If this layer is compromised, the bad actors will typically have access to any files deployed in the container (which may contain other information that will facilitate further leaks). I also include here securing vulnerabilities in the database software used as your backend.
Application level - this refers to your security in the design of your application itself. Typically, problems in the first three layers above will be the responsibility of people setting up the servers and the server software, who are probably a different set of people than the development team. Concerns from this layer onwards are more applicable to the actual application developers. Application-level security can be further broken down into several levels:
- User authentication - this refers to your application’s implementation of user login and session control, and management of things like passwords, password reset requests, two-party authentication, etc. If the implementation is poor, your application may have vulnerabilities like bad actors being able to identify your users’ usernames by enumerating login requests or being able to brute force their passwords.
- Functional security - this refers to how your application grants rights to each user as to which functions in the system he is able to access. As an example, in a typical system, normal users should not be able to access functions restricted to administrator users.
- User-specific Data security - this ensures that only relevant data for each user can be accessed by each user. For example, if I’m logged in as User ID = 1, I shouldn’t be able to view the data for User ID = 2 simply by changing a parameter such as USER_ID=2 in the URL.
- Domain-specific security - this refers to additional security rules specific to the domain your application is involved in. For example, your security rules may demand that sensitive information can only be viewed by users with specific privileges and would need to be masked for other users. This may introduce some special logic in your program development.
- Logging - applications typically log a lot of information to log files and audit trails, to facilitate debugging of problems and often also for statutory purposes. Security restrictions may require making sure that sensitive data is not included in the logs and audit trails.
- Code level - this refers to security in coding practices in both the code used to write your application and also any third-party libraries or frameworks used by the application. This is highly experience-based and requires knowledge of the most common vulnerabilities as a result of coding practices. Organizations such as OWASP regularly publish lists of the most common vulnerabilities, many of them resulting from poor coding decisions such as SQL injection, cross-site scripting, cross-site request forgery, etc. Modern frameworks will typically provide standard ways for developers to minimize the chance of introducing the most well-known vulnerabilities.
- Database level - this refers to security in the design of the database itself. Examples of concerns on this level are whether sensitive information is encrypted or hashed or whether financial data such as credit card information is stored in the database and secured appropriately (for credit card information specifically, there are standards laid out for managing credit card data that you have to comply with in order to be certified to handle credit card transactions). I guess I would also include in this layer how database backups and purged data are handled - are they backed up and encrypted separated from the main database? - although the protection of such backups may be outside the scope of the application developer.
- Third party code - modern day applications use a lot of third party libraries and packages, not to mention code that developers are copying from StackOverflow. And rightly so - since using these dependencies means savings in effort and more productivity. But developers need to be aware of the cost every time we add a new dependency; not only are we dependent on that third party library working correctly, but also that it is generally secure. And if a security issue is found, will it be fixed in a timely manner? And this kind of thing can be difficult to track. There have been stories of websites being exposed to security breaches or even things like embedded crypto miners simply because they added a node.js dependency that had another dependency that had another dependency that was compromised. It’s the kind of thing more senior developers on the team should be aware of and on the lookout for.

In modern settings, developers often specialize into roles like frontend and backend. And one might think that security concerns are limited only to the backend side, but one should not neglect security on the front end either. There are security concerns on the browser/JavaScript level as well, and you have to make sure that your frontend is aligned with the backend’s security paradigms.

Often times many clients will require that a third-party security audit will be done near the end of the project, and the development team will need to address any security concerns that come up during the audit. I’ve been on the receiving end of these a few times and I’ve found the companies doing them to be fairly thorough.

It’s better for your dev team to be generally aware of security best practices from the very beginning, as it may be more costly to correct security problems towards the end of a project. It’s the kind of valuable tribal knowledge among developers that you can carry easily across projects and companies and tech stacks.

Bespoke vs Product Development

hello@roytang.net (Roy Tang) — Tue, 14 Apr 2020 00:00:00 +0000

For most of my time working on software projects, it has always been for bespoke projects. Bespoke basically means a software program or package tailor-made for a specific client. The client provides all the requirements, the team fleshes out more details and specifications, some prototyping may or may not ensure, and implementation proceeds thusly. It’s relatively straightforward compared to product development.

I only started getting involved with “product”-like projects over the in the latter half of my career as a software developer. A “product” is a more general use software program or package, perhaps sold to the mass market or as a service. My experience is mostly with products that are targeted towards the public sector (government agencies and such) and not for general consumer use. Though I imagine many similar issues and pitfalls as I discuss here can occur in other types of product development as well. These are largely issues of “building for multiple clients” as opposed to “building for a specific client”.

The project lifecycle is markedly different: you start with an idea for a product, perhaps based on something you’ve already built before, and shop it around to potential clients who could use such a system. A number of clients put out tenders for projects where they specify it has to be a pre-built product instead of a bespoke solution. Typically you use the requirements from these tenders to direct your development while you don’t have clients signed up yet. That’s right, you start development work ahead of time even when you don’t have clients signed up yet. These tenders typically have significant lead time before the actual implementation, so you kind of hope that if you somehow magically win with your proposal, you have everything (or at least most of it) done and in place and ready for the client.

Tenders can also sometimes have a requirement to have a demonstration during the tendering process. Therein lies the first problem - since you probably don’t have the product ready yet, there’s probably some crunch time needed in preparing and setting up the demo. Some iffy code will need to be written, sacrificing code quality and maintainability in order to meet the timescales for the demo.

Of course, since you want to gain traction as quickly as possible, you try to shop around to as many clients as you can. That means you try to bid for as many tenders as you can, and different clients can have different requirements. This of course creates more problems for the development team in the form of conflicting requirements and even worse, possibly conflicting demo schedules. It’s something that management has to be cognizant of and the company will need to pick and choose its battles. Poorly handled, it could put a lot of pressure on the development team.

I was discussing this the other day with a younger developer working in another company. They were working on a project that was headed in this direction and he was asking whether things would get better once they have more things, and I kind of laughed a little bit before telling him there’s a good chance it would get worse.

The problem of conflicting requirements gets even more complicated when the clients are actually paying for your product because now you have even more responsibility to fulfill their custom requirements. One would think that if you were making an off the shelf product you’re not really subject to the whims of the clients too much, but when you’re trying to gain traction in an existing market you will sometimes need to make compromises to get clients to choose you over more established products, which probably means catering more to their whims.

And really, even clients in the same space needing the same product will have their own requirements, idiosyncrasies and ways of doing things. Sometimes even when there are statutes in place dictating standards (like what data should be recorded and so on), different clients will still have different ways of meeting those standards such that your system may not be able to cater for all of the differences.

Ideally, your product has been planned with such variations in requirements in mind. This means making your system flexible and/or configurable enough to handle any differences between clients, or at least changes can be kept small and manageable between clients. This assumes of course that you had the foresight to predict ahead of time what the differences would be and planned your system architecture accordingly, and that you didn’t have a lot of poor and hard-to-maintain code littered throughout your system because of needing to meet conflicting demo requirements! The more customizable your system needs to be, the more well-structured, modular and lightly coupled the code has to be.

If you don’t have the above, well then likely you’re looking at introducing even more bad code and stretching your code base with code smells everywhere in order to accommodate tight deadlines and conflicting requirements.

There’s also a lot more that would need to be customizable than you might expect. Some things that clients want tailored for their needs are: UI theme or individual UI elements; menu structures; field names and error messages; message/email/notification templates; access control rights; form layouts, what fields are included in each form, field validation logic; reports, what fields are included, and even retrieval logic. These are just some examples, there are many more that are difficult to foresee just how the clients might want them customized, so most likely even with the best-designed system there will still need to be some level of customization done for each client.

How to avoid such pitfalls? It helps if your product has a strong identity and design from the very start, but that means having a strong product manager or management team with a good understanding of the domain required for your system. The product manager will need to manage the clients’ requirements and the direction of future development. The product manager has to be able to assess the client requests and say “no, that’s not how our product works,” when faced with requests that will make things more complicated for the development team.

The above are drawn from my personal experience and from a few other developers’. I’m sure there are ways to handle product development to avoid these problems of course, but I believe that far more projects encounter these difficulties than do not, but it mostly comes down to management understanding the concerns and making adjustments to resources and schedules accordingly.

Naked CSS Day

hello@roytang.net (Roy Tang) — Thu, 09 Apr 2020 11:44:35 +0000

It’s naked CSS day, so the site looks super bare-bones today! Inspired by Laura Kalbag and Eric Meyer.

Trying out the site without the CSS made me realize how I wasn’t setting width and height for the svg icons I was using, so they became huge when the CSS was removed. I also rearranged the templates so the footer is at the end of the document (previously it was being put there by CSS). So at the very least this did lead to some fixes!

(Hopefully I don’t forget to restore the CSS tomorrow!)

Screenshot of the Naked CSS version for posterity:

Browsers and HTTP

hello@roytang.net (Roy Tang) — Tue, 24 Mar 2020 00:00:00 +0000

Drew Devault wrote a great post/rant about the reckless limitless scope of modern web browsers:

I conclude that it is impossible to build a new web browser. The complexity of the web is obscene. The creation of a new web browser would be comparable in effort to the Apollo program or the Manhattan project.

For the past year or so, I’ve been thinking about contributing to an open source project. Haven’t gotten around to it yet, but I have looked at a few projects of interest. Preferably it’s something I use on an every day basis, so Firefox was one of my candidates since it was my web browser of choice. I decided to try it out last month; I followed the instructions on their site for checking out the source and building the code. I assumed there would be a lot of code, and that the build would be a nontrivial amount of time, but just checking out the source and creating a build took the better part of a day; I believe the build itself took somewhere around 4-6 hours (I didn’t really record how long it took, just left it running while I went out and did stuff.) I haven’t actually tried hacking on the code itself yet, because with that kind of build time, I don’t see how one can be productive. (I decided to run the build a second time as I write this post, maybe it speeds up on successive builds?) It’s kind of intimidating to get into.

The point is: Drew is right, and modern browsers are massive pieces of software. The comparison to the Apollo or Manhattan programs may even be understated; even though your browser isn’t taking people to the moon or testing nuclear weapons, it’s probably using way more memory than those programs combined!

Somewhat related: Great post recently over on the Cloudflare blog about the History of the URL. One can’t talk about the history of the URL without talking about the history of the internet of course, and this post let me learn a lot more about those early days of the internet. One of my key takeaways is that a lot of the design decisions around URLs and the early internet made by people like Tim Berners-Lee largely intended the web and the HTTP protocol to be used for serving text-based documents. It’s even in the name of the protocol: hypertext transfer protocol.

And that’s what it was in the early days. People put up documents and web pages, and other people read them and maybe posted comments or other input and that was it. But the modern web (and thus modern web browsers) have tacked on so much functionality onto the presumably text-based protocol. First we added the ability to serve images and videos, then whole applications, formerly the domain of native apps. Just looking at my commonly-used websites, the web browser is used these days for:

full-featured email applications (GMail, ProtonMail)
a complicated social media interface with multiple streams that automatically update (Tweetdeck)
watching videos, tv shows and movies (Youtube, Netflix)
instant messaging with groups and voice chat (Messenger, Discord)
rss feed reader (Inoreader)
note taking and productivity (Trello, Notion)

With systems like Google Stadia, you can even play AAA games via the web browser now!

In the early days of the internet, all of those would still be native apps running outside the browser but these days we overload the poor HTTP protocol to handle all of these functions. The modern browser these days is almost comparable to an entire operating system (and there are of course already ChromeOS and FirefoxOS to drive home this point).

I understand full well the benefits of overloading the browser and the HTTP protocol to handle all of these applications, but the scope creep is definitely unsustainable. The high resource usage of modern browsers and web pages even moreso. Imagine a new web browser that cuts off support for the evolving W3C specs at a certain point to compete on performance and stability instead.

How about one that’s “just a browser”, literally just for browsing documents. Maybe one without JavaScript support completely, if you open a page that has SCRIPT tags, it will prompt you to open the OS-level browsers like Firefox or Chrome? Many modern applications could probably work without JavaScript, except users would have to do without many of the bells and whistles we’ve come to expect from modern webapps. i.e. clicking “Refresh” instead of timelines auto-updating, clicking “Next” instead of having infinite scroll, and so on.

The never-ending scope creep reminds me of the never-ending growth pursued by late-stage capitalism; it can’t be sustainable in the long run, and maybe we need to start looking for different models other than “the browser does everything and will eventually become the OS.” (At that point, everything will have been subsumed by JavaScript)!

Pure CSS Spoilers

hello@roytang.net (Roy Tang) — Thu, 05 Mar 2020 00:00:00 +0000

Edit 2020/04/17: A month and a half later, I found a better way to do this!

I previously had some post that had some content hidden via spoiler tags, using a custom Hugo shortcode. Since I’m an old-school developer I was previously doing this using some Javascript run on load:

    let elements = document.querySelectorAll(".spoiler_header");
    Array.prototype.forEach.call(elements, function(el, i) {
        el.addEventListener( 'click', function( event ) {
            let nextEl = el.nextElementSibling;
            let display = getComputedStyle(nextEl)['display'];
            if (display == 'none') {
                nextEl.style.display = 'block';
            } else {
                nextEl.style.display = 'none';
            }
        }, false);        
    });

This is the modern age however. We should really be trying to minimize our Javascript on the non-app public web as much as possible, so I decided to change it to a Pure CSS solution. Here’s a quick demo:

(Spoilers)

This is hidden text!

The Pure CSS solution uses the following HTML (including Hugo shortcode macros, should be easy to understand; .Inner is the actual spoiler content.):

<div class="spoiler_container">
    <a class="spoiler_link" href="#spoiler-{{ $id }}">(Spoilers)</a> 
    <a class="spoiler_target" id="spoiler-{{ $id }}"></a>
    <div class="spoiler">{{ .Inner }} <a href="#" class="hide_spoiler">Hide Spoilers</a></div>    
</div>
<p />

The CSS for this toggle is:

.spoiler {
    display: none;
}
.spoiler_target:target {
    position: fixed;
}
.spoiler_target:target + .spoiler {
    display: inline;
}

Basically, this takes advantage of the :target CSS pseudo-selector. The “(Spoiler)” anchor changes the URL hash to point to the id of the spoiler_target element, and we use the CSS + operator to display the actual spoiler content when the :target pseudo-selector is activated. I originally tried it with the id being assigned directly to the spoiler content, but this had the problem of “jumping around”, i.e. clicking the “(Spoiler)” link would scroll the page to the anchor location. The workaround for this was separating the target anchor from the spoiler content, making the anchor hidden (no text) and adding position:fixed (to prevent the scrolling). The “Hide” link simply sets the URL hash location back to “#”.

Burnout

hello@roytang.net (Roy Tang) — Thu, 06 Feb 2020 00:00:00 +0000

In the middle of 2015, after 12ish years of working at the same company, I said to myself “I think I’m burnt out.”

I remember that after a few days of ruminating over it I finally decided and said out loud to myself “That’s it, I’m done.” By the end of the year I had left the company. I didn’t write about the burnout then, out of respect for my former employer. But it’s been a while, I think the time has come to write about it.

How do I know I was burnt out? I don’t believe there is any generally accepted consensus of symptoms for burnout, but I had a good set of indicators:

exhaustion, both physical and mental. More and more often I found myself simply tired of everything.
frustration with all the problems at work and the projects I was assigned to, frustration that we kept running into the same issues again and again (mostly relating to scheduling and deadlines and resources), and frustration that any concerns I raised were not being addressed
thinking about work all the time, even on my off hours I would be mentally considered the open Jira tickets, etc
a feeling of helplessness in the face of overwhelming and seemingly never-ending project issues, re:scheduling, resources, quality, and so on
a general feeling that that the effort I was putting in wasn’t worth it or wasn’t appreciated as much as I’d have liked
general lack of free time, and even when there was free time, lack of energy to pursue anything productive
permanently broken sleep cycle (a problem I still struggle with to this day)

The work at my former employer had always been… challenging, to say the least. Overtime was not uncommon, and many people often quit citing the heavy workload. And yet I was generally happy/content working there for the better part of a decade. I kind of enjoyed and thrived in the environment most of the time, especially when facing technical challenges. Things got worse for me personally starting around 2014ish I think. There were a variety of challenges during this time, some of them due to the company/environment and some due to my own personal shortcomings.

Environment challenges:

a large project with constantly changing requirements and overly aggressive schedules (due to sales/marketing push) and almost continuous crunch. It felt like we were always jumping from one crisis to the next
needing to provide support for demo and live systems during off hours, due to the target clients being halfway across the world
some clients and coworkers were challenging to work with
during this period, the traffic situation in Metro Manila had gotten significantly worse, draining even more time and energy

Personal challenges:

I have a very strong sense of duty and responsibility. I realize this doesn’t sound like a negative, but it’d kind of a double-edged sword in that I take my duties as an employee very seriously and will sometimes sacrifice my own well-being if I think it is needed. For the longest time, one of my favorite quotes has been:

“Death is lighter than a feather, but Duty is heavier than a mountain” - Japanese proverb

I find this quote symptomatic of the “hustle culture” that tends to lead to burnout. That’s probably a topic for another time, but the gist of it is that most companies and projects are a never-ending well of work that needs to be done, so much so that oftentimes the reward for good work is often more work. i.e.

“The price for being the best is always having to be the best”

This “sense of duty” was so strong that even when I realized I was burning out, I still gave half a year’s notice for resignation, because I at least wanted to make sure everything was fine before I left. (Spoilers: it would take them a couple more years to actually finish the project). Someone reminded me recently that I expressed regret at leaving “in wartime” since the project got even further delayed and many of the issues were unresolved at that time.
related to the above, I also had a strong sense of pride. I considered myself very good at my job (justifiably so, based on annual reviews etc) and somehow had the mentality that I needed to be able to carry the weight of the project by myself. Whenever there was a resource or scheduling problem, one of my first instincts was to be to offer to take on the needed load myself, no matter how busy I already was.
Somewhat related to the above, but I also had (and often still have) an obsessive-compulsive need to read and reply to all emails and messages and notifications immediately, even outside of work hours
coupled with my strong sense of duty is a lack of ability to say no. As an example, for some reason even after I realized I was burnt out and notified the company I was leaving, I agreed to go live in another country on-site for three months. While it was good to experience another country, the work-life balance while I was there was even worse because the time zone problem still existed (except in reverse, i.e. the dev team was now on the other end of the world), and now I became a bottleneck (since I was one of the only two people on-site) and had to dedicate more time to bridging the gap.

As noted, I think many of these issues existed even before, but perhaps most of them were exacerbated by the fact that we now had to provide support across two time zones seven hours apart. That really took a toll on my work-life balance. Looking back at my emails during that period, I would often be replying late into the night or even early in the morning. I think it was really a question of energy and time being in short supply. I almost burnt myself out again in 2018, but at least I had the sense to recognize it faster and to step back as needed.

I must point out that I bear no ill will towards my former employer or the people who worked there. There was a lot of room for improvement but everyone was just trying the best with what they were given. And I share some of the blame since I was kind of in a leadership position – I should have pushed harder to make things better for everyone involved.

How to avoid burnout:

set reasonable boundaries between work time and personal time. Having a strong sense of duty is probably fine, as long as you know where that ends and to prioritize your own well-being as needed. Guard your free time zealously. As much as possible, don’t accept calls/messages outside of agreed upon work hours. The occasional exception for an emergency is probably fine, but don’t let it become a regular thing. Having well-defined boundaries allows your brain to avoid overlaps - if you’re at home, you should get used to not thinking about work.
understand your own value and your own limits. Having a good understanding of what you bring to the team and the limits of your own ability and capacity means you can more effectively allocate your time and energy.
always be aware of the employee-employer relationship. It’s okay to have a good working relationship with your superiors and coworkers and even to be friends with them, but never lose sight of the fact that to the employer, you are an employee. You aren’t “family” no matter what the company says. You are being paid to do a job, and that job has limits. You don’t owe the company your personal time or your absolute loyalty. It’s a capitalist enterprise, not a nation. Remember that in the end, the company will need to look out for itself (especially for publicly-traded companies), so you will need to look out for yourself as well.
learn to say no. This is coupled with the first two points above. If you know what your boundaries are and what your limits are, it will become easier to say “No” to unreasonable requests. And remember that your personal time is your own, you don’t need to defend or explain why you don’t want to work on weekends or such
regularly reflect on how things are going. Check your energy levels to make sure they aren’t constantly low and that you’re getting enough recharge time. If you think are starting to burn out, pull back, take some time off, and reassess how to move forward. Assess also how the company is addressing your concerns, and if they aren’t or are unwilling to do so, consider whether it may be time to move on.

For people in leadership/management positions:

understand the value and capabilities of your team and build them up. One of the best things about working in larger companies/projects/teams is that you have other people to back you up, to support you and handle things when you reach your limits. Understanding both your own limits and the team’s limits allow you to better allocate tasks to avoid burnout.
it isn’t sufficient to simply be asking your team members “Are you still ok?”, because if they’re anything like me, they may not be aware that they are starting to burn out already. Pay attention to how they are performing and how much time, effort and energy they are putting them in, and don’t push them unnecessarily.
if your team members raise a concern, make sure to not only listen but to make them know what is being done to address their concerns. Unaddressed problems lead to frustration.

In general, software development is a demanding industry, and it tends to attract people like me - industrious people who enjoy and take pride in their work. (Hence the famous quote about laziness, impatience and hubris.) Add to that the common difficulties in estimation, scheduling and planning, and it’s easy to understand why burnout is rampant in this industry. All the more reason for everyone involved to be aware of the risks of burnout and how to avoid it, and perhaps to somehow help reshape the industry to better avoid it.

Tales from the Old Web (Development)

hello@roytang.net (Roy Tang) — Tue, 04 Feb 2020 00:00:00 +0000

There was this great (and long!) article that came out recently about the history of CSS. It reminded me a lot of the old days when I started out in web development. So join me in a walk down memory lane as I reminisce about the trials and tribulations of early web development. (This one isn’t about CSS as much as that linked article.)

I started out with web applications during the heyday of the much-maligned Internet Explorer 6, and pretty much the start of the age of webapps. Most of our projects during this time involved government agencies wanting to upgrade from their old existing client-server style systems. Those were often written in programming languages/tools like Delphi, Oracle Forms, PowerBuilder and such. That meant there was a lot of pressure on the web applications we developed to be able to match most of the client-side functionality available in those kinds of programs, which was often a challenge due to the limitations of the HTTP model.

ActiveX

Typically, the government agencies at that time used only Internet Explorer in their workstations, so we could write our web applications to target just IE specifically. This meant we could use things like ActiveX to attempt to do things browsers normally can’t do. Some of it was for security purposes, like confirming the user’s authentication matches the current workstation user, or for Single-Sign On purposes, things like that. There were also some for interfacing with things outside the web browser, like document scanners.

We also had to use some ActiveX to get around browser quirks. For example, IE6 had this problem with combo boxes/select elements where the width of the drop window would always match the width of the select element, but this means if the option text was longer than that width, they would not be readable. We had to make an ActiveX component just to fix this, by making the drop window wider on document load! Sample screenshot recovered from the vaults:

(Click to view full-size)

Some other ActiveX uses were to do things that would later on become standard features in browsers. Some other examples I can recall:

spellchecking. We used an ActiveX component from a company called WinterTree software. I was surprised to find this particular product of theirs is still available, when practically all modern browsers have spellcheck support! I suppose it’s still useful for legacy environments maybe?
we had this one project that needed a complicated graphing diagram. Think about those pinboards with pictures and documents connected by wires that conspiracy theorists use in memes, something like that. These days that kind of thing can be done using modern HTML+Javascript only, but during the IE6 days we were still pretty limited. We used another third-party ActiveX (can no longer recall the component name), that we wrapped in our own Delphi program for customization purposes.

As far as I can tell, we ended up using ActiveX in some projects up to the early 2010s.

GMail and Ajax

Our initial webapps also did the thing where we would reload the page whenever doing something like going to the next page of results. This was pretty bad user experience-wise, especially compared to the client-server programs of old, which felt a lot more snappy since they were only loading the database results via the database driver, instead of an entire HTML page. By 2005, we were already using Ajax/XMLHttpRequest to get around these limitations in IE6. (The wikipedia entry says XHR didn’t officially exist in IE6, so I’m not sure how we did that, but I’m pretty sure we did). Our main use was for paging search result grids, but we weren’t even fetching XML or JSON back then, the XHR response would be the grid HTML itself, which we just plop right where the previous page was.

I remember I was talking to the SA for the project about ajax, he asked what it was and why we were using it, my explanations was:

Ajax is “Asynchronous Javascript and XML”; it’s a method of using Javascript and XML data that allows us to make server-side (database) calls without having to resubmit the entire page.

We use it because it’s cool =p With Ajax, we can do something like allow the user to re-sort a grid of records without reloading the rest of the page. It is easier for us to meet the performance reqt.

If you’ve seen the Google Maps or GMail webapps, they both use Ajax.

The response was (unedited):

I’ve used Gmail. AJas is client side library??? Seems they process very fast.

(I would later kind of regret using Gmail as a comparison!)

Cross-browser development

Most of our projects were for internal systems where the workstations had controlled environments so we were free to target a single browser only (usually IE). Some projects also had a public-facing portal though, and that required we do some cross-browser frontend stuff. Early on, the typical targets other than IE where Netscape and Firefox. This is where the IE6 compatibility headaches come in. It was usually a game of write the code for IE first, then let the testers find any NS/FF issues, then quash accordingly.

We started checking in Firefox around the quirks mode era, as described in the article linked at the start of this post. The biggest challenge really was getting everything pixel-perfectly the same between browsers. The typical problems were:

browser differences in CSS models (the famous box model issue)
differences in native controls (typically the select and input type='file' controls were the more problematic ones)
for cases where we needed ActiveX plugins for IE6, we would also need to develop corresponding FF add-ins, usually targeting Firefox 2.x (It’s fun to think about how the current version of Firefox is 70 major versions ahead of that!)

We did very little cross-browser work at the start, with the proportion accelerating in tune with Firefox/Chrome’s rise in marketshare. By around 2010, it was common for projects to be testing across at least 2 versions of IE (6 and 7), 2 versions of Firefox (2 and 3), Google Chrome, and often Safari as well. (We didn’t really start thinking about Mac users until around 2007ish).

Google Chrome came out in 2008, I found some old comments I made to our internal tech team when it came out:

I have an installer in t:\roy

It’s not an OS, but it runs browser tabs in separate processes (literally…each one has a different PID), such that a single webapp won’t crash everything else.

More info here: t:\keis\Google_Chrome.pdf

If it gets widespread adoption, it may affect the webapps we develop - it uses Webkit as a rendering engine (used by Safari), so it’s different from Firefox and IE. Haven’t encountered any problems yet though.

I also tested the memory usage of Chrome vs FF back then:

Try this:

Firefox - open 9 tabs. Check the memory consumption. Close 8 tabs. Check the memory consumption. (Remains around the same as previous one)

In Chrome, the memory usage of closed tabs is 0.

This is a far cry from Chrome’s modern reputation as a memory hog! (Although I guess that reputation is probably related to people keeping too many tabs open.)

Side story: while looking up old files for this post, I came upon a thread where we were discussing printing issues across browsers, and there was a problem in printing some pages from the browser because of limitations in IE6, and some management higher-up chimed in with a suggestion to “just tell the client to use IE7” as if the client will magically upgrade all their workstations for that one problem. :shrug:

jQuery

As the linked article mentioned, jQuery raised the bar when it came out, allowing developers to easily paper over cross-browser JavaScript concerns and providing an API that became so prevalent, years later parts of it would become built-in to modern browsers (i.e. document.querySelector).

I shared the link to jQuery via delicious in 2006, but for sure we weren’t using it yet in our projects at that time. Based on my own CV, the first time I used it in a project was in 2009. It quickly became part of our standard libraries. I still remember in one particular project I spent some time thrashing out a bunch of issues because we needed to upgrade from like jQuery 1.3 to 1.4? Or something like that. Normally we’d fix to a certain version, but I think we needed to upgrade because of a jQuery plugin we needed. This was before the modern frontend development frameworks where you already have dependency management tools, so we had to make sure none of our other third-party JS components would break with the upgrade. The company ended up using jQuery as a standard in projects up to the time I left in 2015. (I’m told they later on switched to Vue.js, so more power to them.)

I identify as a “full stack developer”, but these days the “frontend” half of that usually implies usage of one of the modern frontend frameworks: Angular, React, or Vue. While I am cognizant of those frameworks and have tried them all in some form or another over the past few years, my frontend development experience is still largely defined by the jQuery era. So when setting up personal projects or quick prototypes that’s still what I default to.

CSS

The linked article was about CSS, so I figured I should have some CSS stories too, but I find that I don’t have much, other than the cross-browser compat above. My main issue with CSS back then was that in a large project with a nontrivial amount of developers each working on separate screens, the CSS style sheets tended to get longer and denser with time. And with bug fixes and whatnot, it would tend to be full of hacks and workarounds like randomly placing !important in so many places. I just assumed we had a whole bunch of CSS selectors that were no longer necessary but no one would bother cleaning up the style sheets because who knows what you’ll break? Basically a whole chaotic mess. IDK if modern tooling handles this because I haven’t been on larger projects since leaving the company, but I suppose the stronger delineation/specialization between frontend and backend dev work means less people to mess up the stylesheets, and those who do handle the styles are more aware of such pitfalls.

That’s it (for now)

Well, this was a fun trip down memory lane. This post is already longer than I thought it would be, but those are all the old-timey stories I can recall right now. There’s a good chance there’s some more of them lodged in my brain somewhere that will come out sooner or later, so maybe there’s a part 2 sometime in the future.

Flask vs Django

hello@roytang.net (Roy Tang) — Fri, 24 Jan 2020 00:00:00 +0000

In a bid to reduce the number of webapps actually running on my server (for resource consumption reasons), I decided to migrate a small Flask app I had and merge into this larger Django app where I have a lot of my personal data tracking stuff. The Flask app was small enough, mostly containing backend support for this blog (like search and comment submissions) and some Twitter things. The migration was straightforward, taking around half a day, most of that was wrangling with Twitter API rate limits.

It got me thinking, why did I build this using Flask in the first place? And that led to the more general question: when do I prefer using Flask over Django?

Personal experience: I’m much more familiar with Django than Flask, having started using it back in 2008. Professionally, I’ve also worked on more Django projects, compared to only one Flask project that actually made it to production.

The usual quick comparison is that Flask is the more lightweight web framework, suitable to smaller apps like microservices perhaps; while Django is the big monolithic web framework with all the bells and whistles already available. In practice this means Django already provides a lot of functionalities out of the box, while with Flask you have to install and figure out extensions for even the most common tasks like database management. A negative for Flask, but partly due to my inexperience: if you were already well-versed in the use of those extensions, I imagine getting everything together and setting them up will be easier.

If there is need for a traditional RDBMS, I will probably use Django. I’m not super familiar with SQLAlchemy, the usual recommended ORM for Flask, so that’s a big factor. That being said, if I needed to work with something like a NoSQL database or some other nontraditional backend, it might be better to consider Flask since Django isn’t super flexible in that record - most of the pluses are oriented towards RDBMS-backed models.

Admin: The built-in Django admin is a big plus for me; it means that for things like simple record CRUD, if usability and presentation isn’t a big concern, I can just use the Django admin as my backend without needing to write maintenance screens. There is an equivalent Flask Admin plugin, but I’ve not tried that either.

API building: I’m already familiar with the Django REST framework, I’m not sure if there’s an equivalent for Flask. The one Flask project we did, we rolled our own API backend.

I find that I tend to favor Flask when building quick proof-of-concept apps that don’t need a database. An example that was part of the app I migrated recently would be this quick Twitter app I wrote about a year ago, to find intersections between follows/followers. It didn’t need a database, only twitter API calls. (I have since merged it into the Django app, so the current version is using Django now.)

In general I prefer Django, but that is probably due to familiarity and experience. I would love to have a full-scale project that uses Flask though! Perhaps in the future, for now all my personal projects are running on Django.

Is SQL a dying art?

hello@roytang.net (Roy Tang) — Wed, 13 Nov 2019 00:00:00 +0000

I was helping my brother check some database issues the other week, and he mentioned how impressed he was with how quickly I was able to come up with SQL queries on the fly. I told him that SQL was one of the skills I considered myself to have mastery over. This shouldn’t be surprising given my early career path:

for most of my first year working as a software developer, I was working on reports which involved lovingly handcrafted (and oftentimes quite complicated) SQL queries. And there was a lot of Oracle PL/SQL too
by my second year, I was already conducting SQL training for new hires in the company
by my third year and onwards, I was already regularly asked to help other projects with their SQL performance issues

Most of my experience was with Oracle SQL, but as the ANSI standard has evolved quite well, the knowledge translates well to other databases. (Maybe not the PL/SQL though.)

These days I feel like SQL knowledge has become a much rarer skill among developers, chiefly for 2-3 reasons:

the introduction and widespread use of tools like frameworks with ORMs and business intelligence software, minimizing the amount of SQL that needs to be hand-written in most applications
the modern software development stack is much more complicated than it used to be, particularly for web development. When I started out, almost everybody was a “full stack engineer”, and that meant having to be familiar with most technologies you were using, including the database layer. These days, for any nontrivial project, developers on a team will often need to be specialized, with many working only on frontend issues, some on backend, and any SQL or stored procedure work is handled by a specialist developer or a DBA.
a possible secondary reason is the rise of alternative data storage solutions such as NoSQL databases.

Realistically, the need for people who can write good SQL may never go away, but the poor of people who can do so may be getting smaller due to the reasons above. Well, maybe it’s a good thing for me, since scarcity means it may become a more valuable skill as time goes on!

What to do in a production crisis

hello@roytang.net (Roy Tang) — Tue, 12 Nov 2019 00:00:00 +0000

Despite our best efforts as software developers, it can still happen: production goes down. Or some sort of bug introduces catastrophic data error. Hopefully you have a support/DevOps team to handle the response. If not, the dev team themselves have to step in. This usual means a mad rush to figure out what happened and how to fix it, sometimes during off hours and maybe even into the early morning, all while facing pressure from clients and higher-ups. I was advising another developer in such a situation a while back, and it was his first stint as technical lead so he was extra worried. I gave him a few tips to see him through the time of crisis.

Keep calm. In Tagalog, kalma lang. In a crisis, there can be a lot of stress and pressure to fix things immediately. It’s best to keep a level head and make sure the team is keeping it together. Don’t let your emotions carry you away, and try to avoid panic.
Solve one problem at a time until you can go home. This is a lesson from the movie The Martian. In a crisis it can be difficult to focus on the task at hand and if you’re not thinking straight you might just charge in there blindly and move things around until things get better. Follow step #1, then figure out what are the most pressing problems that need to be solved immediately, then address them one at a time. It can also help your team morale to identify which issues can be left until the next day. This gives them a concrete sense of what is set of problems to focus on, and motivates them to solve those so they can go home.
Be extra careful. In times like these, you may find yourself needing to deploy hotfixes directly to production or maybe even update data directly in the production database, in the interest of expediency. It’s also during times like these that you are more likely to make a mistake, especially if your team is under stress and working overtime/during off hours. So it would pay to be extra careful, maybe add an additional layer of review before deploying things directly to production. You’re already in a crisis, don’t make things worse by making a sloppy mistake.

Having to provide tech support during such a crisis is something you shouldn’t wish on any developer, but sometimes it can’t be avoided. (Some might even consider it a kind of rite of passage so to speak). Make sure to have the right mindset to be able to operate well under such stressful conditions.

DevNotes: Python's yield

hello@roytang.net (Roy Tang) — Thu, 07 Nov 2019 00:00:00 +0000

I’ve been using Python for well over 10 years, and I still don’t have an intuitive mastery of one of its keywords: yield. Everytime I see it in someone’s code I need to stop and mentally remind myself what it does. I figured I’d write a devnote to help improve my recall.

Typically, yield is used in a function with a loop, like so:

def some_func(lim):
    for i in range(0, lim):
        yield i

yield means the function returns a “generator” that can be used as an iterable in a loop:

for val in some_func(5):
    print(val)

You can also straight up convert the generator into a list via something like list(some_func(5)).

The equivalent function, without using yield would be something like:

def some_func(lim):
    result = []
    for i in range(0, lim):
        result.append(i)
    return result

This is definitely the kind of function I’ve written often! Now that I’ve actually written it down, yield is rather straightforward, and hopefully it can help me shorten some of the python code I’ll be writing in the future!

The Perils of Handover Documentation

hello@roytang.net (Roy Tang) — Wed, 06 Nov 2019 12:07:37 +0000

A while back I found myself having to figure out how to compile/build/run a mobile application. The developers previously assigned to the project were no longer available to consult with, but they did leave behind some documentation. However, their documentation quality left a lot to be desired. The instructions they left basically amounted to:

npm install
ionic serve
ionic codrova run android/ios

Okay, first sign of trouble is that their instructions were basically commands that anyone who knew the app used Ionic would be able to Google. But ok, I gave this a shot, even though I had not used Ionic/Cordova before, I assumed the simple documentation meant it would be straightforward (spoiler: it was not).

First problem: npm install fails. Okaaaay. I dig around for a bit and figure out there’s an incompatibility between one of the packages and my node version (12). Some more digging around tells me that the previous developers likely were using Node 8. I switch to Node 8 and npm install finishes successfully, somehow.
Second problem: ionic serve throws an error from one of the dependencies. A bit more digging around. Reading ionic docs. Stackoverflow. Turns out there was some extra configuration needed to be added if you’re doing this stuff on Windows. (I later found out the previous developers had used Ubuntu.)
Third problem: ionic serve throws a whole bunch of typescript errors. Why? After facepalming for a bit, I dig around again. And dig some more. Looks like dependency issues. But the package.json has dozens of dependencies, it would take me forever to figure out the problem. I find myself digging through the repository’s commit history trying to figure out if there was a recent change to the package.json that would cause the issue. What I notice instead is that a couple of weeks before the handover, someone deleted package-lock.json from the repository for some unknown reason. Deciding to try it out, I restore the old version of package-lock.json, do the whole npm install dance again, and sure enough the typescript errors vanish. But we’re not yet done.
Fourth problem: ionic serve no longer throws an error, but when it launches the browser, there’s a cyclic error stacktrace shown. Some stackoverflow answers tells me this is because something can’t be serialize it, and I trace the problem to a generic error handler that just calls console.log. I comment this out so that I can see the actual error, which is “cordova doesn’t exist” (or something to that effect). I realize this means that the mobile app uses some native-specific things, so Cordova is required and it’s not going to work when served via web, so why did they include this step?
So instead I do ionic cordova run android (after installing the necessary Android Studio/SDK things…this was after my recent computer troubles, so I had a fresh OS install.) And yes, long story short, I am finally able to get the Android app running in an emulator (and eventually in a device, and then the iOS version as well).

I recounted this because I could use it as lesson to note that even when you do have documentation, there are so many ways the documentation can be insufficient. In this case:

the dependencies/requirements were not specified
they did not have someone actually go through these steps from a fresh install to verify things would work

A case could also be made that they should have just had something like a Docker setup available, but they actually did have a docker-compose.yml in the repository! Sadly, it wouldn’t have worked either because it referenced some environment variables that I had no idea what the value needed to be. And besides, my PC doesn’t support Docker (silly Windows Home), but looking at the docker image did clue me in to what Node version to use.

In this instance, it was also a problem was that the old team was asked to provide the documentation only after the handover (due to circumstances beyond our control). Ideally, a handover would have them available for consultation should there be problems found with their documentation, and the documentation would only be cleared for acceptance after such review. (Realistically, a lot of documentation like user manuals wouldn’t even be looked at, but in this case, the ability to build the application was kind of crucial!)

Python: Markov Chains

hello@roytang.net (Roy Tang) — Wed, 28 Aug 2019 00:00:00 +0000

Back when I was still learning Python in 2008, one of the first “fun” scripts I wrote was a text generator using Markov chains. I’d run it against all the chat logs I had with people at work and serve the results from a webserver on my computer. THe results were often amusing and sometimes hilarious.

Since I’ve been going through my old scripts lately, I thought I’d update that script to Python 3 (read: add parentheses around print params and use pathlib) and run it against all the posts on this here site. I added the script to my deploy script for this site (thus further worsening my build times), so it should generate a new page every hour or so. I also added the generated markdown file to gitignore so it never gets saved to the repo. Every output of this script will be fleeting and ephemeral!

You can view the output here. You can view the script source here.

Devnotes: Python Pathlib

hello@roytang.net (Roy Tang) — Tue, 27 Aug 2019 00:00:00 +0000

Ever since I started learning Python back in 2008ish, I’ve been using it as my primary scripting language for various tasks such as processing log files, organizing my own file system, processing stuff on this blog, and so on. A lot of it is basically moving files around. In the days of Python 2, that involved a lot of imports of different libraries like os, shutil and glob. It can become a bit messy with so many imports, and I often can’t remember which import I need for a particular case and end up having to search for the documentation (or stackoverflow, let’s not kid ourselves here).

With Python 3, a new cleaner option is available to replace all of the above libraries: Pathlib, introduced in Python 3.4, provides an object-oriented way of doing file operations, replacing many of the most common uses I had for the libraries above. Some sample usage:

from pathlib import path

# Declaring a path object is just passing the path string
p = Path("/home/roytang/stuff")

# Other paths can be derived using the / operator
subdir = p / "favorites"

# p.glob replaces stuff like os.walk and glob.glob
for mdfile in p.glob("**/*.md"):
    print(str(mdfile))

    # path objects have properties for individual path elements
    filename = mdfile.name
    stem = mdfile.stem # filename without extension

    # path.exists() replaces os.exists()
    newdir = subdir / stem 
    if not newdir.exists():
        # path.mkdir replaces os.makedirs
        newdir.mkdir(parents=True)

    newfile = newdir / newfile # path objects can be either files or dirs
    # for file copying, you still need shutil!
    import shutil
    shutil.copy(str(mdfile), str(newfile))

For me it’s much cleaner to use than the old methods and require less imports (most of the time). I eventually hope to migrate all my older scripts to use Pathlib (the same way I migrated them away from Python 2 a while back), and moving forward I plan to use it primarily for filesystem operations.

Devnotes: Migrating Mercurial to Git

hello@roytang.net (Roy Tang) — Sat, 24 Aug 2019 00:00:00 +0000

Big news in online repositories this week is that Bitbucket is sunsetting support for Mercurial! This might be the death knell for Mercurial, although Git was already the super popular choice before. Back when I started using online source control for my personal coding projects I started out with Bitbucket over Github because they offered unlimited private repos and Mercurial (which I had already tried out before at work, so at first I preferred it over git). Now that Gitlab and Github both offer unlimited private repos, there’s no reason to stick with Bitbucket either. I had already migrated most of my private Git repos to Gitlab before, but hadn’t realized until now that I also had a couple of Mercurial repos there that needed to be migrated as well. I hadn’t touched those repos in so long that I didn’t even have Mercurial installed locally anymore! (Although I’m still using the code locally!)

Luckily, converting a mercurial repo to git turned out to be straightforward thanks to a project called fast-export. Modified instructions from a convenient stackoverflow post:

brew install hg # install hg on my mac first
cd ~
hg clone https://user@bitbucket.org/user/reponame
git clone git://repo.or.cz/fast-export.git
git init git_repo
cd git_repo
~/fast-export/hg-fast-export.sh -r ~/reponame
git checkout HEAD

The process was fairly quick, and the commit history from the Mercurial side was even retained, I would have been happy just retaining HEAD! After that, I just create a new private repo on Gitlab, set that as the remote, then push to Gitlab as usual.

It’s a good thing I remembered to check my Bitbucket account, as apparently they will be deleting the old Mercurial repos when time is up!

Devnotes: TT Miniproject (Django Rest Framework, Unit Testing, VueJS, Geocoding, Nightwatch e2e Testing)

hello@roytang.net (Roy Tang) — Tue, 20 Aug 2019 00:00:00 +0000

I recently found myself doing a really small project as sort of a proof of concept/demo for a potential client. It often seems that it might be a waste of time to do something like this since you don’t know if the project will actually push through or maybe the client will want something else. To kind of hedge my bets a bit, I decided to take the opportunity to try out some new technologies so that no matter what I at least learned something from all of this. (What is life if not learning?)

Django Rest Framework. I’ve been on projects using DRF before, but it’s my first time building a full CRUD-style API from scratch using it. There is a little bit of a learning curve when it comes to views and serializers, but for the most part it’s straightforward. I did enjoy how the framework has out of the box support for the most common REST API scenarios, and there are available backends for different authentication methods. I used JWT Authentication, which isn’t part of the library by default, but already had a pretty good implementation available elsewhere.

Python Unit Testing. I have dabbled a little bit with Django’s Unit Test framework before, but not having worked with any company that does TDD I’ve never tried doing it the “test first” way. For reference I used an online book: Test Driven Development with DJango. I actually enjoyed the exercise, perhaps a little too much. My impression of TDD has always been that the value lies in how much courage it can give the developer - the courage to try new things without worrying about whether you have broken something that was working before. This support came in handy while I was trying to figure out some issues with DRF especially around permissions etc. Really helpful for backend APIs like this.

Python Social Auth. The proof-of-concept needed some social media logins. I used Python Social Auth for this, really straightforward library with good documentation and supports a lot of backends. I ended up using Twitter and Github OAuth as those were the simplest to implement.

VueJS. The proof-of-concept needed to be a single-page app, so that meant a bunch of Javascript was needed. I willingly admit I’m still behind with the latest web frontend frameworks and trends. In my head I’m still living in a jQuery world. I dabbled a bit with React when trying out React Native, but I wasn’t fond of it. For this project I decided to use VueJS since I had skimmed through their documentation before and things seemed cool. I used it mostly for the dynamic parts and the data binding. I didn’t get too far into using components, my VueJS app was a huge monolith, which is apparently is frowned upon when using these frameworks lol. (Someone familiar with VueJS who looked at the code said it was “very strange”). I have barely scratched the surface of this framework, I will probably need to build something a bit more involved with it to get a better grasp. I will say that I enjoyed the VueJS syntax/structure better than React.

Google Maps API and Geocoding. The miniproject had some components that required me to show stuff on a map and do some geocoding, so I had to set up the API for that. Google’s documentation is fairly good here. One thing I didn’t expect is that you need a Google Cloud account with a billing method in order to access the API. My understanding is I wouldn’t be charged for anything since I was just using the free tier, I hope that was accurate.

Nightwatch e2e Testing. I’ve had some projects with end to end testing before, mostly via scripts or Selenium, but it’s my first time trying out one of these end to end testing frameworks by writing the scripts myself. I chose to go with Nightwatch, it was reasonably strightforward and worked on Mac and Windows. One thing I will note that initially gave me problems during set up is that using the ChromeDriver can be a bit finnicky on Windows. You have to specify the actual location of the executable in your nightwatch.json, like so: "server_path": "node_modules/chromedriver/lib/chromedriver/chromedriver.exe", instead of just the node_modules/.bin/chromedriver the documentation implies.

All in all the mini-project didn’t take much work - maybe 2-3 mandays of effort at most, and I did learn a lot from it, so no matter the outcome it was already a win.

Devnotes: PostgreSQL on the command line

hello@roytang.net (Roy Tang) — Thu, 08 Aug 2019 00:00:00 +0000

I decided to start doing small “devnotes” on developer stuff I’m doing so I can refer to them later (and also because I feel like I could use more technical content on this blog)

Today is about PostgreSQL. I haven’t used it much beyond standard ANSI sql stuff. You won’t always have a graphical interface to access your database, sometimes you need to ssh to prod and query the database from the shell.

The command line for PostgreSQL is psql. You can do:

psql [database-user-name] -d [database-name] and it should prompt you for your password.

But when I tried this today, I got:

psql: FATAL: Peer authentication failed for user "[database-user-name]"

The problem had something to do with the permissions available to my ssh user. The workaround was to tell psql I was doing this from the local machine:

psql: FATAL: Peer authentication failed for user "[database-user-name]" -h 127.0.0.1

Worked fine.

After connecting, you get a prompt that looks like this:

[database-name]=#

Now, when I connected today, I had to do some queries, but I wasn’t superfamiliar with the project’s schema, and would have to muck around to find table names and field names. The psql shell provides some handy shortcuts for that:

\dt - outputs a list of tables

\d [table-name] - describes a specific table, showing field names

You can also just directly run an SQL query, just make sure to end it with a semicolon.

If the command outputs a lot of records, it will show them to you in a page-able vim-like format. You can recognize it when it stops with a : after each page and you can hit spacebar to proceed, and at the end it shows (END). And like vim, you might not know how to get out of this view. I had to google it myself, you just have to hit q to go back to the psql shell.

Upgrading a React Native Project

hello@roytang.net (Roy Tang) — Sat, 20 Jul 2019 05:56:56 +0000

I have a small mobile app that I wrote using React Native (henceforth RN) back in 2017, currently deployed on the Google Play Store and Apple App Store. Shortly before my US trip, I got an email from Google telling me about a required action:

By August 1, 2019, all apps that use native code must provide a 64-bit version in addition to the 32-bit version in order to publish an update. This past January, we reiterated that this is required in order to make way for innovation and in anticipation of future Android devices that only support 64-bit code.

As the deadline approaches, we wanted to remind you that at least one of your apps* uses native code but does not currently offer a 64-bit variant

This illustrates for me one of the big problems with mobile development: that app was perfectly fine when I last deployed it in 2018 and now even though I haven’t changed anything, there’s additional work that needs to be done. I guess I’m too used to web development where we just deploy the app on a server and the server architecture doesn’t suddenly change for no reason. With mobile apps, the platforms and APIs and frameworks change quickly, leading to situations where even if your app doesn’t have any pending changes, you need to do some work on it anyway. (Granted, this can also happen with webservers due to security patches and such, but with managed hosting, that’s not something you worry about as often, and the resolution is often straightforward!)

As I was about to be out of the country on vacation for more than a month, I put this off and only started looking at it when I got back. Luckily, I did actually also have some minor revisions to do on the app, so I would do all the work in one go.

As it turns out, the version of RN I was using was too old and did not support providing the 64-bit version. I had to upgrade to a later version of RN, and after reviewing the process, it seemed like a nontrivial amount of work. There was an upgrade tool, but because I was so many versions behind, it couldn’t work directly. The problem seems to be in the initial boilerplate created by react-native init for the ios and android subprojects; the older boilerplate was out of date and needed to be updated, which was not straightforward given that you may have modified that boilerplate afterwards. I had to use some kind of diff tool to manually find the differences in the boilerplate and merge the changes manually.

That seemed like a lot of work, and the app wasn’t that large and didn’t have that much code, so I figured it was more straightforward to just create a new RN project from scratch, which would be using the latest version of RN (0.60.3, from 0.50.4). Afterwards, I port over the project code into the new project, them swap the new project back into the old project’s repo. That way I only had to update the few dependencies I was using in the project itself.

It was still a nontrivial amount of work, but I eventually got everything working through many iterations of trial-and-error.

I went ahead and upgraded the dependencies to the latest versions as well, to make sure there was no conflict with the new RN version
ReactNavigation had some backward incompatible changes:
- needed to initialize the gesture handler in the java class
- new function createStackNavigator instead of just StackNavigator constructor
- needed to use createAppContainer to initialize the navigator
some components such as WebView and AsyncStorage were no longer part of RN core and had to be imported accordingly
Android now requires that I explicitly prompt for the READ_EXTERNAL_STORAGE permission when trying to access the camera roll
Google now recommends that I provide an upload key for encryption, but since I reset the project, I also accidentally deleted the signing key I was using before. Oops! Momentary panic until I have the sense to do a search and realize that past me was wise enough to backup the file elsewhere.
Test that I can create the app bundle successfully.

By this point I more or less had a working Android build, then shifted over to iOS. Since I only recently got a Macbook Air, I actually hadn’t tried creating and publishing the iOS build, someone was helping me with that part before. Because of reasons, I wanted/had to do it myself this time around, so a bit of a learning curve and trial-and-error again.

clone the project from repo, npm install, try react-native run-ios, oops missing dependencies
find out about CocoaPods. Clone a fresh copy of the project, npm install then pod install (took a while), then try react-native run-ios again
needed to fix some dependencies I was using by manual linking and updating
camera roll also needed to be added to the podfile (a config I possibly lost when I reset the project)
fix a bunch of configs before I can actually run and test the app in the simulator
Success!

At this point, I make my other app changes and test on both my Android phone and the iOS simulator. The changes are easy, so now to try publishing.

Try to publish and upload to iTunesConnect. Takes me a few tries to figure out that I need to be updating stuff like bundle identifier, version number, etc.
iTunesConnect has some new requirements too (like the iPad version needs to support all orientations) which causes a couple more iterations
more screenshots are now required when submitting the app!
I set up TestFlight and verify everything is ok with the published app. (I only tested using the simulator before)
iOS app now submitted and “Waiting for Review”. I anticipate they will have some ridiculous request and it might take me a couple more submission rounds before I can publish
Google Play is more straightforward. gradlew releaseBundle is already working after my previous effort, so I just upload the bundle and distribute.

I see no indication that I have satisfied Google’s action item, but it let me publish the update without complaints, so I’m done!

My initial estimate that the work was nontrivial ends up correct, I spent way more manhours than I expected on the upgrade. Part of it was learning how to do the iOS deployment, so hopefully that ends up easier in the future.

10X Programmers

hello@roytang.net (Roy Tang) — Tue, 16 Jul 2019 05:56:56 +0000

The topic of the mythical “10x programmer” has been the topic of discussion recently on tech twitter, due to a thread listing out the supposed signs of being such a mythical beast.

10x engineers

Founders if you ever come across this rare breed of engineers, grab them. If you have a 10x engineer as part of your first few engineers, you increase the odds of your startup success significantly.

OK, here is a tough question.

How do you spot a 10x engineer?
— Shekhar Kirani (@skirani) July 11, 2019

The thread received a lot of negative responses, mainly because several of these items can be considered red flags indicating someone who doesn’t play well with others - hates meetings, prefer irregular hours, poor mentoring ability, disdain for documentation, find process miserable, works alone to produce great code, etc. These attributes remind me of Steve McConnel’s description of programming heroics in Professional Software Development:

Combine a shortage of skilled workers with the common tendency to set overly optimistic schedules, and the stage is set for the programming hero. Programming heroes take on challenging assignments and write mountains of code. They work vast amounts of overtime. They become indispensable to their projects. Success, it seems, rests squarely on their shoulders.

Project managers both love and fear hero programmers because they are smart, temperamental, and sometimes a little self-righteous, and because the managers don’t see any way to complete the project without them. In a tight labor market, replacing them isn’t an option.

Unfortunately, the reality is that for every programming hero who is capable of monumental coding achievements, there are other pathological programming disasters who just don’t know how to work well with others. They hoard design information and source code. They refuse to participate in technical reviews. They refuse to follow standards established by the team. The sum total of their actions is to prevent other team members from making potentially valuable contributions. A significant number of programming heroes don’t turn out to be heroes at all; they turn out to be prima donna programming ball hogs.

Despite the negative feedback, I found myself relating to the list in the thread above because most if not all of these items could have been used to describe myself at some point or another in my career. Especially in my earlier years, I had a profound dislike for meetings and documentation and process. I think this sort of attitude stems from the transition from being a solo developer (which you largely are in school) who enjoys the act of coding to being part of a team tasked to deliver the software. The first-time team member may find it difficult to appreciate having to do all these non-coding things (meetings, mentoring, documentation) because he doesn’t yet have an appreciation of how they can make things smoother for the entire team.

Not all of the items in the list are bad habits. Luckily, I think I grew out of most of the bad ones as my career went on and I became part of different teams and filled different roles. Not all of these are bad habits though. Going through the list:

10x engineers hate meetings. They think it is a waste of time and obvious things are being discussed. They attend meetings because the manager has called for a “Staff meeting” to discuss the features and status.

I’m still wary of too many unnecessary meetings, but I also understand how getting people away from their IDEs and together for a discussion can be necessary to bring a project back on track.

Timings in the office for 10x engineers is highly irregular. They tend to work when very few folks are around. If there is a crowd or all-hands meeting, they are not visible. Most of them are late-night coders and come late to the office.

I think locally, people tend to come in later during the day instead of earlier, due to Metro Manila traffic. I knew that if I wanted some focused time, I could come in early and not much people would bother me. I think this is generally an ok practice, as long as you’re still available during core hours when your team needs you.

10x engineers laptop screen background color is typically black (they always change defaults). Their keyboard keys such as i, f, x are usually worn out than of a, s, and e (email senders).

This one is just straight-up cargo-cult nonsense. Different people have different preferences. (Okay, I do prefer dark background IDEs).

10x engineers know every line of the code that has gone into production. If a QA or support folks alert an issue, they know precisely where the fault (or bug) is and can fix the same in hours vs days

I have been in this kind of situation, although I guess it’s a bit of an exaggeration. Typically if you’re working in a team with a nontrivial codebase and you “know everything”, some of your knowledge may be shallow, and some other devs who spent more time with each module may have more in-depth knowledge of how a particular module works. Having one guy with good oversight into the codebase can be handy for the team - for the reason cited above, they may easily be identify the cause of an issue, but the team should make sure he’s not the only guy familiar with everything. Some redundancy is needed so that you aren’t always dependent on the same person everytime. It’s also bad for you as the developer who knows everything about the codebase, as it can be very hard to go on vacation, you’ll have a tendency to get calls such as “Hey, just a really quick question, do you know where is the code that does XYZ?”

Most of the 10x engineers are full-stack engineers. For them code is code, they don’t care whether it is front-end, back-end, API, database, serverless, etc. I have rarely seen them doing UI work.

This is still true for me. And just the other day during a call I had to give a disclaimer that my UI work isn’t the best lol. I don’t think this is indicative of disdain for specialists, there should be room in a team for both generalists and specialists.

10x engineers can convert “thought” into “code” in their mind and write it in an iterative fashion. Given a product feature, they can write that entire feature in one or two sittings of 4 to 6 hours with a caffeinated drink without distraction.

I mean, this is just describing a developer who’s in the zone. It’s true that developers who can get into the zone on a regular basis are likely to be much more productive, that’s a tautology.

10x engineers rarely look at help documentation of classes or methods. They know it in memory and can recall from memory. They write code at the same ease as writing English. No breaks, no pauce, just type.

This one is a bit of BS, especially in modern software development where we have the meme of developers coding by stackoverflow. It might have been true in the older days when there weren’t a billion and one APIs and frameworks to know and learn.

10x engineers are always learning new frameworks, languages ahead of everyone in the company. They are not afraid of anything new. If there is something new (e.g. blockchain) they gobble up, setup, experiment before anyone is getting started.

This one is still true for me as of now. I still enjoy exploring new frameworks, languages and technologies, though I don’t always have the time for it. I think this is a good trait for a developer to have, but at the same time it may be dangerous to expect or require this of developers when putting together a team. Not everyone enjoys coding, and developers shouldn’t be penalized for not wanting to do software stuff outside of work hours.

10x engineers are poor mentors as they can’t teach others on what to do OR parcel the work. They always think “It takes too long to teach or discuss with others, I would rather do it myself.” They are also poor interviewers.

I’m sure I have uttered some variation of that quote more than once in my career. The tendency to try to carry all the load yourself is not only a disservice to yourself, it’s a disservice to your team as you’re not helping them grow. I will admit my mentoring skills in the early years wasn’t very good - I’m the type of mentor who tends to throw you into the deep end so you can quickly learn to swim. This is because that’s my personal learning style as well, I learn more from trying stuff than from a lecture or a book. I understand that not everyone learns the same way, so I do actively try to spot opportunities for me to learn to explain things better (I like to think writing in a blog helps!) and for better mentoring. And I think my interviewing skills have improved a bit through the years as well.

10x engineers don’t hack things. They write quality code and know exactly how the code has to evolve, and have a mental model of overall code structure. They write at most one design document, and the rest is in the code.

The idea of the most productive engineers not hacking things can only come from an idealist who hasn’t experienced schedule pressure. While of course quality code is an ideal, the best engineers know when it is necessary to make tradeoffs and workarounds to meet requirements and schedules. And they will write as many documents as needed to make sure things can be understood.

10x engineers rarely job hunt or move out of the company. They move out because you make their life miserable with the process, meetings, training, and other non-value-added activities. If you come across them, hold on to them. Celebrate them.

This was true for me for the longest time, I stayed with my first company for 15 years before deciding to move on. I would suspect this tends to be true for the type of engineer I described who enjoys the coding part and not so much the dealing with people part of the job. Moving companies means a lot more noncoding work: meeting new people, learning new processes and procedures, more meetings and trainings and interviews and what not. Laziness is probably also a factor.

The items listed here remind me of the type of programmer stereotype where one programmer goes into his office, tells nobody not to bother him, then comes out after three days having written all the code and solved all the company’s problems. It’s a ridiculous, dangerous stereotype that doesn’t mesh well with how modern software development teams work.

I’m not claiming that 10x engineers don’t exist, I’m sure they do, and I know from experience many of the above stereotypes can be true. But stereotypes are just that - stereotypes. Every programmer has had a different path and there can be superproductive programmers who also know how to well work with teams and can be great mentors and write great documentation.

I’m also not necessarily claiming to be a 10x programmer even though I claim to relate to the items here (false humility?). I would appreciate being paid a 10x salary though. :D

A second thread on 10x programmers I like, from Alan Cooper:

I’ve worked with more than one “10x” programmer. Their brilliance is astonishing, but their output is rarely practically realized. 1
— Alan Cooper (@MrAlanCooper) July 14, 2019

Learning from failure

hello@roytang.net (Roy Tang) — Wed, 17 Apr 2019 05:56:56 +0000

Success is not final, failure is not fatal: it is the courage to continue that counts.

– Winston Churchill

I already typed the above quote into the post, then realized I had already used it before. Whatever, just goes to show, I’m no stranger to failure.

I was reminded of this quote because recently I prepared a demo for a project that didn’t push through. At first I was annoyed at the wasted effort, but I realized that I had wisely taken the demo project as an opportunity to learn/sharpen some skills. Specifically, I used it to study the Django Rest Framework (for the backend API), and a little bit of VueJS (for the frontend), which means that even if the project isn’t going anywhere, I was at least able to hone some technical skills. The advantage of always looking to try new things is that your worst case scenario still leads to some self-improvement.

QUIZ: Well aimed? How well do you know CSS selectors?

hello@roytang.net (Roy Tang) — Sat, 13 Apr 2019 11:07:55 +0000

Shared via pocket: QUIZ: Well aimed? How well do you know CSS selectors?

A Quick Twitter App I Wrote

hello@roytang.net (Roy Tang) — Mon, 25 Mar 2019 05:56:56 +0000

I wish I had a more concise way to describe it, but I really don’t.

Some time ago this guy I follow on Twitter, visakanv wanted to know how to do a certain search: he wanted to know who a given famous person follows on Twitter, and among those, finds the one who follow him (visakanv), so he could network through them. I might not be explaining the concept too well, here’s the thread.

Anyway, in a fit of “I’m a bit bored, anything interesting I could do?” I figured I could set up a quick webapp to do that. I did a quick PoC in Python/Flask, with Tweepy for accessing the Twitter API. I normally prefer Django for my webapps, but since this was just a small, quick app, I figured Flask would be ideal. That worked out fine, except that I had to use my personal access tokens to for all the API calls. This means that if many people were using the app, it was likely to hit Twitter’s rate limits!

That was a few months back. A couple of days ago, someone emailed me asking about an error he encountered using the app, and sure enough it was due to the rate limits. I had a couple of free hours last weekend, so I figured I’d update the app a bit. I decided to add a Twitter login requirement, that way we can use the individual user’s access token instead of using mine globally. This would allow the rate limits to be per user and thus less likely to be hit. Now, I’ve tried implementing OAuth login some years ago and it can be terribly complicated, but luckily I found out that there’s a nifty Python package called Flask-Dance that made it super easy to add Twitter OAuth to a Flask app. This also let me discard the Tweepy dependency, since the Twitter OAuth session let me query the API endpoints directly. I also did a bit of UI cleanup and error handling and deployed the changes quickly.

The app is available online at roytang.net/apps/twitter. There’s a few other Twitter tools I’ve been meaning to build, so I’ll probably put them here too when I get around to them.

Lies and marketing

hello@roytang.net (Roy Tang) — Sun, 17 Mar 2019 05:56:56 +0000

I mentioned before that as an engineer, I’m not fond of marketing.

(Click to view full-size)

Image credit: Dilbert.com (Disclaimer: Liking the Dilbert comics is not an endorsement of Scott Adams’ politics)

It’s not that I can’t be good at salesmanship either. I have a good grasp of communication skills and think I have a decent chance of writing good copy. My main issue is that I’ve been exposed many times to sales/marketing practices that just seem dishonest downright or scummy. This happens both with me as a customer and with me on the side of the company trying to do the sales/marketing.

An example in the software engineering sphere would be a developer writing project proposals in response to tenders, and he is asked to write a technical response confirming that their product supports a particular tender requirement. Except he knows for a fact that he doesn’t, and when he raises this concern, he’s told not to worry about and that they’ll just implement the feature if/when they win the contract. And apparently that’s a regular practice in some companies. Disclaimer: I’m not saying I have been personally involved in such a situation, but I do know such situations exist.

I understand the motivation from the sales side, but not only is that intellectually dishonest (to say the least), it also puts your engineering teams in a bind should the project push through - you now committed them to do an extra feature without proper estimation or scheduling and so on.

For me it’s okay to sell something by promoting its strengths and advantages, but not by being dishonest about its capabilities or weaknesses. I suspect that this cripples me a bit in the real world where people with less scruples are more willing to bend the truth to make money. But for me personally, honesty and integrity are more important than making more sales.

Don't stay in the same place too long

hello@roytang.net (Roy Tang) — Sat, 16 Mar 2019 05:56:56 +0000

If I could give some advice to someone starting out in their software development career, it would be this: Don’t stay in the same place too long. The first company I worked at, I stayed with them for thirteen years, which I now feel was way too long. I have to admit, the work was hard and challenging, but I was young and had a lot of energy and was willing to work the long hours. I was good at the work and the enjoyed the company of the people I worked with so in a certain sense I got comfortable with the status quo and stayed a bit too long. But given a do-over, I would probably prefer to have left after maybe 4-5 years.

Hopping jobs too quickly can be fantastic for your salary growth, but it has some disadvantages - you don’t build close enough relationships, and too many hops doesn’t look good on your resume. Four to five years seems like a good period to stay, long enough to learn as much as you can and build your network before moving on.

The main advantage of not staying too long comes in the form of personal growth. When you stay at a place too long you tend to stagnate. Sure, it’s software development so somehow you’re always improving, but not improving as quickly as you would be if you were to be thrust into a new environment every few years. Your personal network of like-minded professionals will grow quicker as well when you switch companies. I was thinking about this because I was recently asked if I could refer any senior developers and I realized my personal network of high-level senior technical guys wasn’t that deep because of me staying in the same place for so long.

I don’t consider this a regret, I’m still pretty happy and satisfied with where I am in life. Thinking about this is more of an alternate universe “what if” scenario. I have to admit that staying that long did have some advantages - I got to work with and meet some good people, younger ones even, who became good friends. But this is not an argument in favor of staying that long, as you would still have the chance to build relationships even if you move on. It’s possible in that alternate universe I may have made even better friends or may have even been friends with the same ones I have now somehow.

So, given a chance to give advice to my younger self, I would probably recommend the second path. There is more uncertainty there to be sure, but life is risk, and to live is to explore. Taking the change to move to a new environment optimizes for personal growth, and it’s probably better to be doing it when you are young and full of energy.

What are the pros and cons of a programming career?

hello@roytang.net (Roy Tang) — Tue, 12 Mar 2019 05:56:56 +0000

Another repost from my Quora answers, this time some info for anyone looking to move into programming.

What are the pros and cons of making your career in programming?

Pros:

It is a very rewarding career financially. Software development often ranks in the top 10 highest-earning careers in most countries
There is a lot of scope - you could be developing web applications, mobile applications, embedded applications, client-side, server-side, data analysis, artificial intelligence, games, etc
It is very difficult to be bored. You can always automate away the boring stuff. Different projects always present different challenges. The field is evolving rapidly so there are always new things to learn.
It is both a creative endeavor and a technical endeavor
Lots of opportunities for remote work/working from home, if that is your thing
You get to work with a lot of smart people
Access to a worldwide community of people who think in a similar manner. Most beginner and intermediate problems easily solved through the internet
Generally, working in a programming company is more likely to be a meritocracy - people with better technical skill are more likely to be valued
There are a lot of opportunities - if you are unhappy where you are, you can easily look for another opportunity

Cons:

It is not for everybody. It requires a certain type of thinking that you may or may not be used to. Typically the field needs the ability to think logically and incrementally (step-by-step through how things happen) and to understand how multiple components work and interact together. More advanced challenges will also require significant out of the box thinking
After some time, you will tend to interpret things more literally. This is a side effect of working with computer programs that may affect your social interactions
Many people outside of the field will often find it difficult to understand the amount of work that goes into a computer program. This can lead to difficult conversations, especially with clients
Your nonprogrammer friends and relatives will often ask you for technical help, even though you know nothing about their printer and will most likely just google the information
Typically, the industry is very bad at estimation and scheduling, leading to a lot of overtime and the corresponding stress
The field is evolving rapidly, so there is often a need for a career programmer to self-educate to remain up-to-date with current trends
Encountering a highly difficult problem that no one else has been able to solve may drive you insane

Link to the original answer on Quora

(Annoyingly I just realized that the permalink to a Quora answer may change if the text of the answer changes, since they use the question slug instead of an id in the url.)

Studying a large project codebase

hello@roytang.net (Roy Tang) — Sun, 03 Mar 2019 05:56:56 +0000

Given my recent misgivings about Quora, I thought it might be a good idea to cross-post some of my answers from there into this blog, with some edits even. So here’s the first one! (stuff in italics were added during the cross-post)

How can you read and study a large software project source code?

Attacking a large, existing codebase that you are unfamiliar with can be a daunting endeavor. Don’t expect that you will be able to easily navigate the codebase quickly after just a few days of studying it. Familiarity will come with experience.

Some things that can help:

Don’t try to understand everything all at once. Figure out what is interesting to you and focus your efforts there
Follow the path of data. Easiest way to see how the program works. For example, if I were studying a web application that processes request parameters to generate a search result, I would trace how the program captures the parameters from the requests object, formulates it into a search query, sends it to the database, receives the results, formats the results, then sends it back to the browser. Looking at the typical paths through which data traverses the system can give you a good overview of where everything is. this may be a problem if you’re not familiar with the technology or framework the codebase is using, as some of the details may have too many layers of abstraction to trace. In that case, better to brush up and have a basic understanding of the tech first.
Try to fix a bug or implement some sort of change. If it’s an open source project, this is probably the best way to familiarize yourself with the codebase and get more involved at the same time. I’d suggest starting with minor UI bugs, those tend to be easiest to figure out since a text search of the source will often point you to where the correction needs to be made. For example, if the bug is that a particular error message is being shown under the wrong conditions, you can use a text search to find where the error message is declared, then trace all the code points that generate that message, figure out which one you need to update (probably by trial and error) and fix accordingly

Link to the original answer on Quora

Reddit PH: Software Dev Q&A

hello@roytang.net (Roy Tang) — Wed, 20 Feb 2019 05:56:56 +0000

I had some free time the other day so I randomly decided to post in the PH subreddit’s regular afternoon random discussion thread, asking for questions about software development. I ended up typing some longish answers, I thought I’d copy them over to the blog in case anyone was interested. TBH I meant more like StackOverflow type questions with specific technical problems, but I ended up answering mostly career-related questions, which is fine, but disclaimer: I don’t claim to be an expert, these are just my opinions on things.

capybara_c0de: project/s junior devs should include in his/her portfolio?

I’ve technical interviewed a lot of entry-level/junior level devs. Honestly any kind of project would probably be okay, as long as you’re able to explain what role you played, bonus points if you’re able to discuss technical decisions you made (why was this choice made, and why not this other one etc) or specific technical challenges you were able to overcome. That kind of stuff. Really, if you can demonstrate that you really worked on a project (and not just coasted by on teammates help etc), you’re already an above average interview.

Entry-level candidates might have more trouble since they often won’t have a portfolio, but the quality of candidates is a bit low locally, so if you can demonstrably do simple stuff like loops, recursion, stacks, etc you’re probably already an above average interview. Bigger companies will have written exams to filter out peeps who can’t do that though.

More random technical interviewing tips, even though you didn’t ask for that: being able to communicate well is a big plus, mandatory even. Lack of confidence makes the interviewer worry that you don’t really know what you’re talking about. Sometimes I’ll slip in something I know to be wrong “Diba pag cinall mo yan, ganito mangyayari?”, and see how the candidate reacts

veeequalseyeare: How many years of experience do you have? And what’s your basic salary?

Why do people keep asking about salary? That’s not really a programming or software dev question. I am a freelancing consultant at the moment, so I can’t really give a simple answer to that anyway (it’s almost always “it depends”)

I will answer the experience one - 15ish years in the industry (yeah, I’m old).

Edit: Also, there is a high range in the salaries of software devs in this country (not sure if the same elsewhere, probably), because:

some people are better at selling themselves than others
some devs are way better than others (and thus are more valuable)
each dev’s career path and growth is often unique, so it’s difficult to compare, especially once you rack up some years of experience

i.e. for a given number of years of experience, the salary band will be quite wide.

I get that when you’re starting out, salary is a big deal, but honestly if you’re a decent enough programmer, you can get into any place with above average salary. You can grow your salary a bit faster by hopping jobs every so often, but too many hops looks bad on a resume.

Anyway, my advice is that once you get to a decent salary range that is good for you (i.e. you’re not starving), focus less on comparing yourself to other people, and more on increasing your own value and managing your work/life balance. Money isn’t everything, beyond basic survival.

getmeoutofherealive: Anong advice ang maibibigay mo kapag gusto magcareer shift? Ex: IT Support to Development/Programming.

Shifting careers from IT support isn’t an unreasonable proposition, but you need to already know your stuff before going in for interviews. You will also still need some basic level of programming knowledge. A lot of things you’ll need can be learned on the fly/on the job (like usage of specific languages or tools or libraries), but you need a good basic foundation to build on.

If I’m interviewing someone coming from a nonprogrammming background, I’ll basically be treating her as an entry-level candidate. You will need to be able to justify why I should hire you over say, a fresh grad who still remembers his java programming from his college days. The interviewer will need to know you’re not trying to BS your way into the position. Having some work you can show off (side projects or whatever) can be a big help. You can also cite anything you’re doing in your current domain that might show off some related skillset. Like, if you did some minor scripting to gather support statistics, something like that? IDK if IT support people do that kind of stuff. See my other interview tips in the other answer elsewhere in this thread too.

captainph: web services in web development, ELI5 please

Wait, what do you mean? Web services are like web pages, except they’re endpoints that usually do only one thing and return some machine readable format like JSON or XML instead of HTML that can be rendered and viewed by a human in a browser. Basically you send them an HTTP request and they maybe do some processing and spit back an HTTP response that you consume.

Some types are simpler than others - REST-based web services are usually very simple for example, and can sometimes be invoked directly via the browser address bar. Some follow more complicated protocols (SOAP) that support things like authentication and signing and so on.

Web services are developed the same way as normal web pages, with the usual backend processing and stuff (but there are some frameworks to make this easier, especially for complicated protocols like SOAP), but the client program (the one calling the web services) does not have to be another web application. It can be a mobile app for example.

One of the typical usages would be third-party integration. For example, Philhealth I believe currently has some kind of e-Claims web services in development (maybe? IDK the current status), where the idea is to allow third-party systems to submit Philhealth claims electronically. The idea is that individual hospitals have their own medical records systems, those systems can send HTTP requests to the e-claims web services in order to for example check whether a patient is eligible for a claim.

Ok, sorry I know that explanation doesn’t really work for a 5-year old lol

acequeen21: Do you have plans going abroad? Either work temporarily or move permanently. Why or why not?

Nah, I like living here, in spite of everything, and I’m programming avails me a good living naman. I’ve worked abroad on short stints (pinadala ng company), but TBH, I had never even considered leaving the country long-term, but I started looking into it after 2016. Hassle for me na ma-uproot and to try to start over elsewhere.

Hacktoberfest

hello@roytang.net (Roy Tang) — Thu, 14 Feb 2019 05:56:56 +0000

Last October I participated in #Hacktoberfest, sponsored by DigitalOcean and Github. It’s a “celebration” to promote open source activity, and basically you just need to submit 5 pull requests to any github repository, and they give away swag to anyone who completes the activity. Microsoft held a [counterpart celebration] where they only require you to submit 1 pull request to any Microsoft repository.

I’ve always wanted to start participating in Open Source, but it’s a bit difficult to find a good place to contribute (other than logging issues of course). So when I heard about this activity, I thought “Why not?” Anyway, I’m not ashamed to admit 4 out of my 5 PRs were to small/minor projects, some of them seemingly created specifically to help farm Hacktoberfest PRs lol. I did manage to get one pull request accepted for my current favorite editor, VSCode! So that’s something! It was just a super minor bug fix, but it feels good to know that some future build of VS Code may have some code I wrote in it. This also qualified me to get swag from Microsoft! I haven’t gotten any open source-related swag since the dearly-departed Adobe Flex shirt from 2008 (RIP Adobe Flex).

I’m only posting about this now, four months after the fact, because my Hacktoberfest swag only came in last week and I had to swing by the post office to pick them up. Here’s some pics:

Passed by the post office today to pick up some parcels, turns out it was the #hacktoberfest swag I got for participating last October. Thanks Digital Ocean/Github/Microsoft! (Click to view full-size)

I’m not really a stickers guy, so I’m not sure where I’ll put the stickers. Maybe they’d look good on my Macbook Air? I like the shirts though, since Americans have better shirt sizes from me than locally.

I’m still not “active” really on open source, but at least it was a nice first step. Maybe next October I’ll look for some bigger PRs to do!

TriviaStorm: Text and Answer parsing

hello@roytang.net (Roy Tang) — Sun, 03 Feb 2019 05:56:56 +0000

A while back I started a Twitter trivia bot as a weekend project. That bot is still up and running on Twitter, you can check it out there!

But today, I thought I’d write about the answer-checking mechanism used by the bot. It was a bit interesting to me because it was the first nontrivial use I had for Django’s unit testing framework. I’m not too keen on unit testing web functionality (something I still have to learn), but this seemed an appropriate first use of a unit test framework for several reasons:

the bot had to be able to handle a wide variety of answers
there were a lot of test cases to check and a single checking function handling everything - I couldn’t risk breaking previous working tests
inputs were discrete and outputs were easily checkable
I needed to be able to add new test scenarios all the time as more problematic answers were provided

The project currently isn’t open source, but I did make a gist of the tests.py I used here.

The check function basically accepts three parameters:

a checking mode (currently only supports EXACT and ALL_ANYORDER)
the answer phrase provided by the player
a set of valid answers accepted for the question

There’s a number of test cases already handled:

checking should be case-insensitive
articles should be ignored if they’re at the start of the answer phrase
numbers should be acceptable for the spelled-out versions i.e. “7” should be accepted for “seven”, and vice versa
some minor soundex (phonetic matching) support (via Python Jellyfish)
handling of questions that support multiple answers. This is what ALL_ANYORDER is for - it means that all the given answers must be provided, but they can be in any order. i.e. if the valid answer set is "Huey", "Dewey" and "Louie", then "Louie Huey Dewey" should be accepted as an answer
nonalphanumeric characters should be treated as whitespace, except in some special cases
special case: abbreviations like "don't" or "can't" should be treated as if they were a single term like "dont" or "cant" instead of "don t" or "can t"

The answer checking definitely still isn’t perfect, but I’m pretty happy with where it’s at right now. There is also definitely an element of subjectivity as to which answers should be accepted. One time a player complained that his answer "Batman vs Superman Dawn of Justice" should count for "Batman v Superman: Dawn of Justice", but for this particular question I had chosen not to allow “vs” for “v” because that was the actual movie title, which might be unreasonable now that I think about it!

I do know that I need to implement a better “synonym” handling, i.e. mapping of "v"<->"vs" and other terms like "mr"<->"mister" or "natl"<->"national". The problem with handling things like that is that is the possible combinations of phrases expands when multiple such terms are found in the same answer, so it can’t scale too well. I suppose I need to normalize the answer sets at the time the question is defined. What do you know, I figured out how to do something just by writing a blog post!

I do have a bunch of other enhancements planned for the trivia bot, including support for slack and discord, and a longer time frame roadmap, but I’m not sure when I can commit more time to it. Still, it’s turned out to be a pretty fun endeavor, I’m hoping it leads to something cool!

I still Google the most basic things

hello@roytang.net (Roy Tang) — Thu, 17 Jan 2019 05:56:56 +0000

I’ve been working with Javascript for more than a decade. Last week while helping another developer debug a problem, I had to Google how to check if an element exists in a Javascript array, something superbasic, that one would expect most newbies to know. I’m sure I Google some superbasic thing at least once a week. It’s not embarassing or anything, it’s a common occurrence. I’m surely not alone. Just last night a tweet about this crossed my TL:

In C++ we don't say "Missing asterisk" we say "error C2664: 'void std::vector<block,std::allocator<_Ty>>::push_back(const block &)': cannot convert argument 1 from 'std::_Vector_iterator<std::_Vector_val<std::_Simple_types<block>>>' to 'block &&'" and i think that's beautiful
— mcc (@mcclure111) June 1, 2018

People joke about how developers wouldn’t know what to do if Stackoverflow wasn’t around, but it’s a fact of modern software development that most developers will have access to a ton of easily searchable reference material available online. That’s why things like phone interviews where people ask you to explain what the volatile keyword means in order to test whether you’re really a Java expert are BS. (I have a Java certification and while I’m sure I knew the answer to that at some point, if I needed it today, it would certainly come in the form of a search result.).

Software development is a difficult field, and it can sometimes be intimidating to work with more senior developers or ask them questions or show them your code. But it gets easier when you realize not everyone knows everything all the time, and that not even the most senior people know everything. And that this ability to seek out advice and yes, even stackoverflow results, is part of the skillset you need to be developing.

The nature of the field is that we are almost always encountering new situations and new problems and new challenges (well, either that or be bored doing something irrelevant). That means a lot of the time when we are working on something worthwhile, we are figuring out things as we go.

Finding Time to Learn new Things

hello@roytang.net (Roy Tang) — Mon, 14 Jan 2019 05:56:56 +0000

Someone responded to my post on things to learn in 2019 by asking how one finds the inspiration to learn all of the things. Well, my first answer was that those are just things I find interesting and may look into, but that’s not really an answer for the inspiration part.

Software development is a very wide field, one where the amount of things you can learn increases daily, so it’s almost impossible to keep up with everything. I think that having a natural tendency or curiosity towards learning new things is a distinct advantage in this industry. Having at least some passing familiarity with a wide range of technologies gives you more options in both job hunting and deciding on technical approaches.

When interviewing developers, I often viewed it as a distinct advantage when a candidate invested their own time in trying out new tech. However, I realize that spending your own time to learn things on your own is another form of privilege that I have. Not all developers will have the free time or motivation to do that unless it was part of their work tasks. So insisting that developers should do this in their free time is a bit elitist to say the least.

I read a blog post the other day saying it’s actually beneficial for companies to pay for their employees to learn stuff during work time. Quote:

Giving developers time to learn makes good business sense in a competitive world where attracting and retaining skilled people is an advantage in the marketplace.

While this makes sense, the problem with this is that from experience, many companies are running their projects on very tight schedules, some often almost always behind schedule, which makes giving developers free time for self-improvement will often be unattractive to higher-ups. I know that in my previous jobs, the only way I could justify learning a new technology would be if it was required or important for a specific upcoming project. In a way, having the space to allow developers this privilege is in itself a privilege that not all companies can afford.

Supposedly in the modern softeare industry, demand for developers is still high compared to supply so individuals should still have some leverage here and should vote with our feet. Younger or entry-level developers should ask companies about their policies for upskilling their employees and prefer those which give them more opportunities, and disfavor companies where the devs’ technical knowledge tends to stagnate, so that companies will learn to adjust. Such factors can only be good for their career in the long run.

Git vs CVS

hello@roytang.net (Roy Tang) — Sat, 12 Jan 2019 02:06:56 +0000

A while back we were tasked with helping a client’s internal dev team to migrate their repositories from Subversion to Git. The distributed VCS seemed ideal for their situation - they had a very small in-house dev team managing contributions from external subcontractors. The main rationale was that their process of merging contributions from the external developers was extremely complicated and often resulted in conflicts that were challenging to merge. Before this, I hadn’t actually used Git too deeply myself (aside from cloning stuff from Github), and especially not in a team setting, so the training one of our other engineers gave them was a good opportunity for me to become familiar with Git as well. During the succeeding months we guided them through the first few pull requests and merges; their repository structure and build process was quite complicated so it took us a while to iron out the kinks. There was often some difficulty when merging PRs from old forks and so on. I mostly let our other engineer who had more Git experience handle any merging. Sometimes I wondered about whether the transition was really worth it for the client, but it all seemed to end up okay.

I was reminded of this because I recently tried the whole workflow of fork and change code and pull request and merge with upstream to resolve conflicts in a larger, non-trivial project. (I don’t count the few PRs I did for Hacktoberfest last year, as those were relatively simple). This one was a whole new submodule that I had to merge into the main repository, so there were a lot of changes on my end. I wasn’t expecting any conflicts though, because I was only adding new stuff and not modifying existing code. So I tried to make my PR without merging from upstream, not expecting any issues. Unfortunately, it turns out some CR/LF shenanigans between my Windows machine and the repo meant one of my commits had more “lines changed” than I intended, resulting in a merge conflict. Fortunately, it was easy to simply merge from upstream and resolve the offending files manually. Didn’t take me more than 5 minutes to fix the issue.

My experience with this was a bit amazing in retrospect. Mostly because for a large part of my career to as recently as four years ago, I worked with CVS as a version control system, which is ancient by modern standards. I remember during my first project, we had three branches: DEV, UAT and PROD. And moving changes from any given branch to another was a pain, each file had to be manually re-applied to the other branch. So my instinct has always been to dread any large-scale merge (something I have to unlearn apparently).

Git has a few other advantages of course. One of my favorites is being able to address specific commits individually by hash, and see the file changes and so on. When we were using CVS if you wanted to rollback to a specific change, you had to have the foresight to have tagged that change so that you can check it out again. And CVS for us was always very slow for larger projects with thousands of files, leading to annoyingly long build cycles and wait times. Git seems to be a lot more performant.

I hold no ill will towards my old company for making me use CVS all those years. When I started out it was a reasonable choice, but as time went by better options simply became available, and like many companies it was difficult to leave entrenched systems. I did propose the use of distributed VCS over the years; at first Mercurial and later Git as well, but there was never a good time to migrate without being overly disruptive (this happens when Everything Is Urgent). Probably if I had gotten in on a larger project on the ground floor, but the last time I had a chance for that was around 2007 I think, when Git was still in infancy. Still, I heard from people still there that they’re already in migrating to Git, so good for them (even though they waited til I was gone!).

Tech to Learn in 2019

hello@roytang.net (Roy Tang) — Tue, 08 Jan 2019 05:56:56 +0000

One of the things about self-identifying as a “Full Stack Developer” or “Solution Architect” is that there’s no shortage of things to learn, and oftentimes it’s good for your career-wise to at least have some passing knowledge of a bunch of technologies. It helps that I really like the field as well. I try to make sure I study or learn at least one new programming language or framework every year (though I am willing to stretch that definition as needed). Here’s a list of tech I’m looking at learning in 2019:

Rust or Golang. Between Python and JS, the last few years for me have been mostly focused on dynamically-typed languages that don’t require precompilation. With my recent revisit of C++, I’m a bit interested in checking out a new statically-typed compiled language. Rust and Golang or the primary candidates here, I’ll probably toy around with both to see which one I like better. The problem here is that I typically do web-based programs, so finding a project where it would be worth using one of these might be challenging.
Swift. Since I recently bought a Macbook Air, that opens up the possibility of trying out Swift. I was involved in a mobile project when Swift came out and found it interesting when I was first going through the documentation - maybe because I thought Objective-C was a hot mess (or at least our codebase was back then). I never did get a chance to try it out, so maybe I will this year - if I find an appropriate mobile app to work on.
Kotlin is to Java and the JVM as Swift is to Objective-C and iOS. It’s young, but as a JVM language it has direct interoperability with Java programs, and it’s supposedly a first-class language nowadays for Android development. If I do decide to work on a native mobile app, Kotlin is a prime choice for the Android version. (Assuming I’m willing to dive back into the JVM ecosystem…)
CSS Grid. I’ll be the first to admin I’m a bit behind on frontend practices; my default layout framework is still Twitter’s Bootstrap. If I get the chance, I’ll do some of my personal web projects using CSS grid.
Vue.js. Speaking of frontends, I never got too deep into the current crop of frontend frameworks. Funny story, we were meeting with a database vendor to discuss solution options for a client and I mentioned I had already built a small prototype web application using their product. They asked me what framework I was using on the frontend and when I told them I wasn’t using any, I was told that was “hardcore”. While I do think it’s helpful for full-stack web developers to be familiar with vanilla JS, I am interested in trying out at least one of the three prevalent frontend frameworks - Vue, React, and Angular. I have kind of tried React a bit with React Native, but at this moment, I’m leaning more towards Vue.
Machine Learning. Specifically, I want to try out some text generation using recurrent neural networks, like those posts where people generate screenplays etc. I’m not too conversant with the theory, but reading some examples and tutorials I think this is something doable. Just need a reasonable corpus of texts to work with…
TDD. Yup, test-driven development. I’m way behind the curve on this one. I understand the theory and the benefits, but I’ve never been in the position to have TDD in my projects. It’s difficult sometimes to get management buy-in. At my past job, all of the testing was still manual. I’ve dabbled in TDD a bit, tried out some frameworks, but never full-scale in one of my projects. Maybe I’ll try to write some tests for one of my side projects this year.

That’s it. I think it’s quite a lot actually. I’ll review this post towards the end of the year to see which of these I end up doing. Or maybe I’ll get into a freelance project that involves something new not on this list, who knows?

What about you, what’s your list for 2019?

Solution Architect

hello@roytang.net (Roy Tang) — Mon, 07 Jan 2019 05:56:56 +0000

Although I still primarily identify as a “Full Stack Developer”, during the past few years I’ve also found myself in a role called “Solution Architect”. The thing about being a solution architect is that there isn’t really a clear definition of the role, what it involves, or the scope of responsibility. I suppose it depends largely on the organization and the project. The role mostly involves making techical decisions on a larger scale, like project-wide or organization-wide, rather than on the micro day-to-day technical decisions involved in a typical software development involves. These decisions can encompass not just coding, but also requirements gathering, application design, database design, modelling, testing methodology, processes, deployment, and so on.

This is actually pretty much what I was doing during my last full-time job anyway. I was basically involved in technical decision-making at all levels. So there’s not much change, except that now there’s a formal name/title to the role.

Despite that role, in my past job I still insisted on taking on some of the coding work myself, typically the more difficult ones. I think it’s important as a technical person to maintain some familiarity with the low-level coding details, even when you’re transitioning to higher-level work like management or architectural work. (Also, I still enjoy the vagaries of day-to-day coding.)

For a solution architect in particular, it helps to be familiar with a wide array of technologies, since it informs your technical recommendations. You won’t always get such exposure through official projects either, so it helps to be interested in these things on your own so you can read up or try them out in your spare time.

Interestingly, the term “solution architect” has been increasing in usage over the past few years, if Google Trends is any indication:

(Click to view full-size)

(I originally tried Google Trend’s embed option, but the embedded chart kept crashing my browser tab, so now you get a screen capture instead)

If you’d asked me five years ago I would have said such a role was within the scope of a senior engineer or technical lead. I guess as time goes on, the role becomes more distinct and formalized? At this point, I think the concept might still be a bit too nebulous for anyone to really specialize in it though.