django on Roy Tang

Site Migration from Hugo to Django

hello@roytang.net (Roy Tang) — Tue, 27 Oct 2020 00:00:00 +0000

Leaving Hugo

It’s been a bit under two years since I migrated the site from Wordpress to Hugo. As discussed in this post one year ago, I was very happy with the general workflow of managing posts through markdown files in git, but had big problems with the Hugo build time, largely a consequence of my social media archiving. At that time, I didn’t want to invest effort into migrating to a different backend, but the problem has only gotten incrementally worse since then, so I decided to take the jump and started working on it last month.

Migration goals

The main goal is to get rid of the unnecessary build time. I also don’t want to have to store all the pre-generated HTML files. It’s fine for a small site, but with the large number of posts here, there’s a large amount of unnecessary duplication, considering listing pages and such. That means reverting to a system where each page is generated on the fly.

I want to retain the current workflow of managing content via markdown files, so that rules out something like Wordpress. Still, I figured I needed some kind of database for indexing, so that things like listing and archive pages work properly. Given that we would be migrating away from a static site, I fully expect a performance hit, at least initially, but ideally it’s not too bad.

I’d also like to retain the current content written for Hugo as much as possible. This means the new system will need to be able to parse the custom Hugo shortcodes I’d set up for the site.

I would also be migrating to a new server which I would be managing myself, which means I would need to set up the deployment myself.

Why Django

I naturally fell back on Django, it’s my ORM+backend framework of choice, and the one I’m most familiar with. Also, I already had a handful of small Django apps that I needed to migrate to the new server as well, so no matter what I needed to have a Django app on the server anyway. I also had a bunch of Python scripts that were doing already doing support functions for the site like search and ingestion of social media content, so that’s something to fold into the new backend as well.

Why Bother?

I will fully admit I spent far too much effort unnecessarily doing this, but the realization is that it’s actually work I enjoy. This site is primarily a project for myself, and I enjoy the time working on it and learning stuff and improving things.

That being said, the effort wasn’t too much. There was a lot of code to reuse and migrate, so most of the effort was testing and making sure things worked. Most of the new code was for parsing the markdown files to render the hugo shortcodes as HTML; I basically wrote Django templates for each Hugo shortcode to be supported.

Changes

The migration was largely done in-place: site design remains mostly the same, post URLs should be more or less retained. I did take the opportunity to make one nontrivial change: Photos are now under Notes category. I really only had a separate photos category before because in Hugo I didn’t have a way to query posts that have attached images. Even with everything under Notes, since I have a database now, I can do silly things like show random photos. Also, now the notes section is closer to a microblog/tumblr-like feed.

The source for the site content will still be available publicly via Github, as it was before. Unfortunately, I am not ready to make public the source for this site, because (a) a lot of it is old code that I reused so it looks terrible; and (b) some level of security by obscurity I guess?

Not that the site needs to be secure; there isn’t actually any critical/important data being stored anywhere. But the fact that a database now exists means I need to follow some kind of security practices.

The server is now hosted on DigitalOcean. Here’s my affiliate link. Since I’m on a self-managed VPS now, that means Github webhooks now work, so whenever I push new posts, they take less than a minute to now appear on the site. This is a vast improvement over the old system where it would take anywhere between 30-60 minutes depending on how Travis was feeling.

Speaking of Travis, I set up a new mirror, Travis now publishes to https://mirror.roytang.net once a day, hosted on Netlify. The mirror is still generated using Hugo! I don’t know how long I can keep up having the content generate correctly for both builds, but we’ll see how it goes.

Summary

All of this actually went out much smoother than I thought it would. Took less effort than expected, deployment was a breeze. The listing pages have slightly worse performance, but individual pages are hitting 100 on Pagespeed, so that’s mostly fine.

This post was just the overview of the site changes. There’s still a bunch of technical notes to be written up, so that’s stuff for a future blog post.

The last time I used a Django backend for my website was waaaay back in 2008, and that didn’t last more than a year before I switched to Wordpress. We’ll see how long this one holds out!

Flask vs Django

hello@roytang.net (Roy Tang) — Fri, 24 Jan 2020 00:00:00 +0000

In a bid to reduce the number of webapps actually running on my server (for resource consumption reasons), I decided to migrate a small Flask app I had and merge into this larger Django app where I have a lot of my personal data tracking stuff. The Flask app was small enough, mostly containing backend support for this blog (like search and comment submissions) and some Twitter things. The migration was straightforward, taking around half a day, most of that was wrangling with Twitter API rate limits.

It got me thinking, why did I build this using Flask in the first place? And that led to the more general question: when do I prefer using Flask over Django?

Personal experience: I’m much more familiar with Django than Flask, having started using it back in 2008. Professionally, I’ve also worked on more Django projects, compared to only one Flask project that actually made it to production.

The usual quick comparison is that Flask is the more lightweight web framework, suitable to smaller apps like microservices perhaps; while Django is the big monolithic web framework with all the bells and whistles already available. In practice this means Django already provides a lot of functionalities out of the box, while with Flask you have to install and figure out extensions for even the most common tasks like database management. A negative for Flask, but partly due to my inexperience: if you were already well-versed in the use of those extensions, I imagine getting everything together and setting them up will be easier.

If there is need for a traditional RDBMS, I will probably use Django. I’m not super familiar with SQLAlchemy, the usual recommended ORM for Flask, so that’s a big factor. That being said, if I needed to work with something like a NoSQL database or some other nontraditional backend, it might be better to consider Flask since Django isn’t super flexible in that record - most of the pluses are oriented towards RDBMS-backed models.

Admin: The built-in Django admin is a big plus for me; it means that for things like simple record CRUD, if usability and presentation isn’t a big concern, I can just use the Django admin as my backend without needing to write maintenance screens. There is an equivalent Flask Admin plugin, but I’ve not tried that either.

API building: I’m already familiar with the Django REST framework, I’m not sure if there’s an equivalent for Flask. The one Flask project we did, we rolled our own API backend.

I find that I tend to favor Flask when building quick proof-of-concept apps that don’t need a database. An example that was part of the app I migrated recently would be this quick Twitter app I wrote about a year ago, to find intersections between follows/followers. It didn’t need a database, only twitter API calls. (I have since merged it into the Django app, so the current version is using Django now.)

In general I prefer Django, but that is probably due to familiarity and experience. I would love to have a full-scale project that uses Flask though! Perhaps in the future, for now all my personal projects are running on Django.

Django Blog Application

hello@roytang.net (Roy Tang) — Sun, 28 Oct 2018 05:02:37 +0000

Ten years ago this month, I started studying Django by trying to build my own blog application. I found the code lying around while I was going through some backups lately. It’s way out of date, it uses an early version of django. I thought of bringing it up to speed, but that didn’t seem practical. Instead, for archival purposes, I cleaned it up a bit and uploaded the code to a github repo. (Helpful github immediately warned me that having a very old version of Django was a security risk lol). There’s a lot more information in the README.md of that repo. I actually used this as my main blog engine for a while before I decided the maintenance effort wasn’t worth it and switched to WordPress.

Tangent: I’m not actually 100% happy with WordPress, and when I found this old code I was tempted to maybe trying building Yet Another Blog Application (TM), except maybe using the opportunity to learn some other framework. My 2018 personal to-do list is already way too long though.

New note

hello@roytang.net (Roy Tang) — Thu, 01 Feb 2018 07:11:13 +0000

According to the documentation here: https://djangobook.com/syndication-feed-framework/

If link doesn’t return the domain, the syndication framework will insert the domain of the current site, according to your SITE_ID setting

However, I’m trying to generate a feed of magnet: links. The framework doesn’t recognize this and attempts to append the SITE_ID, such that the links end up like this (on localhost):

<link>http://localhost:8000magnet:?xt=...</link>

Is there a way to bypass this?

New note

hello@roytang.net (Roy Tang) — Thu, 16 Sep 2010 14:16:40 +0000

Like, have them logged to the console?

Reply to 'ValueError in Django' on stackoverflow at https://stackoverflow.com/questions/2032360/valueerror-in-django:

hello@roytang.net (Roy Tang) — Sat, 09 Jan 2010 04:47:32 +0000

Reply to 'ValueError in Django' on stackoverflow at https://stackoverflow.com/questions/2032360/valueerror-in-django:

urlpatterns = patterns('',     
    (r'^salaries/employee/$', list_detail.object_list, 'employee_info'),
)

The third item in the tuple needs to be a dictionary, not a string. Try removing the single quotes around employee_info:

urlpatterns = patterns('',     
    (r'^salaries/employee/$', list_detail.object_list, employee_info),
)

Andy McKay's blog

hello@roytang.net (Roy Tang) — Sun, 06 Sep 2009 04:53:07 +0000

Shared via delicious: Andy McKay's blog

Django snippets: Facebook shell

hello@roytang.net (Roy Tang) — Sun, 06 Sep 2009 04:52:51 +0000

Shared via delicious: Django snippets: Facebook shell

djangozen | find stuff for django

hello@roytang.net (Roy Tang) — Tue, 31 Mar 2009 21:53:48 +0000

Shared via delicious: djangozen | find stuff for django

Stylesheet Switcher using Django

hello@roytang.net (Roy Tang) — Sat, 14 Feb 2009 14:17:32 +0000

You may have noticed the new color scheme and new “Theme Switcher” widget in the sidebar. I had done some CSS work during the past month in the office and it made me want to tweak the stylesheets on this site a bit. I figured I might as well make it easy to switch stylesheets, so I wrote a small Theme Switcher django app. (Well, it’s more of a stylesheet switcher I guess)

The model is simple:

class Theme(models.Model):
    slug = models.SlugField()
    title = models.CharField(max_length=250)
    css_path = models.CharField(max_length=500)

    def __str__(self):
        return self.title

I only needed one view – to switch the current theme.

Added the ff to urls.py:

(r'^theme/([a-zA-Z0-9\-]+)/', 'themer.views.switch'),

The view code simply stores the selected theme as a cookie:

def switch(request, theme_slug):
  """ Sets a cookie to specify the new theme, then redirects to the referer or root url """
  theme = get_object_or_404(Theme, slug=theme_slug)
  referer = "/"
  if "HTTP_REFERER" in request.META:
    referer = request.META["HTTP_REFERER"]
  resp = HttpResponseRedirect(referer)
  resp.set_cookie("theme", theme_slug, 100000)
  return resp

To load the stylesheet, I set up a context processor to pass both the current theme and the list of themes to the context. I wanted to do this using template tags, but I couldn’t figure out how to extract a cookie from within a template tag.

from themer.models import Theme
def context_processor(request):
    theme_list = Theme.objects.all()
    if "theme" in request.COOKIES:
        slug = request.COOKIES["theme"]
        try:
            theme = Theme.objects.get(slug=slug)
        except Theme.DoesNotExist:
            theme = None
    else:
        if theme_list.count() > 0:
            theme = theme_list[0]
        else:
            theme = None
    return {"current_theme" : theme, "theme_list" : theme_list}

Then in my base template file, I add the following to the head section:

<link rel="stylesheet" href="{{ current_theme.css_path }}" type="text/css" />

And add the list of themes into the sidebar.

<h2>Theme Switcher</h2>
<ul>
            {% for theme in theme_list %}
            <li>
            {% ifequal theme.slug current_theme.slug %}
            {{ theme.title }} <span class="note">(current)</span>
            {% else %}
            <a href="{% url themer.views.switch theme.slug %}" title="Switch to {{ theme.title }} theme">{{ theme.title }}</a>
            {% endifequal %}
            </li>
            {% endfor %}
            </ul>

And we’re done!

Of course, I’m still not particularly strong in web design, so the two current “themes” really just switch around the color scheme. The light blue theme is named Azorius, while the old black and green theme is named Golgari, both named after the respective color guilds in Magic: The Gathering’s Ravnica block.

Hopefully this gives me a chance to whip up more interesting themes and polish the old CSS skills some more. Enjoy!

Using Django Pingback

hello@roytang.net (Roy Tang) — Sun, 04 Jan 2009 07:22:20 +0000

I actually had some trouble using django-pingback on my custom blog engine; the django-pingback documentation is mostly fine, but there were some caveats that I had to discover myself through a bit of debugging:

The URL specified for the XML-RPC endpoint in the HTML head needs to be a full absolute url including domain, i.e. http://roytang.net/xmlrpc/, which gave me trouble when I was trying to test using localhost pinging to an online server. I eventually just decided to set it up, deploy on webfaction and test it online before I redirected the domain name.
The documentation mentioned that the pingback handler needs to have the same arguments as the detail view for the post, so I wrote it as follows:
```
# exactly same arguments as 'details' view.
def pingback_blog_handler(catslug, year, month, slug, **kwargs):
    cat = Category.objects.get(slug=catslug)
    return Post.pub_objects.get(date__year=year, date__month=month, slug=slug, category=cat, published=True)
```
But it wasn’t exactly working for me, for some reason the function wasn’t being called. After some debugging I found that I needed to use named parameters for catslug, year, month, slug in the URLConf for the post permalink in order for this to work. I’m not sure if this was because I was doing it wrong or it’s some sort of undocumented requirement.

The pingback client was throwing a KeyError whenever a post I made would have anchor tags without an href attribute. Granted that there isn’t much point in having an anchor tag without the href, but the code shouldn’t just choke on it. I modified the following line in client.py:
```
  links = [a['href'] for a in soup.findAll('a') if is_external(a['href'], url)]
```
Adding the check for the href attribute:
```
  links = [a['href'] for a in soup.findAll('a') if a.has_key('href') and is_external(a['href'], url)]
```
Those are all the changes I made to get pingbacks working. I hope they’re still working now, as I haven’t received any pingbacks since the django version of this blog went live >.

Jannis Leidel - An autocomplete widget for django-tagging form fields

hello@roytang.net (Roy Tang) — Wed, 31 Dec 2008 06:40:40 +0000

Shared via delicious: Jannis Leidel - An autocomplete widget for django-tagging form fields

Colophon 2008

hello@roytang.net (Roy Tang) — Sat, 27 Dec 2008 02:08:22 +0000

Frontend: All page templates are valid (X)HTML. However, I choose not to claim valid XHTML (and no doc type declaration) since I can’t guarantee that blog posts I write are compliant! The site uses standard CSS and uses the Blueprint CSS Framework for the grid layout of the page. The site design is entirely original (if not simple and bland – I’m not very good with website design yet!). The site has minor usage of JQuery Javascript library in some parts.

Backend: The server side uses Python and Django behind mod_python on Apache, with a MySQL database. The following Django apps and Python libraries are used:

custom made blog application by myself
built-in (contrib) apps: admin, comments, sitemaps, flatpages
django-tagging
django-xmlrpc
comment_utils for advanced comment moderation features
template_utils
django-pingback
Beautiful Soup for XML/HTML parsing
Pygments for syntax highlighting. See the Pygments Demo!
Feedparser
Markdown2 so that I can post using Markdown.

As is typical for hobbyist software development, I started working on this site with only a minimal set of features considered, but as I kept working I kept wanting to add more stuff. Some notable custom-made features:

XMLRPC server, supporting Metaweblog API and Pingbacks. I actually only added the metaweblog support so I could post from Flock
Autocard support (same as my previous Autocard plugin for WP) and syntax highlighting of code.
Automatic extraction of twitter and delicious entries into the sidebar
Gravatar support for comments

All entries from the previous “roytang / weblog”, “Roy on Magic” and “Roy on Django” have been imported using a custom data conversion script, but nothing is perfect so some of the older posts may be badly formatted. (Comment on those posts!)

Hosting: Provided by Webfaction.

Fixing up the comments

hello@roytang.net (Roy Tang) — Mon, 01 Dec 2008 04:58:00 +0000

I played around with the built-in comments app today, trying to clean it up.

Some findings:

Some of the moderation views, such as flagging a post or deleting a post, accept a next parameter that determines where the view will redirect to after the operation. However, the way the url’s are set up, there’s no easy way to pass this parameter normally, even through query strings. This lovely bug is documented in http://code.djangoproject.com/ticket/8968. I wasn’t confident enough to try patching it, so as a temporary workaround, I overrode the flag and delete templates and placed the following inside the form:

<input type="hidden" name="next" value="{{ comment.get\_content\_object_url }}" />

Edit: Wow that was stupid. Blogger didn’t want to render the HTML code above!

I also replaced the URL in the “cancel” links for both pages. For some reason, {{ comment.permalink }} wasn’t giving me anything useful.

Took me a bit of work to figure out how the moderation part works. Apparently I have to hook on some of the signals to add my moderation logic. I’m not sure yet how to handle this, or whether I should try customizing the comment form with something like a captcha to prevent comment spam.
I have a pending problem now that submitting comments from the preview or error page does not redirect to the correct place. (i.e. the post page). I tried adding the next parameter to the form, but apparently comment.get_content_object_url returns an empty string here because the comment has not yet been saved to the database. I think the content object should be provided in the context for the preview page. This would be handled by http://code.djangoproject.com/ticket/9268, except apparently no one’s paying attention to it yet.
Some random features I added: Gravatar support (the Gravatar for the email address used to post the comment will be shown), and Markdown support in the comment body. Easy stuff.
Cleaned up an refactored the templates that render the comments, separating them from the parts that render the blog posts. This makes it easier if I want to have comments for other non-Blog post objects later.

Non-comment related:

Added simple search!
The script to import from WordPress has some problems. Since I’m storing the post body in markdown format, I use html2text to convert the HTML body from old posts into markdown, but nice stuff like image alignments and tables are lost. I think I have to make do with editing straight HTML for old posts imported from WordPress, and just clean them up later as needed.

My code is rather messy now (especially from the HTML side). I should clean it up sometime soon.

Fictitious Nonsense » Overriding a Widget in the Django Admin Site

hello@roytang.net (Roy Tang) — Tue, 25 Nov 2008 16:00:16 +0000

Shared via delicious: Fictitious Nonsense » Overriding a Widget in the Django Admin Site

Django Update -- WordPress Import and More

hello@roytang.net (Roy Tang) — Sun, 23 Nov 2008 00:04:00 +0000

I’ve been busy at work so fell a bit behind with Django. Last night I worked on a WordPress importer, so that I could migrate posts from my current blog(s) into the Django-powered blog that I’m coding. I’m using BeautifulSoup to parse the WordPress export file and insert them as Django objects.

Since I was running the script repeatedly, I had to figure out how to easily run it from the command line, without having to run it from inside manage.py shell. After some searching, I found that what I needed to do was set an environment variable DJANGO_SETTINGS_FILE to point to my settings file. After that, the importer script could be run repeatedly.

Also made a few tweaks to my dev copy:

Modified URL mapping to follow the permalinks of the old blog (so that people linking to me don’t suddenly have broken links)
Added a tag cloud (Still using django-tagging)
Added markdown support to the admin maintenance screen. I used to think I would need to code my own form for posting, but I think I can live with just using the admin-provided one (initially at least)

I plan to migrate my main blog over once I take my Christmas break (hopefully 2nd week of December)

Stuff to do:

Use clean, non-hardcoded urls (url template tag and reverse())
Customize the admin screen for posting
Clean up the layout and templates

django-tagging

hello@roytang.net (Roy Tang) — Sun, 26 Oct 2008 04:22:00 +0000

I wanted to add some basic tagging to my blog app so I tried out django-tagging. Unfortunately, the featured downloads on the Google Code site are quite out-of-date and would not work with Django 1.0, so I did a subversion checkout instead. If you’re getting an error like “ImportError: cannot import name parse_lookup”, then you need to get the source code from SVN.

Adding the tagging to the blog was pretty easy:

Add the tagging app to settings.py
Add a tagging.fields.TagField to the Post model
Add a “tags” text field to the post form used.
Modify templates to display the tags.
I used something like “/tag/” url mapping to get all posts associated with a tag. Then you just need to write a wrapper around the object_list generic view:

from tagging.models import Tag, TaggedItem

from django.views.generic.list_detail import object_list

def posts_by_tag(request, tag):

o_tag = Tag.objects.get(name=tag)

queryset = TaggedItem.objects.get_by_model(Post, o_tag)

return object_list(request, queryset)

This view will use the same template used to list out posts normally.

I left it for a while

hello@roytang.net (Roy Tang) — Fri, 24 Oct 2008 06:54:00 +0000

… and now it works!

http://www.randomencounters.bells-n-whistles.net/blog/

I think the apache service needed to restart mod_python or something… (I don’t have SSH access so… )

Deployment Problems

hello@roytang.net (Roy Tang) — Tue, 21 Oct 2008 06:52:00 +0000

So I got a basic blog app up and running.

Posting, paged archives, etc.

Comments implemented using the django.contrib.comments. No problems here, the only caveat being most of the current documentation found by Google searches refer to the pre-1.0 version. Need to peruse the official docs for 1.0 stuff.

RSS feeds implemented using django.contrib.syndication, this one seems fine.

I tested it and it’s running fine on localhost. I also have a free django hosting account at http://bells-n-whistles.net, so I try to upload it there. However, when I access the website there (http://www.randomencounters.bells-n-whistles.net/blog/), I get the following error:

'comments' is not a valid tag library: Could not load template library from django.templatetags.comments, No module named comments

I think the settings.py is not being reloaded - I tried to set Debug = False and I’m still getting the stacktraces.

Bah, I’ll figure it out tomorrow.

Playing with Generic Views and URLs

hello@roytang.net (Roy Tang) — Mon, 20 Oct 2008 13:44:00 +0000

“when redirecting, how can I make the redirect URL decoupled from the urls.py of the parent app?”

-> It turns out that HttpResponseRedirect supports relative paths, so this was fine.

return HttpResponseRedirect("../" + str(post.id) + "/")

I got the basic posting structure up.

/post/new/ -> To make new posts

/post// -> To view a single post

/post/all/ -> To view all posts

I should probably start thinking of a better url scheme. Ideally, I’d want the @login_required views to be indicated as such in the urls. Something like “/admin/post/” for new posts “/admin/manage/” for a screen to manage posts to differentiate it from a screen just to list them out.

Next I think I’ll try to CSS-ify the blog; I’ll probably just reuse stuff from one of the blogger templates as I’m still not very good with the HTML/CSS.

I’ll also look into using some of the date-based generic views to get an archive view. I’ll need more data though. Will consider importing from the WordPress blog.

Starting out

hello@roytang.net (Roy Tang) — Sun, 19 Oct 2008 07:33:00 +0000

The quintessential app to learn from is of course a blog.

Started using a simple Post model. Added the new post form and view. Can now successfully insert posts into the DB.

– create the detail page that will show the post after saving

Figure out:

– when redirecting, how can I make the redirect URL decoupled from the urls.py of the parent app?

i.e. if the parent app has the following mapping:

‘^blog/’ -> pass to blog.urls.urlpatterns

the blog app has mappings for

‘^post/new/’ -> new post

‘^post/([A-Za-z-])/’ -> post detail

inside the view, I want to redirect to “post/”, but with respect to the app, it should be “blog/post/”

I should subscribe to one of the Django mailing lists to ask stuff like this.