The Best (and Worst) of Django

Editor's Notes

• #2: \n
• #3: lead developer, django\nconsultant, revsys, where I help companies scale web apps.\n
• #4: lead developer, django\nconsultant, revsys, where I help companies scale web apps.\n
• #5: This is how many lines of Django code I’ve read over the last 2 years or so.\n\nBy comparison: Django: 120kloc, Python stdlib: 300kloc, pypy 550kloc (or 1m incl stdlib)\n\nMuch of this code has been good; some of it has been great; some very very bad. My goal today is to share with you some of what I’ve learned - to suggest certain successful patterns, and warn you away from other noxious ones.\n
• #6: The theme that’s emerged here: y’all are far to damned clever.\n\n\n
• #7: One of Simon’s favorite quips. Also known as “KISS” and a bunch of other things, but what I like about this particular formulation is the word “possibly”. It’s not “do the simple thing that you know will work”; it’s “do something that *could* work”. If it doesn’t, *then* you can get more complex.\n\nDjango does not lend itself very well to overengineering. Keep it small, keep it simple, keep it clean.\n
• #8: Pattern 1: small is beautiful.\n
• #9: This is easiest to explain by starting with the anti-pattern, so let’s start there. I see, from time to time, Django apps that are just one huge ball of wax - a monolith. Instead of multiple apps, there’s just a single app with all the code in it.\n\nTypically these don’t grow well at all. Because there’s no real thought to factoring, the web gets more and more tangled, and you end up needing to fall back on site-wide things like middleware and context processors to get bits into the right place. At the extreme, people end up inventing their own plugin-like concepts. Remember: Django’s *got that* -- INSTALLED_APPS -- so if you’re reinventing that system you’re cutting against the grain.\n
• #10: \n
• #11: I’ll give one example. Client wanted us to review his code, he told us that it was 20kloc. Sounds fairly small and easy to review, until …\n
• #12: Wait, where’s the rest of the site?\n
• #13: Groan.\n
• #14: I often like to graph the models to get a handle on how the data fits together. The red box is just this one app; something on the order of 200 or so models in there.\n\nThere was stuff in there like “Customer” and “Customer2” — with such a big ball of wax, people were afraid to make changes, not knowing what the ramifications would be.\n
• #15: Large modules are generally considered “code smell.” All-in-one design prevents reuse. You’ll need to introduce random, unrelated code. Monoliths are hellish to test.\n\nMany great Django apps are very small. Even a lot of “simple” Django sites commonly have a dozen or more applications in INSTALLED_APPS. If you’ve got a complex site and a short application list, something’s probably wrong.\n
• #16: \n
• #17: Here’s what (the data parts of) a well-designed site look like. This is a selection of 5 apps from a larger project, but note how easily I can pull them out -- there’s not any tendrils out of here[*]\n\nI want to particularly point out the faqcreator app. The system here is a ticketing and knowledge base system, and one requirement was the ability to quickly convert a ticket to an FAQ entry. I could put this logic in the FAQ system., but then it would create a circular dependency between tickets and faqs, which makes no sense and prevents reuse. So I introduced a faqcreator that depends on both.\n\nRemember: dependencies are just fine, but they should be logical, and they should be linear.\n\n[*] actually there’s an FK to user but I typically omit that -- I consider Django apps fair game for deps.\n
• #18: \n
• #19: \n
• #20: \n
• #21: These make up the bulk of what you’ll write as a Django dev, usually.\n
• #22: \n
• #23: \n
• #24: \n
• #25: \n
• #26: \n
• #27: \n
• #28: Pattern two: model your damned data. aka Django is not a CMS.\n
• #29: A typical “CMS”. This makes me sad sad sad.\n
• #30: ca 2003 Adrian/Simon working at the LJW; content managed in a (reasonably good, custom built) CMS. But you just can’t pull information out of CMSes\n\nexample: gas prices. every morning, a reporter would call a few gas staions in lawrence and ask about gas prices. Then he’d open the cms and type “Shell: $1.82. Quick Stop: $1.88”, etc. This is terrible: no way to get historical data, no way to graph changes over time, no way to build a “closest gas to me”, etc. This is turning “data” into “text”, and readers predictably found these “stories” useless. Eventually the reporter had to stop. Imagine if instead we’d collected that data as data...\n\nDjango started as a reaction against CMSes. This isn’t to say that CMSes are bad. But if you want one, Django isn’t it.\n
• #31: Instead, Django really really wants you to model your data. There’s no such thing as content: there’s gas stations and price measurements; public officials and speaking appearances; photographs and infographics.\n\nAnother example: look at lanyrd. Plenty of people have made conference web sites, but Simon followed (because helped invent!) the Django philosophy, and the site’s incredibly useful because if its powerful respect for the data.\n
• #32: One particularly pernicious form the “content” issue takes is what I all the “everything-is-a” antipattern.\n
• #33: Starts here.\n
• #34: So you do something like this.\n\nRemember, though: databases are *not* OO, so what this actually does is to create a foreign key from (Animal, Vegetable, Mineral) back to BaseObject.\n
• #35: So consider what happens when you do queries.\n\nNow there’s a “hidden join”. Now, the “JOINs are evil” thing is a bit of a myth, at least if you’re using a grown-up database, but it’s clear that the first query is a lot cheaper than the second.ways\n\nThis gets even worse if you’ve got multiple levels of interitance.\n
• #36: So consider what happens when you do queries.\n\nNow there’s a “hidden join”. Now, the “JOINs are evil” thing is a bit of a myth, at least if you’re using a grown-up database, but it’s clear that the first query is a lot cheaper than the second.ways\n\nThis gets even worse if you’ve got multiple levels of interitance.\n
• #37: So consider what happens when you do queries.\n\nNow there’s a “hidden join”. Now, the “JOINs are evil” thing is a bit of a myth, at least if you’re using a grown-up database, but it’s clear that the first query is a lot cheaper than the second.ways\n\nThis gets even worse if you’ve got multiple levels of interitance.\n
• #38: So consider what happens when you do queries.\n\nNow there’s a “hidden join”. Now, the “JOINs are evil” thing is a bit of a myth, at least if you’re using a grown-up database, but it’s clear that the first query is a lot cheaper than the second.ways\n\nThis gets even worse if you’ve got multiple levels of interitance.\n
• #39: So consider what happens when you do queries.\n\nNow there’s a “hidden join”. Now, the “JOINs are evil” thing is a bit of a myth, at least if you’re using a grown-up database, but it’s clear that the first query is a lot cheaper than the second.ways\n\nThis gets even worse if you’ve got multiple levels of interitance.\n
• #40: So consider what happens when you do queries.\n\nNow there’s a “hidden join”. Now, the “JOINs are evil” thing is a bit of a myth, at least if you’re using a grown-up database, but it’s clear that the first query is a lot cheaper than the second.ways\n\nThis gets even worse if you’ve got multiple levels of interitance.\n
• #41: So consider what happens when you do queries.\n\nNow there’s a “hidden join”. Now, the “JOINs are evil” thing is a bit of a myth, at least if you’re using a grown-up database, but it’s clear that the first query is a lot cheaper than the second.ways\n\nThis gets even worse if you’ve got multiple levels of interitance.\n
• #42: So consider what happens when you do queries.\n\nNow there’s a “hidden join”. Now, the “JOINs are evil” thing is a bit of a myth, at least if you’re using a grown-up database, but it’s clear that the first query is a lot cheaper than the second.ways\n\nThis gets even worse if you’ve got multiple levels of interitance.\n
• #43: So consider what happens when you do queries.\n\nNow there’s a “hidden join”. Now, the “JOINs are evil” thing is a bit of a myth, at least if you’re using a grown-up database, but it’s clear that the first query is a lot cheaper than the second.ways\n\nThis gets even worse if you’ve got multiple levels of interitance.\n
• #44: So consider what happens when you do queries.\n\nNow there’s a “hidden join”. Now, the “JOINs are evil” thing is a bit of a myth, at least if you’re using a grown-up database, but it’s clear that the first query is a lot cheaper than the second.ways\n\nThis gets even worse if you’ve got multiple levels of interitance.\n
• #45: So consider what happens when you do queries.\n\nNow there’s a “hidden join”. Now, the “JOINs are evil” thing is a bit of a myth, at least if you’re using a grown-up database, but it’s clear that the first query is a lot cheaper than the second.ways\n\nThis gets even worse if you’ve got multiple levels of interitance.\n
• #46: So consider what happens when you do queries.\n\nNow there’s a “hidden join”. Now, the “JOINs are evil” thing is a bit of a myth, at least if you’re using a grown-up database, but it’s clear that the first query is a lot cheaper than the second.ways\n\nThis gets even worse if you’ve got multiple levels of interitance.\n
• #47: But here’s where it gets really bad -- people *want* base classes to automatically “narrow” themselves to the appropriate subclass, but this isn’t something Django’s ORM can do. Indeed, ORMs that *can* do this (Hibernate, SQLA probably) can’t do it efficiently.\n
• #48: But here’s where it gets really bad -- people *want* base classes to automatically “narrow” themselves to the appropriate subclass, but this isn’t something Django’s ORM can do. Indeed, ORMs that *can* do this (Hibernate, SQLA probably) can’t do it efficiently.\n
• #49: But here’s where it gets really bad -- people *want* base classes to automatically “narrow” themselves to the appropriate subclass, but this isn’t something Django’s ORM can do. Indeed, ORMs that *can* do this (Hibernate, SQLA probably) can’t do it efficiently.\n
• #50: But here’s where it gets really bad -- people *want* base classes to automatically “narrow” themselves to the appropriate subclass, but this isn’t something Django’s ORM can do. Indeed, ORMs that *can* do this (Hibernate, SQLA probably) can’t do it efficiently.\n
• #51: But here’s where it gets really bad -- people *want* base classes to automatically “narrow” themselves to the appropriate subclass, but this isn’t something Django’s ORM can do. Indeed, ORMs that *can* do this (Hibernate, SQLA probably) can’t do it efficiently.\n
• #52: But here’s where it gets really bad -- people *want* base classes to automatically “narrow” themselves to the appropriate subclass, but this isn’t something Django’s ORM can do. Indeed, ORMs that *can* do this (Hibernate, SQLA probably) can’t do it efficiently.\n
• #53: But here’s where it gets really bad -- people *want* base classes to automatically “narrow” themselves to the appropriate subclass, but this isn’t something Django’s ORM can do. Indeed, ORMs that *can* do this (Hibernate, SQLA probably) can’t do it efficiently.\n
• #54: But here’s where it gets really bad -- people *want* base classes to automatically “narrow” themselves to the appropriate subclass, but this isn’t something Django’s ORM can do. Indeed, ORMs that *can* do this (Hibernate, SQLA probably) can’t do it efficiently.\n
• #55: But here’s where it gets really bad -- people *want* base classes to automatically “narrow” themselves to the appropriate subclass, but this isn’t something Django’s ORM can do. Indeed, ORMs that *can* do this (Hibernate, SQLA probably) can’t do it efficiently.\n
• #56: But here’s where it gets really bad -- people *want* base classes to automatically “narrow” themselves to the appropriate subclass, but this isn’t something Django’s ORM can do. Indeed, ORMs that *can* do this (Hibernate, SQLA probably) can’t do it efficiently.\n
• #57: So maybe you try something like this to scope things down. Now you’re doing O(N) queries.\n
• #58: \n
• #59: One of the simplest performance metrics is the number of RTs between your web servers and the db server. As a rule of thumb, you can kinda assume that all queries take roughly the same time; it’s the network round-trip that kills you. Remember: these queries are done synchronously, so even if the db can do a query in only 1ms this page would still take 1.8seconds to render!\n
• #60: At this point I hear you saying “but this *is* a legitimate business need!”\n\nSo give everything a creation date. Do 3 queries (Animal, Veg, Min) and then combine the lists.\n
• #61: At this point I hear you saying “but this *is* a legitimate business need!”\n\nSo give everything a creation date. Do 3 queries (Animal, Veg, Min) and then combine the lists.\n
• #62: One of Simon’s favorite quips. Also known as “KISS” and a bunch of other things, but what I like about this particular formulation is the word “possibly”. It’s not “do the simple thing that you know will work”; it’s “do something that *could* work”. If it doesn’t, *then* you can get more complex.\n\nDjango does not lend itself very well to overengineering. Keep it small, keep it simple, keep it clean.\n
• #63: If you need to get fancy -- you’ve got 60 kinds of content you need to interleave -- then it’s time to denormalize.\n
• #64: Pattern/anti-pattern 3: not understanding PYTHONPATH.\n\nI see a *huge* amount of confusion about imports and the python path.\n
• #65: Brief explanation of what pythonpath is and how imports work, kinda.\n
• #66: Brief explanation of what pythonpath is and how imports work, kinda.\n
• #67: Brief explanation of what pythonpath is and how imports work, kinda.\n
• #68: Brief explanation of what pythonpath is and how imports work, kinda.\n
• #69: Brief explanation of what pythonpath is and how imports work, kinda.\n
• #70: Brief explanation of what pythonpath is and how imports work, kinda.\n
• #71: Brief explanation of what pythonpath is and how imports work, kinda.\n
• #72: Brief explanation of what pythonpath is and how imports work, kinda.\n
• #73: Brief explanation of what pythonpath is and how imports work, kinda.\n
• #74: Brief explanation of what pythonpath is and how imports work, kinda.\n
• #75: Brief explanation of what pythonpath is and how imports work, kinda.\n
• #76: Brief explanation of what pythonpath is and how imports work, kinda.\n
• #77: Brief explanation of what pythonpath is and how imports work, kinda.\n
• #78: Brief explanation of what pythonpath is and how imports work, kinda.\n
• #79: Brief explanation of what pythonpath is and how imports work, kinda.\n
• #80: Brief explanation of what pythonpath is and how imports work, kinda.\n
• #81: Brief explanation of what pythonpath is and how imports work, kinda.\n
• #82: Brief explanation of what pythonpath is and how imports work, kinda.\n
• #83: Consider, a basic project - very simple, just one app.\n
• #84: sadly, manage.py lets you access your models under two separate paths, and this causes no end of confusion.\n
• #85: sadly, manage.py lets you access your models under two separate paths, and this causes no end of confusion.\n
• #86: sadly, manage.py lets you access your models under two separate paths, and this causes no end of confusion.\n
• #87: sadly, manage.py lets you access your models under two separate paths, and this causes no end of confusion.\n
• #88: sadly, manage.py lets you access your models under two separate paths, and this causes no end of confusion.\n
• #89: sadly, manage.py lets you access your models under two separate paths, and this causes no end of confusion.\n
• #90: sadly, manage.py lets you access your models under two separate paths, and this causes no end of confusion.\n
• #91: sadly, manage.py lets you access your models under two separate paths, and this causes no end of confusion.\n
• #92: sadly, manage.py lets you access your models under two separate paths, and this causes no end of confusion.\n
• #93: sadly, manage.py lets you access your models under two separate paths, and this causes no end of confusion.\n
• #94: sadly, manage.py lets you access your models under two separate paths, and this causes no end of confusion.\n
• #95: sadly, manage.py lets you access your models under two separate paths, and this causes no end of confusion.\n
• #96: sadly, manage.py lets you access your models under two separate paths, and this causes no end of confusion.\n
• #97: sadly, manage.py lets you access your models under two separate paths, and this causes no end of confusion.\n
• #98: How you know you’re getting bitten by this.\n
• #99: The problem: manage.py is broken. It adds “.” and “..” to PYTHONPATH. It does this to make the tutorial easier, and to prevent having to explain PYTHONPATH to new django users, but this was a mistake.\n
• #100: Always use app-imports: this makes it easy to spin apps out of the project if you need to.\n\nTry to use relative imports if you’re on 2.5 or higher -- really really makes this easier.\n\nDelete manage.py.\n
• #101: Wait, if I delete manage.py how do I interact with my app?\n\nUse django-admin.py.\n\nThis doesn’t do the pythonpath malarky -- you have to be explicit!\n\nThis --settings thing is important for a semi-related reason; make a note for later.\n
• #102: Wait, if I delete manage.py how do I interact with my app?\n\nUse django-admin.py.\n\nThis doesn’t do the pythonpath malarky -- you have to be explicit!\n\nThis --settings thing is important for a semi-related reason; make a note for later.\n
• #103: Wait, if I delete manage.py how do I interact with my app?\n\nUse django-admin.py.\n\nThis doesn’t do the pythonpath malarky -- you have to be explicit!\n\nThis --settings thing is important for a semi-related reason; make a note for later.\n
• #104: Wait, if I delete manage.py how do I interact with my app?\n\nUse django-admin.py.\n\nThis doesn’t do the pythonpath malarky -- you have to be explicit!\n\nThis --settings thing is important for a semi-related reason; make a note for later.\n
• #105: Wait, if I delete manage.py how do I interact with my app?\n\nUse django-admin.py.\n\nThis doesn’t do the pythonpath malarky -- you have to be explicit!\n\nThis --settings thing is important for a semi-related reason; make a note for later.\n
• #106: Wait, if I delete manage.py how do I interact with my app?\n\nUse django-admin.py.\n\nThis doesn’t do the pythonpath malarky -- you have to be explicit!\n\nThis --settings thing is important for a semi-related reason; make a note for later.\n
• #107: Wait, if I delete manage.py how do I interact with my app?\n\nUse django-admin.py.\n\nThis doesn’t do the pythonpath malarky -- you have to be explicit!\n\nThis --settings thing is important for a semi-related reason; make a note for later.\n
• #108: Wait, if I delete manage.py how do I interact with my app?\n\nUse django-admin.py.\n\nThis doesn’t do the pythonpath malarky -- you have to be explicit!\n\nThis --settings thing is important for a semi-related reason; make a note for later.\n
• #109: Wait, if I delete manage.py how do I interact with my app?\n\nUse django-admin.py.\n\nThis doesn’t do the pythonpath malarky -- you have to be explicit!\n\nThis --settings thing is important for a semi-related reason; make a note for later.\n
• #110: You can automate most of this with virtualenv, and it’s kinda great.\n
• #111: Lastly, I’ll talk a bit about settings. I see people doing the craziest stuff with settings...\n\n(and urls - urls are a special form of setting)\n
• #112: Not too bad... but it *does* defeat the whole purpose of INSTALLED_APPS. The point is to be explicit about what’s loaded. Django’s trying to make sure you’re only using what’s needed. If you circumvent this, things start randomly breaking when you make directories.\n\nI even saw this become a security hole: they allowed uploads into a media/ directory, and I uploaded a “models.py”. Oops.\n
• #113: Then it gets more crazy. People do stuff like this to dynamically load urls because apparently adding a line to urls.py is too much work...\n
• #114: ... and even scary stuff like there.\n\nThere’s actually a couple of bugs here; they’re left as exercises to the reader.\n
• #115: \n
• #116: There’s no good reason not to just make settings simple data structures.\n
• #117: There’s no good reason not to just make settings simple data structures.\n
• #118: The problem: differences between development (mac laptop), testing (ci), staging, production.\n\nI see lots of unneeded complexity here, too. What’s the most Django-esque, least magical solution?\n
• #119: This is a very common pattern, and it needs to die.\n
• #120: Here’s how you explain the pattern. Anyone see what’s wrong?\n
• #121: There are other reasons why this pattern sucks -- e.g. since you’re importing overrides into the “base”, you can’t do stuff like “INSTALLED_APPS.append()”. But EVERYTHING GOES IN SOURCE CONTROL!\n
• #122: \n
• #123: \n
• #124: \n
• #125: \n
• #126: \n
• #127: One of Simon’s favorite quips. Also known as “KISS” and a bunch of other things, but what I like about this particular formulation is the word “possibly”. It’s not “do the simple thing that you know will work”; it’s “do something that *could* work”. If it doesn’t, *then* you can get more complex.\n\nDjango does not lend itself very well to overengineering. Keep it small, keep it simple, keep it clean.\n
• #128: \n