Inércia Sensorial

28 de January de 2014

Remove ._ files from Mac OS X

Filed under: Perils of Software Development — inerte @ 15:43

Run this command on the terminal to remove every ._ file created by Mac OS X on the current path and its descendants:

find . -name '._*' -delete

23 de September de 2013

Replace tabs by space on git history

Filed under: Perils of Software Development — Tags: — inerte @ 21:04

So, I prefer spaces, and when I press tab, it should instead write 4 spaces. I forgot to set this option on Sublime Text and did some commits with tabs. The following git command will rewrite the repo history and replace tabs by 4 spaces.

git filter-branch --tree-filter 'git diff-tree --name-only --diff-filter=AM -r --no-commit-id $GIT_COMMIT | find . -exec perl -pi -e "s/\t/    /g" {} \;' HEAD

12 de November de 2009

memcached monitoring and statistics

Filed under: Perils of Software Development — Tags: — inerte @ 15:54

I wanted to know, overtime, how many cache hits and misses my instance was getting. Searched a lot and finally found this perl script:

http://code.google.com/p/memcache-top/

Usage is very simple. Save the file, give it execution permissions, then just run:

./memcache-top.pl –commands –cumulative

Here’s an output example:

memcache-top.pl output example

Seems like I’m doing fine, caching the right parts! Hit over 80% and only half of my allocated memcached memory being used… (that’s because I had recently restarted it, so usually it keeps filled).

23 de August de 2009

Apache + mod_wsgi + Django + lighttpd

Filed under: Perils of Software Development — Tags: , , , , — inerte @ 18:45

I’ve written how to configure Apache, mod_python and Django and how to put lighttpd behind Apache.

Recently I decided to host my most visited website on a different VPS provider[1], and started a quest to update my knowledge about Django deployment. I did things differently this time, using mod_wsgi (the recommended way of deploying Apache and Django), and configuring Apache behind lighttpd for dynamic content (in other words, lighttpd will serve static media).

I did everything below in the last couple days, and did not wrote things as I was doing it, because it involved a lot of experimentation (trial and error) for me. As such, I am writing this article based on memory and checking my config files. If you encounter any problems, please leave a comment and I will clarify any omissions.

Here’s how to do it:

Install the usual suspects

I choose Ubuntu as my Linux distro, and installation of anything is a breeze on it. sudo apt-get install package-name. This part is well covered around the web, so I won’t comment in details how it’s done. Sufficient to say, some of the packages I’ve installed were apache2, libapache2-mod-wsgi, and lighttpd.

Configure Apache and mod_wsgi to load your project

Since lighttpd will act as the primary server for my domain, I decided to move Apache to port 81:

sudo vi /etc/apache2/ports.conf

Overwrite existing ip:port lines with these:

NameVirtualHost 127.0.0.1:81
Listen 81

Where you put your own Python modules on newer Ubuntu installations has changed to /usr/local/lib/python2.6/dist-packages/. Therefore, I’ve uploaded Django, my project and other necessary modules (which weren’t installed by apt-get) to this directory, leaving me with the following structure:

/usr/local/lib/python2.6/dist-packages/django/
/usr/local/lib/python2.6/dist-packages/my_project_name/

The mod_wsgi documentation has an excellent article on Django integration, but it’s fairly length. You should read it anyway, since there are lots of options that you might want to use. Here’s a cheatsheet:

Create a document root for your domain name:

sudo mkdir /var/www/example.com

Create the file which will be loaded by mod_wsgi with your project configuration:

sudo mkdir /usr/local/lib/python2.6/dist-packages/project_name/apache/
sudo vi /usr/local/lib/python2.6/dist-packages/project_name/apache/django.wsgi

With these contents:

import sys
import os

sys.path.append(‘/usr/local/lib/python2.6/dist-packages’)
os.environ[‘DJANGO_SETTINGS_MODULE’] = ‘project_name.settings’

import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

Create a domain configuration file for Apache:

sudo vi /etc/apache2/sites-available/example.com

With these contents:

<VirtualHost 127.0.0.1:81>
ServerName example.com
ServerAdmin [email protected]

DocumentRoot /var/www/example.com

Alias /media/ /usr/local/lib/python2.6/dist-packages/django/contrib/admin/media/
<Directory /usr/local/lib/python2.6/dist-packages/django/contrib/admin/media>
Options -Indexes
Order deny,allow
Allow from all
</Directory>

Alias /project_media_dir/ /usr/local/lib/python2.6/dist-packages/project_name/templates/project_media_dir/
<Directory /usr/local/lib/python2.6/dist-packages/project_name/templates/project_media_dir>
Options -Indexes
Order deny,allow
Allow from all
</Directory>

WSGIScriptAlias / /usr/local/lib/python2.6/dist-packages/project_name/apache/django.wsgi
WSGIDaemonProcess example.com
WSGIProcessGroup example.com

<Directory /usr/local/lib/python2.6/dist-packages/project_name/apache>
Order deny,allow
Allow from all
</Directory>
</VirtualHost>

Activate it:

cd /etc/apache2/sites-enabled/
sudo ln -s ../sites-available/example.com

Configure lighttpd to proxy non-static media requests to Apache

I used MySQL Performance Blog’s “Lighttpd as reverse proxy” article as a basis for my own configuration. Therefore, we’ll have an url http://example.com/server-status, which will require authentication, enabling us to see Apache’s server status.

Create a directory for error logs:

sudo mkdir /var/log/lighttpd/example.com

Create a domain file configuration for lighttpd:

sudo vi /etc/lighttpd/conf-available/20-example.com.conf

With these contents:

server.modules += ( “mod_auth”,
“mod_status”,
“mod_proxy”,
)

$HTTP[“host”] =~ “(^|\.)example\.com$” {
$HTTP[“url”] !~ “\.(js|css|gif|jpg|png|ico|txt|swf|html|htm)$” {
proxy.server = ( “” => (
( “host” => “127.0.0.1”, “port” => 81 )
))
}

server.document-root = “/var/www/example.com/”
server.errorlog = “/var/log/lighttpd/example.com/error.log”
dir-listing.activate = “disable”

auth.backend = “htpasswd”
auth.backend.htpasswd.userfile = “/var/www/.htpasswd”
auth.require = ( “/server-status” => (
“method” => “basic”,
“realm” => “status”,
“require” => “valid-user”
)
)
}

There are lines worthy of mention in the configuration above:

$HTTP[“host”] =~ “(^|\.)example\.com$” {

This will wrap the directives inside to only apply for example.com requests.

$HTTP[“url”] !~ “\.(js|css|gif|jpg|png|ico|txt|swf|html|htm)$” {
proxy.server = ( “” => (
( “host” => “127.0.0.1”, “port” => 81 )
))
}

These will send any requests for documents not ending in the specified extensions to ip 127.0.0.1, port 81, where Apache lives. Essentially, everything which is static content (or more accurately, specified by the | separated regular expression), will be served by lighttpd.

cd /etc/lighttpd/conf-enabled/
sudo ln -s ../conf-available/example.com.conf

Tell the filesystem where you project and Django’s admin static content are located:

sudo ln -s /usr/local/lib/python2.6/dist-packages/django/contrib/admin/media/ /var/www/example.com/media
sudo ln -s /usr/local/lib/python2.6/dist-packages/project_name/templates/project_media_dir/ /var/www/example.com/project_media_dir/

Finally, restart everything so the newest configuration can be applied

sudo /etc/init.d/apache2 restart
sudo /etc/init.d/lighttpd restart

So, what actually happens?

When a visitor goes to your website (example.com), the request will hit lighttpd first. If the document path does not end with a string in our list of static content extensions, the request will be proxied to Apache on port 81, otherwise lighttpd will serve itself.

And that’s it, if my memory is correct. Did I miss anything? Comment at will.

[1] Linode, if you’re curious. Mainly because bandwidth is cheaper. If you’re looking for a Linode referral, Linode discount code or Linode promotion code, sign up using this link to credit me as referral. Thanks 🙂

23 de January de 2009

Wikipedia with just links

Filed under: Perils of Software Development — Tags: — inerte @ 03:02

So I just read how Britannica is planning a new website version, mostly to “compete” with Wikipedia. Britannica’s CEO also talks about the relationship between Google’s results and Wikipedia’s pages.

Anyway, one criticism is valid, sometimes Wikipedia articles are not the best source of information about a topic. And this is being written by someone who searches daily on Google for terms followed by the word “wikipedia”, because Wikipedia usually have results that are good enough.

So here’s my idea, dump Wikipedia’s database of article titles, and let people submit links to these titles. And let users vote Reddit-Digg style. Find a way to deal with spammer’s bots and you’re ready to do.

For example, the Wikipedia article for “Design_pattern_(computer_science)” (which I have open in a tab right now) would be just a bunch of links for other sites that users can do whatever you want them to do so you can sort the links by relevancy.

As I’ve said on my previous post, I’ve been feeling lazy lately, otherwise I would go and implement this.

You might ask, why not just add the links that users would upvote to Wikipedia? Well, rare are the articles on Wikipedia that accept “full” links on its references section. Most links are put for citations, and these are just used to justify small text/phrases excerpts/facts, not whole webpages. And the “external links” section of Wikipedia articles aren’t sorted, and that’s the whole point of my idea :p

Now go and build it and remember to buy me a beer if you make a buck.

22 de January de 2009

How the fuck is trim in Python?

Filed under: Perils of Software Development — Tags: — inerte @ 20:54

HTFITIP is a website where you can see what a function from one language is called in another (or an implementation).

The idea came to me after the eleventh time I asked myself, how the fuck is trim in Python? It’s called strip by the way.

The website is very simple. On the front page, there’s this form:

How the fuck is _____’s _____ in _____?

The first space is the “source” programming language name, the second one we’ll put the function/method name, and the third one, the “target” programming language. The source language is optional.

A good source for the… source programming language is PHP’s (massive) functions list. First of all, PHP’s seems to have a function for everything, and there’s no namespace so it’s easier for us to build an index of it. Secondly, it’s a widely known language. Thirdly, it’s the first language of a lot of people, so it’s quite possible that a lot of people will want to know what’s the equivalent for PHP’s function X in a new language they’re learning, so if we have to start somewhere might as well be this way.

When the user selects the programming language from the source dropdown, an Ajax call automatically gets all the function names so when the user types a name on the second field, it auto-completes. Then, he selects the “target” language on the third field.

How the fuck is PHP’s trim in Python? (click submit)

If the relationship has already been stablished on the database, we present the answer to the user:

PHP’s trim in Python is called strip. (a link to Python’s manual page on strip would be nice)

If not, we can let the user make this relationship. If it’s not possible, say, Python doesn’t have an array_intersect_uassoc function on any module, we let the user type an implementation, wiki-style.

Web 2.0 baby, the users provide all the content, we make all the money.

And the website backend is simple. A list of programming language and its functions, and the relationship between functions/methods, and somekind of wiki stuff to let people submit implementations.

I’ve been feeling lazy lately so go ahead and implement this if you want. Should be an useful tool.

14 de March de 2008

Easy sharing of screenshots

Filed under: Perils of Software Development — inerte @ 22:57

I like having ideas. Specially when I am a little unhappy on the daily job. I start dreaming how cool would be to have my own company (I will someday). And then I start thinking about what my company will do. Or what software it’ll make. Most of the time it is games, but recently I came up with this one.

Press Print Screen and the image is automatically uploaded to a web server. A tray notification message (a bubble) appears with two options: Copy the URL to the clipboard or edit the image. Editing the image would allow for really simple things to be added, like text, boxes and arrows.

Basically it’s a quick way to show someone what’s on your screen. While developing on the daily job sometimes I want: Show the user/costumer something and ask for feedback. Are the colors nice? The position of a piece of information? The size of the text?

Maybe even allow capture of movies (of my computer usage) and recording of audio (whatever I say on the mic).

Just that. One click sharing of what I am seeing.

The software could benefit from some more features, like user authentication so it’s possible to manage your uploaded screenshots, or to set up permissions. But the basic needs to be there: An URL for my screenshot, the quickest way possible. Eliminate the need to open an image editor, paste the image, save, go to a website, select the image, wait for upload.

Maybe whenever the image is displayed on the web server, show a couple ads…

10 de June de 2007

Lighttpd with Apache

Filed under: Perils of Software Development — inerte @ 00:22

One of my websites is experiencing phenomenal growth and after optimizing the database queries and using memcached, I decided it was time to use another webserver to serve static media. Here’s how to do it.

I recommend that you take a look on a previous post that I wrote about how to configure a webserver to work with Django because I will be manipulating some files which I talked about there. If you’re unsure about the correct paths necessary for your server, use the other post as a guide. I installed Lighttpd and configured Apache on my Slicehost VPS, which is a company that I highly recommend. If you decide to sign-up with them, it would be awesome if you use my email ([email protected]) as a referral, and/or use this link which takes care of setting me as your referral.

We’re going to use Lighttpd on a different port than Apache to serve files from a specific directory based on the domain name, all transparent to the users browsing your website. No changes were made on my website code, server (Python) or client (HTML) side.

Install Lighttpd

My server runs Ubuntu so I just ran the following command to install Lighttpd:

$ apt-get install lighttpd

Edit Lighttpd’s configuration file:

$ vi /etc/lighttpd/lighttpd.conf

Uncomment the server.port line (mine was the 60th):

server.port               = 81

Enable the mod_evhost module. This will help us use a different directory depending on the domain name. Just uncomment the 19th line (inside server.modules).

Add the following somewhere, which says requests from yourdomain.com should look for files on /var/www/yourdomain.com/web/. I did on the 118th line, after the evhost.path-pattern examples.

$HTTP["host"] =~ "yourdomain\\.com" {
    evhost.path-pattern = "/var/www/yourdomain.com/web/"
}

Start Lighttpd:

$ /etc/init.d/lighttpd start

Point your browser to your server to see if Lighttpd is working:

http://your.ip.address:81/

You need to see something like this (click the image for a larger version):

Lighttpd default page

If you have directories under /var/www/, append its names to the url to see if lighttpd is correctly serving them (it should):

http://your.ip.address:81/yourdomain.com

Configure Apache

To transparently serve files using Lighttpd from Apache, you need to enable proxy on Apache:

$ ln -s /etc/apache2/mods-available/proxy.load /etc/apache2/mods-enabled/
$ ln -s /etc/apache2/mods-available/proxy.conf /etc/apache2/mods-enabled/
$ ln -s /etc/apache2/mods-available/proxy_connect.load /etc/apache2/mods-enabled/
$ ln -s /etc/apache2/mods-available/proxy_http.load /etc/apache2/mods-enabled/

Unsecured proxies are used by spammers so Apache comes guarded against bad configurations. You need to edit this file:

$ vi /etc/apache2/mods-available/proxy.conf

To say where the proxy will work and who has access to it. Replace the * with 127.0.0.1, and uncomment the Allow from line. Final result is this:

<Proxy 127.0.0.1>
        AddDefaultCharset off
        Order deny,allow
        Deny from all
        Allow from 127.0.0.1
</Proxy>

Configure the VirtualHost

$ vi /etc/apache2/sites-available/yourdomain.com

Add this somewhere inside <VirtualHost>:

ProxyRequests Off
ProxyPreserveHost On
ProxyPass /web http://127.0.0.1:81/
ProxyPassReverse / http://127.0.0.1:81/

You’re done with all configurations. Now reload the Apache’s modules and restart the server:

$ /etc/init.d/apache2 reload
$ apache2ctl restart

Everything should be working. If you get an error, try to undo some steps and after each change, reload Apache and restart Apache and Lighttpd to see when it broke. If you browse to http://your.ip.address, you should see all your content from /web/ being loaded correctly. How do you tell it’s being served by Lighttpd instead of Apache?

This should tell you Apache is serving the page:

curl -I http://your.ip.address/

While this should tell you Lighttpd is serving the file:

curl -I http://your.ip.address/web/some.file

If you have any doubts or have problems, please comment and I will try to help you.

17 de February de 2007

I am not amused

Filed under: Perils of Software Development — inerte @ 11:12

I am working on a codebase developed by an ex-employee, correcting code like:

1
2
3
4
5
6
7
8
9
10
$x = $y = 0;
while ($x < $interval + 1) {
    $looped_date = date('Y-m-d', mktime(0, 0, 0, date('m'), date('d') + $y, date('Y'));
    $formatted_looped_date = implode('/', array_reverse(explode('-', $looped_date)));
    $looped_date = implode('', array_reverse(explode('/', $formatted_looped_date)));
    $month = "$looped_date[4]$looped_date[5]";
    /* bunch of code */
    $x++;
    $y++;
}

Here are some snippets for the bunch of code above:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
if ($range == 'daily') {
    if ($type == 'credit') {
        $container->cells[] = '&nbsp;';
        $container->cells[] = $value;
        $container->cells[] = '-';
    } else {
        $container->cells[] = $value;
        $container->cells[] = '&nbsp;';
        $container->cells[] = '-';
    }
}
 
if ($range == 'monthly') {
    if ($type == 'credit') {
        $container->cells[] = '&nbsp;';
        $container->cells[] = $value;
        $container->cells[] = '-';
    } else {
        $container->cells[] = $value;
        $container->cells[] = '&nbsp;';
        $container->cells[] = '-';
    }
}
 
// and more 7 if($range)

There’re also functions defined but called just once. Use of Ajax to load a page with an argument, where a function decides 3 other pages to be loaded depending on the argument value. By the way, this value comes straight from the $_GET, with zero validation or treatment. The 3 pages are on a different directory, with no index.*.

One of the Ajax used is activated by an onchange event, and updates a div with an input field, which is raw html returned by one of the 3 files.

Mothership cat is not amused

There was also this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
function _get_account($id)
{
    $sql = "SELECT * FROM table WHERE id = $id";
    $res = mysql_query($sql);
    if ($id == '0') {
        return 'not available';
    }
    /* loop through $res. I don't know why, since the id column is a primary key, and return the result as an array*/
}
 
while ($x < $interval + 1) {
    // Inside the same loop above, after the many if($range)s
    $account = new ourOrmMapper($id);
    $container->cells[] = $account->name;
    $account_details = _get_account($id);
    $container->cells[] = $account_details['amount'];
}

And when I got home, my Windows XP blue-screened before the user login. I can’t reinstall the OS because my HD is SATA, and I don’t have a floppy disk.

I am fucking mad.

22 de November de 2006

Django on SliceHost Virtual Private Server

Filed under: Perils of Software Development — inerte @ 09:47

I’ve recently moved my Django websites from Dreamhost due to poor FastCGI performance to the 256 plan option from SliceHost. If you sign-up with them, please use “[email protected]” as your referral. I have no idea how much they pay for referrals, but any money is good money 😉

Here’s my guide:

Sign-up to SliceHost

Painless registration gave me the root password in 2 minutes. Write down your IP address and your root password somewhere so you won’t forget it.

Setup the packages

SSH (I’ve used Putty) to your server, authenticate, and execute the commands below. I’ve decided to install PHP because I have domains using it. Also, it’s necessary for phpMyAdmin.

$ apt-get install apache2
$ apt-get install libapache2-mod-python2.4
$ apt-get install mysql-server
$ apt-get install python2.4-mysqldb
$ apt-get install php5
$ apt-get install php5-mysql
$ apache2ctl restart
$ /etc/init.d/apache2 reload

Setup Mysql

$ mysql -u root
mysql> UPDATE mysql.user SET Password=PASSWORD ('your_root_mysql_password') WHERE User = 'root';
mysql> FLUSH PRIVILEGES;
mysql> quit

Optional: Install phpMyAdmin

It’s just easier. I’ve decided not to install a FTP server to upload my files. Instead, I’ve used WinSFTP, a sftp client for Microsoft Windows. Download and install it. Open, paste your Slice IP address and browse to the /var/www/ directory, upload phpMyAdmin and follow its install instructions.

Setup Django

$ cd /usr/lib/python2.4/site-packages/
$ svn co http://code.djangoproject.com/svn/django/trunk/ django

Setup your Django project

Open WinSFTP again, browse to the /usr/lib/python2.4/site-packages/ directory and upload your Django project.

Edit its settings.py file.

vi /usr/lib/python2.4/site-packages/your_django_project/settings.py

I’ll show only what you have to change, besides whatever is needed for your project to work (like INSTALLED_APPS):

DATABASE_ENGINE = 'mysql'
DATABASE_NAME = 'your_db_name'
DATABASE_USER = 'your_db_user'
DATABASE_PASSWORD = 'your_db_password'

I use a directory for media files called “web” on most of my projects:

MEDIA_ROOT = '/var/www/your_domain.com/web/'
MEDIA_URL = '/web/'

You’ll also have to change the TEMPLATE_DIRS tuple. Just put whatever you use. Here’s mine for reference:

TEMPLATE_DIRS = (
# Put strings here, like “/home/html/django_templates”.
# Always use forward slashes, even on Windows.
‘/usr/lib/python2.4/site-packages/my_django_project/templates/my_django_project/’,
)

Setup the domain

$ mkdir /var/www/your_domain.com
$ mkdir /var/log/apache2/your_domain.com
$ vi /etc/apache2/sites-available/your_domain.com

Paste this text inside the file:

<VirtualHost *>
ServerName www.your_domain.com
ServerAlias your_domain.com
# The three lines below remove the www from the domain name. I don’t like wwws.
RewriteEngine On
RewriteCond %{HTTP_HOST} ^www\.your_domain\.com [NC]
RewriteRule (.*) http://your_domain.com$1 [R=301,L]

DocumentRoot /var/www/your_domain.com

CustomLog /var/log/apache2/your_domain.com/access.log combined
ErrorLog /var/log/apache2/your_domain.com/error.log

SetHandler python-program
PythonHandler django.core.handlers.modpython
SetEnv DJANGO_SETTINGS_MODULE your_django_project.settings
PythonDebug Off
PythonPath “[‘/usr/lib/python2.4/site-packages/django’] + sys.path”

# My own media directory (as mentioned in the previous section)
<Location “/web/”>
SetHandler None
</Location>
# Necessary for Django’s admin media files
<Location “/media/”>
SetHandler None
</Location>
</VirtualHost>

Symlink your new domain configuration file to the correct directory:

ln -s /etc/apache2/sites-available/your_domain.com /etc/apache2/sites-enabled/your_domain.com

Symlink Django’s admin media files to your domain:

ln -s /usr/lib/python2.4/site-packages/django/django/contrib/admin/media/ /var/www/your_domain.com/media

Edit Apache’s configuration file to tell it your IP address:

vi /etc/apache2/apache2.conf

Paste this before the # Include the virtual host configurations: line (it’s one before the last):

ServerName your.slice.ip.address

Edit Apache’s log rotation to include your new domain:

vi /etc/logrotate.d/apache2

Paste this at the end:

/var/log/apache2/your_domain.com/*.log {
        weekly
        missingok
        rotate 52
        compress
        delaycompress
        notifempty
        create 640 root adm
        sharedscripts
        postrotate
                if [ -f /var/run/apache2.pid ]; then
                        /etc/init.d/apache2 restart > /dev/null
                fi
        endscript
}

DNS Server

I’ve tried to install a DNS Server but not only I thought it’s hard, having a single point of failure is bad. So I’ve signed up with DNS Made Easy and I let them manage this for me. It’s super simple, after joining, add your domain to DNS Made Easy and write down the DNS server addresses. Wait until the domain name is “created” (to me, it varied from 30 minutes to 2 hours), and change the dns servers from your domain name registrar.

Last action

Restart Apache one more time:

$ apache2ctl restart
$ /etc/init.d/apache2 reload

And we’re done! With one caveat: We’ve checked out Django “trunk” directory into your Python’s directory. That means the real django lives a directory below:

/usr/lib/python2.4/site-packages/django/django/

Which means that if you ever want to use Django outside Apache’s mod_python, you’ll have to add the /usr/lib/python2.4/site-packages/django directory to your sys.path. Or, you could checkout Django to somewhere else, and move the “real” django directory to site-packages/ (and change the /etc/apache2/sites-available/your_domain.com) accordingly.

If you’re having problems, write something at the comments here and I will try to help you.

Older Posts »

Powered by WordPress