How to install Xapian 1.2.5 PHP bindings on Ubuntu Lucid Lynx

Starting from version 1.2.x, Xapian repository on Ubuntu does not contain php5-xapian package :( apparently due to the license incompatability between GPL and PHP license(great…)
Issue is discussed somewhat at length here.

But in the meantime, folks suggesting to build PHP bindings for Xapian manually on Ubuntu and Debian. Here is a quick command trail that shows how to install Xapian 1.2.5 PHP bindings on Ubuntu Lucid(10.04), also tested on Ubuntu 10.10 and 11.04:

1. Edit /etc/apt/sources.list and add the following lines to it:

deb http://ppa.launchpad.net/xapian-backports/xapian-1.2/ubuntu lucid main
deb-src http://ppa.launchpad.net/xapian-backports/xapian-1.2/ubuntu lucid main

2. Get some required packages:

sudo apt-get update
sudo apt-get build-dep xapian-bindings
sudo apt-get install php5-dev php5-cli
sudo apt-get install devscripts

3. Fetch sources and build:

apt-get source xapian-bindings
cd xapian-bindings-1.2.5
rm debian/control
env PHP_VERSIONS=5 debian/rules maint
debuild -e PHP_VERSIONS=5 -us -uc

This will generate .deb file in the folder, one level up.

4. Finally, install php5-xapian extenstion:

cd ..
sudo dpkg -i php5-xapian*.deb

5. Verify that you got it running:

php -i | grep Xapian

Information about this process is taken from here and here.

Auto-splitting video file in equal chunks with ffmpeg and python

UPDATE: The source code for this simple script has been moved to GitHub, get it here: https://github.com/c0decracker/video-splitter

Recently I needed to upload a whole bunch of long video files. Maximum allowed length for each video was just few minutes, while the actual length of files I tried to upload were about an hour each. FFmpeg is really great for splitting the video files and Python is quite handy for automating the task. Combining two together in this handy little script(see below). The script below takes a video file and a chunk size in seconds and splits the video file into chunks using ffmpeg, so each chunk is self contained, playable video.

Source code:

#!/usr/bin/env python

import subprocess
import re
import math
from optparse import OptionParser


length_regexp = 'Duration: (\d{2}):(\d{2}):(\d{2})\.\d+,'
re_length = re.compile(length_regexp)

def main():

    (filename, split_length) = parse_options()
    if split_length <= 0:
        print "Split length can't be 0"
        raise SystemExit

    output = subprocess.Popen("ffmpeg -i '"+filename+"' 2>&1 | grep 'Duration'", 
                            shell = True,
                            stdout = subprocess.PIPE
                            ).stdout.read()
    print output
    matches = re_length.search(output)
    if matches:
        video_length = int(matches.group(1)) * 3600 + \
                        int(matches.group(2)) * 60 + \
                        int(matches.group(3))
        print "Video length in seconds: "+str(video_length)
    else:
        print "Can't determine video length."
        raise SystemExit

    split_count = math.ceil(video_length/float(split_length))
    if(split_count == 1):
        print "Video length is less then the target split length."
        raise SystemExit

    split_cmd = "ffmpeg -i '"+filename+"' -vcodec copy "
    for n in range(0, split_count):
        split_str = ""
        if n == 0:
            split_start = 0
        else:
            split_start = split_length * n
        
        split_str += " -ss "+str(split_start)+" -t "+str(split_length) + \
                    " '"+filename[:-4] + "-" + str(n) + "." + filename[-3:] + \
                    "'"
        print "About to run: "+split_cmd+split_str
        output = subprocess.Popen(split_cmd+split_str, shell = True, stdout =
                               subprocess.PIPE).stdout.read()


def parse_options():
    parser = OptionParser()    
    
    parser.add_option("-f", "--file",
                        dest = "filename",
                        help = "file to split, for example sample.avi",
                        type = "string",
                        action = "store"
                        )
    parser.add_option("-s", "--split-size",
                        dest = "split_size",
                        help = "split or chunk size in seconds, for example 10",
                        type = "int",
                        action = "store"
                        )
    (options, args) = parser.parse_args()
    
    if options.filename and options.split_size:

        return (options.filename, options.split_size)

    else:
        parser.print_help()
        raise SystemExit

if __name__ == '__main__':

    try: 
        main()
    except Exception, e:
        print "Exception occured running main():"
        print str(e)


Or download it here: splitting video file script link

Build OpenLDAP 2.3.x from sources on Ubuntu

It happened so that I needed to build a particular version of OpenLDAP on Ubuntu and use that instead of the one in Ubuntu’s repository. Here is a quick guide as to how it worked for me:

1. Get BDB 4.3 sources from Oracle’s site(link)

2. Compile BDB and install it:

tar –xvzf db-4.3.29.tar.gz
cd db-4.3.29/build_unix
./configure --prefix=/usr/local/bdb43
make 
sudo make install

3. Get OpenLDAP source(ftp link)
4. Compile and install it:

tar xzvf openldap-2.3.35.tgz
cd openldap
export CPPFLAGS="-I/usr/local/bdb43/include -D_GNU_SOURCE" 
export LDFLAGS="-L/usr/local/lib -L/usr/local/bdb43/lib -R/usr/local/bdb43lib" 
export LD_LIBRARY_PATH="/usr/local/bdb43/lib"
./configure --prefix=/usr/local/openldap

If you’re installing version 2.3.x or anything before 2.4.15 you will need to manually patch OpenLDAP otherwise you’ll get this error:
../../include/ldap_pvt_thread.h:64: error: missing binary operator before token “(“
Patch file is available in this bug report
Or already patched file for OpenLDAP version 2.3.35 you can download here

If you’re installing anything >= 2.4.15, you can skip the patch.

Once patch is applied just run:

make depend
make
make test  #this will take a while to run 
make install

Common Errors:
configure: error: Berkeley DB version mismatch
Solution: Most likely you didn’t LDFLAGS and LD_LIBRARY_PATH as noted above

getpeereid.c:52: error: storage size of ‘peercred’ isn’t known
You need to include -D_GNU_SOURCE flag, to avoid incompatibility with glibc

../../include/ldap_pvt_thread.h:64: error: missing binary operator before token “(“
Solution: Apply patch as noted above

error while loading shared libraries: libdb-4.3.so: cannot open shared object file: No such file or directory
Solution: Add libdb-4.3 to shared libs cache:

sudo echo "/usr/local/bdb43/lib" > /etc/ld.so.conf.d/slapd.conf
lddconfig -v

Getting around open_basedir restriction on Plesk

Almost every time I do a deploy on Plesk-based hosting server, I hit the same roadblock:

PHP Warning:  realpath() [<a href='function.realpath'>function.realpath</a>]: open_basedir restriction in effect.

Since I tend to keep my app-code outside of webroot, this restriction is somewhat a problem. Well, luckily, the fix is simple enough.

1. Add file vhost.conf to /var/www/vhosts/YOURDOMAIN/conf folder with the following:

<Directory /var/www/vhosts/YOURDOMAIN/httpdocs/>
<IfModule sapi_apache2.c>
php_admin_value open_basedir none
</IfModule>
<IfModule mod_php5.c>
php_admin_value open_basedir none
</IfModule>
</Directory>

2. Reload server configuration, either via Plesk admin or through CLI:

/usr/local/psa/admin/sbin/websrvmng -u --vhost-name=yourdomain

Gnome Do, Docky, upgrade to Ubuntu 10.10 and an odd case of /var/lib/dpkg/tmp.ci/md5sums

So I finally upgraded to 10.10, a bit behind the curve but whatever. For the most part it went surprisingly swimmingly. I only had 2 issues :)

First one – gnome-do docky theme was gone. And apprently it’s gone for good and now it is a separate project. Fix is as simple as:


sudo apt-get install docky

Fixed that right away.

Second issue was a bit more annoying. Problem was with updating linux-firmware package. For some reason dpkg was throwing and error saying that ‘/var/lib/dpkg/tmp.ci/md5sums: Is a directory‘. After messing around fix for that was basically this:

sudo bash
cd /var/lib/dpkg/info
rm -rf linux-firwmare*
apt-get upgrade

What happen is /var/lib/dpkg/info/linux-firmware.md5sums got corrupted and instead of being a file it became a directory point to postfix’s folder(in my case). I had some hard drive issues few weeks ago, so probably fsck did this… oh well, can’t blame it :)

WordPress shortcode for ShareThis buttons

I like using ShareThis for social sharing button, but one thing that was certainly missing for the default ShareThis WordPress plugin were shortcodes. Luckily WordPress makes it really easy adding them :) So to get shortcode for ShareThis integration just add the following few lines to the end of the functions.php in your theme folder:

/**
* Creates sharethis shortcode
*/
if (function_exists('st_makeEntries')) :
add_shortcode('sharethis', 'st_makeEntries');
endif;

Once that’s done you can use [sharethis] shortcode anywhere on the page/post to get sharing buttons rendered there. Short and sweet :)

How to setup local PHP debugging with PHPStorm and Xdebug

I am checking out a new IDE on the block – PHPStorm… well, new for me anyway :)

For Java and Python, Eclipse is all I need, but when coding in PHP, I am not a big IDE guy, for me, Vi is still the weapon of choice. But when it comes to debugging complex apps(yeah you can write those in PHP too :) , Vi doesn’t quite cut it. There are plugins for it to provide debugging capabilities(or rather Vimified interface to Xdebug), but half of the time it doesn’t work and the other half of the time it’s not what I am looking for. So far, I’ve been a user of Komodo IDE, I still have the old version 4 up and running here and do most of my debugging in it. It’s nice on one side, but somewhat cumbersome on the other.

Anyway, one major thing that always kept me from trying new IDEs is debugger setup. I tend to spend 10 minutes trying to setup debugging in new IDE, get frustrated, say fuck it, and get back to Komodo. This short blog post is to capture how I setup debugging in PHPStorm using Xdebug.
There is a video about this as to how to get a remote debugging working, but not much info about how to get a local one for folks who is on Linux(Ubuntu in my case).
So there we go:

Xdebug Setup

Step 0. Get phpize. If you’re on Ubuntu it’s as simple as running sudo apt-get install php5-dev
Step 1. Get Xdebug, it’s pure awesomeness. Just go here: http://xdebug.org/find-binary.php and follow the instructions. Note: you can also try installing it from pecl or Ubuntu repository, but I had problems with these methods before.
Step 2. I recommend not to follow the last step of Xdebug install instructions where it says to add zend_extension line to the php.ini file. Cleaner way of enabling Xdebug and then configuring it is to add a file xdebug.ini to /etc/php5/conf.d/ folder with the following contents:

;xdebug configuration
zend_extension = /usr/lib/php5/20090626/xdebug.so
xdebug.remote_host = 127.0.0.1
xdebug.remote_enable = 1
xdebug.remote_port = 9000
xdebug.remote_handler = dbgp
xdebug.remote_mode = req

This xdebug.ini will be picked up by both CLI and Apache and makes it easier to control without messing with the central config file.
Once apache is restarted and xdebug succesfully loaded, time to look at phpStorm’s side.

Step 3. Get Xdebug helper extension for Chrome or similar for Firefox is very handy. Get one so you can enable/disable debugging sessions from the browser. Make sure you set it up such that is loads up on your local domain. For example, if your project runs on myproject.local, your setup will prob look like this:

PHPStorm Debugging Setup

This is a bit less straightforward, but here I go anyway. Here are my assumptions:
My project is running localy at http://myproject.local
It’s stored locally in /home/user/Projects/myproject/public ,which is also a DocumentRoot for myproject.local Vhost config in Apache. There are ‘library’ and ‘application’ folders outside of DocRoot however.

Step 1. In PHPStorm, setup path to web browser(File->Settings->Web Browsers). It actually detected all paths correctly except for Chrome, which I had to set manually(in my case it was in /usr/bin/google-chrome)

Step 2. Setup run configuration – Run->Edit Configurations. That screen is kinda odd, but here is how I got it going.
* Create new configuration, with type PHP on Server

* Create new server(click on the button next to “Server” dropdown). I picked server type “Local” and set webroot to “http://myproject.local“. On the “Mappings” tab I set local path to /home/user/Projects/myproject/public
* Save that server and select it in the Run Configurations
Ultimately my config looked as following:

This looks like it. Now opening up a file inside that project and ‘running’ it will actually bring up a new Chrome window with a page and put PHPStorm in listening mode on port 9000 for Xdebug connections. You’ll need to activate ‘Xdebug helper’ in that window, after that all interactions with the server will be intercepted by Xdebug and PHPStorm.

How to install Mono on RedHat/Plesk 8.6.x

Mono is a savior if you’re migrating Windows sites to Linux/Apache stack. I had to do this just recently and had to setup Mono on RHEL box with Plesk 8.6. It was relatively straighforward process and here is quick log as to how installed Apache Mono on Plesk:

1. Add Mono repository to yum:


wget -O /etc/yum.repos.d/mono.repo http://ftp.novell.com/pub/mono/download-stable/rhel-4-i386/mono.repo

2. Install Mono:

yum install mono-core mono-web mono-data mod_mono xsp

Note: You may get missing dependency notice.

Missing Dependency: libexif.so.9 is needed by package libgdiplus0-1.9-1.rhel4.novell.i386 (mono)
Error: Missing Dependency: libexif.so.9 is needed by package libgdiplus0-1.9-1.rhel4.novell.i386 (mono)
You could try using --skip-broken to work around the problem
You could try running: package-cleanup --problems
package-cleanup --dupes
rpm -Va --nofiles --nodigest

If that’ the case you’ll need to take a different route. Remove that /etc/yum.repos.d/mono.repo and create a new file centos-5-extras.repo and put the folloging content in there:

[centos-5-extras]
name=CentOS-5 - $basearch - Extras
#baseurl=http://mirror.centos.org/centos/5/extras/$basearch/
mirrorlist=http://mirrorlist.centos.org/?release=5&arch=$basearch&repo=extras
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5
enabled=0

Then run this:

yum --enablerepo=centos-5-extras install mono-core mono-web mono-data mod_mono xsp

This should do it. Just make sure mono is installed afterwards:

mono -V

3. Check & configure mod_mono
Install process should’ve created the configuration file for mod_mono. Check if it exists here: /etc/httpd/conf.d/mod_mono.conf
If it doesn’t exist there create it and put the following contents:

# mod_mono.conf
<IfModule !mod_mono.c>
LoadModule mono_module /usr/lib/httpd/modules/mod_mono.so
AddType application/x-asp-net .aspx
AddType application/x-asp-net .asmx
AddType application/x-asp-net .ashx
AddType application/x-asp-net .asax
AddType application/x-asp-net .ascx
AddType application/x-asp-net .soap
AddType application/x-asp-net .rem
AddType application/x-asp-net .axd
AddType application/x-asp-net .cs
AddType application/x-asp-net .config
AddType application/x-asp-net .Config
AddType application/x-asp-net .dll
DirectoryIndex index.aspx
DirectoryIndex Default.aspx
DirectoryIndex default.aspx
</IfModule>

4. Restart Apache

/usr/local/psa/admin/sbin/websrvmng -r

This did it for me. Along the way I encountered few issues, looking closesly at /var/log/httpd/error_log helped to find solutions(in my case it was missing mono-web package, that brings in System.Web library I guess).

How to move MySQL storage to RamFS or TmpFS partition

Whether moving all MySQL storage to a tmpfs helps with speeding it up or not is questionable but I needed to do for some testing purposes, so this is a short overview of how I did that hopefully will be useful:

First mount tmpfs to a folder:

sudo mkdir /var/ramfs
sudo mount -t ramfs -o size=1G ramfs /var/ramfs/

Here I mounted ramfs to /var/ramfs. I am using ramfs in oppose to tmpfs mainly because:

  • ramfs grows dynamically(tmpfs doens’t)
  • ramfs doesn’t use swap(while tmpfs does)

RAM-backed file system is mounted, so now I need to populate it with MySQL files for processing.
To do that I will need to stop mysql, copy it’s database files over to ramfs, adjust AppArmor and MySQL settings and start mysql server again. Here is the chain of commands to do that:

Copying files:

sudo /etc/init.d/mysql stop
sudo cp -R /var/lib/mysql /var/ramfs/
sudo chown -R mysql:mysql /var/ramfs/mysql

Tweaking MySQL config:

sudo cp /etc/mysql/my.cnf /etc/mysql/original-my.cnf
sudo vim /etc/mysql/my.cnf

Find line with ‘datadir‘ definition(it will look something like datadir = /var/lib/mysql) and change it to

datadir = /var/ramfs/mysql

Next step is to tune apparmor settings:

sudo vim /etc/apparmor.d/usr.sbin.mysqld

Add the following few lines just before the closing curly braces:


/var/ramfs/mysql/ r,
/var/ramfs/mysql/*.pid rw,
/var/ramfs/mysql/** rwk,

Looks like we’re done with settings, let’s see if it will work:


sudo /etc/init.d/apparmor restart
sudo /etc/init.d/mysql start

If mysql daemon starts(double check /var/log/mysql.err for any errors) and you can connect to it, mostlikely now we’re running fully off of a RAM device. To double check it, run this from mysql client:

mysql> show variables where Variable_name = 'datadir' \G
*************************** 1. row ***************************
Variable_name: datadir
Value: /var/ramfs/mysql/
1 row in set (0.00 sec)

That’s pretty much it :)