Simple chrooted FTP setup on EC2 micro instance

Source environment: Ubuntu

1. Install vsftpd
apt-get install vsftpd

2. Edit default config at /etc/vsftpd.conf

Make sure the you enable these:

local_enable=YES
write_enable=YES

chroot_local_user=YES
chroot_list_enable=YES
# (default follows)
chroot_list_file=/etc/vsftpd.chroot_list

Ensure this is disabled:

anonymous_enable=NO

and add the following to the end:

pasv_enable=YES
pasv_max_port=22100
pasv_min_port=22000
pasv_address=123.123.123.123 # REPLACE THIS WITH YOUR IP
port_enable=YES

max, min ports could be anything high enough not to overlap with other services. Those ports will also need to be open in your security group if you’re using EC2

3. Create/edit /etc/vstfp.chroot_list
Add usernames that you don’t want to chroot.

4. Create users for FTP access:

adduser USERNAME

5. Ensure the home folder of a user is not writable(!) This is new since VSFTP 2.3.5 I believe.

chmod a-w /home/USERNAME

6. Create folders under /home/USERNAME for a user to upload stuff to, since a user won’t be able to upload to the root of /home/USERNAME

Ubuntu 11.10 or 12.04 fail to boot after upgrade due to software raid degrade/failure.

Since I first started using Ubuntu back in ’09 with 9.04 I had issues with my software RAID array roughly about every other time I am trying to upgrade to a newer version. Almost everytime the issue lies in GRUB not being able to install/update itself properly, so I end up just doing that manually from the rescue disk – process I have unintentionally learned by heart.

This time around, when I upgraded from 11.04 to 11.10 – it was a different issue. System failed to boot and dropped into initramfs/BusyBox with failure to assemble one of the software RAIDs. Apparently there was an update introduced in 11.10(I believe) that prevents system to boot if there is any software RAID array that it could not assemble fully. This could be an issue if for example your drives got mixed or, like in my case, I had one older RAID array defined that was not properly removed, but was always deactivated.

There is a pretty long, yet interesting conversation here on this matter: https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/872220

The way to solve this for me was to hit Ctrl-D when it dropped into initramfs/BusyBox, select ‘root shell’ and fix the issue – properly deactive the array I didn’t need and fix my working RAID array, that got degraded and needed to rebalance.

Oh, well… The Ubuntu upgrade process is still not there.

How to install Xapian 1.2.5 PHP bindings on Ubuntu Lucid Lynx

Starting from version 1.2.x, Xapian repository on Ubuntu does not contain php5-xapian package :( apparently due to the license incompatability between GPL and PHP license(great…)
Issue is discussed somewhat at length here.

But in the meantime, folks suggesting to build PHP bindings for Xapian manually on Ubuntu and Debian. Here is a quick command trail that shows how to install Xapian 1.2.5 PHP bindings on Ubuntu Lucid(10.04), also tested on Ubuntu 10.10 and 11.04:

1. Edit /etc/apt/sources.list and add the following lines to it:

deb http://ppa.launchpad.net/xapian-backports/xapian-1.2/ubuntu lucid main
deb-src http://ppa.launchpad.net/xapian-backports/xapian-1.2/ubuntu lucid main

2. Get some required packages:

sudo apt-get update
sudo apt-get build-dep xapian-bindings
sudo apt-get install php5-dev php5-cli
sudo apt-get install devscripts

3. Fetch sources and build:

apt-get source xapian-bindings
cd xapian-bindings-1.2.5
rm debian/control
env PHP_VERSIONS=5 debian/rules maint
debuild -e PHP_VERSIONS=5 -us -uc

This will generate .deb file in the folder, one level up.

4. Finally, install php5-xapian extenstion:

cd ..
sudo dpkg -i php5-xapian*.deb

5. Verify that you got it running:

php -i | grep Xapian

Information about this process is taken from here and here.

Auto-splitting video file in equal chunks with ffmpeg and python

UPDATE: The source code for this simple script has been moved to GitHub, get it here: https://github.com/c0decracker/video-splitter

Recently I needed to upload a whole bunch of long video files. Maximum allowed length for each video was just few minutes, while the actual length of files I tried to upload were about an hour each. FFmpeg is really great for splitting the video files and Python is quite handy for automating the task. Combining two together in this handy little script(see below). The script below takes a video file and a chunk size in seconds and splits the video file into chunks using ffmpeg, so each chunk is self contained, playable video.

Source code:

#!/usr/bin/env python

import subprocess
import re
import math
from optparse import OptionParser


length_regexp = 'Duration: (\d{2}):(\d{2}):(\d{2})\.\d+,'
re_length = re.compile(length_regexp)

def main():

    (filename, split_length) = parse_options()
    if split_length <= 0:
        print "Split length can't be 0"
        raise SystemExit

    output = subprocess.Popen("ffmpeg -i '"+filename+"' 2>&1 | grep 'Duration'", 
                            shell = True,
                            stdout = subprocess.PIPE
                            ).stdout.read()
    print output
    matches = re_length.search(output)
    if matches:
        video_length = int(matches.group(1)) * 3600 + \
                        int(matches.group(2)) * 60 + \
                        int(matches.group(3))
        print "Video length in seconds: "+str(video_length)
    else:
        print "Can't determine video length."
        raise SystemExit

    split_count = math.ceil(video_length/float(split_length))
    if(split_count == 1):
        print "Video length is less then the target split length."
        raise SystemExit

    split_cmd = "ffmpeg -i '"+filename+"' -vcodec copy "
    for n in range(0, split_count):
        split_str = ""
        if n == 0:
            split_start = 0
        else:
            split_start = split_length * n
        
        split_str += " -ss "+str(split_start)+" -t "+str(split_length) + \
                    " '"+filename[:-4] + "-" + str(n) + "." + filename[-3:] + \
                    "'"
        print "About to run: "+split_cmd+split_str
        output = subprocess.Popen(split_cmd+split_str, shell = True, stdout =
                               subprocess.PIPE).stdout.read()


def parse_options():
    parser = OptionParser()    
    
    parser.add_option("-f", "--file",
                        dest = "filename",
                        help = "file to split, for example sample.avi",
                        type = "string",
                        action = "store"
                        )
    parser.add_option("-s", "--split-size",
                        dest = "split_size",
                        help = "split or chunk size in seconds, for example 10",
                        type = "int",
                        action = "store"
                        )
    (options, args) = parser.parse_args()
    
    if options.filename and options.split_size:

        return (options.filename, options.split_size)

    else:
        parser.print_help()
        raise SystemExit

if __name__ == '__main__':

    try: 
        main()
    except Exception, e:
        print "Exception occured running main():"
        print str(e)


Or download it here: splitting video file script link

Build OpenLDAP 2.3.x from sources on Ubuntu

It happened so that I needed to build a particular version of OpenLDAP on Ubuntu and use that instead of the one in Ubuntu’s repository. Here is a quick guide as to how it worked for me:

1. Get BDB 4.3 sources from Oracle’s site(link)

2. Compile BDB and install it:

tar –xvzf db-4.3.29.tar.gz
cd db-4.3.29/build_unix
./configure --prefix=/usr/local/bdb43
make 
sudo make install

3. Get OpenLDAP source(ftp link)
4. Compile and install it:

tar xzvf openldap-2.3.35.tgz
cd openldap
export CPPFLAGS="-I/usr/local/bdb43/include -D_GNU_SOURCE" 
export LDFLAGS="-L/usr/local/lib -L/usr/local/bdb43/lib -R/usr/local/bdb43lib" 
export LD_LIBRARY_PATH="/usr/local/bdb43/lib"
./configure --prefix=/usr/local/openldap

If you’re installing version 2.3.x or anything before 2.4.15 you will need to manually patch OpenLDAP otherwise you’ll get this error:
../../include/ldap_pvt_thread.h:64: error: missing binary operator before token “(“
Patch file is available in this bug report
Or already patched file for OpenLDAP version 2.3.35 you can download here

If you’re installing anything >= 2.4.15, you can skip the patch.

Once patch is applied just run:

make depend
make
make test  #this will take a while to run 
make install

Common Errors:
configure: error: Berkeley DB version mismatch
Solution: Most likely you didn’t LDFLAGS and LD_LIBRARY_PATH as noted above

getpeereid.c:52: error: storage size of ‘peercred’ isn’t known
You need to include -D_GNU_SOURCE flag, to avoid incompatibility with glibc

../../include/ldap_pvt_thread.h:64: error: missing binary operator before token “(“
Solution: Apply patch as noted above

error while loading shared libraries: libdb-4.3.so: cannot open shared object file: No such file or directory
Solution: Add libdb-4.3 to shared libs cache:

sudo echo "/usr/local/bdb43/lib" > /etc/ld.so.conf.d/slapd.conf
lddconfig -v

Getting around open_basedir restriction on Plesk

Almost every time I do a deploy on Plesk-based hosting server, I hit the same roadblock:

PHP Warning:  realpath() [<a href='function.realpath'>function.realpath</a>]: open_basedir restriction in effect.

Since I tend to keep my app-code outside of webroot, this restriction is somewhat a problem. Well, luckily, the fix is simple enough.

1. Add file vhost.conf to /var/www/vhosts/YOURDOMAIN/conf folder with the following:

<Directory /var/www/vhosts/YOURDOMAIN/httpdocs/>
<IfModule sapi_apache2.c>
php_admin_value open_basedir none
</IfModule>
<IfModule mod_php5.c>
php_admin_value open_basedir none
</IfModule>
</Directory>

2. Reload server configuration, either via Plesk admin or through CLI:

/usr/local/psa/admin/sbin/websrvmng -u --vhost-name=yourdomain