Simple chrooted FTP setup on EC2 micro instance

Source environment: Ubuntu

1. Install vsftpd
apt-get install vsftpd

2. Edit default config at /etc/vsftpd.conf

Make sure the you enable these:

local_enable=YES
write_enable=YES

chroot_local_user=YES
chroot_list_enable=YES
# (default follows)
chroot_list_file=/etc/vsftpd.chroot_list

Ensure this is disabled:

anonymous_enable=NO

and add the following to the end:

pasv_enable=YES
pasv_max_port=22100
pasv_min_port=22000
pasv_address=123.123.123.123 # REPLACE THIS WITH YOUR IP
port_enable=YES

max, min ports could be anything high enough not to overlap with other services. Those ports will also need to be open in your security group if you’re using EC2

3. Create/edit /etc/vstfp.chroot_list
Add usernames that you don’t want to chroot.

4. Create users for FTP access:

adduser USERNAME

5. Ensure the home folder of a user is not writable(!) This is new since VSFTP 2.3.5 I believe.

chmod a-w /home/USERNAME

6. Create folders under /home/USERNAME for a user to upload stuff to, since a user won’t be able to upload to the root of /home/USERNAME

Ubuntu 11.10 or 12.04 fail to boot after upgrade due to software raid degrade/failure.

Since I first started using Ubuntu back in ’09 with 9.04 I had issues with my software RAID array roughly about every other time I am trying to upgrade to a newer version. Almost everytime the issue lies in GRUB not being able to install/update itself properly, so I end up just doing that manually from the rescue disk – process I have unintentionally learned by heart.

This time around, when I upgraded from 11.04 to 11.10 – it was a different issue. System failed to boot and dropped into initramfs/BusyBox with failure to assemble one of the software RAIDs. Apparently there was an update introduced in 11.10(I believe) that prevents system to boot if there is any software RAID array that it could not assemble fully. This could be an issue if for example your drives got mixed or, like in my case, I had one older RAID array defined that was not properly removed, but was always deactivated.

There is a pretty long, yet interesting conversation here on this matter: https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/872220

The way to solve this for me was to hit Ctrl-D when it dropped into initramfs/BusyBox, select ‘root shell’ and fix the issue – properly deactive the array I didn’t need and fix my working RAID array, that got degraded and needed to rebalance.

Oh, well… The Ubuntu upgrade process is still not there.

How to install Xapian 1.2.5 PHP bindings on Ubuntu Lucid Lynx

Starting from version 1.2.x, Xapian repository on Ubuntu does not contain php5-xapian package :( apparently due to the license incompatability between GPL and PHP license(great…)
Issue is discussed somewhat at length here.

But in the meantime, folks suggesting to build PHP bindings for Xapian manually on Ubuntu and Debian. Here is a quick command trail that shows how to install Xapian 1.2.5 PHP bindings on Ubuntu Lucid(10.04), also tested on Ubuntu 10.10 and 11.04:

1. Edit /etc/apt/sources.list and add the following lines to it:

deb http://ppa.launchpad.net/xapian-backports/xapian-1.2/ubuntu lucid main
deb-src http://ppa.launchpad.net/xapian-backports/xapian-1.2/ubuntu lucid main

2. Get some required packages:

sudo apt-get update
sudo apt-get build-dep xapian-bindings
sudo apt-get install php5-dev php5-cli
sudo apt-get install devscripts

3. Fetch sources and build:

apt-get source xapian-bindings
cd xapian-bindings-1.2.5
rm debian/control
env PHP_VERSIONS=5 debian/rules maint
debuild -e PHP_VERSIONS=5 -us -uc

This will generate .deb file in the folder, one level up.

4. Finally, install php5-xapian extenstion:

cd ..
sudo dpkg -i php5-xapian*.deb

5. Verify that you got it running:

php -i | grep Xapian

Information about this process is taken from here and here.

Build OpenLDAP 2.3.x from sources on Ubuntu

It happened so that I needed to build a particular version of OpenLDAP on Ubuntu and use that instead of the one in Ubuntu’s repository. Here is a quick guide as to how it worked for me:

1. Get BDB 4.3 sources from Oracle’s site(link)

2. Compile BDB and install it:

tar –xvzf db-4.3.29.tar.gz
cd db-4.3.29/build_unix
./configure --prefix=/usr/local/bdb43
make 
sudo make install

3. Get OpenLDAP source(ftp link)
4. Compile and install it:

tar xzvf openldap-2.3.35.tgz
cd openldap
export CPPFLAGS="-I/usr/local/bdb43/include -D_GNU_SOURCE" 
export LDFLAGS="-L/usr/local/lib -L/usr/local/bdb43/lib -R/usr/local/bdb43lib" 
export LD_LIBRARY_PATH="/usr/local/bdb43/lib"
./configure --prefix=/usr/local/openldap

If you’re installing version 2.3.x or anything before 2.4.15 you will need to manually patch OpenLDAP otherwise you’ll get this error:
../../include/ldap_pvt_thread.h:64: error: missing binary operator before token “(“
Patch file is available in this bug report
Or already patched file for OpenLDAP version 2.3.35 you can download here

If you’re installing anything >= 2.4.15, you can skip the patch.

Once patch is applied just run:

make depend
make
make test  #this will take a while to run 
make install

Common Errors:
configure: error: Berkeley DB version mismatch
Solution: Most likely you didn’t LDFLAGS and LD_LIBRARY_PATH as noted above

getpeereid.c:52: error: storage size of ‘peercred’ isn’t known
You need to include -D_GNU_SOURCE flag, to avoid incompatibility with glibc

../../include/ldap_pvt_thread.h:64: error: missing binary operator before token “(“
Solution: Apply patch as noted above

error while loading shared libraries: libdb-4.3.so: cannot open shared object file: No such file or directory
Solution: Add libdb-4.3 to shared libs cache:

sudo echo "/usr/local/bdb43/lib" > /etc/ld.so.conf.d/slapd.conf
lddconfig -v

Gnome Do, Docky, upgrade to Ubuntu 10.10 and an odd case of /var/lib/dpkg/tmp.ci/md5sums

So I finally upgraded to 10.10, a bit behind the curve but whatever. For the most part it went surprisingly swimmingly. I only had 2 issues :)

First one – gnome-do docky theme was gone. And apprently it’s gone for good and now it is a separate project. Fix is as simple as:


sudo apt-get install docky

Fixed that right away.

Second issue was a bit more annoying. Problem was with updating linux-firmware package. For some reason dpkg was throwing and error saying that ‘/var/lib/dpkg/tmp.ci/md5sums: Is a directory‘. After messing around fix for that was basically this:

sudo bash
cd /var/lib/dpkg/info
rm -rf linux-firwmare*
apt-get upgrade

What happen is /var/lib/dpkg/info/linux-firmware.md5sums got corrupted and instead of being a file it became a directory point to postfix’s folder(in my case). I had some hard drive issues few weeks ago, so probably fsck did this… oh well, can’t blame it :)

How to install Mono on RedHat/Plesk 8.6.x

Mono is a savior if you’re migrating Windows sites to Linux/Apache stack. I had to do this just recently and had to setup Mono on RHEL box with Plesk 8.6. It was relatively straighforward process and here is quick log as to how installed Apache Mono on Plesk:

1. Add Mono repository to yum:


wget -O /etc/yum.repos.d/mono.repo http://ftp.novell.com/pub/mono/download-stable/rhel-4-i386/mono.repo

2. Install Mono:

yum install mono-core mono-web mono-data mod_mono xsp

Note: You may get missing dependency notice.

Missing Dependency: libexif.so.9 is needed by package libgdiplus0-1.9-1.rhel4.novell.i386 (mono)
Error: Missing Dependency: libexif.so.9 is needed by package libgdiplus0-1.9-1.rhel4.novell.i386 (mono)
You could try using --skip-broken to work around the problem
You could try running: package-cleanup --problems
package-cleanup --dupes
rpm -Va --nofiles --nodigest

If that’ the case you’ll need to take a different route. Remove that /etc/yum.repos.d/mono.repo and create a new file centos-5-extras.repo and put the folloging content in there:

[centos-5-extras]
name=CentOS-5 - $basearch - Extras
#baseurl=http://mirror.centos.org/centos/5/extras/$basearch/
mirrorlist=http://mirrorlist.centos.org/?release=5&arch=$basearch&repo=extras
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5
enabled=0

Then run this:

yum --enablerepo=centos-5-extras install mono-core mono-web mono-data mod_mono xsp

This should do it. Just make sure mono is installed afterwards:

mono -V

3. Check & configure mod_mono
Install process should’ve created the configuration file for mod_mono. Check if it exists here: /etc/httpd/conf.d/mod_mono.conf
If it doesn’t exist there create it and put the following contents:

# mod_mono.conf
<IfModule !mod_mono.c>
LoadModule mono_module /usr/lib/httpd/modules/mod_mono.so
AddType application/x-asp-net .aspx
AddType application/x-asp-net .asmx
AddType application/x-asp-net .ashx
AddType application/x-asp-net .asax
AddType application/x-asp-net .ascx
AddType application/x-asp-net .soap
AddType application/x-asp-net .rem
AddType application/x-asp-net .axd
AddType application/x-asp-net .cs
AddType application/x-asp-net .config
AddType application/x-asp-net .Config
AddType application/x-asp-net .dll
DirectoryIndex index.aspx
DirectoryIndex Default.aspx
DirectoryIndex default.aspx
</IfModule>

4. Restart Apache

/usr/local/psa/admin/sbin/websrvmng -r

This did it for me. Along the way I encountered few issues, looking closesly at /var/log/httpd/error_log helped to find solutions(in my case it was missing mono-web package, that brings in System.Web library I guess).

How to move MySQL storage to RamFS or TmpFS partition

Whether moving all MySQL storage to a tmpfs helps with speeding it up or not is questionable but I needed to do for some testing purposes, so this is a short overview of how I did that hopefully will be useful:

First mount tmpfs to a folder:

sudo mkdir /var/ramfs
sudo mount -t ramfs -o size=1G ramfs /var/ramfs/

Here I mounted ramfs to /var/ramfs. I am using ramfs in oppose to tmpfs mainly because:

  • ramfs grows dynamically(tmpfs doens’t)
  • ramfs doesn’t use swap(while tmpfs does)

RAM-backed file system is mounted, so now I need to populate it with MySQL files for processing.
To do that I will need to stop mysql, copy it’s database files over to ramfs, adjust AppArmor and MySQL settings and start mysql server again. Here is the chain of commands to do that:

Copying files:

sudo /etc/init.d/mysql stop
sudo cp -R /var/lib/mysql /var/ramfs/
sudo chown -R mysql:mysql /var/ramfs/mysql

Tweaking MySQL config:

sudo cp /etc/mysql/my.cnf /etc/mysql/original-my.cnf
sudo vim /etc/mysql/my.cnf

Find line with ‘datadir‘ definition(it will look something like datadir = /var/lib/mysql) and change it to

datadir = /var/ramfs/mysql

Next step is to tune apparmor settings:

sudo vim /etc/apparmor.d/usr.sbin.mysqld

Add the following few lines just before the closing curly braces:


/var/ramfs/mysql/ r,
/var/ramfs/mysql/*.pid rw,
/var/ramfs/mysql/** rwk,

Looks like we’re done with settings, let’s see if it will work:


sudo /etc/init.d/apparmor restart
sudo /etc/init.d/mysql start

If mysql daemon starts(double check /var/log/mysql.err for any errors) and you can connect to it, mostlikely now we’re running fully off of a RAM device. To double check it, run this from mysql client:

mysql> show variables where Variable_name = 'datadir' \G
*************************** 1. row ***************************
Variable_name: datadir
Value: /var/ramfs/mysql/
1 row in set (0.00 sec)

That’s pretty much it :)

Apache 2 and HTTP Authentication with PAM

There are 2 ways(at least that I know of) to get Apache 2 to use PAM for http auth:

  • Old mod_auth_pam, which I believe is not developed anymore and also posses some security risks
  • Newer mod_authnz_external and pwauth

This little write up shows how to get Apache and PAM going on Ubuntu using the mod_authnz_external.
To get started, let’s install some packages:

sudo apt-get install libapache2-mod-authnz-external pwauth
sudo apt-get install libapache2-mod-authz-unixgroup
sudo a2enmod authnz_external authz_unixgroup

Edit config file for the Virtual Host you’d like to get them PAM-based HTTP Authentication going, such that it contains the following clause:


<IfModule mod_authnz_external.c>
AddExternalAuth pwauth /usr/sbin/pwauth
SetExternalAuthMethod pwauth pipe
</IfModule>

And the final bit of configuration goes to your Directory definition inside of vhost block:

<Directory /var/www/yourlocation>
AuthType Basic
AuthName "Restricted Area"
AuthBasicProvider external
AuthExternal pwauth
Require user john

# some other configuration statements
</Directory>

This will allow user john to access the resource.

Now if you also want to have PAM authentication by users group you’ll need to make few extra steps. Missing bit of puzzle here is called ‘unixgroup’ script and for some reason it is not in Ubuntu’s pwauth package where it ought to be. You will need to grab it from here and copy it over to /usr/sbin/unixgroup and make it executable. Here is a quick snippet to do that:


wget "http://pwauth.googlecode.com/files/pwauth-2.3.9.tar.gz"
tar xzvf ./pwauth-2.3.9.tar.gz
sudo cp pwauth-2.3.9/unixgroup /usr/sbin/
sudo chmod a+x /usr/sbin/unixgroup

Once that’s done, you’ll need to few more lines to you Virtual Host config, so it will look something like this:

<IfModule mod_authnz_external.c>
AddExternalAuth pwauth /usr/sbin/pwauth
SetExternalAuthMethod pwauth pipe
AddExternalGroup unixgroup /usr/sbin/unixgroup
SetExternalGroupMethod unixgroup environment

</IfModule>

<Directory /var/www/yourlocation>
AuthType Basic
AuthName "Restricted Area"
AuthBasicProvider external
AuthExternal pwauth
GroupExternal unixgroup
Require user john# some other configuration statements
</Directory>

Hopefully this is helpful to someone besides myself :) Let me know if you got stock somewhere along the way.

Installing Mercurial 1.5, 1.6 or 1.7 on Ubuntu Lucid Lynx 10.04

Lycid Lynx’ repository by default set to install Mercurial 1.4.x, which is great, but as of today that’s already 3 major releases behind :( To get yourself on the latest and greatest version of Mercurial there are 2 options: build from the source or just add mercurial-releases repository like this:

sudo add-apt-repository ppa:mercurial-ppa/releases
sudo apt-get update
sudo apt-get install mercurial

Make sure it did the right thing:

$ hg --version
Mercurial Distributed SCM (version 1.6)

Copyright (C) 2005-2010 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Another thing that will go nicely with Mercurial 1.6.x or 1.7.x is up-to-date TortoiseHg. To install TortoiseHg on Ubuntu 10.04 you’ll need to run the following:

sudo add-apt-repository ppa:tortoisehg-ppa/releases
sudo apt-get update
sudo apt-get install tortoisehg-nautilus

XDebug doesn’t show local variables in Komodo, NetBeans or Eclipse PDT

Lovely surprise that came sometime after upgrading Lucid Lynx(and thus to PHP 5.3.2) – Xdebug doesn’t show local variables. Fortunately that’s a known issue that is fixed in Xdebug 2.1.0RC, but since it’s not released, it is not yet in Ubuntu’s repository. This should be pretty easy to fix though :) let’s try this:

sudo apt-get install php5-dev php-pear

That’ll get us ready for manual xdebug building. Then we need to get the sources. Gotta tell you, Xdebug guys are pretty awesome – they put together this nice little tool that helps you find and build the right version: http://xdebug.org/find-binary.php

It requires output of phpinfo() or in ‘php -i’ command and based on that gives you set by step instructions.

For standard Lucid Lynx install it boils down to the following commands:

wget http://xdebug.org/files/xdebug-2.1.0RC1.tgz
tar -xvzf xdebug-2.1.0RC1.tgz
cd xdebug-2.1.0RC1
phpize
./configure
make
sudo cp modules/xdebug.so /usr/lib/php5/20090626
sudo /etc/init.d/apache2 restart

Before restarting the server though(last command) you may want to double check the the following line exists in either /etc/php5/apache2/php.ini or in /etc/php5/apache2/conf.d/xdebug.ini

zend_extension = /usr/lib/php5/20090626/xdebug.so