TheNitai

Migrating one of my customers the other day, bought up an ugly error when I had to create 2000 directories on one go. The error was:

Cannot run program "chmod": java.io.IOException: error=24, Too many open files

While, “too many open files” usually means one can raise the limit of open files under Linux (check out ulimit -a) it unfortunately did not help in this situation. I even rebooted the whole server and made sure that no other service was running, except Java that is. Still no success.

I then looked at my code in the CFML (Coldfusion) template. In order to create the directories I used a simply:

< cfinclude action="create" directory="..." mode="775" >

Normal code, right? Well, as it turns out, I simply had to remove the “mode” part in order to overcome this error. Not sure, why this caused a “too many open files” error, but it worked in my situation. I can only imagine that the server tried to put all 2000 directories into memory and then write them in one go (I have a high value for the open files limit set and 12GB RAM).

In any case, hope this helps someone out here.

I hit a really strange issue for some time now with one of my Ubuntu 10.04 LTS servers where, despite doing recent kernel updates (the latest is 2.6.32-33) it always booted into the kernel 2.6.32-28. No matter what I did (update-grub, etc.), the server was sticked to 2.6.32-28.

After searching and reading a lot of posts and wiki pages I still couldn’t find a solution for it (most blogs and wiki pages talk about compiling a new kernel or installing a new one, but none talked about selecting the proper kernel or fixing it manually).

But since I had some other server, that booted into the correct kernel issue, I luckily had some config files to compare. The one that I was after is the “menu.lst” which is a GRUB file and is located at “/boot/grub”. Looking at the menu.lst from the working server and comparing it to the “not working” one, revealed that the “non working” one had UUID’s declared for each server while the working one not UUID but “root (hd0,0)” and hard coded root paths of “/dev/sda3″.

Working config:

title   Ubuntu 10.04.2 LTS, kernel 2.6.32-33-server
root    (hd0,0)
kernel  /vmlinuz-2.6.32-33-server root=/dev/sda3 ro quiet splash
initrd  /initrd.img-2.6.32-33-server
quiet

Not-Working config:

title   Ubuntu 10.04.2 LTS, kernel 2.6.32-33-server
uuid  9823d540-6bf0-467a-8640-39d33c7544fb
kernel  /vmlinuz-2.6.32-33-server root=UUID=d9e79ad9-6d53-4cb5-85d6-e9f1eea712f2 ro quiet splash 
initrd  /initrd.img-2.6.32-33-server
quiet

As you can see from the this, the UUID and the root are quite different. I can’t actually explain why this configuration took place on this particular server. This is even more surprising since both servers are being “kept in sync” (with updates and such).

In any case, the resolution (for me at least) was to copy certain parameters over to the “not-working” menu.lst. Those were, the root path (/devsda3), the “root (hd0,0)”, the “groot=(hd0,0)” plus the “kopt=root=/dev/sda3 ro” lines.

It took a long time to fix this issue and and I hope this helps someone else.

Somehow with the recent Ubuntu 10.04 LTS updates or maybe with a subversion update, I received some errors message of the type “locale…”. To be more precise the errors are;

svn: warning: cannot set LC_CTYPE locale
svn: warning: environment variable LC_CTYPE is UTF-8
svn: warning: please check that your locale name is correct

So, if you want to fix this all you have to do is to set the “LC_ALL” variable manually. To make it permanent just edit the file “/etc/environment” and add the line:

LC_ALL=C

Save the file and exit the editor. In order for it to apply you have to logout of the current shell session. The next time you log in, the issue with SVN will be gone.

When you access a server over SSH you usually get asked for a password that you trustfully type into the terminal window. But doing so is insecure for many different reasons (I’m sure there are many people who wrote about this before and describe it better then I ever could). So, what is a better way to log into your server then? The best way so far is a method called “public key authentication”.

So, since we want to add security to our belt, we can simply use this technique for our SSH access as well. On MacOS X it is actually very easy to setup.

First up, you need to create your own keys. Doing so, is straight forward, all you have to do is to open up a Terminal window and type “ssh-keygen”. This will then prompt you some questions, where to put the keys (use default) and for the passphrase (I would suggest you use a good password). In the end, it will save your keys (your private one and a public one) to your .ssh directory.

Now what you got your public key, all there is left to do is to copy your public key to your server. In case you have root access to your server, it is simply a manner of doing it with “scp”, like:

“scp ~/.ssh/id_rsa.pub root@{yourserverdomain}:.ssh/authorized_keys”

This will copy your public key to the “authorized_keys” of the server.

Once done, you can now simply log into your server with ssh root@{yourserverdomain} without the need to enter a password since your server and you exchange keys for authentication.

Troubleshooting

When you copy your key to server you might get a error that the file “authorized_keys” is not found. If so, then simply create the file on the server and issue the copy command again.

If you happen to see the message “Failed to open the relay log…” in your MySQL error log file (sometimes it is good to look into it once in a while) then you either have your replication setup incorrectly or you forgot to remove the salve information in the master.

If the later is the case all you have to do is to login to the MySQL server and issue:

Stop Slave;
Reset Slave;

There is no need to restart MySQL. But if you want to see that it actually works now, then restart MySQL and look into the mysql.log file. You will now see that the error message is gone.

As an application maintainer you always look for the best performance in your application and website. At one point in your quest for the best performance you will undoubtedly trip over memcached.

In short memcached is (quote); Free & open source, high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load. Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.

That said, installing is a no brainer as well. On Ubuntu you simply need to do the following:

apt-get install memcache
apt-get install php5-memcached

That’s it. Your system takes care of the rest and you will have your first memcached server up and running. Of course, the final step will be to restart apache in order for php to pick up the changes.

Now, memcached alone is of no good use, if your code/application can’t work with it. Thus here I’ll outline 2 examples.

WordPress: Memcached with the W3C total cache plugin

First off, if you aren’t using the awesome W3C Total Cache plugin you should now run install the plugin immediately (just search for w3 cache in the plugin section of the wordpress administration). Even if you are not using memcached it will boost the performance of your WordPress site manifold.

Now, to enable memcached for your WordPress site is as simple as selecting the memcached option for the cache. with the plugin you can even select what you want to place into the memcached cache. Quit slick.

Configuring phpBB to use memcached

Actually it took some time to figure this out, since the setting were not so apparent, so I’m hopping this helps others also. phpBB by default used the local disk for caching. This can be chanced in the config.php file in phpBB folder. Open it and ADD or change the following lines:

$acm_type = 'memcache';
@define('PHPBB_ACM_MEMCACHE_HOST', 'localhost'); // Memcache server hostname
@define('PHPBB_ACM_MEMCACHE_PORT', 11211); // Memcache server port
@define('PHPBB_ACM_MEMCACHE_COMPRESS', false); // Compress stored data
$load_extensions = 'memcache';

Especially the last line with “load_extensions” is important. Save the file and restart apache. Now phpBB will use the memcached server(s). Alone on a board with 800 users I have seen a massive speed improvement.

That’s it. Next up is to get all my CFML apps to work with memcached

I remember when I installed Linux the first time (many many moons ago) and it was all cryptic for me. All that starred at me, was a black screen with some strange symbols and a pointer blinking.

So, this is Linux, I thought and tried to get my way around it. To be honest, it took a couple of re-installs and some learnings to come to the level I’m at now. Nowadays, all of my applications run on Linux servers (my favorite one is Ubuntu server) and I have to say that I’m more then happy how Linux performs.

Actually, my next step is to adopt Linux (Ubuntu) on my laptop, but to move to Linux on my desktop, I really need to have a application like Aperture of Adobe Lightroom. Apart from that, I think Linux on the desktop has a big chance to succeed. Especially, Ubuntu 11 with Unity will probably make this move apparent for a lot of users.

In any case, if you run Linux on your servers or thinking of migrating to Linux, you own it to yourself to watch the below anniversary video and head over to the dedicated “20 years of Linux” site.

On a Windows machine I came upon the other day (forcefully and not intentional:-) ) I saw that the default engine was “search-results.com”. This URL was called whenever the user entered a keyword in the URL bar (you do know that Google Chrome and Firefox 4 will automatically search for the words you enter in the URL field, don’t you?). Key was, that I wanted to change this URL.

Easy right? Well, as it turned out, it took some small effort to find the right value.

First off, I entered the “about:config” and searched for “browser.search.defaultenginename”. Double clicked on it and changed it to “Google”. Immediately, I restarted Firefox, but unfortunately the search still was being directed to the other site.

Ok, hitting again the “about:config” and this time searching for “search” only revealed that there was another setting called “browser.search.defaultengine” which also pointed to another search engine. Changing it to “Google” also did not help.

Finally, after looking for some more, I find out that the config value “keyword.url” is the one setting that needs to be changed. Low and behold, Firefox even has a nice page on this topic (you just need to know what you need to be looking for, right…).

In short changing the value back to the default value (or any other you want) “http://www.google.com/search?ie=UTF-8&oe=utf-8&q=” fixed it.

Hope this helps.

Oracle recently released the latest MySQL 5.5.x versions. Despite of having tones of fixes and supposedly running much faster and (finally) making InnoDB the default storage engine, it is always good to keep up to date with latest releases.

But, as in the past, MaxOS X users always have to battle with getting it run correctly on their platform of choice. With this release it is no difference. This time, we are hit by permission issues and wrong path settings that prohibits MySQL to start at all and also to make it automatically startup during boot time.

Fixing startup

After installing the preference pane, you should be able to simply click the button to start/stop the MySQL server. Point is that “it should”, but it wont. This is due to a wrong path setting in the startup file. In order to make it work you have to edit a file. Jump into your favorites tool (no, it is not iTunes), but the Terminal application and enter:

nano /usr/local/mysql/support-files/mysql.server

(Nano is a editor in your shell, you can also use “vi” or “emacs”, but I guess “nano” is easy to use)

Then hit “ctrl + W” (for search) and enter “basedir=”, then hit Enter”. This will search for the string entered. Once found, change it to:

basedir=/usr/local/mysql

Once changed, hit “ctrl + W” again and search for “mysqld_pid_file_path=$datadir/`hostname`.pid” and change it to:

mysqld_pid_file_path=$datadir/`/bin/hostname`.pid

After these two changes simply press “ctrl + X” and enter “y” and then enter. This will save the changes you just made and will close the editor.

Fixing Startupitem

During reboot you might have seen the error message:

“Insecure Startup Item disabled. /Library/StartupItems/MySQLCOM has not been started because it does not have the proper security settings.”

This is because the StartUpItem has been installed with the improper permission settings. This can simply be solved with the following commands you have to enter in the terminal:

chown -R root:wheel /Library/StartupItems/MySQLCOM

That’s it. After next reboot the MySQL server will startup as expected and you will also be able to start/stop the server from the preference panel.

For some of my Ubuntu Servers (10.04) the Cron jobs did not work or only sporadically executed. I’ve asked around, but none of my friends could explain why this is so. This happened on physical as also on virtual servers. Thus I set out today to finally figure out what caused this. To mention it upfront, I did not find the cause of it, but I got my Cron jobs working again. (and learned a lot along the way)

System Crontab and User Crontab

In the past I always set cron jobs in the /etc/crontab file. This is also the way I’ve used cron in the last couple of years. By switching to Ubuntu server 9.x this worked, but somehow since 10.04 they only worked sporadically. As explained, I don’t know why this is so, but moving those cron jobs into the user (in my case the root account) with “crontab -e” they started working again.

The only difference is that one does not need to pass the user the jobs has to run in with the command, since we edit the user crontab file with “crontab -e”, but everyting else works now as expected.

Getting cron to log

Under Ubuntu logging of any cron action is disabled by default. Since, I wanted to see if my jobs are running and execute successfully I had to enable it. Enabling logging of cron is done in the system default files with:

vim /etc/rsyslog.d/50-default.conf

Uncommenting the ‘#’ before the line that says cron and it will enable logging for cron:

cron.* /var/log/cron.log

and save the file. Now simply reload or restart the rsyslog utility with:

service rsyslog restart

Note: In my case, I had to create the file “/var/log/cron.log” manually in order for cron to write the log.

Enable or preventing cron to send eMails

I guess by default, cron will send eMails on errors or completed actions to the user. In my case, I got a lof of eMails with completed actions and more from cron shortly after. Controlling sending of eMails is done by the global variable ” MAILTO=”" “. One can also direct the output to null for disabling sending of emails.

To disable cron from sending for all jobs:

Add MAILTO=”" to the top of cour crontab file.

To disable sending of eMails for individual jobs:

0 1 5 10 * /path/to/script.sh &> /dev/null