Archive | open source RSS feed for this section

OpenBD and MongoDB: Can’t get easier then this

I’m currently working on a project that uses MongoDB (yes, that’s the awesome NoSQL database everyone talks about). Now, my favorites language of choice (CFML) and especially my favorite open source CFML engine OpenBD, has support for MongoDB built in which makes it a snap to use for any ColdFusion developer.

MongoDB’s query syntax is quite different then the SQL you might got used to. A good overview how to find (select) records with MongoDB can be found at SQL to Mongo Mapping Chart. In short, the MongoDB find() syntax is close to the “chained” commands you might know from the likes of JQuery. Important to remember is that 99% of the time everything you pass to MongoDB is a JSON structure and has a JSON like syntax.

Since OpenBD is using the Java Driver and we apply a CFML tag for it, the query syntax differs slightly from the one you would enter in a mongo shell. Please find below some examples.

Find all records

<cfset myarray = MongoCollectionfind(
   datasource="mongosource",
   collection="users",
   query={}
)>

The above MongoCollectionfind() will return an Array with all records of the Collection and all fields.

Return only certain fields

It’s a common practice not to return all fields as in a “SELECT column1, column2 FROM…”. With MongoCollectionfind() you would do something like:

<cfset myarray = MongoCollectionfind(
datasource="mongosource",
collection="users",
query={},
fields={email:true,firstname:true}
)>

Again the MongoCollectionfind() would return an Array with all records, but this time only with the columns “email” and “firstname”. Note: The unique value of “_id” is always returned!

Find a specific user

Of course, returning all records is 99% of the time not wanted. MongoDB has a very sophisticated approach to finding records. As you can see from the SQL to MongoDB map. Ok, now let’s find user by his eMail address.

<cfset myarray = MongoCollectionfind(
datasource="mongosource",
collection="users",
query=({email:"nitai@openbd.com"}),
fields={email:true,firstname:true}
)>

To find the user with his eMail address AND his first name you would do:

<cfset myarray = MongoCollectionfind(
datasource="mongosource",
collection="users",
query=({email:"nitai@openbd.com",firstname:"nitai"}),
fields={email:true,firstname:true}
)>

The same find() as above but this time with OR would be:

<cfset myarray = MongoCollectionfind(
datasource="mongosource",
collection="users",
query=( { $or : [ { email : "nitai@openbd.com" } , { first_name : "nitai" } ] } ),
fields={email:true,firstname:true}
)>

As you can see form the examples above, working with OpenBD and the MongoDB syntax is straight forward and easy. Hope this helps.

Comments { 0 }

htaccess being hacked and detecting malware on Linux

Unfortunately, I’ve had to learn a thing or two about server intrusion, .htaccess being hacked, backdoors and more in the last days. Well, one could say it is a good education… In any case, one of the major issues I was confronted with, is that on one PHP based server (with WordPress) the .htaccess file was manipulated. The hack wasn’t obvious, since it was targeted to any user coming to the site from a search engine and then being redirected to some (random) russian site.

Now, removing the .htaccess file, editing it or changing the file permission did not help, as the file itself was being reset to the hacked version every 30 minutes or so. It was obvious that someone was able to upload a backdoor to the server and calling the file remotely. The question was now, how to find the infected file or uploaded on in order to close the hole?

Luckily, the people at R-FX Network made the malware script available (under the GPL open source license), that lets you scan your server for any malware and infected files. It uses many different scans to find any malware. I especially like, that users can send their infected files to them and they then update the scanner engine. Very effective method to keep your server secure.

In my case, the malware script quickly found the file in question, which was then quarantined. Since then, the WordPress blog worked again. Needless to say, just running this script wont secure your server. You need to have a firewall in place, update your server regularly, read your logs and so on.

If you want to read more about the difference way of hack and how to protect the server I’ve found this blog post from Unmask Parasites very valuable. Also, they have a video of Matt Cutts talking about malware.

 

Comments { 0 }

Your PHP installation appears to be missing the MySQL extension which is required by WordPress and some other PHP strangeness

I just went trough some issues with a PHP5 installation on some of our WordPress servers. To spare you of the details, why I had to do that, I simply post the solutions here (strange enough searching for these issues online gives you “trillions” of results, but no solution…).

1. Removing PHP5 and re-installing

When you are in need to remove and re-install PHP5, you should use the “–purge” parameter like; “apt-get –purge remove php5″. Now, while this seams to work and when you re-install PHP5 it strangely gives you feedback of “Not replacing deleted config file /etc/php5/apache2/php.ini” during re-installation.

In short, the re-installation does NOT install a php.ini file. While this might not be a issue for some, it was in my case. Luckily, there are some default php.ini files that one can copy. Depending on your server they might be in different places (mine were at /usr/share/php5). So, all there was to do is to copy the file with: “cp /usr/share/php5/php.ini-production /etc/php5/apache2/php.ini”.

2. Your PHP installation appears to be missing the MySQL extension which is required by WordPress

While the above might not be major issue, the message I received from our WordPress installation of “Your PHP installation appears to be missing the MySQL extension which is required by WordPress” was worrying. Nothing that I tried, re-installing PHP5, enabling modules, checking Apache and MySQL solved it. Also, searching for this online, did not help at all, since almost every entry simply tells you to re-install php5.

Ok, so how did I get it solved? After hours of hours the solution is too simple. All I had to do, was to explicitly, enable the extension in the php.ini file with:

extension=mysql.so
extension=mysqli.so

Saved the file, restarted Apache and finally I got the MySQL parameters in the PHP Info page and all blogs worked again. One would think that the apt-get installation routine would do this automatically since I was using “apt-get install php5-mysql”, but alas as I found out the hard way, this is not so.

Hope this helps someone out there.

Comments { 0 }

Why Coldfusion / CFML has its place and is worth to learn it

I actually never indulge in conversations why one programming language is better then another, because what is right to you, does not automatically mean, it is right for someone else. So, for me ColdFusion, or as we call the language itself – CFML, works very well.

Nevertheless, in this post I like to clear up some confusion that has been around. I can see where the confusion comes from as Coldfusion has gone from Allaire to Macromedia and is now “in the hands” of Adobe.

In any case, here are some reasons why CFML is still worth for you to learn.

  • CFML is open source and yes you can use it for free. I guess, many people are put off by the fact that Coldfusion has had a big price tag on it in the past. True, given the nature of PHP, Java, Phyton, Ruby, etc. being free, there was actually no reason to shell out your hard earned buck for some application server and on top of it even learn the language. Thanks to the short sighted business decisions of Macromedia and now Adobe or shall we say with the greedy money making mentality of its management, Coldfusion has been faced with a drainage of developers.
    Fortunately, this has all changed with the advent of OpenBD – the first real open source CFML server, followed shortly by Railo, another popular open source CFML server. In other words, to learn CFML and to deploy your applications, is now free and free to be.
  • Write less code. Compared to PHP, Java, C++, even Ruby and Python – CFML allows you to write the same program with much much fewer lines of code. Why would you want spend your precious time writing more code when you can do it for less? As a matter of fact, you can write your application in CFML so efficiently, that the same application written by you alone would probably need a team with Java, Ruby, etc. This is a proven fact.
  • Well designed. The CFML language is well designed and many required functions already exists for you to use. There is no need to write a wrapper for a email sending function. I mean, you don’t even need a framework, to achieve a simple tag like “<cfmail…>”. There is no obnoxious, framework to learn or write functions for this. Compare this to Java, PHP, Ruby, etc. you are very well off with CFML. (again all without a framework)
  • Build web applications fast. Due to the nature of writing less code with CFML and with the built in function, you will be writing your next web application in weeks, instead of months. On top of that, you will have a full scalable enterprise model on your hand to scale when your startup takes off. Heard about the stories of PHP web apps, that had to be converted to xyz language just to scope with the traffic. Again, if you would deploy your web application with CFML and OpenBD, you can deploy on any Java application server (Tomcat, JBoss, Websphere, etc.), connect to any database (MongoDB, H2, Oracle, MySQL, MS SQL, DB2, you name it…) and have your cluster, load balancing, caching setup done.

If you are in for writing less code and building your next web application the fast way, then I simply urge you to give CFML a try. I’m certain that you will get your project done in half the time then in another language. There is simply nothing to loose for you!

 

Comments { 0 }

CFML and Cannot run program “chmod”: java.io.IOException: error=24, Too many open files

Migrating one of my customers the other day, bought up an ugly error when I had to create 2000 directories on one go. The error was:

Cannot run program "chmod": java.io.IOException: error=24, Too many open files

While, “too many open files” usually means one can raise the limit of open files under Linux (check out ulimit -a) it unfortunately did not help in this situation. I even rebooted the whole server and made sure that no other service was running, except Java that is. Still no success.

I then looked at my code in the CFML (Coldfusion) template. In order to create the directories I used a simply:

<cfinclude action="create" directory="..." mode="775">

Normal code, right? Well, as it turns out, I simply had to remove the “mode” part in order to overcome this error. Not sure, why this caused a “too many open files” error, but it worked in my situation. I can only imagine that the server tried to put all 2000 directories into memory and then write them in one go (I have a high value for the open files limit set and 12GB RAM).

In any case, hope this helps someone out here.

Comments { 0 }

How to search & replace in MySQL

Thinking of a way to quickly search & replace data in a MySQL column got me to this solution:

update table
set
column = replace(column,'thistext','thattext')

With this method I was quickly able to change millions of records today. Quick and easy.

Comments { 0 }

Ubuntu 10.04 LTS server always selecting older kernel despite updates

I hit a really strange issue for some time now with one of my Ubuntu 10.04 LTS servers where, despite doing recent kernel updates (the latest is 2.6.32-33) it always booted into the kernel 2.6.32-28. No matter what I did (update-grub, etc.), the server was sticked to 2.6.32-28.

After searching and reading a lot of posts and wiki pages I still couldn’t find a solution for it (most blogs and wiki pages talk about compiling a new kernel or installing a new one, but none talked about selecting the proper kernel or fixing it manually).

But since I had some other server, that booted into the correct kernel issue, I luckily had some config files to compare. The one that I was after is the “menu.lst” which is a GRUB file and is located at “/boot/grub”. Looking at the menu.lst from the working server and comparing it to the “not working” one, revealed that the “non working” one had UUID’s declared for each server while the working one not UUID but “root (hd0,0)” and hard coded root paths of “/dev/sda3″.

Working config:
[code]]czoxODE6XCJ0aXRsZSAgIFVidW50dSAxMC4wNC4yIExUUywga2VybmVsIDIuNi4zMi0zMy1zZXJ2ZXINCnJvb3QgICAgKGhkMCwwKQ17WyYqJl19Cmtlcm5lbCAgL3ZtbGludXotMi42LjMyLTMzLXNlcnZlciByb290PS9kZXYvc2RhMyBybyBxdWlldCBzcGxhc2gNCmluaXRyZCAgL3tbJiomXX1pbml0cmQuaW1nLTIuNi4zMi0zMy1zZXJ2ZXINCnF1aWV0DQpcIjt7WyYqJl19[[/code]

Not-Working config:
[code]]czoyMzg6XCJ0aXRsZSAgIFVidW50dSAxMC4wNC4yIExUUywga2VybmVsIDIuNi4zMi0zMy1zZXJ2ZXINCnV1aWQJOTgyM2Q1NDAtNmJ7WyYqJl19ZjAtNDY3YS04NjQwLTM5ZDMzYzc1NDRmYg0Ka2VybmVsCS92bWxpbnV6LTIuNi4zMi0zMy1zZXJ2ZXIgcm9vdD1VVUlEPWQ5ZTc5YXtbJiomXX1kOS02ZDUzLTRjYjUtODVkNi1lOWYxZWVhNzEyZjIgcm8gcXVpZXQgc3BsYXNoIA0KaW5pdHJkCS9pbml0cmQuaW1nLTIuNi4zMi0ze1smKiZdfTMtc2VydmVyDQpxdWlldA0KXCI7e1smKiZdfQ==[[/code]

As you can see from the this, the UUID and the root are quite different. I can’t actually explain why this configuration took place on this particular server. This is even more surprising since both servers are being “kept in sync” (with updates and such).

In any case, the resolution (for me at least) was to copy certain parameters over to the “not-working” menu.lst. Those were, the root path (/devsda3), the “root (hd0,0)”, the “groot=(hd0,0)” plus the “kopt=root=/dev/sda3 ro” lines.

It took a long time to fix this issue and and I hope this helps someone else.

Comments { 0 }

svn: warning: cannot set LC_CTYPE locale

Somehow with the recent Ubuntu 10.04 LTS updates or maybe with a subversion update, I received some errors message of the type “locale…”. To be more precise the errors are;

svn: warning: cannot set LC_CTYPE locale
svn: warning: environment variable LC_CTYPE is UTF-8
svn: warning: please check that your locale name is correct

While all SVN commands still worked, it was something I didn’t tackle with immediately. But today, I set out to fix it. Well, it only took a minute or so :)

So, if you want to fix this all you have to do is to set the “LC_ALL” variable manually. To make it permanent just edit the file “/etc/environment” and add the line:

LC_ALL=C

Save the file and exit the editor. In order for it to apply you have to logout of the current shell session. The next time you log in, the issue with SVN will be gone.

 

Comments { 0 }

Seamless server access from MacOS X to Ubuntu with SSH public keys

When you access a server over SSH you usually get asked for a password that you trustfully type into the terminal window. But doing so is insecure for many different reasons (I’m sure there are many people who wrote about this before and describe it better then I ever could). So, what is a better way to log into your server then? The best way so far is a method called “public key authentication”.

So, since we want to add security to our belt, we can simply use this technique for our SSH access as well. On MacOS X it is actually very easy to setup.

First up, you need to create your own keys. Doing so, is straight forward, all you have to do is to open up a Terminal window and type “ssh-keygen”. This will then prompt you some questions, where to put the keys (use default) and for the passphrase (I would suggest you use a good password). In the end, it will save your keys (your private one and a public one) to your .ssh directory.

Now what you got your public key, all there is left to do is to copy your public key to your server. In case you have root access to your server, it is simply a manner of doing it with “scp”, like:

“scp ~/.ssh/id_rsa.pub root@{yourserverdomain}:.ssh/authorized_keys”

This will copy your public key to the “authorized_keys” of the server.

Once done, you can now simply log into your server with ssh root@{yourserverdomain} without the need to enter a password since your server and you exchange keys for authentication.

Troubleshooting

When you copy your key to server you might get a error that the file “authorized_keys” is not found. If so, then simply create the file on the server and issue the copy command again.

 

Comments { 0 }

Installing memcached on Ubuntu for wordpress and phpbb

As an application maintainer you always look for the best performance in your application and website. At one point in your quest for the best performance you will undoubtedly trip over memcached.

In short memcached is (quote); Free & open source, high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load. Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.

That said, installing is a no brainer as well. On Ubuntu you simply need to do the following:

[code]]czo1NjpcImFwdC1nZXQgaW5zdGFsbCBtZW1jYWNoZQ0KYXB0LWdldCBpbnN0YWxsIHBocDUtbWVtY2FjaGVkXCI7e1smKiZdfQ==[[/code]

That’s it. Your system takes care of the rest and you will have your first memcached server up and running. Of course, the final step will be to restart apache in order for php to pick up the changes.

Now, memcached alone is of no good use, if your code/application can’t work with it. Thus here I’ll outline 2 examples.

WordPress: Memcached with the W3C total cache plugin

First off, if you aren’t using the awesome W3C Total Cache plugin you should now run install the plugin immediately (just search for w3 cache in the plugin section of the wordpress administration). Even if you are not using memcached it will boost the performance of your WordPress site manifold.

Now, to enable memcached for your WordPress site is as simple as selecting the memcached option for the cache. with the plugin you can even select what you want to place into the memcached cache. Quit slick.

 

Configuring phpBB to use memcached

Actually it took some time to figure this out, since the setting were not so apparent, so I’m hopping this helps others also. phpBB by default used the local disk for caching. This can be chanced in the config.php file in phpBB folder. Open it and ADD or change the following lines:

[code]]czoyNzM6XCIkYWNtX3R5cGUgPSBcJ21lbWNhY2hlXCc7DQpAZGVmaW5lKFwnUEhQQkJfQUNNX01FTUNBQ0hFX0hPU1RcJywgXCdsb2NhbGhvc3tbJiomXX10XCcpOyAvLyBNZW1jYWNoZSBzZXJ2ZXIgaG9zdG5hbWUNCkBkZWZpbmUoXCdQSFBCQl9BQ01fTUVNQ0FDSEVfUE9SVFwnLCAxMTIxMSk7e1smKiZdfSAvLyBNZW1jYWNoZSBzZXJ2ZXIgcG9ydA0KQGRlZmluZShcJ1BIUEJCX0FDTV9NRU1DQUNIRV9DT01QUkVTU1wnLCBmYWxzZSk7IC8vIHtbJiomXX1Db21wcmVzcyBzdG9yZWQgZGF0YQ0KJGxvYWRfZXh0ZW5zaW9ucyA9IFwnbWVtY2FjaGVcJztcIjt7WyYqJl19[[/code]

Especially the last line with “load_extensions” is important. Save the file and restart apache. Now phpBB will use the memcached server(s). Alone on a board with 800 users I have seen a massive speed improvement.

That’s it. Next up is to get all my CFML apps to work with memcached :-)

Comments { 0 }