SQL Paging with Oracle, MySQL and MS SQL

I have to admit, I love databases. To hold data in a central place and be able to hook up any application from wherever you are to it, is just convenient to say the least. But there are moments I wish that vendors would “follow” a standard. I guess, when it came down to paging, every vendor thought his way of doing it is the best. No wonder, paging is the one SQL syntax that differs the most. Thus I dedicate, this post to paging and outlining the difference, meaning the solution to achieve it with Oracle, MySQL and MS SQL.

Paging with Oracle

Oracle knows of the internal column “ROWNUM”. Thus with “rownum” we can page trough the records, but have to do this with subselects. Look at the example;

SELECT rn, columns
   FROM (
  SELECT ROWNUM AS rn, columns
    FROM (
      SELECT columns
      FROM table
      WHERE id = 1
      AND something = 2
      ORDER BY columns
      )
    WHERE ROWNUM <= 10
  )
WHERE rn > 0

In this example we select the first 10 records, if you want to select the next 10 records then you will need to increase the values for “ROWNUM” with 20 and for “rn” to 10.

Paging with MySQL

There is one thing I like MySQL, that is the “LIMIT” clause. Not only can you limit how many records to retrieve, but can also add a offset to the limit clause. Thus we can do very elegant paging with “LIMIT offset, number”. Again, we limit the query below to return us 10 records:

SELECT columns
FROM table
WHERE id = 1
AND something = 2
ORDER BY columns
LIMIT 0, 10

Now, this will return the first 10 records. So, what if we want to do a paging here? Actually, quite easy, all we have to do is to increase the offset by one. It is important to know, that the offset start with a “0″ (think of it as the FIRST page), thus if we want to retrieve the next 10 records we use the limit with “LIMIT 1, 10″. This will give us the next 10 records. Elegant, no?

Paging with MS SQL

Now paging with MS SQL took me some time to figure out, since some said that MS SQL know of “rownumber” as well, but honestly, I couldn’t get anything to work with it. So the most easy method to do paging in MS SQL is to use the TOP syntax combined with a NOT IN subselect. Makes sense, right? Here is the example;

SELECT TOP 10 columns
FROM table
WHERE xxxx
AND id NOT IN (
  SELECT TOP 0 id
  FROM table
  WHERE xxxx
  )

As in the Oracle example above, if you would like to retrieve the next 10 records you would have to increase the TOP values by 10 records, meaning the FIRST TOP would have a value of 20 and the inner select TOP a value of 10.

So here you have it. How paging works for Oracle, MySQL and MS SQL.

Comments { 23,575 }

FFmpeg — here we go again

My two other posts on FFmpeg entitled “Installing ffmpeg on CentOS 5” and “SELinux with ffmpeg” already explained in deep how to get FFmpeg up and running.

Unfortunately, today one of our servers just reported a plain:

ffmpeg: error while loading shared libraries: libfaad.so.0: cannot open shared object file: No such file or directory

when trying to run any ffmpeg command. Now, what was that and foremost why? Especially, since it worked 2 days ago. I guess, “someone” must have done some update. In any case, it was a good lesson to reinstall ffmpeg and bring all libraries up to date.

So, since my last installation, some things must have obviously been changed behind the scenes, because when I tried to configure ffmpeg with the same commands I run into another error which was:

libx264 version must be &gt;= 0.78

Even thought I installed x264 from the latest GIT repository it still showed me the above error! Looking around the Videolan.org website I saw that the nightly snapshots differ in size a lot. So, either their nightly script is broken or something else is going on. In any case, I went with the x264-snapshot-20091031-2245 one.

Configured and installed it. Then did a “ldconfig -v” (in order to see that it really took the latest one) and went on with the ffmpeg configuration (I took the latest code from SVN (Revision 20525)) and low and behold, everything compiled and installed without problem.

Important: Update your libraries after the installation of ffmpeg again with “ldconfig”! Else you will get the “ffmpeg: error while loading shared libraries: ….” error again.

Comments { 626 }

Tomcat and directories

I run into a nasty issue the other day with “alike” directories in my web application.

That is, I had a copy of a “WEB-INF” directory called “WEB-INF–”. While my web application did not bother with copy of the WEB-INF directory, Tomcat threw all kinds of errors in the catalina.out log file. Also, I had some issues with Tomcat restarting and strange Java error messages, like:

Java.lang.NullPointerException
com.nary.util.Localization.convertCharSetToCharEncoding(Unknown Source)
com.naryx.tagfusion.cfm.engine.cfEngine.getDefaultEncoding(Unknown Source)
com.naryx.tagfusion.cfm.engine.cfFormData.<init>(Unknown Source)
com.naryx.tagfusion.cfm.engine.cfFormData.<init>(Unknown Source)
com.naryx.tagfusion.cfm.engine.variableStore.<init>(Unknown Source)
com.naryx.tagfusion.cfm.engine.cfSession.<init>(Unknown Source)
com.naryx.tagfusion.cfm.engine.cfEngine.service(Unknown Source)
com.naryx.tagfusion.cfm.cfServlet.service(Unknown Source)
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

After browsing the log files, I saw that Tomcat tried to read the directory and threw a bunch of critical errors while reading those copied directories. I guess, Tomcat got totally confused and tried to overload libraries.

Lesson learned; Never have any copy of the WEB-INF folder within the web application again.

Comments { 806 }

Upgrade to most recent ImageMagick version on CentOS 5.x

My favorite choice for running a Linux Server is CentOS, since it is based on the RedHat distribution you can rest assured you will get a top notch enterprise offering and stability. As with all things “enterprise” the priority is on stability and security and not on the latest code releases. This works 99% of the time, but sometimes you still need some update.

In the case of ImageMagick, CentOS comes with version 6.2.8, it was a bug that was fixed with PSD conversion and thus I needed to get the latest version installed. So, here are the steps to install ImageMagick 6.5.7 on CentOS 5.x. Mind you, that you will loose the internal patch upgrading from yum, but all you need to install to the next version is just to follow these steps again.

Uninstall current version
Uninstall the current version with:

yum erase ImageMagick*

This will uninstall ImageMagick 6.2.8 and if you have any other versions installed, like the devel one.

Install the needed dependencies
ImageMagick depends on a couple of additional libraries to convert to different formats. Let us just make sure, that they are all installed with:

yum install tcl-devel libpng-devel libjpeg-devel ghostscript-devel bzip2-devel freetype-devel libtiff-devel

Download and extract latest ImageMagick version
You can always get the latest ImageMagick version directly from their website. Code below will download and extract the file.

wget ftp://ftp.imagemagick.org/pub/ImageMagick/ImageMagick.tar.gz
tar xcvf ImageMagick.tar.gz
cd ImageMagick-6.5.7-5

Configure and make ImageMagick
With the below configure command we are configuring ImageMagick with the most needed options. Feel free to adjust it to your needs. As always issue a “–help” to see all the available options.

configure --prefix=/usr/local --with-bzlib=yes --with-fontconfig=yes --with-freetype=yes --with-gslib=yes --with-gvc=yes --with-jpeg=yes --with-jp2=yes --with-png=yes --with-tiff=yes

Wait until configure has finished. At the end you will see all the enable options. When you think all went well issue:
make

Now is a good time to make yourself some coffee or continue coding your next big killer application because make will take some time to finish. When it’s done, issue:
make install

That’s it! You are done. Wasn’t so bad, was it? Check with:

convert --version

That ImageMagick is properly installed and that you got the current version up and running. If all went well you should see something similar to this:
Version: ImageMagick 6.5.7-5 2009-11-08 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2009 ImageMagick Studio LLC

Comments { 1 }

Oracle backup with RMAN

Deploying an Oracle database is one thing, but doing a good recovery backup plan is another matter. Good for us, Oracle comes with its own tool for doing backups called “RMAN”.

Sure, one can do backup with the Enterprise Manager, but with it you will need to shut down the database in order to do a backup. Not exactly, what is needed with a production database that is online 24/7 and can’t afford a shutdown. In order to do a “online backup” one needs to run a script for RMAN.

RMAN uses the same login settings as your SQLPLUS login and thus it should be rather easy to do so. In our case we have also configured a backup catalog within Oracle with its own tablespace. Once done a backup script can be run each night with crontab.

Here is how to setup the backup catalog:
[code]]czoyNzA6XCINCnNxbHBsdXMgc3lzDQpTUUwmZ3Q7IGNyZWF0ZSB1c2VyIHJtYW4gaWRlbnRpZmllZCBieSBybWFuOw0KU1FMJmd0OyB7WyYqJl19YWx0ZXIgdXNlciBybWFuIGRlZmF1bHQgdGFibGVzcGFjZSB0b29scyB0ZW1wb3JhcnkgdGFibGVzcGFjZSB0ZW1wOw0KU1FMJmd0O3tbJiomXX0gYWx0ZXIgdXNlciBybWFuIHF1b3RhIHVubGltaXRlZCBvbiB0b29sczsNClNRTCZndDsgZ3JhbnQgY29ubmVjdCwgcmVzb3VyY2Use1smKiZdfSByZWNvdmVyeV9jYXRhbG9nX293bmVyIHRvIHJtYW47DQpTUUwmZ3Q7IGV4aXQ7DQpcIjt7WyYqJl19[[/code]

in case you don’t have a tablespace called “tools” you need to create it with:
[code]]czoxNTQ6XCINCmNyZWF0ZSB0YWJsZXNwYWNlIHRvb2xzDQogIGxvZ2dpbmcNCiAgZGF0YWZpbGUgXCcvZGJmMS90b29scy5kYmZcJyANCntbJiomXX0gIHNpemUgMzJtIA0KICBhdXRvZXh0ZW5kIG9uIA0KICBuZXh0IDMybSBtYXhzaXplIDIwNDhtDQogIGV4dGVudCBtYW5hZ2VtZW50e1smKiZdfSBsb2NhbDsNClwiO3tbJiomXX0=[[/code]

Now you can log in to RMAN create the catalog schema with:
[code]]czo3OTpcIg0Kcm1hbiBjYXRhbG9nIHJtYW4vcm1hbg0KUk1BTj4gY3JlYXRlIGNhdGFsb2cgdGFibGVzcGFjZSB0b29sczsNClJNQU57WyYqJl19PiBleGl0Ow0KXCI7e1smKiZdfQ==[[/code]

Then register the database with;
[code]]czo3NzpcIg0Kcm1hbiBjYXRhbG9nIHJtYW4vcm1hbiB0YXJnZXQgc3lzL3Bhc3N3b3JkQE9SQ0wNClJNQU4+IHJlZ2lzdGVyIGRhdGF7WyYqJl19YmFzZTsNClwiO3tbJiomXX0=[[/code]

Make sure you connect to the database that you need to backup!

I then created a script that will keep my backup for 2 days. I also have a script that will compress the backup files and move them to another server. Here is my backup script:
[code]]czo3NjQ6XCINCkNPTkZJR1VSRSBDT05UUk9MRklMRSBBVVRPQkFDS1VQIE9OOw0KQ09ORklHVVJFIENPTlRST0xGSUxFIEFVVE9CQUN7WyYqJl19S1VQIEZPUk1BVCBGT1IgREVWSUNFIFRZUEUgRElTSyBUTw0KXCcvb3B0L29yYWNsZS9iYWNrdXAvYXV0b2JhY2t1cF9jb250cm9sX2Z7WyYqJl19aWxlJUZcJzsNCkNPTkZJR1VSRSBSRVRFTlRJT04gUE9MSUNZIFRPIFJFQ09WRVJZIFdJTkRPVyBPRiAyIERBWVM7DQpydW4gew0KQUx7WyYqJl19TE9DQVRFIENIQU5ORUwgUk1BTl9CQUNLX0NIMDEgVFlQRSBESVNLOw0KQ1JPU1NDSEVDSyBCQUNLVVA7DQpCQUNLVVAgQVMgQ09NUHtbJiomXX1SRVNTRUQgQkFDS1VQU0VUIERBVEFCQVNFIEZPUk1BVA0KXCcvb3B0L29yYWNsZS9iYWNrdXAvZGF0YWJhc2VmaWxlc18lZF8ldV8lc3tbJiomXX1fJVRcJzsNCnNxbCBcJ0FMVEVSIFNZU1RFTSBBUkNISVZFIExPRyBDVVJSRU5UXCc7DQpCQUNLVVAgQVMgQ09NUFJFU1NFRCBCQUNLVVBTe1smKiZdfUVUIEFSQ0hJVkVMT0cgQUxMIEZPUk1BVA0KXCcvb3B0L29yYWNsZS9iYWNrdXAvYXJjaGl2ZWxvZ3NfJWRfJXVfJXNfJVRcJyBERUxFVHtbJiomXX1FIElOUFVUOw0KQkFDS1VQIEFTIENPTVBSRVNTRUQgQkFDS1VQU0VUIENVUlJFTlQgQ09OVFJPTEZJTEUgRk9STUFUDQpcJy9vcHQvb3tbJiomXX1yYWNsZS9iYWNrdXAvY29udHJvbGZpbGVfJWRfJXVfJXNfJVRcJzsNCkNST1NTQ0hFQ0sgQkFDS1VQOw0KREVMRVRFIE5PUFJPTVBUIHtbJiomXX1PQlNPTEVURTsNCkRFTEVURSBOT1BST01QVCBFWFBJUkVEIEJBQ0tVUDsNClJFTEVBU0UgQ0hBTk5FTCBSTUFOX0JBQ0tfQ0gwMTsNe1smKiZdfQp9DQpleGl0Ow0KXCI7e1smKiZdfQ==[[/code]

This will backup your control file, archive log and database. It will also compress the backup and will remove the old files.

In my case, I saved the above script as RMA_backup.txt and have a RMAN_script.sh with the following inside:
[code]]czo2OTpcIg0KIyEvYmluL2Jhc2gNCnJtYW4gdGFyZ2V0IHN5cy9wYXNzd29yZEBPUkNMIEAvb3B0L3JtYW5fYmFja3VwLnR4dDsNClwie1smKiZdfTt7WyYqJl19[[/code]

Troubleshooting
I have seen that sometimes we got errors running the above script with:
[code]]czoyMDE6XCINClJNQU4tMDYyMDc6IFdBUk5JTkc6IDUgb2JqZWN0cyBjb3VsZCBub3QgYmUgZGVsZXRlZCBmb3IgRElTSyBjaGFubmV7WyYqJl19bChzKSBkdWUNClJNQU4tMDYyMDg6ICAgICAgICAgIHRvIG1pc21hdGNoZWQgc3RhdHVzLiAgVXNlIENST1NTQ0hFQ0sgY29tbWFuZHtbJiomXX0gdG8gZml4IHN0YXR1cw0KUk1BTi0wNjIxMDogTGlzdCBvZiBNaXNtYXRjaGVkIG9iamVjdHMNClwiO3tbJiomXX0=[[/code]

With RMAN you can check your backup with the following commands:
[code]]czoxMDA6XCINCkNST1NTQ0hFQ0sgYmFja3VwIG9mIGRhdGFiYXNlOyAgDQpDUk9TU0NIRUNLIGJhY2t1cCBvZiBjb250cm9sZmlsZTt7WyYqJl19DQpDUk9TU0NIRUNLIGFyY2hpdmVsb2cgYWxsOyANClwiO3tbJiomXX0=[[/code]

If all goes well, you should not get any errors running these commands. Still we had the above errors, even thought I issued the command to delete obsolte control files with;
[code]]czo1NTpcIg0KREVMRVRFIE5PUFJPTVBUIE9CU09MRVRFIFJFQ09WRVJZIFdJTkRPVyBPRiAyIERBWVM7DQpcIjt7WyYqJl19[[/code]

The key to success was when I used the “FORCE” command as;
[code]]czo2MTpcIg0KREVMRVRFIEZPUkNFIE5PUFJPTVBUIE9CU09MRVRFIFJFQ09WRVJZIFdJTkRPVyBPRiAyIERBWVM7DQpcIjt7WyYqJl19[[/code]

Hope this helps anybody. There is a good reference for backups over at the Oracle FAQ.

Comments { 561 }

Why GPL

Matt Mullenweg of the famous WordPress application talks in the below video about GPL and why WordPress uses it. I recently had some discussions with some people why Razuna, our Open Source Media Management (Digital Asset Management) Solution, uses the AGPL and why we chose it. It think Matt brings it right to the point and I agree with him totally :-)

Comments { 581 }

One of the reasons why the internet and open source works

Social theorist Jonathan Zittrain gave a very entertaining and enlightening talk about the random kindness on the Internet and how we can all benefit from this worldwide “kindness”.

Listening to his talk, you will also understand why open source works and why people spend many many hours contributing to projects without getting any financial rewards for it. It is the kindness and the good within the human nature to do “good” and to feel “belonging” to a greater good. I guess, open source projects, touch on this very common need of the humans.

I for myself, feel very satisfied to see so many people taking up our Open Source Digital Asset Management and contribute to it in any way possible. At this point, thank you all :-)

Enjoy the talk.

Comments { 921 }

SELinux and FFMpeg

SELinux is a good thing, but it also requires you to watch out on some libraries. Sometimes, something just fails, because SELinux does not allow it to run. This so happens when you have SELinux installed and want to run FFMpeg.

I just run into this when I tried to run FFMpeg and got this error message:

“error while loading shared libraries: /usr/lib/libavcodec.so.52: cannot restore segment prot after reloc: Permission denied”

In order to run FFMpeg without problems I had to add the remove the restrictions for SELinux with:

chcon -t textrel_shlib_t ‘/usr/lib/libavutil.so.49.15.0′
chcon -t textrel_shlib_t ‘/usr/lib/libavcodec.so.52.20.0′
chcon -t textrel_shlib_t ‘/usr/lib/libavformat.so.52.31.0′
chcon -t textrel_shlib_t ‘/usr/lib/libavformat.so.52.31.0′
chcon -t textrel_shlib_t ‘/usr/lib/libswscale.so.0.7.1′

Comments { 675 }

Griffin Express Card/34 and Snow Leopard

Like many others I installed Snow Leopard on my Mac and like many I’m most of the time very happy with it. But there are some things that still don’t work as expected. Especially troublesome is when working Hardware can not be used anymore.

Unfortunately, I had one of those experiences today. As a owner of the Late 2008 MacBook Pro series, I don’t have a Smart Card Reader built in, thus I bought myself a Griffin eSata Express Card, which I use to read/write to Smart Cards. But as I found out today, after wanting to use it for the first time since upgrading to Snow Leopard, the Smart Card could not be read. As a fact, the Card Reader is not even recognized by the System itself.

Apparently, Apple disables some drivers during the installation as stated in the Technote called “About incompatible Software”. One of those drivers is the one from Sillicon Image which enables the Smart Card Reader. After some searching I found the same driver again over at the Sillicon Image download page and installed the latest driver (from 2007) again. Low and behold the card works again.

Not sure, how long this will work, but for now this will hopefully help anybody else.

Update:
After some more searching I found that updated Snow Leopard ExpressCard drivers are available from FirmTek. So far, I can report successful installation working with this driver as well.

Comments { 796 }

Awesome informative video

If you are moving on the web and are surrounded by many little gadgets or simply want to be informed where the society is heading you own it to yourself to watch the Fall Edition of the famous “Shift Happens” series. This one is called “Did you know 4.0″.

Comments { 492 }