After a month without X11, I've finally have X11 working on my new iBook. The installer that apple has online DOES NOT WORK. Apple mentions:
10.4 customers can install X11 by using the Tiger DVD installer disk.
In fact, you must use the installer from the Tiger disk, as the downloadable installer doesn't work, reporting that you already have a newer version of the package.
Unfortunatelly the installer is kinda hidden so here are the steps:
Insert your Tiger (10.4) install disk.
Scroll Down to the Optional Packages
Then choose X11 under packages