Xen 3.2 on CentOS 5.2 x86_64 / RHEL 5.2 x86_64

Posted At : July 7, 2008 3:02 PM | Posted By : Ruslan Sivak
Related Categories: Xen
Since my drivers are not signed, you will need to temporarily disable signature checking for yum.

# vi /etc/yum.conf
Make sure it says gpgcheck=0

Download the file and untar it.

# tar xvzf xen-3.2.0-CentOS5.2.tar.gz
xen-3.2.0-CentOS5.2/
xen-3.2.0-CentOS5.2/xen-3.2.0-0xs.x86_64.rpm
xen-3.2.0-CentOS5.2/xen-libs-3.2.0-0xs.x86_64.rpm
xen-3.2.0-CentOS5.2/xen-devel-3.2.0-0xs.x86_64.rpm

# cd xen-3.2.0-CentOS5.2

Install the RPMs.

#yum install -y xen-3.2.0-0xs.x86_64.rpm xen-libs-3.2.0-0xs.x86_64.rpm xen-devel-3.2.0-0xs.x86_64.rpm

It will download a bunch of dependencies and install them. Among them will be a new kernel - kernel-xen.

# vi /etc/yum.conf
Set gpgcheck back to the old value: gpgcheck=1 if it was originally 1.

The only problem at this point is that yum for some reason doesn't update the boot entry, so we have to do this manually.

# vi /boot/grub/menu.lst

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00
# initrd /initrd-version.img
#boot=/dev/sda
default=1
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-92.1.6.el5xen)
root (hd0,0)
kernel /xen.gz-2.6.18-92.1.6.el5
module /vmlinuz-2.6.18-92.1.6.el5xen ro root=/dev/VolGroup00/LogVol00
module /initrd-2.6.18-92.1.6.el5xen.img
title CentOS (2.6.18-92.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-92.el5 ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.18-92.el5.img

We want to update the CentOS (2.6.18-53.1.21.el5xen) entry, and set it to use the proper kernel - xen.gz-3.2

This is what my updated menu.lst looks like - yours may differ. I have also set the default to 0, so that it boots my xen kernel automatically.

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00
# initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-92.1.6.el5xen)
root (hd0,0)
kernel /xen.gz-3.2
module /vmlinuz-2.6.18-92.1.6.el5xen ro root=/dev/VolGroup00/LogVol00
module /initrd-2.6.18-92.1.6.el5xen.img
title CentOS (2.6.18-92.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-92.el5 ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.18-92.el5.img

Once you reboot you should automatically boot into the xen kernel and be able to create virtual machines.

Comments (74) | Print | Send | Download | del.icio.us | Linking Blogs
Comments
[Add Comment]
Mark's Gravatar Nice tutorial. Can you create some rpm's for xen 3.2.1? =)
# Posted By Mark | 7/10/08 3:15 AM
Brian's Gravatar Have you had any luck with the virt-manager tools? After installing 3.2, I can no longer create new virtual machines using those methods.
# Posted By Brian | 7/15/08 6:27 PM
Paul's Gravatar Brian,
Did you ever get your virt-manager to work after upgrading to 3.2? I am experiencing the same issue.
# Posted By Paul | 7/16/08 9:35 PM
Boris's Gravatar virt-install works fine.
I can manage Domains via xm , as i did on Solaris Nevada since build 75a
# Posted By Boris | 7/17/08 11:42 AM
Sascha's Gravatar Hello,

this tutorial is really pretty good, but i have some addtional stuff. Thank you Ruslan!

@Brian + Paul:
This is a known bug and i've fixed that with the "xen-timer-mode.patch"
There is a yum repository with fixed packages on: http://www.gitco.de/repo/xen3.2

@Mark:
Yes, you can use the yum repository on: http://www.gitco.de/repo/xen3.2.1
I've just built the packages for this release. Please have a look on this:

======================================================================
[root@slab005 /]# cat /etc/yum.repos.d/XEN.repo
[xen]
name=CentOS-$releasever - XEN
baseurl=http://www.gitco.de/repo/xen3.2.1
gpgcheck=0
[root@slab005 /]#

[root@slab005 /]# yum install xen xen-libs xen-devel
Loading "fastestmirror" plugin
Loading mirror speeds from cached hostfile
* xen: www.gitco.de
* extras: ftp.plusline.de
* updates: ftp.plusline.de
* base: ftp.plusline.de
* addons: ftp.plusline.de
Setting up Install Process
Parsing package install arguments
Resolving Dependencies
--> Running transaction check
---> Package xen-devel.x86_64 0:3.2.1-0xs set to be updated
---> Package xen-libs.i686 0:3.0.3-64.el5_2.1 set to be updated
---> Package xen-libs.x86_64 0:3.2.1-0xs set to be updated
---> Package xen.x86_64 0:3.2.1-0xs set to be updated
--> Processing Dependency: python-virtinst for package: xen
---> Package xen-devel.i686 0:3.0.3-64.el5_2.1 set to be updated
--> Running transaction check
---> Package python-virtinst.noarch 0:0.300.2-8.el5 set to be updated
--> Processing Dependency: libvirt-python >= 0.2.1 for package: python-virtinst
--> Processing Dependency: virt-viewer for package: python-virtinst
--> Running transaction check
---> Package virt-viewer.x86_64 0:0.0.2-2.el5 set to be updated
--> Processing Dependency: libvirt.so.0()(64bit) for package: virt-viewer
---> Package libvirt-python.x86_64 0:0.3.3-7.el5 set to be updated
--> Running transaction check
---> Package libvirt.x86_64 0:0.3.3-7.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================
Package Arch Version Repository Size
=============================================================================
Installing:
xen x86_64 3.2.1-0xs xen 4.4 M
xen-devel x86_64 3.2.1-0xs xen 226 k
xen-devel i686 3.0.3-64.el5_2.1 updates 215 k
Installing for dependencies:
libvirt x86_64 0.3.3-7.el5 base 915 k
libvirt-python x86_64 0.3.3-7.el5 base 74 k
python-virtinst noarch 0.300.2-8.el5 base 161 k
virt-viewer x86_64 0.0.2-2.el5 base 25 k
xen-libs i686 3.0.3-64.el5_2.1 updates 141 k
xen-libs x86_64 3.2.1-0xs xen 365 k

Transaction Summary
=============================================================================
Install 9 Package(s)
Update 0 Package(s)
Remove 0 Package(s)

Total download size: 6.4 M
Is this ok [y/N]: y
Downloading Packages:
(1/8): xen-3.2.1-0xs.x86_ 100% |=========================| 4.4 MB 00:07
(2/8): xen-devel-3.0.3-64 100% |=========================| 215 kB 00:00
(3/8): libvirt-python-0.3 100% |=========================| 74 kB 00:00
(4/8): xen-libs-3.2.1-0xs 100% |=========================| 365 kB 00:00
(5/8): libvirt-0.3.3-7.el 100% |=========================| 915 kB 00:01
(6/8): virt-viewer-0.0.2- 100% |=========================| 25 kB 00:00
(7/8): python-virtinst-0. 100% |=========================| 161 kB 00:00
(8/8): xen-devel-3.2.1-0x 100% |=========================| 226 kB 00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing: xen-libs ######################### [1/9]
Installing: libvirt ######################### [2/9]
ln: creating symbolic link `/etc/libvirt/qemu/networks/autostart/default.xml' to `../default.xml': File exists
Installing: virt-viewer ######################### [3/9]
Installing: libvirt-python ######################### [4/9]
Installing: xen-libs ######################### [5/9]
Installing: python-virtinst ######################### [6/9]
Installing: xen-devel ######################### [7/9]
Installing: xen-devel ######################### [8/9]
Installing: xen ######################### [9/9]

Installed: xen.x86_64 0:3.2.1-0xs xen-devel.x86_64 0:3.2.1-0xs xen-devel.i686 0:3.0.3-64.el5_2.1
Dependency Installed: libvirt.x86_64 0:0.3.3-7.el5 libvirt-python.x86_64 0:0.3.3-7.el5 python-virtinst.noarch 0:0.300.2-8.el5 virt-viewer.x86_64 0:0.0.2-2.el5 xen-libs.i686 0:3.0.3-64.el5_2.1 xen-libs.x86_64 0:3.2.1-0xs
Complete!

[root@slab005 /]# xm dm | head -n5
__ __ _____ ____ _
\ \/ /___ _ __ |___ / |___ \ / |
\ // _ \ '_ \ |_ \ __) | | |
/ \ __/ | | | ___) | / __/ _| |
/_/\_\___|_| |_| |____(_)_____(_)_|
[root@slab005 /]#


==========================================================================



IMPORTANT: I can't ensure the stability of this packages. So feel free if you deploy that packages in a live environment!!!

Thanks!

Cheers Sascha
# Posted By Sascha | 7/17/08 11:57 AM
Paul's Gravatar Hey Man,

Thanks so much for updating me on this information so quickly. Thank you as well for sharing this with the internet world as well. It saved me a lot of time by using your method to update my CentOS with your RPM packages. I really appreciate the response.

Cheers,

Paul
# Posted By Paul | 7/17/08 1:39 PM
Daniel's Gravatar Hi Ruslan,

Great work! Did you compile the kernel with "PCI-device backend driver" for VT-d support?

Cheers,
Daniel
# Posted By Daniel | 7/17/08 5:15 PM
aneip's Gravatar Why when I trying to install it still show old version?

Installing:
xen i686 3.0.3-64.el5_2.1 updates 1.8 M
xen-devel i686 3.0.3-64.el5_2.1 updates 215 k

any idea??
# Posted By aneip | 7/25/08 9:51 AM
Boris's Gravatar Recompile Xen 3.2.1 for CentOS 5
http://lxer.com/module/newswire/view/106076/index....
# Posted By Boris | 7/27/08 11:03 AM
Timothy A. Geist's Gravatar I was wondering if there is a similar repo such as the one you mentioned but for a 32bit system?


Tim
# Posted By Timothy A. Geist | 7/28/08 4:26 PM
Bosis's Gravatar View Grant's McWilliams blog (it works for 64 and 32 bit system).
Follow his quite understandable instructions on 32-bit system. You 'll be done
in the same way. You don't need any third party repositories.
Start is always the same
# rpm -iv http://bits.xensource.com/oss-xen/release/3.2.0/ce... 3.2.0-0xs.centos5.src.rpm 2 > /dev/null
# cd /usr/src/redhat/SOURCES ;
wget http://bits.xensource.com/oss-xen/release/3.2.1/xe...
. . . . . . . . .
# cd /usr/src/redhat/RPMS/i386
You'll get i386 rpms under this folder
# Posted By Bosis | 7/29/08 2:15 PM
Bosis's Gravatar Grant's McWilliams blog

http://grantmcwilliams.com/index.php?option=com_co...
# Posted By Bosis | 7/29/08 2:26 PM
Timothy A. Geist's Gravatar Hi Boris

I followed it and it worked perfectly.

Thank you.
# Posted By Timothy A. Geist | 7/30/08 2:28 PM
Boris's Gravatar Ruslan,
Please, fix a typo in my name in two the most recent comments
been done by me.
My name is Boris.
Thanks.
# Posted By Boris | 7/30/08 2:48 PM
Sunny's Gravatar I have too : " Installing:
xen i686 3.0.3-64.el5_2.1 updates 1.8 M
xen-devel i686 3.0.3-64.el5_2.1 updates 215 k"

Looks like repo http://www.gitco.de/repo/xen3.2.1 wrong...
# Posted By Sunny | 8/19/08 11:57 AM
Sascha's Gravatar Hello Sunny,

the repo has only packages for arch x86_64. I have a running CentOS5.2 (x86_64) and can't found the packages as mentioned:

Take a look:

[root@slab005 ~]# uname -a
Linux slab005.labor.lan 2.6.18-92.1.6.el5xen #1 SMP Wed Jun 25 14:13:10 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

[root@slab005 ~]# yum list 'xen*'
Loading "fastestmirror" plugin
Loading mirror speeds from cached hostfile
* xen: www.gitco.de
* extras: mirror.silyus.net
* updates: mirror.silyus.net
* base: mirror.silyus.net
* addons: mirror.silyus.net
Installed Packages
xen.x86_64 3.2.1-0xs installed
xen-devel.x86_64 3.2.1-0xs installed
xen-libs.x86_64 3.2.1-0xs installed
Available Packages
xen-devel.i686 3.0.3-64.el5_2.1 updates
xen-devel.i386 3.0.3-64.el5_2.1 updates
xen-libs.i686 3.0.3-64.el5_2.1 updates
xen-libs.i386 3.0.3-64.el5_2.1 updates
[root@slab005 ~]#

Have you running RHEL or CentOS ?

Cheers,
Sascha
# Posted By Sascha | 8/19/08 1:24 PM
Mark's Gravatar How about Xen 3.3 now that its been released as stable? =P
# Posted By Mark | 8/24/08 2:10 PM
Simon's Gravatar Yeah I'll make sure to check on your repos for a /3.3.0/
# Posted By Simon | 8/25/08 8:40 AM
Boris's Gravatar For 3.2 following src rpm was needed:-
http://bits.xensource.com/oss-xen/release/3.2.0/ce...
and that's all
As soon as xen-3.3.0-0xs.centos5.src.rpm will be on the Net. I believe it won't be a problem
to upgrade Xen Hypervisor to 3.3 , but kernel will stay the same.
Cloning Xen 3.3 Mercurial Repository and building xen 3.3 from source requires "git" to be installed before running "make world" and takes more time then for 3.2.
However, works fine on CentOS 5.2.
# Posted By Boris | 8/26/08 4:23 AM
Sascha's Gravatar Hi all,

i've created the XEN-3.3.0 RPMs today and done a quick test.
On a HVM, it was possible to get a VNC session, but within the virt-manager,
no chance. The virt-manager gives "The console is currently unavailable".
Okay, after a short action of investigation, i had to build a new version of libvirt, too.

These are the packages:

libvirt-0.4.4-3.x86_64.rpm
libvirt-devel-0.4.4-3.x86_64.rpm
libvirt-python-0.4.4-3.x86_64.rpm
xen-3.3.0-2.x86_64.rpm
xen-devel-3.3.0-2.x86_64.rpm
xen-libs-3.3.0-2.x86_64.rpm

You can found them on: http://www.gitco.de/repo/xen3.3.0

And this is the repo file as described above-mentioned:

[root@slab005 /]# cat /etc/yum.repos.d/XEN.repo
[xen]
name=CentOS-$releasever - XEN
baseurl=http://www.gitco.de/repo/xen3.3.0
gpgcheck=0
[root@slab005 /]#

Have a lot of fun ...


IMPORTANT: I can't ensure the stability of this packages. So feel free if you deploy that packages in a live environment!!!

Thanks!

Cheers,
Sascha
# Posted By Sascha | 8/28/08 5:03 PM
Boris's Gravatar Xen 3.3 & libvirtlibvirt-0.4.4-3 rpms installed on CentOS 5.2
Solaris 10 HVM DomU and CentOS 5.2 PV DomU
created via virt-install with no problems.
Thank you
# Posted By Boris | 8/29/08 9:56 AM
Nathan's Gravatar Transaction Check Error:
file /etc/rc.d/init.d/libvirtd from install of libvirt-0.4.4-3 conflicts with file from package libvirt-0.3.3-7.el5
file /etc/sysconfig/libvirtd from install of libvirt-0.4.4-3 conflicts with file from package libvirt-0.3.3-7.el5
file /usr/share/man/man1/virsh.1.gz from install of libvirt-0.4.4-3 conflicts with file from package
libvirt-0.3.3-7.el5
# Posted By Nathan | 8/29/08 9:43 PM
boris's Gravatar [root@ServerRHL51 InstallXen3.3]# yum install xen-3.3.0-2.x86_64.rpm \
> xen-devel-3.3.0-2.x86_64.rpm xen-libs-3.3.0-2.x86_64.rpm \
> libvirt-0.4.4-3.x86_64.rpm libvirt-devel-0.4.4-3.x86_64.rpm \
> libvirt-python-0.4.4-3.x86_64.rpm
Loading "installonlyn" plugin
Setting up Install Process
Setting up repositories
xen 100% |=========================| 951 B 00:00
Reading repository metadata in from local files
primary.xml.gz 100% |=========================| 3.7 kB 00:00
################################################## 7/7
Parsing package install arguments
Examining xen-3.3.0-2.x86_64.rpm: xen - 3.3.0-2.x86_64
Marking xen-3.3.0-2.x86_64.rpm as an update to xen - 3.2.1-0xs.x86_64
Examining xen-devel-3.3.0-2.x86_64.rpm: xen-devel - 3.3.0-2.x86_64
Marking xen-devel-3.3.0-2.x86_64.rpm as an update to xen-devel - 3.2.1-0xs.x86_64
Examining xen-libs-3.3.0-2.x86_64.rpm: xen-libs - 3.3.0-2.x86_64
Marking xen-libs-3.3.0-2.x86_64.rpm as an update to xen-libs - 3.2.1-0xs.x86_64
Examining libvirt-0.4.4-3.x86_64.rpm: libvirt - 0.4.4-3.x86_64
Marking libvirt-0.4.4-3.x86_64.rpm as an update to libvirt - 0.3.3-7.el5.x86_64
Examining libvirt-devel-0.4.4-3.x86_64.rpm: libvirt-devel - 0.4.4-3.x86_64
Marking libvirt-devel-0.4.4-3.x86_64.rpm to be installed
Examining libvirt-python-0.4.4-3.x86_64.rpm: libvirt-python - 0.4.4-3.x86_64
Marking libvirt-python-0.4.4-3.x86_64.rpm as an update to libvirt-python - 0.3.3-7.el5.x86_64
Resolving Dependencies
--> Populating transaction set with selected packages. Please wait.
---> Package xen.x86_64 0:3.3.0-2 set to be updated
---> Package xen-devel.x86_64 0:3.3.0-2 set to be updated
---> Package xen-libs.x86_64 0:3.3.0-2 set to be updated
---> Package libvirt.x86_64 0:0.4.4-3 set to be updated
---> Package libvirt-devel.x86_64 0:0.4.4-3 set to be updated
---> Package libvirt-python.x86_64 0:0.4.4-3 set to be updated
--> Running transaction check
--> Processing Dependency: /usr/bin/qemu-img for package: libvirt
--> Processing Dependency: iscsi-initiator-utils for package: libvirt
--> Processing Dependency: cyrus-sasl-md5 for package: libvirt
--> Restarting Dependency Resolution with new changes.
--> Populating transaction set with selected packages. Please wait.
---> Downloading header for cyrus-sasl-md5 to pack into transaction set.
cyrus-sasl-md5-2.1.22-4.x 100% |=========================| 15 kB 00:00
---> Package cyrus-sasl-md5.x86_64 0:2.1.22-4 set to be updated
---> Downloading header for qemu to pack into transaction set.
qemu-0.9.0-4.x86_64.rpm 100% |=========================| 13 kB 00:00
---> Package qemu.x86_64 0:0.9.0-4 set to be updated
---> Downloading header for iscsi-initiator-utils to pack into transaction set.
iscsi-initiator-utils-6.2 100% |=========================| 12 kB 00:00
---> Package iscsi-initiator-utils.x86_64 0:6.2.0.868-0.7.el5 set to be updated
--> Running transaction check

Dependencies Resolved

=============================================================================
Package Arch Version Repository Size
=============================================================================
Installing:
libvirt-devel x86_64 0.4.4-3 libvirt-devel-0.4.4-3.x86_64.rpm 1.0 M
Updating:
libvirt x86_64 0.4.4-3 libvirt-0.4.4-3.x86_64.rpm 6.0 M
libvirt-python x86_64 0.4.4-3 libvirt-python-0.4.4-3.x86_64.rpm 339 k
xen x86_64 3.3.0-2 xen-3.3.0-2.x86_64.rpm 15 M
xen-devel x86_64 3.3.0-2 xen-devel-3.3.0-2.x86_64.rpm 843 k
xen-libs x86_64 3.3.0-2 xen-libs-3.3.0-2.x86_64.rpm 1.0 M
Installing for dependencies:
cyrus-sasl-md5 x86_64 2.1.22-4 base 46 k
iscsi-initiator-utils x86_64 6.2.0.868-0.7.el5 base 575 k
qemu x86_64 0.9.0-4 extras 4.5 M

Transaction Summary
=============================================================================
Install 4 Package(s)
Update 5 Package(s)
Remove 0 Package(s)

Total download size: 29 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): cyrus-sasl-md5-2.1 100% |=========================| 46 kB 00:00
(2/3): qemu-0.9.0-4.x86_6 100% |=========================| 4.5 MB 02:35
(3/3): iscsi-initiator-ut 100% |=========================| 575 kB 00:19
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Updating : xen-libs ####################### [ 1/14]
Installing: iscsi-initiator-utils ####################### [ 2/14]
Installing: qemu ####################### [ 3/14]
Installing: cyrus-sasl-md5 ####################### [ 4/14]
Updating : libvirt ####################### [ 5/14]
Updating : libvirt-python ####################### [ 6/14]
Updating : xen ####################### [ 7/14]
Updating : xen-devel ####################### [ 8/14]
Installing: libvirt-devel ####################### [ 9/14]
Cleanup : xen ####################### [10/14]
Cleanup : xen-devel ####################### [11/14]
Cleanup : xen-libs ####################### [12/14]
Cleanup : libvirt ####################### [13/14]
Cleanup : libvirt-python ####################### [14/14]

Installed: libvirt-devel.x86_64 0:0.4.4-3
Dependency Installed: cyrus-sasl-md5.x86_64 0:2.1.22-4 iscsi-initiator-utils.x86_64 0:6.2.0.868-0.7.el5 qemu.x86_64 0:0.9.0-4
Updated: libvirt.x86_64 0:0.4.4-3 libvirt-python.x86_64 0:0.4.4-3 xen.x86_64 0:3.3.0-2 xen-devel.x86_64 0:3.3.0-2 xen-libs.x86_64 0:3.3.0-2
Complete!
[root@ServerRHL51 InstallXen3.3]#
# Posted By boris | 8/30/08 3:18 AM
Sascha's Gravatar Hi,

@Nathan:
------------
I think you have installed libvirt.i386, too. For this package, i haven't got built a new RPM.

Take a look on this:

[root@slab005 /]# yum list 'libvirt*'
Loading "fastestmirror" plugin
Loading mirror speeds from cached hostfile
* xen: www.gitco.de
* extras: mirror.fraunhofer.de
* updates: mirror.fraunhofer.de
* base: mirror.fraunhofer.de
* addons: mirror.fraunhofer.de
xen 100% |=========================| 951 B 00:00
extras 100% |=========================| 1.1 kB 00:00
updates 100% |=========================| 951 B 00:00
base 100% |=========================| 1.1 kB 00:00
addons 100% |=========================| 951 B 00:00
Installed Packages
libvirt.x86_64 0.4.4-3 installed
libvirt-python.x86_64 0.4.4-3 installed
Available Packages
libvirt.i386 0.3.3-7.el5 base
libvirt-devel.x86_64 0.4.4-3 xen
libvirt-devel.i386 0.3.3-7.el5 base
[root@slab005 /]#

I only installed libvirt.x86_64 and libvirt-python.x86_64 and not the .i386 version!

@Boris:
---------
Thank you for your comments. I checked this for HVM on Windows Server 2003(x32)/2008(x64) and on WinXP SP3 (x32). And it works, too! A colleague had check this with the newest OpenSolaris (x64) as PVM. No problem at all!



Cheers,
Sascha
# Posted By Sascha | 8/30/08 5:36 AM
Boris's Gravatar I didn't mention Solaris Nevada (b94) PV DomU install only because it was done via
phyton profile referencing "pygrub" as bootloader ( not virt-install)
I also have to notice, that several time graphical console didn't pop up itself after guest's
OS reboot and i had to "vncviewer localhost:0" from other terminal session.
But, it's a minor issue.
# Posted By Boris | 8/30/08 6:03 AM
Mickaël's Gravatar Hi,

It appears to lack pv-grub.gz kernel in Xen 3.3.0 RPMs provided in http://www.gitco.de/repo/xen3.3.0

Is it normal ? How can I have it ? I would like to test this new feature.

THanks a lot
# Posted By Mickaël | 9/4/08 10:25 AM
Sascha's Gravatar Hello Mickaël,

i'll build new RPMs with stubdom on the next days.

Regards,
Sascha
# Posted By Sascha | 9/4/08 12:11 PM
Mark's Gravatar Does anyone have Xen 3.3 and Virt-Manager working fine on CentOS 5.2 x86_64. I basically cant get Virt manager to do much at all anymore =/. Especially create new domu's.
# Posted By Mark | 9/6/08 5:58 PM
Boris's Gravatar Virt-install works fine.
# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 5946 2 r----- 1004.7
RHEL52PV 2048 1 146.6
# xm start RHEL52PV
# vncviewer localhost:0
# Posted By Boris | 9/9/08 3:04 PM
Mark's Gravatar Um, xm commands arent the same as using virt-manager (gui app) for managing xen domu's.
# Posted By Mark | 9/9/08 3:19 PM
Sascha's Gravatar Hi all,

i've created new rpms with stubdom support.

[root@slab004 /]# rpm -q xen
xen-3.3.0-3
[root@slab004 /]# rpm -ql xen |egrep 'stub|pv'
/etc/xen/xmexample.hvm-stubdom
/etc/xen/xmexample.pv-grub
/usr/lib/xen/bin/stubdom-dm
/usr/lib/xen/boot/ioemu-stubdom.gz
/usr/lib/xen/boot/pv-grub-x86_32.gz
/usr/lib/xen/boot/pv-grub-x86_64.gz
[root@slab004 /]#

But i didn't test it :-(

I found a problem when i am check with tap:aio:

disk =[ 'tap:aio:/data/xen-images/vm01/hda.img,xvda,w', ]

Dom0 and DomU will freeze ... I'm on investigation ....

This works stable:
disk =[ 'file:/data/xen-images/vm01/hda.img,xvda,w', ]

@MARK: Have you also installed the new libvirt rpms ?


Cheers,
Sascha
# Posted By Sascha | 9/9/08 3:45 PM
Mickaël's Gravatar Hi,

I'm testing pv-grub with latests RPMs.

I can't get it work with VNC vfb (works fine without it).

Did someone try this ?

This is my VM conf file:

name = "ns1"
maxmem = 512
memory = 256
vcpus = 1
#vfb = [ 'type=vnc,vnclisten=0.0.0.0,vncdisplay=0' ]
disk = [ "drbd:ns1,xvda,w" ]
vif = [ "mac=00:16:3e:34:39:D5,bridge=xenbr0" ]
kernel = "/usr/lib/xen/boot/pv-grub-x86_32.gz"
extra = "(hd0,0)/grub/grub.conf"
# Posted By Mickaël | 9/10/08 3:19 AM
Mickaël's Gravatar There is a bug in Xen 3.3.0 that prevent using block-drbd device helper script (and maybe other helper scripts).

You have to apply this patch:

--- /usr/lib64/python2.4/site-packages/xen/xend/server/blkif.py.orig   2008-09-10 09:07:46.000000000 +0200
+++ /usr/lib64/python2.4/site-packages/xen/xend/server/blkif.py   2008-09-10 09:08:13.000000000 +0200
@@ -56,9 +56,9 @@
else:
try:
(typ, params) = string.split(uname, ':', 1)
- if typ not in ('phy', 'file', 'tap'):
+ if typ not in ('phy', 'file', 'tap', 'drbd'):
raise VmError(
- 'Block device must have "phy", "file" or "tap" '
+ 'Block device must have "phy", "file", "tap" or "drbd" '
'specified to type')
except ValueError:
raise VmError(


Cheers
# Posted By Mickaël | 9/11/08 4:41 AM
Sascha's Gravatar Hi Mickaël,

i will add this asap and release new rpms again.

Regards,
Sascha
# Posted By Sascha | 9/11/08 4:48 AM
Sascha's Gravatar Hi,

today we checked with pv-grub the first time.

With that, you can use vfb without any problems.
Our test was with SLES-10 SP2 x86_64 as PVM

name = "pvgrub-test"
maxmem = 512
memory = 256
vcpus = 1
vfb = [ 'type=vnc' ]
disk = [ 'file:/var/lib/xen/images/pvgrub-test/hda.img,hda,w' ]
vif = [ "bridge=xenbr0" ]
kernel = "/usr/lib/xen/boot/pv-grub-x86_64.gz"
extra = "(hd0,0)/grub/menu.lst"


Cheers,
Sascha
# Posted By Sascha | 9/12/08 12:48 PM
Mickaël's Gravatar I tried with 64bits VM, but I still can't get vfb work with pv-grub...

I can see grub menu, but vnc server crashes just after (netstat report that it's not listening on the specified port).

I can't any releavant message in any log (xm dmesg, dmesg, /var/log/messages, /var/log/xen/xend.log)

DOes someone have any suggests ?
# Posted By Mickaël | 9/15/08 4:23 AM
Mickaël's Gravatar I can't get stubdom working.

My configuraiton files:

# cat xp_pro_sp2_eng|grep -v "^#"|grep -v ^$
kernel = "/usr/lib/xen/boot/hvmloader"
builder='hvm'
memory = 256
name = "xp_pro_sp2_eng"
vif = [ 'type=ioemu, bridge=xenbr0' ]
device_model = '/usr/lib/xen/bin/stubdom-dm'
boot="dc"
sdl=0
opengl=0
vnc=1
vnclisten="0.0.0.0"
vncpasswd=''
stdvga=0

# cat xp_pro_sp2_eng-dm|grep -v "^#"|grep -v ^$
kernel = "/usr/lib/xen/boot/ioemu-stubdom.gz"
vif = [ '', 'type=ioemu, bridge=xenbr0' ]
disk = [ 'file:/tmp/xp_pro_sp2_eng.img,hda,w', 'file:/tmp/XP_PRO_SP2_ENG.iso,hdc:cdrom,r' ]

* xp_pro_sp2_eng starts paused
* xp_pro_sp2_eng-dm starts
* xp_pro_sp2_eng is unpaused
* xp_pro_sp2_eng-dm stops

then I have (strange state):

# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 15847 8 r----- 4598.5
xp_pro_sp2_eng 8 256 1 ------ 0.0

Does anybody get it work successfully ?
Does anybody knows what I'm doing wrong ?
# Posted By Mickaël | 9/15/08 10:05 AM
Boris's Gravatar Mickaël,
To get stubdom working on getco's Xen 3.3 port to CentOS 5.2
Create in /etc/xen files:-

[root@ServerCentOS52 xen]# cat WinXPhvm-stubdom
kernel = "/usr/lib/xen/boot/hvmloader"
builder='hvm'
memory = 512
name = "WinXPhvm"
vif = [ 'type=ioemu, bridge=eth0' ]
device_model = '/usr/lib/xen/bin/stubdom-dm'
boot="c"
sdl=0
opengl=0
vnc=0
# vnclisten="0.0.0.0"
# vncpasswd=''
stdvga=0

[root@ServerCentOS52 xen]# cat WinXPhvm-dm
kernel = "/usr/lib/xen/boot/ioemu-stubdom.gz"
vif = [ ' ', 'bridge=eth0']
vfb = [ 'type=sdl' ]
disk = ['file:/etc/xen/isos/winxp.iso,hdc:cdrom,r','file:/etc/xen/images/disk.img,hda,w','file:/tmp/test,hdb,r']

Per Xen 3.3 Stubdom Release Notes run:-
# mkdir -p /exports/usr/share/xen/qemu
# ln -s /usr/share/xen/qemu/keymaps /exports/usr/share/xen/qemu
# mkdir -p /exports/var/lib
# ln -s /var/lib/xen /exports/var/lib
# /usr/sbin/fs-backend &

Switch from console to another terminal session :-

# xm create WinXPhvm-stubdom
# Posted By Boris | 9/27/08 9:18 AM
Boris's Gravatar In original hvm profile both vnc and sdl are disabled.
Selection is made *-dm Domain's profile.
Just follow carefully release notes.
Install goes hardly alive with long outages , looks like it got stuck
I've set up mem_dom0 to 4GB on 8GB box, but
WinXPhvm-dm allocated only 32 MB, no matter of mem_dom0.
I would guess bad Xen Hypervisor tuning on Dom0 due
to limited experience from my side.
Otherwise, it's worse then usual HVM DomU
# Posted By Boris | 9/27/08 9:27 AM
Boris's Gravatar That's a final draft.

Getting stubdom working on Xen 3.3 port via http://www.gitco.de/repo/xen3.3.0 to CentOS 5.2

http://bderzhavets.wordpress.com/2008/09/27/gettin...
# Posted By Boris | 9/27/08 11:43 AM
Boris's Gravatar Ubuntu 8.04.1 & Debian Etch R2 in Stub Domains at Xen 3.3 CentOS 5.2 Dom0 (all 64-bit).
http://bderzhavets.wordpress.com/2008/10/02/ubuntu...
# Posted By Boris | 10/2/08 12:49 PM
Sascha's Gravatar Hi Boris,

really a great job. Thank you!

Cheers,
Sascha
# Posted By Sascha | 10/2/08 5:16 PM
Boris's Gravatar Attempt to load CentOS 5.2 PV DomU via pv-grub.
GRUB installed in /boot in DomU.

Profile :-
[root@ServerCentOS52 vm]# cat rhel52.pv-grub
name = "rhel-pvgrub"
memory = 2048
vcpus = 1
vfb = [ 'type=vnc' ]
disk = [ 'phy:/dev/sda10,xvda,w!' ]
vif = [ "bridge=eth0" ]
kernel = "/usr/lib/xen/boot/pv-grub-x86_64.gz"
extra = "(hd0,0)/grub/grub.conf"

[root@ServerCentOS52 ~]# xm console rhel-pvgrub
Thread "kbdfront close": pointer: 0x2080130010, stack: 0x15f0000
close fb: backend at /local/domain/0/backend/vfb/28/0
close kbd: backend at /local/domain/0/backend/vkbd/28/0
WARN: No handler for port 5 when unbinding
Booting 'CentOS PVG (2.6.18-92.el5xen)'

root (hd0,0)
Thread "kbdfront close" exited.
Filesystem type is ext2fs, partition type 0x83
kernel /vmlinuz-2.6.18-92.el5xen ro root=/dev/VolGroup00/LogVol00 rhgb quiet
................
Hangs

GRUB installed in / in DomU
Profile -
[root@ServerCentOS52 vm]# cat rhel52.pv-grub
name = "rhel-pvgrub"
memory = 2048
vcpus = 1
vfb = [ 'type=vnc' ]
disk = [ 'phy:/dev/sda10,xvda,w!' ]
vif = [ "bridge=eth0" ]
kernel = "/usr/lib/xen/boot/pv-grub-x86_64.gz"
extra = "(hd0,0)/boot/grub/grub.conf"

[root@ServerCentOS52 xen]# xm console rhel-pvgrub
Thread "kbdfront close": pointer: 0x2080130010, stack: 0x15f0000
close fb: backend at /local/domain/0/backend/vfb/24/0
close kbd: backend at /local/domain/0/backend/vkbd/24/0
WARN: No handler for port 5 when unbinding
Booting 'CentOS PVG (2.6.18-92.el5xen)'

root (hd0,0)
Thread "kbdfront close" exited.
Filesystem type is ext2fs, partition type 0x83
kernel /boot/vmlinuz-2.6.18-92.el5xen ro root=/dev/xvda1 rhgb quiet
.............
Hangs

However, for OpenSuse 11 PV DomU pv-grub works fine.
# Posted By Boris | 10/5/08 10:32 AM
Boris's Gravatar Sorry, GRUB is always in /dev/xvda (MBR for DomU)
Two configs tested :-
1. vmlinuz-xen, initrd-xen in /boot folder.
2. vmlinuz-xen, initrd-xen in / folder (as in OpenSUSE 11 case)
# Posted By Boris | 10/5/08 11:06 AM
Boris's Gravatar OK. It works. Remove vfb=[ type='vnc'] from profile.

[root@ServerCentOS52 vm]# cat rhel52.pv-grub
name = "rhel-pvgrub"
memory = 2048
vcpus = 1
disk = [ 'phy:/dev/sda10,xvda,w!' ]
vif = [ "bridge=eth0" ]
kernel = "/usr/lib/xen/boot/pv-grub-x86_64.gz"
extra = "(hd0,0)/grub/menu.lst"

Start PV DomU :-

[root@ServerCentOS52 vm]# xm create -c rhel52.pv-grub
------------------------------------------------
GRUB MENU it text mode - Press Enter
------------------------------------------------


Booting 'CentOS (2.6.18-92.el5xen)'

root (hd0,0)
Filesystem type is ext2fs, partition type 0x83
kernel /vmlinuz-2.6.18-92.el5xen ro root=/dev/VolGroup00/LogVol00 rhgb quiet
initrd /initrd-2.6.18-92.el5xen.img

close blk: backend at /local/domain/0/backend/vbd/34/51712
PCI: Fatal: No PCI config space access function found
rtc: IRQ 8 is not free.
i8042.c: No controller found.

CentOS release 5.2 (Final)
Kernel 2.6.18-92.el5xen on an x86_64

rhel52pvm login: root
Password:
Last login: Sun Oct 5 16:26:58 on xvc0
[root@rhel52pvm ~]# vncserver

New 'rhel52pvm:2 (root)' desktop is rhel52pvm:2

Starting applications specified in /root/.vnc/xstartup
Log file is /root/.vnc/rhel52pvm:2.log

[root@rhel52pvm ~]# cat /boot/grub/menu.lst
default=0
timeout=15
splashimage=(hd0,0)/grub/splash.xpm.gz
# hiddenmenu
title CentOS (2.6.18-92.el5xen)
root (hd0,0)
kernel /vmlinuz-2.6.18-92.el5xen ro root=/dev/VolGroup00/LogVol00 rhgb quiet
initrd /initrd-2.6.18-92.el5xen.img
# Posted By Boris | 10/5/08 4:47 PM
Boris's Gravatar PV-Grub on CentOS 5.2 PV DomU at Xen 3.3 CentOS 5.2 Dom0 (all 64-bit)
Final draft :-
http://bderzhavets.wordpress.com/2008/10/05/pv-gru...
# Posted By Boris | 10/6/08 8:07 AM
Sascha's Gravatar Hi,

i received a couple of mails, with the question, if the packages are broken ...
Because they get these errors after "yum update xen":

Transaction Check Error:
file /etc/rc.d/init.d/libvirtd from install of libvirt-0.4.4-3 conflicts with file from package libvirt-0.3.3-7.el5
file /etc/sysconfig/libvirtd from install of libvirt-0.4.4-3 conflicts with file from package libvirt-0.3.3-7.el5
file /usr/share/man/man1/virsh.1.gz from install of libvirt-0.4.4-3 conflicts with file from package libvirt-0.3.3-7.el5

Yes, it's normal, because these are only x86_64 packages. You have to remove the i386 version of libvirt.

[root@vh0005 ~]# yum remove libvirt.i386
......

Is this ok [y/N]: y
Downloading Packages:
Erasing : libvirt ######################### [1/1]

Removed: libvirt.i386 0:0.3.3-7.el5
Complete!

[root@vh0005 ~]# yum update xen
..... ####################### [13/13]

Dependency Installed: cyrus-sasl-md5.x86_64 0:2.1.22-4 iscsi-initiator-utils.x86_64 0:6.2.0.868-0.7.el5 qemu.x86_64 0:0.9.0-4
Updated: libvirt.x86_64 0:0.4.4-3 libvirt-python.x86_64 0:0.4.4-3 python-virtinst.noarch 0:0.300.3-7 xen.x86_64 0:3.3.0-3 xen-libs.x86_64 0:3.3.0-3
Complete!


That's all ....

Cheers,
Sascha
# Posted By Sascha | 10/7/08 5:35 PM
Nathan Stratton's Gravatar I did an yum update tonight and upgraded to 2.6.18-92.1.13.el5xen kernel. I get the following error in xend-debug.log:

sysctl operation failed -- need to rebuild the user-space tool set?
Exception starting xend: (13, 'Permission denied')

Any ideas?

-Nathan BlinkMind, Inc. - http://www.blinkmind.com
# Posted By Nathan Stratton | 10/14/08 10:44 PM
Mickaël Canévet's Gravatar Verify that the kernel line in your grub.conf is pointing to /xen.gz-3.3.0.
# Posted By Mickaël Canévet | 10/15/08 2:34 AM
jan's Gravatar http://www.social21.com/Forum/forum_posts.asp?TID=...
# Posted By jan | 10/16/08 8:59 AM
yasfg's Gravatar http://www.social21.com/Forum/forum_posts.asp?TID=...
# Posted By yasfg | 10/16/08 9:01 AM
sdgsdj's Gravatar http://www.social21.com/Forum/forum_posts.asp?TID=...
# Posted By sdgsdj | 10/16/08 9:03 AM
dfhdfhh's Gravatar http://www.social21.com/Forum/forum_posts.asp?TID=...
# Posted By dfhdfhh | 10/16/08 9:04 AM
sdgsdgfh's Gravatar http://www.social21.com/Forum/forum_posts.asp?TID=...
# Posted By sdgsdgfh | 10/16/08 9:05 AM
tredg's Gravatar http://www.cce.ufl.edu/LEED/forum/forum_posts.asp?...
# Posted By tredg | 10/17/08 5:33 PM
qre's Gravatar http://www.cce.ufl.edu/LEED/forum/forum_posts.asp?...
# Posted By qre | 10/17/08 5:36 PM
kij's Gravatar http://www.cce.ufl.edu/LEED/forum/forum_posts.asp?...
# Posted By kij | 10/17/08 5:39 PM
pil's Gravatar http://www.cce.ufl.edu/LEED/forum/forum_posts.asp?...
# Posted By pil | 10/17/08 5:41 PM
kijuh's Gravatar http://www.cce.ufl.edu/LEED/forum/forum_posts.asp?...
# Posted By kijuh | 10/17/08 5:43 PM
oyun's Gravatar Thanks
# Posted By oyun | 10/25/08 7:40 PM
Boris's Gravatar Install Ubuntu Intrepid Server PV DomU at Xen 3.3 (port via http://getco) CentOS 5.2 Dom0
http://bderzhavets.wordpress.com/2008/10/28/instal...
# Posted By Boris | 10/28/08 3:42 AM
lok's Gravatar http://www.state.sd.us/gis/wwf/forum/forum_posts.a...
# Posted By lok | 11/10/08 5:13 PM
ned's Gravatar http://www.state.sd.us/gis/wwf/forum/forum_posts.a...
# Posted By ned | 11/10/08 5:15 PM
nes's Gravatar http://www.state.sd.us/gis/wwf/forum/forum_posts.a...
# Posted By nes | 11/10/08 5:17 PM
merf's Gravatar http://www.state.sd.us/gis/wwf/forum/forum_posts.a...
# Posted By merf | 11/10/08 5:18 PM
milll's Gravatar http://www.state.sd.us/gis/wwf/forum/forum_posts.a...
# Posted By milll | 11/10/08 5:19 PM
Sascha's Gravatar Hi all,

today I have built CentOS 5 RPMS (x86_64) for XEN-3.3.1_RC1_pre.

[root@vh0002 ~]# cat /etc/yum.repos.d/XEN-Gitco.repo
[xen]
name=CentOS-$releasever - XEN
baseurl=http://www.gitco.de/repo/xen3.3.1_rc1_pre
gpgcheck=0

[root@vh0002 ~]#

[root@vh0002 ~]# yum update
Loading "fastestmirror" plugin
Loading mirror speeds from cached hostfile
* xen: www.gitco.de
* extras: ftp.hosteurope.de
* updates: ftp.plusline.de
* base: ftp.hosteurope.de
* addons: ftp.plusline.de
xen 100% |=========================| 951 B 00:00
primary.xml.gz 100% |=========================| 4.1 kB 00:00
xen : ################################################## 8/8
extras 100% |=========================| 1.1 kB 00:00
updates 100% |=========================| 951 B 00:00
base 100% |=========================| 1.1 kB 00:00
addons 100% |=========================| 951 B 00:00
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package xen-libs.x86_64 0:3.3.1_rc1_pre-0 set to be updated
---> Package xen.x86_64 0:3.3.1_rc1_pre-0 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================
Package Arch Version Repository Size
=============================================================================
Updating:
xen x86_64 3.3.1_rc1_pre-0 xen 8.8 M
xen-libs x86_64 3.3.1_rc1_pre-0 xen 158 k

Transaction Summary
=============================================================================
Install 0 Package(s)
Update 2 Package(s)
Remove 0 Package(s)

Total download size: 9.0 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): xen-3.3.1_rc1_pre- 100% |=========================| 8.8 MB 00:14
(2/2): xen-libs-3.3.1_rc1 100% |=========================| 158 kB 00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Updating : xen-libs ######################### [1/4]
Updating : xen ######################### [2/4]
Cleanup : xen-libs ######################### [3/4]
Cleanup : xen ######################### [4/4]

Updated: xen.x86_64 0:3.3.1_rc1_pre-0 xen-libs.x86_64 0:3.3.1_rc1_pre-0
Complete!
[root@vh0002 ~]#

[root@vh0002 ~]# xm info
host : vh0002.gitco.tld
release : 2.6.18-92.1.18.el5xen
version : #1 SMP Wed Nov 12 09:48:10 EST 2008
machine : x86_64
nr_cpus : 2
nr_nodes : 1
cores_per_socket : 1
threads_per_core : 2
cpu_mhz : 3200
hw_caps : bfebfbff:20100800:00000000:00000180:0000641d:00000000:00000000:00000000
virt_caps :
total_memory : 3575
free_memory : 3127
node_to_cpu : node0:0-1
node_to_memory : node0:3127
xen_major : 3
xen_minor : 3
xen_extra : .1-rc1-pre
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
cc_compiler : gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)
cc_compile_by : root
cc_compile_domain : gitco.tld
cc_compile_date : Fri Nov 14 23:39:29 CET 2008
xend_config_format : 4
[root@vh0002 ~]#

Cheers,
Sascha
# Posted By Sascha | 11/14/08 6:18 PM
Gerdek gecesi gelenekleri's Gravatar http://www.oyunlars.net/games/hedef.html
http://cizgi-film-izle-seyret.blogspot.com/
http://www.gazetetr.net
http://www.arkadasfm.com
http://www.oyun112.com
http://www.komikce.net
http://www.gazetetr.net
http://www.arkadasfm.com
http://www.oyun112.com
http://www.oyun112.com/Cocuk.html
http://www.oyun112.com/Macera.html" target="_blank">http://www.oyun112.com/Macera.html
http://www.oyun112.com/Dovus.html" target="_blank">http://www.oyun112.com/Dovus.html
http://www.oyun112.com/Yaris.html
http://www.oyun112.com/Zeka.html" target="_blank">http://www.oyun112.com/Zeka.html
http://www.oyun112.com/Zeka.html" target="_blank">http://www.oyun112.com/Zeka.html
http://www.oyun112.com/Spor.html
http://www.oyun112.com/Komik.html
http://www.oyun112.com/oyunlari.php?t=K?z" target="_blank">http://www.oyun112.com/oyunlari.php?t=K?z

http://www.oyunlars.net
http://www.oyunlars.net/games/kiz.html" target="_blank">http://www.oyunlars.net/games/kiz.html
http://www.oyunlars.net/games/cocuk.html
http://www.oyunlars.net/games/macera.html
http://www.oyunlars.net/games/dovus.html" target="_blank">http://www.oyunlars.net/games/dovus.html
http://www.oyunlars.net/games/yaris.html" target="_blank">http://www.oyunlars.net/games/yaris.html
http://www.oyunlars.net/games/zeka.html
http://www.oyunlars.net/games/spor.html
http://www.oyunlars.net/games/aksiyon.html" target="_blank">http://www.oyunlars.net/games/aksiyon.html
http://www.oyunlars.net/games/komik.html
http://www.oyunlars.net/games/hedef.html
http://cizgi-film-izle-seyret.blogspot.com/
# Posted By Gerdek gecesi gelenekleri | 11/15/08 3:42 AM
[Add Comment]
BlogCFC was created by Raymond Camden.