Search This Blog

Tuesday, May 27, 2014

How to use F5 Wireshark Plugin for LTM troubleshooting

In this post we are going to look how to use F5 Wireshark Plugin to troubleshoot networking issues on BigIP LTM.
  • Download the and install the plugin in your Wireshark
The full instruction are here F5 Wireshark Plugin. In essence you needed to copy the f5ethtrailer.dll file into C:\Program Files (x86)\wireshark\wireshark16\WiresharkPortable\ and restart my Wireshark.

Once you restart wireshark go to menu Help - About Wireshark, Plugins tab. You should be able to see the plugin listed there if properly installed.

  • The plugin is useful only if you take a capture on LTM with 'noise' information.
The noise is an internal information that TMM is attaching and managing for every packet when is being processed. To have a capture with noise these are the minimal options you need to specify:

tcpdump -w /var/tmp/capture.pcap -s0 -i _interface_:nnn

where the _interface_ can be:
    •  1.1 - example of an physical interface
    • dmz_vlan - a name you gave to your vlan when created
    • 0.0 - is the equivalent of 'any' interface what means capture on all interfaces and all vlans
My favourite syntax is usually something like this:

tcpdump -s0 -nn -w /var/tmp/test1-$(date +%s).pcap -i 0.0:nnn '(host _ip_ and port _port_ ) or arp or not ip' 
  • Open the capture in wireshark as normal
Once you open you will noticed that there is additional section in the packet details.

  • The most useful part of using this plugin is that you can quickly and easily find the client and server site traffic in the capture (It can be a challenging when you have multiple tcp streams and OneConnect profile):
    • Find a single packet of the flow you are interested in (search for VIP or client ip for example).
    • Find the "Flow ID" from the F5 Ethernet trailer (see the picture above for example).
    • Click with right mouse taste on the Flow ID field and select "Prepare as Filter".
    • In the Filter box (on top ) it will pre-populate the syntax for you.
    • Copy the hex value and delete the '.flowid == hex' part and start typing '.'  (dot).
    • It will mediately give you a list of possible options, select anyflowid and copy the hex back as it was originally. Example:
The original filter         : f5ethtrailer.flowid == 0x0d2e6dc0
Filter after modifications  : f5ethtrailer.anyflowid == 0x0d2e6dc0
    • Press Apply button
This filter is gong to find the client and server site flows for you. You can then analyse them packet by packet to find out and understand how and why LTM load balance it to one or another pool member.

References

https://devcentral.f5.com/wiki/advdesignconfig.F5WiresharkPlugin.ashx
https://devcentral.f5.com/questions/tcpdump-with-multiple-pool-members
SOL13637: Capturing internal TMM information with tcpdump

Wednesday, May 21, 2014

Simple MySQL and SQL exercises

How to create a sample MySQL data base and user

You can download an example data base sql file from here: http://www.mysqltutorial.org/mysql-sample-database.aspx. After unziping you should find following file:
 
rado2@ubuntu12-04:~$ ls -la mysqlsampledatabase.sql
-rw-rw-r-- 1 rado2 rado2 190711 May 23  2013 mysqlsampledatabase.sql
 
rado2@ubuntu12-04:~$ more mysqlsampledatabase.sql
/*
http://www.mysqltutorial.org
*/

CREATE DATABASE /*!32312 IF NOT EXISTS*/`classicmodels` /*!40100 DEFAULT CHARACTER SET latin1 */;

USE `classicmodels`;

/*Table structure for table `customers` */

DROP TABLE IF EXISTS `customers`;

CREATE TABLE `customers` (
  `customerNumber` int(11) NOT NULL,
  `customerName` varchar(50) NOT NULL,
  `contactLastName` varchar(50) NOT NULL,
  `contactFirstName` varchar(50) NOT NULL,
....

We don't want to use a root user to manipulate our data base records. To create a separate user you can run these commands:
 
$ mysql -u root -p
mysql> use information_schema;
mysql> CREATE USER 'rado2'@'localhost';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'rado2'@'localhost';
mysql> select * from USER_PRIVILEGES ;

To import and inspect the database we can use this commands:
 
$ mysql -u rado2 < mysqlsampledatabase.sql

$ mysql -u rado2
mysql> show databases;
mysql> show tables;
+-------------------------+
| Tables_in_classicmodels |
+-------------------------+
| customers               |
| employees               |
| offices                 |
| orderdetails            |
| orders                  |
| payments                |
| productlines            |
| products                |
+-------------------------+
8 rows in set (0.00 sec)

mysql> select * from employees  LIMIT 5;
+----------------+-----------+-----------+-----------+---------------------------------+------------+-----------+----------------------+
| employeeNumber | lastName  | firstName | extension | email                           | officeCode | reportsTo | jobTitle             |
+----------------+-----------+-----------+-----------+---------------------------------+------------+-----------+----------------------+
|           1002 | Murphy    | Diane     | x5800     | dmurphy@classicmodelcars.com    | 1          |      NULL | President            |
|           1056 | Patterson | Mary      | x4611     | mpatterso@classicmodelcars.com  | 1          |      1002 | VP Sales             |
|           1076 | Firrelli  | Jeff      | x9273     | jfirrelli@classicmodelcars.com  | 1          |      1002 | VP Marketing         |
|           1088 | Patterson | William   | x4871     | wpatterson@classicmodelcars.com | 6          |      1056 | Sales Manager (APAC) |
|           1102 | Bondur    | Gerard    | x5408     | gbondur@classicmodelcars.com    | 4          |      1056 | Sale Manager (EMEA)  |
+----------------+-----------+-----------+-----------+---------------------------------+------------+-----------+----------------------+

mysql> select * from offices  LIMIT 5;
+------------+---------------+-----------------+--------------------------+--------------+------------+---------+------------+-----------+
| officeCode | city          | phone           | addressLine1             | addressLine2 | state      | country | postalCode | territory |
+------------+---------------+-----------------+--------------------------+--------------+------------+---------+------------+-----------+
| 1          | San Francisco | +1 650 219 4782 | 100 Market Street        | Suite 300    | CA         | USA     | 94080      | NA        |
| 2          | Boston        | +1 215 837 0825 | 1550 Court Place         | Suite 102    | MA         | USA     | 02107      | NA        |
| 3          | NYC           | +1 212 555 3000 | 523 East 53rd Street     | apt. 5A      | NY         | USA     | 10022      | NA        |
| 4          | Paris         | +33 14 723 4404 | 43 Rue Jouffroy D'abbans | NULL         | NULL       | France  | 75017      | EMEA      |
| 5          | Tokyo         | +81 33 224 5000 | 4-1 Kioicho              | NULL         | Chiyoda-Ku | Japan   | 102-8578   | Japan     |
+------------+---------------+-----------------+--------------------------+--------------+------------+---------+------------+-----------+

mysql> show COLUMNS FROM employees
+----------------+--------------+------+-----+---------+-------+
| Field          | Type         | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+-------+
| employeeNumber | int(11)      | NO   | PRI | NULL    |       |
| lastName       | varchar(50)  | NO   |     | NULL    |       |
| firstName      | varchar(50)  | NO   |     | NULL    |       |
| extension      | varchar(10)  | NO   |     | NULL    |       |
| email          | varchar(100) | NO   |     | NULL    |       |
| officeCode     | varchar(10)  | NO   | MUL | NULL    |       |
| reportsTo      | int(11)      | YES  | MUL | NULL    |       |
| jobTitle       | varchar(50)  | NO   |     | NULL    |       |
+----------------+--------------+------+-----+---------+-------+

mysql> show COLUMNS FROM offices ;
+--------------+-------------+------+-----+---------+-------+
| Field        | Type        | Null | Key | Default | Extra |
+--------------+-------------+------+-----+---------+-------+
| officeCode   | varchar(10) | NO   | PRI | NULL    |       |
| city         | varchar(50) | NO   |     | NULL    |       |
| phone        | varchar(50) | NO   |     | NULL    |       |
| addressLine1 | varchar(50) | NO   |     | NULL    |       |
| addressLine2 | varchar(50) | YES  |     | NULL    |       |
| state        | varchar(50) | YES  |     | NULL    |       |
| country      | varchar(50) | NO   |     | NULL    |       |
| postalCode   | varchar(15) | NO   |     | NULL    |       |
| territory    | varchar(10) | NO   |     | NULL    |       |
+--------------+-------------+------+-----+---------+-------+
9 rows in set (0.00 sec)

Exercise 1: select all employees from offices in USA only
 
mysql> SELECT * FROM employees as e, offices as o  where e.officeCode = o.officeCode and o.country='USA';
+----------------+-----------+-----------+-----------+---------------------------------+------------+-----------+--------------------+------------+---------------+-----------------+----------------------+--------------+-------+---------+------------+-----------+
| employeeNumber | lastName  | firstName | extension | email                           | officeCode | reportsTo | jobTitle           | officeCode | city          | phone           | addressLine1         | addressLine2 | state | country | postalCode | territory |
+----------------+-----------+-----------+-----------+---------------------------------+------------+-----------+--------------------+------------+---------------+-----------------+----------------------+--------------+-------+---------+------------+-----------+
|           1002 | Murphy    | Diane     | x5800     | dmurphy@classicmodelcars.com    | 1          |      NULL | President          | 1          | San Francisco | +1 650 219 4782 | 100 Market Street    | Suite 300    | CA    | USA     | 94080      | NA        |
|           1056 | Patterson | Mary      | x4611     | mpatterso@classicmodelcars.com  | 1          |      1002 | VP Sales           | 1          | San Francisco | +1 650 219 4782 | 100 Market Street    | Suite 300    | CA    | USA     | 94080      | NA        |
|           1076 | Firrelli  | Jeff      | x9273     | jfirrelli@classicmodelcars.com  | 1          |      1002 | VP Marketing       | 1          | San Francisco | +1 650 219 4782 | 100 Market Street    | Suite 300    | CA    | USA     | 94080      | NA        |
|           1143 | Bow       | Anthony   | x5428     | abow@classicmodelcars.com       | 1          |      1056 | Sales Manager (NA) | 1          | San Francisco | +1 650 219 4782 | 100 Market Street    | Suite 300    | CA    | USA     | 94080      | NA        |
|           1165 | Jennings  | Leslie    | x3291     | ljennings@classicmodelcars.com  | 1          |      1143 | Sales Rep          | 1          | San Francisco | +1 650 219 4782 | 100 Market Street    | Suite 300    | CA    | USA     | 94080      | NA        |
|           1166 | Thompson  | Leslie    | x4065     | lthompson@classicmodelcars.com  | 1          |      1143 | Sales Rep          | 1          | San Francisco | +1 650 219 4782 | 100 Market Street    | Suite 300    | CA    | USA     | 94080      | NA        |
|           1188 | Firrelli  | Julie     | x2173     | jfirrelli@classicmodelcars.com  | 2          |      1143 | Sales Rep          | 2          | Boston        | +1 215 837 0825 | 1550 Court Place     | Suite 102    | MA    | USA     | 02107      | NA        |
|           1216 | Patterson | Steve     | x4334     | spatterson@classicmodelcars.com | 2          |      1143 | Sales Rep          | 2          | Boston        | +1 215 837 0825 | 1550 Court Place     | Suite 102    | MA    | USA     | 02107      | NA        |
|           1286 | Tseng     | Foon Yue  | x2248     | ftseng@classicmodelcars.com     | 3          |      1143 | Sales Rep          | 3          | NYC           | +1 212 555 3000 | 523 East 53rd Street | apt. 5A      | NY    | USA     | 10022      | NA        |
|           1323 | Vanauf    | George    | x4102     | gvanauf@classicmodelcars.com    | 3          |      1143 | Sales Rep          | 3          | NYC           | +1 212 555 3000 | 523 East 53rd Street | apt. 5A      | NY    | USA     | 10022      | NA        |
+----------------+-----------+-----------+-----------+---------------------------------+------------+-----------+--------------------+------------+---------------+-----------------+----------------------+--------------+-------+---------+------------+-----------+
10 rows in set (0.00 sec)

References

http://www.mysqltutorial.org/mysql-sample-database.aspx
http://en.wikipedia.org/wiki/Join_%28SQL%29
https://answers.yahoo.com/question/index?qid=20080520200936AAmD1Mt
http://www.cyberciti.biz/tips/mysql-auto-completion-for-database-table-names.html

Tuesday, May 20, 2014

What is the difference between XenServer vs Xen vx XCP vs XAPI

It wasn't clear to me at fist what the differences are between XenServer, Xen and XCP. To make it even more confusing the documentation in many place were referring to XAPI and its importance in managing the hypervisors.

To understand what the XAPI is and how it can be used please take a look at this demo I wrote: How to install ipython on XenServer and test XAPI. As we can see the XAPI is an elegant way on top of the hypervisor itself that exposes some more advance API operation to help to control and managed the VM and hypervisor live cycle.

In a very simplistic way you can think of Xen as a 'hypervisor kernel'. The kernel itself may be difficult to use so we need some management software bundled with it.

It is similar comparing Linux kernel and a distribution together. It is hard to use the kernel on its own, we need a more user friendly tools to do this and this is the place where GNU toolchain is coming into play.

Once we understand this it is now easy to understand this FAQ: What's the difference between Xen hypervisor (from xen.org) and Citrix XenServer or XCP?

If you understood what the last link is about please take a look at these for more advance comparisons:
Here is an example showing the differences between the XenServer and Xen management cli:

How to install ipython on XenServer and test XAPI

We've been using the more user friendly shell to interact with python before: ipython. The example below are showing first how to install and enable EPEL repository to be able to install ipython. Next we are going to write a simple XAPI demo program.

Install ipython
  • Find the distro your XenServer is based on
cat /etc/issue.net
CentOS release 5.7 (Final)
Kernel \r on an \m
  • Check enabled repository 
yum repolist
  • From the EPEL install the relevant rpm packets that will add new repository to your yum
# http://fedoraproject.org/wiki/EPEL
# http://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/epel/5/i386/repoview/epel-release.html

rpm --force -i http://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
  • Update the repo info
yum list 
yum list | grep ipython
  • Install ipython
yum install ipython.noarch
  • Start and verify that ipython is working fine
ipython

In [3]: import sys
In [4]: sys.version
Out[4]: '2.4.3 (#1, Sep 21 2011, 20:06:00) \n[GCC 4.1.2 20080704 (Red Hat 4.1.2-51)]'

Xapi example using ipython

References

XAPI:
http://blogs.citrix.com/2011/05/18/so-what-is-xenserver-xapi/
http://docs.vmd.citrix.com/XenServer/6.2.0/1.0/en_gb/sdk.html#language_bindings-python

Packages:
http://xmodulo.com/2012/05/how-to-install-additional-packages-in.html
http://thomas-cokelaer.info/blog/2012/01/installing-repositories-under-centos-6-2-to-get-ipython-r-and-other-packages/
http://fedoraproject.org/wiki/EPEL#How_can_I_use_these_extra_packages.3F


Monday, May 19, 2014

Create a VM on an isolated network

For experimenting and testing we want to have a VM that is attached to an isolated network.
In the script below in the part 1) :
  • Take a clone of an existing VM
  • Create a new private network
  • Create and attached new interface to our VM
 Next in part 2) we configure statically an IP of 192.168.32.1 on this new interface.

References

http://blogs.citrix.com/2013/03/18/virtual-hypervisor/
https://wiki.debian.org/NetworkConfiguration
http://docs.vmd.citrix.com/XenServer/4.0.1/reference/ch03s02.html

Saturday, May 17, 2014

Using xe how to boot a VM in XenServer

After we have installed XenServer  there is a time to spin up an VM to test. Below are is a litle script that create a VM (code is based on: http://wiki.xen.org/wiki/Installing_Linux_on_Kronos ).

This create and starts the VM. You need to connect over the console port and follow the installer questions. We don't need anything media or ISO files. The installer will automatically download all necessary files.

Once your VM is created you can shutdown it, export it to XVA image. Later on we can restore the VM by simply importing it back. We can use the XVA file as well to create further more VMs for testing.

References

http://wiki.xen.org/wiki/Installing_Linux_on_Kronos
http://krypted.com/tag/list_domains

Docs on Citrix site:

http://docs.vmd.citrix.com/XenServer/6.2.0/1.0/en_gb/

XenServer 6.2.0 Technical FAQ
XenServer 6.2.0 Release Notes
XenServer Product Documentation
/

Linux Bash cheat sheet

I've spend some time googling for bash shortcats using phrases like: bash readline shortcat, copy and paste text to bash clipboard, etc  ... I always forget how to do this, especially when I don't work on Linux for a while.

Below is a list of my favorite (hard to remember) bash shortcats and tricks I like to use.

Bash shortcats

Ctrl + w  Cut the Word before the cursor to the clipboard
Ctrl + y  Paste the last thing to be cut (yank)
Alt + r  Cancel the changes and put back the line as it was in the history (revert).

Bash tricks to speed up typing 
  • How to copy the last command 
  • How to copy and paste the last command output
This one is my favorite because it allows me to refer to a previous command output text without having to copy and paste it with mouse.

# readline function
shell-expand-line (M-C-e)

Example 1:
$ myvar="/etc/passwd"
$ echo $myvar
$ ls $(echo $myvar)

Before you press enter press now (M-C-e) and the line will turn into

ls /etc/passwd-rrr

Example 2:
$ ls -l /etc/passwd
$ echo !!

Before you press enter press now (M-C-e) and the line will turn into

echo ls -l /etc/passwd

Example 3:

$ ls -l /etc/passwd
$ echo $(!!)

Before you press enter press now (M-C-e) and the line will turn into

echo -rw-r--r-- 1 root root 1399 May 17 02:19 /etc/passwd

References

http://ss64.com/bash/syntax-keyboard.html
http://superuser.com/questions/304519/how-to-copy-the-results-from-a-grep-command-to-the-bash-clipboard
http://superuser.com/questions/421463/why-does-ctrl-v-notpaste-in-bash-linux-shell
http://unix.stackexchange.com/questions/15850/how-to-use-keyboard-instead-of-mouse-middle-click-for-copy-paste
http://stackoverflow.com/questions/749544/pipe-to-from-clipboard
https://wiki.archlinux.org/index.php/Keyboard_Shortcuts
http://rtomaszewski.blogspot.co.uk/2013/06/linux-and-bash-cheat-sheet.html



XenServer installation over iKVM and redirected ISO CD-ROM option in JViewer

I've had a problem using my XenServer on VMware Workstation. I needed instead to install it on my dedicated server i have. Some of my notes:
  • Enabled iKVM on the server: How to enable IPMI settings in BIOS on Tyan S8225 motherboard.
  • In BIOS make sure you have enabled the VGA graphic output and disabled the graphics output on the PCI-X bus (as per  the link above)
  • The XenServer installer was not able to recognize my AHCI disk driver so I needed to enable a regular 'Native IDE' driver in BIOS (see screen shots below)
  • Disable java security permission in Windows (search for 'configure java' under Windows menu)
  • Mount the ISO image from the JViewer (java iKVM from ASUS) restart the server and follow the installation instructions :)
  • At the beginning of installation you will see that the phase 'Loading /install.img' is taking very long time. Don't panic, watch the network card stats and wait for the installer to start. It can take even up to 10min.


References

http://www.davethijssen.nl/2013/07/install-citrix-xenserver-62-from-usb.html

Wednesday, May 14, 2014

How to install XenServer on VMware Workstation in Windows

How to install XenServer 6.2 on VMware Workstation 8

Note: 17 May 2014:
The installation was fine. The XenServer boots fine in the VM. Unfortunately it hangs once in a while constantly. Couldn't find out what causes it. I've installed XenServer on hw instead here.

It is possible to install Type 1 hypervisor like XenServer within a virtualized environment like VMware Workstation by using nested virtualization technology. It may be shocking at first look because this is the software stack we are going to create:
  • Regular operating system, I'm using Windows 7
  • Install in Windows (Type 2) hypervisor VMware Workstation
  • Create a VM within VMware 
  • Install XenServer with in VM
  • Boot VM and run (Type 1) hypervisor XenServer
Installation steps
  • Create a new VM and chose guest: OS VMware ESX and version: VMware ESXi 5 (my config for the VM can be found here: vmware-workstation-xenserver.conf)
  • Download the XenServer-6.2.0-install-cd.iso 
  • For the VM chose to boot from the iso above
  • Boot the VM and follow the installation instruction
  • Installer at the end will reboot the VM
  • Before new boots power off VM and deselect the ISO file
  • Boot the VM and enjoy your XenServer 
  • You may want to install XenServer-6.2.0-XenCenter.msi to graphically manged your XenServer or use CLI over SSH
 XenServer 6.2 doesn't boot after installation on VMware Workstation 8

You may run into the following issues when attempting the installation procedure above

  • Wrong VM guest type. I've seen this initially when I try to use the type: Other, version: Other 64-bit.
BUG: recent printk recursion!
clocksource/1: Time went backwards
PCI: Bar 13: no parent found for of bridge



  • You can boot the VM and from the Xen boot loader select 'safe'. Unfortunately the XenServer will report this time I/O error.
ata2.01: qc timeout (cmd 0xa0)
ata2.01: TEST_UNIT_READY failed (err_mask=0x4)
Unhanded error code
Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
end_request: I/O error, dev sda, sector 0
buffer I/O error on device sda, logical block 0




References

http://bjtechnews.org/2013/07/01/how-to-install-citrix-xenserver-6-2-0-on-vmware-workstation-9-0/
http://discussions.citrix.com/topic/329733-xenserver-freeze-on-reboot/
http://discussions.citrix.com/topic/324048-virtual-machines-in-xenserver-6-on-wmware-workstation/
http://www.vi-tips.com/2011/10/how-to-run-xenserver-60-on-vsphere-5.html
http://vstorage.wordpress.com/2010/06/06/running-xenserver-5-6-on-vmware-workstation/