Jump to content


Photo

Match IOSTAT devices to VHD?

Started by Tonny Andersson , 17 December 2011 - 09:29 PM
9 replies to this topic

Tonny Andersson Members

Tonny Andersson
  • 539 posts

Posted 17 December 2011 - 09:29 PM

When running iostat i get a list of devices (tda, tdb, tdc, tdd etc). I understand that tda is the VHD of the first started VM, tdb is the VHD of the second started vm and so on.

Is there some way to easy identify which device belongs to which vm..?



Tobias Kreidl CTP Member

Tobias Kreidl
  • 18,052 posts

Posted 17 December 2011 - 09:56 PM

Look in /dev/disk and in the subdirectories below that (by-id, by-label, by-path, etc.) and see if you can find a correlation that way.
--Tobias



Tonny Andersson Members

Tonny Andersson
  • 539 posts

Posted 17 December 2011 - 10:09 PM

In my case the VHD's is on NFS-storage. Under /dev/disk/by-id it seems like there are only local disks?



Tobias Kreidl CTP Member

Tobias Kreidl
  • 18,052 posts

Posted 17 December 2011 - 10:21 PM

Not even in the /dev/disk/by-id area? When a drive gets mapped, there should be a device entry created on it somewhere on the XenServer host... there are logical entries, for example, under /sys/block.



paolo nardini Members

paolo nardini
  • 226 posts

Posted 11 October 2013 - 01:44 PM

Thought I'd share some knowledge about this:

In Dom0 run iostat -xn to get the devices;here you can see IOPs other stats etc. Run #iostat -xn 1 to get second stats to view real time to identify issue

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.01 3.50 1.60 11.64 166.03 440.88 45.85 0.30 22.83 3.31 4.39
sda1 0.00 2.63 0.03 1.35 1.37 31.88 23.99 0.06 42.63 6.51 0.90
sda2 0.00 0.00 0.00 0.00 0.00 0.00 8.81 0.00 0.91 0.88 0.00
sda3 0.01 0.86 1.56 10.29 164.66 408.99 48.41 0.24 20.52 3.13 3.71
sdb 0.00 0.00 0.00 0.00 0.00 0.00 8.81 0.00 0.47 0.46 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 24.76 0.00 4.04 3.98 0.00
dm-1 0.00 0.00 0.04 0.14 1.81 3.85 31.62 0.00 10.11 7.01 0.13
tda 0.00 0.00 0.05 0.16 1.81 3.84 27.04 0.00 10.92 6.28 0.13
tdc 0.00 0.00 0.01 0.00 0.26 0.26 39.80 0.00 45.08 7.60 0.01
dm-2 0.00 0.00 0.01 0.06 0.35 13.24 208.57 0.00 20.94 6.38 0.04
tdd 0.00 0.00 0.01 0.16 0.35 13.20 79.27 0.01 43.50 2.50 0.04
tdb 0.00 0.00 0.00 0.04 0.05 0.57 16.27 0.01 234.02 48.05 0.18
tde 0.00 0.00 0.00 0.00 0.01 0.00 46.69 0.00 28.39 15.06 0.00
tdf 0.00 0.00 0.02 0.01 0.89 0.19 38.63 0.00 42.88 8.79 0.02
tdg 0.00 0.00 0.00 0.00 0.02 0.00 29.51 0.00 13.94 10.11 0.00

You should be able to identify the disk issue you want to address or get a view on - for this example I'll use tdd

[root@veskdrxenserver ~]# cat /sys/block/tdd/dev
253:3

Note the minor number - in this case it's 3

Run tap-ctl with minor number to get the VHD

[root@veskdrxenserver ~]# tap-ctl list -m 3
7657 3 0 vhd /dev/VG_XenStorage-aa99c4a4-4c7d-4597-3ecb-7c28aff7a76d/VHD-0f9f6097-d97c-438a-b6c1-bb91217e1c2f

Run xe vdi-list uuid-"VHD name" to get the VM name so you can hop on and see what it's doing

[root@veskdrxenserver ~]# xe vdi-list uuid=0f9f6097-d97c-438a-b6c1-bb91217e1c2f
uuid ( RO) : 0f9f6097-d97c-438a-b6c1-bb91217e1c2f
name-label ( RW): restore local
name-description ( RW):
sr-uuid ( RO): aa99c4a4-4c7d-4597-3ecb-7c28aff7a76d
virtual-size ( RO): 193273528320
sharable ( RO): false
read-only ( RO): false

Thanks Paolo



Ed Goff Members

Ed Goff
  • 42 posts

Posted 13 November 2013 - 02:30 PM

Thank you, this is exactly the information I needed! That doesn't happen often.



Rachel Berry Citrix Employees

Rachel Berry
  • 597 posts

Posted 13 November 2013 - 10:37 PM

A lot of iostat data is available via the rrd metrics so you can view it quickly via xencenter follow the adding graphs instructions in 9.1.3.1 in http://support.citrix.com/article/CTX137828 a lot of metrics will be available per vbd or vif.

You'll see the partial uuid attached to the metric when you get the full list under add graphs.

Any use?
Rachel



Francesco Abeni Members

Francesco Abeni
  • 4 posts

Posted 09 December 2015 - 04:24 PM

Thank you @Paolo for the useful advice.

 

As I understand it, the tda, tdb, tdc, ..., order depends on the boot order of the VM, so if I reboot the Xen machine, the relationship with the VDI may change.

 

In other words, currently tdf is linked to a given VDI, but before the latest reboot it may have been linked to a different VDI. 

 

Is my assumption correct?



paolo nardini Members

paolo nardini
  • 226 posts

Posted 15 December 2015 - 05:08 PM

Hi Francesco,

 

Correct. Also there's always the potential to boot on another host so using this for graphing doesn't work unfortunately.

 

However, for real time troubleshooting it's very good.

 

FYI - On my new XenServer installs for 6.5 Iostat isn't returning data for some reason so I use the below script instead. The devices presented as domains then disk so you can tell which VM they correlate to by using list_domains

 

/opt/xensource/debug/xsiostat



Nekta Singh Citrix Employees
  • #10

Nekta Singh
  • 6 posts

Posted 18 April 2016 - 06:01 PM

To find the iostat of a VDI, complete the following procedure (for example, VDI with UUID=7cd40609-4a53-48e3-83f1-98dc6ad262a5):

  1. Run the following command:
    [root@sydeny tde]# tap-ctl list
    You receive the following output:
    10636    1    0        vhd /dev/VG_XenStorage-46059437-ba85-5699-cd30-57175caeaa26/VHD-d84531eb-93c9-4a42-9b21-ffe5591170f4
    11274    2    0        vhd /dev/VG_XenStorage-631b10d8-cd21-7694-7801-c5839e02be8f/VHD-f455e552-2c0a-44e1-8686-bddea55eda40
    11454    3    0        vhd /dev/VG_XenStorage-631b10d8-cd21-7694-7801-c5839e02be8f/VHD-4959f268-f0ba-4b8b-bd33-68780da799e9
    12469    4    0        vhd /dev/VG_XenStorage-46059437-ba85-5699-cd30-57175caeaa26/VHD-7cd40609-4a53-48e3-83f1-98dc6ad262a5
      9583    0    0        vhd /dev/VG_XenStorage-631b10d8-cd21-7694-7801-c5839e02be8f/VHD-18714d84-9fc4-44ef-b1bf-853e8cfb57b7

    In this output, the second column is minor number and third column is major number.

    Minor Number signifies the corresponding Tap Disk:
    If minor number is 0 it means tda
    If minor number is 1 it means tdb
    If minor number is 2 it means tdc
    If minor number is 3 it means tdd
    If minor number is 4 it means tde

  2. Run the following command:
    [root@sydeny /]# xe vdi-list uuid=7cd40609-4a53-48e3-83f1-98dc6ad262a5
    uuid ( RO)                : 7cd40609-4a53-48e3-83f1-98dc6ad262a5
              name-label ( RW): citrix-license disk
        name-description ( RW): Created by template provisioner
                 sr-uuid ( RO): 46059437-ba85-5699-cd30-57175caeaa26
            virtual-size ( RO): 8589934592
                sharable ( RO): false
               read-only ( RO): false
    [root@sydeny /]# cat /sys/block/tde/dev
    253:4

  3. Run the following command to find out the I/O of Tap Disk:
    [root@sydeny /]# iostat -xn
    -x and -n options are mutually exclusive.
    Linux 2.6.32.43-0.4.1.xs1.8.0.847.170785xen (sydeny)    08/13/2014
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
               0.62    0.00    0.05    0.12    0.04   99.18
     
    Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
    sda               0.02     1.87  0.55  2.97    49.13    90.42    39.66     0.01    2.66   1.39   0.49
    sda1              0.01     1.84  0.05  0.97     1.73    22.42    23.70     0.00    3.43   2.87   0.29
    sda2              0.00     0.00  0.00  0.00     0.00     0.00     7.68     0.00    2.61   2.57   0.00
    sda3              0.00     0.04  0.50  2.00    47.40    68.00    46.16     0.01    2.35   0.82   0.20
    dm-0              0.00     0.00  0.00  0.00     0.00     0.00    21.93     0.00    3.93   2.86   0.00
    dm-1              0.00     0.00  0.23  0.64    34.21    43.52    89.36     0.00    5.41   1.35   0.12
    tda               3.77     4.57  0.55  0.86    34.52    43.45    55.45     0.01    9.90   0.89   0.12
    sdb               0.01     0.00  1.11  1.35    69.63    50.80    49.07     0.01    5.00   1.66   0.41
    dm-2              0.00     0.00  0.00  0.00     0.03     0.00    24.73     0.00    2.47   2.38   0.00
    dm-5              0.00     0.00  0.94  0.05    48.53    12.80    62.38     0.00    3.74   2.33   0.23
    tdb               0.07     0.00  0.99  0.13    47.37     4.25    46.03     0.00    4.24   2.04   0.23
    dm-6              0.00     0.00  0.26  1.06    12.98    19.10    24.28     0.00    1.00   0.71   0.09
    tdc               0.02     0.00  0.32  1.07    13.04    19.06    23.03     0.00    1.53   0.70   0.10
    dm-7              0.00     0.00  0.00  0.34     0.14     5.37    15.97     0.00    1.05   0.39   0.01
    tdd               0.00     0.33  0.01  0.33     0.16     5.36    16.17     0.00    1.35   0.46   0.02
    dm-8              0.00     0.00  0.04  0.02     1.17     0.23    23.04     0.00    9.31   4.88   0.03
    tde               0.00     0.27  0.04  1.21     0.93    19.66    16.47     0.01    5.44   0.75   0.09
    dm-3              0.00     0.00  0.00  0.00     0.01     0.02    48.49     0.00    6.10   1.69   0.00
    dm-4              0.00     0.00  0.01  1.15     0.03    19.45    16.82     0.01    4.45   0.48   0.06
    dm-9              0.00     0.00  0.06  0.10     2.25     3.20    35.67     0.00    3.95   1.19   0.02
    Device:                  rBlk_nor/s   wBlk_nor/s   rBlk_dir/s   wBlk_dir/s   rBlk_svr/s   wBlk_svr/s     ops/s    rops/s    wops/s 

    root@sydeny /]# iostat -xnm tde
    -x and -n options are mutually exclusive.
    Linux 2.6.32.43-0.4.1.xs1.8.0.847.170785xen (sydeny)    08/11/2014
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
               0.62    0.00    0.05    0.13    0.04   99.17
    Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
    tde               0.00     0.26  0.05  1.19     0.00     0.01    16.54     0.01    5.53   0.80   0.10
    Device:                   rMB_nor/s    wMB_nor/s    rMB_dir/s    wMB_dir/s    rMB_svr/s    wMB_svr/s     ops/s    rops/s    wops/s

    tps: Indicates the number of transfers per second that were issued to the device. A transfer is an I/O request to the device. Multiple logical requests can be combined into a single I/O request to the device. A transfer is of indeterminate size.
    rrqm/s: The number of read requests merged per second that were queued to the device.
    wrqm/s: The number of write requests merged per second that were queued to the device. 
    r/s: The number of read requests that were issued to the device per second. 
    w/s: The number of write requests that were issued to the device per second.
    rMB/s: The number of megabytes read from the device per second.
    wMB/s: The number of megabytes written to the device per second. 
    avgrq-sz: The average size (in sectors) of the requests that were issued to the device. 
    avgqu-sz: The average queue length of the requests that were issued to the device. 
    await: The  average  time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the
                         time spent servicing them. 
    svctm: The average service time (in milliseconds) for I/O requests that were issued to the device. 
    %util: Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.