Deploy VMs using Ansible

[Bojan Vitnik1709158530]

Hi guys,

 

Sorry if this looks like shameless self promotion but for a few weeks now I've been working on Ansible module for managing XenServer VMs - first of it's kind if I'm not mistaken. The module is in line with what Ansible offers for VMWare and there is a pull request to mainline it. What I would love is for anyone willing to help, and already familiar with Ansible, is to spare some time to test the module since the only way to test it is against real hardware. Of course, the module should not be tested in production environments.

 

UPDATE: 2019-03-01

 

Some of my Ansible modules have been merged upstream and are currently available in development version of Ansible. Ansible 2.8 will be the first official release to include these modules.

 

My work is currently involved around these three modules:

 

xenserver_guest - upstreamed - used for deployment of new VMs from templates and reconfiguration of existing VMs.

xenserver_guest_facts - upstreamed - used for getting VM facts (useful XenServer VM params).

xenserver_guest_powerstate - upstreamed - used for controlling VM power state (running/halted/suspended), graceful shutdown and reboot etc.

 

For anyone interested in using/testing these modules, upstreamed modules can be acquired by cloning Ansible repo from GitHub (or downloading the ZIP archive) and running Ansible from source. Instructions can be found here:

 

https://docs.ansible.com/ansible/devel/installation_guide/intro_installation.html#running-from-source

 

To use the modules, you will also need XenAPI.py from here:

 

https://raw.githubusercontent.com/xapi-project/xen-api/master/scripts/examples/python/XenAPI.py

 

Copy the file to your Python site-packages (e.g. /usr/lib/python2.7/site-packages/ on CentOS 7).

 

You can get module documentation by running this command:

$ ansible-doc <module_name>

or from official Ansible docs (upstreamed modules only):

• https://docs.ansible.com/ansible/devel/modules/xenserver_guest_module.html#xenserver-guest-module
• https://docs.ansible.com/ansible/devel/modules/xenserver_guest_facts_module.html#xenserver-guest-facts-module
• https://docs.ansible.com/ansible/devel/modules/xenserver_guest_powerstate_module.html#xenserver-guest-powerstate-module

 

CHANGELOG:

• Fixed a bug on XenServer 7.1 with Cumulative Update where a version could not be properly detected, causing an exception.

 

Thanks.

 

[John Thompson1709154691]

Hey Bojan,

 

I think I found another small bug.

The module doesn't seem to handle spaces in the network name.

i.e. If I have a network name in xencenter with "Network 4" or "VLAN 41", the vm create seems to fail.

 

If I take rename the NIC in Xencenter with no space and then update the playbook, everything works.

[Bojan Vitnik1709158530]

I'm also struggling to find the best way to provision VMs at scale. By now, I've implemented three completely different ways to do that. I can't share them here but I can give you some tips:

 

1) Don't use with_* loop controls. They will be deprecated. Use plain loop.

 

2) For a var containing hierarchy of VMs, I'd use something like this:

    Machines:
      Connectors:
        - hostname: CCConn-0001
          num_cpus: 4
          num_cpu_cores_per_socket: 2
          memory_mb: 8192
        - hostname: CCConn-0002
          num_cpus: 4
          num_cpu_cores_per_socket: 2
          memory_mb: 8192
      Storefronts:
        - hostname: SFPrinci-0001
          num_cpus: 4
          num_cpu_cores_per_socket: 2
          memory_mb: 8192

So I lost one dict level and used lists. Now you can do something like this:

  tasks:
  - name: Create VMs from a template
    xenserver_guest:
      hostname: 10.8.47.11
      username: 
      password: 
      validate_certs: no
      #folder: /home/testvms
      name: '{{ item['hostname'] }}'
      state: poweredon
      template: W2K16_RTM_64_EN_ans
      disks:
      - size_gb: 100
        name: ''
        sr: XenRTVol
      linked_clone: yes
      hardware:
        num_cpus: '{{ item['num_cpus'] }}'
        num_cpu_cores_per_socket: '{{ item['num_cpu_cores_per_socket'] }}'
        memory_mb: '{{ item['memory_mb'] }}'
      cdrom:
        type: iso
        iso_name: guest-tools.iso
      networks:
      - name: vlan40
      wait_for_ip_address: no
    delegate_to: localhost
    register: deploy
    loop: '{{ Machines['Connectors'] + Machines['Storefronts'] }}'

For a simple use case, that should do it.

 

3) Learn how to use jinja filters that manipulate lists and dicts. The most powerful one of them is json_query (you'll have to learn some jmespath to use it efficiently).

 

[Bojan Vitnik1709158530]

A milestone has been achieved  . All of my modules have been merged upstream. I've updated the first post with new info.

 

That will most probably be all for Ansible 2.8. I'm keeping some improvements and possibly more modules for Ansible 2.9. Currently, I'm thinking about what other modules could be useful so I could implement them. Any suggestion or wish would be much appreciated.

 

[John Thompson1709154691]

Thanks for all of this.  I really wish I had access to these tools 3 years ago.

Currently I have to use a lot of internal tools to do what I need here at citrix (and they have their pros and cons- and obviously I can't share them), but, I really think this ansible stuff would be beneficial to anyone using citrix products (or the open source xen).

 

I created some windows boot scripts that makes renaming a "syspred'd" virtual machine a lot easier and runs the "configure remoting" for ansible script as well.

I wonder where would be a good place to share those? 

 

Do you have any deletion capability?  (Obviously that would only speed up automated testing, not really production environments I guess)

 

If I think of anything else, I will let you know.  Thanks again.

 

 

 

[Bojan Vitnik1709158530]

For deleting VMs, use "state: absent". If VM is running, also use "force: yes".

 

For sharing your scripts, I don't really know. Maybe put it in some GitHub repo and make a nice blog post on some Citrix/XenServer blog with introduction and usage :).

 

[Bojan Vitnik1709158530]

johntho, can you ask around Citrix if someone is willing to update XenAPI pip package?

 

https://pypi.org/project/XenAPI

 

It's very old. Maybe Citrix could give access to the pypi project to a community maintainer (I can be one)? That would also be great.

[John Thompson1709154691]

Hey sorry, for the delayed response.  I've been a sidetracked a bit over the past few days.

I'll ask in our internal xenserver chat channels and see what they say.

[John Thompson1709154691]

Bojan,

 

Your request generated quite a few discussions among some of the maintainers.

Looks like they are pro, mirroring it to pypi.

 

I don't know how long this will take.  They have another .NET xen api repo they are trying to update as well.

I think there are some internal JIRA tickets being opened to begin the process :)

 

I didn't hear much back on whether they would allow community maintainers.

 

I don't feel comfortable throwing out the emails of the guys driving it (because I don't know what the etiquette is), so I'll just try to keep tabs on it for you.

[Bojan Vitnik1709158530]

Thanks a lot johntho. That's some great news. I hope they will be able to maintain releases on regular basis.

[John Thompson1709154691]

No problem.  I'll try and keep an eye on it, as I'm interested as well.

 

On a side note:

Can the ansible xenserver_guest module take snapshots?

I.e., A use case might be, take a snapshot before I use chocolatey to install a piece of software

 

Or is that considered trivial since is probably a one line xe command?

[Bojan Vitnik1709158530]

The answer is no. xenserver_guest module can't be used to make snapshots. That would be hard to express trough module parameters so the best option is to implement different module for snapshot management. VMware already has one.

 

So basically, that's a good idea for new module :). I'm currently exploring possibilities to implement module for managing XenServer networks. Module for managing snapshots could come next.

[Bojan Vitnik1709158530]

First post has been updated with fresh info. Another module got merged upstream. A fix for XenServer 7.1 bug has also been merged upstream.

[Bojan Vitnik1709158530]

It should work with arbitrary network names. In fact, networks in my testing environment contain spaces in their names and I have not encountered any problems.

 

Can you run ansible-playbook with -vvv option and send me the output of the task that fails? Also, task definition from playbook of the task that fails would be helpful.

 

[John Thompson1709154691]

So this is strange.

 

The first time I saw it fail, the NIC name was 'Network 4' in Xencenter.

I re-named it to Network4 and it worked

 

I then just now renamed it back to 'Network 4' and it worked.

 

I seem to recall this behavior with some internal xenapi tools we have, but, I was never able to really track down why.

 

At any rate, perhaps this is a false alarm.  I guess its not that serious.

 

See attached for logs, etc.

 

Here is the task.

 

- hosts: xenservers
  vars_files:
    - /home/johntho/ansible-dev/cqe/ansible-goodies/group_vars/vars/machines.yml

  tasks:
  - name: Create VMs from a template
    xenserver_guest:
      hostname: XXXXX
      username: root
      password: blabla
      validate_certs: no
      #folder: /home/testvms
      name: '{{item.value.hostname}}'
      state: poweredon
      template: 'WS 2016 updated 2018-08-03'
      #template_uuid: 506e4d22-ade3-69fe-cf40-2bd251fede57
      disks:
      - size_gb: 100
        name: '{{item.value.hostname}}'
        sr: 'Cam-ss-2 SMB virtual storage'
      linked_clone: yes
      hardware:
        num_cpus: '{{item.value.num_cpus}}'
        num_cpu_cores_per_socket: '{{item.value.num_cpu_cores_per_socket}}'
        memory_mb: '{{item.value.memory_mb}}'
      cdrom:
        type: iso
        iso_name: guest-tools.iso
      networks:
      - name: 'Network 4'
      wait_for_ip_address: no
    delegate_to: localhost
    register: deploy
    loop: "{{ Machines | dict2items }}"

  # - name:
  #   debug:
  #     msg: "{{ item.key }} - {{ item.value.hostname }}"
  #   with_dict: "{{ Machines }}"

And the var file:

Machines:
  # Connector0:
  #   hostname: jwtConn0
  #   machine_group: Connectors
  #   memory_mb: 8192
  #   num_cpu_cores_per_socket: 2
  #   num_cpus: 4
  Connector1:
    hostname: jwtConn1
    machine_group: Connectors
    memory_mb: 8192
    num_cpu_cores_per_socket: 2
    num_cpus: 4

 

check_log.txt

run_log.txt

[Bojan Vitnik1709158530]

Your playbook and logs look fine to me. Network name is properly quoted. This could be XenAPI or Python issue.

 

If it happens to fail again, please catch an error message. That's the only way I can get any clue on what happened.

 

P.S. Just an advice. Add line "gather_facts: no" just under "hosts: xenservers" line. You probably don't need facts of XenServer host itself and it will greatly speed up your playbook.

 

[John Thompson1709154691]

Bojan,

 

Maybe I should start a new thread at this point, but, I had another question- This time on xenserver_guest_facts.

 

 I was trying to use it to build an inventory file of IPs

 

However, I'm not sure how to extract just the IP from the dict of facts in this example.
I can get the network dict blob however...

 

- hosts: xenservers
  gather_facts: no

  tasks:
  - name: Gather facts
    xenserver_guest_facts:
      hostname: 0000
      username: admin
      password: somepassword
      name: jwtConn0
    delegate_to: localhost
    register: facts

  - name: Print facts
    debug:
      msg: "{{facts.instance.networks}}"
  
  - local_action: 
      module: copy 
      content: "{{ facts.instance.networks }}"
      dest: /home/johntho/facts.txt

 

The IP seems to be a list within another dict?

[Bojan Vitnik1709158530]

I usually use Python notation for dicts so it looks like this:

{{ facts['instance']['networks'][0]['ip'] }}

Thats for first (zeroth) network interface.

 

For Python object notation, I think it should look like this:

{{ facts.instance.networks[0].ip }}

 

[John Thompson1709154691]

Bojan,

 

I may have found another small bug.

 

When force deleting vms, you often get this error.

"msg": "XAPI ERROR: ['HANDLE_INVALID', 'VM', 'OpaqueRef:67e1d4db-03ab-462e-9ed9-ec24323cd63b']"

 

I captured the -vvv output.

However, the machines do delete.

 

I also thought about opening an issue here, but, I wasn't sure if thats the proper place...

https://github.com/ansible/ansible/issues

 

See attacehd.

ansible_xapi_errors.txt

 

This was on xenserver 7.6 with latest patches (at least for March anyway).

 

Ansible play:

- hosts: xenservers
  gather_facts: no
  vars_files:
    - /home/johntho/ansible-dev/cqe/ansible_goodies/group_vars/vars/machines.yml

  tasks:
  - name: Create VMs from a template
    xenserver_guest:
      hostname: 
      username: 
      password: 
      validate_certs: no
      #folder: /home/testvms
      name: '{{item.value.hostname}}'
      state: absent
      force: yes
      template: 'JWTWin2016Temp_AnsibleVM_UK'
      #template_uuid: 506e4d22-ade3-69fe-cf40-2bd251fede57
      disks:
      - size_gb: '100'
        name: '{{item.value.hostname}}'
        sr: 'Cam-ss-2 SMB virtual storage'
      linked_clone: yes
      hardware:
        num_cpus: '{{item.value.num_cpus}}'
        num_cpu_cores_per_socket: '{{item.value.num_cpu_cores_per_socket}}'
        memory_mb: '{{item.value.memory_mb}}'
      cdrom:
        type: none
        #iso_name: guest-tools.iso
      networks:
      - name: 'Network 4'
      wait_for_ip_address: yes
    delegate_to: localhost
    register: deploy
    loop: "{{ Machines | dict2items }}"

 

[Bojan Vitnik1709158530]

Yes, https://github.com/ansible/ansible/issues is the place to report any bugs. Please report it there so it can be tracked properly.

 

This bug seems to be related to "wait_for_ip_address" which is invalid/meaningless when deleting VM. I'll fix it soon.

 

[Bojan Vitnik1709158530]

johntho,

 

A quick question for you. In you opinion, should I just ignore "wait_for_ip_address" when used with "state: absent" or should I ignore it and display a warning? Warnings can be annoying too even if they don't cause task to fail.

 

Also what if "wait_for_ip_address" is used with "state: present" and VM is halted? I think module should still fail with an error message in that case.

[John Thompson1709154691]

My first thought is...

 

Any of those, "where it doesn't make sense scenarios", just have ansible validate the playbook and fail.

 

Don't even allow wait_for_ip to be set to yes, when you are deleting.  That way the user knows, how to build the playbook properly.

 

I suppose you could ignore it, but, philosophically I'm not so sure I like that, because its trying to fix an "illogical" set of instructions to ansible.

[Bojan Vitnik1709158530]

That was my intention from start - to fail on any "illogical set of instructions" and if not for a small overlook in code, it would show a proper error message in your case (state: absent and wait_for_ip_address: yes) instead of a generic XAPI error message.

 

But, looking at your playbook made me wonder. So you basically made a task with a bunch of module parameters, some values taken from variables, deployed some VMs and then decided you want to destroy them. Then you just flipped "state: poweredon" -> "state: absent" and forgot to comment out "wait_for_ip_address" or make it "no". That is rather a common use case. More so if you happen to use "state: {{ state_var }}" to dynamically assign some state to a VM. Then you would also need to make "wait_for_ip_address" depend on same variable. I've talked to some Ansible devs and they think it's better to ignore "wait_for_ip_address" when it does not make any sense (like when VM is destroyed). Some Ansible modules already implement this behavior.

 

The problem boils down to this. "state: poweredon" and "wait_for_ip_address" shouldn't even be there. xenserver_guest_powerstate module should be used instead. It's much more straight forward. Early in module development I was advised not to support power states in xenserver_guest module exactly because of problems described here but I thought it was better to support at least "poweredon" so you can deploy and power on your VM in same task. Maybe I was wrong :D.

[John Thompson1709154691]

Well I would say its easy enough to remove the poweredon feature out of the guest module and put it in the powerstate one.

I didn't even know there was a powerstate one until now, but, I'm just learning still.

 

Like any "system" design, I think you just have to get the overall opinion of the users as you go.

I would think once this hits the stable version, you will see a lot more usage.

Xen (not just xenserver), is still pretty popular I think.

[John Thompson1709154691]

Also... officially I think citrix xenserver team only supports their current version and 1 to 2 LTSR versions, if I remember right.

That is if I'm reading this correctly:

https://www.citrix.com/support/product-lifecycle/milestones/citrix-hypervisor.html

 

So it may be a waste of your time to try and get 7.2, 7.3, 7.4, 7.5 to work?

 

Current version is 7.6

[Bojan Vitnik1709158530]

Any feedback... good... bad?