Jump to content


Photo

Small pool R230 with QNAP NFS

Started by Mike Maddison , 17 March 2017 - 06:11 PM
12 replies to this topic

Mike Maddison Members

Mike Maddison
  • 154 posts

Posted 17 March 2017 - 06:11 PM

Greetings,
 
I'm setting up a small pool for about 40 small VMs to run on a set of 3 Dell servers.  These are pretty lightweight VMs for web servers and they're not mission critical.  We already have some Qnap storage servers that we'll be splitting the storage loads onto.  I was hoping to get some tips.  The layout is pretty basic I think, just three R230 Servers with 4 NICs bonded going into a Dell switch and then into the 2 QNAP servers.  I'm thinking of using NFS for the storage network.
 
3 x Dell 230 PowerEdge Servers
Intel® Xeon® E3-1220 v5
64GB RAM
2 on board GBe NIC
4 additional GBe NIC (Intel 350)
1 TB HDD (For OS - cheapest option)
4 x GBe NICs Bonded for storage network
1 GBe NIC dedicated to management network
1 GBe NIC dedicated to public Internet
 
Dell X1052 Switch
 
NAS 1
Qnap TS-1635
12 x 1TB WD HDD
4 x SSD 850 EVO
2 Port 1GBe Network Card add-on (4 total)
4 x GBe NICs Bonded for storage network
 
NAS 2
Qnap TS-453A
4x2TB WD2005FBYZ
4 x GBe NICs Bonded for storage network

 



Alan Lantz Members

Alan Lantz
  • 6,989 posts

Posted 17 March 2017 - 06:18 PM

Keeping with NFS is what I would do, no reason to overly complicate with iSCSI, especially with a single switch. For your nics I would do bonding two nics for storage/management/public internet (I assume this is what we typically call VM traffic).  

 

--Alan--



Mike Maddison Members

Mike Maddison
  • 154 posts

Posted 17 March 2017 - 06:54 PM

Hello Alan,

For bonding do you mean each usage is bonded separately like:
2 Bonded for Management

2 Bonded for Public
2 Bonded for Storage

Or do you mean use one bonded pair for all of the above combined?

I care more about the storage performance than either management or public.  I'm hoping to 100MB/s out of the storage.  The overall disk usage is not a lot but I'd like it to be reasonably fast.

Currently I only dedicated one NIC to management and one to Public and it's been quite reliable and the performance is great so I'm happy to continue that.  Hopefully that makes sense!

 

Thanks for the confirmation on NFS.  



Tobias Kreidl CTP Member

Tobias Kreidl
  • 18,294 posts

Posted 17 March 2017 - 11:46 PM

Separate bonds for each of the three, if you have 6 NICs. The goal is to isolate the network traffic as much as possible into each one of these distinct areas.

 

-=Tobias



Alan Lantz Members

Alan Lantz
  • 6,989 posts

Posted 18 March 2017 - 12:40 AM

Yes, 3 separate bonds, sorry I lacked clarity on that. The bonds for storage and management are more for availability than for throughput. A second switch for switch redundancy would also be desirable.

 

--Alan--



Mike Maddison Members

Mike Maddison
  • 154 posts

Posted 18 March 2017 - 04:06 PM

Great, thanks for the clarity.  I think I'll use a separate switch for storage. I'm feeling confident about this configuration.  Hopefully the performance of the equipment will be reasonable.



Tobias Kreidl CTP Member

Tobias Kreidl
  • 18,294 posts

Posted 20 March 2017 - 12:28 AM

Qnap and Synology make pretty nice drive configurations, especially the smaller ones that are often more than enough and way less costly than something with 8 or 12 or more drive bays.

 

-=Tobias



Mike Maddison Members

Mike Maddison
  • 154 posts

Posted 20 March 2017 - 09:20 PM

That's great to hear.  I've had good luck with most Qnap devices over the years mostly as backup/repositories.  
I spoke to Dell today and apparently I can have as many as 18 network ports in an R230.  I've added another pair to each machine so I can pair up all of the adapters as suggested and still have 4 for storage.

2 Bonded for Management

2 Bonded for Public
4 Bonded for Storage

 

 



Tobias Kreidl CTP Member

Tobias Kreidl
  • 18,294 posts

Posted 21 March 2017 - 12:08 AM

Mike,

That sounds like a reasonable network configuration!

 

-=Tobias



Mike Maddison Members
  • #10

Mike Maddison
  • 154 posts

Posted 08 April 2017 - 03:34 AM

Ok.  So I have the R230's set up in a pool for testing. I'm testing performance with different combinations of bonded NICs on both the QNAP and the R230's and finding strange results.

Pool1.PNG

  •  I've set up 6 identical CentOS7 VMs.  1 on Server1, 3 on Server2 and 2 on Server3.
  • Each server has 4 Intel 350 NICs bonded in LACP.
  • Each 4 ports matching each 4 NICS on the server are configured with LACP on the same Key in the switch.  

Server1 LACP Bond in Switch.PNG

 

 

To see the performance I'm watching the stats on the QNAP performance graphs and reading the bytes from dd:
dd if=/dev/zero of=/root/testfile bs=1200M count=1 oflag=direct; cat /root/testfile; rm -f /root/testfile

This command tells me one direction results but the graph on the QNAP shows both so I can see well what's happening.  XenServer appears to only use 1 of the NICs for all VMs configured on the host server.  If there's one VM I get 114MB/s  If there's 2 I get 60MB/s and if there's three VMs I get 33MB/s.  

If I try just doing the 3 VMs on one host I still only get 33MB/s.  The Qnap has 3 NICs bound so I am expecting to see something closer to 3 x 114MB/s instead.

 

I've tried switching the QNAP from LACP to Balance-rr or Balance-alb and so far the graphing suggests that balance-rr is giving the best results on the QNAP side, but XenServer side doesn't take advantage of the positive change. First-qnap-rr_Second-qnap-lacpMAC.PNG

I've tried switching from 4 bound NICs per machine to just 2 and it makes no difference either.  It seems like either XenServer sticks to just one NIC no matter which bonding mode I pick, or the switch has faulty LACP.

I've tried to condense this here as I've done about 8 hours of tests on this now so there's a lot more I've done as well.  Any ideas on what I should do to try and get the combined NICs to actually load balance?  I know I need to test a different switch but unfortunately I don't have one handy right now so I can still test other ideas until a new one gets here.

Thanks.


 

 

 

Attached Thumbnails

  • XenServer Networking.PNG


Alan Lantz Members
  • #11

Alan Lantz
  • 6,989 posts

Posted 08 April 2017 - 03:47 AM

LACP for storage is more for failover and availability that bandwidth. What I suspect is happening is you QNAP is only presenting itself as a single ip address and/or a single mac address. Therefore, you will only ever use one nic when you are accessing you QNAP storage. You do do a poor man's load balancing of sorts and depending on the device maybe you could assign 4 different NFS exports to each of those QNAP interfaces so when you access them you are spreading the load across your nics. 

 

--Alan--



Mike Maddison Members
  • #12

Mike Maddison
  • 154 posts

Posted 08 April 2017 - 08:03 PM

Ok. Thanks for help me out!

It seems I'm learning a lesson the hard way here.  Very frustrating!  Take this for example:

Misleading-Bond-Speeds.PNG

This seems to be falsely advertising that I have a 4000Mbps connection when really I have just 1000Mbps with 3 spares.  So before I ordered my equipment I was researching all over about aggregating gigabit connections and it seemed like this was a common and normal thing to do to gain speed and redundancy.   In fact testing on the Qnap with has even shown me that the Qnap using Balance-rr actually achieves this just fine.  I've even successfully unplugged one of the 3 Trunked NICs and it handles the dropped connection just fine.
RR-unplugged-NIC.PNG

So bad on me for not buying just one machine and testing the XenServer side properly first.  Had I discovered this in testing I would have probably (reluctantly) spent the extra $4-5000 to move to 10Gbe copper or opted to do things differently.  I got excited when I saw 2000Mbps & 4000Mbps in the XenServer NIC screen and lept to conclusions.  Funny fact for me is that this is why I've always stuck to internal storage using RAID cards.  The #1 reason I'm even trying SAN is that XenServer (Linux in general?) support for RAID cards seems to be out of fashion.

Moving forward now I have to figure out how I'm going to make the best of this.  I had hoped for higher speeds but I can make due with 1000Mbps but only if I have at least 2 or 4 x 1000Mbps available on each machine.  1000Mbps / 20VMs is not good.  4000Mbps / 20VMs is fine.

So now I'm looking for ideas.  The QNAP only has 4 NICs and I was hoping to reserve one for management, backups, etc.  I don't want to run anything other than storage on the storage network if I can avoid it.  

Option 1: Bind the Qnap NICs in pairs using Balance-rr and then bind the NICs on XenServer in pairs LACP, create 2 storage repositories, and manually split the VMs across the repositories?  The problem I see with this is I don't know what machines each VM will start on in the pool so I could end up with all VMs on Storage A being on the same VM and then we're back down to 1000Mbps and one bond is sitting there doing nothing.

Option 2:  Buy more Qnaps.  Similar issues as Option 1.

Option 3:  Try setting up a Balance-rr configuration on XenServer
http://louwrentius.com/achieving-450-mbs-network-file-transfers-using-linux-bonding.html

Not officially supported by XenServer and it might screw something up with the Pool?

Option 4:  Upgrade to 10Gbe - spend $5000+ on switches and cards and 10Gbe ready Qnap.  Problem Solved.

I guess there are some other options out there as well. 
 



Alan Lantz Members
  • #13

Alan Lantz
  • 6,989 posts

Posted 08 April 2017 - 11:03 PM

balance-rr is not supported and your switch has to support that to make it work, but it would give you more throughput for your bonding. Upgrading to 10Gb is an option, but yes, an expensive one (but I wouldn't go back !). If your QNAP supports iSCSI and multipathing you could have two switches and go that route which would double your bandwidth. With one link you can get 80MBps and likely around 150MBps with multipathing. Its harder to setup, but maybe an option. I'm not sure how many VM's you are running or how disk intensive they are, but I would think if you have less than 50 VM's 1Gb links and should suffice.

 

--Alan--




Also tagged with one or more of these keywords: Dell, R230, QNAP, NFS