Filesystem filling up with mastools-xxx files

[Marc Wilhelmi]

Hi,

 

we have a Netscaler/ADC HA-Pair filling up the /var filesystem with files in /var/core/<number>/, all named mastools-<4-digit number> i.e. mastools-2787

every 2 minutes a new file is generated with 7.8M in size

there is no unusual network or cpu load

 

this behavior started 2 days ago and only affects one HA-pair, all others look fine.

 

Any ideas what causes this behavior and how to stop it?

[Christopher Hsu1709152190]

same here, started a couple of days ago, filled up our ADC, had to reboot it in order to get back in and clear some space. 

 

NS13.0 91.13.nc - Azure flavor

[Marc Wilhelmi]

Update:

reboot does not fix the issue here

Killing the process mastools_monit.sh stops this

 

ps -ax | grep mastools_monit.sh
 1458  ??  I      0:00.02 /bin/sh mastools_monit.sh

 

today the process is running again on both nodes, but only on one node is just one new mastools-xxx file, the other node has no new files.

 

 

 

[JEFF LESLIE]

same here.  Noticed that the files were on both of our HA Paired ADC's and not the stand along one.  Cleaned up the files and now only the secondary node in the pair is having the files generated.  This has only started since we updated to NS13.0.91.13.ns

[Diana Wong]

Following as well.... We are also experiencing the same issue with the /var/core folder filling up with mastools-#### files since upgrading to   NS13.0.91.13.ns

[Marcel Laven1709164016]

Same here, after upgrading both our NS clusters experience this issue. Had to delete all MASTOOLS-XXXX files to get them back up and running.

[Peder Christian Mikkelsen]

Hi. I have seen the same issue with two different customers. Hopefully Citrix will find a solution

[Chris Gundry]

We have the same issue, on a single ADC and then also on a HA pair.

 

From what I can see the mastools scripts updated on 8th of August, then the issue started.

The scripts updated again on 10th of August around 1700.

Then around 0700 on the 11th the issue stopped and no more files being generated...

 

I raised a support call on the 9th, when we saw the issue, but Citrix support as usual are being useless. They have said there is a known issue in 13.0.90.13 and the workaround is:

go to shell

go to /var/mastools/scripts/

type "mastoolsd stop

 

We are on 13.0 91.13, and supposedly the issue still exists in that version. I queried why the issue didn't start when we updated to 13.0 91.13, which was several weeks ago. They have no answer for that. I personally don't believe the issue we are seeing is related to the 'known issue' in 13.0.90.13 as:

1. It started well after the update to 13.0 91.13 was completed.

2. It started shortly after mastools scripts updated themselves

3. It stopped on it's own

 

I asked Citrix to confirm:

1. Why the scripts have updated themselves? We are not familiar with this process.

2. If Citrix know about an issue with the mastools scripts which started on the 8th and has now been resolved somehow with a further updated mastools.

 

As of yet they are not giving anymore info, they are being useless.

[Peder Christian Mikkelsen]

hi. I think I have seen an error that is related to this 13.0.91.13 - and perhaps related to this mastools. After upgrading to this version Netscaler seems to lose the registration to ADM in Citrix Cloud. The Netscaler is configured to use the build-in ADM serviceagent. I have tried to reregister the agent as specified in https://docs.netscaler.com/en-us/citrix-application-delivery-management-service/getting-started/initiate-built-in-agent - but the ADM fails to import the Netscalers with the following error: "Failed to get netscaler version for "ip_service" Error: Error in config/nshostname.  And the ADM was able to communicate before the upgrade.

[James Golden1709160441]

I agree with Chris Gundry.  Something else happened.  It wasn't the update.  We updated on July 18th because of the crit vulnerabilities and I see these files started showing up on August 8th!

 

It also appeared that it "mysteriously" stopped creating the files between August 10th at 23:00 and August 11 02:00 on all 4 of our netscalers.  I am just finding this out yesterday 8/21 when I couldn't login on them.

 

Thanks for posted those steps and dealing with Citrix support, they have been useless for years!  Your post helped me get things right again.

 

 

 

[Gaurav Hirwaniya1709163771]

Dear Members,

 

Kindly follow these steps to fix the issue.

 

1. Clear out old NSTRACE, CORE/CRASH and SupportBundle Files (to ensure enough space on device)
2. Login to NetScaler CLI and type "shell"
3. execute "/var/mastools/scripts/mastoolsd stop"
4. execute "rm /var/core/mastools*"
5. execute " /var/python/bin/python /var/mastools/scripts/mastools_upgrade.py" (upgrade the function manually)

Gaurav Hirwaniya

Senior Escalation Engineer  - NetScaler