Monitoring VMware Cloud on AWS vCenter alarms within vRealize Log Insight Cloud

vRealize Log Insight Cloud (vRLIC) gives us unified visibility across public and private clouds through robust log aggregation, analytics and faster root cause determination. The great news is that it is also included as part of your subscriptions to VMware Cloud on AWS (VMC) and as of VMworld Europe 2019 also now includes additional features and functionality:

Check out the official blog article here

With the core version, VMC customers now get access to real-time reporting which is what we are going to look at today. In a previous post, I talked using creating VMC vCenter alarms and setting notifications for specific events. The event we were particularly interested in was if the VSAN datastore reaches 70% utilisation because at 75% a new host will be added to ensure we stay within SLA. In the example, we used an alert that would trigger if the datastore was less than 100% utilised as this would ensure the alert would always trigger. We are now going to use vRLIC to query for the alert and then send us a notification once it has been triggered.

vRLIC is automatically configured to ingest logs for VMC and can be accessed via the Cloud Services Portal so there is nothing that you need to do to start using it, simply launch the application:

The initial landing page gives us a great overview of recent alerts and event observations over the last hour. It is definitely worth spending some time with vRLIC to see the level of information and default alerts that are available to VMC customers:

I've re-created the vCenter alert that triggers when VSAN Datastore Usage is below 100 percent just to ensure the log is sent instantly to vRLIC:

If we explore the logs and query for the alert name VSAN Datastore Usage is below 100 percent with a timeframe of the last ten minutes then we can see the triggered alert. We know this is the alert because we can see it change its state from Gray to Red:

Now that we have the query needed we can click on the save icon:

Give the query a suitable name and description and click Save:

Once we save the query we can click on the alert icon to create an alert based on the query:

Give the alert a suitable name and description and click Save:

The Alert Definition screen will appear which will allow you to customise the alert. Remember to add the Email address where you would like the alert to be sent, set the trigger to evaluate on every match and enable it before clicking on the save icon:

It's also worth sending a test alert to ensure you receive the notification:

Hopefully, if everything is set up correctly next time the alert is triggered you should receive a notification via email and also see it in the Recent Alerts:

With the example above we are triggering an email notification when we see any log that has been ingested and contains the text, VSAN Datastore Usage is below 100 percent. This is not ideal because it will also trigger the notification when any changes to the alarm are made i.e. reset to green or disabling and re-enabling. I tried testing this on alarm name as well as the text gray to red which is sent when the state of the alarm changes but during testing, I noticed that this was not always sent on certain alarm configuration changes which I have fed back to the BU and will be addressed in the future. I don't envisage these changes being made regularly in customer environments so it should not cause an influx of emails.

A point that I would like to highlight is that vRLIC currently runs out of one of the US AWS regions so if there are issues with logs residing outside of the UK/EU then please get in touch and I will continue to raise this internally.

VMware Cloud on AWS vCenter Alarms

A lot of VMware customers use vCenter alarms and notifications for monitoring their on-premises environment and the same goes for when they move to VMware Cloud on AWS. I was recently asked by a customer on how they can receive a notification when the VSAN storage capacity is getting close to 75% full. For those who are not aware we need 25% slack space for VSAN and will automatically add a host once storage utilisation reaches 25%, which is documented in the Service Level Agreement for VMware Cloud on AWS.

Creating an alarm can either be completed directly in the vCenter client or via the Cloud Gateway Appliance. Simply browse to the WorkloadDatastore, select Configure and then Alarm Definitions. From here you need to Add a new alarm:

Give the alarm a suitable Name and Description and click Next:

In the example, I am setting the alarm to be triggered if the utilization is less than 100% (Which will always be the case) to ensure that I receive a notification. Typically you would set this to is above 70% or whatever threshold you feel comfortable with. Once you have the correct parameters enter the email address to whom you would like the notification sent to and click Next:

Set the email notification if you want to be notified once the condition clears otherwise click Next:

Review your settings and click Next when ready:

Since our alarm was set to trigger if storage utilisation was less than 100% we can see that it triggered straight away within vCenter:

We also received an email notification:

Since currently there is no sender address there is a chance this might get picked up by your email spam filtering software so you might have to create a rule to allow it through. Due to this annoyance in the next article, I will show you how to trigger an alert from vRealize Log Intelligence Cloud.