August 3, 2021
Table of Contents
Errors with VCF 4.2.1 Configuration Drift Bundle
Precheck Failures Preventing Upgrades
The Issue? “NSX-T Audit Check Failures”
Ordinarily, upgrading VCF versions should be easy. Unfortunately, this isn’t always the case. There are a host of issues that could surface during an upgrade. The focus of this article is the dreaded NSX-T Audit Check Failures.
If the pre-check fails and there is in fact an issue with SDDC Manager’s audit of NSX-T, you will not be able to proceed with the upgrade. Keep in mind, NSX-T is upgraded after vCenter is upgraded.
Identifying The Issue
To confirm the issue you have is the issue that is being resolved in this article, refer to the screen shot and log excerpt below.
The error message you are greeted with in SDDC Manager should look like the screenshot below.
To ensure it is definitely the same issue, use a client like PuTTY and ssh onto SDDC Manager. The log file you will need to look at is the lcm.log file. Issue the command less /var/log/vmware/vcf/lcm/lcm.log, once the log is open press shift + g to scroll to the bottom. For other commands, this URL is handy.
Once you have the log open, the scrolling up a little should reveal the last error. If your log matches the excerpt below, then you likely have the same issue, and this article should be of assistance.
[c.v.e.s.l.p.impl.nsxt.NsxtAuditImpl,pool-6-thread-6] Error auditing NSX-T Cluster sm-nsx01.shank.com with exception {}
com.vmware.evo.sddc.lcm.model.error.LcmException: Failed to load NSX-T Cluster from the Inventory
at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtInventoryLoader.loadNsxtInventory(NsxtInventoryLoader.java:87)
at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtAuditImpl.doAudit(NsxtAuditImpl.java:68)
at com.vmware.evo.sddc.lcm.audit.NsxtAuditService.doAudit(NsxtAuditService.java:120)
at com.vmware.evo.sddc.lcm.audit.NsxtInventoryAuditScheduler.auditNsxtInventory(NsxtInventoryAuditScheduler.java:84)
at com.vmware.evo.sddc.lcm.audit.NsxtInventoryAuditScheduler$$FastClassBySpringCGLIB$$2ae06d0f.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
at org.springframework.cloud.sleuth.instrument.scheduling.TraceSchedulingAspect.traceBackgroundThread(TraceSchedulingAspect.java:73)
at sun.reflect.GeneratedMethodAccessor298.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
at com.vmware.evo.sddc.lcm.audit.NsxtInventoryAuditScheduler$$EnhancerBySpringCGLIB$$474e096c.auditNsxtInventory()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.NoSuchElementException: No value presentThe errors to be mindful of are:
- Error auditing NSX-T Cluster sm-nsx01.shank.com with exception {} com.vmware.evo.sddc.lcm.model.error.LcmException: Failed to load NSX-T Cluster from the Inventory
- Caused by: java.util.NoSuchElementException: No value present
Resolving the NSX-T Audit Check Failure
The issue in this case was that SDDC Manager failed to audit the NSX-T Cluster based on what it thought was deployed. I’ve always said, if you are changing anything that was deployed out of SDDC Manager, be very careful and ensure it is supported to do so.
In this case, I did not take my own advice and at some point renamed one of the host transport nodes in NSX-T. I Removed the FQDN name and changed it to shortname, this was causing the audit check to fail. Refer to the image below.
In this case I had to rename the host back to what it was originally, the FQDN that matches the naming structure of the other hosts. Once that was done, a re-run of the pre-check was then successful.
Conclusion
The underlying issue was the renaming of a SDDC Manager deployed object. In most cases, you are better off not changing anything out of SDDC Manager, unless it is supported to do so. Reverting the change allowed the pre-check to complete successfully. You may or may not have renamed something in NSX, however, if anything was renamed and wasn’t supported, you may see similar issues.
For more VCF Troubleshooting tips check out my Unofficial VCF Troubleshooting Guide
All my other VCF related articles can be found here
