Hope you are doing all great, for today’s post I wanted to put together some of the commands/troubleshooting I’ve had used with VMware vSAN,

Identify a partitioned node from a VSAN Cluster (Hosts)

What a single partitioned node looks like:

~ # esxcli vsan cluster get
Cluster Information
 Enabled: true
 Current Local Time: 2020-10-25T10:35:19Z
 Local Node UUID: 507e7bd5-ad2f-6424-66cb-1cc1de253de4
 Local Node State: MASTER
 Local Node Health State: HEALTHY
 Sub-Cluster Master UUID: 507e7bd5-ad2f-6424-66cb-1cc1de253de4
 Sub-Cluster Backup UUID:
 Sub-Cluster UUID: 52e4fbe6-7fe4-9e44-f9eb-c2fc1da77631
 Sub-Cluster Membership Entry Revision: 7
 Sub-Cluster Member UUIDs: 507e7bd5-ad2f-6424-66cb-1cc1de253de4
 Sub-Cluster Membership UUID: ba45d050-2e84-c490-845f-1cc1de253de4
~ #

What a full 4-node cluster looks like (no partition)

~ # esxcli vsan cluster get
Cluster Information
 Enabled: true
 Current Local Time: 2020-10-25T10:35:19Z
 Local Node UUID: 54188e3a-84fd-9a38-23ba-001b21168828
 Local Node State: MASTER
 Local Node Health State: HEALTHY
 Sub-Cluster Master UUID: 54188e3a-84fd-9a38-23ba-001b21168828
 Sub-Cluster Backup UUID: 545ca9af-ff4b-fc84-dcee-001f29595f9f
 Sub-Cluster UUID: 529ccbe4-81d2-89bc-7a70-a9c69bd23a19
 Sub-Cluster Membership Entry Revision: 3
 Sub-Cluster Member UUIDs: 54188e3a-84fd-9a38-23ba-001b21168828, 545ca9af-ff4bfc84-dcee-001f29595f9f, 5460b129-4084-7550-46e1-0010185def78, 54196e13-7f5f-cba8-
5bac-001517a69c72
 Sub-Cluster Membership UUID: 80757454-2e11-d20f-3fb0-001b21168828

Check if there are ongoing/stuck resync operations (hosts)

while true;do echo " ****************************************** " ; echo "" > /tmp/resyncStats.txt ;cmmds-tool find -t DOM_OBJECT -f json |grep uuid |awk -F \" '{print $4}' |while read i;do pendingResync=$(cmmds-tool  find -t DOM_OBJECT -f json -u $i|grep -o "\"bytesToSync\": [0-9]*,"|awk -F " |," '{sum+=$2} END{print sum / 1024 / 1024 / 1024;}');if [ ${#pendingResync} -ne 1 ]; then echo "$i: $pendingResync GiB";fi;done |tee -a /tmp/resyncStats.txt;total=$(cat /tmp/resyncStats.txt |awk '{sum+=$2} END{print sum}');echo "Total: $total GiB" |tee -aa /tmp/resyncStats.txt;total=$(cat /tmp/resyncStats.txt  |grep Total);totalObj=$(cat /tmp/resyncStats.txt|grep -vE " 0 GiB|Total"|wc -l);echo "`date +%Y-%m-%dT%H:%M:%SZ` $total ($totalObj objects)" >> /tmp/totalHistory.txt; echo `date ` ; sleep 60 ;done

No resync operations

Total: 0 GiB
Sun Oct 25 00:30:29 UTC 2020

Pending resync operation:

ba8d625b-6457-b3ac-6da9-e4434b016608: 212.125 GiB
Total: 212.125 GiB
Sun Oct 25 00:43:29 UTC 2029

Check the state of the components (Hosts)

cmmds-tool find -f python | grep CONFIG_STATUS -B 4 -A 6 | grep 'uuid\|content' | grep -o 'state\\\":\ [0-9]*' | sort | uniq -c 

Good state:

772 state\": 7 --healthy 

Good state:

2 state\": 13 --inaccessible objects 
6 state\": 15 --absent/degraded 

Find the UUID of the healhty (state 7), inaccessible objects (state 13) and absent/degraded (stage 15). Replace the stage number to filter out the results per state, for example, here are the two inaccessible objects:

cmmds-tool find -f python | grep CONFIG_STATUS -B 4 -A 6 | grep 'uuid\|content' | grep 'state\\\": 13' -B 1 | grep uuid | cut -d "\"" -f4 
7328b659-9d15-cb40-68eb-a81e846157fb 
e52bb359-d0e0-3ce2-7070-a81e846153d9 

If you find objects on state 13 or 15, you better contact VMware ASAP to see if they can be recovered.

Disk Balance Test (Hosts)

esxcli vsan health cluster get -t "vSAN Disk Balance"

A balanced scenario

vSAN Disk Balance green
Overview
Metric Value
---------------------------------
Average Disk Usage 10 %
Maximum Disk Usage 12 %
Maximum Variance 4 %
LM Balance Index 2 %

Disk Balance
Host Device Rebalance State Data To Move
------------------------------------------------------------------------------------------------------------------------

A not balanced scenario

vSAN Disk Balance yellow

Overview
Metric Value
---------------------------------
Average Disk Usage 27 %
Maximum Disk Usage 45 %
Maximum Variance 37 %
LM Balance Index 19 %

Disk Balance
Host Device Rebalance State Data To Move
------------------------------------------------------------------------------------------------------------------------
172.16.11.246 Local TOSHIBA Disk (naa.50000398b821a7c5) Proactive rebalance is in progress 420.5244 GB

Verify which devices from a particular host are in which disk groups  (Hosts)

[root@esxi-1:~] vdq -iH 

Good mapping

Mappings: 
DiskMapping[0]: 
SSD: naa.58ce38ee2016ffe5 
MD: naa.5002538a4819e540 
MD: naa.5002538a4819e510 
MD: naa.5002538a4819e3e0 

DiskMapping[2]: 
SSD: naa.58ce38ee2016fe55 
MD: naa.5002538a48199ca0 
MD: naa.5002538a48199e20 
MD: naa.5002538a48199e00 

Something is not right… (missing disks)

Mappings: 
DiskMapping[0]: 
SSD: naa.58ce38ee2016ffe5 
MD: naa.5002538a4819e3e0 

DiskMapping[2]: 
SSD: naa.58ce38ee2016fe55 
MD: naa.5002538a48199ca0 
MD: naa.5002538a48199e20 
MD: naa.5002538a48199e00 

Find devices that are in PDL state (Hosts)

Run the command vdq –qH and search for IsPDL? if the value is equal to 1 the device is in PDL state.

[root@esxi-04:~] vdq -qH
DiskResults:
 DiskResult[0]:
 Name: naa.600508b1001c4b820b4d80f9f8acfa95
 VSANUUID: 5294bbd8-67c4-c545-3952-7711e365f7fa
 State: In-use for VSAN
 ChecksumSupport: 0
 Reason: Non-local disk
 IsSSD?: 0
IsCapacityFlash?: 0
 IsPDL?: 0
 <<truncated>>
 DiskResult[18]:
 Name:
 VSANUUID: 5227c17e-ec64-de76-c10e-c272102beba7
 State: In-use for VSAN
 ChecksumSupport: 0
 Reason: None
 IsSSD?: 0
IsCapacityFlash?: 0
 IsPDL?: 1          = Device is in PDL state

Determine if there are enough reources remaining in the cluster to rebuild by simulating a host failure scenario (RVC – vCenter)

RVC command vsan.whatif_host_failures helps to know if there are the sufficient amount of resources by simulating a host failure, make sure to connect to your vCenter-RVC.

/localhost/virtuallyvtrue/computers> vsan.whatif_host_failures 0
Simulating 1 host failures:
+-----------------+-----------------------------+-----------------------------------+
| Resource | Usage right now | Usage after failure/re-protection |
+-----------------+-----------------------------+-----------------------------------+
| HDD capacity | 5% used (3635.62 GB free) | 7% used (2680.12 GB free) |
| Components | 3% used (11687 available) | 3% used (8687 available) |
| RC reservations | 0% used (3142.27 GB free) | 0% used (2356.71 GB free) |
+-----------------+-----------------------------+-----------------------------------+
/localhost/virtuallyvtrue/computers>

Testing Virtual SAN functionality – deploying VMs – (RVC – vCenter)

/localhost/virtuallyvtrue/computers> diagnostics.vm_create --
datastore ../datastores/vsanDatastore --vm-folder ../vms/Discovered\ virtual\ machine 0
Creating one VM per host ... (timeout = 180 sec)
Success
/localhost/virtuallyvtrue/computers>

Start and Stop proactive rebalances (RVC – vCenter)

/vcsa-01.virtuallyvtrue.local/virtuallyvtrue/computers> vsan.proactive_rebalance_info 0
2020-10-25 14:14:27 +0000: Retrieving proactive rebalance information from host esxi-02.virtuallyvtrue.local ...
2020-10-25 14:14:27 +0000: Retrieving proactive rebalance information from host esxi-04.virtuallyvtrue.local ...
2020-10-25 14:14:27 +0000: Retrieving proactive rebalance information from host esxi-01.virtuallyvtrue.local ...
2020-10-25 14:14:27 +0000: Retrieving proactive rebalance information from host esxi-03.virtuallyvtrue.local ...



Proactive rebalance is not running!
Max usage difference triggering rebalancing: 30.00%
Average disk usage: 5.00%
Maximum disk usage: 26.00% (21.00% above mean)
Imbalance index: 5.00%
No disk detected to be rebalanced

Start the proactive rebalance - vsan.proactive_rebalance -s 0
/vcsa-01.virtuallyvtrue.local/virtuallyvtrue/computers> vsan.proactive_rebalance -s 0
2020-10-25 14:15:05 +0000: Processing Virtual SAN proactive rebalance on host esxi-02.virtuallyvtrue.local ...
2020-10-25 14:15:05 +0000: Processing Virtual SAN proactive rebalance on host esxi-04.virtuallyvtrue.local ...
2020-10-25 14:15:05 +0000: Processing Virtual SAN proactive rebalance on host esxi-01.virtuallyvtrue.local ...
2020-10-25 14:15:05 +0000: Processing Virtual SAN proactive rebalance on host esxi-03.virtuallyvtrue.local ...


Proactive rebalance has been started!

/vcsa-01.virtuallyvtrue.local/virtuallyvtrue/computers> vsan.proactive_rebalance_info 0
2020-10-25 14:15:11 +0000: Retrieving proactive rebalance information from host esxi-02.virtuallyvtrue.local ...
2020-10-25 14:15:11 +0000: Retrieving proactive rebalance information from host esxi-04.virtuallyvtrue.local ...
2020-10-25 14:15:11 +0000: Retrieving proactive rebalance information from host esxi-01.virtuallyvtrue.local ...
2020-10-25 14:15:11 +0000: Retrieving proactive rebalance information from host esxi-03.virtuallyvtrue.local ...

Proactive rebalance start: 2020-10-25 14:13:10 UTC
Proactive rebalance stop: 2020-10-25 14:16:17 UTC
Max usage difference triggering rebalancing: 30.00%
Average disk usage: 5.00%
Maximum disk usage: 26.00% (21.00% above mean)
Imbalance index: 5.00%
No disk detected to be rebalanced

Stop the proactive rebalance – vsan.proactive_rebalance -o 0

/vcsa-01.virtuallyvtrue.local/virtuallyvtrue/computers> vsan.proactive_rebalance -o 0
2020-10-25 14:15:45 +0000: Processing Virtual SAN proactive rebalance on host esxi-01.virtuallyvtrue.local ...
2020-10-25 14:15:45 +0000: Processing Virtual SAN proactive rebalance on host esxi-02.virtuallyvtrue.local ...
2020-10-25 14:15:45 +0000: Processing Virtual SAN proactive rebalance on host esxi-04.virtuallyvtrue.local ....
2020-10-25 14:15:45 +0000: Processing Virtual SAN proactive rebalance on host esxi-03.virtuallyvtrue.local ...
Proactive rebalance has been stopped!

Verify the health of the Virtual SAN Cluster by their objects’ states (RVC – vCenter)

/vcsa-01.virtuallyvtrue.local/virtuallyvtrue/computers> vsan.obj_status_report 0
2020-10-25 12:47:04 +0000: Querying all VMs on VSAN ...
2020-10-25 12:47:04 +0000: Querying all objects in the system from esxi-02.virtuallyvtrue.local ...
2020-10-25 12:47:04 +0000: Querying all disks in the system from esxi-02.virtuallyvtrue.local ...
2020-10-25 12:47:05 +0000: Querying all components in the system from esxi-02.virtuallyvtrue.local ...
2020-10-25 12:47:05 +0000: Querying all object versions in the system ...
2020-10-25 12:47:06 +0000: Got all the info, computing table ...
Histogram of component health for non-orphaned objects

+-------------------------------------+------------------------------+
| Num Healthy Comps / Total Num Comps | Num objects with such status |
+-------------------------------------+------------------------------+
| 11/11 (OK)						  | 30                           |
| 3/3 (OK) 							  | 6                            |
+-------------------------------------+------------------------------+
Total non-orphans: 36

Histogram of component health for possibly orphaned objects

+-------------------------------------+------------------------------+
| Num Healthy Comps / Total Num Comps | Num objects with such status |
+-------------------------------------+------------------------------+
+-------------------------------------+------------------------------+
Total orphans: 0

Total v1 objects: 0
Total v2 objects: 0
Total v2.5 objects: 0
Total v3 objects: 0
Total v5 objects: 36

Check if the unicast agents are able to see all nodes (Hosts)

[root@esxi-03:~] esxcli vsan cluster unicastagent list
NodeUuid                              IsWitness  Supports Unicast  IP Address    Port  Iface Name
------------------------------------  ---------  ----------------  -----------  -----  ----------
5bcf2026-d387-c17e-9755-246e96ccc730          0              true  10.62.45.71  12321
5aed2ebf-7377-e34e-95a7-246e96ccc790          0              true  10.62.45.73  12321
5aedafb5-2826-7ba4-7d56-246e96ccd6b0          0              true  10.62.45.70  12321
[root@b-p-vxrail-03:~]

Check if all Virtual SAN hosts are in the same network segment

[root@esxi-03:~] esxcli network ip neighbor list -i vmk3
Neighbor     Mac Address        Vmknic   Expiry  State  Type
-----------  -----------------  ------  -------  -----  -------
10.62.45.73  00:50:56:67:c0:87  vmk3    625 sec         Unknown
10.62.45.71  00:50:56:65:72:29  vmk3    627 sec         Unknown
10.62.45.70  00:50:56:60:a6:8e  vmk3    628 sec         Unknown

Validate if the vSAN is communication with each other through the UDP port 12321 <Requires vSAN’s vmk>

[root@esxi-h03:~] tcpdump-uw -i vmk3 port 12321
tcpdump-uw: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmk3, link-type EN10MB (Ethernet), capture size 262144 bytes
22:47:22.734356 IP 10.11.107.22.12321 > 10.11.107.20.12321: UDP, length 400
22:47:27.736888 IP 10.11.107.22.12321 > 10.11.107.23.12321: UDP, length 400
22:47:27.737094 IP 10.11.107.22.12321 > 10.11.107.21.12321: UDP, length 400
22:47:27.737344 IP 10.11.107.21.12321 > 10.11.107.22.12321: UDP, length 200
22:47:27.737396 IP 10.11.107.20.12321 > 10.11.107.22.12321: UDP, length 464

Check the disk groups available

[root@esxi-03:~] esxcli vsan storage list | grep "VSAN Disk Group UUID:" | sort | uniq -c
      4    VSAN Disk Group UUID: 520243a4-8bb7-c95a-a6e9-28655e26febd
      4    VSAN Disk Group UUID: 52f0561b-e656-8f9a-17da-adb120a1544a

Check disk’s location

[root@esxi-03:~]esxcli storage core device physical get -d naa.5002538a9823cfd0
   Physical Location: enclosure 1, slot 6

Hope you enjoyed this post and don’t forget to share and comment.