How to resolve close waits from the OS
Lets look at the close waits and see how many and the offending port.
The following will show us the port and IPof the close wait
netstat -an |grep -i close
tcp 1 0 icudbaup02-priv.xxx.:53494 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53447 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53451 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53452 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53454 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53470 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53415 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53418 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53421 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53435 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53436 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53376 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53400 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53119 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53114 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53115 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53112 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53113 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53096 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53095 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53088 icudbaup01-priv.xxx.:61020 CLOSE_WAIT tcp 1 0 icudbaup02-priv.xxx.:53085 icudbaup01-priv.xxx.:61020 CLOSE_WAIT
The following will show us the offending process (note, this must be done as the root user
sudo lsof -I :53494
cat /usr/local/grid/log/grac2/cssd/ocssd.log | egrep -i 'Removal|evict|30000|network HB'