File System aparece lleno pero no tiene ficheros

En algunas ocasiones nos damos cuenta que en un servidor tenemos problemas de espacio y al realizar un du dentro del file system indicado realmente no esta ocupando lo que dice. Para solventar esto vamos a realizar una serie de comprobaciones para ver que proceso esta ocasionando el problema .

Entramos al servidor y comprobamos al realizar un df que esta ocuado gran cantidad del espacio.

bash

rokitoh@redorbita:/# df -h
Filesystem Size Used Avail Use% Mounted on

/dev/dm-0 12G 2.0G 9.3G 18% /

/dev/mapper/vg_display-lv_tmp 5.4G 12M 5.1G 1% /tmp

/dev/mapper/vg_display-lv_opt 10G 9.0G 520M 95% /opt

/dev/mapper/vg_display-lv_home 5.4G 991M 4.2G 20% /home

/dev/mapper/vg_display-lv_usr 9.3G 3.2G 5.7G 36% /usr

/dev/mapper/vg_display-lv_var 13G 12G 1G 16% /var

Accedemos al file system y comprobamos que realmente no tenemos 12GB ocupados tal y como nos indica el DF

bash

rokitoh@redorbita:/var# du -hs *
4.0K agentx

1.4G app

9.9M backups

202M cache

4.0K crash

339M lib

4.0K local

0 lock

14M log

16K lost+found

4.0K mail

4.0K opt

0 run

36K spool

12K tmp

Vemos si se ha quedado algun proceso colgado de borrado en cache:

bash

rokitoh@redorbita:/var# ls -l /proc/*/fd/* | grep deleted | grep /var | awk ‘{print $9}’ | xargs -i ls -lhL {}
-rw-rw-rw- 0 root root 11G Aug 4 11:02 /proc/12673/fd/7

-rw-r–r– 0 tomcat7 tomcat7 2.7K Jul 24 13:09 /proc/1521/fd/12

-rw-r–r– 0 tomcat7 tomcat7 0 Jul 24 13:09 /proc/1521/fd/13

-rw-rw-rw- 0 root root 11G Aug 4 11:02 /proc/15247/fd/7

-rw-rw-r– 0 root root 22K Jul 24 21:14 /proc/1680/fd/16

-rw-rw-r– 0 root root 22K Jul 24 21:14 /proc/1680/fd/3

-rw-rw-r– 0 root root 22K Jul 24 21:14 /proc/1680/fd/5

-rw-rw-r– 0 root root 22K Jul 24 21:14 /proc/1680/fd/7

-rw-rw-r– 0 root root 22K Jul 24 21:14 /proc/1680/fd/9

-rw-r—– 0 root root 151 Jul 24 13:09 /proc/1/fd/11

mediante el comando lsof vemos cuantos procesos de borrado se han quedado colgados:

bash

rokitoh@redorbita:/var#lsof | grep «/var» | grep deleted

init 1 root 11w REG 252,4 151 651557 /var/log/upstart/systemd-logind.log.1 (deleted)

java 1521 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1522 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1522 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1523 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1523 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1524 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1524 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1525 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1525 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1526 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1526 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1527 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1527 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1528 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1528 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1529 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1529 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1530 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1530 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1531 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1531 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1532 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1532 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1533 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1533 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1534 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1534 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1535 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1535 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1536 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1536 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1541 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1541 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1762 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1762 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1763 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1763 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1764 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1764 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

ruby 12673 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby-time 12673 12675 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12676 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12677 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12678 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12679 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12702 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12726 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12727 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12728 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12729 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12730 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12731 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12732 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12733 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12734 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby-time 15247 15249 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15251 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15252 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15253 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15254 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15271 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15286 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15287 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15288 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15289 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15290 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15291 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15292 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15293 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15294 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

Como podemos observar el tomcat y una aplicación estan bloqueando el borrado de multiples logs.

Reiniciamos el servicio tomcat y el aplicativo y comprobamos que se libera el espacio.

bash

rokitoh@redorbita:/# df -h
Filesystem Size Used Avail Use% Mounted on

/dev/dm-0 12G 2.0G 9.3G 18% /

/dev/mapper/vg_display-lv_tmp 5.4G 12M 5.1G 1% /tmp

/dev/mapper/vg_display-lv_opt 10G 9.0G 520M 95% /opt

/dev/mapper/vg_display-lv_home 5.4G 991M 4.2G 20% /home

/dev/mapper/vg_display-lv_usr 9.3G 3.2G 5.7G 36% /usr

/dev/mapper/vg_display-lv_var 13G  2.0G   11G  16%

un saludo.

:wq!

Articulos relacionados

OpenCode Enterprise: Implementación para Platform Engineering

Implementación de un Bastión de Seguridad con Kernel Hardening (Grsecurity/Self-Protection) y Cifrado de Memoria en Linux

Gestión de Versiones de Terraform con tfenv

Comentarios