Red Orbita
Home Linux & Systems Cybersecurity Cloud & DevOps Networks & Infrastructure SIEM & Monitoring DFIR & Threat Intel Development & Other All categories Projects About Tools

File System appears full but has no files

Leer en espanol
August 1, 2016 Linux & Systems rokitoh
File System appears full but has no files

Table of contents

On some occasions we realize that on a server we have space problems and when performing a du within the indicated file system it is not really taking up what it says. To solve this, we are going to carry out a series of checks to see what process is causing the problem.

We enter the server and verify by performing a df that a large amount of space is occupied.

Bash
rokitoh@redorbita:/# df -h
Filesystem Size Used Avail Use% Mounted on

/dev/dm-0 12G 2.0G 9.3G 18% /

/dev/mapper/vg_display-lv_tmp 5.4G 12M 5.1G 1% /tmp

/dev/mapper/vg_display-lv_opt 10G 9.0G 520M 95% /opt

/dev/mapper/vg_display-lv_home 5.4G 991M 4.2G 20% /home

/dev/mapper/vg_display-lv_usr 9.3G 3.2G 5.7G 36% /usr

/dev/mapper/vg_display-lv_var 13G 12G 1G 16% /var

We access the file system and verify that we really do not have 12GB occupied as the DF indicates.

Bash
rokitoh@redorbita:/var# du -hs *
4.0K agentx

1.4G app

9.9M backups

202M cache

4.0K crash

339M lib

4.0K local

0 lock

14M log

16K lost+found

4.0K mail

4.0K opt

0 run

36K spool

12K tmp

We see if any process has been left hanging from cache deletion:

Bash
rokitoh@redorbita:/var# ls -l /proc/*/fd/* | grep deleted | grep /var | awk ‘{print $9}’ | xargs -i ls -lhL {}
-rw-rw-rw- 0 root root 11G Aug 4 11:02 /proc/12673/fd/7

-rw-r–r– 0 tomcat7 tomcat7 2.7K Jul 24 13:09 /proc/1521/fd/12

-rw-r–r– 0 tomcat7 tomcat7 0 Jul 24 13:09 /proc/1521/fd/13

-rw-rw-rw- 0 root root 11G Aug 4 11:02 /proc/15247/fd/7

-rw-rw-r– 0 root root 22K Jul 24 21:14 /proc/1680/fd/16

-rw-rw-r– 0 root root 22K Jul 24 21:14 /proc/1680/fd/3

-rw-rw-r– 0 root root 22K Jul 24 21:14 /proc/1680/fd/5

-rw-rw-r– 0 root root 22K Jul 24 21:14 /proc/1680/fd/7

-rw-rw-r– 0 root root 22K Jul 24 21:14 /proc/1680/fd/9

-rw-r—– 0 root root 151 Jul 24 13:09 /proc/1/fd/11

Using the lsof command we see how many deletion processes have been left hanging:

Bash
rokitoh@redorbita:/var#lsof | grep «/var» | grep deleted

init 1 root 11w REG 252,4 151 651557 /var/log/upstart/systemd-logind.log.1 (deleted)

java 1521 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1522 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1522 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1523 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1523 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1524 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1524 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1525 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1525 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1526 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1526 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1527 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1527 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1528 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1528 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1529 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1529 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1530 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1530 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1531 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1531 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1532 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1532 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1533 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1533 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1534 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1534 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1535 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1535 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1536 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1536 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1541 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1541 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1762 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1762 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1763 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1763 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

java 1521 1764 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)

java 1521 1764 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)

ruby 12673 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby-time 12673 12675 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12676 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12677 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12678 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12679 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12702 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12726 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12727 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12728 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12729 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12730 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12731 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12732 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12733 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 12673 12734 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby-time 15247 15249 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15251 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15252 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15253 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15254 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15271 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15286 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15287 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15288 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15289 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15290 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15291 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15292 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15293 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

ruby 15247 15294 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

As we can see, tomcat and an application are blocking the deletion of multiple logs.

We restart the tomcat service and the application and check that the space is freed.

Bash
rokitoh@redorbita:/# df -h
Filesystem Size Used Avail Use% Mounted on

/dev/dm-0 12G 2.0G 9.3G 18% /

/dev/mapper/vg_display-lv_tmp 5.4G 12M 5.1G 1% /tmp

/dev/mapper/vg_display-lv_opt 10G 9.0G 520M 95% /opt

/dev/mapper/vg_display-lv_home 5.4G 991M 4.2G 20% /home

/dev/mapper/vg_display-lv_usr 9.3G 3.2G 5.7G 36% /usr

/dev/mapper/vg_display-lv_var 13G  2.0G   11G  16%

all the best.

:wq!

Comments