Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Forensics On a Cracked Linux Server

kdawson posted more than 7 years ago | from the hmmm-ls-looks-funny dept.

Security 219

This blog entry is the step-by-step process that one administrator followed to figure out what was going on with a cracked Linux server. It's quite interesting to me, since I have had the exact same problem (a misbehaving ls -h command) on a development server quite a while back. As it turns out, my server was cracked, maybe with the same tool, and this analysis is much more thorough than the one I was able to do at the time. If you've ever wondered how to diagnose a Linux server that has been hijacked, this short article is a good starting point.

Sorry! There are no comments related to the filter you selected.

Story is FUD from a M$ shill (2, Funny)

Anonymous Coward | more than 7 years ago | (#20345887)

A Cracked Linux Server? Ha! He should live so long!

Re:Story is FUD from a M$ shill (0, Funny)

Anonymous Coward | more than 7 years ago | (#20346033)

Cracked Linux server? Oh Noes, that's unpossible! Teh Lunix is UNBREAKABLE!

Re:Story is FUD from a M$ shill (0, Offtopic)

FST777 (913657) | more than 7 years ago | (#20346625)

Larry, is that you?

twitter (0, Troll)

Anonymous Coward | more than 7 years ago | (#20347019)

Oh please twitter, please post in this thread and tell us all how this is Microsoft's fault. I can't wait for your explanation for this one!

Yeah obvious FUD article (5, Funny)

Anonymous Coward | more than 7 years ago | (#20345933)

Why Slashdot would such obvious anti-Linux FUD is beyond me. Maybe the M$ advertising dollars are turning their heads.

The bottom line is that a LINUX SERVER CAN'T BE CRACKED.

Maybe this admin got his login info phished by Nigerian scammers, I don't know. The guy probably is wondering why his Ebay account has a bunch of negative feedback and his MySpace is all jacked up and hasn't put 2 and 2 together with that time he responsed to that clever email asking for the triple whammy of MySpace/Ebay/root on your servers so that you could clear the money transfer.

That or he didn't have his updates turned on and had an outdated BIND. And its not like BIND means Linux is unsecure.

Even not that the idea that Linux is crackable is laughable and not worht front page at digg let alone slashdot. You don;t see Technorait or Bruce Perens' site posting garbage like this ever so why slashdot editors can't see thru it i dont kno.

Re:Yeah obvious FUD article (0, Redundant)

Inakizombie (1081219) | more than 7 years ago | (#20346233)

Break out the BBQ! Its flame bait!

Re:Yeah obvious FUD article (1, Insightful)

PPH (736903) | more than 7 years ago | (#20346381)

The bottom line is that a LINUX SERVER CAN'T BE CRACKED.
Its not impossible. There are admins dim enough to configure a system so as to be crackable. Its not like a Windows system. It takes work, but idiocy knows no bounds.

Replace 'LINUX' with another version of Unix (the name of which will be withheld to protect the innocent). Some years ago, I ventured out onto the shop floor where I worked and encountered a terminal logged on to a critical production server. Nobody responsible seemed to be around. Typing 'whoami' returned 'root'. I promptly called the IT department's computing security group, informed them of the problem and hung around to see who showed up. After about 15 minutes with neither the original user or IT security appearing, I just logged the system off and left.

Who knows what damage could have been done to that system before I arrived?

Re:Yeah obvious FUD article (5, Funny)

ATMD (986401) | more than 7 years ago | (#20346565)

*whoosh*

Re:Yeah obvious FUD article (1)

stephanruby (542433) | more than 7 years ago | (#20347647)

Forgetting to log out happens all the time. In College, anyone who forgot to log out from our unix student lab would find the next day that they had sent obscene emails and love poems to a few select faculty members.

Re:Yeah obvious FUD article (3, Insightful)

JeremyGNJ (1102465) | more than 7 years ago | (#20346487)

Great attitude to have. It's like saying "no one can pick my front door lock". Vulnerabilities are found all the time, and just because they are found and patched, doesn;t mean that someone couldn't have exploited them before that point.

Don't be blinded by your religion.

Re:Yeah obvious FUD article (1)

Inakizombie (1081219) | more than 7 years ago | (#20346723)

Don't be blinded by your religion. <3

Re:Yeah obvious FUD article (5, Funny)

Anonymous Coward | more than 7 years ago | (#20347203)

. o <- Joke

..O <- You
./|\
./ \

Re:Yeah obvious FUD article (0, Redundant)

suggsjc (726146) | more than 7 years ago | (#20347383)

ASCII art is lame If you really want to blast them Then try a haiku

Re:Yeah obvious FUD article (5, Funny)

suggsjc (726146) | more than 7 years ago | (#20347457)

Dang HTML Formatted default, forgot the <br>'s

ASCII art is lame
If you really want to blast them
Then try a haiku

So in my rage, I wrote this (and used the code layout):
Today I posted
Today I looked like an ass
It is Friday, beer

Re:Yeah obvious FUD article (0)

Anonymous Coward | more than 7 years ago | (#20347445)

Sorry, I don't get it...

Re:Yeah obvious FUD article (2, Interesting)

DrSkwid (118965) | more than 7 years ago | (#20347033)

I had a co-lo rental from Pipex. Linux 2.2. They noticed it was broken in to, cut us off, charged us to re-image the box on which they had left a tar of the drive. OK sounds fair enough, but they re-imaged it with EXACTLY the same Linux 2.2 install and it was infiltrated again by the time I got the email telling me it was back on. I fixed it by hand and never told them lest they charge the company again. Happily I quite soon after.

Re:Yeah obvious FUD article (0)

Anonymous Coward | more than 7 years ago | (#20347209)

Why Slashdot tolerates ignorant responses like this is beyond me.

Any software, as long as it's connected to a network or otherwise accessible by a malicious third party, is open to attacks. It's quite difficult to make something 100% bug free, uncrackable piece of software unless it's the "Hello World" program that you wrote in some C for Beginners class. Actually, even with something so simple there can be errors that can be exploited. The cost of reaching that 100% bug free state becomes prohibitively high with any piece of software that's relatively complex.

The bottom line is that any piece of software with any degree of complexity *CAN* and probably does have bugs that can be exploited. If Linux were truly bug free, then by definition things like patches would not exist for Linux.

What's really laughable is that you truly believe that turning on updates makes you impervious to exploits.

Re:Yeah obvious FUD article (0)

Anonymous Coward | more than 7 years ago | (#20347597)

Do you think if you took that stick out of your ass then you might be able to smell flowers and fresh cut grass, hear laughing happy children, pet a kitten, and get the joke that everyone else is enjoying?

Re:Yeah obvious FUD article (1)

32771 (906153) | more than 7 years ago | (#20347553)

I actually had to look up 'ls -h'. It supposedly prints out file sizes in a human readable format (man ls says "1K 234M 2G"). I thought that mere humans aren't granted access to UNIX machines.

I wonder what the crack did to ls. It probably printed smilies instead of 'K', 'M' or 'G', i.e. big smilies for G and a scowl for no extra letter because of the low efficiency or what not.

Another option could have been ls output in rosy warm colors - eek ...

     

Forensics (5, Insightful)

DrDevil (90608) | more than 7 years ago | (#20345961)

Where did the word forensics come from? This is the completely wrong approach if working forensically. Can slashdot please use not use sensational titles! "Analysis of a cracked box" maybe more appropriate.

Re:Forensics (1)

Leftist Troll (825839) | more than 7 years ago | (#20346179)

Troll? I hardly see how looking at your bash history qualifies as "forensics".

Don't get me wrong, the article is somewhat interesting. It's just not an accurate headline.

Re:Forensics (2, Insightful)

extrasupermario (1084831) | more than 7 years ago | (#20346209)

For those that do not knowingly experience 'cracked' linux boxes (re: not knowing everything to look for), articles like this are a great way to learn from others. Kudos to 'lars' for sharing his findings with the world and reminding us all that security is an evolving process.

Re:Forensics (4, Insightful)

eln (21727) | more than 7 years ago | (#20346277)

This article is somewhat helpful as it does show one way to catch crackers, although he goes about it somewhat clumsily (an "ls" command that doesn't accept a flag you know to be valid, especially when that flag has been aliased on your own shell for months, should instantly tell you you have a cracked box) and the method by which he finds out where the rootkit is is due to a mistake that most non-moron crackers would not make (neglecting to remove the .bash_history file).

It's unfortunate that this cracker made such an elementary mistake, it would have been interesting to see more advanced techniques in detecting rootkits. However, his analysis of the rootkit itself does provide some good information as to what a typical rootkit will generally do (replace core binaries, hide itself, use innocuous-looking names, etc).

Re:Forensics (5, Funny)

Anonymous Coward | more than 7 years ago | (#20346595)

On the one server I have backdoor access to .bash_history is symbolically linked to /dev/random

It makes for an interesting read :)

Anonymous in case the admin actually reads slashdot.

Re:Forensics (0, Flamebait)

slightcrazed (973882) | more than 7 years ago | (#20347103)

I'm pretty sure it's not. AND, I'm pretty sure you're an idiot.

Re:Forensics (1, Interesting)

Anonymous Coward | more than 7 years ago | (#20347599)

[~]:apache$
lrwxr-xr-x 1 root wheel 11 Dec 20 2006 .bash_history -> /dev/random

I'm pretty sure it is. I didn't use any crazy exploits or anything. It's an old computer that I once had access too when I was in school. It's just a lesser used machine and all I use it for is bit torrent (on a .edu).

I created a few users such as "apache" and "sendmail". I'm not claiming to be a haxor by any means, and I just use it, like I said, for bit torrent.

'apache's root directory is actually a mounted DMG file that I have mounted to /tmp.

With OSX it's pretty easy.
Create DMG: /usr/bin/hdiutil create -size 1t -fs HFS+ -type SPARSE -encryption -stdinpass -volname objc_sharing_ppc_23 data

Attach DMG: /usr/bin/hdiutil attach -readwrite -private -mountroot /tmp -nobrowse -stdinpass "/Library/Application Support/LiveType/LiveFonts/Pro Series/Script.ltlf/data.sparseimage"

Detach DMG: /usr/bin/hdiutil detach /tmp/objc_sharing_ppc_23 >> /dev/null

128 bit encryption on that home directory. No one really questions large files in /Library/.

Re:Forensics (0, Redundant)

SIIHP (1128921) | more than 7 years ago | (#20346665)

"Can slashdot please use not use sensational titles!"

??? ...

BWAAAAHAHAHAHHAHAHAHAHAHAAHAHAHAHHAHAHAHAHAHAHAHAH AHAHHAHHAHAHAHA (cough cough).

Thank You. Really, that was awesome.

How did he get access and On tools (5, Insightful)

morgan_greywolf (835522) | more than 7 years ago | (#20345963)

And the most important question is, how did he get access in the first time? The server was running Ubuntu 6.06 LTS (i386) and was fairly updated. The compromised could be caused by:

        * An exploit unknown to the public.
        * A user accessing this server from an already compromised host. The attacker could then sniff the the password.
It's a very good question, because if the guy was keeping his server up-to-date, then these two are the most likely scenarios.

On tools...it's important to note that in forensics on a Linux box, your friends are ethereal (for watching packets on open connections), netstat (to see what's listening), and strace (shows you what UNIX API calls a running process makes, which gives you very good idea about what's going on.)

Other tools: nmap may be useful for seeing what's going on with 62.101.251.166 and 83.18.74.235. The service detection options, in particular. Always do this on a sandboxed host. Something running in a VM might be useful in this regard.

Anyway, nice article. This is almost exactly how I proceeded when one of my own servers was hacked a few years ago.

Further discussion... (5, Informative)

meringuoid (568297) | more than 7 years ago | (#20346275)

Bruce Schneier posted this a few days back [schneier.com] . Consensus is that it's not that good an analysis, but that the attacker was even worse. Some discussion also of whether it is better to take the machine offline immediately (and risk alerting the attacker that he has been rumbled) or to begin your analysis with the machine still live and operational. I for one side with the 'shut that thing down NOW' faction.

Re:Further discussion... (1)

morgan_greywolf (835522) | more than 7 years ago | (#20346423)

It's not a great analysis, no. In my case, I was actually able to find the hole (an unpatched BIND with a known exploit -- ouch! That'll teach me to keep my patches up to date!), the attackers IP address, (which was not easy to find. I had to sleuth around a bit a contact a few sysadmins before I traced him down to a cybercafe in China) but it's a good start.

Taking tha machine offline immediately -- bleh. It depends on the box. In my case, my box was nothing more than a old machine being used as a firewall. He was never successful in getting through to my boxes behind the firewall, he tried...but something he saw must've spooked him or made him disintrested, because he stopped looking and just left the box open as a zombie. My guess is that's all he was after was a zombie anyway. So shutting down the box would have saved me exactly what?

Re:Further discussion... (0)

Anonymous Coward | more than 7 years ago | (#20346569)

. So shutting down the box would have saved me exactly what?

His zombie spending the time spamming anyone and everyone while your network gets added to every spam list, even those that are next to impossible to get off of.

Re:Further discussion... (1)

morgan_greywolf (835522) | more than 7 years ago | (#20347001)

Bleh. I had already blocked off the mail port with IPtables. It was the fact that it was sending thousands of mails an hour that I noticed it was a problem at all.

Re:Further discussion... (4, Interesting)

Andy Dodd (701) | more than 7 years ago | (#20346731)

On the other hand, shutting down the box ASAP makes it much harder to find the guy.

For example, one of Vodafone Greece's first reactions to finding that some of their switching systems had been rootkitted was to remove the offending software. This removal was one of the main contributing factors to the authorities having no chance to ever find the group that had compromised the system, that along with a couple of other screwups led to Vodafone getting fined a pretty hefty sum.

http://en.wikipedia.org/wiki/Greek_telephone_tappi ng_case_2004-2005 [wikipedia.org]

IEEE Spectrum had a recent article that had MUCH better information than Wikipedia though, I don't have it with me at the moment unfortunately.

Re:How did he get access and On tools (4, Funny)

eln (21727) | more than 7 years ago | (#20346329)

I think it's probably the fact that the owner of this system had the root password set to "GOD" as all good sysadmins do. The hacker's extensive experience hacking the Gibson made getting into this system a cakewalk.

Clearly, we as sysadmins should rethink the long-standing policy of setting all root passwords to either love, secret, sex, or god. Perhaps we should at least add another password to the list, like "unhackable" or something truly secure like that.

Re:How did he get access and On tools (1)

jimicus (737525) | more than 7 years ago | (#20346371)

There's a few things which immediately spring to mind:

1. We already know that it was meant to be running Apache. Perhaps there was some PHP application which wasn't very secure? Even so, if that were the case then the exploit they used must have been fairly convoluted because it probably wouldn't have got them root access immediately.

2. We don't know what other services were supposed to be running, how/if they were firewalled and secured. SSH, for instance, is only as secure as the weakest password on the box - for best results you probably want to combine it with minimising the number of shell accounts, only allowing root access through private/public keys and using denyhosts (or similar) to automatically block bruteforce dictionary attacks.

3. We don't know how secure the desktop PC which was used to administer this box is. There is an awful lot of Windows-based malware out there - it wouldn't surprise me if there's more than one piece which looks around for when you start a connection to a host on port 22, enables a keylogger and sends the results back.

Re:How did he get access and On tools (0)

Anonymous Coward | more than 7 years ago | (#20346519)

On tools...it's important to note that in forensics on a Linux box, your friends are ethereal (for watching packets on open connections), netstat (to see what's listening), and strace (shows you what UNIX API calls a running process makes, which gives you very good idea about what's going on.)
Yes, I think if you know what all this means, your friends are definitely ethereal.

Likely dictionary ssh attack on a random user (0)

Anonymous Coward | more than 7 years ago | (#20346531)

Linux has many local exploits and very little effort to fix them sadly. If you have 'regular' users, you need to keep cracklib hooked in to all password change methods and try to use john the ripper often on password files. OpenBSD has a likely safer userland, but in general local root exploits happen on many services that are necessary on a server. The best thing would be to have an OpenBSD jump server with no extra services/tools for users that can easily be monitored/rebuilt. Even better, expose daemon servers (apache,etc) by NFS only if users must touch their actual filesystems.

Re:Likely dictionary ssh attack on a random user (1)

morgan_greywolf (835522) | more than 7 years ago | (#20347187)

Huh? If you're running a server with 'regular' users, and you're using even remotely dictionary-based passwords, you deserve to get hacked.

Re:How did he get access and On tools (5, Informative)

arivanov (12034) | more than 7 years ago | (#20346733)

All of these will help only if it is cracked by amateur sr1pt k1dd10tz like in this case. If it is cracked properly you will not see anything or spook off the intruder. He will either go underground or destroy the box with all of your data (not that you should try to use it as it may have been altered).

I have seen a number of rootkits for Linux as far back as 97-98 which were considerably more advanced. It was a bit of an arms race between the admins (including me) and the guys who were breaking in. By the end the best rootkits could:

1. Load a whole hidden fs with tools into a ramdisk or hidden area on the filesystem not visible using normal tools.
2. Hide all sockets, processes and files belonging to the rootkit completely. You simply could no longer see them using netstat, ps and other similar tools.
3. Monitor network driver state for the promisc flag and "scrub" backdoor traffic out of it so it is no longer visible using tcpdump and ethereal.
4. Adjust memory totals and df so that you do not see them. This was also the only way we found to catch it. Try to allocate 95% of the remaining free memory and see the system oops magestically.
5. Doctor logs so that you could not notice anything.
6. The rootkit itself handled all connections via something that looked like ssh. I never managed to figure out how it loaded. One of the executables in the system loaded at startup was backdoored. Probably sendmail or one of the other daemons it could not do without.
7. The rootkit managed to masq changed files completely. Tripwire and md5sums were reporting all OK while executables were being changed.

That was a the tech level in 97. I would expect 10 years later a good rootkit to be even better. Looking at the blog post I can only laugh.

If you suspect a system is cracked:

1. Take it offline and take the disks out. Analyse the system completely offline looking at the disk from another system mounted as ro (on SCSI discs use the RO jumper). Never ever even try to start it. Nowdays knoppix is a great help. Most importantly - do not fsck systems before mounting as the rootkit may hide in orphaned areas which fsck will fix.

2. If you are monitoring traffic, monitor it on a switch span port or create yourself a simple multiple interface box which serves as a firewalling bridge (so you can hijack the more interesting bits and alter them). Lex Book PCs are a good choice as they can run either Linux or BSD and are as portable as a laptop. A recent Via with 2 Ethernet ports is also a good choice as it can handle up to 1GB of traffic across as a bridge.

Re:How did he get access and On tools (5, Informative)

sootman (158191) | more than 7 years ago | (#20347669)

If you suspect a system is cracked:
1. Take it offline and take the disks out.


And I've been told don't use the 'shutodwn' command--instead, pull the power plug out of the wall. A rootkit could include a cleanup routine that gets run at shutdown time.

cat .bash_history (0)

Anonymous Coward | more than 7 years ago | (#20345971)

I though even script kiddies knew unset HISTFILE... hmm...

Re:cat .bash_history (0)

Anonymous Coward | more than 7 years ago | (#20346023)

If I bust your server, you'll see me upload, compile, install (to /sbin/sh) and exec my own shell.
If you're lucky, you might find I the source behind.
The shell is a working Bourne shell, so nothing interesting there. However, being a Bourne shell, there's no history file generated from there.

*Bourne* Shell? (4, Funny)

Spy der Mann (805235) | more than 7 years ago | (#20346303)

The shell is a working Bourne shell

I knew it! Jason Bourne was involved in this!

Re:*Bourne* Shell? (1)

nschubach (922175) | more than 7 years ago | (#20346815)

If Jason Bourne were involved in this he'd be standing right next to you...

Re:*Bourne* Shell? (0)

Anonymous Coward | more than 7 years ago | (#20347153)

Was this a very poor attempt at making a joke, or are you seriously ignorant of the most popular shell ever [wikipedia.org] ?

Re:cat .bash_history (0)

Anonymous Coward | more than 7 years ago | (#20346229)

You really need "ttyrpld". (Just hope your logging server does not get hacked too.)

Looks as if there was another way... (3, Funny)

sphealey (2855) | more than 7 years ago | (#20346029)

Looks as if there was another way to crash his server...

sPh

This is not forensics (5, Informative)

Gandalf_the_Beardy (894476) | more than 7 years ago | (#20346055)

Forensics has to be useful in court. This is not - it's tainted evidence. Now if they took the original disk out, copied it with DD or similar to a file and mounted it as loopback and worked on that, then that's a first start to a forensic analysis.

Re:This is not forensics (1, Insightful)

andreMA (643885) | more than 7 years ago | (#20346213)

Uh, just because the term "forensics" is sometimes used in a limited sense in the legal sphere doesn't mean it can't be used in a more casual sense elsewhere. If he'd called it a "postmortem" would you be complaining that it wasn't performed by a licensed medical examiner?

Re:This is not forensics (1)

Gandalf_the_Beardy (894476) | more than 7 years ago | (#20346281)

No but postmortem is better - it's clearly not being dissected by scalpels. Just that computer forensics is a strict discipline that has a chain of custody and doesn't tamper evidence, and this would give the misleading impression that the actions carried out are OK. As a learning step by step article for finding out what happened and doing a post-mortem, then I think it is actually a pretty darn good article and one that I'll be saving to show to people who want to learn. Just not one that you would be able to use in criminal court.

Re:This is not forensics (1)

eln (21727) | more than 7 years ago | (#20346385)

How is postmortem better? Postmortem means "occurring after death," but this box is not dead by any means.

Re:This is not forensics (1)

Gandalf_the_Beardy (894476) | more than 7 years ago | (#20346509)

From the article. "....most notably the web-server apache refused to start...." OK so it probably did other things besides serve web pages, but as a web server it's as dead as a dodo since the logfiles were hosed.

Re:This is not forensics (2, Insightful)

Quarters (18322) | more than 7 years ago | (#20346447)

"...sometimes used in a limited sense in the legal sphere..."

The definition of the word forensics is, "The use of science and technology to investigate and establish facts in criminal or civil courts of law." The original poster's argument is correct. This was not forensics. It was an analysis.

Sorry, nice try, no (0, Flamebait)

SIIHP (1128921) | more than 7 years ago | (#20346747)

"The definition of the word forensics is..."

No, that's A definition. Here's another

1 : an argumentative exercise

OP was wrong, and so are you.

Still a good read. (1)

Seakip18 (1106315) | more than 7 years ago | (#20346541)

He was collecting a good bit of data there. If he pulled the drive out before doing that, he would of lost all volatile data, including possible info that hadn't been garbage collected. Granted, a dump of the RAM should have been his first command, since everything before it risks trampling de-referenced addresses.

If your going for a court case, your better off with the mountain of information than just a sheet of what really matters.....unless your the RIAA, then you make accusations at dead grandmothers.

How server was accessed in the first place is what I really want to know.

Ssshhh.... Secrets Revealed... (-1, Flamebait)

Anonymous Coward | more than 7 years ago | (#20346059)

AC because I am a MS insider (so the C part of AC is automatic, anyway). Story was posted by my boss, who would also like to direct your attention to Get the---Compare http://www.microsoft.com/windowsserver/compare/def ault.mspx [microsoft.com] .

See also http://linux.slashdot.org/article.pl?sid=07/08/23/ 2254254 [slashdot.org] .

Re:Ssshhh.... Secrets Revealed... (1)

FuzzyDaddy (584528) | more than 7 years ago | (#20346561)

AC because I am a MS insider

Suuuure you are.

Re:Ssshhh.... Secrets Revealed... (5, Funny)

dedazo (737510) | more than 7 years ago | (#20346785)

I am a MS insider

The 220,000 or so members of the Slashdot Members Who Post Authoritative Statements On The Inner Workings Of Microsoft To Support Their Arguments warmly welcomes you to the club.

Heh heh (-1, Flamebait)

Anonymous Coward | more than 7 years ago | (#20346097)

Yup. I authored quite a few tools like this, though I don't think my work was used for these servers in question. Why do I write these tools? Because Linux users are smug. I'll admit, it is quite a bit easier to jack a Windows PC, but that has a lot more to do with the ignorant user base. But Linux users are arrogant, and it is fun to have a laugh at their expense. Most of the time, they haven't the faintest idea that I have been in and can get in whenever I want.

Re:Heh heh heh heh hehe hehehe eheheheheheehehehe (0)

Anonymous Coward | more than 7 years ago | (#20346365)

You still don't know I 0wnz0red you teh hole t1m3?

Hahahaa, stupid kid. Check your log files @ /dev/null

God I feel smug _now_. But then again, I AM a Lunix user.

> Go back to play with your nintari. Press A to start and B to stop, you knows...

Not enough information (0, Redundant)

downix (84795) | more than 7 years ago | (#20346113)

What was his setup? How did they access? And who had access?

Re:Not enough information (1)

Stormx2 (1003260) | more than 7 years ago | (#20346547)

From TFA:

And the most important question is, how did he get access in the first time? The server was running Ubuntu 6.06 LTS (i386) and was fairly updated. The compromised could be caused by: * An exploit unknown to the public. * A user accessing this server from an already compromised host. The attacker could then sniff the the password.

rkhunter anyone? (3, Informative)

jshriverWVU (810740) | more than 7 years ago | (#20346153)

I have rkhunter on all of my machines, sends a nice email letting me know of any changes in system files.

Re:rkhunter anyone? (0)

kwalker (1383) | more than 7 years ago | (#20346467)

So does mine, along with an alert e-mail because it doesn't work with SELinux running most of the time, it doesn't like /dev/.udev, and it isn't updated for my newest distros. it's to the point that I'm about to uninstall it because of all the false positives.

Re:rkhunter anyone? (1)

Mr2cents (323101) | more than 7 years ago | (#20346529)

But if your system is compromized, do you still trust rkhunter?

Does rtkhunter... (4, Insightful)

Anonymous Coward | more than 7 years ago | (#20346563)

Does rtkhunter send you a email when the cracker changes /usr/bin/rtkhunter so that it won't email you the attacker's changes?

If you think that rtkhunter will protect you from a Linux kernel module rootkit your completely delusional. NOTHING will _reliably_ locate a LKM rootkit. That's the point of it.

Think about it. Rtkhunter relies on the ability of the kernel to accurately indicate files sizes, file names, and running proccesses as well as a bunch of other little detail things that normal rootkit makers tend to get wrong. When that kernel is subverted and controlled by it's new owner to give rtkunter, as well as other processes (such as your bash shell) false information about the system then those things are completely worthless.

It's the same as virus scanning on Linux (or any other system). Once the attacker gets root access then they have access to the kernel. Once they have access to the kernel they can use the kernel against you to hide what they are doing. Since userspace runs on top of the kernel then any sort of activity can be hidden by making the kernel lie to anything running in userspace.

This includes logging daemons, rootkit detection software, administrators, virus detection, rpm checksums, or anything else that people use to give themselves a FALSE sense of security.

There are two ways to reliably detect a rooted machine.

The first way is to use a network-based Intrusion Detection System (IDS). One of the best ones is commercially supported open source application called Snort. These guys can be hooked up to networks in a passive and completely undetectable way and are used to monitor traffic. They will alert administrators to any unusual network activity.

Network based IDS can be fooled, but as a administrator your at least operating on the same playing feild since your own software isn't used against you.

The second, and more reliable way, is to use a checksum-style IDS. MD5deep, AIDE, or Tripwire are 3 very good examples of this.

However how people use these things are completely worthless. If you keep the checksums and run the checksum software on the same machine as the one your trying to detect, then it's not good. Since they rely on the kernel any kernel-level rootkit can defeat them and the attacker can edit and substitute incorrect checksums.

In order for stuff like AIDE to be usefull it needs to be ran from read-only media and from a different operating system then the one your checking. (for example booted up in a knoppix cdrom, or a removable disk in a dedicated unconnected-to-any-network 'Tripwire' machine)

Both forms of IDS are very expensive and difficult to correctly use. Virtual machines make this stuff somewhat easier, but it's still much better to have dedicated machines for these things.

rtkhunter is nice if it's job is to make you feel good. If it's job is to make sure your machine is secure then it's shit. (no offense to the rtkhunter authors, I am sure they understand it's role and effectiveness.. to bad their users don't tend to) It's only good for kiddies that don't know better and if your being owned by kiddies then you have bigger problems.

Re:Does rtkhunter... (1)

mutterc (828335) | more than 7 years ago | (#20347643)

There's an interesting third approach, used by Sysinternals's (now part of MS) RootkitRevealer for Windows.

Basically, enumerate all the files on the system using the usual OS APIs. Then, scan the entire raw disk, and enumerate all the files on the system by manually interpreting the directory structures stored on-disk. Any files whose directory entries exist on-disk, but don't show up in the OS's API (with a few standard system exceptions) are being hidden from the OS API layer by a rootkit.

It's certainly theoretically possible to fool, by having your rootkit hook the APIs used to read the raw disk, and returning innocuous values, but that's a good bit harder to do than the other stuff rootkits usually do. Some rootkits fooled it by not hiding their files if the process trying to look them up was named RootkitRevealer.exe, so the tool took to making a randomly-named copy of itself and executing that.

Re:rkhunter anyone? (0)

Anonymous Coward | more than 7 years ago | (#20347115)

"I have rkhunter on all of my machines, sends a nice email letting me know of any changes in system files."

Can't you just set permissions so that nobody can change those files?

After all, if someone has sufficient access to change the file without the file-owner's permission, then presumably they also have sufficient access to modify your email script?

Re:rkhunter anyone? (1)

Obsi (912791) | more than 7 years ago | (#20347525)

Root bypasses permissions. Otherwise, how'd you delete a file some noob chmod 000?

They got the webserver too (1)

EvilRyry (1025309) | more than 7 years ago | (#20346155)

Looks like the server is down for some forensic analysis following a break-in as well. Too bad. Wonder how they are going to do the analysis on the server without TFA?

Taking Bets... (1)

sanimalp (965638) | more than 7 years ago | (#20346193)

Ill bet his root password was "password"... oh, wait, "password1" is the new norm now..

Re:Taking Bets... (1)

nschubach (922175) | more than 7 years ago | (#20346855)

Actually, this month I believe it's "password08"

Can't read TFA (1)

PPH (736903) | more than 7 years ago | (#20346205)

I guess the rootkit on my system prevents me from reading any articles on how to detect and clean up rookits....


Time to put my tinfoil hat back on.

Hey (1)

Joseph1337 (1146047) | more than 7 years ago | (#20346207)

You`re SURE that he wasn`t running Windows?

I had to do this once. (4, Funny)

Anonymous Coward | more than 7 years ago | (#20346237)

We had a cracked linux server at work one time and I took it upon myself to find out who did it. Long story short: some server monkey decided it would be a fun idea to ride his bike around inside the data center and smashed into one of the racks.

Re:I had to do this ounce. (0)

Anonymous Coward | more than 7 years ago | (#20346837)

Hey if it was a "racked" server to begin with, it was only one Letterman-gets-evil-off-his-meds move away from being "cracked" or "hacked".

Re:I had to do this once. (2, Funny)

CompMD (522020) | more than 7 years ago | (#20347555)

im in ur datacenter breakin ur racks

Mirror (3, Informative)

W2k (540424) | more than 7 years ago | (#20346257)

Meta-cracking (5, Funny)

CopaceticOpus (965603) | more than 7 years ago | (#20346283)

Oh, I see, it's a clever DOS attack:

1. Infect Linux server of some guy with a blog.
2. Guy blogs about how he dealt with said infection.
3. Blog posting gets linked to on Slashdot.
4. Millions of computers attempt to access the blog, hence bringing down the server.

Don't you see? We've a socially engineered botnet!

(And please, for the love of all that is sacred and funny, don't reply to this and add steps for "???" and "Profit". It's just tired and completely not funny. And the clever little variation on that theme you're thinking about posting right now isn't funny either.)

Re:Meta-cracking (1, Funny)

Anonymous Coward | more than 7 years ago | (#20346379)

5. ???
6. Profit!

(oh, come on, you asked for it)

Re:Meta-cracking (5, Funny)

Anonymous Coward | more than 7 years ago | (#20346401)

1. Find clever little variation that is funny
2. ????
3. Profit!

Re:Meta-cracking (-1, Redundant)

Anonymous Coward | more than 7 years ago | (#20346453)

5. ???
6. Profit!!

Re:Meta-cracking (1)

slightcrazed (973882) | more than 7 years ago | (#20347277)

4. Prophet? 5. ????? 6. Profit!

Wish I would have known... (1)

DoctorPepper (92269) | more than 7 years ago | (#20346295)

I got hacked back in February - March 2001 time-frame. I made the mistake of setting up my Linux server as a router, and left my Samba and NFS shares active. This kind of info would have really helped me then.

Casual approach to forensics (1)

Vario (120611) | more than 7 years ago | (#20346299)

Before everybody complains how he could have done the analysis much better I think it reflects quite well the approach a lot of people would use here. If my friend would ask me about a failing apache server my first reaction would not to dd the whole system.

Unfortunately the article is a little low on details about the running configuration. Ubuntu 6.06 seems like a solid distribution security wise, so where all current patches installed, was there a weak root password? Was root ssh login enabled?

It is quite lucky that the attacker was not really experienced and more or less just used the scripts he downloaded somewhere without knowing exactly what they were doing. Otherwise without anything like tripwire this might have gone unnoticed for quite some time.

Re:Casual approach to forensics (1)

mr_mischief (456295) | more than 7 years ago | (#20346543)

Okay, I've not yet RTFA. Did it specifically say, "bog standard Ubuntu 6.06 with absolutely no additional software and only bare necessary configuration changes needed for system differentiation purposes"?

I ask because everyone seems to be looking very closely at the initial OS distro, and almost any server that's been put into useful production has been tweaked in some way from the official packages. Stuff gets compiled from source. Custom stuff gets coded. Packages get installed out of third-party repositories or straight from vendor sites. Daemon configurations get changed, firewall rules may be tweaked, and additional modules for existing server daemons get added.

Hell, they could have done something as stupid as allowing root logins through unencrypted telnet, then actually using that "feature".

Oh, well, off to RTFA to see if it contains any of the answers.

Re:Casual approach to forensics (1)

mr_mischief (456295) | more than 7 years ago | (#20346707)

Okay, I read a cache of the article. It doesn't answer any questions about what might not be stock Ubuntu 6.06, but simply assumes that uname/motd says it all.

Another option for a perfectly secure box that wasn't mentioned in TFA is that the friend could have run a Trojan that opened the initial hole.

All the log files will have been changed to: (0)

Anonymous Coward | more than 7 years ago | (#20346319)

j00'v3 b33n PWN3D! I 4M 3r337.

Re:All the log files will have been changed to: (0)

Anonymous Coward | more than 7 years ago | (#20347407)

"3r337"? So, "hacked by chinese?"

Raise your hand (5, Funny)

tie_guy_matt (176397) | more than 7 years ago | (#20346557)

Raise your hand if you typed "ls -h" on your box just to make sure it still works right.

Re:Raise your hand (5, Funny)

Anonymous Coward | more than 7 years ago | (#20346617)

C:\>ls -h
'ls' is not recognized as an internal or external command,
operable program or batch file.


Oh noes!

selinux? (4, Insightful)

burnin1965 (535071) | more than 7 years ago | (#20346613)

Does Ubuntu install selinux and a policy in a default installation, or is it necessary to add it later?

I've only performed one Ubuntu install and most of my experience is with Red Hat and Fedora linux distros. Fedora installs selinux with a targeted policy enforcing by default which I think is a good thing. I had an experimental Fedora web server with PHPbb installed which was comprimised via the PHPbb application but looking through the log files it appeared that selinux had thwarted attempts to root the box or setup a zombie to connect to an irc server.

Other than the mistake of an outdated PHPbb application I also made the mistake of allowing execution of code in /tmp, lesson learned. But it was interesting to see selinux do its job and I'd be curious if it was utilized in this instance.

Re:selinux? (1)

Neil Watson (60859) | more than 7 years ago | (#20347265)

I think SElinux is still a mixed bag when it comes to distribution support. My attempts at using SElinux with Debian have been disappointing. Red Hat AS4's SElinux works out of the box but, it is not enabled by default.

Re:selinux? (2, Informative)

OmegaBlac (752432) | more than 7 years ago | (#20347321)

Does Ubuntu install selinux and a policy in a default installation, or is it necessary to add it later?
No, one must install it manually. Getting SELinux into a default installation for future release is being worked on though: https://wiki.ubuntu.com/SELinux?highlight=%20selin ux%20#2910857737223089520 [ubuntu.com]

Ineffective rootkit (1)

whoever57 (658626) | more than 7 years ago | (#20346845)

The "rootkit" does not seem to be very effective at hiding itself and the malware processes:

These two processes show up using (our backdoored) "ps", so I guess that why the attacker renamed it to "smbd":
root@server1:/var/.x/psotnic# ps axuw | grep smb
root 3799 0.0 0.4 8592 2156 ? S 11:00 0:00 /usr/sbin/smbd -D
In fact, the whole crack of the server seems to be pretty amateurish. Still, even if the analysis was not very good, it is interesting article.

That's it, I'm switching to Windows (3, Funny)

Maltheus (248271) | more than 7 years ago | (#20346915)

Security is very important to me, I can't be screwing around with something that can be so easily cracked.
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?