Findings from day 2

+ ISO image mount into /mnt/isoimage was detected 
+ C file compilation was detected (root escalation) 
+ syslog was clean 
+ brute-force attack was discovered on user produser 
+ user soval was banned 
+ several jobs were restarted 
+ SSH daemon was discovered to be compromised, ssh daemon was restored 
+ skynet root kit used (info on seclists) 
+ Machines were booted under version control 
+ mtab was modified as attacks started (ISO image was mounted) 
+ executable was detected and killed (several times over) 
+ problem with creamce was detected (a+r/w) 
+ both sshd and ssh binaries were compromised (permissions and attributes were screwed up) 
+ original config was restored 
+ ssh was locked down to root and produser only 
+ user dgram logged in to the site w/pub key 
+ user produser's private key was probably ripped off 
+ exploit code was found in ~produser 
+ telnet backdoor detected 
+ ssh binaries were reinstalled 
+ privilege escalation is unclear 
+ modified executables and config files were found and fixed 
+ attack was run from produser 
+ suspect python script was found 
+ user hekate was found & banned 
+ tampering with .bashrc was detected 
+ /etc/backup.tgz was deleted 
+ "from" statements were added to /root/.ssh/authorized_keys 
+ /var/tmp/...* 
+ forecast service was not brought back up