This is a re-post (or posting to my blog) of an interview I had done for Linux Security in September of 2002. The original article is available here.
Just to give everyone an idea about who writes these articles and feature stories that we spend so much of our time reading each day, I have decided to ask Brian Hatch and Duane Dunston, the newest members of the LinuxSecurity.com team, a few questions. I’d also like to begin by thanking both Brian Hatch and Duane Dunston for allowing us to conduct this interview.
Before we get to that, let me give you a brief introduction as to who these folks are, although I promise you that you will receive plenty of insight the further you read:
Brian Hatch is Chief Hacker at Onsight, Inc.[1] where he is a Unix/Linux and network security consultant. His clients have ranged from major banks, pharmaceutical companies and educational institutions to major California web browser developers
and dot-coms that haven’t failed. He has taught various security, Unix, and programming classes for corporations
through Onsight and as an adjunct instructor Northwestern University. He is also co-maintainer of Stunnel[2], an Open Source secure SSL wrapper used around the world to encrypt clear text protocols.
Brian Hatch has been interviewed several times on his stances on security, full disclosure, and Open Source technologies. (See interview 1 and interview 2.) Rather than retrace this ground, we thought we’d ask questions geared toward his personal rather than professional endeavors.
Duane Dunston is a Computer Security Analyst at STG Inc. for the National Climatic Data Center[3] in Asheville, NC. He received his B.A. and M.S. degrees from Pfeiffer University and he has his GSEC certification from SANS[14]. Not only does he enjoy his work in computer security, he also likes to get involved in its ever-growing technologies.
LS: How did you attain your security knowledge? Did you get a CS degree, or was it on your own?
Brian: I had your formal four year college education, but during that time had only one official computer class – Pascal – because they wouldn’t teach C at the time. All my Unix, C, and security knowledge was self taught from the man pages, which is definitely not the way I’d suggest for maintaining sanity. Nowhere is there an ‘asterisk means pointer’ man page. Suffice it to say, my early C code was ugly as sin. Luckily I’ve deleted every copy, and burned the tape backups.
Frequently I’m brought into security gigs in early stages to make decisions and get infrastructure in place, and then hand off the project to an employee, which means I frequently find myself in a position where I make hiring decisions. Because of my non-formal background, I am able to go in without preconceptions that certifications are the only mark of quality, and am able to get some extremely qualified folks hired even though they are sans certifications. (Pun intended.)
Had I been a CS major, there still would have been no chance to take security-related classes at the time anyway – there was no security emphasis at the time. There was a small group of us who would compare notes and tricks, and this was invaluable, more than any formal classes would have been. Hopefully more educational institutions are integrating security concepts into their curriculum, because without it the code being whipped up throughout the world will continue to be largely crap. Programs with buffer overflows are unconscionable, and yet they pop up in every product out there still, despite being a known quantity with well defined solutions.
Duane: I graduated from Pfeiffer University in 1997 with a BA in Sociology, the day after I ended up in Thaliand for about 5 months teaching English. When I returned there wasn’t much work for a Sociology major straight from Thailand teaching English.
I had some computer experience prior to that, though. My first PC was a Gateway 486 which I got back in ’94. I started shadowing my best friend and former boss at work and there I was introduced to Linux.
I continued to use Linux through Graduate school and learn more and more about security. After realizing I didn’t want to be a Manager, which is what I have my graduate degree in, I decided I wanted to stay with computers but with Security. Nearly all of what I have learned has been on my own from websites, books, and having support from my managers to just experiment.
LS: What are some of your other hobbies?
Brian: Between writing books, writing the Linux Security: Tips, Tricks and Hackery newsletter[4], preparing for a Linux security class I’ll be teaching in January, answering questions about Stunnel and maintaining the Stunnel website, trying to keep up with current security issues on mailing lists like Bugtraq and VulnWatch, web pages like LinuxSecurity.com and Help net Security, something called a “Day Job”, and a somewhat foreign concept in the computer geek circle called “wife and kids”, I’ve very little free time.
I like to read a lot, though the topic depends on the location. By myself, I usually pick security and geeky titles – I’ll re-read W. Richard Stevens’ books now and then for fun. But much more of my time is spent reading books such as Dinosaur Roar, Tumble Bumble, or Sheep in a Jeep to my daughter. If she stops paying attention, I start sneaking new characters in, such as Dmitri Dog, who is hounded by the evil Ashcroft Crow. I figure I need to sneak in anti-DMCA[12] sentiments early via allegory.
Duane: I don’t have many hobbies outside of computers. My interest tend to be around a particular aspect of computers. In particular, parallel computing, cryptography, finding good uses of open source in the business world, and computer forensics. I do have an interest in the arts. In particular, what is going through an artist’s mind as he or she is working. Two of my friends are artist and I really enjoy being around them, and other artists, and listening to them brainstorm ideas or talk about what their work means to them.
LS: Supposing you had free time, what would you be doing with it?
Brian: I’d devote some time to helping out the Linux Security Module project. I hope to help port systrace to LSM next year. Currently it is a kernel patch, and I think the community would be served better in the long run by having it available as an LSM module, which would make it more accessible to those who fear kernel compilation.
And some day I hope to get around to turning some of the megs of perl code I’ve written over the years into well defined Perl modules for CPAN. Then I won’t be the only one supporting this spaghetti code. 😉
If I had infinite time, I’d learn to play the Hammered Dulcimer and French Horn. There’s nothing in the world as musical as a well-played French Horn.
LS: In your opinion, what is the most interesting thing about Linux and Security?
Brian: The first thing is that, with Linux, security is a possibility. It is not an end point – you must constantly keep abreast of new attacks and revisit your security posture – but there is nothing that is unavailable to you if you want to look. Closed source systems can never offer this. By design, be it chosen for monetary reasons or to prevent competition, closed source products always hide details from the users and administrators that could be critical to understanding how thing function, and how they can be broken.
One of the beauties of Linux (and other open systems, such as *BSD) is that you can use them to boost the security of those closed source machines. By the liberal application of Linux machines throughout your infrastructure, you can keep those exploits-waiting-to-happen locked down where they can do less harm. For more of my ranting on this topic, see my article Linux is Securable — I won’t waste time rambling here.
What is most intriguing right now on the Linux horizon is the evolution of security controls. In the beginning, all you had to work with were file permissions. Root could do absolutely anything unchecked, and root access was required for some things such as binding low network ports or opening raw sockets, which meant use of set userid bits on programs, which frequently were broken to gain root access.
Next came capabilities, where each bit of root’s power was defined in more specific terms. When determining if a process could bind port 80 originally you’d check to see if uid==0. Now you’d check if the process had the CAP_NET_BIND_SERVICE capability. In theory, you could now remove capabilities from the system – for example removing the ability to load kernel modules ever again, which is good for defending against malicious LKMs.
This compartmentalization was a good evolutionary step, but it didn’t get used much in the real world. You had only global access to turn off a capability, and you’d never have it available again, except by rebooting. You couldn’t assign a capability on a process-by-process level, which made it much less useful.
Then came some excellent security kernel patches. LIDS[7], Grsecurity[8], SELinux[9], and others. Finally we had ways to give abilities to only the processes that needed them. We could completely lock down our machine such that any deviation from what we have allowed will fail. Should an unknown buffer overflow occur, the cracker can’t bind an inbound rootshell on port XYZ, it won’t be able to read files outside those we’ve allowed, and won’t be able to write to the file system anywhere. If they break in as root, they still can’t change anything in /etc unless they know yet another password, they won’t be able to Trojan files in /bin, or insert a loadable kernel module into the system. Life is good.
But, life could be better. These patches are not integrated into the kernel. You won’t get them on your Red Hat CD, or your Debian download. (Though some places do ship hardened kernels, such as Owl which contains the OpenWall[10] kernel security patches, or EnGarde[11], which comes with LIDS.) So normal users need to go out of their way to patch and recompile the kernel – a daunting task.
So, the current bright spot on the Linux security horizon in my opinion is the Linux Security Module infrastructure, which promises to make it possible to load these advance Linux security policy behaviors without a kernel recompile, thus removing a significant barrier to the new Linux administrators. If we can get more of the LSM modules to be stackable where appropriate, we’d be able to pick and choose the best of the features of each, something which currently involves manually merging kernel patches – quite a painful endeavor.
So, as we enter this new era of Linux security, you bet I’m excited. Just tell your manager that it’s a “Paradigm shift” and they’ll give you time to play with it too.
Duane: Computer forensics reigns supreme in my opinion. These experts provide us
the information we need to fine tune the tools we currently use and the tools needed to prevent attacks from occurring in the future. Sifting through megabytes or gigabytes of data to find one clue as to how someone compromised a system has to be tedious but the information they find helps us quickly learn how to patch the vulnerability and create tools to help detect how a system was compromised. I just don’t think the security industry can survive without forensic experts. There are many people who aren’t forensic “experts” who can and do identify how a system was compromised and they also provide the community with valuable information. However, the forensic experts can dig down and find the root of the problem and the information they provide can then be used to help fine tune currently used tools and methods.
LS: What are some of the lessons you you have learned in your years in computer security?
Brian: Well, in no particular order, here are a few things that pop to mind.
- Never bother with a technological hack when a social engineering one takes ten seconds.
- Posting a piece of paper listing a ‘contest for the most creative and verifiable username/password’ in a college campus food court will get you several hundred valid entries you can use at will.
- The easiest way to get access to a friend’s account is to get them to ‘xhost +’ to play network xtank.
- The next easiest is to untar /etc/shadow from the backup tapes that the administrator leaves next to the tape drive.
- Sometimes the perfect hack is just one click away.
- When you have a frustrating week, let the script kiddies amuse you. Set up a honeypot with as much known-buggy software as you can find. Break almost all the tools. Don’t include vi or emacs, make them learn ex. Include perl, but no modules. Install the Fortran compiler, but no others. Include netcat, but not telnet. Make the default shell tcsh, and remove all Bourne shell variants. Delete all terminfo/termcap definitions. Don’t include more,
less, or any other pager. Compile a version of cat that randomly swaps letters every so often. - Never put that honeypot anywhere near your actual machines.
- Use mutt. A good mailer is key to getting work done, and you can’t be secure if you’re too busy wading through your email.
- If you can’t do it from the command line using Netcat, you don’t actually understand how it works.
- If you can give the root password to your enemy and he can’t compromise your system, you’ve done a good job securing it.
- Every now and then, lock the root password on your own machine and try to recover it. You’ll learn where your own weakness are, and hone your skills.
- Make sure you have a throttle on any software that can page you.
- Never rename /usr/lib/libc.a.
- Always have a statically linked shell, in case another administrator renames libc.a and you need to re-create it using nothing but shell built-ins.
- And check your logs. Always check your logs.
Duane: The most important lesson I have learned is that management support is critical to any security person’s job. If you don’t have management support to shut down services, put up a firewall, install
file-integrity, enforce policies, etc. then there is not much use to be around. All you are going to be doing is fighting fires and that can get old real quick. As security professionals we have to keep managers informed with
what is going on in the security world in plain terms. Some managers don’t understand the language of security people. A little bit of downtime to enable security is a small trade off compared to the downtime of a compromise or a well=publicized hack. I have been very fortunate to have very supportive supervisors. If you want to get their support then provide them hard facts and not theory. That doesn’t mean you have to run an exploit to prove your point but you can show them logs from your Intrusion Detection Sensor’s or logs from the firewall or from any other
devices you run.
LS: What got you interested in the field of computers/Linux/security?
Brian: Apparently I was born with an inherent distrust and paranoia. When I got my first computer, an Apple ][, I spent ages developing a completely secure ‘hello’ program. In order to boot off the disk, you needed several passwords, and could not drop out of it using any of the existing methods such as ctrl-reset, etc. You couldn’t boot off a separate disk to access the files because I slightly modified the disk layout to be not compatible. When I try to think back towhy I did this, I have no idea. There wasn’t a damn thing on those
disks that was private or, for that matter, interesting even to me. Just the ‘help Brian to learn to spell’ program that my mother and I wrote.
My actual interest in computers started from that very machine. I was playing Snake Byte (ahh, I miss that game) and hit ctrl-reset by mistake. I was dropped down to an asterisk prompt, and discovered it could do simple integer math. 2+2==4, 6+3==9. However when I got to higher results, such as 5+7 I was confused by the answer “C”. It took me a while to figure out what was going on, and soon I learned to think in hexadecimal, though it was many years before I learned what it was called. Knowing only some apple DOS and BASIC commands, I tried ‘list’ to see what the program looked like, but at the assembler prompt. “L” meant “List a page of assembly instructions”, “I” meant inverse, “S” meant “Step one instruction”, and “T” meant “trace execution”. So instead of getting a look at a BASIC program, I was accosted by pages of machine code in a manner less friendly that gdb.
I think that was my “Eureka!” moment where suddenly I knew there was something completely foreign out there, and I wanted to learn everything about it.
Since then, my drive for security has come from both directions. I’d be on a project where we were supposed to RCS everything, but the administrators failed to put us in the proper groups to be able to do our work. Some of their build process used sudo to chown files to the install user before packaging them up. This meant we all had unrestricted chown access, although that fact was buried deep inside the Makefiles. So, a quick ‘sudo chown bri /etc/group; vi /etc/group; sudo chown root /etc/group’ was all it took for us to get our work done and make our deadline.
The same personality traits that make me want to find the weaknesses in systems is the one that makes me want to lock things down as much as possible too, and I think most security freaks are the same way.
Duane: I got into security after a security incident occurred at an organization I was affiliated with. It became a sort of obsession trying to understand why someone hacked their server. The server that was attacked had nothing significant on it. However, I soon learned that there has tobe nothing of significance on the computer by simply having high-speed access, large storage space, or just having an Internet connection was
enough for someone to compromise a server. That year SANS had some of their courses freely available and I went through those documents and audio files with a fine-toothed comb. I remember spending Christmas that
year at my alma matr’s computer systems learning how to interpret syslog messages, how to enable tcp_wrapper support, using Crack, and how to run nmap. My supervisor, Bob, and my best friend, Bone, both knew security
was important and welcomed and supported any advice.
LS: Brian, I see that you’re donating all online proceeds from sales of Hacking Linux Exposed, Second edition to the Electronic Frontier Foundation.
Brian: I’m extremely worried about many recent changes that are affecting the world of technology — the DMCA and the continued threat of decreased privacy being the most obvious. I’m a member of the EFF[13], and thought this would be another good way to support them. We’ve set up ‘associates’ accounts with Amazon and Barnes and Noble, and all the money we get through those accounts we’ll send to the EFF. This actually goes for any purchases that ‘originate’ at our site too. See our books page for more info.
LS: Duane, during the many email conversations we have had, you have expressed some ideas about security related education, would you mind explaining those a little?
Duane: I would like to see a more systematic approach to teaching the basics of security and making this information freely-available. If System Administrators took care of the most common vulnerabilities then that would greatly reduce their chances of being compromised and make their job less frustrating. I know the time investment it takes to administer many servers but there has to be time for security.
- Onsight Inc. – http://www.onsight.com/
Onsight offers security design and implementation consulting, as well as on-site training in Unix and Network security, Basic Perl programming, Advanced Perl programming, CGI programming using Perl, Tcl/Tk, XML and JavaScript.
- Stunnel – http://www.stunnel.org/
Stunnel is a program that allows you to encrypt arbitrary TCP connections inside SSL (Secure Sockets Layer) available on both Unix and Windows.
- National Climatic Data Center – http://lwf.ncdc.noaa.gov/oa/ncdc.html
NCDC is the world’s largest active archive of weather data.
- Linux Security – Tips, Tricks, and Hackery – http://lists.onsight.com/
This newsletter shows both defensive and offensive security tricks you can apply to your Linux (and other Unix-like) systems.
- Linux Security Module – http://lsm.immunix.org/
The Linux Security Modules (LSM) project provides a lightweight, general purpose framework for access control.
- Systrace – http://www.citi.umich.edu/u/provos/systrace/
Systrace enforces system call policies for applications by constraining the application’s access to the
system. - LIDS – Linux Intrusion Detection System – http://www.lids.org/
LIDS is a kernel patch and admin tool to enhance the linux kernel security.
- Grsecurity – http://www.grsecurity.net/
Grsecurity is an extensive set of security patches to the 2.4 tree of Linux kernels.
- SELinux– http://www.nsa.gov/selinux/
SELinux, the Security-Enhanced Linux (previously known as NSA Linux) provides a flexible Mandatory Access Control
system. - OpenWall – http://www.openwall.com/linux/
The OpenWall patch is a collection of security-related features for the 2.2 Tree of Linux kernel,
- EnGarde Secure Linux – http://www.guardiandigital.com/
EnGarde Secure Linux is a comprehensive software solution that provides all the tools necessary to build a complete
online presence, including DNS, Web, e-mail services, and more. - Anti-DMCA – http://anti-dmca.org/
The Anti Digital Millenium Copyright Act Website.
- Electronic Frontier Foundation – http://www.eff.org/
EFF is a donor-supported membership organization working to protect our fundamental rights regardless of technology.
- SysAdmin, Audit, Network, Security – http://www.sans.org/
The SANS Institute is a cooperative research and education organization.
