Brian Hatch & Duane Dunston Interview

This is a re-post (or posting to my blog) of an interview I had done for Linux Security in September of 2002. The original article is available here.


Just to give everyone an idea about who writes these articles and feature stories that we spend so much of our time reading each day, I have decided to ask Brian Hatch and Duane Dunston, the newest members of the LinuxSecurity.com team, a few questions. I’d also like to begin by thanking both Brian Hatch and Duane Dunston for allowing us to conduct this interview.

Before we get to that, let me give you a brief introduction as to who these folks are, although I promise you that you will receive plenty of insight the further you read:

Brian Hatch is Chief Hacker at Onsight, Inc.[1] where he is a Unix/Linux and network security consultant. His clients have ranged from major banks, pharmaceutical companies and educational institutions to major California web browser developers
and dot-coms that haven’t failed. He has taught various security, Unix, and programming classes for corporations
through Onsight and as an adjunct instructor Northwestern University. He is also co-maintainer of Stunnel[2], an Open Source secure SSL wrapper used around the world to encrypt clear text protocols.

Brian Hatch has been interviewed several times on his stances on security, full disclosure, and Open Source technologies. (See interview 1 and interview 2.) Rather than retrace this ground, we thought we’d ask questions geared toward his personal rather than professional endeavors.

Duane Dunston is a Computer Security Analyst at STG Inc. for the National Climatic Data Center[3] in Asheville, NC. He received his B.A. and M.S. degrees from Pfeiffer University and he has his GSEC certification from SANS[14]. Not only does he enjoy his work in computer security, he also likes to get involved in its ever-growing technologies.

LS: How did you attain your security knowledge? Did you get a CS degree, or was it on your own?

Brian: I had your formal four year college education, but during that time had only one official computer class – Pascal – because they wouldn’t teach C at the time. All my Unix, C, and security knowledge was self taught from the man pages, which is definitely not the way I’d suggest for maintaining sanity. Nowhere is there an ‘asterisk means pointer’ man page. Suffice it to say, my early C code was ugly as sin. Luckily I’ve deleted every copy, and burned the tape backups.

Frequently I’m brought into security gigs in early stages to make decisions and get infrastructure in place, and then hand off the project to an employee, which means I frequently find myself in a position where I make hiring decisions. Because of my non-formal background, I am able to go in without preconceptions that certifications are the only mark of quality, and am able to get some extremely qualified folks hired even though they are sans certifications. (Pun intended.)

Had I been a CS major, there still would have been no chance to take security-related classes at the time anyway – there was no security emphasis at the time. There was a small group of us who would compare notes and tricks, and this was invaluable, more than any formal classes would have been. Hopefully more educational institutions are integrating security concepts into their curriculum, because without it the code being whipped up throughout the world will continue to be largely crap. Programs with buffer overflows are unconscionable, and yet they pop up in every product out there still, despite being a known quantity with well defined solutions.

Duane: I graduated from Pfeiffer University in 1997 with a BA in Sociology, the day after I ended up in Thaliand for about 5 months teaching English. When I returned there wasn’t much work for a Sociology major straight from Thailand teaching English.

I had some computer experience prior to that, though. My first PC was a Gateway 486 which I got back in ’94. I started shadowing my best friend and former boss at work and there I was introduced to Linux.

I continued to use Linux through Graduate school and learn more and more about security. After realizing I didn’t want to be a Manager, which is what I have my graduate degree in, I decided I wanted to stay with computers but with Security. Nearly all of what I have learned has been on my own from websites, books, and having support from my managers to just experiment.

LS: What are some of your other hobbies?

Brian: Between writing books, writing the Linux Security: Tips, Tricks and Hackery newsletter[4], preparing for a Linux security class I’ll be teaching in January, answering questions about Stunnel and maintaining the Stunnel website, trying to keep up with current security issues on mailing lists like Bugtraq and VulnWatch, web pages like href="http://www.linuxsecurity.com/">LinuxSecurity.com and Help net Security, something called a “Day Job”, and a somewhat foreign concept in the computer geek circle called “wife and kids”, I’ve very little free time.

I like to read a lot, though the topic depends on the location. By myself, I usually pick security and geeky titles – I’ll re-read W. Richard Stevens’ books now and then for fun. But much more of my time is spent reading books such as Dinosaur Roar, Tumble Bumble, or Sheep in a Jeep to my daughter. If she stops paying attention, I start sneaking new characters in, such as Dmitri Dog, who is hounded by the evil Ashcroft Crow. I figure I need to sneak in anti-DMCA[12] sentiments early via allegory.

Duane: I don’t have many hobbies outside of computers. My interest tend to be around a particular aspect of computers. In particular, parallel computing, cryptography, finding good uses of open source in the business world, and computer forensics. I do have an interest in the arts. In particular, what is going through an artist’s mind as he or she is working. Two of my friends are artist and I really enjoy being around them, and other artists, and listening to them brainstorm ideas or talk about what their work means to them.

LS: Supposing you had free time, what would you be doing with it?

Brian: I’d devote some time to helping out the Linux Security Module project. I hope to help port systrace to LSM next year. Currently it is a kernel patch, and I think the community would be served better in the long run by having it available as an LSM module, which would make it more accessible to those who fear kernel compilation.

And some day I hope to get around to turning some of the megs of perl code I’ve written over the years into well defined Perl modules for CPAN. Then I won’t be the only one supporting this spaghetti code. ;-)

If I had infinite time, I’d learn to play the Hammered Dulcimer and French Horn. There’s nothing in the world as musical as a well-played French Horn.

LS: In your opinion, what is the most interesting thing about Linux and Security?

Brian: The first thing is that, with Linux, security is a possibility. It is not an end point – you must constantly keep abreast of new attacks and revisit your security posture – but there is nothing that is unavailable to you if you want to look. Closed source systems can never offer this. By design, be it chosen for monetary reasons or to prevent competition, closed source products always hide details from the users and administrators that could be critical to understanding how thing function, and how they can be broken.

One of the beauties of Linux (and other open systems, such as *BSD) is that you can use them to boost the security of those closed source machines. By the liberal application of Linux machines throughout your infrastructure, you can keep those exploits-waiting-to-happen locked down where they can do less harm. For more of my ranting on this topic, see my article Linux is Securable — I won’t waste time rambling here.

What is most intriguing right now on the Linux horizon is the evolution of security controls. In the beginning, all you had to work with were file permissions. Root could do absolutely anything unchecked, and root access was required for some things such as binding low network ports or opening raw sockets, which meant use of set userid bits on programs, which frequently were broken to gain root access.

Next came capabilities, where each bit of root’s power was defined in more specific terms. When determining if a process could bind port 80 originally you’d check to see if uid==0. Now you’d check if the process had the CAP_NET_BIND_SERVICE capability. In theory, you could now remove capabilities from the system – for example removing the ability to load kernel modules ever again, which is good for defending against malicious LKMs.

This compartmentalization was a good evolutionary step, but it didn’t get used much in the real world. You had only global access to turn off a capability, and you’d never have it available again, except by rebooting. You couldn’t assign a capability on a process-by-process level, which made it much less useful.

Then came some excellent security kernel patches. LIDS[7], Grsecurity[8], SELinux[9], and others. Finally we had ways to give abilities to only the processes that needed them. We could completely lock down our machine such that any deviation from what we have allowed will fail. Should an unknown buffer overflow occur, the cracker can’t bind an inbound rootshell on port XYZ, it won’t be able to read files outside those we’ve allowed, and won’t be able to write to the file system anywhere. If they break in as root, they still can’t change anything in /etc unless they know yet another password, they won’t be able to Trojan files in /bin, or insert a loadable kernel module into the system. Life is good.

But, life could be better. These patches are not integrated into the kernel. You won’t get them on your Red Hat CD, or your Debian download. (Though some places do ship hardened kernels, such as Owl which contains the OpenWall[10] kernel security patches, or EnGarde[11], which comes with LIDS.) So normal users need to go out of their way to patch and recompile the kernel – a daunting task.

So, the current bright spot on the Linux security horizon in my opinion is the Linux Security Module infrastructure, which promises to make it possible to load these advance Linux security policy behaviors without a kernel recompile, thus removing a significant barrier to the new Linux administrators. If we can get more of the LSM modules to be stackable where appropriate, we’d be able to pick and choose the best of the features of each, something which currently involves manually merging kernel patches – quite a painful endeavor.

So, as we enter this new era of Linux security, you bet I’m excited. Just tell your manager that it’s a “Paradigm shift” and they’ll give you time to play with it too.

Duane: Computer forensics reigns supreme in my opinion. These experts provide us
the information we need to fine tune the tools we currently use and the tools needed to prevent attacks from occurring in the future. Sifting through megabytes or gigabytes of data to find one clue as to how someone compromised a system has to be tedious but the information they find helps us quickly learn how to patch the vulnerability and create tools to help detect how a system was compromised. I just don’t think the security industry can survive without forensic experts. There are many people who aren’t forensic “experts” who can and do identify how a system was compromised and they also provide the community with valuable information. However, the forensic experts can dig down and find the root of the problem and the information they provide can then be used to help fine tune currently used tools and methods.

LS: What are some of the lessons you you have learned in your years in computer security?

Brian: Well, in no particular order, here are a few things that pop to mind.

  • Never bother with a technological hack when a social engineering one takes ten seconds.
  • Posting a piece of paper listing a ‘contest for the most creative and verifiable username/password’ in a college campus food court will get you several hundred valid entries you can use at will.
  • The easiest way to get access to a friend’s account is to get them to ‘xhost +’ to play network xtank.
  • The next easiest is to untar /etc/shadow from the backup tapes that the administrator leaves next to the tape drive.
  • Sometimes the perfect hack is just one click away.

  • When you have a frustrating week, let the script kiddies amuse you. Set up a honeypot with as much known-buggy software as you can find. Break almost all the tools. Don’t include vi or emacs, make them learn ex. Include perl, but no modules. Install the Fortran compiler, but no others. Include netcat, but not telnet. Make the default shell tcsh, and remove all Bourne shell variants. Delete all terminfo/termcap definitions. Don’t include more,
    less, or any other pager. Compile a version of cat that randomly swaps letters every so often.

  • Never put that honeypot anywhere near your actual machines.
  • Use mutt. A good mailer is key to getting work done, and you can’t be secure if you’re too busy wading through your email.
  • If you can’t do it from the command line using Netcat, you don’t actually understand how it works.
  • If you can give the root password to your enemy and he can’t compromise your system, you’ve done a good job securing it.
  • Every now and then, lock the root password on your own machine and try to recover it. You’ll learn where your own weakness are, and hone your skills.

  • Make sure you have a throttle on any software that can page you.

  • Never rename /usr/lib/libc.a.
  • Always have a statically linked shell, in case another administrator renames libc.a and you need to re-create it using nothing but shell built-ins.

  • And check your logs. Always check your logs.

Duane: The most important lesson I have learned is that management support is critical to any security person’s job. If you don’t have management support to shut down services, put up a firewall, install
file-integrity, enforce policies, etc. then there is not much use to be around. All you are going to be doing is fighting fires and that can get old real quick. As security professionals we have to keep managers informed with
what is going on in the security world in plain terms. Some managers don’t understand the language of security people. A little bit of downtime to enable security is a small trade off compared to the downtime of a compromise or a well=publicized hack. I have been very fortunate to have very supportive supervisors. If you want to get their support then provide them hard facts and not theory. That doesn’t mean you have to run an exploit to prove your point but you can show them logs from your Intrusion Detection Sensor’s or logs from the firewall or from any other
devices you run.

LS: What got you interested in the field of computers/Linux/security?

Brian: Apparently I was born with an inherent distrust and paranoia. When I got my first computer, an Apple ][, I spent ages developing a completely secure 'hello' program. In order to boot off the disk, you needed several passwords, and could not drop out of it using any of the existing methods such as ctrl-reset, etc. You couldn't boot off a separate disk to access the files because I slightly modified the disk layout to be not compatible. When I try to think back towhy I did this, I have no idea. There wasn't a damn thing on those
disks that was private or, for that matter, interesting even to me. Just the 'help Brian to learn to spell' program that my mother and I wrote.

My actual interest in computers started from that very machine. I was playing Snake Byte (ahh, I miss that game) and hit ctrl-reset by mistake. I was dropped down to an asterisk prompt, and discovered it could do simple integer math. 2+2==4, 6+3==9. However when I got to higher results, such as 5+7 I was confused by the answer "C". It took me a while to figure out what was going on, and soon I learned to think in hexadecimal, though it was many years before I learned what it was called. Knowing only some apple DOS and BASIC commands, I tried 'list' to see what the program looked like, but at the assembler prompt. "L" meant "List a page of assembly instructions", "I" meant inverse, "S" meant "Step one instruction", and "T" meant "trace execution". So instead of getting a look at a BASIC program, I was accosted by pages of machine code in a manner less friendly that gdb.

I think that was my "Eureka!" moment where suddenly I knew there was something completely foreign out there, and I wanted to learn everything about it.

Since then, my drive for security has come from both directions. I'd be on a project where we were supposed to RCS everything, but the administrators failed to put us in the proper groups to be able to do our work. Some of their build process used sudo to chown files to the install user before packaging them up. This meant we all had unrestricted chown access, although that fact was buried deep inside the Makefiles. So, a quick 'sudo chown bri /etc/group; vi /etc/group; sudo chown root /etc/group' was all it took for us to get our work done and make our deadline.

The same personality traits that make me want to find the weaknesses in systems is the one that makes me want to lock things down as much as possible too, and I think most security freaks are the same way.

Duane: I got into security after a security incident occurred at an organization I was affiliated with. It became a sort of obsession trying to understand why someone hacked their server. The server that was attacked had nothing significant on it. However, I soon learned that there has tobe nothing of significance on the computer by simply having high-speed access, large storage space, or just having an Internet connection was
enough for someone to compromise a server. That year SANS had some of their courses freely available and I went through those documents and audio files with a fine-toothed comb. I remember spending Christmas that
year at my alma matr's computer systems learning how to interpret syslog messages, how to enable tcp_wrapper support, using Crack, and how to run nmap. My supervisor, Bob, and my best friend, Bone, both knew security
was important and welcomed and supported any advice.

LS: Brian, I see that you're donating all online proceeds from sales of Hacking Linux Exposed, Second edition to the Electronic Frontier Foundation.

Brian: I'm extremely worried about many recent changes that are affecting the world of technology -- the DMCA and the continued threat of decreased privacy being the most obvious. I'm a member of the EFF[13], and thought this would be another good way to support them. We’ve set up ‘associates’ accounts with Amazon and Barnes and Noble, and all the money we get through those accounts we’ll send to the EFF. This actually goes for any purchases that ‘originate’ at our site too. See our books page for more info.

LS: Duane, during the many email conversations we have had, you have expressed some ideas about security related education, would you mind explaining those a little?

Duane: I would like to see a more systematic approach to teaching the basics of security and making this information freely-available. If System Administrators took care of the most common vulnerabilities then that would greatly reduce their chances of being compromised and make their job less frustrating. I know the time investment it takes to administer many servers but there has to be time for security.


  1. Onsight Inc.http://www.onsight.com/

    Onsight offers security design and implementation consulting, as well as on-site training in Unix and Network security, Basic Perl programming, Advanced Perl programming, CGI programming using Perl, Tcl/Tk, XML and JavaScript.

  2. Stunnelhttp://www.stunnel.org/

    Stunnel is a program that allows you to encrypt arbitrary TCP connections inside SSL (Secure Sockets Layer) available on both Unix and Windows.

  3. National Climatic Data Centerhttp://lwf.ncdc.noaa.gov/oa/ncdc.html

    NCDC is the world’s largest active archive of weather data.

  4. Linux Security – Tips, Tricks, and Hackeryhttp://lists.onsight.com/

    This newsletter shows both defensive and offensive security tricks you can apply to your Linux (and other Unix-like) systems.

  5. Linux Security Modulehttp://lsm.immunix.org/

    The Linux Security Modules (LSM) project provides a lightweight, general purpose framework for access control.

  6. Systracehttp://www.citi.umich.edu/u/provos/systrace/

    Systrace enforces system call policies for applications by constraining the application’s access to the
    system.

  7. LIDS – Linux Intrusion Detection Systemhttp://www.lids.org/

    LIDS is a kernel patch and admin tool to enhance the linux kernel security.

  8. Grsecurityhttp://www.grsecurity.net/

    Grsecurity is an extensive set of security patches to the 2.4 tree of Linux kernels.

  9. SELinux- http://www.nsa.gov/selinux/

    SELinux, the Security-Enhanced Linux (previously known as NSA Linux) provides a flexible Mandatory Access Control
    system.

  10. OpenWallhttp://www.openwall.com/linux/

    The OpenWall patch is a collection of security-related features for the 2.2 Tree of Linux kernel,

  11. EnGarde Secure Linuxhttp://www.guardiandigital.com/

    EnGarde Secure Linux is a comprehensive software solution that provides all the tools necessary to build a complete
    online presence, including DNS, Web, e-mail services, and more.

  12. Anti-DMCAhttp://anti-dmca.org/

    The Anti Digital Millenium Copyright Act Website.

  13. Electronic Frontier Foundationhttp://www.eff.org/

    EFF is a donor-supported membership organization working to protect our fundamental rights regardless of technology.

  14. SysAdmin, Audit, Network, Securityhttp://www.sans.org/

    The SANS Institute is a cooperative research and education organization.

First Attempt at Poetry: No Substitutes

Although I already posted my poem on Hanukah first, that was just because it was Hanukah and I wanted to get it up there. This is actually the first one that I wrote. Its shabby, but it got me going.



# No Substitutes

foreach (%challenge) {
  $i; wait;
  $i =~ study each %challenge;

  foreach ($programming{language}) {
    $i = tr/y/$_/;
    $i; push @away and goto Perl;
    $no. sub { FOR: { $perl } } if $Perl{exists};
  }

AND:
  foreach ($problem{solved}) {
    my $knowledge++ and $Perl{love}++;
  }
}

What Is Slapper?

This is a re-post (or posting to my blog) of an article I had written for Linux Security in September of 2002. The original article is available here.


The question of the week: What is Slapper? Let me begin by telling you I am not only describing the Slapper worm, but I am also describing the Apache/mod_ssl worm, the bugtraq.c worm, and the Modap worm. In effect, this is just 4 different names for the same nasty worm.
On the always lucky day of Friday the 13th (of September) the first
reports appeared on Bugtraq of an active worm exploiting the OpenSSL buffer overflow vulnerability reported at the end of July. The next day, CERT issued an advisory CA-2002-27, the Apache/mod_ssl Worm.

A quote directly from the CERT issued advisory prior to the release of the worm:

Compromise by the Apache/mod_ssl worm indicates that a
remote attacker can execute arbitrary code as the apache user on the
victim system. It
may be possible for an attacker to subsequently leverage a local
privilege escalation exploit in order to gain root access to the
victim system. Furthermore, the DDoS capabilities included in the
Apache/mod_ssl worm allow victim systems to be used as platforms to
attack other systems.

By Sunday September 15th, F-Secure Corporation reported
13,000 infected servers out of “over 1,000,000 active OpenSSL
installations in the public web.” Businesswire gave a more in-depth
view into just how F-Secure got their numbers:

During the weekend following Friday the 13th, F-Secure
engineers have reverse engineered the peer-to-peer protocol that the
worm uses. F-Secure has now infiltrated the Slapper peer-to-peer attack
network, posing as an infected web server. Through this fake server,
the exact number of infected machines and their network names can be
identified.

Updates to fix the problem, including backports to earlier versions of OpenSSL, had been available
for over a month from the OpenSSL project, Caldera, Conectiva, Debian,
EnGarde, Eridani, Gentoo, Mandrake, OpenPKG, Red Hat, SuSE, Trustix and
Yellow Dog.

SecurityFocus has completed and released a full analysis (PDF format) of the worm in addition to their initial incident Alert (PDF format). F-Secure is maintaining a “Virus Description” of this worm with lots of interesting information.

The Linux.Slapper.Worm spreads in similar fashion to last year’s
Nimda and Code Red worms, by scanning for, and then infecting,
vulnerable systems. Because this worm establishes peer-to-peer links
among infected servers, experts fear it could create a powerful
platform to launch denial-of-service attacks against virtually any
target on the Internet.

Some of the more noteworthy (interesting) things thats the
Slapper Worm does are similar to the Apache Scalper worm. A major
difference is that Slapper creates a hierarchical network structure.
The Security Focus Analysis states:

The Modap Worm, like Scalper, implements many
innovative structures, including a hierarchical network structure in
which it keeps track of the systems it has infected, the system that
infected it, as well as a list of other infected systems and how many
hops away they are. All of the internal communication between hosts
infected with Modap is accomplished through an implementation of a
stateful protocol transmitted over UDP.

Once the worm has infected a system and created the necessary file
(below), it executes itself with at one command line parameter. If it
is not executed with at least one command line parameter, then it
displays an error message and does not run. Now that the worm is
running, the first thing it attempts to do is bind to UDP port 2002.
The bot (worm) then sends out a packet to register itself on the
network [of other worms]. Now that the worm is bound to a port, it
enters a daemon mode and forks and installs signal handlers for SIGCHLD
and SIGHUP which point to an empty function. The worm now enters a
while loop where it just scans and propagates.

The way the worm propagates is it begins by scanning for hosts
that are listening on port 80. Once a system is found, it send the
following string:

GET / HTTP/1.1\r\n\r\n

Since a “400 Bad Request” reply is generated, the worm now has
information about the server to look at. It parses the information
given with the response and determines weather or not it has just
contacted an apache server. The worm checks the response string to see
if the version of apache as well as the operating system are
vulnerable. If the operating system or the apache version don’t match
anything the worm has, then it uses the default attack.

F-Secure has charts which illustrate how many hosts are/were
infected at a given time. Although the count was nearing 20,000 hosts
as of 17 September, the number has been drastically reduced between
patching and emails to system administrators. The number is supposedly
down to below 1,000 at the time of this article writing.

One of the main characteristics associated with Slapper is the
file names that it creates. It creates 3 files within the /tmp
directory:

/tmp/.bugtraq This is the copy of the worm that is running on the infected system.
/tmp/.bugtraq.c This is the source code to the worm that is running on the infected system.
/tmp/.uubugtraq This is the uuencoded copy of the worm that is running on the
infected system. This file is also used by the worm to propagate itself
to other systems.

Media References include:

RUS-CERT has made available a tool to remotely detect vulnerable servers. However, Eric Rescorla has observed behavior different from what that tool expects.

If you have yet to apply a patch, I would strongly urge you to
do so now. If reading this article has not convinced you, then go apply
the patch to spite me. If you are unsure of where to obtain a patch for
your version of linux, Linux Security Advisories has a list of all the advisories by vendor.

Much of the information stated in this document is available via the sources and references listed throughout this document.

Happy Hanukah

I don’t often write poetry. In fact, I am not usually a great writer of poetry (since I don’t read much of it anyway). However, I do, for some odd reason, find it fun to write programming poetry and Perl just makes it so easy. So admist all the holiday cheer (mostly Christmas) and in light of seeing this Christmas Poem, I decided to throw together my own version for the Jewish population. Enjoy and Happy Hanukah!

BEGIN {
  if ($kids) { write $santa; dump $kids; }
  foreach (@night) { study $prayers and $stories; }
  select $gentiles; #to
    join "you"; if ($gentiles) { open $MIND,"ed"; }
}

foreach $night (1..8) {
  wait until $sundown;
  y/light/candles/;
  tell $oil_story;
  seek $matches, $candle, 0;
  bless $candle, $prayers;

  LIGHTING: {
    $candle unless $lit; next $candle;
    redo LIGHTING until last, menorah, is, lit;
  }
  goto FOOD unless ($young_children);
  join $gifts, $child;
  open ALL, $gifts;

  FOOD:
    do {
      unpack "FOOD", $table;
      chop $potato_latkes;
      sort @tater_tots;
      chop $brisket;
      open $applesauce, "packages";
      sin unless (bagles and lox); exists $on{table};
      require fork;
      if (exists $jewish{mother}) { use constant feeding, "rules"; };
      while ($jewish_household) { }
        continue { $eating until ($stomach > $full); }
    } while ($food eq "Kosher");

  TRADITIONS: {
    tell $stories;
    listen $stories, $history;
    setpriority $children, $jewish{spouse}, $marriage;
    do { $jewish_geography if (local $family); }
  }
  accept $gelt, $family;
  push @dreidel, $around_room;
  read($it,$happened,$over_there) unless (local $Israel);
}

do $mitzvot until $tired;

require $all; tell $everyone; { "Happy Hanukah"; }
Posted in Perl, Poetry. 1 Comment »