http://packetlife.net/blog/2010/mar/10/the-science-of-network-troubleshooting/

The essentials which I won’t cover:

ls, grep, sed, awk, cat, less, head, tail, ..

I’m sure there are others. These should automatically just be extensions of your brain — you need to be intimately familiar with them to be productive on a command line.  If you aren’t, I suggest you search around the net, there are thousands of tutorials to bring you up to speed on each one individually, and then you can progress to chaining them together.

On to the real meat of this post:

netstat
This really should be included above, but it does have some special uses for application debugging. Useful flags:
netstat -anlp | grep PID

nc
Netcat has become the multi-tool of connection testing, for what we used to use “telnet” to establish simple tcp outbound connections, nc can now provide that, plus a listening mode to receive incoming connections. This is especially useful for validating firewall configurations before your applications ever get installed. Plus, combining nc with chained commands such as tar or gzip can make for some very quick file transfer mechanisms (bypassing ssh/scp’s performance limitations). Common uses:
nc host port — Connect outbound to a host:port
nc -l 8080 — Listen for a connection on 8080 and exit when closed.

lsof
Handy way to list the open files/handles/sockets from a process. Common flags:
lsof -nPp PID

strace
Awesome utility to monitor the system calls an application makes. Having problems debugging an app that doesn’t seem to read your configs? Or hangs every 30 seconds? Fire up strace and attach to the pid, to find out that it’s reading the wrong path, or connecting to a downed service! Want to find the longest or most frequent running system calls? No problem! The volume of info and ease of use strace provides makes it an essential part of your toolkit. Common flags:
strace -cp PID
Will give you a nice table that counts the syscalls and sorts them, as well as the time spent executing.
strace -ttTp PID
Spits out the timing down to the microsecond of system calls.
Add -f to follow forked processes as well (handy for things like apache pre-fork, or any similar threaded/forking application). Make sure to use -o FILE to write out your output, it can move pretty quick!

gdb
The glorious debugger. If you are here, it’s probably because you have a poorly behaving app, that is core dumping. strace can only go so far.. you want to find the problem code, and kick it back to the developers. GDB will help here. We can attach to a process (which will pause it), then continue the process, and have the application perform whatever causes the core dump. At that point, GDB should spit back the problem line, and hopefully provide you a window into the problem. Common use:
gdb [binary name] [PID]
Issue “c” to continue the app. Generate your problem/seg fault, and observe the console output.
Beej has a pretty slick guide to gdb: http://beej.us/guide/bggdb/