What is the best resource for really understanding Linux deeply

The best way to understand Linux is to break it, badly, and then fix it.

I'm not sure, when you say a "deeper level", if you're talking about getting more into the command line (some people never leave the GUI that comes with their distro, if you can imagine that), or with really gritty low-level stuff like writing kernel drivers. I'm going to assume more of the former than the latter.

I'd advise you to set reasonable goals for things you'd like to do using Linux and applications running on Linux (run a web server, serve files to Windows hosts, serve DHCP, run a graphical desktop, etc). It's my firm belief that you only learn when you're "doing". Once you've got a goal in mind, pursue it. That means reading man pages (lots and lots of man pages), "HOWTO" documentation, mailing lists archies, random blog posts, and, of course, the documentation that comes with the various programs you're installing or compiling to run on your boxes.

Having taught classroom-based IT certification courses for several years, I believe I can say with some degree of authority that the students who I saw make the most progress were the ones that were doing crazy projects of their own design, and learning by the seats of their pants.

As I said before, man pages, "HOWTO" documentation (http://tldp.org/docs.html and in many, many other places on the 'net) and mailing list archives are your friends. I'd steer clear of any books that talk about kernel internals, at least at this stage. You don't need that kind of deep knowledge to get started.

Talk to other people. Server Fault looks like it's turning out to be a great place to get good advice. Ask questions here, and don't think that you're going to look stupid doing so. If you can get some "face time" with people who are familiar with Linux, go for it. (Though I'd recommend you try and separate opinion from fact. There are as many "holy wars" in the Linux community over differing opinions of how to do things as in any other community-- perhaps more, given the nature of the community.)

To get really "deep" knowledge of the Unix heritage of Linux, you might go for some older Unix-specific administration or reference manuals. I'd steer clear of these early on (at least until you can appreciate the historical nature of the "paths not taken" with Linux as compared to some of the Unix-derived operating sysems).

Not knowing what your overall skill-level is, I will throw a shout-out to a book that my or may not be heplful. I highly recommend "TCP/IP Illustrated" (http://www.amazon.com/Illustrated-Volumes-Addison-Wesley-Professional-Computing/dp/0201776316). It's not Linux-specific at all, but you will be doing so many things that deal with TCP/IP that knowing it "cold" is a no-brainer.

I guess, in short, I'm saying that there's no magic book or books, no super-secret videos that the "masters" learned from, but absolutely no limit to what you can learn if you stick to it and aren't afraid to get your hands dirty.

Only years of experience playing with individual components will give you a deep understanding. Having said that, the vast majority of Oreilly Press books are really well written and perfect if you're not a dummy and have more than 24 hours. :)

There is a project called Linux From Scratch, which shows you how to build a linux distribution from nothing. You may find it educational to do once. It'll be time consuming as you need to compile everything from source and you'll throw it all away at the end.

I find LWN.net's kernel page invaluable for keeping up with how things work in the kernel at a high level.

Other than using it everyday (Which I think is a great thing!), I recommend you start thinking about services that could be implemented for your network on a linux box. Start designing it based on Linux services, research and implement once your ready. There will be hopefully mistakes in the process, and that experience will help you understand the OS at a deeper level.

You will find several free guides at The Linux Documentation Project, guides page. This is a short list i pick from there, but you should scroll through the page to find what you need.

• Introduction to Linux - A Hands on Guide; Jun 2008
• Linux on the Road; Nov 2005
• The Linux System Administrators' Guide; July 2005
• Advanced Bash-Scripting Guide; Mar 2009 (yes, this is good for understanding linux too)

You should look for specific HOWTOs for topics you are interested in. Another date sorted HOWTO list.

There are some starters at the Linux reviews beginners page.

Finally, this is a small book available online -- Linux Kernel in a Nutshell. Its in PDF form of the 2007 edition.

You should use these references only as a feeler to start you own search for things that you need.

For general Unix philosophy and an excellent introduction to the command line, there is Brian Kernighan and Rob Pike's classic The Unix Programming Environment.

Also, IBM's DeveloperWorks web site has over 900 articles in its Linux section. You can browse the article list for topics you find interesting.

Finally, once you're ready to get into the nitty-gritty, go to the source. Grab the source code for the kernel version you're using at kernel.org (your distribution will also have kernel source packages you can install) and check out the Documentation directory. You'll find lots of reference material on kernel internals and configuration settings.

School. To really understand the Linux system you need a wide array of Computer Science systems backgrounds. Compilers, Computer Architecture and Operating Systems. And you need a guide who can show you what's important and correct any misunderstandings you have or form.

Once you understand how modern processors work, and how C compilers exploit them, you can dive into books like Minix 3:

This is the latest version of the book that Torvalds, author of the Linux kernel, (loosely) based his work on. You'll learn about the fundamental components and algorithms of an operating system, and how exactly to implement one: the text comes with a nearly complete printing of the source code to Minix for reference and instructional purposes. Check out the interrupt handler to get a complete understanding of how the system works and where control flows.

And before you scoff at schooling and Computer Science, it's important to note that the jobs Operating Systems are asked to do are generally NP-complete. So understanding a wide variety of available algorithms is critical to performance tuning, since there won't be a provably optimal algorithm.

In addition to the kernel, there's a number of other areas. Because Linux is open source, this is an academic gold mine. Systems like Debian and Ubuntu make their source available, and it's dead simple:

apt-get source package-name

Many upstreams also host their code in revision control, so you can read the most up to date version of a program's source, or even see how it was built years ago.

Just start building your own distro. I've done it for yourself and you know what? After 4 month I've spend on that I know about Linux internals more than guys around me who are using Linux more then 10 years.

The source.

That, of course, if you mean the kernel.

If you are trying to learn the system above the kernel and the API, I would start at learning old school Unix. Maybe try some FreeBSD/OpenBSD/NetBSD/DragonflyBSD. A little Minix could help too. Then move to Slackware, Arch, Debian and Gentoo. They are all different (and come with great documentation), and in those diferencies, your curiousity will make you search why it is so, you will gain a lot of technical, comercial, political and historical knowledge.

Another way would be go diggin' in /etc, old Unix manuals, and of course, Google.

Linux from scratch is a great way of learning GNU/Linux.

Learning in any of the *nix environments is a holistic approach. It involves a series of epiphanies and experiences. This is not by accident. It is the deliberate byproduct of its design, as the original environment was designed around and for computer programmers. It is also its greatest shortcoming, as the focus is on technical aspects and not user experience. It is "a house with a sturdy frame that will last 100 years without service, yet its siding is haphazard and the paint clearly neglected".

Contrast this with Windows, which is the mirror image of this philosphy - do something that gives the user a tangible experience, but shield them from the inner workings of things. It is "a house with beautiful trim and paint, but the foundation that has been built-over several times with multiple work-arounds".

To really start learning, I would suggest building custom kernel images that have options specific to your hardware, and installing them with the options you want. You should fully expect going into this that something, somewhere will break, and you may or may not be able to back things out to "normal". This is a normal part of that learning process and you should approach this as if things will break (i.e. don't make this a primary install, use a separate drive or virtual machine or something...)

I've done an answer on a similar question, so to reduce typing I'll crosslink it here. You'll also want to read the first few paragraphs on this page, which will give you a better feel for what you're in for.