Solaris Performance and Tools: DTrace and MDB Techniques for Solaris 10 and OpenSolaris [Englisch] [Gebundene Ausgabe]

A Sun colleague recently noted that the consistency of interfaces in Solaris isn't a strong point, and she's right. Anyone who understands much of Solaris has to manage many odd and subtle details. While the concepts that drive Unix variants are indeed powerful, it doesn't mean every contributing engineer grasps and implements them the same way.

As a result, there are differing views in topic areas like performance management, including: proper methodology, or "best practices"; which statistics are useful and how to interpret them; which reports may be significant, trivial, or misleading; and of course, which tools help you get them. As a contributing author to Sun Microsystem's course on Solaris performance, I heard many of those views from many experienced trainers, Sun engineers, and other interested parties. The complexity of the topic leads many people to believe they understand it "the one way it is supposed to be understood." The passion is great, so long as it doesn't lead to a narrow-minded zeal.

Solaris Performance and Tools punts on such religious matters. In my view there are some good and some disappointing outcomes. The book covers two primary areas. One, it is a detailed looks at programs used to measure system and process performance. The coverage ranges from the obvious and everyday to the highly technical and obscure. Second, there are some brief but helpful introductions to mdb and Dtrace, the killer analysis tool introduced with Solaris 10. This book doesn't often propose a method or application of these tools. It does present what the authors feel are 'the' important ways to measure CPU, disk, and I/O efficiency, but relies more on lots of output from lots of tools, commenting on them only occasionally.

There are a lot of listings: command output, script or C code, grepped output. As with the companion book, Solaris Internals, they are not indexed or captioned. In this book, however, these grey boxes aren't annotated either! They are simply left for the reader to study. This idea of printing a book would bring little more than a shrug ten years ago. There wasn't much else you could do with a closed codebase and so few online references. Has nothing happened to improve on that situation?

The code listings appear in Bourne shell, Perl, Dtrace, or C, so the reader must know how to interpret them all to profit from the discussion. But even for a peer technical reader, some kind of analysis, key-line commentary, or occasional emphasis on nonbovious lines...some help would be nice. I know programmers find commenting a time-consuming chore, but a peer reader could do much of the work this book shows on their own, and spare the trees.

The command-line output does illuminate the discussion, as it should. However, it feels like filler after a ehile when you're reading sample output for ping, traceroute, snoop, output for multiple prstat and ps options, not to mention numerous trivial examples of various process tools, such as pkill and pstop. What are we getting from this? If there was something important to say about them, fine, but again, there's no commenting provided.

The notes on observing CPU, disk and I/O measurements are detail-driven and idiomatic. The focus seems to be on subtlety and non-obvious aspects of statistics that either aren't well-explained in other references, or are widely misinterpreted.

I'm happy for the discussion on mdb and kstats. These are hard subjects to absorb. The online documentation for them is lengthy, hard to gloss, and (of course) poorly-commented where sample code or output is shown. This book gets down to the point and makes the task of learning these tools seem far less daunting.

A key stength of this book is the thorough review of tools and what they do. The book would serves well as a reference when a terminal window is not available. The Dtrace Toolkit is reviewed at length, but there is equally useful coverage and more examples online.

Be advised: the front matter and back matter of this book are the same as the Solaris Internals book, not including the table of contents and index. I mention this because it seemed peculiar that the bibliography for a book on performance, mdb, and dtrace referencesd nothing published in the last six years.

"Solaris Internals" and its predecessor "Sun Performance and Tuning" are wonderful books for giving you the knowledge to know whats actually happening under the covers, but many SA's admit struggling when it comes to translating that into usable day-to-day understanding of the systems on which they manage. Just knowing how it works isn't enough to be really useful, what you need is the ability to look at the system and work out how what your seeing fits what you know.

"Solaris Performance and Tools" bridges that gap. Every page, cover-to-cover is filled with practical examples and explanations of the tools that let you actually see what Solaris is doing. If you've tended to rely on only a handful of tools such as vmstat, iostat, netstat, sar, and prstat, then you really want to get this book and start digging much deeper. Even as a Sr Admin I found that there were wonderful tools available that I didn't even know existed (such as "intrstat").

In particular, this book unlocks two powerful tools in Solaris 10 that can be as complex as they are powerful: DTrace and mdb. Both of these give you unparalleled power to dig your fingers into the system, but using them beyond simple one liners is more difficult than most people admin. This book gives you a great step-by-step approach to learning both. While a one-line DTrace script found in a blog might help you here and there, you won't truly understand how powerful DTrace can be untill you've built

a firm foundation on which to build your own. This book is the best way to jump start that process.

This truly is the only book available that opens the window to whats possible in Solaris in such a practical way. You'll find things you didn't know, you'll start understanding how things work by putting practical numbers on YOUR system together with the knowledge you aquired from "Solaris Internals" and you'll start solving problems rathan than just knowing why somethings broken or slow.

Every Solaris SA should have a copy of this book on their desk for quick and easy reference. Stop guessing, start knowing.

I will make this as short as I can, unlike the one for the companion book, Solaris Internals. I have been troubleshooting Sun Solaris for 15 years, in one version or another. Crash dump analysis was the main way to get data from within the kernel and only if the system blew a gasket. There have been different methods through the years,crash, kdb, and mdb are the main ones, but now with Solaris 10 you can add a powerful tool to your knowledge tool box, DTrace. This is built in to the system code so its not a seperate program that you run, it lives in Solaris and you enable the probes you want to see. Interpreting the data is not easy if you dont know what you are looking at, so the Companion book tells you what the internal workings are so you can know what you are looking at. This book tells you how to find the most used issues or problems. It covers these things in more detail than you can find unless you work in and engineering lab and program apps for Solaris. Solaris 10 has many things in it that can throw an admin, Zones for instance, can throw you if you are having some type of performance issue, but what can you do to get the data from the kernel to watch the internal processes deep under the hood? DTrace should be the first thing out of your mouth. This is a top notch book and I understand other people's issues or questions with it, however, assume you have not touched Solaris 10 in production and your company is doing a technology refresh and migration to new Sun Hardware and Solaris 10. How are you going to help your company troubleshoot issues in this new envrionment? You will use DTrace and any other tools you can. I use DTrace almost every day. I did today.

So if you want to know how Dtrace and mdb and other utility commands are used, buy this book and sit down on a Solaris 10 box and start typing. If you want to see other people's take on it, go to one of the Author's web sites. Brendan Gregg. You can look him up on the web. This is a fine fine add on book and I have two copies of it too. One at home and one in my book bag. It is one of my top 10 carried books when I travel.

Kudo's to Brendan, Rich, and Jim.

Thanks and keep up the good work.

Sincerely

Bill Branson