Juan Paul

July 20, 2010

Bringing Dev and Ops closer together through metrics and dashboards

Filed under: DevOps — juan paul @ 4:30 pm
Tags: ,

I recently had a great visit from Damon Edwards and John Willis over at DevOps Cafe. They stopped by Shopzilla’s Los Angeles HQ to chat about how we use metrics, monitoring and dashboards to not only bridge gaps between our application engineering and operations orgs, but to create measurable applications for practical use by any customer within our shop – product managers and business VPs included. I’ll do you a bit of a favor and highlight the 57-minute video cast available at their operations open mic series:

  • Shopzilla’s ~150 engineer shop produces nearly 100 measurable services, all of which are candidates to have their entire life-cycle’s statistics broadcast to all employees in the company.
  • Software statistics measured range from unit test coverage, documentation coverage, complexity, performance test results, production SLAs, requests per second, exceptions (both distinct lists and aggregate counts) and uptime (measured internally and externally.)
  • These statistics come in at an all-told rate of about 85,000 per second. Measurement speed and scalability in presentation is key.
  • Initially, application engineers were reluctant to put their workings ‘on display’, however once the predictive value of these metrics was exhibited, the notion of broadcasting all this information became a fundamental part of considering a new project complete. The request flow changed from, ‘Please expose some metrics for us to measure. ‘ to ‘Please measure these metrics for us.’
  • Measure everything, and once this data gets too cumbersome for one display, allow the user to customize their own view of normalized data from multiple systems. For example, Shopzilla creates operational ‘mashboards’ that allow a user to define which revenue graph, which logging output, which system health checks they want to see. Once these displays can be customized, you basically have a user driven CMS application creating personal views of whatever data they deem important. If they still don’t use it, then what data do they need? Grab that, and present it in your normalized view as well.
  • If you don’t measure from the beginning (requirements gathering, early in the dev proces,) then how will you know what you’re missing when you get to production? You wouldn’t install a new version of Oracle/Red Hat without reading the release notes, so don’t release internal software without doing the same.
  • Tools used in our Shopzilla model: Sonar (for java unit test and documentation coverage,) Hudson (for continuous builds, deployments and pushing instrumentation to Sonar,) Splunk (for log aggregation,) Keynote (for external site monitoring through synthetic transactions,) Clearstone (for Oracle Coherence grid stats,) Cassandra (someone has got to put those 85k numbers into a meaningful place,) and Graphite (make it pretty.) Mix in a myriad of perl and python code for presentation, and you can abstract any additional tools at your leisure.

If video is more your thing and those bullets are not quite enough, you can still view the entire, hour long video over at DevOps Cafe. Enjoy, and please feel free to comment and/or disagree here or with the DevOps folks.

Advertisements

May 23, 2010

Building teams out of individual people

Filed under: Leadership — juan paul @ 1:58 am
Tags: , , ,

I’ve been called a lot of things at work. No, not just things. Mostly, I’ve been called a lot of letters. I’ve been an I, an N, a T, a J, a low D, a high I (not to be confused with the previously mentioned, just-plain I), a mid-C with a splash of S…I’ve been midway between Gen-X and Gen-Y…I’ve been a chicken and a pig, I’ve been a do-er and a, well, don’t-er. I’ve been a leader and a follower (although I’ve never been good at getting out of the way), a promoter and a provider, yet I’ve never really felt like any of those things – or letters – helped me find my role within my organization generically enough to work the same way with my next team. I could never help but wonder if all of these generalizations help us think we have all the Is and Fs and chickens all figured out, but if I did not feel ‘figured out’, how could I assume that of my people?

Perhaps I am not any or all of those things/letters, and perhaps what I am will never be featured in an employer’s blog, but I am unique. In fact, the most common thing about me is that I am unique, and so is everyone on your team. <ObligatoryCredibilityEstablisher>For 15 years or so, I’ve been in leadership positions of information technology departments</ObligatoryCredibilityEstablisher> ranging in size and culture with very few things in common, outside of monitors and keyboards.  I’ve been fortunate enough to work with talented teams with a mesh of personalities that luckily, I am able to understand and help craft into a system that gets things done. I rarely take an opportunity to categorize common themes between employees, nor do I keep some distance from folks outside my teams in an effort to extend local focus. Those tactics seem like shortcuts to me, and not nearly a suitable replacement for actually learning about the people I work with specifically enough to find which piece they might fit in which puzzle. Each time, each person, individually, sometimes even through conversation – imagine that!

Of course, I fear discussing this tactic with other leaders around me. It doesn’t scale. It takes too much time. It violates a chapter in the en vogue leadership book everyone is reading. How can I ever lead groups of more than a handful of people if I have to lead them all individually? Yet even the sharpest minds around me do ask how I can take teams of varying size and create harmony, inspire pride and quality, groom future leaders and still meet deadlines.

Get this: you can talk to your people. Learn what motivates them, even if it is outside of their predetermined profile. (I get a different Myers-Briggs profile each time I am analyzed.) Use these motivations to find the right pairings and assignments within your team. Here is the real key: keep it small. Keep yourself responsible for a handful of people at most (7 +/- 2) by leading through leaders in your own org. Teach those leaders how to learn about their employees through the same conversation patterns and make sure that when you meet with them, eye to I, you keep listening. And when you find yourself just waiting for your turn to talk because you’ve already figured out their personality and that of their team, start over. Make sure you reconnect and keep the connections flowing throughout your team. First and foremost as a leader, you have to make sure the seeds are growing more than you have to actually produce the fruit. Make sure your leaders keep it small and grow the next generation of leaders, under what motivations might make them grow. Voila, you can now scale to a team hundreds. Or a time of five. Whatever motivates you…or I.

This might violate hours of training you’ve received recently. Training meant to help you understand the difference between generations, genders, introverts and extroverts. This might simplistically contradict volumes of books handed out throughout your organization; books meant to show you how to hire, retain and satisfy every person in the employment life-cycle. But it will help you create a synergy with your business and people because they will feel connected to rest of the farm. Actually hear them, and while you can still apply the lessons in the books and the personality profiles, try not to feel forced to overwhelm them with generalizations. Your best and brightest might appreciate being listened to (or maybe not, but you’re going to learn that now, right?)

If you feel like you haven’t been listened to, drop me a line. Leave a comment on this post. Let’s chat, face to face, over lunch. I bet we’d have a lot to talk about, U and I.

PS.  This originally felt better as the ‘About Juan Paul’ section of this blog than the original post, but then I’d have to name it, ‘About I’ which would not get me much S, E or any O. I wouldn’t anticipate that future entries would be so much about me, or I, but then, that was half the point here.

Blog at WordPress.com.