Visualising Production Data For The Team - Retrospective 1

Posted By Duncs ~ 15th January 2013

Today I pitched my idea on visualising production data so that the team can see what kind of experience our customers are having. This is a brief update on my progress…

I invited my fellow Tester, my Test Manager, my Dev Lead & the Dev Manager - All of them turned up, what a great start! I kept the demo to 30 minutes as I wanted to keep it brief & to the point. My starting point was that us Testers are spending far too long manually testing in production.

What made me mad

Nothing

What made me sad

I’m not convinced I fully got my point across about the benefits.
We had a conversation about graphs being good for retrospective trending, not for live monitoring, but I mentioned the idea was retrospective trending over the half / hour period of the release but that didn’t wash.
My 2nd / 3rd order ignorance of our metrics & how they are measured really came to light.
My ignorance of New Relic & Splunk showed in the demo - even though I talked about what I had planned with condensing the dashboard into fewer graphs & that this was a Proof Of Concept the demo could have been more polished.
At times I felt a bit like they were thinking “Why is a Tester doing this?” Perhaps it was just my paranoia. As its my role being impacted, I definitely feel it is up to me to try & sort, hopefully with the support of my team.

What made me glad

I haven’t been told to stop!
Everyone in the room appeared to like the idea & agreed that there definitely was some scope for this kind of dashboard.
They questioned why I chose the metrics I did & I received some great feedback about the sort of metrics we should be looking at & how both New Relic & Splunk can be used in conjunction to serve this purpose
I got some ideas on visualisation other than graphs. This was interesting. I was initially reserved about the idea, but I think a graph might be well complimented by a gauge or a dial (Splunk)
I got some pointers on the path I should be taking now
I got great support from my Test Manager for the angle of reducing testing time in Production as a primary driver / motivation for the dashboard.
Plans are afoot to get Apache data into Splunk (as well as other key logs)
Talk of looking at other monitoring tools (guess what, I’m pushing Graphite)

What else?

Aside from the demo, I enquired about getting another machine to do my experimenting on. This went down well in that I haven’t been denied, but due to a limitation in the number of ports available they’d rather I use a virtual machine. If it does the same job, I’m not fussy. Previous Actions - actions in green have been completed

Find out why we’re not using New Relic RUM
Find out if New Relic RUM might actually provide value for us
Learn how to include several metrics in one graph to save space (e.g. logins, deposits, bets placed)
Demo my idea / current dashboard to the relevant people
Take feedback / ideas on how to improve the dashboard
See if we’re ever going to get Apache data into Splunk (I know the Sys Admins want some other data logging as well)
If successful, get some monitors up on the walls
look into setting up the dashboard / monitor on a web server & embed the charts in an HTML page to see if that saves space
Having the dashboard / monitor on a web server will help with different people / locations being able to see the dashboard via a URL
Investigate Reload & Revolver to see if they are having memory problems or find an alternative

Whats Next

New Relic RUM didn’t come in the conversation - I’d like to look into that further
Investigate some other metrics & how they relate to different levels of the technology stack (without losing focus on customer experience)
Investigate some other ideas for visualisation of data
Acquire physical / virtual machine to test my dashboard on
Getting dashboard on a big screen on a wall is still an action, but now its a bit further off…

Finally, work is ramping up a bit so I need to focus on this side project somewhat less. Unfortunately I will be without family at the end of January as the wife & girls return to the UK to catch up with family & friends. I am going to use that time to create a much slicker dashboard which really sells my idea. Catch up with you on this in February, hopefully with something quite productive…