QCon London 2012

March 27, 2012

I’m late with my write-up of QCon, and what’s worse, it will be partial - “sadly” I was in Lanzarote on a training week with the running club from the Thursday (8th) so I missed most of it. A sacrifice I had to make for 7 days in the sunshine…

Firstly, me me me

I presented the talk I previewed at Skillsmatter the previous week, something I was calling the User’s Guide to the Disruptor, but actually turned out to be how-can-Trish-fill-95-slides-with-pictures-and-finish-in-under-40-minutes.

The audience was different to the Skillsmatter event, not surprisingly. What was surprising is that I expected people at the conference to be less aware of the Disruptor, and those who came to the Disruptor-only LJC event to have had more exposure to it. It was a (pleasant) surprise to see how many of the standing-room-only audience had not only heard of the Disruptor but had read stuff about it (I always love it when people have read my blog), played with it and were even using it in anger.

Because of that, I think if anything the talk did not go into enough detail, or enough new stuff, to please everyone. Tough crowd! But it was gratifying to hear the audience correct me in some of my answers, and answer other people’s questions - it’s always nice to know people are listening.

Of course, I will post a link to the presentation when it’s available. For now only the slides are online.

I enjoyed QCon

QCon was noticeably different to the other conferences I’ve been to in the last six months. For one, it’s not a Java conference - sure, I was hosted on the Java track, but QCon is wider-ranging than just one technology platform. I’m not sure if it’s because of this, or because it was based in the notoriously impatient London, but I felt like there was a message of “Look, let’s just get stuff done, OK?”. Ultimately we get paid to deliver stuff for the business, and since my favourite question is always “but what are we trying to achieve?” I like to hear ideas around how to actually deliver. Don’t get me wrong, I’ve loved the technology conferences - I like to hear about new stuff I had no idea about, and I really like the community vibe from them. But it’s a nice change to be shaken up into thinking exactly why we do all this.

The Data Panorama

Firstly, Rebecca Parsons and Martin Fowler from my old employer ThoughtWorks put Big Data into perspective. Previously I hadn’t really cared about it - we process lots of data at LMAX but we don’t really have to dig into it, so Big Data is not top of my “oooh I really worry about that” problems. There were quite a few interesting points I took out of this:

In the past, it was easy (and possibly even correct) to model the whole application based on the data you were collecting or manipulating (and probably storing in a relational database). These days it’s not just the data from your app you need to worry about (and that can get big enough), but also all the news, blogs, twitter, and Facebook stuff generated by you and about you. In addition, your data might not even be located with your app - the cloud has made the physical question of location redundant. All of this pushes you towards an architecture which has to separate data from the application, and forces you to ponder your design. I heard good arguments for Domain Driven Design here, which is nice because we like that at LMAX.
Reporting and analytics on Big Data must be more fluid. There’s so much data about you, your users, your application, out there that you don’t even know the questions to ask. Instead you want to be able to spot patterns in data you didn’t even know you had, I thought this was dead interesting - I studied Computer Science and Artificial Intelligence at university, and we were told data mining using AI was going to be fundamental to companies who wanted to be on the bleeding edge. Only now is it looking like people realise it’s becoming that important.
Martin referred to Data Scientists and said he was suspicious of scientists. I’d been reading The Black Swan on the train in that morning and couldn’t help but wonder if he’s read the same thing - that was talking about how you should treat someone with suspicion if they suggest you can apply science and logic to anything that is… well, actually to anything other than actual science (by which he meant physics I believe). By saying it’s a science you suggest it’s predictable and follows rules. And if it was predictable, it wouldn’t be hard. Big Data is anything other than predictable - the data could be corrupt (it’s safest to assume some of it is); it’s generated by people (and we all know how fickle they are); and if you’re collecting lots of it practically randomly, then cause/effect/correlation are not guaranteed. So Martin suggests the term Data Journalist. There was a storm on Twitter which suggested a certain amount of disagreement with this term, but I like it. But then, I like to pretend I’m a writer and not a programmer.
They gave some examples of using data to drive economic growth (e.g. Kenya) - what I thought was interesting about this is that it was a win-win situation - expose the data to grow technology skills, but get a lot of interesting/useful/socially responsible applications in return.
Something I liked was about the idea of embracing “bad” data, stuff that’s not trustworthy for whatever reason. You can assume that the good data overwhelms the bad but you can’t be sure. By looking for more fluffy patterns, vague correlations, in your data, you might expose something interesting even if the data’s not “correct”. As humans, we can’t expect that we won’t make a mistake - it’s better assume we will and work out how to deal with it.
There was a call to not passively accept requirements, but to play an active role. I like to think this ties in to my post about working with your customer. But then I would.
I got a lot from this keynote, even if it was just a feeling of “I knew it! I was right!”.

Highly available systems in Erlang

Joe Armstrong was a very interesting person to listen to, clearly someone who’s been there and done that. Even without the presentation, the slides are interesting in their own right as they contain a lot of information and guidelines.

The points I took away are:

If part of the system fails, it’s not up to that part to fix itself. You need special help to deal with failures. If you fell over with a heart attack, you wouldn’t try and heal yourself, you’d get a medic
Isolation between threads/programs will mean that those different things cannot interfere with each other (i.e. no shared state).
My favourite quote was “If you make things synchronous you’ll bugger things up”. In theory, it’s so much easier for us humans (programmers) to think synchronously. But whenever you design your system to by asynchronous, you find that your system actually becomes simpler and not more complicated.

JVM performance optimizations at Twitter’s scale

I had a terrible view in the fully packed room, so I was just picking up phrases. I heardAttilamention the Uncanny Valley, a phenomenon I was introduced to by my sister when she was studying her Cybernetics PhD (Google it, I found it fascinating).
There was a lot of really useful information about how the Java GC works. It seemed to back up the (rough) premise we work on - stuff that’s very short-lived is fine, and stuff that lasts “forever” is fine - it’s the stuff in between which cripples your system when it keeps getting shoved around.

Decisions Decisions

Dan North was, as always, an excellent speaker. He entertained us but got us all thinking. Five years ago at QCon London (my very first conference ever, and the thing that motivated me to start a professional blog), I saw Dan speaking and I was inspired to think about my working practices. He was talking about BDD at the time, which was a relatively new concept to me. I came away from that QCon with clearer ideas of what awesome development practices should look like. Never did I imagine that five years later, not only would I be working in an environment which follows a lot of those practices (and pioneers many more) but that I would actually be speaking at the same conference.

I’ve come a long way since then, and of course, Dan isn’t talking about the same things either. I’ve been to a lot of conferences this year, and I heard nothing really preaching to the “meta-agile” LMAX (still have a post pending about our agile practices…). When it comes to agile, there seem to be very few people who we can learn off - don’t get me wrong, there are lots of things we want to improve on, which is why we’re looking for people to learn off. But most people are still preaching TDD and we want to know “what next?”.

Well, Dan took everything we thought we knew, and ripped it to pieces. Taught us to challenge everything we think we know.

It was irritating actually because he had no answers. But he did tell us that the answers we think we already have might not be the right ones…

Well worth watching his talk when it comes online, I really can’t summarise it here.

Developers Have a Mental Disorder

I nearly missed this ending keynote. I’m so glad I didn’t, Greg Young is awesome at ranting. He said developers have a disease - we overcomplicate things, when we try to simplify things. We want to abstract stuff, we love to look for patterns and reuse when actually sometimes we just need to solve the problem.

In my notes I have “People come to conferences for answers, when they should be remembering to use their brains”. I assume that’s a quote from him, and not a comment I thought of at the time, but I wholeheartedly agree with it - if a shiny new technology solved your problems, you’d be out of a job. Your job is to take the problem and figure out how to solve it. Not to drag and drop an answer into place.

Another talk that I can’t do justice to, watch the video when it comes online.

…and finally…

At the end of the day, the Atlassian-sponsored community night was awesome. I got to chat to (be prepared for gratuitous name dropping here) Martin Fowler; my old friend Simon Brown; my ubiquitous LJC colleagues Martijn and John; the Zero Turnaround guys; a couple of ex-colleagues from Evolution / Detica; a heap of LMAX and ex-LMAX guys; and, of course, bundles and bundles of new interesting people.

Summary

(i.e the short version for those who can’t be bothered to read this whole post)

Maybe it’s wishful thinking, but the messages I took from QCon were:

If you want to solve the problem your business has, you might want to model your system around their world. Funny, that sounds suspiciously like Domain Driven Design.
Synchronous is bad, mmmkay?
Hardcore understanding of what the computer is really doing seems to be coming back into fashion. Hmmm, I wonder who started that…? (tongue firmly in cheek, we can’t have been the only ones)
“You can’t tune something you don’t understand” - testing and monitoring is kinda important.
Our business is all about trade offs. There is no perfect solution, they pay us because it’s very very hard to work out something that’s Good Enough.
Ultimately it feels like back to basics: understand the problem; model the domain, and have sympathy for the hardware that’s running your solution.

My Corporate Bit

QCon overall turned out to be a bit of an LMAX fest in the end, with Mike & Martin and Andy Stewart (our Chief Lord Business Analyst), all giving presentations there as well as me. It’s nice to be on home turf, and it’s very cool to see that we have such a range of things to talk about that so many of us are invited to speak.

PS

I just found out there’s a QCon in New York. My invitation seems to have got lost in the post. Don’t suppose anyone wants me to speak there…?