Sine Waves, Sorting and real Sim Cities at London JS Conference

I just came back from the latest edition of London JS, hosted by the guys from Poke and Pearson. There was a big focus on d3 and data visualisation, and there was three very interesting talks by three very interesting people. 

d3.js with Google Analytics

 Edd Sowden (I bet he has trouble flying with that name these days) showed us how to build line charts with data from Google Analytics. His talk covered the basics of d3's data binding and selections. I was quite impressed by the way he went about it: he used a sine wave as example data to illustrate how to create, remove and update data on d3.js.

He then went on to tell us about google analytics, describing what an OLAP Cube is and how to access its data through Google Analytics JSONP API. He has created a handy authentication module that handles OAuth connections to the Analytics API, all done client side:

Edd's diagram of Google Analytics' JSONP API authentication

Edd's diagram of Google Analytics' JSONP API authentication

He demoed a project called Browser Matrix, which allows you to break down your site's traffic by browser type. His work reveals some interesting patterns with browser usage, and their users. Did you know that iOS Safari traffic to his site drops dramatically on weekdays, while it massively spikes from friday night onwards? More interestingly, IE traffic drops like bad stock on the weekends, but it ranks consistently high during the weeks.

Using GA and D3.js you can learn a lot about your users. This will enable you to tailor your content and experience to the people who actually use your site, rather than targeting specific browsers.

You can check out Edd's code on github.

11 things you didn't know about d3.js

Anna Powell-Smith gave a fantastic lightning talk about 11 things about d3.js that she discovered while reading its documentation. She covered a lot of very interesting things:

  1. You can listen to the 'progress' event of your XHR queries so you can inform the user when loading big amounts of data.
  2. D3.js is very efficient at parsing any character delimited file with the `d3.dsv` function.
  3. You can easily perform 'Group-By' operations on d3 by using the `nest` function on your data.
  4. D3.js makes ES6's newly proposed Object implementation available right now  with the d3.map() function.
  5. Timers on D3 are far superior to using setTimeout/setInterval, as they use requestAnimationFrame when available, degrading gracefully when it isn't available and helping you save CPU cycles by detecting browser focus, stopping when in your tab is in the background.
  6. She showed a great example of Matrix Decompsition by Mike Bostock
  7. D3.js enables you to do perceptual color calculations: rather than calculating RGB, you can make a color '.darker()' or '.brighter()'.
  8. In much the same way, D3.js gives you the ability to format times and dates, and perform advanced date calculations. To know how many mondays since a birthday, you can just do `d3.time.mondays(new Date("1988-01-01"), new Date())`.
  9. She also covered the plethora of map projections available on the geo package, improving on a diagram from the Guardian about People's Republic of Korea missile range capabilities, by using an azimuthal projection rather than a standard mercator one.
  10. She showed awesome live sorting of huge tables, with rows actually flying around to their sorted position. Really cool.
  11. You can use D3.js for UX tests and development.
  12. It's really easy to use D3.js in node by just installing it with NPM. 

Her slides aren't up yet, but in the meantime check out her github repos 

ViziCities - SimCity for the real world

Robin Hawkes gave the last talk, a lenghty look at his project, ViziCities, where him and Pete Smart have managed to visualise London's buildings, traffic, crime statistics and transport systems in near-real-time. His inspiration came from playing SimCity V and wanting to re-create the game's famously detailed city data visualisations with real data.

His idea started with an email to himself at late hours. In a few days, he had plotted every building in central London on a WebGL visualisation. He estimated the heights of buildings in the area, creating a 3D map of the city.  With that canvas in hand, he then set out to collate data from most public APIs (police.uk, Ordnance Survey, the census, TFL, OpenStreetMap).

The demo he showed us uses Three.js and sported a very polished look. He used shaders to create  Ambient Occlusion, DOF effects, and tilt-shift, to give it a miniature-like look. The team also created autonomous vehicles and set them out to drive around on what he claimed to be London's first complete road network. 

A video of Vehicle AI in ViziCities

One of the their greatest challenges was to produce a 3D map of London's underground, complete with station depths and live train information. Unfortunately, as is the case when collating data in an unified visualisation, some of the information lies behind hard to find sources, such as huge, impractical excel sheets or unparesable PDF files from Freedom of Information Act requests. 

I really liked the way in which they use heuristics to estimate real life data, such as using Open Street Map layers to gauge road heights and tunnels. They also faced a very difficult challenge when defining where the boundaries of a City lie. It turns out that administrative divisions can be impractical, so they took a plinth approach, where they show only a square of a city and get the user to navigate it.

I really enjoyed his encouragement to distrust convention and push the boundaries of your ideas to find novel approaches, rather than reusing techniques that might not give you the breakthrough you need when facing a difficult problem.

This edition of London JS was a very enjoyable one. I spoke to the organisers about (hopefully) being a speaker on their February edition, and they showed quite a lot of interest. I now just have to find something cool to speak about - maybe grunt.js?