To this end, I wanted to do a quick tour of the technology stack we used to create, to give you an idea of exactly how a data-driven app like it functions. The focus of the architecture underlying Perception is repetition, and the ability for all the components to support this repetition will become clear.
Factory and shopfront
At the highest level, Perception is split into two parts – the server and the host. The server is where the application components are built, and the host is simply the location where they are stored, ready for consumption from the outside world. A good metaphor here is the idea of the server being the factory where the goods are made, and the host being the shopfront where they are sold. Word of warning – I’m going to flog this factory metaphor to death.
A well organised warehouse
Databases give you a structured way to ask for data, so you can be sure you are getting the right thing in the same way every time you ask. It lets you store and access all your data consistently, over and over again. Compare a well organised warehouse to a hole in the ground where you throw stuff to keep it safe. PostgreSQL is a powerful leading open source database that has been around since the 1990s. Its advantages and disadvantages are hard to quantify in many respects, but an easy one is comparing the licensing cost. Common proprietary databases are hideously expensive to license (an enterprise licence for an Oracle database can set you back $40,000 plus, yikes), and PostgreSQL gives you all the functionality you need for relatively simple data storage (relatively simple, here, means in the realm of megabytes or even gigabytes of data rather terabytes). As an atlas, we also needed a way to store geographic information (i.e. complex polygon geometries) in this database. The beauty of using an Open Source database is that the community has already provided us with this functionality, in the form of the PostGIS plugin for PostgreSQL.
Machinery driving the factory
The logistics team
The final component in the server environment is Devops. Devops is the way in which you guarantee that your build process is repeatable and triggerable. In our factory metaphor, it’s probably best comparable to the team that handles logistics. Devops is used to set up what processes need to be done, when, and gives you an interface to manage the jobs – it bridges the gap between “development” (i.e. coding and app creation) and “operations” (i.e. the people actually using the app). The advantage of putting Devops in place is that you can rapidly, efficiently, and consistently re-release your app, with new data or new features. For example, when I have a new dataset to add to Perception, I work with the data locally until it looks the way that I want, then I upload it to the database. Once it is in the database, all I need to do is say “Hey, Devops. There’s some new data.” and in 15 minutes, without having to twiddle so much as one knob, there is a new version of Perception on the website ready for its adoring public. We use Jenkins here at SV, which is fantastically solid Devops software styled after a fictional butler who organises all your jobs for you (which I suppose makes me the bumbling aristocrat that he’s constantly picking up after)
Using Jenkins, we can define jobs, set job triggers, and then have them cascade into each other, so that you can set off a whole chain of deployment events just by sending an update of your changes. For example, we have a build-data job, which runs a NodeJS script to extract data from the database. After this job is finished, the build-app job automatically starts, and creates a functional version of the web application. Once this is done, the whole package is arranged and shipped to the host server, where it is ready for consumption. The whole update process is automated and easily repeatable.
For mapping in the browser, Perception makes use of the popular Leaflet mapping library. Leaflet plugs into AngularJS, converting the standalone mapping package into a data-driven machine. Likewise, the charts you see in Perception are produced with a third-party library plugged into AngularJS. D3 is a visualisation library created by the famous Mike Bostock (well, famous in certain circles). Used to create SVG charts, it comes pre-canned with a huge variety of graphs that use the data that is already driving your AngularJS app.
And finally, the whole app needs to look nice, doesn’t it? That’s where Bootstrap comes in. One of the greatest challenges for a web developer for what felt like the longest time was making sure that your website looked good on all devices. Often (and you’ll still see this occasionally) you would need to produce two completely different versions of your website – one for desktop and one for mobile. Good riddance to that. Nowadays, consistently styled layouts and responsive sizing can all be catered for from a single composition, thanks to libraries like Bootstrap. Bootstrap is the system of measurements and standardised elements that you use for arranging your shop so that it suits anybody walking into it (although the metaphor does break down slightly here since nobody designs a shop to shrink and grow depending who the customer is…)
And that is the tech stack for Perception!
Use the Buttons to Share this Blog: