I regularly walk through the Turnpike Lane Bus Station, there\u2019s a pretty big sign pointing to it. It\u2019s a major node for North London public transport and yet, a few years back, I found out that it did not exist\u2026 in the data, at least. I used to run the official data set of bus stops for the UK Government\u2014a rather obscure dataset that made its way into powering a few popular journey planners like Google Maps and City Mapper. <\/p>\n\n\n\n
This was 2020 during COVID, and one of my colleagues wanted a list of all bus stations in the country in order to send posters which advertised social distancing. While the dataset contained over 500,000 points, it did not contain this bus station. The problem data definitions: the dataset listed bus stops, which were not the same things as bus stations. While the words \u201cbus station\u201d have a common sense meaning in our minds as a collection of bus stops, that meaning was not translated into the dataset. The individual bus stops making the bus station are all in the datasets, except there was no way to group them together other than trying to infer<\/em> they\u2019re part of the same bus station because of their proximity. <\/p>\n\n\n\n I found other interesting issues in the dataset. Some were easy to spot, like bus stations in the middle of the North Sea. Other stations were a few meters away from their real location, which would not have a huge impact unless we were trying to use the dataset to get self-driving buses to park automatically. So, why weren\u2019t these groupings captured in the first place? The process that created and populated the data never asked itself \u201care we capturing everything that we need about this bus stop?\u201d. As a result, the dataset wasn\u2019t quite fit for the purpose we were looking to deliver. The translated definitions of common sense concepts into data is a major element of making sure that a dataset is usable and stays current, and having a process that allows that question to emerge is an ingredient of good data management. At the time, to my surprise, we didn\u2019t have either. <\/p>\n\n\n\n Disappointingly, data that may appear suitable to your purpose is not always; and if you are in the fortunate position of being the owner of a dataset, always ask: are there any use cases that would be out of scope for this dataset, and is it worth expanding?<\/p>\n\n\n W.E.B. Du Bois is widely remembered for his infographics about the conditions of African Americans at the end of the nineteenth century. What I always hail him for was having shown that a lack of data should not stop a good data project and that sometimes the hard work is putting data together. When he realised the US Census lacked data about African Americans, he assembled his own survey and team,collecting data that resulted in his now famous infographics. Incomplete or missing data is something that I\u2019ve regularly had to cope with and decide whether to pursue the initial project or pivot to something different. Once again, during the pandemic, we were trying to see if there was a way to check the density of people on pavements, and entered a tunnel trying to find the accurate measurements of pavements for the whole of the UK\u2014an impossible task. This is when I realised using a proxy<\/em> would have given informative enough results, as did The Economist in the chart below, created by collecting, over time, Google Places \u201cbusy times\u201d for major points of interest in major cities. Simple, effective, but not based on anywhere close to \u201ccomplete\u201d data.<\/p>\n\n\n Sometimes missing data should make us reflect. In one of my projects while working for public healthcare in the UK, a team of dermatologists came asking if my team could develop an AI algorithm to grade a type of skin condition. The intent was very positive: in their clinical research, they realised human medics were biased, resulting in less accurate grading for people who are not white, and were looking for AI to help correct that bias. We found that the collection of images we could find about this condition were themselves biased, so any AI model trained on them would have not addressed the issue. The image below captures what dermatologists call the Fitzpatrick scale\u2014the official measure of skin darkness. <\/p>\n\n\n We realised we had as many images as we wanted about Fitzpatrick scale I, and increasingly less as we went towards scale VI. Developing a model would have been unethical, and we and the team of dermatologists agreed to pivot and start collecting an unbiased dataset. This experience taught me to reframe success: we successfully detected bias and used the tools at our disposal to try and correct it. Unbiased data means more ethical and accurate models. During this project I learned a lot also about the concept of intersectionality<\/em>, thanks to data journalist and humanizer Donata Columbro, who suggested in her book \u201cDentro l\u2019algoritmo\u201d (2022) that in order to understand and correct bias in data, you can apply intersectional thinking as the framework that analyses the embedded injustice in the administration of power.<\/p>\n\n\n\n …or so they said. One possible critique of du Bois\u2019 work is that it wasn\u2019t based\u2014for good reasons, as we\u2019ve seen\u2014on the official data. We also use the words authoritative<\/em> for data that comes from official administrations, suggesting it has a superior authority; yet that\u2019s not always the case. A personal example of this is the chart below, from the Financial Times, representing how UK parliamentarians increasingly mention the words \u201cNHS\u201d (National Health Services) in their speeches. It\u2019s personally interesting for me because if you look at the source, it references \u201cParli-N-Grams,\u201d a website I built that allows that analysis. I always make the joke that I called my mum to say \u201cmum, I\u2019ve become a source.\u201d What\u2019s going on here and what\u2019s the lesson?<\/p>\n\n\n Well, while I\u2019m obviously flattered that the Financial Times used my website to make this analysis, it is remarkable that I\u2019m not the official source of the data. In fact, I\u2019m not even using the original source myself, as I used the easily usable extracts released by TheyWorkForYou<\/a>, a project of civic tech charity mySociety. This is because the original data is really hard to use, so data journalists are discouraged from doing so. This is clearly a minor problem in an article like this, but it makes you think about the potential for news manipulation. In my job in the public sector, I always think about how I can make the data as usable as possible.<\/p>\n\n\n\n There are cases where it\u2019s just simply wrong or misleading beyond authoritative data not being ideal. The most famous case is that of borders and their representation in data and maps. There are plenty of border disputes around the world, including in unexpected places. For example, France and Italy are broadly at peace but take different views on where the border around Mont Blanc falls, to the point that the official map of Switzerland is more accurate than those of the two countries. And Switzerland itself has a three-way border dispute with Germany and Austria which has reached several courts. Representing all these fuzzy borders in data and maps can be difficult and it can lead to litigation. Mikel Maron, once head of the Open Street Map Data Working Group, tried to address the issue of maps around Jerusalem, where there are obviously two famously different views about borders. His message \u201cJerusalem is an edge case of everything\u201d sadly resonates, but I\u2019d take the slightly less optimistic view that it is not just Jerusalem to have edge cases\u2014our reality is full of edge cases, and representing it in data is difficult and imperfect.<\/p>\n\n\n\n We use a lot of data to back our statements with evidence. But is data easy to understand for people who are not trained? And as data journalists and advocates, how can we make it easy to understand? Often this is, again, a problem of clarifying definitions. I often challenge my audiences to raise their hand if they know how to calculate the average; most people will raise their hand. I ask some people how, and they will tell me the well-known formula of the arithmetic average\u2014sum all items and divide by the number of items. And here\u2019s the catch: in many official statistics, we don\u2019t actually use the arithmetic mean. <\/p>\n\n\n\n In the UK, the Office for National Statistics uses the geometric mean to publish the average house prices. Most people don\u2019t remember the formula nor its meaning; some people might even be unaware of its existence. Other times, the definitions we use are disconnected from people\u2019s reality and, while they make a lot of sense from a statistical point of view, they might not chime with people\u2019s common sense understanding of a problem. For example, the official definition of \u201cemployed\u201d in the UK is \u201ca person aged 16 years or over who did one hour or more of paid work per week in the previous week\u201d. Having worked an hour per week might not meet the expected definition of \u201cemployed\u201d of someone without statistical training; this can trigger confusion. The lesson I\u2019ve learned through engaging with using these examples is that communicating data requires us to clarify definitions, explain why they were chosen & why they were better than the alternatives, and clarify the meaning of the data we are publishing.<\/p>\n\n\n\n This also means, sometimes, recognising that the data is not neutral or impartial. See the two maps below. Would you imagine that they are based on the same data?<\/p>\n\n\n\n They are both based on the data from the Italian General Election in 2018, which was a particularly contentious election with three coalitions, and all outcomes were possible according to the polls, due to the intricacies of the electoral system. <\/p>\n\n\n\n The map on the left is one I made. It\u2019s a dot-map, so it shows a dot for every several thousand votes\u2014blue for the right-wing coalition, red for the left-wing coalition, yellow for the non-aligned \u201c5 star\u201d movement. The story I told with this map was that Italy was an electorally confused country; it does indeed look pretty confused on the map.<\/p>\n\n\n\n On the right you see the map that the Financial Times released, a much more beautiful map than mine! You see the stark difference\u2014the FT team coloured the map based on the winner in each constituency, getting a country that is pretty much split into three parts: centre-right in the north, 5 star in the South, and the centre-left only featuring in its strongholds around Romagna, Tuscany, and less expectedly in Sudtyrol. This map illustrated a story about Italy being a divided country. <\/p>\n\n\n\n So which is it, was Italy divided or confused? The lesson here is that data-driven doesn\u2019t mean neutral or impartial. There\u2019s an agenda in every data use, sometimes unwillingly. In this case, the \u201cagenda\u201d is the fact that the data is initially collected for one purpose which gives it an initial \u201cspin\u201d\u2014administering an election\u2014and then we make a choice to tell a story that we detect in the data, which gives it another spin.<\/p>\n\n\n\n Creating live systems, where data or models concur in delivering a service, has been one of my areas of work. Here, the basic problems are the same: using the right definitions, making sure we remove bias, etc, but there are some lessons specific to the live nature of the data. For example, you might have heard of the concept of model decay\u2014if you create a predictive model based on data and the specific problem is one where data is generated frequently, then the model will slowly lose its predictive power. Correcting this requires strong collaboration between experts of the problem, data scientists, and data engineers. I\u2019ve seen this in an interesting project where we used neural networks in order to predict how long a patient would likely stay in hospital at the moment of presentation. While the model was pretty accurate at the point of creation, having been trained on over one million patient admissions and capturing over 300 data points per admission, it would have quickly lost its predictive power without allowing for re-training in the live pipeline. <\/p>\n\n\n\n This project was also remarkable for another lesson that became apparent: asking \u201cso what\u201d about every data-driven project. The problem here was that, despite the model being very accurate, we had no idea how to use it in practice. What action would doctors and nurses take that was different thanks to the prediction? We couldn\u2019t work it out. And there lies the lesson: any data-driven model needs to be built together with those who will use it. Capturing user needs with adequate user research, bringing subject matter experts together with data scientists, is the only recipe for success. <\/p>\n\n\n\n This brings me back to my initial point: data is multidisciplinary. The last example would have been much more successful if we had, from the outset, worked not just with data experts but also with those running the hospital operations\u2014nurses and doctors\u2014who would have guided our development of the model more effectively. <\/p>\n\n\n\n I hope that sharing these lessons will help. Working in data, whether it is data visualization, data analysis, or creating data-driven systems, is fascinating, but it really requires a lot of knowledge that comes from different areas: technology, maths and statistics, ethics, law. The best data practitioners use all these to challenge their assumptions and become better at their data uses.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/dvsnightingale.wpenginepowered.com\/wp-content\/uploads\/2025\/07\/bus-station-1024x768.jpg?resize=720%2C540&ssl=1\")
Sometimes the data is really incomplete or missing<\/strong><\/h2>\n\n\n\n
![\"\"](\"https:\/\/i0.wp.com\/dvsnightingale.wpenginepowered.com\/wp-content\/uploads\/2025\/07\/economist-social-distancing.jpeg?resize=712%2C400&ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/dvsnightingale.wpenginepowered.com\/wp-content\/uploads\/2025\/07\/wikipedia-fitzpatrick-1024x393.png?resize=720%2C276&ssl=1\")
Just use the official data and you\u2019ll be fine\u2026<\/strong><\/h2>\n\n\n\n
![\"\"](\"https:\/\/i0.wp.com\/dvsnightingale.wpenginepowered.com\/wp-content\/uploads\/2025\/07\/FT-nhs-parli-1024x851.png?resize=720%2C598&ssl=1\")
Communicating data should clarify reality<\/strong><\/h2>\n\n\n\n
![\"\"](\"https:\/\/i0.wp.com\/dvsnightingale.wpenginepowered.com\/wp-content\/uploads\/2025\/07\/dot-italy.webp?resize=670%2C716&ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/dvsnightingale.wpenginepowered.com\/wp-content\/uploads\/2025\/07\/FT-italy-1-1024x935.jpeg?resize=720%2C657&ssl=1\")
Enter production: data-driven services<\/strong><\/h2>\n\n\n\n