Erratum: UK and supplementary units
Between the 26/06 and the 20/07, the UK was sometimes silently dropped from trade partners, and series with supplementary units had miscalculated prices.
Both these errors happened after I switched both from Eurostat's dissemination API to Eurostat's bulk download facility, and from pandas to polars.
In the midst of this migration, I made two mistakes.
- I did not realise immediately that the country code used for the United Kingdom varies between Eurostat's API endpoints. Sometimes, it's "UK", and sometimes, it's "GB". I had hardcoded the wrong value when switching to polars, and so dropped inadvertently the United Kingdom from our trade partners.
- The data from the bulk facility goes back all the way to 1988, and I did not notice that many supplementary units have changed designation over time. For example, milk was initially measured in hectolitres, before the database switched to a normalised litre unit. And my initial strategy was to drop the entire series when they featured uncomparable units. The logic behind that choice is that I did not want to calculate prices between incompatible quantities. For example, kWh and kg do not mix. But the side effect is that, in my initial algorithm, hL and L appeared as different, not comparable units, and so the entire data series was dropped. This led to misreported prices.
Now, those two issues (and a series of slightly more trivial ones) have been resolved, and I set up a webpage, available here, where users can go and check whether Eurostat's live API (pulled on demand by the server) and my own processing pipeline (built on the bulk download files) report the exact same quantities and values. So far, all the health checks I performed were conclusive. I do expect some discrepancies to happen whenever new data is published by Eurostat, as my server will take around three to four days to pick them up.
I had good reasons to operate both switches in spite of the temporary mishaps. The dissemination API is very practical to use, since it provides ready-to-be processed dataframes. There is no data homogeneity issue, and little to no metadata issue, as identifying partners or reporters is straightforward. However, it is slow, and this made the website a bit sluggish. Plus, it is not as complete as the bulk download files. Candidate countries' trade data is not available, and sometimes, supplementary units (price per kWh, or per piece, for example) are missing.
Conversely, the problem with the bulk download facility is that the complete Comext database is fairly large. Compressed, the files I need for this website take around 16GiB. Uncompressed, they occupy around 140GiB. Thus, unpacking the entire database and searching for relevant rows through it can easily become a challenge on a resource-limited server like the one running the dashboard currently. Thus, soon after I had replaced on-demand downloads from Eurostat's API with local on-disk reads of the full database, I started getting out-of-memory issues.
To be precise, most customs codes do not create an issue. Compressed, they occupy no more than 400KiB, and in RAM, no more than 4MiB. However, the top end of the tail is heavy, with more than 10% of the database taking 200MiB or more in RAM, and a few codes taking more than 500MiB in RAM. That's just the loading part, before any computation happens. Moreover, the website allows users to compare several products at once, and I had to put a hard limit on that. Most of the time, above 30 codes loaded concurrently, the server crashes. A mid or upper range workstation with 16GiB RAM or more would have no trouble with that load, but then, since the current website is not profitable, I need to control server costs. Users can still contact me directly if they want more RAM-intensive analyses than what the server can provide.
In order to manage this memory issue, I switched to polars. Initially, the entire backend was built on pandas. I am very familiar with pandas since I have been using it for more than ten years now. I was already using it during my PhD in order to store the data scraped from EUR-Lex and to plot graphs of networks of lawyers working on EU trade policy. I am much less familiar with polars, and I am still learning the ropes. But, for large datasets, polars is pretty much essential. Its lazy-scanning algorithm dropped both the RAM consumption and processing time by a factor ten.