How São Paulo's Digital Archives Became a Graveyard of Duplicate Images — and Why Cleaning Them Up Now Matters

São Paulo's public institutions are drowning in copies of themselves. Across municipal servers managed by the Secretaria Municipal de Gestão (SMG), archivists and IT managers have been quietly confronting a problem years in the making: duplicate digital images, sometimes dozens of identical or near-identical files stored under different names, have clogged government databases, slowed public-access portals, and inflated storage costs that city auditors flagged as wasteful in budget reviews dating back to 2023.

The issue did not appear overnight. It is the direct product of how digitisation was handled — or mishandled — across roughly a decade of overlapping, underfunded, and poorly coordinated projects that began in earnest around 2014 when the Prefeitura launched its first major push to scan historical documents held at the Arquivo Histórico Municipal Washington Luís, a nineteenth-century building tucked behind the Viaduto do Chá in the city centre. Each successive administration added new scanning campaigns without retiring old workflows, and without enforcing common file-naming or deduplication standards.

A System Built to Replicate Mistakes

The Arquivo Histórico Municipal is just one node in a much larger problem. The Museu da Cidade de São Paulo, based in the Parque do Estado in Ipiranga, ran separate digitisation drives. So did the Instituto de Pesquisas Tecnológicas, based on the Cidade Universitária campus in Butantã. Each institution used different scanning equipment, different file formats — TIFF, JPEG, PNG — and different metadata schemas. When files were later aggregated into shared drives or uploaded to the city's open-data platform, dados.prefeitura.sp.gov.br, the duplicates came with them, quietly multiplying.

Procurement cycles made things worse. Contracts for digitisation services were typically awarded in annual tranches, meaning a new vendor arriving in January often had no visibility into what the previous contractor had already scanned the prior December. City IT staff interviewed for this article — speaking in a general capacity about institutional practice rather than about specific active investigations — described a culture where completing a scan quota mattered more than checking whether that image already existed somewhere on the network.

The practical consequences are not abstract. A 2024 report by the Tribunal de Contas do Município de São Paulo found that redundant data storage across municipal secretariats was contributing to infrastructure inefficiencies, though the tribunal stopped short of quantifying the full cost attributable to image duplication alone. Independent estimates from the São Paulo tech sector, where data-management firms like Neoway and Stefanini have studied analogous problems in financial institutions, suggest that unmanaged duplicate files can account for between 20 and 40 percent of total storage consumption in organisations that lack automated deduplication pipelines — a range that, applied to the city's known server footprint, implies significant unnecessary expenditure annually.

The Push Toward a Fix

Pressure to act has been building from two directions simultaneously. Federally, the Lula administration's programme for digital government modernisation, the Estratégia de Governo Digital 2024–2027, has set interoperability and data quality as explicit conditions for municipalities seeking federal technology grants. São Paulo, which applied for funding under that framework, cannot afford to present auditors with archives in disarray. Locally, Mayor Ricardo Nunes has championed a smart-city agenda centred on Avenida Paulista's tech corridor, and the contrast between that forward-looking branding and antiquated back-end infrastructure has become uncomfortable for city hall to explain.

Practical steps are now underway. The SMG issued internal guidance in the first quarter of 2026 directing municipal departments to audit image repositories using hash-comparison tools — software that generates a unique digital fingerprint for each file and flags exact duplicates for human review before deletion. The Arquivo Histórico Municipal Washington Luís is understood to be piloting this approach across its photograph collection, which runs to several hundred thousand scanned items.

For institutions, researchers, and journalists who rely on São Paulo's public digital archives, the immediate advice is straightforward: download and locally preserve any image files you are currently using from city portals, since the deduplication review process may temporarily pull files offline for verification. Longer term, the city's move toward a unified metadata standard — modelled in part on Dublin Core protocols already used by the Biblioteca de São Paulo on Avenida Cruzeiro do Sul — should mean that what remains after the cleanup is leaner, better labelled, and actually findable.