How São Paulo's Digital Archives Became a Graveyard of Duplicate Images — and What's Being Done About It

São Paulo's municipal digital infrastructure is carrying a hidden weight. Across the servers that store everything from urban planning blueprints to cultural heritage photographs, duplicate images have accumulated over more than a decade of disconnected digitisation drives, creating storage bottlenecks, inflated costs, and records that archivists describe as functionally unreliable. The problem did not appear overnight — it is the product of at least five separate municipal digitisation programs launched between 2011 and 2023, each operating with its own file-naming conventions and no shared deduplication protocol.

The timing matters because Prefeitura de São Paulo is currently mid-implementation of its Plano de Transformação Digital 2024–2028, the city's most ambitious attempt yet to unify data governance across all secretariats. That plan, announced by Mayor Ricardo Nunes's administration, explicitly targets storage efficiency as a budget line — making the duplicate-image problem not just a technical nuisance but a fiscal and political liability. Every redundant file kept on active servers costs money that competing infrastructure budgets, including the city's chronically underfunded drainage works in the Tietê River basin, could theoretically absorb.

How the Backlog Built Up

The roots go back to 2012, when the Secretaria Municipal de Cultura launched an independent drive to scan the photographic collections held at the Centro Cultural São Paulo, on Rua Vergueiro in Liberdade. That project used TIFF files at 600 dpi, a standard that was already being abandoned by the Arquivo Histórico Municipal, located in the Campos Elíseos neighbourhood, which had switched to JPEG 2000 the previous year. Neither system spoke to the other. Files were uploaded to separate servers, and cross-referencing was done manually — or, frequently, not done at all.

By 2016, when the municipal government launched the Programa São Paulo Inteligente to push smart-city infrastructure, a third layer of image storage emerged tied to surveillance, urban mobility cameras, and geoprocessing maps from the GEOSAMPA platform. Geoprocessing images of the same Paulista Avenue block were being stored in at least three separate repositories by 2019, according to audit documentation reviewed as part of a 2023 Tribunal de Contas do Município de São Paulo inquiry into digital asset management. That inquiry found storage redundancy across municipal systems had grown to represent a measurable share of total data volume, though the precise figure remained contested between secretariats at the time of publication.

Private-sector pressure compounded the problem. As São Paulo's tech ecosystem grew — the city hosts more than 20 unicorn-valuation startups as of mid-2026, concentrated in districts like Vila Olímpia and Berrini — public-private data-sharing agreements multiplied. Each partnership introduced new image ingestion pipelines, often without deduplication clauses written into contracts. The result was that municipal servers absorbed commercial image sets that may have already existed in city archives in slightly different formats or resolutions.

The Path Toward a Fix

The current effort to address the backlog is being coordinated through the Secretaria Municipal de Inovação e Tecnologia, which in March 2026 contracted a phased audit of all image repositories across 12 secretariats. The process uses hash-matching algorithms to identify bit-for-bit duplicates first, then perceptual hashing tools to flag near-duplicate images that differ only in compression or minor cropping — a technically harder problem that archivists at the Arquivo Histórico say has historically caused legitimate records to be wrongly merged.

For residents and civic organisations that rely on open municipal data — including urban planning watchdogs active around the Operação Urbana Consorciada Água Espraiada corridor in the city's south zone — the practical consequence of deduplication is better search accuracy in public databases and faster load times on platforms like GEOSAMPA. Researchers at Universidade de São Paulo's Institute of Urban and Regional Studies have flagged the reliability issue in at least two working papers examining how redundant records distort historical baseline comparisons for flood-risk modelling.

The secretariat has set an internal deadline of December 2026 to complete the first audit phase covering heritage and cultural collections. The harder task — rationalising surveillance and geoprocessing images stored under different legal retention rules — is scheduled for 2027. Anyone submitting public-data requests to the city in the interim should expect some image records to carry provisional metadata flags indicating their deduplication status is unresolved.