A vo-centric data lake with automated embargo for FAI data archive

Abstract

The study aims to modernize the scientific and technical infrastructure of the Fesenkov Astrophysical Institute and the services of the Kazakhstan National Virtual Observatory to integrate heterogeneous astronomical data into the global research workflow. The methodology is based on designing a scalable file storage split into logical sections (partitioned Data Lakes), complying with International Virtual Observatory Alliance standards, implementing automated streaming FITS reduction pipelines, and applying the 'Scientific Showcase' access control model. The key result of the work is the development and deployment of an end-to-end automated pipeline for astronomical data ingestion, sorting, calibration, and cataloging. A software suite for metadata indexing was established, and a four-class data embargo policy was administratively institutionalized via a public license agreement. Pilot testing of the pipeline on the photometric archive of the 1-meter Zeiss-1000 telescope validated the architecture's stability, successfully cataloging over 800,000 files totaling more than 8 terabytes. The novelty of this work lies in the creation of a hybrid model that separates the physical storage layer from international publishing protocols while reliably protecting the priority rights of principal investigators. The practical significance of the results is that we reorganized and structured our local observational archives: fragmentation was eliminated, unified standardization was introduced, and open access to the data was provided. The created software and hardware infrastructure is easily scalable, allowing new datasets to be integrated without changing the core architecture.