Abstract
Contributed Talk - Splinter E-Science
Tuesday, 19 September 2017, 14:20 (HS2)
5 years of Daiquiri - Lessons learned and further developments
J. Klar1, G. Matijevic1, A. Galkin1, K. Riebe1, H. Enke1
1Leibniz Institute for Astrophysics Potsdam (AIP)
The publication of astronomical research data from observational surveys, as well as from computer simulations, remains a challenge to collaborations and data centers. These datasets, reaching hundreds of terabytes in volume, need to be selected and filtered before download, limiting the amount of transfered data to reasonable quantities. In recent years, queries written in SQL, or it's VO dialect ADQL, submitted through dedicated web services, became the preferred method for this process. This, however, results in a significant effort for the development and maintenance of these applications, in particular when hosting several different archives. Already in 2012 we addressed this problem with the creation of the Daiquiri framework. Daiquiri enables us to create different highly customizable web applications for data publication, while only maintaining one common code base. At AIP, we employ Daiquiri to host the CosmoSim database, the APPLAUSE archive, the data releases of the RAVE survey, and our mirror of the Gaia archive. Daiquiri is open-source software and available on GitHub. While Daiquiri is a great tool to keep our portfolio of data publications maintainable, several problems became evident in the past. Some of our choices regarding the used technology were not optimal and lead to a insufficient extensibility of the code. Also, more tools and libraries where recently developed and are now available to us. Therefore, we started a refactoring process towards a second version of Daiquiri written in Python. In Fall of 2017 we will publish this new django-daiquiri package. In my talk, I will give an overview on our experiences with Daiquiri and discuss the different issues, which lead to the current rewrite. In addition, I will show a first web site created with django-daiquiri and how it can be applied by other institutions.