ELI: Petawatt nebo Petabyte kdo z koho? Pavel Bastl, Jaroslav Charfreitag CESNET Community Fórum 2014 - Zpracování a ukládání velkých dat
Local DAQ Based on rack mount computer with PCIe interface Amount of PCIe x8 interfaces should be available Optical fiber based NIC for DAQ network At the moment 4x SFP+ 10Gb/s NIC card (40 Gb/s overall throughput) Some SFP+ cages may be connected locally during local control More than one NIC may be inserted when needed and PCIe slots are available Digitizers Other standard measurement cards are available on the market Big amount of cheap DDR RAM is available Front access to devices in PCIe slots must be provided ELI ELI BL 4442 PRE 00000000-A
Projekt ELI Extreme Light Infrastructure (ELI) 3 + 1 laser = ELI ELI rovná se tři laserová centra sdružená pod jednou hlavičkou První centrum (ELI Beamlines) bude umístěna v České republice a bude vytvářet novou generaci sekundárních zdrojů pro mezioborové aplikace (fyzika, medicína, biologii, materiálové inženýrství ) Druhé centrum (ELI Attosecond) se staví v Maďarsku a má se věnovat fyzice ultrakrátkých optických pulsů v řádu attosekund. Třetí centrum (ELI Nuclear Physics) orientované na fotonukleární fyziku, by mělo sídlit v Rumunsku. Čtvrté infrastruktury se v současné době jedná.
Projekt ELI Beamlines Centrum Dolní Břežany První centrum (ELI Beamlines)
Projekt ELI Plán umístění systémů 1. patro Podpůrné technologie Přízemí Laserové haly Podzemí Kompresorová hala Distribuce svazků ve vakuu Specializované experimentální haly (E1 E6)
Projekt ELI Technologie a zdroje dat Obory a oblasti Stavební Strojní Laserové Experimentální Řídící (SCADA, kontrolní systémy) Zpracování dat (DAQ systémy) největší zdroj Ostatní HR, finance, evidence, běžný IT provoz
Zpracování dat (DAQ systémy) DAQ = Data Acquisition data z experimentů, měření, kontrolních systémů řídicích systému Řádově stovky TB dat z jednoho měření!!! ELI = možnost až 6 experimentů současně ELI = 4 laserové dráhy různého typu ELI = zdroj dat v éře BigData
Agenda DAQ system Data acquisition introduction Data acquisition devices Data acquisition server structure Data acquisition online processing Global level data acquisition Local level data acquisition
Data acquisition introduction Instrumentation Data acquisition - DAQ
Data acquisition introduction Instrumentation Data acquisition - DAQ Frame rate up to ~khz level High volume of data High speed throughput is required Data tame-stamping
Data acquisition structure Data acquisition input devices Matrix detectors CCD CMOS Scalar detectors Digitizers Network based interfaces GigE Visision 10GigE Visison (formally GigE Vision ver 2.0) Local bus interfaces USB 3.0 Metallic cable Optical fiber PCIe CoaXPress CameraLink PXI/PXIe CameraLink TCA
Data acquisition structure Data acquisition input devices Matrix detectors CCD CMOS Scalar detectors Digitizers Network based interfaces GigE Visision 10GigE Visison (formally GigE Vision ver 2.0) Local bus interfaces USB 3.0 Metallic cable Optical fiber PCIe CoaXPress CameraLink PXI/PXIe CameraLink TCA
Global level DAQ structure hierarchical data storage Online data view NFS/CIFS mounts
DAQ server Hierarchical data storage High speed disks - Tier 1 Low speed disks - Tier 2 Tape library Tier 3
DAQ server Hierarchical data storage High speed disks - Tier 1 Low speed disks - Tier 2 Tape library Tier 3
DAQ server Hierarchical data storage High speed disks - Tier 1 Low speed disks - Tier 2 Tape library Tier 3 Multinode DAQ server Shared DRAM - Tier0
DAQ server Hierarchical data storage High speed disks - Tier 1 Low speed disks - Tier 2 Tape library Tier 3 Multinode DAQ server Shared DRAM - Tier0 Network interface At minimum 4xSFP+ 40Gb interface
DAQ server FPGA & Timestamping FPGA online processing FPGA card in PCIe DDR RAM buffer 10Gb SFP+ interfaces FPGA IP cores & SW OpenCL support DDR memory IP core 10Gb ethernet IP core PCIe IP core
DAQ server FPGA & Timestamping FPGA online processing FPGA card in PCIe DDR RAM buffer 10Gb SFP+ interfaces FPGA IP cores & SW OpenCL support DDR memory IP core 10Gb ethernet IP core PCIe IP core Time stamping support Timing system based card Available only for facility level devices PTP support Suitable for user computer or devices
Global level DAQ - interfaces LC connector MPO/MTP connector Full duplex two fibers Standard in data networks and in ELI Twelve fibers in one connector Used in service HUBs FC connector Standard for instrumentation
Local DAQ Based on rack mount computer with PCIe interface Amount of PCIe x8 interfaces should be available Optical fiber based NIC for DAQ network At the moment 4x SFP+ 10Gb/s NIC card (40 Gb/s overall throughput) Some SFP+ cages may be connected locally during local control More than one NIC may be inserted when needed and PCIe slots are available Digitizers Other standard measurement cards are available on the market Big amount of cheap DDR RAM is available Front access to devices in PCIe slots must be provided
Local DAQ - interfaces PCIe PXI/PXIe Standard interface Data acquisition cards available Optical network is supported USB 3.0 is supported Timing and triggering not supported by the bus NI proprietary interface open? Data acquisition and general measurement cards available General measurement cards CPU cards are very expensive and have not enough memory for high speed data storage Optical network not supported USB 3.0 interface not supported Backplane supports timing and triggering μtca Advanced telecommunication architecture open, very flexible, complicated Data acquisition and general cards available Optical network is supported USB 3.0 is supported Backplane supports timing and triggering Becomes to be popular in facilities (DESY, Elettra, SLAC etc.)
Local DAQ - interfaces Based on rack mount computer with PCIe interface Amount of PCIe x8 interfaces should be available Optical fiber based NIC for DAQ network At the moment 4x SFP+ 10Gb/s NIC card (40 Gb/s overall throughput) Some SFP+ cages may be connected locally during local control More than one NIC may be inserted when needed and PCIe slots are available Digitizers Other standard measurement cards are available on the market Big amount of cheap DDR RAM is available Front access to devices in PCIe slots must be provided These requirements lead to server based computer
ELI vlastní datové uložiště (vývoj) Komerční uložiště X Vlastní uložiště cena, rozšiřitelnost, morální udržitelnost, doba dodání - 4U/5U velikost pro montáž do rack skříně, duální napájení Možnosti klasické disky/ssd disky/kombinace dle požadavku na IOPs 45 disků 3,5, hrubá kapacita až 270 TB (450 TB na přelomu 2015/2016) Možnost FC, iscsi a InfiniBand (FDR/QDR) rozhraní dle potřeb
Institute of Physics ASCR, v. v. i. Na Slovance 2 182 21 Prague 8 Czech Republic eli-cz@fzu.cz www.eli-beams.eu Děkujeme za pozornost a pozvání sdružení Kontakt: Fyzikální ústav AV ČR v.v.i., Na Slovance 2, 182 21 Praha 8 OC Harfa, Českomoravská 2420/15, 190 00 Praha 9 (do září 2015) Pavel Bastle, Pavel.Bastl@eli-beams.eu Jaroslav Charfreitag, Jaroslav.Charfreitag@eli-beams.eu