CASPER CorrelatorsPocket Correlators, FX Correlators, and the PAPER ProjectGriffin Foster, UC BerkeleyJPL CASPER Workshop June 11-12, 20091Thursday, June 11, 2009FX Correlators for Large-N ArraysFor small N arrays an XF correlator has traditionally been preferred. FX Correlators are a good general solution to Large-N arrays. F is the FFT portion of the design, X is the conjugation portion of the design.2Thursday, June 11, 2009Pocket Correlator: Designing a Correlator on a single FPGAThe PoCo design was originally built for the 4 antenna PAPER deployment in western Australia. This design fits on a single iBOB with 2 iADC card.The PoCo design allows for up to 1024 channels over a bandwidth of 27 MHz up to 250 MHz with ~100 ms dump times.This design has been extended to become the basis for the Packetized Correlator(PaCo) which uses iBOBs, BEEs and 10 GbE interconnects3Thursday, June 11, 2009F-Engine on PoCoThe PoCo design uses the early astro_lib pink blocks, and updated version is expected soon.The generic F engine design is ADC->DDC(optional)->FIR->FFT . The F engine design for PoCo is the same that is used in the full scale CASPER Packetized Correlator4Thursday, June 11, 2009X-Engine and Accumulation on PoCoEqualization block is used to select 4 bits coming out of the FFTFor the 4 inputs there are 10 correlations, 4 auto, 6 cross. With this general design we can either use 4 single pol inputs or two dual pol inputs which will produce full Stokes data.5Thursday, June 11, 2009Orbcomm Satellite Tracking6Thursday, June 11, 2009Determining an Accurate Beam Model with RFI7Thursday, June 11, 2009PoCo-8 on ROACHPoCo-8 Plans:8 input QuADC boards, 2-4k Channels, 20-100MHz, ~1ms accumulations sent over 10 GbE8Thursday, June 11, 2009FX-CorrelatorBy using an FX design the correlator is broken up into modular pieces which use off the self interconnectsF Engines are self contained on iBOBs/ROACH boards, the X Engines are run off BEEs/ROACHEs. By using standardized packet formats and 10 GbE the boards can interface without expensive custom interfaces9Thursday, June 11, 2009F-Engine on iBOBThe meat of the F Engine design is to digitize data, select a part of the band with a DDC, then the streaming voltages are pasted through a FIR/FFT to generate spectra. The spectra are accumulated with a vector accumulator and sent to the X Engine over XAUI.10Thursday, June 11, 2009XAUI Communication between iBOB and BEEXAUI is a streaming data interface using CX-4 cables, we use this interface to connect between the F and X EnginesThe interface between the X Engines on separate chips also uses the CX-4 cables but the instead of XAUI communication we use the industry standard 10 GbE Interface11Thursday, June 11, 200910 GbE Interface/LoopbackThe ‘heart’ of the X Engine design is actually the 10 GbE Interface. Since the design is a packetized correlator the arrival of every packet at the right time is not guaranteed, we need some logic to line the data up 12Thursday, June 11, 2009X-Engine on BEEThe X Engine design is a very simple idea, each spectra we generate in the F Engine is conjugated by every other spectra, including itself(auto correlation).In practice this is a little harder since we are growing the data from N to N^2, we become limited by the amount memory we have access to so there are a number of tricks we use to do the same logic in different parts of the board13Thursday, June 11, 2009Modularity of the DesignModularity is central to the CASPER correlator design, we want to design only 1 correlator.Any specific correlator that is designed should be based on this design with its own specifications. A correlator is a simple idea, but hard to build. The first 90% is the same for every correlator design, we can build that and leave the final 10% to the specific project 14Thursday, June 11, 2009Software interface with BORPHThe BEE and ROACH boards run a small version of Linux called BORPH which interfaces with the logic of the FPGA design via snap blocks and software registers. BORPH masks these interfaces as files which allows us to treat them as a standard UNIX file. For the correlator designs we use a set of Python/C scripts for control and monitoring.Design ConceptUNIX processSystem Call Device DriverHardware DeviceDesign ConceptBORPH processBORPH SyscallDevice DriverFPGA gatesDesign ConceptFPGA gates Various CAD ToolsSW HW Hybrid15Thursday, June 11, 2009Current and Planned DeploymentsProjectDeploymentAntennasBandwidthChannelsAcc Time(s)SKA ItlaySept. 200832, full stokes32 MHz2k0.1PAPER, GBNov. 200816, full stokes100 MHz2k~15PAPER, WASept. 200932, full stokes100 MHz2k1-10KAT-720098, full stokes500 MHz4k0.1GMRT200932, full stokes400-500 MHz4-8k1ATA200944, full stokes500-600 MHz x 44k0.001CARMA201023, full stokes500 MHz x 161k0.5FASR201164, full stokes500 MHz4k0.02The general PAPER/CASPER FX-Correlator design can be easily reconfigured to fit a number of interferometric designs and science goals. Currently there are a number of correlator designs that have been deployed in the field.16Thursday, June 11, 2009Next Up:ROACH/QuadADC Correlators17Thursday, June 11, 2009PAPER as a test bed for CASPER CorrelatorsPrecision Array for Probing the Epoch of ReionizationPAPER Team:UC Berkeley: Don Backer, Aaron Parsons, Griffin Foster, Mel WrightNRAO: Richard Bradley, Erin Benoit, Chris Carilli,UPenn: James Aguirre, Daniel JacobsUVA: Nicole Gugliucci, Chaitali ParashareUCT: Jason Manley18Thursday, June 11, 2009Requirements for PAPER CorrelatorThe PAPER correlator requirements are relatively simple and sparse which makes it a great design for our CASPER Standard correlator design. The terms PaCo, CASPER correlator, and PAPER correlator are interchangeable. The PAPER array uses 4m dipoles which operate between 100-200 MHz to detect the redshifted 21cm line between z=7 to 11. For this project we need a correlator which can produce 2k channels over 100 MHz of bandwidth with ~1 second accumulations. We don’t use phase switching or delays though these may be added in the future at the number of antennas grow. The final goal is to have a 128 antenna correlator which will be deployed in western Australia.19Thursday, June 11, 2009PWA-420Thursday, June 11, 2009PGB-8The Precision Array for Probing the Epoch of Reionization: 8 Station Results (http://arxiv.org/abs/0904.2334)21Thursday, June 11, 2009PGB-1622Thursday, June 11, 2009Data RecordingThe python correlator code runs on the BEE and a Linux server, recording