Config files explained

The aim of these config files is to automate running the weak lensing measurements and post-processing. The module xpipe.paths automatically tries to read the configs upon import, but you can also specify them later.

In the pipeline mode xpipe keeps track of reading lens catalogs, splitting them up into parameter bins, measuring the lensing signal and estimating covariances and contaminations automatically. However in order to do this, first we need to specify the details of these tasks in the below config files:

  • params.yml defines the measurement parameters

  • inputs.yml defines the available data files, and short aliases for them

These files do not exist yet when you first clone the repository, however there is a default_params.yml and default_inputs.yml which you should use as a reference. These defaults are set up such that when you create params.yml and inputs.yml they will be automatically looked for and read.

Config files are defined as yaml files, and are read as dictionaries, each entry consisting of a key and a corresponding value (note that this includes nested dictonaries).

Note that in yaml one should use null instead of None

Load order

  1. default_params.yml

  2. looks for params.yml

  3. tries to read custom_params_file from params.yml

from this point the load is recursive, e.g. param files are loaded as long as there is a valid custom params file defined in the last loaded config. Each new config file only updates the settings, such that keys which are not present in the later files are left at their latest value.

The parameters defined here are loaded into the dictionary xpipe.paths.params

params.yml

Key reference

  • custom_params_file: params.yml

    If you want to use an other parameter file, then specify it here. It must be in the same directory

  • custom_data_path: False

    Absolute path to the data directory of the pipeline. If False: uses default project_path + /data

  • mode: full

    The pipeline supports two modes: full and dev. This is primarily used in setting up the input files for the measurement. e.g. you can define two binning schemes: one really complex for the full run, and a simple, quicker for dev

  • tag: default

    Prefix for all files (with NO trailing “_”). In addition this will be the name of the directory wher input and output files are written to.

  • shear_style: reduced

    Format of the source galaxy catalog. Available formats are reduced, lensfit and metacal

  • cat_to_use: default

    Alias for the lens catalog to be used (in this case the default). Aliases are defined in inputs.yml

  • shear_to_use: default

    Alias for the source catalog to be used (in this case the default). Aliases are defined in inputs.yml

  • param_bins_full

    Parameter bins defined for mode: full, e.g.:

    param_bins_full:
        q0_edges: [0.2, 0.35, 0.5, 0.65]
        q1_edges: [5., 10., 14., 20., 30., 45., 60., 999]
    

    q0 and q1 refer to the zero-th and first quantities (in this order) you want to split your lens catalog by. For defining what these relate to see lenskey and randkey. In the above example q0 is redshift, and q1 is optical richness.

    In general you can define an arbitrary number of quantities keeping the notation that the binning edges for quantity n are written as q[n]_edges.

  • param_bins_dev

    Parameter bins defined for mode: dev, e.g.:

    param_bins_dev:
        q0_edges: [0.2, 0.35]
        q1_edges: [45, 60]
    

    q0 and q1 refer to the zero-th and first quantities (in this order) you want to split your lens catalog by. For defining what these relate to see lenskey and randkey. In the above example q0 is redshift, and q1 is optical richness.

    In general you can define an arbitrary number of quantities keeping the notation that the binning edges for quantity n are written as q[n]_edges.

  • lenskey

    Aliases for the columns of the lens data table (assuming fits-like record table):

    lenskey:
      id: MEM_MATCH_ID
      ra: RA
      dec: DEC
      z: Z_LAMBDA
      q0: Z_LAMBDA
      q1: LAMBDA_CHISQ
    

    q0 and q1 refer to the zero-th and first quantities (in this order) you want to split your lens catalog by (see param_bins_*). In general you can define an arbitrary number of quantities keeping the notation that the alias for quantity n are written as q[n]. In the above example q0 is redshift, and q1 is optical richness.

  • randkey

    Aliases for the columns of the random points data table (assuming fits-like record table):

    randkey:
      q0: ZTRUE
      q1: AVG_LAMBDAOUT
      ra: RA
      dec: DEC
      z: ZTRUE
      w: WEIGHT
    

    q0 and q1 refer to the zero-th and first quantities (in this order) you want to split your random points catalog by. In general you can define an arbitrary number of quantities keeping the notation that the alias for quantity n are written as q[n]. In the above example q0 is redshift, and q1 is optical richness

    Note that for random points you have to specify the same quantities as for the lens catalog.

  • nprocess: 2

    Number of maximum processes or CPU-s to use at the same time (OpenMP-style parallelization).

  • njk_max: 100

    Maximum number of Jackknife regions to use in resampling. Actual number is max(n_lens, njk_max)

  • nrandoms

    Number of random points to use:

    nrandoms:
      full: 50000
      dev: 1000
    
  • seeds

    Random seed for choosing the random points random_seed, and for generating rotated shear catalogs shear_seed_master:

    seeds:
      random_seed: 5
      shear_seed_master: 10
    
  • cosmo_params

    Cosmology parameters defined as:

    cosmo_params:
      H0: 70.
      Om0: 0.3
    
  • radial_bins

    Logarithmic (base 10) radial bins from rmin to rmax:

    radial_bins:
      nbin: 15
      rmin: 0.0323
      rmax: 30.0
      units: Mpc
    

    Available units: Mpc, comoving_mpc or arcmin

  • weight_style: "optimal"

    Source weight style in the xshear lensing measurement. Use optimal when estimating \Delta\Sigma and uniform when measuring \gamma.

  • pairlog

    Specifies the amount of source-lens pairs to be saved, and for which radial range:

    pairlog:
     pairlog_rmin: 0
     pairlog_rmax: 0
     pairlog_nmax: 0
    

    Note that the pair limit is considered for each call of xshear separately. That is if you separate lenses into Jackknife regions then this is applicable for a single region.

  • lens_prefix: y1clust

    Prefix for lens-files

  • rand_prefix: y1rand

    Prefix for random points files

  • subtr_prefix: y1subtr

    Prefix for lens - random points files

  • fields_to_use: ['spt', 's82']

    List of names of observational fields to use (as defined below)

  • fields

    Definition of observational field boundaries:

    fields:
      spt:
        dec_top: -30.
        dec_bottom: -60.
        ra_left: 0.
        ra_right: 360.
      s82:
        dec_top: 10.
        dec_bottom: -10.
        ra_left: 300.
        ra_right: 10.
      d04:
        dec_top: 10.
        dec_bottom: -30.
        ra_left: 10.
        ra_right: 250.
    

    These can be approximate, the only requirement is that they divide the lens dataset into the appropriate chunks

  • pzpars

    Parameters for the boost factor extraction:

    pzpars:
      hist:
        nbin: 15
        zmin: 0.0
        zmax: 3.0
        tag: "zhist"
      full:
        tag: "zpdf"
      boost:
        rbmin: 3
        rbmax: 13
        refbin: 14
    

    There are two modes histogram hist which relies on Monte-Carlo samples of redshifts and is less robust, and full which uses the full P(z) of each source galaxy.

    • tag defines the name appended to the corresponding files.

    • boost defines the radial range for the boost estimation in radial bins

  • pdf_paths: null

    Regular expression matching the absolute paths of the BPZ output files containing the full redshift PDF. (e.g. /home/data/*.h5).

    NOTE This is only required for estimating the Boost factors, and can be safely left null in a simple lensing run.

inputs.yml

This config file lists the available data products. Currently all products are listed under the local key, indicating that they are found on disk, (as opposed to downloaded from some network location).

The two major sub-headings are:

  • shearcat

    Lists the available xshear-style source catalog files located within:

    [custom_data_path]/shearcat/
    

    where [custom_data_path] is the absolute path to the data folder specified by the corresponding key in params.yml

    Each input file has it’s key as an alias for the file name, such that you can use the key you define here for a valid value of shear_to_use for params.yml, e.g.:

    shearcat:
      default: default.dat
      im3shape: im3shape_shear_catalog.dat
      metacal: metacal_shear_catalog.dat
    

    These input files should be written in ASCII

  • lenscat

    Lists the available lens catalog files located within:

    [custom_data_path]/lenscat/
    

    where [custom_data_path] is the absolute path to the data folder specified by the corresponding key in params.yml

    Each dataset has it’s key as an alias, which you can use to define the lens dataset for a valid value of cat_to_use for params.yml. In addition, each dataset is implicitely assumed to consist of a lens catalog, and a corresponding catalog of random points, such that for each key there are two sub-keys: lens and rand. Both of these files should be written in fits format:

    lenscat:
      y1clust:
          lens: des_y1_lens_catalog.fits
          rand: des_y1_rand_catalog.fits
      svclust:
          lens: des_sv_lens_catalog.fits
          rand: des_sv_rand_catalog.fits
      testclust:
          lens: test_catalog.fits
          rand: null
    

    In case there are no random points available for the dataset you are using, it is safe to leave the rand field empty, but in this case make sure you also use the --norands flag when exectuing the pipeline scripts.

    In case the input catalog is defined in multiple files (for example when the parameter bins are not trivial to define), a list of filenames can be defined for lens and rand:

    lenscat:
        pre_binned_data:
            lens: [
                [des_y1_lens_catalog_bin-0-0.fits, des_y1_lens_catalog_bin-0-1.fits],
                [des_y1_lens_catalog_bin-1-0.fits, des_y1_lens_catalog_bin-1-1.fits],
            ]
    

    Note: The defined files will be assumed to correspond to separat parameter selections, and thus this mode cannot be used together with the definiton of parameter bins in params.yml