Config files explained¶

The aim of these config files is to automate running the weak lensing measurements and post-processing. The module xpipe.paths automatically tries to read the configs upon import, but you can also specify them later.

In the pipeline mode xpipe keeps track of reading lens catalogs, splitting them up into parameter bins, measuring the lensing signal and estimating covariances and contaminations automatically. However in order to do this, first we need to specify the details of these tasks in the below config files:

params.yml defines the measurement parameters
inputs.yml defines the available data files, and short aliases for them

These files do not exist yet when you first clone the repository, however there is a default_params.yml and default_inputs.yml which you should use as a reference. These defaults are set up such that when you create params.yml and inputs.yml they will be automatically looked for and read.

Config files are defined as yaml files, and are read as dictionaries, each entry consisting of a key and a corresponding value (note that this includes nested dictonaries).

Note that in yaml one should use null instead of None

Load order¶

default_params.yml
looks for params.yml
tries to read custom_params_file from params.yml

from this point the load is recursive, e.g. param files are loaded as long as there is a valid custom params file defined in the last loaded config. Each new config file only updates the settings, such that keys which are not present in the later files are left at their latest value.

The parameters defined here are loaded into the dictionary xpipe.paths.params

params.yml¶

Key reference

custom_params_file: params.yml

If you want to use an other parameter file, then specify it here. It must be in the same directory
custom_data_path: False

Absolute path to the data directory of the pipeline. If False: uses default project_path + /data
mode: full

The pipeline supports two modes: full and dev. This is primarily used in setting up the input files for the measurement. e.g. you can define two binning schemes: one really complex for the full run, and a simple, quicker for dev
tag: default

Prefix for all files (with NO trailing “_”). In addition this will be the name of the directory wher input and output files are written to.
shear_style: reduced

Format of the source galaxy catalog. Available formats are reduced, lensfit and metacal
cat_to_use: default

Alias for the lens catalog to be used (in this case the default). Aliases are defined in inputs.yml
shear_to_use: default

Alias for the source catalog to be used (in this case the default). Aliases are defined in inputs.yml
param_bins_full
Parameter bins defined for mode: full, e.g.:
param_bins_full: q0_edges: [0.2, 0.35, 0.5, 0.65] q1_edges: [5., 10., 14., 20., 30., 45., 60., 999]
q0 and q1 refer to the zero-th and first quantities (in this order) you want to split your lens catalog by. For defining what these relate to see lenskey and randkey. In the above example q0 is redshift, and q1 is optical richness.

In general you can define an arbitrary number of quantities keeping the notation that the binning edges for quantity n are written as q[n]_edges.
param_bins_dev
Parameter bins defined for mode: dev, e.g.:
param_bins_dev: q0_edges: [0.2, 0.35] q1_edges: [45, 60]
q0 and q1 refer to the zero-th and first quantities (in this order) you want to split your lens catalog by. For defining what these relate to see lenskey and randkey. In the above example q0 is redshift, and q1 is optical richness.

In general you can define an arbitrary number of quantities keeping the notation that the binning edges for quantity n are written as q[n]_edges.
lenskey
Aliases for the columns of the lens data table (assuming fits-like record table):
lenskey: id: MEM_MATCH_ID ra: RA dec: DEC z: Z_LAMBDA q0: Z_LAMBDA q1: LAMBDA_CHISQ
q0 and q1 refer to the zero-th and first quantities (in this order) you want to split your lens catalog by (see param_bins_*). In general you can define an arbitrary number of quantities keeping the notation that the alias for quantity n are written as q[n]. In the above example q0 is redshift, and q1 is optical richness.
randkey
Aliases for the columns of the random points data table (assuming fits-like record table):
randkey: q0: ZTRUE q1: AVG_LAMBDAOUT ra: RA dec: DEC z: ZTRUE w: WEIGHT
q0 and q1 refer to the zero-th and first quantities (in this order) you want to split your random points catalog by. In general you can define an arbitrary number of quantities keeping the notation that the alias for quantity n are written as q[n]. In the above example q0 is redshift, and q1 is optical richness

Note that for random points you have to specify the same quantities as for the lens catalog.
nprocess: 2

Number of maximum processes or CPU-s to use at the same time (OpenMP-style parallelization).
njk_max: 100

Maximum number of Jackknife regions to use in resampling. Actual number is max(n_lens, njk_max)
nrandoms
Number of random points to use:
nrandoms: full: 50000 dev: 1000
seeds
Random seed for choosing the random points random_seed, and for generating rotated shear catalogs shear_seed_master:
seeds: random_seed: 5 shear_seed_master: 10
cosmo_params
Cosmology parameters defined as:
cosmo_params: H0: 70. Om0: 0.3
radial_bins
Logarithmic (base 10) radial bins from rmin to rmax:
radial_bins: nbin: 15 rmin: 0.0323 rmax: 30.0 units: Mpc
Available units: Mpc, comoving_mpc or arcmin
weight_style: "optimal"

Source weight style in the xshear lensing measurement. Use optimal when estimating $\Delta\Sigma$ and uniform when measuring $\gamma$ .
pairlog
Specifies the amount of source-lens pairs to be saved, and for which radial range:
pairlog: pairlog_rmin: 0 pairlog_rmax: 0 pairlog_nmax: 0
Note that the pair limit is considered for each call of xshear separately. That is if you separate lenses into Jackknife regions then this is applicable for a single region.
lens_prefix: y1clust

Prefix for lens-files
rand_prefix: y1rand

Prefix for random points files
subtr_prefix: y1subtr

Prefix for lens - random points files
fields_to_use: ['spt', 's82']

List of names of observational fields to use (as defined below)

fields

Definition of observational field boundaries:

fields:
  spt:
    dec_top: -30.
    dec_bottom: -60.
    ra_left: 0.
    ra_right: 360.
  s82:
    dec_top: 10.
    dec_bottom: -10.
    ra_left: 300.
    ra_right: 10.
  d04:
    dec_top: 10.
    dec_bottom: -30.
    ra_left: 10.
    ra_right: 250.

These can be approximate, the only requirement is that they divide the lens dataset into the appropriate chunks

pzpars
Parameters for the boost factor extraction:
pzpars: hist: nbin: 15 zmin: 0.0 zmax: 3.0 tag: "zhist" full: tag: "zpdf" boost: rbmin: 3 rbmax: 13 refbin: 14
There are two modes histogram hist which relies on Monte-Carlo samples of redshifts and is less robust, and full which uses the full P(z) of each source galaxy.
- tag defines the name appended to the corresponding files.
- boost defines the radial range for the boost estimation in radial bins
pdf_paths: null

Regular expression matching the absolute paths of the BPZ output files containing the full redshift PDF. (e.g. /home/data/*.h5).

NOTE This is only required for estimating the Boost factors, and can be safely left null in a simple lensing run.

inputs.yml¶

This config file lists the available data products. Currently all products are listed under the local key, indicating that they are found on disk, (as opposed to downloaded from some network location).

The two major sub-headings are:

shearcat
Lists the available xshear-style source catalog files located within:
[custom_data_path]/shearcat/
where [custom_data_path] is the absolute path to the data folder specified by the corresponding key in params.yml

Each input file has it’s key as an alias for the file name, such that you can use the key you define here for a valid value of shear_to_use for params.yml, e.g.:
shearcat: default: default.dat im3shape: im3shape_shear_catalog.dat metacal: metacal_shear_catalog.dat
These input files should be written in ASCII
lenscat
Lists the available lens catalog files located within:
[custom_data_path]/lenscat/
where [custom_data_path] is the absolute path to the data folder specified by the corresponding key in params.yml

Each dataset has it’s key as an alias, which you can use to define the lens dataset for a valid value of cat_to_use for params.yml. In addition, each dataset is implicitely assumed to consist of a lens catalog, and a corresponding catalog of random points, such that for each key there are two sub-keys: lens and rand. Both of these files should be written in fits format:
lenscat: y1clust: lens: des_y1_lens_catalog.fits rand: des_y1_rand_catalog.fits svclust: lens: des_sv_lens_catalog.fits rand: des_sv_rand_catalog.fits testclust: lens: test_catalog.fits rand: null
In case there are no random points available for the dataset you are using, it is safe to leave the rand field empty, but in this case make sure you also use the --norands flag when exectuing the pipeline scripts.

In case the input catalog is defined in multiple files (for example when the parameter bins are not trivial to define), a list of filenames can be defined for lens and rand:
lenscat: pre_binned_data: lens: [ [des_y1_lens_catalog_bin-0-0.fits, des_y1_lens_catalog_bin-0-1.fits], [des_y1_lens_catalog_bin-1-0.fits, des_y1_lens_catalog_bin-1-1.fits], ]
Note: The defined files will be assumed to correspond to separat parameter selections, and thus this mode cannot be used together with the definiton of parameter bins in params.yml