cx2d_lib module

Module with basic functions to convert output from CorrelX/CX to DiFX/SWIN+PCAL.

The objective of this library is to connect the CorrelX processing chain with: | *vex2difx+calcif2 (input): for generating the configuration files. | *difx2mark4 (output): for generating a mark4 fileset that can be processed with fourfit.

Conventions CorrelX

Make sure that the same file const_mapred.py used for the correlation is imported here, since the fields in the header are

accesed using the constants const_mapred.INDEX_*.

Conventions DiFX

Conventions are as defined in: | DiFX/SWIN [DIFX_*]: CSIRO - http://www.atnf.csiro.au/vlbi/dokuwiki/doku.php/difx/files#the_swin_output_data_format (2016.01.12) | DiFX/PCAL [PCAL_*]: [Br14] NRAO - A Guide to the DiFX Software Correlator (Version 2.2), Section 6.8.2. Pulse cal data files. (2014.06.23) | DiFX/SWIN [.im,.input]: CSIRO - http://www.atnf.csiro.au/vlbi/dokuwiki/doku.php/difx/files (2016.01.12)

Sections of code

CX output (read)

CX output (debug)

DiFX/SWIN (write)

DiFX/SWIN (read)

Processing zoombands: CX -> CX_zoomed

Conversion CorrelX/CX -> DiFX/SWIN

Conversion CorrelX/CX -> DiFX/PCAL

Conversion CorrelX/CX -> DiFX/SWIN+PCAL

DiFX/.im,.input parser tools

Conversion DiFX/.im -> CorrelX/delay_model+sources

Conversion DiFX/.input -> CorrelX/stations+correlation+media

(!) Limitations

Single source. source_index=0

Single configuration. configuration_index=0

Single phase_centre. phase_centre=0

Single pulsar_bin. pulsar_bin=0

Data weight is forced to be 1

u,v,w are forced to be 0

Averaging currently only implemented for zoom bands

Zoom bands currently implemented during postprocessing

Notes

All constants are kept in this file due to strong dependence on external references.

TO DO

Create separate library and move there the CX output file processing functions.

Configuration conversion libraries: experimental, see limitations for input_to_media().

Print remainder and do checks for CX_IMPORT_CONST_MAPRED.

Remove back_compat option (replace by custom const_mapred import).

cx2d_lib.append_pcal_records(records, chan_freq_out_mega, pcal_freq_out_mega, pcal_ind, datac, n_bins, pol_char, conjugate_pcal_values, pcal_scaling)

Append a set of new phase calibration tone records for this band. Record is the “group of four numbers” in [Br14].

records : list of str

records (previously generated for other bands for the same accumulation period.

chan_freq_out_mod_mega : int or float

frequency of the first phase calibration tone in this band[Mhz].

pcal_freq_out_mega : int or float

phase calibration tone frequency separation [Mhz].

pcal_ind : list of ints

positions with phase calibration indices.

datac : complex 1D numpy array

phase calibration results (DFT).

n_bins : int

number of bins in the phase calibration DFT.

pol_char : char

polarization.

conjugate_pcal_values : int

conjugate all valid results if 1.

pcal_scaling : float

multiply all valid results by this value.

records : list of str

appended list of str including (the new elements are the records for this band).

TO DO:

Check if N == n_bins and simplify code.

Add offset from configuration.

cx2d_lib.compute_baseline_num_swin(st0, st1)

Takes two integers with IDs for stations are computes baselines number as 256*([st0]+1)+([st1]+1).

st0,st1 : int

values for station 0 and station 1 starting at zero.

baseline_num : int

baseline number

cx2d_lib.compute_stations_num_swin(baseline_num)

Returns two integers with IDs for station based on SWIN header baseline number. See compute_baseline_num_swin().

Values for st0 and st1 starting at zero.

cx2d_lib.convert_cx2d(doutput_file, cxoutput_file, correlation_ini_file, media_ini_file, forced_pol_list=[], only_half=0, duplicate=0, freq_ids=[], v=1, back_compat=1, forced_accumulation_period=-1, divide_vis_by=1, conjugate_vis_values=0)

Convert visibilities from an output file from CorrelX/CX to DiFX/SWIN format.

doutput_file : str

output file name (will append DIFX_...) for new SWIN file.

cxoutput_file : str

path to CX file to be converted

correlation_ini_file : str

path to correlation.ini.

media_ini_file : str

path to media.ini.

forced_pol_list : list of str, optional

used to override polarizations in ini file.

only_half : optional

see create_bytes_list_visibilities_swin().

duplicate : optional

see create_bytes_list_visibilities_swin().

freq_ids : unused

[remove]

v : int, optional

verbose if 1.

back_compat : int

[remove] see notes.

forced_accumulation_period : int, optional

see create_bytes_list_visibilities_swin().

divide_vis_by : int

see create_bytes_list_visibilities_swin().

conjugate_vis_values : int

see create_bytes_list_visibilities_swin().

doutput_file : str

path to newly created SwIN output file.

CX accumulation periods referenced by start time, SWIN by middle time.

It is assumed that all the polarizations [0,1,2,...] (as many as used) are defined in the media.ini file.

(!) Limitations:

See limitations on top of this file (forced values...).

TO DO:

Remove freq_ids.

Remove back_compat and add version for CX file header instead.

cx2d_lib.convert_cx2dpc(inout_folder, file_in, file_out, correlation_ini_file, media_ini_file, stations_ini_file, v=0, only_half=0, duplicate=0, freq_ids=[], back_compat=1, forced_accumulation_period=-1, divide_vis_by=1, forced_file_list=[], pcal_scaling=0, conjugate_pcal_values=0, conjugate_vis_values=0)

Main routine to convert CorrelX/CX into DiFX/SWIN+PCAL.

inout_folder : str

path to folder containing CX file, and where newly created SWIN+PCAL files will be placed.

file_in : str

CX filename.

file_out : str

filename for new SwIN file.

correlation_ini_file : str

path to correlation.ini associated with CX file.

media_ini_file : str

path to media.ini associated with CX file.

stations_ini_file: path to stations.ini associated with CX file.

See convert_cx2d() for the rest of the arguments.

pcal_file_list : list of str

filenames for new PCAL files (None if error).

cx2d_lib.create_bytes_list_visibilities_swin(data_complex_list, only_half=0, duplicate=0, divide_vis_by=1, conjugate_vis_values=0)

Generate binary representation for one set of visibilities.

data_complex_list : 1D numpy arrayof complex

visibilities (components from 0 to N-1).

only_half : optional,testing

if 1 takes components [0,N/2-1], if 2 it takes [N/2,N-1].

duplicate : optional,testing

duplicate components [consider removing].

divide_vis_by : optional,testing

divide all coefficients by this value.

conjugate_vis_values : optional,testing

conjugate all visibilities.

header_list : byte array

visibilities ready to be written to a file.

TO DO:

Consider removing duplicate.

cx2d_lib.create_header_swin(st0, st1, vis, chan, pol0, pol1, mjd, seconds_start, accumulation_period, polarization_chars_list, source_index=0, config_index=0, pulsar_bin=0, header_version=1, forced_weight=1.0, forced_u=0.0, forced_v=0.0, forced_w=0.0)

Byte representation for one header.

st0 : int

first station in the baseline.

st1 : int

second station in the baseline.

vis : int

accumulation period id.

chan : int

channel id.

pol0 : char

polarization id for st0.

pol1 : char

polarization id for st1.

mjd : int

MJD for this accumulation period.

seconds_start : float

seconds corresponding to the middle of the first accumulation period.

accumulation_period : float

duration of the accumulation period in seconds.

polarization_chars_list : list of str

polarization characters to accessed by id (pol0 and pol1).

source_index : int,optional

See constants on top of this file.

config_index : int,optional pulsar_bin : int,optional header_version : int,optional forced_weight : float,optional forced_u : float,optional forced_v : float,optional forced_w : float,optional

header_list : byte array

header ready to be written to a file.

Optional arguments are replaced by forced values.

Note that secnods start is not at the start of the first accumulation period but at the middle!

Values for st0,st1,vis,chan,pol0,pol1 starting at 0.

TO DO:

Change input floats by double as applicable.

cx2d_lib.create_symb_media(inout_folder, file_input_v, forced_files='', symb_dir=None)

Create symbolic links for media. This links are also part of the configuration.

inout_folder : str

path to folder containing newly created ini files.

file_input_v : list of str

paths to media files.

forced_files : str

comma separated list of paths to media files to override file paths from .input file. If “”, then the pahts from the .input file are used.

symb_dir : str,optional

path relative to inout_folder where links will be created.

file_list : list of str

filenames to be used in media.ini.

files_str : str

comma separated list of filenames.

cx2d_lib.cxpcal2d(doutput_folder, cxoutput_file, correlation_ini_file, media_ini_file, stations_ini_file, forced_file_list=[], pcal_scaling=0, conjugate_pcal_values=0, v=1)

Generate phase calibration tone files (“pulse cal data files”) DiFX/SWIN from CorrelX/CX.

doutput_folder : str

path to place new PCAL files.

cxoutput_file : str

path to CX file to read.

correlation_ini_file : str

path to correlation.ini.

media_ini_file : str

path to media.ini.

stations_ini_file : str

path to stations.ini.

forced_file_list : optional,[testing]

consider only these media files (consider all if []).

pcal_scaling : float, optional

if not 0, multiply all phase calibration tones by this value.

conjugate_pcal_values : optional,[testing]

conjugate phase calibration tone values.

v : int

verbose if 1.

name_file_list : list of str

names of the newly created PCAL files (no path).

TO DO:

(!) This function needs to be simplified.

Assuming that number of channels and polarizations equals the max id plus one.

Phase calibration line string currently hardcoded (pc).

Untested for real data.

fs=2fs, check this, consider reading one frame of the media to determine if complex/real data.

LSB/USB.

Offset for pcal.

cx2d_lib.get_char_from_header(header, i)

Read char from a np.uint8 array (header) starting at position i.

cx2d_lib.get_coeff(line)

Process station clock information from .input file.

cx2d_lib.get_difx_filename(mjd_start, seconds_start)

Get filename for SWIN file.

TO DO:

Add missing funcionality for filling DIFX_FILENAME_SUFFIX.

cx2d_lib.get_double_from_header(header, i)

Read double from a np.uint8 array (header) starting at position i.

cx2d_lib.get_error_indicator(file_vis_1, file_vis_2, force=0, path_src='')

Provides the sum of the L2-norm between all pairs of visibilities. Use only for debugging comparing two CorrelX output files (e.g. testing changes).

file_vis_1 : str

path to the reference file with visibilities.

file_vis_2 : str

path to the test file with visibilities.

force : int

continue execution even if metadata differ.

path_src : str

path to location of source file const_mapred.py.

None

Assumptions / TO DO

Currently assuming that then number of coefficients does not change for lines in the same file.

Currently no error checking if forcing execution (e.g. different number lines).

cx2d_lib.get_field_im(line)

Get the value+units from a line of DiFX configuration file.

cx2d_lib.get_float_from_header(header, i)

Read float from a np.uint8 array (header) starting at position i.

cx2d_lib.get_header_dm(mjd, seconds, interval, src, ant)

Create header for delay_model.ini.

TO DO: consider moving into lib_ini_files.py.

cx2d_lib.get_id_param(line, num)

Get the id that is in the parameter, in the position num.

cx2d_lib.get_int_from_header(header, i)

Read integer from a np.uint8 array (header) starting at position i.

cx2d_lib.get_last_num(line)

Get the id that is in the parameter, in the last position.

cx2d_lib.get_lines_pcal_header(mjd_start_str, seconds_start_str, stations_name_str)

Get header lines for PCAL file.

mjd_start_str : str

start MJD for this accumulation period.

seconds_start_str: str

start seconds for this accumulation period.

station_name_str : str

name of the station.

header_pcal_v : list of str

lines for the PCAL file header.

cx2d_lib.get_list_meta(const_file='const_mapred.py', path_src='')

Get list with metadata fields from source file const_mapred.py.

Use for debugging.

const_file : str

path to const_mapred file.

path_src : str

path to location of source file const_mapred.py.

line_st_v : list of str

all metadata fields (without INDEX_).

Hardcoded values based on const_mapred.py, use only for debugging.

cx2d_lib.get_pcal_filename(mjd_start_str, seconds_start, station_name_str)

Get filename for PCAL file.

cx2d_lib.get_pcal_line(meta_pcal, records)

Get a line for the PCAL file.

meta_pcal : str

metadata.

records : list of str

records.

cx2d_lib.get_pcal_meta(mjd_start_str, seconds_start, acc_period, seconds_duration, station_name, st0, tot_channels, num_tones_pcal)

Create metadata for PCAL record.

mjd_start_str : str

start MJD for the scan.

seconds_start : float

start seconds for the scan.

acc_period : int

accumulation period id.

seconds_duration : float

accumulation period duration

station_name : two char

station name.

st0 : int

station id.

tot_channels : int

number of channels or bands.

num_tones_pcal : int

number of phase calibration tones per band.

meta_pcal : str

metadata preceding the records for one accumulation period.

TO DO:

Check offset +seconds_duration/2.

st0 should be datastream id?

cx2d_lib.get_pcal_record_valid(pcal_value, tone_freq_mega, pol_char)

Get record with valid result for phase calibration tone.

pcal_value : complex

phase calibration tone.

tone_freq_mega: int or float

pcal tone freq [MHz].

pol_char : char

polarization.

cx2d_lib.get_pcal_tone_positions(N, bw, chan_freq, pcal_freq, num_tones_pcal)

Get positions of the phase calibration tones in the results.

N : int

number of coefficients in the results (DFT of accumulated windows).

bw : int or float

bandwidth of the channel [Hz].

chan_freq : int or float

lower edge frequency of the channel [Hz].

pcal_freq : int or float

phase calibration tone frequency.

num_tones_pcal: number of phase calibration tones expected for this band.

pcal_ind_mod : list of int

position for the coeffients containing the phase calibration tones.

extreme_value : int

value used to indicate invalid index.

cx2d_lib.get_pos_in_fft(f, fz, fft_size, bw)

Get the coefficient corresponding to a certain frequency in the visibilities.

f : float

lower edge frequency of the band.

fz : float

frequency between f and f+bw (which associated coefficient is to be found).

fft_size : int

number of coefficients of the visibilities.

bw : float

bandwidth of the full band.

output : int

position of the first coefficient.

TO DO:

LSB/USB.

cx2d_lib.get_src_ant_im(line)

Get two str, one with source id and another with station id from the line in .im file.

cx2d_lib.get_value_im(line)

Get the value (no units) from a line of DiFX configuration file.

cx2d_lib.get_vector_im(line)

Get a list of str with polynomial coefficients from line in .im file.

cx2d_lib.get_zoom_list(params_array_media, params_array_corr, v=0, average_channels=-1)

Get the information required to process the zoom bands.

params_array_media : list

list with media configuration (see lib_ini_files.py).

params_array_corr : list

list with correlation configuration (see lib_ini_files.py).

v : int

verbose if 1.

average_channels : int

number of coefficients that are average into one (here only used for reporting).

z_list : list

list of elements [band_id, first_sample_fft, last_sample_fft, new_band_id] where:

band_id: id of the processed band.

first_sample_fft: position of the first element of the zoom in the full fft.

last_sample_fft: position of the last element of the zoom in the full fft.

new_band_id: id of the newly created band (zoom).

Assumptions:

It is assumed that a channel has only one associated sideband (this is checked when computing the

limits of the band, if lower sideband the upper edge is given.

It is assumed that zoom bands are always upper side band.

TO DO:

LSB/USB for zoom band definitions...

cx2d_lib.im_to_delay_model(inout_folder, file_in, file_out, filter_sources=[], v=0)

Convert DiFX/.im into CorrelX/delay_model.ini.

inout_folder : str

path to folder containing .im file, and where newly created delay_model.ini file will be placed.

file_in : str

.im filename.

file_out: str

delay_model.ini filename.

filter_sources: list of str

source names. If not [], information for sources that are not in this list will be dismissed.

v : int

verbose if 1.

None

cx2d_lib.im_to_sources(inout_folder, file_in, file_out, v=0)

Convert DiFX/.im into CorrelX/sources.ini.

inout_folder : str

path to folder containing .im file, and where newly created sources.ini file will be placed.

file_in : str

.im filename.

file_out : str

sources.ini filename.

v : int

verbose if 1.

None

cx2d_lib.input_to_correlation(inout_folder, file_in, file_out, v=0)

Convert DiFX/.input into CorrelX/correlation.ini.

inout_folder : str

path to folder containing .input file, and where newly created correlation.ini file will be placed.

file_in : str

.input filename.

file_out : str

correlation.ini filename.

v : int

verbose if 1.

None

Assumptions:

Assuming one data stream per station for computing the number of stations.

cx2d_lib.input_to_media(inout_folder, file_in, file_out, forced_files='', v=0)

Convert DiFX/.input into CorrelX/media.ini, create symbolic links for media files

and generate report with summary (for reporting/debugging).

inout_folder : str

path to folder containing .input file, and where newly created media.ini file will be placed.

file_in : str

.input filename.

file_out : str

media.ini filename.

forced_files : str, required

list of files to be used in the media files

v : int

verbose if 1.

None

Limitations:

Currently limited to sets of files with the same configuration (i.e. no missmatched bands).

Assumptions:

The function is strongly dependent on the current order of the fields in the .input file.

The zoom bands are expected to appear after the normal bands.

Assuming incremental id starting at 0 in input file.

TO DO:

Read forced files from the .input file.

cx2d_lib.input_to_stations(inout_folder, file_in, file_out, forced_stations=[], v=0)

Convert DiFX/.input into CorrelX/stations.ini.

inout_folder : str

path to folder containing .input file, and where newly created stations.ini file will be placed.

file_in : str

.input filename.

file_out : str

stations.ini filename.

forced_files : list of str,optional

list of str with station names (for overriding values from .input).

v : int

verbose if 1.

None

cx2d_lib.plot_pcal_tones(datac, pcal_ind, extreme_value)

Plot phase calibration tones in red, overlaying the DFT with all results. Use for debugging.

datac : numpy array of complex

DFT with pcal results.

pcal_ind : list of int

positions of the phase calibration tones.

extreme_value : int

value used to indicate invalid index.

cx2d_lib.plot_vis_cx(file_input, title_figure, mode_in='px', max_lines=-2, interval_start=0, interval_end=-1, filter_str='')

Basic function for plotting output, one plot per band.

file_input : str

path to the visibilities file.

title_figure : str

prefix for the title of the figures

mode_in : str

prefix of cx file lines.

max_lines : int,optional

maximum number of lines to read.

interval_start: int,optional

first coefficient to plot.

interval_end : int,optional

last coefficient to plot.

filter_str : str,optional

filter lines with this string (for example only a certain baseline).

N/A

cx2d_lib.process_zoom_band(inout_folder, file_in, file_out, correlation_ini_file='correlation.ini', media_ini_file='media.ini', stations_ini_file='stations.ini', v=1, average_channels=-1, filter_acc_periods=[])

Generate a new CX file with the zoom bands from an existing CX file with results for the full band.

inout_folder : str

path to folder containing CX file, and where newly created CX_zoom file will be placed.

file_in : str

CX filename.

file_out : str

filename for new CX file with zoom bands results.

correlation_ini_file : str

path to correlation.ini.

media_ini_file : str

path to media.ini.

correlation_ini_file : str

path to correlation.ini.

stations_ini_file : str

path to stations.ini.

v : int

verbose if 1.

average_channels : int

number of coefficients to average (-1 for no averaging).

filter_acc_periods : list of int

ids (starting at 0) for accumulation periods to process. Will process all if [].

fft_read : int

number of coefficients in the last visibilities read from the file.

Assumptions:

Assuming a regular configuration where all the stations have the same zoom bands (i.e.: missmatched bands not suppported).

TO DO:

Migrate this functionality into lib_fx_stack.py so that missmatched band support can be provided.

“px” harcoded, move to const_mapred.

cx2d_lib.read_bytes_from_file(f, n_values, read_type='bytes', v=0)

Read groups of one or four bytes from a binary file.

f : handlder to binary file (‘rb’)

binary file to read from.

n_values : int

number of groups to read.

read_type : { “bytes” , “floats” }

“bytes” to read unsigned integer 8 bits, “floats” to read unsigned integer 32 bits.

v : int

verbose if 1.

words_array : numpy array of unsigned int of 8 or 32 bits (as configured in read_type).

read bytes.

TO DO:

Change notation for floats.

cx2d_lib.read_cxoutput(cxoutput_file, v=1, back_compat=0)

Read cx output file into list.

cxoutput_file : str

path to cx file.

v : int

verbose if 1.

back_compat : int,optional

[default 0][remove]

list_output : list

list of [st0,st1,vis,chan,pol0,pol1,diff_st] elements where:

st0: first station in the baseline.

st1: second station in the baseline.

vis: accumulation period id.

chan: channel id.

pol0: polarization id for st0.

pol1: polarization id for st1.

datac: complex 1D np.array with visibilities.

diff_st: field used for sorting.

TO DO

Sorting convention based on sorting based on actual SWIN files. Check method and find proper reference.

Remove back_compat in libraries calling this function.

cx2d_lib.read_doutput(doutput_file, complex_vector_length, vis_limit=10, filter_bands=[], filter_pols=[], filter_seconds=[], v=0, interval_start=0, interval_end=-1)

Plot visibilities from a SWIN file.

doutput_file : str

path to SWIN file.

complex_vector_length : int

number of coefficients in the visibilities.

vis_limit : int, optional

number of visibilities to read (-1 for no limit).

filter_bands : list of int, optional

band ids to include ([] to include all). E.g.: [0,1]

filter_pols : list of str, optional

band ids to include ([] to include all). E.g.: [“LR”,”RL”]

filter_seconds : list of ints, optional

band ids to include ([] to include all). E.g.: [“0.16”,”0.48”]

v : int, optional

verbose if 1.

N/A

Configuration:

Constant ENABLE_PLOTTING=1 to display plots.

Notes:

Visbilities are displayed into two figures: magnitude and phase.

TO DO:

Add checks for header.

Consider automating the computation of complex_vector_length.

cx2d_lib.read_line_cx(line, fft_size=-1)

Read a line from a CX file (and check number of visibilities if required).

line : str

line from CX file.

fft_size : int,optional

number of coefficients in the visibilities (or pcal bins).

meta : str

line header.

st0 : int

first station of the baseline.

st1 : int

second station of the baseline.

key : int

absolute key (see msvf.get_pair_str().key_value).

vis : int

accumulation period.

chan : int

band id.

pol0 : int

polarization id for st0.

pol1 : int

polariation id for st1.

predata : str

metadata fields.

datac : complex 1D np.array

visibilities.

cx2d_lib.replace_channel_in_key(meta, new_band_id)

Replace the band id in the CX header key.

meta : str

CX line header.

new_band_id : int

id for the new (zoom) band.

new_meta : str

new CX line header.

Configuration:

INDEX_KEY_CHANNEL: const_mapred.py (location of the channel id in the key (SF_SEP), to be replaced by new channel id [zoom])

TO DO:

Create general funcionts to create and read key.

cx2d_lib.show_line_cx(file_in, line_start, line_count, filter_line='px', v=1, path_src='')

Display metadata fields and number of coefficients in the visibilities.

Use for debugging.

file_in : str

path to CX file.

line_start : int

first line to read.

line_count : int

number of lines to read (-1 for no limit).

filter_line : str

pattern to filter lines (exact match at line start) to be displayed.

v : int

verbose if 1.

path_src : str

path to location of source file const_mapred.py.

results : list of [str,list of float]

keys and the associated visibilities.

cx2d_lib.sort_str_list(list_in)

Return sorted set of elements of a list.

cx2d_lib.sort_swin_records(output_list)

Sort SWIN output records.

output_list : list of [st0,st1,vis,chan,pol0,pol1,header,values_bytes,diff_st]

see convert_cx2d.output_list.

Sorting is as follows:

1. accumulation period

2. term based on stations ids (see read_cxoutput())

3. first station of the baseline

4. second station of the baseline

5. band

6. polarization for first station

7. polarization for second station

TO DO:

x[2] duplicated, remove second one

cx2d_lib.split_line_correct_tab(line)

Separate key and value in CX line.

Update if changes to msvf.get_pair_str are done (key separator FIELD_SEP+KEY_SEP).

cx2d_lib.write_lines_to_f(lines_out, full_output_file, str_info='results')

Write a list of strings into a file, one per line, and report.

cx2d_lib.write_pcal_file(pcal_file, mjd_start_str, seconds_start, station_name_str, records_v)

Write PCAL file.

pcal_file : str

path to pcal file.

mjd_start_str : str

start MJD.

seconds_start : str

start seconds.

station_name_str : two char.

Two-character code for the station.

records_v : list of str

lines with pcal records.