lib_ini_exper module

Routines for processing the experiment configuration.

lib_ini_exper.check_errors_ini_exper(data_dir, ini_folder, ini_stations, ini_sources, ini_delay_model, ini_media)[source]

Check experiment configuration for errors. Currently simply checking existence of paths and files.

data_dir : str
path to media folder.
ini_folder : str
path to experiment folder.
ini_stations : str
path to stations.ini (absolute).
ini_sources : str
path to sources.ini (absolute).
ini_delay_model : str
path to delay_model.ini (absolute).
ini_media : str
path to media.ini (absolute)
init_success : int
0 if errors, 1 otherwise.

TO DO:

Add more checks. Consider checking ini files syntax, range of values, etc.
lib_ini_exper.gen_delays_ini_file(file_stations_ini, params_array_stations, file_sources_ini, params_array_sources, file_delay_model_ini, params_array_delay_model, file_delays_ini, mjd_start, seconds_ref, tot_steps, step_seconds=1, seconds_offset=0, v=1)[source]

Generate file with delay polynomials from delay model.

file_stations_ini : str
path to stations.ini.
params_array_stations : list
configuration of stations.ini.
file_sources_ini : str
path to sources.ini.
params_array_sources : list
configuration of sources.ini.
file_delay_model_ini : str
path to delay_model.ini.
params_array_delay_model : list
configuration of delay_mode.ini.
file_delays_ini : str
path to delays.ini [will write].
mjd_start : int
MJD for the start of the scan.
seconds_ref : int
seconds in MJD for the start of the scan.
tot_steps : int
number of accumulation periods in the scan.
step_seconds : str(float)
accumulation period [s].
seconds_offset
0
v : int
verbose if 1.
s_delay
None if error, ini configuration to write if successful.
lib_ini_exper.get_num_partitions_red(one_baseline_per_task, accumulation_time, signal_duration, stations, auto_stations, max_num_channels, max_num_pols, v, file_log)[source]

Get number of reducers based on the experiment configuration.

one_baseline_per_task : int
[default 0] 0 for all baselines per task, 1 for single baseline per task.
accumulation_time : str(float)
accumulation period in seconds.
signal_duration : str(float)
signal duration in seconds.
stations : int
number of stations.
auto_stations : int
1 if compute autocorrelations, 0 otherwise.
max_num_channels : int
maximum number of channels per media file.
max_num_pol : int
maximum number of polarizations per media file.
v : int
verbose if 1.
file_log : file handler
handler to log file.
total_partitions : int
number of reducers.
This function is critical (together with the custom partitioner) to parallelize the
reduce phase and keep a well balanced load among the reducers.
lib_ini_exper.process_ini_correlation(params_array_corr, file_log, v=1)[source]

Process correlation.ini file.

params_array_corr
list with correlation configuration.
file_log
handler for log file.
v
verbose if 1.
stations : int
number of stations.
fft_size : int
number of coefficients in DFT.
accumulation_time : str(float)
accumulation time in seconds.
windowing : str
window type to be applied before DFT.
mjd_start : int
MJD for the start of the scan.
seconds_start : int
number of seconds in MJD for the start of the scan.
seconds_duration : int
duration of the scan in seconds.
phase_calibration : int
1 if phase calibration tones will be extracted, 0 otherwise.
auto_stations : int
1 if compute autocorrelations, 0 otherwise.
auto_pols : int
1 if crosspolarizations (see msvf.py).
first_frame_num : int
[testing only] start reading after this frame id (-1 to read from first frame).
num_frames : int
[testing only] read this number of frames per second (-1 to read all frames)
num_pols : int
[testing only] default -1.
lib_ini_exper.process_ini_files(data_dir, ini_stations, ini_sources, ini_delay_model, ini_delays, ini_media, ini_correlation, one_baseline_per_task, v=1, file_log=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)[source]

Process .ini files.

data_dir : str
path to folder with media files.
ini_stations : str
path to stations.ini.
ini_sources : str
path to sources.ini.
ini_delay_model : str
path to delay_model.ini.
ini_delays : str
path to delays.ini.
ini_media : str
path to media.ini.
ini_correlation : str
path to correlation.ini.
one_baseline_per_task : int
default 0.
v : int
verbose if 1.
file_log : file handler
handler for log file.
stations_serial_str : str
serialized version of stations.ini.
media_serial_str : str
serialized version of media.ini.
correlation_serial_str : str
serialized version of correlation.ini.
delays_serial_str : str
serialized version of delays.ini.
auto_stations : int
1 if compute autocorrelations, 0 otherwise.
auto_pols : int
1 if compute cross-polarization correlations, 2 if only same-polarization correlations (see msvf.py).
fft_size : int
requested number of coefficients in the FFT.
accumulation_time : str
integration period duration [s].
stations : int
number of stations (it will be used to compute number of mappers...).
ref_epoch : int
MJD for the start of the scan.
signal_start : int
seconds after MJD for the start of the scan.
signal_duration : float
duration of the scan [s].
input_files : list of str
filenames for media files.
first_frame_num : int
[default -1] This can be used to force the mapper to start reading frames after this id.
num_frames : int
[default -1] This can be used to force the mapper to read only this number of frames for each second.
codecs_serial : str
[default “0”] Serialized version of the codebook if using compression.
max_packet_size : int
number of bytes in the largest frame of the media files.
total_frames : int
number of frames in all the media files.
total_partitions : int
number of reducer tasks.
windowing : str
windowing as defined in correlation.ini.
phase_calibration : int
1 if extract phase calibration tones, 0 otherwise.
delay_error : None or str
None if no errors during delay computations, str with error otherwise.
error_str_v :
list of str with errors for media files not existing.
num_pols : int
-1

TO DO:

Remove num_pols.
Remove vq.
lib_ini_exper.process_ini_media(params_array_media, data_dir, v, file_log)[source]

Process media.ini file.

params_array_corr : list
list with media configuration.
data_dir : str
folder with media files.
v : int
verbose if 1.
file_log : file handler
handler for log file.
v_stations : list of str
all stations ids corresponding to all the media files listed.
max_num_channels : int
maximum number of channels per media file.
max_num_pols : int
maximum number of polarizations per media file.
input_files : list of str
filenames for media files.
total_frames : int
total frames considering all media files.
max_packet_size : int
maximum frame size per media file.
error_str_v : list of str
errors on missing media files ([] if no errors).

Considerations:

Currently assuming that all the media files have frames with the same size.
Currently assuming that all the files listed are processed, and that there is a section for each
of the listed files.
lib_ini_exper.process_ini_stations(params_array_stations, v_stations)[source]

Process stations.ini file.

params_array_corr
list with stations configuration.
v_stations
list of str with all stations ids corresponding to all the media files listed (from media).
v_id_stations
list of int with the ids corresponding to v_stations.