lib_vdif module

Module with basic functions to read and write VDIF headers and data.

– VDIF Specification version [VD01]–

VLBI Data Interchange Format (VDIF) Specification

Release 1.0 (w/24 August 2009 corrections/clarifications highlighted)

Ratified 26 June 2009

Notes

-bitarray library still used for VDIF file generation.

Updated 2015.10.15:

-frame length (//8 when writing header, *8 when reading header)

-changed endiannes to be correct

2015.12.03: changed bits2int(bitsin)

2015.12.03: read_samples is now more efficient, but may not work with numbers of bits per sample that do not divide 32.

TO DO: implement in “read_samples” the functionality to read bits per sample that do not divide 32.

TO DO: testing on epoch and seconds conversions.

2016.09.21: bitarray reader implementation removed in favor of raw which is faster.

lib_vdif.bits2int(bitsin)

lib_vdif.compute_range(word_size, bits_per_sample)

Get starting positions in word for groups of bits_per_sample bits.

Example:

word_size=32

bits_per_sample=4

list(compute_range(word_size,bits_per_sample))

>>> [0, 4, 8, 12, 16, 20, 24, 28]

lib_vdif.create_header_vdif(seconds_fr=17, invalid=False, legacy=True, ref_epoch=15, frame_num=1, vdif_version=1, log_2_channels=1, frame_length=20000, data_type=0, bits_per_sample=2, thread_id=1, station_id=1)

TO DO: check epoch...

lib_vdif.date_to_vdif(year, month, day, hour, minute, second, offset_seconds=0)

Compute VDIF fields for epoch and seconds from date.

lib_vdif.disp_list(a)

lib_vdif.get_vdif_num_frames(filename, packet_limit=-1, forced_packet_size=0, offset_bytes=0, only_offset_once=1, v=1)

Get information from vdif file by processing headers. As many headers as packet_limit are processed. Each of the output parameters are vectors with all the possible values found.

See get_vdif_stats().

num_frames : int

number of frames read.

TO DO:

This is a modified version of get_vdif_stats(), merge with it.

lib_vdif.get_vdif_stats(filename, packet_limit=-1, forced_packet_size=0, offset_bytes=0, only_offset_once=1, v=1, short_output=0)

Get information from vdif file by processing headers. As many headers as packet_limit are processed. Each of the output parameters are vectors with all the possible values found.

filename : str

path to VDIF file.

packet_limit : int

[default -1] maximum number of packets, if -1 no limit.

forced_packet_size : int

[default 0] if >0 will read this number of bytes instead those in the VDIF header.

offset_bytes : int

[default 0] skip this number of bytes before reading the first VDIF header.

only_offset_once : int

[default 1] if ==0 will skip offset_bytes before every frame to be read.

v : int

[default 1] verbose mode if 1.

short_output : int

[default 0] group very long vector (frames) if 1.

v_stations : list

list with unique ids found for stations in all frames read.

v_seconds : list

list with unique seconds found for all frames read.

v_frames : list

list with unique frame numbers for all frames read.

v_sizes : list

list with unique frame sizes for all frames read.

total_size : int total size for all frames read.

lib_vdif.int2bits(value)

lib_vdif.print_header_mark5(sync_word, years, user_data, bit_t, frame_num, time_code_w1, time_code_w2, crcc)

[DELETE]

Untested. Consider deleting.

lib_vdif.print_header_vdif(seconds_fr=17, invalid=False, legacy=True, ref_epoch=15, frame_num=1, vdif_version=1, log_2_channels=1, frame_length=20000, data_type=0, bits_per_sample=2, thread_id=1, station_id=1)

Show header of vdif frame, for logging.

See read_header_vdif_from_raw() output.

lib_vdif.print_header_vdif_info(brief)

Show stats results header.

lib_vdif.print_header_vdif_row(id=0, seconds_fr=17, invalid=False, legacy=True, ref_epoch=15, frame_num=1, vdif_version=1, log_2_channels=1, frame_length=20000, data_type=0, bits_per_sample=2, thread_id=1, station_id=1)

lib_vdif.read_bytes_offset_file(f, n_bytes, v=0)

Used to skip some offset when reading a binary file.

f : file handler

(sys.stdin).

n_words : int

number of words of type TYPE_WORD.

v : int

[0 by default] verbose mode if 1.

lib_vdif.read_header_mark5(header)

[DELETE]

Untested. Consider deleting.

lib_vdif.read_header_vdif_from_raw(header)

Reader vdif header.

header : numpy array with four np.unit32 words.

[seconds_fr,invalid,legacy,ref_epoch,frame_num,vdif_version,log_2_channels,

frame_length,data_type,bits_per_sample,thread_id,station_id]: where:

seconds_fr: VDIF frame seconds (integer).

invalid: VDIF invalid bit.

legacy: VDIF legacy bit.

ref_epoch: VDIF frame epoch (float with MJD (TBC)).

frame_num: VDIF frame number (integer).

vdif_version: VDIF frame version (integer).

log_2_channels: VDIF logarithm in base 2 of the number of channels in the frame.

frame_length: VDIF frame length (integer with number of bytes).

data_type: VDIF frame data type bit.

bits_per_sample: number of bits per sample.

thread_id: VDIF thread identifier field (integer).

station_id: VDIF station id field (integer).

lib_vdif.read_samples_from_raw(words, bits_per_sample, word_size=32)

Extract samples from unit32 words into numpy array of integers.

words : numpy array of np.unit32. bits_per_sample : int word_size : int

Limitations:

bits per sample should be a divisor of word_size.

lib_vdif.read_vdif_frame(f, show_errors=0, forced_frame_length=0, offset_bytes=0, v=0, return_raw_header=0)

Main routine for reading one VDIF frame.

f : file handler

input file handler (typically sys.stdin).

show_errors : int

[0 by default] display information on errors if 1.

forced_frame_length : int

[0 by default] number of bytes to read including header, if 0 will take value from header (recommended).

offset_bytes : int

[0 by default] number of bytes to skip before reading header.

v : int

[0 by default] verbosed mode if 1.

return_raw_header : int

[0 by default] if 0 returns a list of integers with the header fields, if 1 returns bytes.

header : list of int

list of integers with the fields in the VDIF header:

[seconds_fr,invalid,legacy,ref_epoch,frame_num,vdif_version,log_2_channels, | frame_length,data_type,bits_per_sample,thread_id,station_id].

See read_header_vdif_from_raw() for details.

If errors, header is None.

samples : 1D numpy array of int

sample components. E.g. for VDIF complex frame, [I0, Q0, I1, Q1, ...]

if errors, samples is None.

check_size_samples : int

1 if read as many samples as expected from the frame length (and rest of metadata), 0 otherwise.

Configuration:

HEADER_VDIF_WORDS

HEADER_BYTES

WORD_SIZE_BYTES

Procedure:

1. Read VDIF header.

2. Read rest of the frame based on frame length information in header.

lib_vdif.read_words_from_file_to_raw(f, n_words, v=0)

Read binary file directly into numpy array.

f : file handler

(sys.stdin).

n_words : int

number of words of type TYPE_WORD.

v : int

[0 by default] verbose mode if 1.

words_array : numpy array with number->n_words type->TYPE_WORD words.

Configuration:

TYPE_WORD: [np.uint32 by default] type of words to read.

lib_vdif.reshape_samples(allsamples, data_type, samples_in_frame, num_channels)

Corner-turning.

allsamples : 1D array

with all samples read from VDIF file.

data_type : int

0 for real, 1 for complex.

samples_in_frame : int

number of samples components per channel.

num_channels : int

number of channels.

samples : 2D array with num_channels rows, with each row containing only its associated samples.

Procedure:

# example

aa=list(range(0,20))

bb=np.reshape(aa,(-1,5,2))

cc=np.transpose(bb,(1,0,2))

dd=np.reshape(cc,(5,-1))

dd

lib_vdif.show_headers_vdif(filename, limit_frames=-1, skip_frames=-1, brief=0)

Get tabulated information from vdif file headers. The first “limit_frames” after “skip_frames” are read.

filename : str

path to file.

limit_frames : int

maximum number of lines to display (-1 for no limit).

skip_frames : int

number of frames to skip before starting displaying frames (default <=0).

brief : int

1 to avoid displaying descriptions for columns.

N/A

lib_vdif.strbitarray2int(a)

lib_vdif.vdif_epoch_seconds_to_epoch_seconds_datetime(epoch_six, tot_seconds)

Compute reference epoch and seconds from VDIF standard to MJD and seconds. Using datetime.

epoch_six : int

epoch directly from VDIF header.

tot_seconds : int

seconds directly from VDIF header.

epoch

epoch MJD.

seconds

epoch seconds.

lib_vdif.write_samples(samples, bits_per_sample, word_size=32)

Write vdif samples. Used by vdif signal generator.

lib_vdif.write_samples_raw(samples, bits_per_sample, word_size=32)

Write vdif samples.

Untested. Limited to bits_per_samples that are divisor of 8.

lib_vdif.write_words_to_file(f, words)

lib_vdif.write_words_to_file_raw(f, words)