Database access¶

The `tod_info` Table¶

The tod_info table stores lots of useful information about the merged TODs. To get some tod_info, run:

from moby2.instruments import actpol
db = actpol.TODDatabase()

db.get_record('1378762587.1378762626.ar1')

To see a summary of all the information in a TODRecord, use the to_string method:

print tod_record.to_string()

mce_data_acq_id : 6864L
array           : 'AR1'
mean_alt        : 50.0411
rms_alt         : 0.00013
min_az          : 276.919
max_az          : 281.514
scan_speed      : 1.32151
pwv             : None
min_dec         : -13.0941
max_dec         : -10.3552
min_RA_south    : 204.329
min_RA_north    : 205.45
max_RA_south    : 206.397
max_RA_north    : 207.518
ctime_start     : 1378762588L
ctime_end       : 1378763082L
obs_type        : 'planet'
obs_detail      : 'venus'
obs_drift       : 'setting'
malfunction     : None
datafile_id     : 20597L
basename        : '1378762587.1378762626.ar1'
ctime           : 1378762587L
merged_ctime    : 1378762626L

Each field listed is available as an attribute in the TODRecord object. All fields except for datafile_id, basename, ctime, and merged_ctime can be used for selecting TODs using keyword arguments to the select_tods function as described in the next section.

Selecting TODs¶

The select_tods method returns a TODRecordList of TODRecords.

# Get a list of TODRecord instances
tod_list = db.select_tods(obs_detail='venus')

# Print basename of last item
print tod_list[-1].basename

The TODRecordList is like a normal list, but an additional get_fields method returns an array of values for a desired attribute (or list of attributes):

ctime = tod_list.get_fields('ctime_start')
min_dec, max_dec = tod_list.get_fields(['min_dec', 'max_dec'])
print 'Venus mean declination: ', (min_dec + max_dec).mean() / 2

Keyword arguments passed to the select_tods method are interpreted as expressions to match in the tod_info table. In addition to simple matching to a single value:

If a list is passed, then records will be return that match any one of the values listed.
If a tuple is passed, then it should have a length two and will be interpreted as describing a range of values to match.

For ctime fields, you can pass in a date string like ‘2013-09-21 12:00:00’ instead of an integer or float and it will be converted automatically.

Examples:

# Get all TODs for deep fields 1 or 2
tod_list = db.select_tods(obs_detail=['deep1','deep2'])

# Get all planet TODs with 1234560000 <= ctime_start < 1234570000
tod_list = db.select_tods(obs_type='planet',
                          ctime_start=(1234560000, 1234570000))

# Get all stare TODs from Sep 15 UTC:
tod_list = db.select_tods(obs_type='planet',
                          ctime_start=('2013-09-15 00:00:00',
                                       '2013-09-16 00:00:00'))

Getting filenames¶

Datafile locations are stored in the ACTpol database.

from moby2.instruments import actpol
filebase = actpol.Filebase()

# Find the filename associated with my Venus observation
filename = filebase.filename_from_name('1378762587.1378762626.ar1', single=True)
print filename

The actpol.Filebase configuration can be passed in to the constructor, or provided in the .moby2 file with a block like:

# Config for actpol.Filebase
actpol_filebase = {
    'manifest_config': '/ACTpol/etc/manifest.conf',
    'nodes': [('merlin','raid')
              ('mce1','raid',None,'prefix5')],
}

# Use actpol.Filebase as the default filebase
filebase = {
    'type': 'actpol_manifest',
}

The second block is used by scripts to decide how to instantiate a default filebase.

Finding Calibration and Housekeeping Data¶

SQUID tuning, IV curve and bias step data from the MCE are not merged into dirfiles. These acquisitions are recorded in the mce_data_acq table of the database and the data are stored in the raw data archive.

To find calibration data, use select_acqs. The important things to tell it are the data suffix and array. You can also filter on ctime, as one would when using select_tods.

Popular values for suffix:

iv: IV curves. The IV analysis (extension .out) is included with the data.
bc1_step, bc2_step, bc3_step: Bias steps.
sqtune: Analysis block from SQUID setup.
RCs_ssa, RCs_sq2servo, RCs_sq1servo, RCs_sq1ramp, RCs_sq1rampc: SQUID tuning raw data.

Example:

from moby2.instruments import actpol
database = actpol.TODDatabase()
filebase = actpol.Filebase()

ctimes = ('2013-09-18 00:00:00', '2013-09-18 08:00:00')
ar = 'ar1'
recs = database.select_acqs(suffix='iv', ctime=ctimes,
                            array=ar)

print filebase.filename_from_name(recs[0].mce_name, array=ar)

Auto-doc¶

class moby2.instruments.actpol.TODRecord[source]¶

classmethod from_database(tdb, mce_id=None)[source]¶: Load TOD information from ACTpol manifest over database connection tdb, which must be an actpol.TODDatabase instance. mce_id should be the mce_data_acq_id, or basename.

to_string()[source]¶: Return textual summary of data in this record.

class moby2.instruments.actpol.TODDatabase(config_file=None, season=None)[source]¶

get_record(acq_id)[source]¶: Return a TODRecord for the acq_id specified, which must be either the mce_data_acq_id or a basename/filename.

select_acqs(fields=None, clauses=[], id_only=False, order=None, **kwargs)[source]¶

Find acquisitions in the mce_data_acq table. This handles kw_args in the same way as select_tods.

Returns a TODRecordList of AcquisitionRecord objects. If that’s too slow, get raw DB rows by passing id_only=True or passing a list of desired fields in fields.

Example:

tdb = TODDatabase()
recs = tdb.select_acqs(array='ar1', suffix='iv',
                       ctime=(1379703312, 1379789712))
prints recs[0].to_string()

select_hk(clauses=[], order=None, **kwargs)[source]¶

Find HK dirfiles in the merlin table. This handles kw_args in the same way as select_tods.

Example

tdb = TODDatabase() recs = tdb.select_hk(ctime=(1379703312, 1379789712)) prints recs[0].to_string()

select_tods(fields=None, clauses=[], id_only=False, order=None, **kwargs)[source]¶

Select records from ACTpol TOD database. Returns either a list of records, or a list of the mce_data_acq_ids if id_only=True.

If you are only interested in certain fields from the tod_info table, pass fields=[…]. This bypasses the individual record queries that get the basename, etc. It’s way faster. In this case the raw DB row data are returned.

To select records using tod_info fields, pass in strings in the “clauses” list, or use kwargs to specify values or ranges for particular tod_info fields.

E.g.:

db = actpol.TODDatabase()

# Match some values exactly:

tods = db.select_tods(array=’AR1’, obs_detail=’saturn’)

# Match a range of values – pass a tuple:

tods = db.select_tods(ctime_start=(1234560000,1234570000))

# Match any from a list of values – pass a list:

tods = db.select_tods(obs_detail=[‘venus’, ‘saturn’])

class moby2.instruments.actpol.Filebase(manifest_config=None, nodes=None, db=None)[source]¶

Provide filenames for merged TODs in simple colossus storage.

Requires that a connection be made to the ACTpol manifest database.

filename_from_id(id_, single=False)[source]¶: Same behavior as filename_from_name, except that instead of a basename the user should pass in the datafile_id.

filename_from_name(name, single=False, array=None)[source]¶

Find files on present storage nodes and return a list of full paths. This works for TODs, MCE data, and merged housekeeping. “name” is the either:

TOD basename, in the form <ctime>.<ctime>.<ar>
MCE filename, <ctime>_suffix[extension]
Housekeeping basename, <ctime>.<ctime>.hk

For MCE filenames, the “array” must also be specified.

If single=True then instead of a list, one of the filenames is returned, or None is returned if there were no matches.