Metadata Browser

Irena heavily relies on use of wave notes in Igor to store metadata for both data - on import, for example, Irena records which data were imported, where the file was located, how the data were processed and modified etc. Every time some processing is done in Irena tools, more metadata are added to wave note. As result, there is basically giant pile of information which is really difficult to read and understand. Igor provides rudimentary way to look inside the wave notes using DataBrowser. By selecting a wave in DataBrowser and making sure checkbox “info” is selected, one can see the wave note.

Irena will format its records in the wave notes as keyword=value; list. Thanks to this clear definition fo format, Irena can provide relatively easy to use interface Metadata Browser for users to extract the data from wave notes.

Irena keywords are hopefully understandable for humans. They are picked as reasonably as possible and generally should be possible (hopefully) to understand.

• When importing data Irena uses names which seem logical and related to step when used (date, file name,…).

• If metadata from the data container (header, Nexus metadata,…) are used, same names which were in the container are used. These are often least understandable…

• When Irena (or Nika) tool generates its own metadata, it uses its internal names of the tool, which are hopefully meaningful for those who understand the tool use.

There is NO list of keywords available anywhere You will need to figure it out or get in touch with me and I can help you. Sorry, too many options. Also, there is no guarantee metadata keywords are unique, which can cause issue. If you find conflicts, let me know and I will change the keywords somewhere. And finally, there is no guarantee a specific number is actually saved. If you find something missing, let me know and I will add it.

Metadata Browser GUI organization

If you look at the Figure with MetadataBrowser GUI, you can see that is divided in few parts:

• Data Controls & Selection - Left 1/3 column of the GUI, from top to bottom, is part dedicated to data type selections and controls. There are few controls for data type selection and left vertical listbox which lists available data in the current Igor experiment, which match the controls selection. Here user can select which data sets will be probed when processing (extracting, mining for the data).

• Metadata Selection This is middle vertical listbox (with few controls above it) and here user can pick names of metadata.

• List of metadata to process this is right most listbox and contains names of metadata, which will be extracted from the selected folders.

• Controls at the bottom right which are used to process selected folders, display table of extracted results. It also allows users to plot selected results or show the results in the Igor folder.

Data Controls & Selection

Left top side of the panel is Data selection. In the Data selection you need to define sufficiently the data you want to look inside. There is detailed description on how to use this widget system Multi Data selection. Please refer to that page for details.

Data type Irena recognizes few data types directly:

• USAXS data type = this is naming system for data generated by APS USAXS instrument. Ignore, unless you have data from this instrument. If you have our data, you should know enough to use this.

• QRS data type = this is the default data naming system for SAXS/WAXS data in Irena and Nika. For details see here QRS data type.

• Irena results = any fit and modeling results generated by Irena. Most tools will save some type of data - size distribution, fits, pddf, diffraction peaks,… All of these data types can be seen as “Irena results”

• Any = if all checkboxes are unchecked, user can define Regular expression, which will tell irena which wave is x, y, and optionally error. Keep in mind, that the first wave matching the regular expression will be picked. This may require some testing or help from me, if you want to use it.

Start Fldr Here you can select at which location in data tree code will start looking for the data. Pick suitable place, for example root:SAXS may be a good start. Picking suitable start where to look for data makes the code run faster.

Folder Match (RegEx) this allows users to look for only some of the folders. A short summary on regular expressions is at the bottom of the page, below the Listbox with folder. Google it, understanding regular expressions will be very helpful.

Invert? this checkbox inverts the Regular expression meaning. So if you insert in the “Folder Match” field string 00034, only data which have in name 00034 will show. If you check this checkbox, selection is inverted and all files which do NOT contain this string in the name will show.

Sort Folders This sorts the folders using one of many methods implemented. As result, this will group folders in order which may be helpful for processing. For example, some tools create list of results in the order the samples were processed. Having proper order helps plotting results after the analysis properly.

Metadata selection

This is the middle column. The top above the listbox shows in red which data name is currently selected. Clicking on any name in the left Data Selection Listbox will change the name here. At that time code will also extract wave note and fill the middle list box with the wave note, one pair of keyword=value on each line. User can now look for the metadata which he/she wants.

Since the list is surely long and it is difficult to find what one wants, one can use Regex Key name - regular expression which is used to reduce which Keywords are displayed.

Double clicking on the Keyword=value string in the listbox adds the Keyword to the listbox on the right hand side, which lists which keywords will code look for and extract into folder as waves.

Folder name of the data is always included in the list to provide reliable way to assign extracted values to the data source.

In case the Folder name contains useful information - for example it has _xyzC in the name, Pull down menu Extract From Folder Name will enable user to extract the values in separate wave.

To remove keyword which you do not want to process, you can double click that name in the right listbox or to remove all, use the button below the listbox.

In the figure above, we are looking for Unified fit results, specifically in the wave called UnifiedFitIntensity, in its note. If there would be multiple generations of results we would pick the most recent one (the highest order number). Metadata are listed in the middle listbox - BUT only those which contain anywhere in their name string “Level1”. By double clicking on Level1B line in the middle Listbox user adds this keyword to those which will be extracted from data selected in the left side Listbox. I also added extraction of temperature from the folder name. Since the name contains a common way of recording temperature. Optionally, if this temperature would be written in the metadata, it can be extracted as usual metadata value.

Metadata extraction and utilization

Now, that I have decided what to extract, I can select the data to process. Shift-click will select range of data, ctrl/cmd click enables select/deselect one line etc. I selected all except the first one which is measurement without temperature. Result is in next image…

The run finished and code created table of results. These are waves, which are stored in folder with named in Save to: field on the panel. Note, the folder will be created, but the name must be acceptable for Igor as Folder name. Also, you need to start with root: and add a name which you wish to use. In this case folder contains waves named: FolderNameWv, Level1B, and TemperatureWv. Temperature wave was created by extracting _xyzC from Folder name assuming it is temperature in degrees C. FolderNameWv is created always (and it is text wave). Level1B is Keyword name from the wave note. If the value is number, the wave will be numerical wave. If value is string, wave will be string and if the string is one of few recognized date/time representations, you will get wave with date-time converted to Igor time (seconds since some day in 1972 or whatever).

It is possible to generate easily plot of extracted data, use X: and Y: popups and select what will be X and Y axis. Push the large button Plot Selected. Next image shows plot of Level1B (aka Porod constant if P=4) against Temperature.

Last feature is ability to Delete old results. This button will try to close graph and table using the results and delete folder which is in Save To: field. Alternatively, same can be achieved by using DataBrowser from Igor Pro and deleting the folder form there, manually closing all tables and graphs using data from teat folder first.