Custom SAXS/WAXS tasks#

How to use

These tasks come from ewoksbm29 ewoksid02 ewoksid11 ewoksid22 ewoksid31 ewoksoptics. It can be installed with

pip install ewoksbm29 ewoksid02 ewoksid11 ewoksid22 ewoksid31 ewoksoptics

ℹ️ These tasks are used at the following ESRF beamlines: BM29, ID02, ID06, ID11, ID22, ID31

DahuHplcSummary#

Rebuild the complete chromatogram with basic analysis.

In addition to the inputs from DahuJobWithIspybUpload:

Required inputs:

  • integrated_files (List[str]): Azimuthally integrated SAXS data.

Optional inputs:

  • hplc_summary_parameters (dict): Extra subtract parameters (see HplcSummaryParameters).

Identifier:
ewoksbm29.tasks.hplc_summary.DahuHplcSummary
Task type:
class
Inputs:
integrated_files*
config_directory
dahu_job_index
dahu_parameter_file
dahu_url
extra_dahu_parameters
hplc_summary_parameters
ispyb_metadata
ispyb_url
nobackup
timeout
Outputs:
dahu_job_id
dahu_job_index
dahu_result

DahuIntegrate#

Azimuthal integration of BM29 SAXS data.

In addition to the inputs from DahuJobWithIspybUpload:

Required inputs:

  • scan_data_slice (dict): Subset of parameters related to the scan data (see IntegrateParameters).

Optional inputs:

  • integrate_parameters (dict): Extra integrate parameters (see IntegrateParameters).

Identifier:
ewoksbm29.tasks.integrate.DahuIntegrate
Task type:
class
Inputs:
scan_data_slice*
config_directory
dahu_job_index
dahu_parameter_file
dahu_url
extra_dahu_parameters
integrate_parameters
ispyb_metadata
ispyb_url
nobackup
timeout
Outputs:
dahu_job_id
dahu_job_index
dahu_result

ReadScanDataSlice#

Return a subset of IntegrateParameters for a SAXS scan index range covered by the lima file with index lima_file_index.

Optional inputs (online & offline):

  • lima_file_index (int): Lima file index that determines the scan slice.

  • dahu_to_counter_name (dict): Map counter names to IntegrateParameters keys.

  • storage_ring_current (float): Storage ring current in mA.

  • exposure_time (float): Scan point exposure time in s.

Optional inputs (online):

  • scan_key (str): Blissdata scan key (tries offline when fails).

  • retry_timeout (float): Timeout when trying to access the Lima image dataset or wait for the scan to be PREPARED.

  • retry_period (float): Period in retry loops.

Optional inputs (offline):

  • scan_file_path (str): Bliss scan file name.

  • scan_number (str): Bliss scan number.

Outputs:

  • scan_data_slice (dict): Subset of IntegrateParameters related to the scan data.

  • has_data (bool): scan_data_slice contains a non-empty scan data slice.

  • lima_file_index (int): The lime file index to which scan_data_slice belongs.

  • next_lima_file_index (int): The next Lima file index.

Identifier:
ewoksbm29.tasks.read.ReadScanDataSlice
Task type:
class
Inputs:
dahu_to_counter_name
exposure_time
lima_file_index
retry_period
retry_timeout
scan_file_path
scan_key
scan_number
storage_ring_current
Outputs:
has_data
lima_file_index
next_lima_file_index
scan_data_slice

AccumulateDahuJobResults#

Accumulate Dahu job results with the job index as a key.

.. code: python

dahu_results[dahu_job_index] = {"dahu_job_id": dahu_job_id, "dahu_result": dahu_result}

Required inputs:

  • dahu_job_index (int): Dahu job index for ordering results.

  • dahu_job_id (Optional[int]): Dahu job id (None means no Dahu job was executed).

  • dahu_result (Optional[dict]): Dahu result (None means no Dahu job was executed).

Outputs:

  • dahu_results (Dict[int, dict]): Add the Dahu result to results from previous executions within the same workflow.

Identifier:
ewoksbm29.tasks.results.AccumulateDahuJobResults
Task type:
class
Inputs:
dahu_job_id*
dahu_job_index*
dahu_result*
Outputs:
dahu_results

DahuSubtract#

Subtract average integrated buffer signal from the sample signal.

In addition to the inputs from DahuJobWithIspybUpload:

Required inputs:

  • sample_file (str): Azimuthally integrated SAXS data.

  • buffer_files (List[str]): Azimuthally integrated SAXS data of the buffer.

Optional inputs:

  • subtract_parameters (dict): Extra subtract parameters (see SubtractParameters).

Identifier:
ewoksbm29.tasks.subtract.DahuSubtract
Task type:
class
Inputs:
buffer_files*
sample_file*
config_directory
dahu_job_index
dahu_parameter_file
dahu_url
extra_dahu_parameters
ispyb_metadata
ispyb_url
nobackup
subtract_parameters
timeout
Outputs:
dahu_job_id
dahu_job_index
dahu_result

AverageTask#

The AverageTask class is responsible for calculating the average of datasets in the ID02 SAXS pipeline. It extends the ID02ProcessingTask class and provides additional functionality for handling averaging-specific inputs and processing logic. If azimuth_range is not provided, a full average will be performed.

Optional Inputs: - dataset_sum_signal (numpy.ndarray): Sum of signal, non-normalized from an ai.integrate2d result. - dataset_sum_normalization (numpy.ndarray): Sum of normalized pixels, from an ai.integrate2d result. - dataset_sum_variance (numpy.ndarray): Sum of variance, from an ai.integrate2d result. - radial_array (numpy.ndarray): Radial axis array for the dataset. - azimuth_array (numpy.ndarray): Azimuthal axis array for the dataset. - Dummy (float): Value to replace invalid pixels in the dataset. - unit (str): Unit for the radial axis (e.g., “q_nm^-1”). - azimuth_range (list of tuples): Azimuthal ranges for averaging (e.g., [(0, 90), (270, 360)]). - pca_parameters (dict): Parameters for Principal Component Analysis (PCA) if applicable. Outputs Parameters: - dataset_average_signal_norm (numpy.ndarray): Normalized average signal dataset. - radial_array (numpy.ndarray): Radial axis array for the averaged data.

Identifier:
ewoksid02.tasks.averagetask.AverageTask
Task type:
class
Inputs:
Dummy
azimuth_array
azimuth_range
beacon_host
dataset_sigma
dataset_signal
dataset_sum_normalization
dataset_sum_signal
dataset_sum_variance
dataset_variance
datatype
detector_name
do_process
do_save
filename_data
filename_lima
gc_collect
headers
index_range
index_range_last
info
info_history
lima_index_number_format
lima_url_template
lima_url_template_args
log_level
loop_nb
max_slice_size
pca_parameters
processing_filename
processing_subtitle
radial_array
reading_node
save_metadata
save_sigma
save_variance
scan_memory_url
scan_nb
streams_subscan1
streams_subscan2
subtitle
unit
Outputs:
continue_pipeline
dataset_average_signal_norm
dataset_sigma
dataset_signal
dataset_variance
detector_name
filename_data
filename_lima
headers
index_range_last
info_history
loop_nb
radial_array
scan_nb
streams_subscan1
streams_subscan2

AzimuthalTask#

The AzimuthalTask class is responsible for performing azimuthal integration on datasets in the ID02 SAXS pipeline. It extends the ID02ProcessingTask class and provides additional functionality for handling azimuthal integration-specific inputs and processing logic using the pyFAI library.

Optional Inputs: - filename_mask_azimuthal (str): Path to the mask file for masking invalid pixels. - filename_dark_azimuthal (str): Path to the file with a dark-current correction. - npt2_rad (int): Number of radial bins for the integration. - npt2_azim (int): Number of azimuthal bins for the integration. - unit (str): Unit for the radial axis (e.g., “q_nm^-1”). - Dummy (float): Value to replace invalid pixels in the dataset. - DDummy (float): Tolerance for dummy pixel replacement. - Center_1 (float): Beam center in the first dimension. - Center_2 (float): Beam center in the second dimension. - PSize_1 (float): Pixel size 1. - PSize_2 (float): Pixel size 2. - BSize_1 (float): Pixel binning 1. - BSize_2 (float): Pixel binning 2. - SampleDistance (float): Sample to detector distance in meters. - WaveLength (float): Wavelength of beam in meters. - DetectorRotation_1 (float): rot2 of pyFAI. - DetectorRotation_2 (float): rot1 of pyFAI. - DetectorRotation_3 (float): rot3 of pyFAI. - method (str): Integration method to be used by pyFAI (e.g., “bbox”, “csr”). - integration_options (dict): Additional options for pyFAI integration. - do_variance_formula (bool): Flag to enable variance calculation using a formula. Default is False. - variance_formula (str): Formula for calculating variance in the dataset. - save_sum (bool): To save or not the arrays sum_signal, sum_normalization and sum_variance. Outputs: - dataset_sum_signal (numpy.ndarray): Summed signal dataset after azimuthal integration. - dataset_sum_normalization (numpy.ndarray): Summed normalization dataset after azimuthal integration. - dataset_sum_variance (numpy.ndarray): Summed variance dataset after azimuthal integration. - radial_array (numpy.ndarray): Radial axis array for the integrated data. - azimuth_array (numpy.ndarray): Azimuthal axis array for the integrated data.

Identifier:
ewoksid02.tasks.azimuthaltask.AzimuthalTask
Task type:
class
Inputs:
BSize_1
BSize_2
Center_1
Center_2
DDummy
DetectorRotation_1
DetectorRotation_2
DetectorRotation_3
Dummy
PSize_1
PSize_2
SampleDistance
WaveLength
beacon_host
dataset_sigma
dataset_signal
dataset_variance
datatype
detector_name
do_process
do_save
do_variance_formula
filename_dark_azimuthal
filename_data
filename_lima
filename_mask_azimuthal
gc_collect
headers
index_range
index_range_last
info
info_history
integration_options
lima_index_number_format
lima_url_template
lima_url_template_args
log_level
loop_nb
max_slice_size
method
npt2_azim
npt2_rad
processing_filename
processing_subtitle
reading_node
save_metadata
save_sigma
save_sum
save_variance
scan_memory_url
scan_nb
streams_subscan1
streams_subscan2
subtitle
unit
variance_formula
Outputs:
azimuth_array
continue_pipeline
dataset_sigma
dataset_signal
dataset_sum_normalization
dataset_sum_signal
dataset_sum_variance
dataset_variance
detector_name
filename_data
filename_lima
headers
index_range_last
info_history
loop_nb
radial_array
scan_nb
streams_subscan1
streams_subscan2

CavingTask#

The CavingTask class is responsible for applying a “caving” operation to datasets in the ID02 SAXS pipeline. The caving is applied to those pixels whose intensity matches with a dummy value + those pixels provided by a mask file.

Optional Inputs: - Dummy (float): Value to replace invalid pixels in the dataset. - Center_1 (float): Beam center in the first dimension. - Center_2 (float): Beam center in the second dimension. - filename_mask_static (str): Path to the mask file used for the caving operation. - filename_mask_reference (str): Path to the reference mask file (kind of a negative mask). - flip_caving (bool): Cave the image with its flipped projection, both horizontal and vertical. WARNING: it is physically not correct! - algorithm_cave (str): Implementation to perform the caving (numpy or cupy).

Identifier:
ewoksid02.tasks.cavingtask.CavingTask
Task type:
class
Inputs:
Center_1
Center_2
Dummy
algorithm_cave
beacon_host
dataset_sigma
dataset_signal
dataset_variance
datatype
detector_name
do_process
do_save
filename_data
filename_lima
filename_mask_reference
filename_mask_static
flip_caving
gc_collect
headers
index_range
index_range_last
info
info_history
lima_index_number_format
lima_url_template
lima_url_template_args
log_level
loop_nb
max_slice_size
processing_filename
processing_subtitle
reading_node
save_metadata
save_sigma
save_variance
scan_memory_url
scan_nb
streams_subscan1
streams_subscan2
subtitle
Outputs:
continue_pipeline
dataset_sigma
dataset_signal
dataset_variance
detector_name
filename_data
filename_lima
headers
index_range_last
info_history
loop_nb
scan_nb
streams_subscan1
streams_subscan2

CavingGapsTask#

The CavingGapsTask inherits ‘CavingTask’. Here, it will take the parameters from the headers to perform the caving operation and cover the gaps in the dataset and avoid the beamstop. Hence, no explicit mask filenames are needed.

Identifier:
ewoksid02.tasks.cavingtask.CavingGapsTask
Task type:
class
Inputs:
Center_1
Center_2
Dummy
algorithm_cave
beacon_host
dataset_sigma
dataset_signal
dataset_variance
datatype
detector_name
do_process
do_save
filename_data
filename_lima
filename_mask_reference
filename_mask_static
flip_caving
gc_collect
headers
index_range
index_range_last
info
info_history
lima_index_number_format
lima_url_template
lima_url_template_args
log_level
loop_nb
max_slice_size
processing_filename
processing_subtitle
reading_node
save_metadata
save_sigma
save_variance
scan_memory_url
scan_nb
streams_subscan1
streams_subscan2
subtitle
Outputs:
continue_pipeline
dataset_sigma
dataset_signal
dataset_variance
detector_name
filename_data
filename_lima
headers
index_range_last
info_history
loop_nb
scan_nb
streams_subscan1
streams_subscan2

CavingBeamstopTask#

The CavingBeamstopTask inherits ‘CavingTask’. Here, it will take the parameters from the headers to perform the caving operation and cover the beamstop. Hence, no explicit mask filename is needed.

Identifier:
ewoksid02.tasks.cavingtask.CavingBeamstopTask
Task type:
class
Inputs:
Center_1
Center_2
Dummy
algorithm_cave
beacon_host
dataset_sigma
dataset_signal
dataset_variance
datatype
detector_name
do_process
do_save
filename_data
filename_lima
filename_mask_reference
filename_mask_static
flip_caving
gc_collect
headers
index_range
index_range_last
info
info_history
lima_index_number_format
lima_url_template
lima_url_template_args
log_level
loop_nb
max_slice_size
processing_filename
processing_subtitle
reading_node
save_metadata
save_sigma
save_variance
scan_memory_url
scan_nb
streams_subscan1
streams_subscan2
subtitle
Outputs:
continue_pipeline
dataset_sigma
dataset_signal
dataset_variance
detector_name
filename_data
filename_lima
headers
index_range_last
info_history
loop_nb
scan_nb
streams_subscan1
streams_subscan2

SingleDetector#

Trigger DAHU in the workflow.

Identifier:
ewoksid02.tasks.dahuprocessingtask.SingleDetector
Task type:
class
Inputs:
parameters*
Outputs:
files
logging
runtime

EndTask#

Identifier:
ewoksid02.tasks.end.EndTask
Task type:
class
Inputs:
detector_name
filename_data
index_range_last
loop_nb
scan_memory_url
scan_nb
Outputs:
detector_name
filename_data
index_range_last
loop_nb
scan_memory_url
scan_nb

SaveGalleryTask#

Identifier:
ewoksid02.tasks.gallerytask.SaveGalleryTask
Task type:
class
Inputs:
nxdata_url*
do_save
Outputs:
filename_png

ID02ProcessingTask#

This class contains processing support methods and saving methods in the ID02 SAXS pipeline. It extends the ID02LoopTask class and provides additional functionality for handling metadata, processing flags, and saving processed data to HDF5 files.This class is designed to be used as part of the ID02 pipeline.It does not contain a process method, that has to be implemented in the child class.

Optional Inputs: - detector_name (str): Name of the detector used for data acquisition. This is the only mandatory input. - scan_memory_url (str): URL for accessing scan memory in online processing. - beacon_host (str): Host and port to plug blissdata to the correct beacon server. Only for online processing. - reading_node (bool): Flag to indicate if the task should read data from the node. - filename_data (str): Path to the dataset file (Master file, Nexus writer) for offline processing. - filename_lima (str): Path to the first Lima file, the only place where some detector metadata can be found. - scan_nb (int): Scan number for identifying the dataset. - headers (dict): Only for Online processing. Dictionary containing headers information. - max_slice_size (int): Maximum number of frames to process in one iteration. Default is 20. - dataset_signal (numpy.ndarray): Signal dataset to be processed. - dataset_variance (numpy.ndarray): Variance dataset to be processed. - dataset_sigma (numpy.ndarray): Sigma dataset to be processed. - datatype (str): Datatype to be used to save the 2D data. Default and recommended is float32. - lima_url_template (str): Format string to locate the Lima file and the path to the data inside that file. - lima_url_template_args (dict): Dictionary to format the lima_url_template. - log_level (str): Logging level for the task. Default is "warning". - processing_filename (str): Full path to the (new) output file. - processing_subtitle (str): Additional subtitle for the processing task. - subtitle (str): Subtitle for the processing task to be added to the output filename. - do_process (bool): Flag to enable or disable processing. Default is True. - do_save (bool): Flag to enable or disable saving of processed data. Default is True. - save_variance (bool): Flag to enable or disable saving of variance dataset. Default is False. - save_sigma (bool): Flag to enable or disable saving of sigma dataset. Default is True. - save_metadata (bool): Flag to enable or disable saving of metadata. Default is True. - last_index_read (int): Index of the last frame read in the dataset. Default is 0. - range_index_read (list): Range of indices to read from the dataset. This parameter is not propagated to the next task. - loop_nb (int): Current loop iteration number. Default is 0. - info (dict): Additional metadata to save. - info_history (dict): Additional metadata to propagate and save, creating a history of processing. - gc_collect (bool): Manually collect garbage at the end of every task. - lima_index_number_format (str): format to find the first Lima file (02%d by default) Outputs: - last_index_read (int): Updated index of the last frame read. - loop_nb (int): Updated loop iteration number. - dataset_signal (numpy.ndarray): Processed signal dataset. - dataset_variance (numpy.ndarray): Processed variance dataset. - dataset_sigma (numpy.ndarray): Processed sigma dataset. - continue_pipeline (bool): Flag to indicate whether the pipeline should continue. - info_history (dict): Additional metadata to propagate and save, creating a history of processing.

Identifier:
ewoksid02.tasks.id02processingtask.ID02ProcessingTask
Task type:
class
Inputs:
beacon_host
dataset_sigma
dataset_signal
dataset_variance
datatype
detector_name
do_process
do_save
filename_data
filename_lima
gc_collect
headers
index_range
index_range_last
info
info_history
lima_index_number_format
lima_url_template
lima_url_template_args
log_level
loop_nb
max_slice_size
processing_filename
processing_subtitle
reading_node
save_metadata
save_sigma
save_variance
scan_memory_url
scan_nb
streams_subscan1
streams_subscan2
subtitle
Outputs:
continue_pipeline
dataset_sigma
dataset_signal
dataset_variance
detector_name
filename_data
filename_lima
headers
index_range_last
info_history
loop_nb
scan_nb
streams_subscan1
streams_subscan2

ID02ProcessingTask#

The ID02LoopTask class is a base task designed to handle iterative data processing in the ID02 SAXS pipeline. It provides functionality for reading datasets, managing processing loops, and controlling the pipeline flow. This class is intended to be extended by more specific tasks, such as ID02ProcessingTask. This class could be seen also a Reading node.

Inputs: - detector_name (str): Name of the detector used for data acquisition. This is the only mandatory input. Optional Inputs: - scan_memory_url (str): URL for accessing scan memory in online processing. - filename_data (str): Path to the dataset file (Master file, Nexus writer) for offline processing. - scan_nb (int): Scan number for identifying the dataset. - subscan (int): Subscan number for processing. Default is 1. - max_slice_size (int): Maximum number of frames to process in one iteration. Default is 20. - last_index_read (int): Index of the last frame read in the dataset. Default is 0. - range_index_read (list): Range of indices to read from the dataset. This parameter is not propagated to the next task. - loop_nb (int): Current loop iteration number. Default is 0. - dataset_signal (numpy.ndarray): Signal dataset to be processed. - dataset_variance (numpy.ndarray): Variance dataset to be processed. - dataset_sigma (numpy.ndarray): Sigma dataset to be processed. - reading_node (bool): Flag to indicate if the task should read data from the node. - lima_url_template (str): Format string to locate the Lima file and the path to the data inside that file. - lima_url_template_args (dict): Dictionary to format the lima_url_template. - beacon_host (str): Host and port to plug blissdata to the correct beacon server. Only for online processing. - log_level (str): Logging level for the task. Default is "warning". - info (dict): Additional metadata to save. - info_history (dict): Additional metadata to propagate and save, creating a history of processing. Outputs: - last_index_read (int): Updated index of the last frame read. - loop_nb (int): Updated loop iteration number. - dataset_signal (numpy.ndarray): Processed signal dataset. - dataset_variance (numpy.ndarray): Processed variance dataset. - dataset_sigma (numpy.ndarray): Processed sigma dataset. - continue_pipeline (bool): Flag to indicate whether the pipeline should continue. - info_history (dict): Additional metadata to propagate and save, creating a history of processing.

Usage:

This class is intended to be used as part of a larger pipeline for SAXS data processing. It handles the reading and propagation of data:

  • If ‘reading_node’ is set to True, it will try to get data from Blissdata (online processing) or from a static file (offline processing).

  • If ‘reading_node’ is set to False (default) and it receives a dataset, it will just propagate the dataset to the next task. Since the SAXS pipeline is a loop, if reading_node to True, the task will become a kind of entry-exit valve for the pipeline.

Identifier:
ewoksid02.tasks.looptask.ID02ProcessingTask
Task type:
class
Inputs:
detector_name*
beacon_host
dataset_sigma
dataset_signal
dataset_variance
datatype
do_process
do_save
filename_data
filename_lima
force_saving
headers
info
info_history
last_index_read
lima_url_template
lima_url_template_args
log_level
loop_nb
max_slice_size
processing_filename
processing_subtitle
range_index_read
reading_node
save_metadata
save_sigma
save_variance
scan_memory_url
scan_nb
slow_counters
subscan
subtitle
Outputs:
continue_pipeline
dataset_sigma
dataset_signal
dataset_variance
info_history
last_index_read
loop_nb

LoopTask#

The LoopTask class is a base task designed to handle iterative data processing. It provides functionality for reading datasets, managing processing loops, and controlling the pipeline flow. This class could be seen also a Reading node.

Optional Inputs: - detector_name (str): Name of the detector used for data acquisition. This is the only mandatory input. - scan_memory_url (str): URL for accessing scan memory in online processing. - filename_data (str): Path to the dataset file (Master file, Nexus writer) for offline processing. - scan_nb (int): Scan number for identifying the dataset. - subscan (int): Subscan number for processing. Default is 1. - max_slice_size (int): Maximum number of frames to process in one iteration. Default is 20. - last_index_read (int): Index of the last frame read in the dataset. Default is 0. - range_index_read (list): Range of indices to read from the dataset. This parameter is not propagated to the next task. - loop_nb (int): Current loop iteration number. Default is 0. - dataset (numpy.ndarray): Signal dataset to be processed. - reading_node (bool): Flag to indicate if the task should read data from the node. - lima_url_template (str): Format string to locate the Lima file and the path to the data inside that file. - lima_url_template_args (dict): Dictionary to format the lima_url_template. - beacon_host (str): Host and port to plug blissdata to the correct beacon server. Only for online processing. Outputs: - last_index_read (int): Updated index of the last frame read. - loop_nb (int): Updated loop iteration number. - dataset (numpy.ndarray): Processed signal dataset. - continue_pipeline (bool): Flag to indicate whether the pipeline should continue.

Identifier:
ewoksid02.tasks.looptask.LoopTask
Task type:
class
Inputs:
beacon_host
dataset
detector_name
filename_data
last_index_read
lima_url_template
lima_url_template_args
loop_nb
max_slice_size
range_index_read
reading_node
scan_memory_url
scan_nb
subscan
Outputs:
continue_pipeline
dataset
last_index_read
loop_nb

NormalizationTask#

The NormalizationTask class is responsible for normalizing datasets in the ID02 SAXS pipeline. It extends the ID02ProcessingTask class and provides additional functionality to apply a standard pyFAI normalization:

  • Methods to read monitor values from the metadata file or from blissdata

  • Methods to read normalization parameters from the metadata file or from the headers

  • Methods to cache pyFAI azimuthal integrator and apply normalization

  • Applies corrections such as masking, dark frame subtraction, flat field correction, and polarization adjustments.

Optional Inputs: - filename_mask_normalization (str): Path to the mask file for correcting detector gaps or bad pixels. - filename_dark_normalization (str): Path to the file for dark current correction. - filename_flat_normalization (str): Path to the file for flat field correction. - Dummy (float): Value to replace invalid pixels in the dataset. - DDummy (float): Tolerance for dummy pixel replacement. - NormalizationFactor (float): Factor for normalizing the dataset. - polarization_factor (float): Factor for polarization correction. - polarization_axis_offset (float): Axis for polarization correction. - Center_1 (float): Beam center in the first dimension. - Center_2 (float): Beam center in the second dimension. - PSize_1 (float): Pixel size 1. - PSize_2 (float): Pixel size 2. - BSize_1 (float): Pixel binning 1. - BSize_2 (float): Pixel binning 2. - SampleDistance (float): Sample to detector distance in meters. - WaveLength (float): Wavelength of beam in meters. - DetectorRotation_1 (float): rot2 of pyFAI. - DetectorRotation_2 (float): rot1 of pyFAI. - DetectorRotation_3 (float): rot3 of pyFAI. - pin_monitor (str): Pin to the monitor stream. - header_pin_monitor (str): Header key used to monitor values. - variance_formula (str): Formula for calculating variance in the dataset. - algorithm_normalization (str): Implementation to perform the normalization (cython or cupy). - dark_filter (str): Filter to use in case the dark-current file is a multi-frame scan file. - dark_filter_quantil_lower (str): In case the dark_filter is quantil, lower limit. - dark_filter_quantil_upper (str): In case the dark_filter is quantil, upper limit.

Identifier:
ewoksid02.tasks.normalizationtask.NormalizationTask
Task type:
class
Inputs:
BSize_1
BSize_2
Center_1
Center_2
DDummy
DetectorRotation_1
DetectorRotation_2
DetectorRotation_3
Dummy
NormalizationFactor
PSize_1
PSize_2
SampleDistance
WaveLength
algorithm_normalization
beacon_host
dark_filter
dark_filter_quantil_lower
dark_filter_quantil_upper
dataset_sigma
dataset_signal
dataset_variance
datatype
detector_name
do_process
do_save
filename_dark_normalization
filename_data
filename_flat_normalization
filename_lima
filename_mask_normalization
gc_collect
header_pin_monitor
headers
index_range
index_range_last
info
info_history
lima_index_number_format
lima_url_template
lima_url_template_args
log_level
loop_nb
max_slice_size
pin_monitor
polarization_axis_offset
polarization_factor
processing_filename
processing_subtitle
reading_node
save_metadata
save_sigma
save_variance
scan_memory_url
scan_nb
streams_subscan1
streams_subscan2
subtitle
variance_formula
Outputs:
continue_pipeline
dataset_sigma
dataset_signal
dataset_variance
detector_name
filename_data
filename_lima
headers
index_range_last
info_history
loop_nb
scan_nb
streams_subscan1
streams_subscan2

ScalersTask#

The ScalersTask class is responsible for creating a scalers file, that contains a large chunk of the metadata for reprocessing. It extends the ID02ProcessingTask class and provides additional functionality to apply a standard pyFAI normalization:

  • Methods to read monitor values from the metadata file or from blissdata

  • Methods to read normalization parameters from the metadata file or from the headers

  • Methods to cache pyFAI azimuthal integrator and apply normalization

Identifier:
ewoksid02.tasks.scalerstask.ScalersTask
Task type:
class
Inputs:
beacon_host
dataset_sigma
dataset_signal
dataset_variance
datatype
detector_name
do_process
do_save
filename_data
filename_lima
gc_collect
headers
index_range
index_range_last
info
info_history
lima_index_number_format
lima_url_template
lima_url_template_args
log_level
loop_nb
max_slice_size
processing_filename
processing_subtitle
reading_node
save_metadata
save_sigma
save_variance
scan_memory_url
scan_nb
streams_subscan1
streams_subscan2
subtitle
Outputs:
continue_pipeline
dataset_sigma
dataset_signal
dataset_variance
detector_name
filename_data
filename_lima
headers
index_range_last
info_history
loop_nb
scan_nb
streams_subscan1
streams_subscan2

SecondaryScatteringTask#

The SecondaryScatteringTask class is responsible for calculating and correcting the scattering coming from the window that separates the wagon from the flying tube. It can be used to correct any scattering source that is close to the detector.

Optional Inputs: - filename_window_wagon (str): Path to the mask file used for defining the scattering window WAXS pattern. - WindowRoiSize (float): Distance parameter for subdata extraction during secondary scattering correction. - Dummy (float): Value to perform a pre-caving step (to mask the detector gaps) - Center_1 (float): Beam center in the first dimension. - Center_2 (float): Beam center in the second dimension. - algorithm_2scat (str): Implementation to perform the secondary scattering correction and pre-caving (numpy or cupy). - pre_caving (bool): To perform a caving step before the correction. - filename_mask_static (str): Path to the mask file used for the caving operation. - filename_mask_reference (str): Path to the reference mask file (kind of a negative mask). - flip_caving (bool): Cave the image with its flipped projection, both horizontal and vertical. WARNING: it is physically not correct! - save_secondary_scattering (bool): Flag to save the secondary scattering dataset. Default is False. - BSize_1 (float): Pixel binning 1. - BSize_2 (float): Pixel binning 2. Outputs: - secondary_scattering (numpy.ndarray): Dataset with the calculated secondary scattering.

Identifier:
ewoksid02.tasks.secondaryscatteringtask.SecondaryScatteringTask
Task type:
class
Inputs:
BSize_1
BSize_2
Center_1
Center_2
Dummy
WindowRoiSize
algorithm_2scat
beacon_host
dataset_sigma
dataset_signal
dataset_variance
datatype
detector_name
do_process
do_save
filename_data
filename_lima
filename_mask_reference
filename_mask_static
filename_window_wagon
flip_caving
gc_collect
headers
index_range
index_range_last
info
info_history
lima_index_number_format
lima_url_template
lima_url_template_args
log_level
loop_nb
max_slice_size
pre_caving
processing_filename
processing_subtitle
reading_node
save_metadata
save_secondary_scattering
save_sigma
save_variance
scan_memory_url
scan_nb
streams_subscan1
streams_subscan2
subtitle
Outputs:
continue_pipeline
dataset_sigma
dataset_signal
dataset_variance
detector_name
filename_data
filename_lima
headers
index_range_last
info_history
loop_nb
scan_nb
secondary_scattering
streams_subscan1
streams_subscan2

SumTask#

Add two numbers

Identifier:
ewoksid02.tasks.sumtask.SumTask
Task type:
class
Inputs:
a*
b
delay
Outputs:
result

SumTask1#

Add two numbers

Identifier:
ewoksid02.tasks.sumtask.SumTask1
Task type:
class
Inputs:
a*
b
delay
Outputs:
result

SumTask2#

Add two numbers

Identifier:
ewoksid02.tasks.sumtask.SumTask2
Task type:
class
Inputs:
a*
b
delay
Outputs:
result

PdfGetXAverage#

Average a list of 1D XRPD patterns from provided output of IntegrateBlissScan

Inputs: - nxdata_url: .h5 path with NXData url Outputs: - radial: 1D array - intensity: 1D array - info: dict with radial_units and wavelength

Identifier:
ewoksid11.pdfgetx_average.PdfGetXAverage
Task type:
class
Inputs:
nxdata_url*
Outputs:
info
intensity
radial

PdfGetXConfig#

Parse pdfgetx config parameters

Identifier:
ewoksid11.pdfgetx_config.PdfGetXConfig
Task type:
class
Inputs:
filename
pdfgetx_options_dict
Outputs:
pdfgetx_options

PdfGetXProcessor#

Extracts the PDF signal from provided intensities and radial

Inputs: - radial: 1D array - intensity: 1D array - info: dict with unit and wavelength - pdfgetx_options: PDFConfig object Outputs: - result: PDFGetter object with attributes defined by outputtype. Attributes are tuples with [0] X (radial) and [1] Y (intensity) - pdfgetx_options: PDFConfig object

Identifier:
ewoksid11.pdfgetx_processor.PdfGetXProcessor
Task type:
class
Inputs:
info*
intensity*
pdfgetx_options*
radial*
Outputs:
pdfgetx_options
result

PdfGetXSaveAscii#

Saves the PDF result (iq, sq, fq, gr) in separate ASCII files

Inputs: - filename: h5file name that would have been used to save data from XrpdProcessor - result: PDFGetter object with attributes defined by outputtype. Attributes are tuples with [0] X (radial) and [1] Y (intensity) Outputs: - saved: True if saved - filename: list of saved filenames

Identifier:
ewoksid11.pdfgetx_save_ascii.PdfGetXSaveAscii
Task type:
class
Inputs:
filename*
result*
Outputs:
filenames
saved

PdfGetXSaveNexus#

Saves the PDF result (iq, sq, fq, gr) in a NeXus file

Inputs: - output_filename: h5file name where the data must be saved - scan: scan number - detector name: name of the detector - result: PDFGetter object with attributes defined by outputtype. Attributes are tuples with [0] X (radial) and [1] Y (intensity) Outputs: - output_url: URL of the NXprocess containing the saved data

Identifier:
ewoksid11.pdfgetx_save_nexus.PdfGetXSaveNexus
Task type:
class
Inputs:
detector_name*
output_filename*
pdfgetx_options*
result*
scan*
lima_url_template
lima_url_template_args
prioritize_non_native_h5items
retry_period
retry_timeout
subscan
Outputs:
output_url

CalibAverage#

Average a list of 2D images from bliss data url

Inputs: - bliss_data_url: format like /data/visitor/maXXXX/idXX/20240101/sample/dataset/dataset.h5::1.1/measurement/detector Outputs: - image: 2D image averaged from list of 2D images

Identifier:
ewoksid11.pyfai_calib_average.CalibAverage
Task type:
class
Inputs:
bliss_data_url*
Outputs:
image

CalibSave#

Saves the geometry from calibration result in output_path. By default, the geometry file will be: {calibrant}_YYYYMMDD_hhmm.poni

Inputs: - geometry: dictionnary, result of CalibrateSingle - calibrant: string with calibrant name used in CalibrateSingle - detector: string, default is detector - detector_config: dict with key splineFile - output_path: list of strings, where the poni will be saved - energy: float, in keV

Outputs: - output_path: list of strings, path of the poni files saved

Identifier:
ewoksid11.pyfai_calib_save.CalibSave
Task type:
class
Inputs:
calibrant*
detector*
detector_config*
energy*
geometry*
output_path*
Outputs:
output_path

ID22H5ToSpec#

Identifier:
ewoksid22.convert.ID22H5ToSpec
Task type:
class
Inputs:
filename*
ascii_extension
entries
outdirs
outprefix
primary_outdir
rebin_filename
retry_timeout
Outputs:
entries
outfile

ID22TopasExtract#

Identifier:
ewoksid22.extract.ID22TopasExtract
Task type:
class
Inputs:
filename*
inp_file*
entries
full_tth
inp_step
outdirs
outprefix
primary_outdir
startp
tth_max
tth_min

ID22Rebin#

Identifier:
ewoksid22.rebin.ID22Rebin
Task type:
class
Inputs:
filename*
parsfile*
debug
delta2theta
device
endp
energy
entries
iter
outdirs
outprefix
phi
pixel
primary_outdir
range
retry_timeout
startp
step
unscramble_method
wavelength
width
Outputs:
entries
outfile

ID22Sum#

Identifier:
ewoksid22.sum.ID22Sum
Task type:
class
Inputs:
binsize*
filename*
advanced
ascii_extension
entries
lowtth
outdirs
outprefix
primary_outdir
raw_filename
resfile
scaling
sum_all
sum_single
Outputs:
outfile

WaitScansFinished#

Identifier:
ewoksid22.wait.WaitScansFinished
Task type:
class
Inputs:
filename*
entries
retry_timeout
Outputs:
entries
filename

FlatFieldFromEnergy#

Interpolate an energy-stack of flat-field images.

The resulting flat-field image can be used as follows to correct diffraction patterns for flat field:

.. code::

Icor = I / flatfield

Output:

  • flatfield: 2D numpy.ndarray

Identifier:
ewoksid31.tasks.flatfield.FlatFieldFromEnergy
Task type:
class
Inputs:
newflat* : str

Filename of ID31 P3 detector flatfield

Examples:
  • '/data/id31/inhouse/P3/flats.mat'
oldflat* : str

Filename of ID31 P3 detector ‘old’ flatfield

Examples:
  • '/data/id31/inhouse/P3/flats_old.mat'
energy* : float | str

X-ray energy in KeV or URL of hdf5 dataset storing its value

Examples:
  • 75.0
  • '/path/to/file.f5::/2.1/instrument/positioners/energy'
enabled : bool= True
Outputs:
flatfield

SaveNexusPatternsAsId31TomoHdf5#

Save integrated XRD patterns into an HDF5 file compatible with ID31 tomography workflow.

Output:

  • filename (str): Path to the written HDF5 file, or empty string if skipped.

Identifier:
ewoksid31.tasks.tomo.SaveNexusPatternsAsId31TomoHdf5
Task type:
class
Inputs:
scan_entry_url* : str

HDF5 URL to the NXentry group containing the ‘measurement’ group

Examples:
  • '/path/to/raw_data/file.f5::/2.1'
rot_name* : str

Dataset name in ‘measurement’ for rotation angles

Examples:
  • 'nth'
y_name* : str

Dataset name in ‘measurement’ for horizontal positions

Examples:
  • 'ny'
nxdata_url* : str

HDF5 URL to the NXdata group containing integrated patterns

Examples:
  • '/path/to/processed_data/file.f5::/2.1/p4_integrate/integrated'
output_filename* : str

Path to the output HDF5 file

Examples:
  • '/path/to/processed_data/tomo.f5'
enabled : bool= True
overwrite : bool= False
retry_timeout : float | None= None

Duration in seconds for which to try to open HDF5 files

retry_period : float | None= None

Amount of time in seconds between each attempt to open HDF5 files

Outputs:
filename

GenerateIniFile#

Generate the ini file needed for the detector distorsion. Save it in output_file. Inputs: - template_file: string, path of the ini template file - mapping_data: dict[str, any], A dictionary of key-value pairs used for substitution in the template file - output_file: string, the full path (including filename) where the ini file will be saved Outputs: - output_file: string, the full path (including filename) where the ini file is saved

Identifier:
ewoksoptics.tasks.generate_ini_file.GenerateIniFile
Task type:
class
Inputs:
mapping_data*
output_file*
template_file*
Outputs:
output_file