Workflows

Primitive components

Workflows are directed acyclic graphs (DAGs) of individual MUSES module executions, or “processes”, constructed using composite structures based on four primitive components:

process: the fundamental workflow type that executes a single MUSES module
group: a set of components that execute in parallel
chain: a linear sequence of components

Composition

A workflow is constructed as a list of process definitions and an ordered list of components that build an increasingly complex DAG from those processes. Components must be defined in the order of their dependency.

Process names and component names share a namespace in which they must be unique.

Examples

The examples below use YAML format to define workflow configurations, because this format is easy to read for humans while supporting the rigorous syntax required to unambiguously define a data structure suitable for machines. Ultimately the worflow definition must be rendered in JSON format suitable for the Calculation Engine API, but as demonstrated in the tutorial, this conversion can be done transparently by any number of libraries such as Python requests.

Chain

A chain is a sequence of components that are executed in order, where previous components in the sequence must successfully complete before the next component is processed. Chain components are requied when components are causally dependent on one another.

In the example below, the Chiral EFT module pipes its output to the Lepton module, which must only run if the first process completes successfully.

processes:
  - name: chiral_eft_eos
    module: chiral_eft
    config:
      run_name: 'test_chiral_eft_lepton'
      chiraleft_parameters:
        fitted_parameter_set: 'n3lo-450'
      calculation_options:
        use_quadratic_asymmetry_expansion: true
  - name: lepton-module
    module: lepton
    pipes:

    config:
      derivatives:
        relative_step_size: 3.0e-6
        precision: 3
    pipes:
      input_eos:
        label: ChEFT_Output_Lepton
        module: chiral_eft
        process: chiral_eft_eos
components:
  - type: chain
    name: chiral_eft-lepton
    sequence:
      - chiral_eft_eos
      - lepton-module

Group

A group is set of components that are allowed to run in parallel. Concurrent execution is not actually guaranteed, however, because that depends on the Calculation Engine task queue system and dynamic worker load. Parallel here means that the output of the components in a group do not causally depend on one another.

In the example below, there are two chains that execute in parallel. One chain outputs the EoS generated by CMF to the Lepton module. The other chain outputs the EoS generated by Chiral EFT to an independent Lepton module process.

Note:

The name of the Lepton module processes must be unique for unambiguous reference when defining components.
The order of the process definitions in the processes block does not matter.
The order of the chain components (chain1 and chain2) in the group1 definition does not matter.

processes:
  - name: chiral_eft_eos
    module: chiral_eft
    config:
      run_name: 'test_chiral_eft_lepton'
      chiraleft_parameters:
        fitted_parameter_set: 'n3lo-450'
      calculation_options:
        use_quadratic_asymmetry_expansion: true
  - name: cmf
    module: cmf_solver
    config:
      variables:
        chemical_optical_potentials:
          muB_begin: 1000.0
          muB_end: 1400.0
          muB_step: 200.0
          muQ_begin: -300.0
          muQ_end: 0.0
          muQ_step: 150.0
  - name: lepton1
    module: lepton
    pipes:

    config:
      derivatives:
        relative_step_size: 3.0e-6
        precision: 3
    pipes:
      input_eos:
        label: ChEFT_Output_Lepton
        module: chiral_eft
        process: chiral_eft_eos
  - name: lepton2
    module: lepton
    pipes:

    config:
      global:
        use_beta_equilibrium: false
    pipes:
      input_eos:
        label: CMF_for_Lepton_baryons_only
        module: cmf_solver
        process: cmf
components:
  - type: chain
    name: chain1
    sequence:
      - chiral_eft_eos
      - lepton1
  - type: chain
    name: chain2
    sequence:
      - cmf
      - lepton2
  - type: group
    name: group1
    group:
      - chain1
      - chain2

Singleton

A so-called “singleton” workflow consists of a single process. This means that the workflow is identical whether a “group” or a “chain” component is defined.

processes:
  - name: cmf
    module: cmf_solver
    config:
      variables:
        chemical_optical_potentials:
          muB_begin: 1000.0
          muB_end: 1400.0
          muB_step: 200.0
          use_hyperons: false
          use_decuplet: false
          use_quarks: false
components:
  - type: group
    name: run_cmf_test
    group:
      - cmf

Complex workflow

An incomplete “sketch” of the config for the complex workflow depicted in the diagram below is provided here. The purpose is to illustrate how to break down the desired structure and construct it logically piece by piece.

It is left as an exercise for the reader to complete the missing “pipes” connecting the output of processes to the consuming processes and to add the missing config: specification for each process.

  - name: chiral
    module: chiral_eft
  - name: cmf
    module: cmf_solver
  - name: crust
    module: crust_dft
  - name: lepton1
    module: lepton
    pipes:
      input_eos:
        label: CMF_for_Lepton_baryons_only
        module: cmf_solver
        process: cmf
  - name: lepton2
    module: lepton
    pipes:
      input_eos:
        label: CMF_for_Lepton_baryons_only
        module: cmf_solver
        process: cmf
  - name: lepton3
    module: lepton
    pipes:
      input_eos:
        label: ChEFT_Output_Lepton
        module: chiral_eft
        process: chiral
  - name: lepton4
    module: lepton
    pipes:
      input_eos:
        label: e4mma w/o lepton
        module: crust_dft
        process: crust
  - name: synthesis1
    module: synthesis
    pipes: {}
  - name: synthesis2
    module: synthesis
    pipes: {}
  - name: synthesis3
    module: synthesis
    pipes: {}
  - name: qlimr
    module: qlimr
    pipes:
      input_eos:
        label: CMF_for_Lepton_baryons_only
        module: cmf_solver
        process: cmf
  - name: flavor
    module: flavor_equilibration
    pipes: {}
components:
  - type: group
    name: group-leptons
    group:
      - lepton1
      - lepton2
  - type: chain
    name: chain-cmf-leptons
    sequence:
      - cmf
      - group-leptons
      - synthesis1
  - type: chain
    name: chain-chiral-lepton
    sequence:
      - chiral
      - lepton3
  - type: chain
    name: chain-crust-lepton
    sequence:
      - crust
      - lepton4
  - type: group
    name: group-chiral-crust
    group:
      - chain-chiral-lepton
      - chain-crust-lepton
  - type: chain
    name: chain-chiral-crust-synthesis
    sequence:
      - group-chiral-crust
      - synthesis2
  - type: group
    name: eos-syntheses
    group:
      - chain-cmf-leptons
      - chain-chiral-crust-synthesis
  - type: chain
    name: final-synthesis
    sequence:
      - eos-syntheses
      - synthesis3
  - type: group
    name: group-observables
    group:
      - qlimr
      - flavor
  - type: chain
    name: final-observables
    sequence:
      - final-synthesis
      - group-observables