Loading...
 
Page actions
Print
Applications

Introduction

Applications are B-Fabric entities which represent external programs. There are two kinds of input:
  1. physical samples: a physical sample is entered into an instrument which is controlled by some special program; this program performs some kind of analysis/measurement.
  2. output files of other programs: one or more files, which are already registered in B-Fabric, are processed/analyzed.
This distinction is modeled as application type: in the first case, an application is considered to be of type "Import", in the second case, it is of type "Analysis". Regardless of this distinction, the purpose of all applications is the same: they contain a part of information necessary for the creation of workunits (the other part is entered by the user when creating the concrete workunit). The created output is written into files which are copied to dedicated hosts (storage hosts), where they are stored persistently and registered as resources in B-Fabric.
Summary: Application is a kind of template for the workunit, which can have none(Import) or one(Analysis) preceding workunits.

Register/Create Applications

Applications can only be created by users having the role applicationManager. At FGCZ, all employees have this role implicitly. After a new application is created, the creator will automatically be set as supervisor for the new application. Only the admin can exchange the supervisor of an application, and only the admin or the supervisor will be able to edit or delete the application. There are further conditions an applications needs to fulfill in order to be deletable: all workunits created with this application have to be deleted, and all import resources connected to the application have to be deleted as well.
Summary: Applications can not be deleted if there are references (existing workunits).
For manage applications, one has to navigate to the "Applications" menupoing.
Following attributes have the same meaning for all types of applications:
  • Name: The name of the application must not have more than 256 characters. Note that the application name will appear in several places, e.g. in buttons to invoke the application. Therefore, choose a compact but reasonable name.
  • Technology: the technology to which the application belongs.
  • Help: The URL to an external web page where one can get more info about this application. This link is presented to the user in various places, like run-application.html screen.
  • Description: The description shall contain all relevant information about this application, e.g. what it does and which input it expects.
  • Hidden: When set to TRUE, the application cannot be invoked.
  • For Employees Only: When set to TRUE, only employees will be able to invoke this application.
  • Send Mail Notification: When set to TRUE, an email notification is sent to all members of the container on setting the status to "available" of the workunit with this application.
  • Preceding Applications: The resources which belong to workunits created by applications defined as preceding applications will be considered as input resources of the application.
  • Succeeding Applications: The resources which belong to workunits created by the application will be considered as input resources for the defined succeeding applications.
  • Annotation Required: All resources of the workunit created by this application must be annotated.
Depending on the configuration of attributes Type, Pageflow, Storage, Application Executable, Wrapper Creator and Submitter, the application will be considered to be used in different ways.
Screenshot from B-Fabric:
Image

Scenario

Files are stored in the correct place on a host which is registered in B-Fabric as storage. It is only necessary to make B-Fabric aware of those files.

Configuration

  • Type: Import
  • Pageflow: Import
  • Storage: none or appropriate storage
  • Application Executable: none
  • Wrapper Creator: none
  • Submitter: none
Important: The application executable must NOT be set if this application is considered to work as a "Link Import" application.

Creation of ImportResource Entities

The ImportResource entities for this application must contain the storageid/relativePath if they are to be presented to the user. The following XML request shows an example with minimal amount of information.
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:end="http://endpoint.server.webservice.bfabric.org/">
   <soapenv:Header/>
   <soapenv:Body>
      <end:save>
         <parameters>
            <login>?</login>
            <password>?</password>
            <importresource>
               <name>jonas_20080804_p438_fqms_data/20080719_44_yeast_OGE_12_12_b02.RAW</name>
               <applicationid>7</applicationid>
               <projectid>474</projectid>
               <relativepath>p474/Proteomics/LTQ_1/jonas_20080804_p438_fqms_data/20080719_44_yeast_OGE_12_12_b02.RAW</relativepath>
               <size>101948114</size>
               <storageid>2</storageid>
            </importresource>
         </parameters>
      </end:save>
   </soapenv:Body>
</soapenv:Envelope>

When a workunit is created, the selected ImportResource entities will be connected to this workunit. This will be visible in the show-workunit.html screen as well as when fetching the workunit information through web services. For more information about ImportResource entities and web services, refer to Endpoint ImportResource.

"Trigger" Import

Scenario A

Files are stored in the correct place on a host which is registered in B-Fabric as storage, but just making B-Fabric aware of those files may not be enough; it may be necessary to trigger further actions (outside of B-Fabric) when a user creates a workunit with this application like importing sample information from another system, automatically assign extracts to resources, notify the project coach that files are imported...

Scenario B

Files are not stored in the correct place, but in a temporary location; when they are imported in B-Fabric, the copy process to the correct storage is to be triggered (as well as further actions, as described in scenario A).

Configuration

  • Type: Import
  • Pageflow: Import
  • Storage: appropriate storage
  • Application Executable: appropriate application executable
  • Wrapper Creator: appropriate wrapper creator
  • Submitter: appropriate submitter

Creation of ImportResource Entities

The ImportResource entities for this application must contain the URL if they are to be presented to the user. The following XML request shows an example with minimal amount of information.
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:end="http://endpoint.server.webservice.bfabric.org/">
   <soapenv:Header/>
   <soapenv:Body>
      <end:save>
         <parameters>
            <login>?</login>
            <password>?</password>
            <importresource>
               <name>20120608-G25_1.fastq.gz</name>
               <applicationid>101</applicationid>
               <projectid>788</projectid>
               <url>rsync://fgcz-s-024.uzh.ch/srv/gstore/projects/p788/Illumina_20120608/20120608-G25_1.fastq.gz</url>
            </importresource>
         </parameters>
      </end:save>
   </soapenv:Body>
</soapenv:Envelope>

When a workunit is created, the selected ImportResource entities will be connected to this workunit. This will be visible in the show-workunit.html screen as well as when fetching the workunit information through web services.

"Local File" Import

Scenario

A user uploads a file to the B-Fabric host, in a temporary location. After saving the workunit, the uploaded file will be copied to the correct storage host. This scenario is actually a special case of "Trigger" Import.

Configuration

  • Type: Import
  • Pageflow: Local File Import
  • Storage: appropriate storage
  • Application Executable: appropriate application executable
  • Wrapper Creator: appropriate wrapper creator
  • Submitter: appropriate submitter

Scenario

Resources created in B-Fabric are used as input for an application executable, but the processing is not triggered through B-Fabric. The selection of input resources, dataset and parameters is performed outside of B-Fabric. After the execution is complete, the result files are copied to a storage machine and finally registered in B-Fabric. At the time of writing, there is only one such registered application, namely "MS Search Result Import", which does not utilizes web services but a specially created SQL function. For a possible way how to do this by utilizing web services, have a look at the example 2 of Webmethod Save.

Configuration

  • Type: Analysis
  • Pageflow: none
  • Storage: empty or appropriate storage
  • Application Executable: none
  • Wrapper Creator: none
  • Submitter: none

"Trigger" Analysis

Scenario

Resources created in B-Fabric are used as input for an application; the processing is triggered through B-Fabric. The selection of input resources, dataset and parameters is performed through the B-Fabric GUI.

Configuration

  • Type: Analysis
  • Pageflow: appropriate pageflow
  • Storage: empty or appropriate storage
  • Application Executable: appropriate executable
  • Wrapper Creator: appropriate wrapper creator
  • Submitter: appropriate submitter

Workunit Creation: How the different components work together

The application executable is a B-Fabric entity which represents a real-world executable; it is either uploaded in B-Fabric, or the attribute "program" of the executable contains either a command, an absolute path or similar. This executable contains the complete processing logic from the biological point of view, it is not necessarily aware of B-Fabric. If such an executable is to be triggered from B-Fabric, it must be extended by all necessary additional functionality, which means that one (or several) wrappers are to be created. This additional functionality may differ: some executables are able to copy the input files from the storage to the local scratch, others expect that the input files are already there; some do copy the output files from the temporary location to the storage, others are done after the creation of the output file is done. The way of the wrapper creation is determined by the selection of the Wrapper Creator. After the wrapper executables are created, they need to be executed on appropriate machines. The information about how and on which machines exactly the wrappers have to be run is determined by the Submitter selection. The following steps are performed after a user determined all the input and clicked on the "Save" button:
  • B-Fabric will create an external job (assume that this external job has the ID 1000) for the wrapper creator executable. The external job will have "Workunit" in the "Client Entity Class Name" attribute, and the workunit ID in "Client Entity Class ID".
  • B-Fabric will trigger the wrapper creator executable (it is assumed that this is a shell executable uploaded in B-Fabric) as follows: 
    /bin/bash /path/to/wrappercreator/executable.sh -j 1000
  • The wrapper creator executable is supposed to create all wrappers and upload them to B-Fabric.
  • After the wrapper creation/upload is done, the wrapper creator executable sets the status of the external job with ID 1000 to "done". Through this, it announces to B-Fabric that the wrapper creation process is finished and that further steps might be performed. If the status is set to "failed", then B-Fabric will also set the status of the workunit to "failed" and will stop any further triggering.
  • If the wrapper creation was successful, then B-Fabric will create an external job (assume that this has the ID 1010) for the submitter executable and trigger it as follows: 
    /bin/bash /path/to/submitter/executable.sh -j 1010
  • The submitter executable is supposed to download all wrappers and submit them. In case of success, the status of the external job with ID 1010 will be set to "done", otherwise to "failed".
  • The submitted wrappers are supposed to announce their outcome by setting the status of the corresponding external job to "done" or "failed". If the status of all external jobs of context "workunit" are set to "done" (as soon as the last external job status was changed), B-Fabric will update the status of the workunit and set it to "available" (in case of "import" applications to "imported") as well as update the end date. In case that external processes are killed without the posibility to tell B-Fabric what happend, B-Fabric must be "informed manually" by webservice, which means that the status of those external jobs has to be set to "failed" manually.

What to do when an application does not run (correctly), workunits are missing/incorrect etc.

It is important to know the workflow of the workunit creation:
  1. Just after the creation (pressing Save button), an external job entity is created which is to be processed by the wrapper creator executable. This executable is triggered with the ID of the created external job as input. B-Fabric expects that the status of this external job will be changed to "done", in which case B-Fabric will proceed.
  2. If the previous step was successful, B-Fabric creates an external job entity for the submitter executable and triggers it. This will submit the uploaded workunit specific executables.
  3. For each workunit specific executable, an external job was created, and after the processing completed successfully, its status must be updated to "done", otherwise to "failed".
If something goes wrong, then one should have a look at which step the error occured and try to find out how this happened. If the processing did not run, then the person responsible for the application executable should be contacted.

Soap XML

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:end="http://endpoint.server.webservice.bfabric.org/">
   <soapenv:Header/>
   <soapenv:Body>
      <end:save>
         <parameters>
            <login>?</login>
            <password>?</password>
            <!--Zero or more repetitions:-->
            <application>
               <!--Zero or more repetitions:-->
               <customattribute>
                  <name>?</name>
                  <!--Optional:-->
                  <type>?</type>
                  <value>?</value>
               </customattribute>
               <!--Optional:-->
               <id>?</id>
               <name>?</name>
               <!--Optional:-->
               <description>?</description>
               <!--Optional:-->
               <annotationrequired>?</annotationrequired>
               <!--Optional:-->
               <enabled>?</enabled>
               <!--Optional:-->
               <executableid>?</executableid>
               <!--Optional:-->
               <foremployeesonly>?</foremployeesonly>
               <!--Optional:-->
               <help>?</help>
               <!--Optional:-->
               <hidden>?</hidden>
               <!--Optional:-->
               <importresourcesrequired>?</importresourcesrequired>
               <!--Optional:-->
               <outputfileformat>?</outputfileformat>
               <!--Optional:-->
               <pageflowname>?</pageflowname>
               <!--Zero or more repetitions:-->
               <precedingapplicationid>?</precedingapplicationid>
               <!--Optional:-->
               <predecessorid>?</predecessorid>
               <!--Optional:-->
               <sendmailnotification>?</sendmailnotification>
               <!--Optional:-->
               <storageid>?</storageid>
               <!--Optional:-->
               <submitterid>?</submitterid>
               <!--Zero or more repetitions:-->
               <succeedingapplicationid>?</succeedingapplicationid>
               <!--Optional:-->
               <supervisorid>?</supervisorid>
               <technologyid>?</technologyid>
               <type>?</type>
               <!--Optional:-->
               <valid>?</valid>
               <!--Optional:-->
               <weburl>?</weburl>
               <!--Optional:-->
               <wrappercreatorid>?</wrappercreatorid>
            </application>
         </parameters>
      </end:save>
   </soapenv:Body>
</soapenv:Envelope>

Created by amarko. Last Modification: Friday February 2, 2024 13:32:53 CET by tuerker.