eSiMon

Overview

News
  • 2/1/11 – eSiMon 1.0 released. See the Download & Installation tab to download eSiMon alpha and get the user’s manual.
  • 9/17/10 – eSiMon (alpha) released. See the Download & Installation tab to download eSiMon alpha and get the user’s manual.

Description

The eSiMon dashboard is a collaborative web-based system that provides scientists to monitor and analyze their simulations. The “live” version of the dashboard is physically located at Oak Ridge National Laboratory (ORNL) and can be accessed with a National Center for Computational Sciences (NCCS) account at https://esimmon.ccs.ornl.gov. This version of the dashboard gives an overview of ORNL and National Energy Research Scientific Computing Center (NERSC) computers. Users can quickly determine which systems are up or down, which are busy and where they would like to launch a job. Users can also view the status of their running and past jobs as well as of their collaborators’. The workflow/dashboard system at ORNL consistently lists and archives simulation runs (we call shot). Scientists can select a particular run to view and manipulate the data generated by this simulation. The workflow coordinates and executes all the mundane simulation tasks (moving files, converting formats, producing images and movies etc.), recording its every step in the eSiMon “Data Store” using its integrated provenance tracking system. It is the eSiMon “data feeder”. The main goal of the web interface is to be an easy-to-use user-centered front end that shields users from the intricacies of dealing with directories, files, file formats etc. Transferring the complexity of the system to the back end creates a lighter weight and more accessible web portal and a more inclusive and diverse user base; users can have a common access point to the simulation data while their level and area of expertise may vary from theoretical sciences, to performance , to visualization and analysis etc.

A portable version of eSiMon is available to allow other centers/scientists to take advantage of this system. At ORNL the workflow/provenance system feeds the web application with metadata on the simulation in order to point eSiMon to the simulation data. The portable dashboard is installed with simplified PHP and C methods to fill in the eSiMon database. The PHP API explains how the dashboard works and the information needed to connect the simulation to the web interface. The C API can be used to directly link a running simulation to eSiMon. Both will be available in the final release.

The dashboard front-end code was developed using Flex, framework to create Flash Applications. The code is written in Action Script and MXML. Flash is a common choice for Rich Internet Applications (RIAs); it creates dynamic pages with local interaction and asynchronous server communication. The Flash Media Player is fast, robust, scalable and reliable. In addition to natively supporting good quality videos, Flash has other advantages such as consistent appearance and native support for vector graphics and 3D graphics. These are desired characteristics to scrutinize simulation data from different perspectives. The server code is implemented in PHP which is a widely used general purpose scripting language.

The interface offers some basic features such as visualizing images, videos and textual information about the simulation. By simply dragging-and-dropping variables names from a Tree View on the left of the simulation monitoring page onto the main canvas, users can view graphics associated with these variables at a particular time stamp (step). They can use VCR features to observe the variables changing over time. Scientists can take electronic notes on the simulation as well as annotate movies. Other features include vector graphics with zoom/pan capabilities, viewing data lineage and downloading processed and raw data onto their local machines. Future versions will include hooks into external software and user customized analysis and visualization tools.

Screenshot

eSiMon Screenshot

Download & Installation

Below are links for the eSiMon download and user manual. For additional support, please contact help@nccs.gov.

Download/Install (UNIX)

To install eSiMon, execute the following steps:

  1. Download eSiMon installation script (buildesimmon.sh)
  2. Move the script to the user’s home directory on the target machine. Alternatively you can directly download the setup script onto the target machine using the following command: wget http://users.nccs.gov/~rbarreto/esimmon/packagingtools/buildesimmon.sh
  3. Change the permissions buildesimmon.sh by running the following command: chmod +x buildesimmon.sh
  4. Run the script as root: sudo ./buildesimmon.sh
  5. Follow the onscreen instructions and wait until the script finished execution. The script will download and install all the packages required for the application. The script will also check the compiler versions to insure the correct building of all the packages. The installation takes place in three phases.
    1. Check the version of the g++, ruby and python compilers available in the target machine
    2. Build the basic software that eSiMon and the associated tools depends on, namely zlib, libpng, mxml, mpi
    3. Build the tools used by eSiMon, namely netcdf, adios, mplayer, ffmpeg, flvtool2, xmgrace, libcurl
    4. The optional srm-lite package is installed and the java run time environment required for it is verified
    5. The eSiMon source code
  6. Once it is done, navigate using a web browser to http://youripaddress/esimmon to view the eSiMon dashboard (make sure you replace with the actual ip address of the machine you just ran the scrpt on). You may be asked to download the Flash Media Player for your browser upon entering the URL

Note: For general and specific requirements about different Linux based Operating Systems see Appendix A of the User’s Manual.

Dependencies

The eSiMon package includes the core of the code: the Graphical User Interface (Flash application) and the server code used to implement the basic eSiMon features. However without a monitoring API that feeds information from a simulation to the eSiMon database, it remains only a front-end application. In other words, users would have to understand and implement methods to give it content. We plan to provide C API and enable users to automatically connect their running simulations to eSiMon. In this case, we depend on additional software to generate Flash (.flv) movies from images generated by the simulation. Moreover, our approach to eSiMon has been to provide a single access point to users’ analysis tools. In future releases, we plan to provide and support hooks into external software for data management and analysis. This alpha version of eSiMon includes a hook into a Storage Resource Manager SRM-Lite.

  1. Data Description Software
    1. NetCDF: NetCDF (network Common Data Form) is a set of software libraries and machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.
    2. ADIOS: The Adaptable IO System provides a simple, flexible way for scientists to describe the data in their code that may need to be written, read, or processed outside of the running simulation. By providing an external to the code XML file describing the various elements, their types, and how you wish to process them this run, the routines in the host code (either Fortran or C) can transparently change how they process the data.
  2. Movie Generation Software
    1. Mplayer/Mencoder: Mplayer is a movie player for LINUX and many other Unices. Mencoder is a converter that supports many input video format.
    2. FFmpeg: FFmpeg is a complete, cross-platform solution to record, convert and stream audio and video. It is used to generate movies from eSiMon.
    3. FLVTool2: FLVTool2 is a manipulation tool for Macromedia Flash Video files (FLV). We use it to add cur points to the .flv files and keep track of simulation time steps.Note: FLVTool2 has a Ruby dependency. The tagging script for the movies is written in Python.
  3. Storage Resource ManagerSRM Lite allows efficient data movement across wide area network. It is a simple command-line based tool with pluggable file transfer protocol supports developed at the Lawrence Berkeley National Laboratory (https://sdm.lbl.gov/srmlite/). It is Java-based. The PHP and C API explain how to register provenance information about images generated during the simulation on the eSiMon database. Provided that the path to SRM-Lite is set and the data lineage information is available in the eSiMon data store, users can conveniently download raw simulation data files from the monitoring simulation page.

    Note: SRM-Lite has a Java dependency.

User’s Manual and Installation

Press & Publications

  1. Cummings, Klasky, Podhorszki, Barreto, Lofstead, Schwan, Docan, Parashar, Sim, Shoshani, “EFFIS: and End-to-end Framework for Fusion Integrated Simulation”, submitted to PDP 2010, http://www.pdp2010.org/ <http://www.pdp2010.org/> .
  2. Roselyne Tchoua, Scott Klasky, Norbert Podhorszki, Brad Grimm, Ayla Khan, Emanuele Santos, Claudio Silva, Pierre Mouallem, Mladen Vouk: “Collaborative Monitoring and Analysis for Simulation Scientists” 2010 International Symposium on Collaborative Technologies and Systems, (CTS 2010), Chicago, Illinoi, USA, May 2010.
  3. E. Santos, J. Tierny, A. Khan, B. Grimm, L. Lins, J. Freire, V. Pascucci, C. Silva, S. Klasky, R. Barreto, N. Podhorszki, “Enabling Advanced Visualization Tools in a Web-Based Simulation Monitoring System”, in IEEE International Conference on eScience 2009.
  4. Pierre Mouallem, Mladen Vouk, Scott Klasky, Norbert Podhorszki and Roselyne Barreto: “Tracking Files Using the Kepler Provenance Framework” Proceedings of 21st International Conference on Scientific and Statistical Database Management, SSDBM’09, LNCS 5566, pp. 273-282, New Orleans, LA, USA, June 2009.
  5. R. Barreto, S. Klasky, N. Podhorszki, P. Mouallem, M. Vouk: “Collaboration Portal for Petascale Simulations” 2009 International Symposium on Collaborative Technologies and Systems, (CTS 2009), pp. 384-393, Baltimore, Maryland, USA, May 2009. doi: 10.1109/CTS.2009.5067505.