In this project, the potential of novel interaction and visualization techniques for processing node-link diagrams (e.g. UML diagrams, business process models, etc.) was explored. The goal was to find the most promising approaches (e.g. multi-touch gestures) to facilitate the editing and exploration of node-link diagrams. The developed techniques were realized using iterative development processes and evaluated by laboratory studies. They are implemented in a system that can serve as a basis for further research in the field of diagram interaction. However, they can also be transferred to other application areas.

Creation and editing of node-link diagrams on interactive displays

mathias_project_1

Visual representations of graphs in the form of node-link diagrams play an important role in many application areas, for example in software modeling, biology or process management. Besides exploration and analysis, in many situations diagrams are created and edited by hand. For this purpose they are drawn on paper or whiteboards or edited in digital diagram editors. However, interaction with mouse-based editors is often cumbersome, and there is no seamless integration of hand-drawn sketches.
Therefore, this project explored the potential of novel technologies, such as interactive displays and digital pen and paper, for processing node-link diagrams. Especially the combination of multi-touch and pen input played an important role. It allows not only the simultaneous execution of several actions, but also fast mode switching with gestures, where finger and pen interaction can complement each other in a useful way.

Results

Within the scope of this project a prototype was developed, which offers the following functionalities:

  • Interaction with finger and pen gestures: The realized gesture set is based on a collection of gestures suggested by users.
  • Consideration of two predominant mental models: Structured editing (creating diagrams by selecting items from menus) and sketching diagrams (free drawing as on electronic whiteboards).
Results.

Multitouch layout techniques for interactive displays

The precise alignment of graphic objects and the creation of layouts are elementary processes when working with various software tools, such as graphics and presentation applications or diagram editors. The use of interactive displays seems promising for such applications. The goal of this project was the development and evaluation of interaction techniques as well as virtual tools that support the user in layout tasks in tabletop environments.

Results

As a result of this project the NEAT system (Natural and Effective Layout Techniques) was developed. It consists of three core components:

  • Interactive Background Grid: This is an interactive grid that is displayed on the background of the drawing area. It allows precise positioning of graphical objects using snap dragging. The cell sizes of the grid can be interactively adjusted and changed by multi-touch input. Furthermore, it is possible to quickly and seamlessly transform an orthogonal grid into a radial grid.mathias_project_2_1mathias_project_2_2
  • Multitouch Alignment Guides: These guides are created explicitly, for example by drawing them, and can be freely positioned with the help of a handle. The guideline can take any shape (from straight lines and curves to circles or rectangles) and allow graphical objects to be bound. These can then be positioned and aligned along the line using multitouch interaction.mathias_project_2_3mathias_project_2_4
  • Layout Gestures: A set of multi-touch and pen gestures has been developed that allow graphic objects to be quickly duplicated and aligned with each other with pixel precision. The gestures are abstract in nature, so they have to be learned, and explicitly address the needs of modeling experts.mathias_project_2_5
Full video of NEAT.

Off-Screen Visualization for Node-Link Diagrams

Node-link diagrams, such as software models, become so large in many situations that they are difficult to display in their entirety on standard displays. As a result, the contents of overview views are difficult to read and correlations are often difficult to recognize. This project deals with the use of modern visualization techniques to make large graphs easier to understand and navigate.

Results

A focus and context technique was developed to support the exploration of large diagrams. It allows to quickly navigate to currently invisible nodes of a diagram. When enlarging a diagram section, representative objects for the nodes outside the visible area are displayed at the edge of the display. These interactive proxy objects provide the user with context information. By clicking on them, they allow quick automatic navigation to the linked diagram element.

mathias_project_2_7mathias_project_2_8
mathias_project_2_9

Full video of Visualizing Off-Screen Elements of Class Diagrams.

Puplications

  • @inbook{fixme,
       author = {Mathias Frisch and Raimund Dachselt},
       editor = {Thomas Schlegel},
       title = {Kombinierte Multi-Touch und Stift-Interaktion: Ein Gesten-Set zum Editieren von Diagrammen},
       booktitle = {Multi-Touch: Interaktion durch Ber\"{u}hrung},
       series = {Xpert.press},
       year = {2014},
       month = {2},
       isbn = {978-3-642-36112-8},
       pages = {89--116},
       doi = {10.1007/978-3-642-36113-5_5},
       url = {http://dx.doi.org/10.1007/978-3-642-36113-5_5},
       publisher = {Springer Berlin Heidelberg}
    }

    List of additional material

  • @article{Frisch-2013-InfoVisOffScreen,
       author = {Mathias Frisch and Raimund Dachselt},
       title = {Visualizing Off-Screen Elements of Node-Link Diagrams},
       booktitle = {Information Visualization},
       volume = {12(2)},
       year = {2013},
       month = {4},
       pages = {133--162},
       numpages = {30},
       publisher = {SAGE}
    }

    List of additional material

  • Visualization and Interaction Techniques for Node-Link Diagram Editing and Exploration

    Visualization and Interaction Techniques for Node-Link Diagram Editing and Exploration.

    Frisch, M.;

    Verlag Dr. Hut,2012.

    @phdthesis{fixme,
       author = {Mathias Frisch},
       title = {Visualization and Interaction Techniques for Node-Link Diagram Editing and Exploration},
       series = {Informatik},
       year = {2012},
       month = {6},
       isbn = {978-3-8439-0563-3},
       location = {Otto-von-Guericke-Universit\"{a}t Magdeburg},
       numpages = {253},
       url = {http://www.dr.hut-verlag.de/978-3-8439-0563-3.html},
       publisher = {Verlag Dr. Hut},
       address = {M\"{u}nchen}
    }

    List of additional material

    Web

  • NEAT: A Set of Flexible Tools and Gestures for Layout Tasks on Interactive Displays

    NEAT: A Set of Flexible Tools and Gestures for Layout Tasks on Interactive Displays.

    Frisch, M.; Langner, R.; Dachselt, R.;

    In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces 2011.ITS '11, Kobe, Japan.ACM,1-10,2011.10.1145/2076354.2076356

    @inproceedings{frisch-2012-ITS,
       author = {Mathias Frisch and Ricardo Langner and Raimund Dachselt},
       title = {NEAT: A Set of Flexible Tools and Gestures for Layout Tasks on Interactive Displays},
       booktitle = {Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces 2011},
       series = {ITS '11},
       year = {2011},
       month = {11},
       isbn = {978-1-4503-0871-7},
       location = {Kobe, Japan},
       pages = {1--10},
       numpages = {10},
       doi = {10.1145/2076354.2076356},
       url = {http://doi.acm.org/10.1145/2076354.2076356},
       acmid = {2076356},
       publisher = {ACM},
       address = {New York, NY, USA},
       keywords = {alignment, bimanual interaction, constraints, expert mode, gestures, multi-touch, pen input, tabletops}
    }

    List of additional material

    Video

  • Evaluating a User-Elicited Gesture Set for Interactive Displays

    Evaluating a User-Elicited Gesture Set for Interactive Displays.

    Heydekorn, J.; Frisch, M.; Dachselt, R.;

    In Proceedings of Mensch und Computer 2011.MuC 2011, Chemnitz, Germany.Oldenbourg Publishing,191--200,2011.

    @inproceedings{Heydekorn-2011-MuC,
       author = {Jens Heydekorn and Mathias Frisch and Raimund Dachselt},
       title = {Evaluating a User-Elicited Gesture Set for Interactive Displays},
       booktitle = {Proceedings of Mensch und Computer 2011},
       series = {Mensch und Computer 2011},
       year = {2011},
       month = {9},
       isbn = {978-3-486-71235-3},
       location = {Chemnitz, Germany},
       pages = {191----200},
       numpages = {10},
       publisher = {Oldenbourg Publishing},
       address = {Munich, Germany}
    }

    List of additional material

  • Grids & Guides: Multi-Touch Layout and Alignment Tools

    Grids & Guides: Multi-Touch Layout and Alignment Tools.

    Frisch, M.; Kleinau, S.; Langner, R.; Dachselt, R.;

    In Proceedings of the Conference on Human Factors in Computing Systems.CHI '11, Vancouver, BC, Canada.ACM,1615-1618,2011.10.1145/1978942.1979177

    @inproceedings{frisch-2011-CHI,
       author = {Mathias Frisch and Sebastian Kleinau and Ricardo Langner and Raimund Dachselt},
       title = {Grids \& Guides: Multi-Touch Layout and Alignment Tools},
       booktitle = {Proceedings of the Conference on Human Factors in Computing Systems},
       series = {CHI '11},
       year = {2011},
       month = {5},
       isbn = {978-1-4503-0228-9},
       location = {Vancouver, BC, Canada},
       pages = {1615--1618},
       numpages = {4},
       doi = {10.1145/1978942.1979177},
       url = {http://doi.acm.org/10.1145/1978942.1979177},
       acmid = {1979177},
       publisher = {ACM},
       address = {New York, NY, USA},
       keywords = {interactive grids, layout techniques, manual diagram layout, multi-touch alignment guides, snap-dragging}
    }

    List of additional material

    Video

  • A Multi-Touch Alignment Guide for Interactive Displays

    A Multi-Touch Alignment Guide for Interactive Displays.

    Frisch, M.; Langner, R.; Kleinau, S.; Dachselt, R.;

    In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces 2010.ITS '10, Saarbrücken, Germany.ACM,255-256,2010.10.1145/1936652.1936703

    @inproceedings{fixme,
       author = {Mathias Frisch and Ricardo Langner and Sebastian Kleinau and Raimund Dachselt},
       title = {A Multi-Touch Alignment Guide for Interactive Displays},
       booktitle = {Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces 2010},
       series = {ITS '10},
       year = {2010},
       month = {11},
       isbn = {978-1-4503-0399-6},
       location = {Saarbr\"{u}cken, Germany},
       pages = {255--256},
       numpages = {2},
       doi = {10.1145/1936652.1936703},
       url = {http://doi.acm.org/10.1145/1936652.1936703},
       acmid = {1936703},
       publisher = {ACM},
       address = {New York, NY, USA},
       keywords = {alignment, interactive surfaces, multi-touch}
    }

    List of additional material

    Video

  • Editing and Exploring Node-Link Diagrams on Pen- and Multi-Touch-operated Tabletops

    Editing and Exploring Node-Link Diagrams on Pen- and Multi-Touch-operated Tabletops.

    Frisch, M.; Heydekorn, J.; Schmidt, S.; Nacenta, M.; Dachselt, R.; Carpendale, S.;

    In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces 2010.ITS '10, Saarbrücken, Germany.ACM,304,2010.10.1145/1936652.1936732

    @inproceedings{fixme,
       author = {Mathias Frisch and Jens Heydekorn and Sebastian Schmidt and Miguel Nacenta and Raimund Dachselt and Sheelagh Carpendale},
       title = {Editing and Exploring Node-Link Diagrams on Pen- and Multi-Touch-operated Tabletops},
       booktitle = {Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces 2010},
       series = {ITS '10},
       year = {2010},
       month = {11},
       isbn = {978-1-4503-0399-6},
       location = {Saarbr\"{u}cken, Germany},
       pages = {304},
       numpages = {1},
       doi = {10.1145/1936652.1936732},
       url = {http://doi.acm.org/10.1145/1936652.1936732},
       acmid = {1936732},
       publisher = {ACM},
       address = {New York, NY, USA}
    }

    List of additional material

    Download

  • Off-Screen Visualization Techniques for Class Diagrams

    Off-Screen Visualization Techniques for Class Diagrams.

    Frisch, M.; Dachselt, R.;

    In Proceedings of the ACM Symposium on Software Visualization 2010.SOFTVIS'10, Salt Lake City, USA.ACM,163--172,2010.10.1145/1879211.1879236

    @inproceedings{fixme,
       author = {Mathias Frisch and Raimund Dachselt},
       title = {Off-Screen Visualization Techniques for Class Diagrams},
       booktitle = {Proceedings of the ACM Symposium on Software Visualization 2010},
       year = {2010},
       month = {10},
       isbn = {978-1-4503-0028-5},
       location = {Salt Lake City, USA},
       pages = {163----172},
       numpages = {10},
       doi = {10.1145/1879211.1879236},
       url = {http://doi.acm.org/10.1145/1879211.1879236},
       acmid = {1879236},
       publisher = {ACM},
       keywords = {UML, contextual view, interaction, navigation, node-link diagrams, off-screen visualization}
    }

    List of additional material

    Video

  • Eine Architektur zum flexiblen Einsatz von gestischer Interaktion

    Eine Architektur zum flexiblen Einsatz von gestischer Interaktion.

    Heydekorn, J.; Frisch, M.; Dachselt, R.;

    In Proceedings of Mensch und Computer 2010.MUC 2010, Duisburg, Germany.Oldenbourg Publishing,321--330,2010.

    @inproceedings{fixme,
       author = {Jens Heydekorn and Mathias Frisch and Raimund Dachselt},
       title = {Eine Architektur zum flexiblen Einsatz von gestischer Interaktion},
       booktitle = {Proceedings of Mensch und Computer 2010},
       year = {2010},
       month = {9},
       isbn = {978-3486704082},
       location = {Duisburg, Germany},
       pages = {321----330},
       numpages = {10},
       publisher = {Oldenbourg Publishing}
    }

    List of additional material

  • Diagram Editing on Interactive Displays Using Multi-Touch and Pen Gestures

    Diagram Editing on Interactive Displays Using Multi-Touch and Pen Gestures.

    Frisch, M.; Heydekorn, J.; Dachselt, R.;

    In Proceedings of the International Conference on the Theory and Application of Diagrams 2010.Diagrams 2010, Portland, Oregon.Springer,182--196,2010.

    @inproceedings{fixme,
       author = {Mathias Frisch and Jens Heydekorn and Raimund Dachselt},
       title = {Diagram Editing on Interactive Displays Using Multi-Touch and Pen Gestures},
       booktitle = {Proceedings of the International Conference on the Theory and Application of Diagrams 2010},
       year = {2010},
       month = {8},
       isbn = {978-3-642-14599-5},
       location = {Portland, Oregon},
       pages = {182----196},
       numpages = {15},
       url = {http://portal.acm.org/citation.cfm?id=1884734.1884759},
       acmid = {1884759},
       publisher = {Springer},
       keywords = {digital pen, interaction techniques for diagrams, mental models, multi-touch and pen interaction, sketching, tabletops, user-centered approach}
    }

    List of additional material

  • Prospects of user elicited gestural interaction techniques

    Prospects of user elicited gestural interaction techniques.

    Heydekorn, J.; Frisch, M.; Dachselt, R.;

    In CHI 2010 Workshop on Natural User Interfaces: The Prospect and Challenge of Touch and Gestural Computing.2010.

    @article{fixme,
       author = {Jens Heydekorn and Mathias Frisch and Raimund Dachselt},
       title = {Prospects of user elicited gestural interaction techniques},
       booktitle = {CHI 2010 Workshop on Natural User Interfaces: The Prospect and Challenge of Touch and Gestural Computing},
       year = {2010},
       month = {4},
       address = {Atlanta, Georgia, USA}
    }

    List of additional material

  • Investigating Multi-Touch and Pen Gestures for Diagram Editing on Interactive Surfaces

    Investigating Multi-Touch and Pen Gestures for Diagram Editing on Interactive Surfaces.

    Frisch, M.; Heydekorn, J.; Dachselt, R.;

    In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces 2009.ITS '09, Banff, Alberta, Canada.ACM,167-174,2009.10.1145/1731903.1731933

    @inproceedings{fixme,
       author = {Mathias Frisch and Jens Heydekorn and Raimund Dachselt},
       title = {Investigating Multi-Touch and Pen Gestures for Diagram Editing on Interactive Surfaces},
       booktitle = {Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces 2009},
       series = {ITS '09},
       year = {2009},
       month = {11},
       isbn = {978-1-60558-733-2},
       location = {Banff, Alberta, Canada},
       pages = {167--174},
       numpages = {8},
       doi = {10.1145/1731903.1731933},
       url = {http://doi.acm.org/10.1145/1731903.1731933},
       acmid = {1731933},
       publisher = {ACM},
       address = {New York, NY, USA},
       keywords = {bimanual input, diagram editing, hand gestures, multi-touch, node-link diagrams, pen interaction, tabletop}
    }

    List of additional material

    Video, Talk

  • Towards a Framework for Supporting Software Modeling Activities Through Novel Interaction and Visualization Techniques

    Towards a Framework for Supporting Software Modeling Activities Through Novel Interaction and Visualization Techniques.

    Frisch, M.; Dachselt, R.;

    In Software Engineering - Companion Volume, 2009. ICSE-Companion 2009. 31st International Conference on Software Engineering.ICSE'09, Vancouver, Canada. 359--362,2009.10.1109/ICSE-COMPANION.2009.5071021

    @article{fixme,
       author = {Mathias Frisch and Raimund Dachselt},
       title = {Towards a Framework for Supporting Software Modeling Activities Through Novel Interaction and Visualization Techniques},
       booktitle = {Software Engineering - Companion Volume, 2009. ICSE-Companion 2009. 31st International Conference on Software Engineering},
       year = {2009},
       month = {5},
       location = {Vancouver, Canada},
       pages = { 359----362},
       numpages = {4},
       doi = {10.1109/ICSE-COMPANION.2009.5071021},
       url = {http://dx.doi.org/10.1109/ICSE-COMPANION.2009.5071021}
    }

    List of additional material

    Additional Poster

  • Towards seamless semantic zooming techniques for UML diagrams

    Towards seamless semantic zooming techniques for UML diagrams.

    Frisch, M.; Dachselt, R.; Brückmann, T.;

    In Proceedings of the 4th ACM Symposium on Software Visualization (Ammersee, Germany, September 16-17, 2008).SOFTVIS '08, Ammersee, Germany.ACM,207 - 208,2008.10.1145/1409720.1409758

    @inproceedings{fixme,
       author = {Mathias Frisch and Raimund Dachselt and Tobias Br\"{u}ckmann},
       title = {Towards seamless semantic zooming techniques for UML diagrams},
       booktitle = {Proceedings of the 4th ACM Symposium on Software Visualization (Ammersee, Germany, September 16-17, 2008)},
       year = {2008},
       month = {9},
       isbn = {978-1-60558-112-5},
       location = {Ammersee, Germany},
       pages = {207 -- 208},
       numpages = {2},
       doi = {10.1145/1409720.1409758},
       url = {http://doi.acm.org/10.1145/1409720.1409758},
       acmid = {1409758},
       publisher = {ACM},
       address = {New York, NY, USA},
       keywords = {UML, diagrams, interaction techniques, models, semantic zooming, visualization, zoomable user interface}
    }

    List of additional material

    Additional Poster

  • Enhancing UML sketch tools with digital pens and paper

    Enhancing UML sketch tools with digital pens and paper.

    Dachselt, R.; Frisch, M.; Decker, E.;

    In Proceedings of the 4th ACM Symposium on Software Visualization (Ammersee, Germany, September 16-17, 2008).SOFTVIS '08, Ammersee, Germany.ACM,203-204,2008.10.1145/1409720.1409756

    @inproceedings{fixme,
       author = {Raimund Dachselt and Mathias Frisch and Eike Decker},
       title = {Enhancing UML sketch tools with digital pens and paper},
       booktitle = {Proceedings of the 4th ACM Symposium on Software Visualization (Ammersee, Germany, September 16-17, 2008).},
       year = {2008},
       month = {9},
       isbn = {978-1-60558-112-5},
       location = {Ammersee, Germany},
       pages = {203--204},
       numpages = {2},
       doi = {10.1145/1409720.1409756},
       url = {http://doi.acm.org/10.1145/1409720.1409756},
       acmid = {1409756},
       publisher = {ACM},
       address = {New York, NY, USA},
       keywords = {Anoto, UML, sketching, software design}
    }

    List of additional material

    Additional Poster

  • Benefits of interactive display environments in the software development process

    Benefits of interactive display environments in the software development process.

    Frisch, M.; Dachselt, R.;

    In Proceedings of the International Workshop on Cooperative and Human Aspects of Software Engineering 2008.CHASE '08, Leipzig, Germany.ACM,53-56,2008.10.1145/1370114.1370128

    @article{frisch-2008-CHASE,
       author = {Mathias Frisch and Raimund Dachselt},
       title = {Benefits of interactive display environments in the software development process},
       booktitle = {Proceedings of the International Workshop on Cooperative and Human Aspects of Software Engineering 2008},
       year = {2008},
       month = {5},
       isbn = {978-1-60558-039-5},
       location = {Leipzig, Germany},
       pages = {53--56},
       numpages = {4},
       doi = {10.1145/1370114.1370128},
       url = {http://doi.acm.org/10.1145/1370114.1370128},
       acmid = {1370128},
       publisher = {ACM},
       address = {New York, NY, USA},
       keywords = {UML, diagrams, interaction techniques, models, multi touch, semantic zooming, software development process, visualization, zoomable user interface}
    }

    List of additional material