All.md



========================================================
1fA35u
1. fixes the code carefully although it is not clear
[F] would prefer the table to be left aligned and not spanning the whole screen
2. participant takes time to understand the code and explains it thoroughly (but not on a high level, i.e. only explains what each rule is doing in terms of what combinations and checks are done).
participant shows a correct understanding of how the rules are working, also answers the questions correctly 
explanation and back and forth about what the rule is doing
"after using nev I think more easy, especially when thinking about more complicated programs"
3. finding the path of the transitive closure done correctly and explained without problem
replacing the rule also done very quickly -- clear understanding of the tool
4. performance issue walkthrough shows the [F] bug with the table count queieing, but the participant easily identifies the most time consuming rule.
participant checks the time and number of facts easily, too.
fix to the performance issue on the first recursion of the transitive rule (should be the second), so we take some time to check if they can reach the correct solution on their own
checking the amount of facts goes without problems. 
[F] "The one bug is a bit confusing, right." (count queues)
5. extending the path goes without problems
finding the path in the table leads to finding the [F] that the table below sometimes doesn't show all the facts. 
6. super quick, gets it immediately
[F] could show predicates / rules out of sight at the bottom as a single line or so.
7. "It doesn't scale very well. Still much better than just using nemo"




========================================================
3vVQWR
1. we explained how datalog rules work (left implied by right), what facts are, and that ? denotes variables
   "let me get back into this", it has been a while since participant worked with this type of code
   completed renaming task, but "error message a little confusing, maybe lack of familiarity from my side". Could be something like unnamed variable"
2. what does program do? unclear
   checks for minimum of c 
   unclear where rules get their predicates from 
   open tables
   correctly tells where the rules get initialized (e predicates), and that tables are identical
   [F] create floating rules
   does not name "transitive closure", but indicates in broader terms what the program is doing (min distances)
   renaming takes a bit longer
   "I think here (nev) it's more intuitive"
3. completes tasks, we indicate easier way (see path at bottom)
   "I don't think that was helpful" (the right of the t(1, 10, X) idea)
4. appreciates focus on rule
   [F] compute time could be sticky between trees, and shown on table view
   [F] don't show focus on rule button if tree is already single rule
   identifies transitive closure as taking the longest, 
   corrects the performance issue
5. finds lowest cost, opens a lot of edges
   realizes this solution without much effort
6. easily completes task
7. same as before. 

General feedback
"Pretty intuitive, very nicely done"
"something to move this around" 
[F] cut also from the rules
[F] zoom on tree (sidebar size is limited with the indentation)





========================================================
5EGu2C
participant mumbles a lot.
1. everything clear with this, "errors look good."
2. what does the program do? A: dunno yet
  - is a bit confused by terminology, but completes the tasks with little issue otherwise
  when selecting position of current root: "both are the same, so it shouldn't matter", which is correct! 
  - understands un/restrict
  [F] bug: ghost preview remained after hovering on the rule
  explains for a long time about what parameters go where and get summed but does not reach an answer about the rule's meaning.
  "renaming is hard" 
  "it is helpful to have this tree witht he derivations, and adding/removing rules, also restricting and unrestricting"
  "it would be nice to have some definitions of aggregates and so on"
  "SUM is only over the Cs" language issue
3. reaches the path relatively easily. (1,4,6,9,10)
  initially doesn't see the shorter path, but then finds it. 
  "nev also helps a lot since I can plug in the values and see the relations that lead to certain results and exchange them". 
4. "it would be nice to see what the maximum or total amount of pages is"
  solution to improve performance reached quickly. 
  "finding the fix was not hard to imagine"
  "[nev] helps know how many times are derived and the execution time per rule (...) it is very helpful in this regard"
  [F] it would be nice to check the total time (per rule). 
5. participant found the path very easily. 
  [F] bug: facts apparently don't always show up in the table at the bottom
  "it was a bit cumbersome to not be able to keep count, but otherwise easy to do".
6. task on finding path 1 to 100 completed easily. 
  "exactly as simple, but a bit cumbersome to follow the path"
7. also understands the task easily. How about scalability? 
  "nev is probably not made to read a path along such a long derivation, but even then it works well" 
  "at some point it will become unmanageable"

General feedback:
- sidebar has some problems, 
"visualization helps a lot, at least [it is] the most intuitive mechanism, maybe".




========================================================
f8M412
1. solves the errors, doesn't look too deep into the program yet
2. doesn't spend time trying to understand the program, jumps directly into derivations. 
inspects the UI in close detail. 
asks whether the reconstructed tree should already be for the same fact (already knows the tree is unrestricted)
"It is difficult to think aloud and also think". "I want to look at a few examples".
"I'm not good at understanding things when I think I'm under pressure" 
"I think the names are not bad because they are short"
"I think for me it is easier to understand it just from the program, because I've never used the interface. (...) But I actually think it's a pretty cool feature especially when there are not too many rules". 
"For me it feels like something new and I was very impressed that it just shows the derivation"
3. finds path but hesitates a lot with the interface. Probably due to the pressure described earlier. 
[F] would like to add a new rule by typing / i.e. had not understood that it only shows rules from the program. 
"It is hard for me but I think after exploring it more it becomes very useful"
4. finding the time it took to compute by rule was a bit difficult
a bit clearer guidance is needed for how the computing time is calculated/shown per node. 
5. finds the path by cutting rules above, perceives the tool to be intuitive due to the repeating pattern of the tree towards the left. 
6. finds the task easily, a bit confused by the change of names. 
7. faced with the largest graph, the question comes up that only reading through the graph is not that useful and would perhaps prefer e.g. a long string.
[F] documentation! (referring to nemo, but also true for nev)




========================================================
HLApIN
1. "what does 'unsafe variable' mean?" 
"I have no clue what I changed, just tried to do something for the error to disappear"
2. what does the program do? right now, no idea. 
corrections to the code by the participant lead to a more complicated tree when computing for the transitive closure on the minimum path. 
part. is a bit confused by the 'sum' aggregate, seems to be taking a bit longer to understand the trees
"so what does the 'r' predicate do?" > interviewer: that's for you to figure out later! 
despite the difference in the reconstructed tree, the participant remembers that the difference is the "unrestrict"
understanding the r predicate takes longer than intended, could be due to the different fix used at the start
"so this is... very clever!" when told that the renaming feature distinguishes between predicates and variables (symbol rename)
"the UI is very easy to understand, I would say"
3. finding a path. With pretty much no help, the participant finds the correct view (focus on fact on transitive rule), but gets confused with further cutting the tree
to find a shorter path, participant gets a bit confused again with the remove/add rule
4. the performance issue was corrected relatively easily but some interface bugs still are a notable bother: e.g. the queued requests on the pagination, showing only the time for the shown rules, etc. 
there's again a small confusion between constants and parameters when doing the second optimization. 
"I don't know other tools that show the times like this"
5. finding the path took a while, since naturally it is more bending what nev can do 
[F] a bug was identified with the pagination [check video around ~1hr18min], shortly before jumping to next task
6. quickly finds the path this time. 
7. "what is maybe weird is that it grows to the left" but after thinking of a way to change this, thinks there's no point in changing that
"would be good to have something like a list"
and what about beyond this example? "it's still pretty clear" paraphrasing: as you can just go for what you are interested in.

"in general I'm happy with this tool because it is easy to navigate and do stuff"
"by just reading the rule it is very hard to see what it does"
[F] color the variables



========================================================
if7X19
1. no issues with the syntax highlighting
2. has a pretty good understanding of the code, already guessing it is about a graph
participant often clicks on the rule nodes, we ask what they expect to happen when they click there: "maybe a map that shows how this [predicate below] connects to this [predicate above]"
there are some language issues that make the walkthrough take a bit longer, as the participant struggles to follow instructions such as "click the crosshair icon on top of the rule"
reconstruction of the tree goes well and the participant describes well the process of getting there. Not clear what the relation with Unrestrict is
"After I unrestrict it, I can't do operation like restricting this [the top of node]", so interviewer explains again that focusing on a row still allows this. 
participant fluctuates between showing good understanding of rule-based programming and getting stuck with UI and spoken instructions. 
3. understands the path feature quickly and explains the tree derivation 
still takes some time to spell out the path due to the question if we need "the shortest" or "a shorter" path. 
4. "it all takes 0 ms, I think there's something wrong" - another reminder that showing times for only the visible rows is very difficult to understand intuitively
the performance optimization takes a long time and is very much guided by the interviewer. 
Once the changes are finished the participant takes still quite some time to do tree manipulations
[F] nemo could provide suggestions on how to improve the program
5. creating a path was very difficult 
participant at one point looked at all the paths from 1 to 10, which would have been another way to more quickly get to a path with more edges (e.g. since a higher cost likely means more edges, too)
6. "I thought you would provide some features to see this more easily"
describes some way to see paths that follow a specific pattern, perhaps as some form of log on the right or so.
7. participant finds it funny as most and does believe that there may be easier ways to print out the path in more scalable

mentions font weight and size, as you can also adjust this in vscode
participant requests more documentation and the ability to load the data not directly in the code but elsewhere
"I'm not quite sure what's the point of this" (the indented tree on the left)
would like the lines to be more distinguishable, as in the vscode extension GitGraph
"I'm very sorry, sometimes I feel like I didn't understand your guidance"



========================================================
LJe8zG
Experienced candidate. Takes time to read the code before doing anything. 
1. "It was but of course it would've been much harder without proper syntax highlighting". 
"It follows what I expect an editor to look like"
"It's also boring, which I consider to be a good thing" 
2. What does the code do? Already has the right idea: computing minimal cost paths
"This makes sense!" 
Unrestricts to understand how the tree shape/pattern works (does it very often!)
Reconstruction of the tree very fast, immediately understands what to do.
Mentions that without seeing it in action, they wouldn't know what the green button does, but after knowing, it's very easy to use. 
Figuring out the meaning of a rule: "basically it computes a transitive edge" 
What is table m: "the length of the shortest path [in terms of cost] from each node"
Already knows F2 is the key to rename
[F] bug: the renaming has some troubles 
"not very hard because I have seen programs like this before" 
"with nev, I think it makes it easier because I can see how the values flow through the program" "it definitely increased [the value towards easiest in SEQ] substantially". 
"not obtrusive, it is minimalistic, so very much on the supportive side"
"I like the snappiness (...) once I understood what the buttons do I can easily perform any task"
"It's not usual, I don't think I've seen interfaces like this before"
3. "maybe I shouldn't have picked such long names!"
"the path is already traced" (edges at the bottom)
shortest (not min cost) path. Also thinks of trying to switch another of the rules (transitive). This however, leads to a bug.
pleasant: "it is tedious, not difficult, but it is just not pleasant" (as a task)
4. performance: 
[F] bug: queueing several changes to the pagination
without checking specific times, already has an idea of which rule is doing the heavy lifting ("this rule is quadratic")
optimization almost instant, but renaming start variable brings some trouble: "?start = 1"
"it's hard to use it for this type of thing" "the interface doesn't center around rules (...) you have to connect which ones are influenced by what"
"nemo shows the % time for each rule"
"it's not the way I look at the program usually (...) but not the right tool for the job. I did like the construction of these (tree) shapes, which I think is easy to do" 
didn't like the see more option ("if you hadn't told me I would not have seen it (...) and this interface just doesn't work" (referring to the bug in the pagination counts))
5. constructs the path very quickly but misses (10 nodes: 9 vs 10 edges)
"you would write a specific program for this" (saying how it would be pretty difficult without nev)
6. "its easier now because I am more used to the interface". 
"I guess it would have been even harder without but you can say it's just as easy (with nev)"
7. "In this case it's just tedious either way"
"It's a bit underspecified (the tasks) because I'm typically not interested in the path, I want to do something with it."
"I would expect undo/redo on top left because that's where I look" Could not find the "Unrestrict" button when told about it. 
[F] I would like to declutter rules that don't matter to me. Something like hiding (in explore mode) for all rules of a certain kind. 
"Pagination thing is very very sad" 




========================================================
n166ua
1. paraphrased: first I thought about trying to figure out what the letters meant, or why the triplets were there. But that was not needed to solve the task"
2. what does the program do? recursive model that sums the last number of the triplet to check against the sum of all possible values. 
(very good idea without using nev)
participant mentions they would also like to look at the code again 
focus on rule and notices that the tables are the same because it's the first step of the recursion. 
also understands and mentions that it ressembles the unrestricted tree. 
correctly understands now that r is a transitive rule, as well as the limit (t)
correclty understands the program, too! min cost path from 1 to all
renaming works well 
"I was unfamiliar with solving tasks with this type of visualization"
3. finding the path works out smoothly but the participant first spends some time looking at the path rule on its own (from 1 to 6)
spends some extra time trying to find an even shorter path after replacing one "path" to "edge"
"I'm getting for familiar with nev so it gets more easy over time"
4. performance task a bit confusing with whether the time in the green nodes corresponds or not to the rule below them
had to give away a bit the answer to performance issue but then shows a lot how it is a solution that makes a lot of sense. 
5. constructing length 10 path was very easy but it wasn't as efficient
6. compares how in task 3 the tree layout is balanced, whereas in this one it is skewed to the left
7. easy to get to the solution, but does it find it funny how the trace shows. 
"it would help if there was an additional type of query" [F] filter by predicate name
"not as good scalable as some other solutions, I would think"

"For a total newcomer it would be nice to see the corresponding program and have a direct to which parts of the program corresponds to which tree shape"
"Of course there were bugs, but you're probably aware of them"




========================================================
OFUWOI
1. identifies the problem with the variables and already fixes it correctly.
2. language is a bit of a barrier in following the instructions
the thinking aloud is a bit difficult, too. The participant inspects the tables in a lot of detail without explaining their reasoning. 
reconstructs the tree correctly! but does not recognize the unrestricted nature of the tree. After telling them, they claim its clear. 
with a bit of assistance, the participant figures out the transitive rule and soon after explains the code. Slightly misunderstands which rule minimum is working over, but understands that the code is computing minimal paths. 
code renaming issues with keyboard layout (english vs german)
3. a bit of confusion with whether the rule is computing direct paths or any path (multiple nodes)
the path is constructed along several trees of the lower levels of the recursion (e.g a tree for 1 to 6 [1, 4, 6] then another for 6 to 10 [6, 9, 10])
interviewer explains that this can be done by looking at the higher level of recursion to see all edges in the same tree
although there are some issues understanding how to find the alternatives paths, eventually the solution is reached and the participant explains the interface showing that they correctly understand it.
participant gives a lot of praise to the tool and claims "To find suboptimal paths it is a bit confusing, but it is probably a me problem" - interviewer says that they should just write about their own experience (i.e. in the scores)
4. finding the slowest rule takes long, although the participant does not encounter the bugs of the interface
solution to the performance issue not achieved. 
in a mix of tiredness and again language problems, it appears the participant does not remember the interface buttons too easily. 
5. the concept of recursion in the tree is really not clear -- participant doesn't seem to understand why the first rule application to get a path is the copyofe initial instance. 
[F] line break was confusing in the modal! seeing the rules one on a single line and the other in multiple led to a lot of confusion
[F] bug: table at the bottom does not show all results when updating count some times
[F] would like to add n rule applications at once
6. [F] cut parts of the tree that are the same to reduce complexity. 
7. [F] quickly jump to the bottom or top
[F] minimap / miniview of the whole trace to be able to navigate faster




========================================================
swMccD
1. goes without issues. 
2. what does the program do? takes a while but it is not clear
  - speaks very softly 
  - goes through the tasks pretty effortlessly
  [F] opened nodes should not close up when focusing on fact on the root
  - after the task what does the graph do? A: calculates paths with under a certain value
  [F] bug? renaming variables. Small issue with the F2+double click
  "the one with the graph looked good, graph expanding was smooth"
  "the buttons on the right are a bit far, so as one focuses on the center they are easy to oversee"
  "replacing (if it worked) would be pretty convenient"
3. [F] "if I could copy-paste [parts of the graph] next to each other"
  - how to restrict query: focus on a fact, EDIT query. 
  Deleting from the upper rule, but "I always try to right click, or use the keyboard first" 
  "you have to learn it once and then it is clear, but yes, one has to learn it"
4. [F] one could think about paginating only when the program is taking too long to return results, instead of always having to increase pagination. A global default count would also help. 
  - fixed the performance issue without any issue. Participant is careful with the execution. 
  [F] replace only some predicates (children) of a rule
  "It is a bit confusing that you only see the time for the facts that are displayed"
5. "Without Nemo/Nev this would be much harder, because you have to look at all possible paths"
6. It is the same, yes, you have to look a bit more but it is fine. 
  [F] probably needs a filter function on the table. 
  "With a 1000 nodes this would be difficult"
7. "It is just a bit more time intensive". 
  Would like to extract content from the trace [F] export subselection from the tree? 

General commentary/discussion:
  - A manual would be very good
  - If you know how to use it, then it is pretty good. But you need this first "introduction" to it. 
  - Copy/pasting: you want to have two disconected trees and derive something from them both. "Instead of having two trees and derive a conclusion, with the current version you have to first make the conclusion and then  




========================================================
TYfxOA
1. confident, directly to the errors in the code, and talks aloud nicely. Solves it by removing one of the troublesome variables.
"error messages were quite useful, didn't need to look at the rule or understand it at all" 
"it would have taken me longer without the messages + syntax highlighting, but I still don't know what the program is doing, of course". 
2. visually executes the program, talks about recursive rules and about the possible differences with the intended solution after removing the +c2
does not figure out the program without tracing. 
very actively tries out the features in nev as they are presented.
"there are multiple rules with r, but why not the other" (not showing other possible derivations)
follows the explanations without issue, and answers the questions correctly (e.g. identical tables, which predicates go into initialization of each rule) 
adding and removing rules also not a problem, nor is identifying unrestricted vs. restricted trees. 
"I think I would spend more time looking at the rule and not at concrete instantiations" (when asked to figure out what the specific rule is doing)
despite the change in the sum computation, participant correctly describes the meaning of the rule and also realizes the sum should not have been changed. 
explain what the program does now: find minimum cost to get to specific nodes. [correct]
renaming goes smoothly.
"[nev] helps to undestand the structure of the rule"
"after knowing this [that adding parts of the tree restricts it further], it is very intuitive"
"useful for debugging why a rule doesn't trigger"
"this can give you a very quick view of which rules led up to this point (...) but to understand each rule you still need to look at the rule itself"
3. trying to construct the path tree by adding further branches at the bottom, eventually finds the new root necessary to create the path. 
reaches the path easily.
"what I would actually do is change the program to take into account the path length instead of the cost"
after a little hint (restricting rule by adding an instantiation), they find the path.
"especially if you have much bigger graphs, this would be much easier with nev" (finding a shorter path)
"I think I wouldn't have gotten to this trick of instantiation on my own" because it makes some assumptions that there is a shortcut from the first edge.
4. finds the rules much easier than by focusing on the rule: just cuts to only have the predicate
first instinct is to add inequalities to rule (0.4 second gain, which is already good). With an additional hint, also fixes the double transitive rule use. 
finds the different amount of facts easily. 
"I find it very cool that it still shows times even for restricted queries"
[F] in nemo-web already with a heatmap encoding one could show which rules are taking the longest
"without nev this would be very hard when rules depend on each other"
"the only thing I found a bit confusing is the (see more) as I would have to trust that it tells me if there is more"
5. to find the minimal path, participant uses the min rule at root after adding the additional edges to get to the length 10 path. 
since the participant enforced no loops as one of the optimizations, they get a different answer with a higher cost. 
[F] text selection goes away if one clicks on the search bar. 
"it would get a bit cumbersome with paths of 20 (length) but it still works very well and would be useful in other ways"
6. despite the slight differences between the prepared version and the longer graph version
"I think this was really manageable and way easier than how I thought it would be"
7. "I mean it's still very responsive -- but reading it off... yeah"
[F] bug: participant also finds the issue of the side bar on the left, that it doesn't grow longer than half of the screen. 
[F] color different rules with different colors (to see from afar whether one or some of the rules are not the same) 
"Seems like a very powerful thing when developing a Datalog program"
"many things you can do with Datalog to build these visualizations, but this seems a lot faster".
(about difficulty) "It depends what you want to do with this, if you want to read it off, it is more work"
[F] also Search for specific predicates or full strings





========================================================
whgCgP
1. doesn't run or see the error messages in the editor but fixes the program based on what seems right after looking at the code for a while
2. what does the program do? "I don't see it right now"
[F] download buttons maybe too close to table headers in nemo web
answers pretty confidently (and correctly) about the derivation tree, seems easy to understand. 
correctly understands the r rule (transitive closure) 
understands t as an upper bound to prevent infinite loops
3. gets to the path very quickly reading the tree in nev! (great success)
also finds the shorter (more costly path) quickly and compares the total cost
"it was quite straight forward to get the path, without [it would] probably not [be] so easy"
4. performance task a bit less smooth, due to the interface not communicating very clearly how long each rule takes
once identified, fixes the program almost immediately! very good understanding
"not as intuitive to get the times for how long the rules [took], but it is helpful to know which rules to focus on"
5. to find the path, participant uses the add rule above feature
"it was not so hard. Without, probably more complicated"
6. also no problem. Doesn't say much. 
7. no problem either. "A bit overwhelming to see such a big tree, but zooming in and out is quite intuitive, so it's still good to keep an overview on how things work"
"Without any guidance it would be much more complicated to figure out, but I guess that's what tutorials are for"





========================================================
WLSaM8
1. "Not exciting because I think it would be distracting if there was a lot going on"
2. identifies that the code is working with a graph
shows awe when looking at nev, completes the tasks without much issue
participant looks at various features in detail, curious about what they do and triggers various features for them. 
"I would need some more complete documentation to know exactly what to do" 
"Just by clicking you can find out a lot, but maybe with mouse over, help or documentation this would be better"
3. finding the path is done rather quickly, but the participant rightfully compares the experience to just using an algorithm specifically for this (e.g. dijkstra)
4. finding the rule which takes the longest takes some time due to the UI issues and associated
uses the count of the pagination to trigger reload and check new execution times
[F] would like to choose one of the parameters from the predicate nodes
[F] reload nev with the existing shape (if possible, i.e. if the code has not changed too dramatically). Requires encoding type 2 in url somehow
[F] difficult to see in one go which rule is eating up the most time 
[F] clear - confusing: without a dedicated resource it is difficult to figure out these
[F] line count on the indented tree view 
5. finding the path not the intended solution and thus it is a bit confusing
[F] bug: table doesn't always show all results after updating count
[F] measuring nesting degree if a regular pattern is found in the tree
6. still feasible but of course a list would be easier to use. 
"This being a graphical tool, there maybe inputs that are just not easy to read"
[F] export graph into text file / perhaps add a copy as text to the indented tree view.
7. "This displays what I mentioned: as a graphical tool, reading these is not feasible anymore after a certain size"
[F] show specific values in the indented tree instead of the open variables. 
[F] statistics to the overall tree or to a selected rule. e.g. accumulated time, how often it was nested, etc. 
"It's pretty clear that this isn't intended to handle such big trees"
"For what it's intended for, it's pretty usable"
"I would be hard pressed to use this as it is without you explaining it to me"





========================================================
zBcJD4
1. likes the underlines in red and error message, saying that the problem is in the head and not body. 
"would have been even nicer if you could hover on the error message and it showed the line that it corresponds to"
2. what does program do? has an idea of what everything is doing but doesn't guess what the program does. 
[F] do checking directly in the table 
building derivation trees: does it without issues. 
does not notice the difference between restricted and unrestricted. Opens everything to check what the specific rule does. 
participant not super familiar with terminology 
"shortest path" but not for length, for lowest weight. 
[F] bug: renaming might be related to the language server.
"what I found quite unintuitive at first at least is how the structure is displayed" 
[F] edges could go to the text position where the predicate is used in the rule. Think about conditions versus other types of applications. 
3. finds the query for the transition (transitive(1, 10, x)) but does not see the path at the leaves 
[F] source of confusion when trimming could be that the position of the root remains 
[F] graying out when focusing on fact [magnifying glass] could work the same as when cutting using a button.
"if you spend some time with it then it becomes clearer what to do"
4. [F] "show all" button
performance issue not fixed on their own (several hints provided, e.g. recursion, only changing one of the calls, etc) 
still takes a while to get to specific buttons wrt tasks. 
5. adds the edges pretty easily but gets confused by the expected amount of edges, and by tracing facts on the leaves. 
[F] expand a number of times versus just once. Could be cool if there is a recursive rule. Something like "open n times or until you can't open anymore"
6. unconvential way to solve the task, by creating the path from edge instead of just checking for the minimal table.
7. completes task easily. 
[F] recursive rule different representation.
[F] highlight where predicates are used in the rules. 
[F] reposition arguments from the nodes. 
[[there might have been a misunderstanding in the last questions since they include nemo but the participant might have meant a distinction between only the vis and only an editor]]
