prep/notes for beta test, get feature weights/rates

7 files changed, 168 insertions, 66 deletions

diff --git a/inc/help.inc.php b/inc/help.inc.php
index 2bd511e..9f2e7f6 100644
--- a/inc/help.inc.php
+++ b/inc/help.inc.php
@@ -1,41 +1,47 @@
 <p>Press the 'Next' button to <strong>submit</strong> your post-edit and to request the next segment for post-edition.
-Alternatively, in the textual interface, you may just press return when you finished the post-edit ('Target' text area is in focus).</p>
+Alternatively, in the textual interface, you may just press return when the post-edit is finished ('Target' text area is focused).</p>
 
-<p>The session can be paused at any time and continued later; However, if you have to pause your session, wait until the activity notification disappears and then press 'Pause', as we are collecting timing information. You may also just reload this site and re-request the segment to reset the timer.</p>
+<p>The session can be paused at any time and continued later; However, if you have to pause your session, wait until the activity notification disappears and then press 'Pause', as we are collecting timing information. You may also just reload this site upon your return and re-request the segment to reset the timer.</p>
 
-<p>Please use only a <strong>single browser window</strong> at the same time. Going back to earlier examples is not possible, please take great care when interacting with the system.</p>
+<p>Please use only a <strong>single browser window</strong> per session at the same time. Going back to earlier examples is not possible, please take great care when interacting with the system.</p>
 
 <p><span style="border-bottom:1px solid #ccc">Instructions for the graphical interface:</span></p>
-<p>To submit a post-edition in the graphical interface all phrases have to be marked as finished.</p>
+<p>To submit a post-edition in the graphical interface all phrases have to be marked as finished (phrases are highlighted in dark gray color).</p>
 <ul>
-  <li><strong>Moving around:</strong> Press <strong>'S'</strong>, then select phrases using the arrow keys.</li>
+  <li><strong>Moving around:</strong> Press <strong>'S'</strong>, then select phrases using the arrow keys. This is the default mode.</li>
   <li><strong>Editing text:</strong> Double click on a phrase or press <strong>'E'</strong> to edit the contents of the current phrase. Press 'Return' to save.</li>
-  <li><strong>Reordering of target phrases:</strong> Press <strong>'M'</strong>, then use the arrow keys to move the selected phrase. Press 'Return' to fix the position.</li>
-  <li><strong>Mark phrase as finished:</strong> Press <strong>'Return'</strong> to mark phrases as finished (press 'Return' again to undo). Moving, editing or deleting of finished phrases is not possible.</li>
-  <li><strong>Adding target phrases:</strong> To add a phrase right next to the currently selected one press <strong>'A'</strong>.</li>
+  <li><strong>Reordering of target phrases:</strong> Press <strong>'M'</strong>, then use the arrow keys to move the selected phrase. Press 'Return' to fix its position.</li>
+  <li><strong>Mark phrase as finished:</strong> Press <strong>'Return'</strong> to mark phrases as finished (press 'Return' again to undo). Moving, editing, aligning or deleting of finished phrases is not possible.</li>
+  <li><strong>Adding target phrases:</strong> To add a phrase right next to the currently selected one, press <strong>'A'</strong>.</li>
   <li><strong>Removing target phrases:</strong> Press <strong>'D'</strong> to delete the currently selected phrase.</li>
-  <li><strong>Adding/removing alignments:</strong> Select a source phrase by clicking on it, then click on a suitable target phrase to connect or disconnect. Click the selected source phrase again to cancel.</li>
+  <li><strong>Adding/removing alignments:</strong> Select a source phrase by clicking on it, then click on a target phrase to connect or disconnect it with the selected source phrase. Click the selected source phrase again to cancel.</li>
   <li><strong>Undo:</strong> Press <strong>'U'</strong> to undo alignments, text edits and deletion of phrases.</li>
   <li><strong>Reset:</strong> Click 'Reset' button to start from scratch.</li>
 </ul>
 
-<p>The interface was tested with Firefox 31, 38 and 43.</p>
+<p>The interface was tested with <strong>Firefox</strong> 31, 38 and 43.</p>
 
 <p><span style="border-bottom:1px solid #ccc">Known issues:</span></p>
 <ul>
   <li>The width of the canvas of graphical editor may be to small when adding a lot of phrases.</li>
   <li>The in-line editor may change height and span several lines.</li>
   <li>When editing phrases that have no contents, the input box is lower than normal.</li>
-  <li>Post-edits in the session overview may be partially tokenized due to phrase segmentation.</li>
-  <li>Horizontal scrollbar doesn't follow highlighted phrase.</li>
+  <li>The horizontal scrollbar doesn't follow highlighted phrase.</li>
   <li>Mouseover is not detected for undoing.</li>
   <li>The interface only works with Firefox.</li>
 </ul>
 
+<p><span style="border-bottom:1px solid #ccc">Notes on debugging:</span></p>
+<ul>
+  <li>If the <em>string</em> of the translation is not changed (compared to the original MT output), there will be no pairwise ranking update. </li>
+  <li>New rules are added (and known rules are annotated) in a leave-one-out fashion. Amongst other things, this implies that 'new' rules will first receive a weight upon their second occurrence. As OOV rules are added prior to translation, they are added immediately.</li>
+  <li>Most tables are sortable (click on respective column in header row).</li>
+</ul>
+
 <p class="tiny">
   Support: <a href="mailto://simianer@cl.uni-heidelberg.de">Mail</a>
 </p>
-<p class="tiny">Session: #<?php echo $_GET["key"]; ?> |
+<p class="tiny">Session: <?php echo $_GET["key"]; ?> |
   <a href="http://postedit.cl.uni-heidelberg.de:<?php echo $db->port; ?>/debug" target="_blank">Debug</a>
 </p>
 
diff --git a/index.php b/index.php
index 74d64a3..d10bc2e 100644
--- a/index.php
+++ b/index.php
@@ -26,37 +26,36 @@
 <p>Beta test:
 <select class="small">
 
-<optgroup label="________________">
+<!--<optgroup label="________________">
   <option value="product_de-en_toy_example" onclick="document.getElementById('key').value=this.value;">toy example</option>
-</optgroup>
+</optgroup>-->
 
-<optgroup label="________________">
-  <option value="product_de-en_toy_example" onclick="document.getElementById('key').value=this.value;">toy example</option>
-  <option value="product_de-en_beta_test_A" onclick="document.getElementById('key').value=this.value;">A de-en</option>
-  <option value="product_en-de_beta_test_A" onclick="document.getElementById('key').value=this.value;">A en-de</option>
-  <option value="product_de-en_beta_test_1_A" onclick="document.getElementById('key').value=this.value;">A* de-en</option>
-  <option value="product_en-de_beta_test_1_A" onclick="document.getElementById('key').value=this.value;">A* en-de</option>
+<optgroup label="Session A">
+  <option value="product_de-en_beta_test_A" onclick="document.getElementById('key').value=this.value;">de-en</option>
+  <option value="product_en-de_beta_test_A" onclick="document.getElementById('key').value=this.value;">en-de</option>
+<!--  <option value="product_de-en_beta_test_1_A" onclick="document.getElementById('key').value=this.value;">A* de-en</option>
+  <option value="product_en-de_beta_test_1_A" onclick="document.getElementById('key').value=this.value;">A* en-de</option>-->
 </optgroup>
 
-<optgroup label="________________">
-  <option value="product_de-en_beta_test_B" onclick="document.getElementById('key').value=this.value;">B de-en</option>
-  <option value="product_en-de_beta_test_B" onclick="document.getElementById('key').value=this.value;">B en-de</option>
-  <option value="product_de-en_beta_test_1_B" onclick="document.getElementById('key').value=this.value;">B* de-en</option>
-  <option value="product_en-de_beta_test_1_B" onclick="document.getElementById('key').value=this.value;">B* en-de</option>
+<optgroup label="Session B">
+  <option value="product_de-en_beta_test_B" onclick="document.getElementById('key').value=this.value;">de-en</option>
+  <option value="product_en-de_beta_test_B" onclick="document.getElementById('key').value=this.value;">en-de</option>
+<!--  <option value="product_de-en_beta_test_1_B" onclick="document.getElementById('key').value=this.value;">B* de-en</option>
+  <option value="product_en-de_beta_test_1_B" onclick="document.getElementById('key').value=this.value;">B* en-de</option>-->
 </optgroup>
 
-<optgroup label="________________">
-  <option value="product_en-de_beta_test_C" onclick="document.getElementById('key').value=this.value;">C en-de</option>
-  <option value="product_de-en_beta_test_C" onclick="document.getElementById('key').value=this.value;">C de-en</option>
-  <option value="product_en-de_beta_test_1_C" onclick="document.getElementById('key').value=this.value;">C* en-de</option>
-  <option value="product_de-en_beta_test_1_C" onclick="document.getElementById('key').value=this.value;">C* de-en</option>
+<optgroup label="Session C">
+  <option value="product_en-de_beta_test_C" onclick="document.getElementById('key').value=this.value;">en-de</option>
+  <option value="product_de-en_beta_test_C" onclick="document.getElementById('key').value=this.value;">de-en</option>
+<!--  <option value="product_en-de_beta_test_1_C" onclick="document.getElementById('key').value=this.value;">C* en-de</option>
+  <option value="product_de-en_beta_test_1_C" onclick="document.getElementById('key').value=this.value;">C* de-en</option>-->
 </optgroup>
 
-<optgroup label="________________">
-  <option value="product_de-en_beta_test_D" onclick="document.getElementById('key').value=this.value;">D de-en</option>
-  <option value="product_en-de_beta_test_D" onclick="document.getElementById('key').value=this.value;">D en-de</option>
-  <option value="product_de-en_beta_test_1_D" onclick="document.getElementById('key').value=this.value;">D* de-en</option>
-  <option value="product_en-de_beta_test_1_D" onclick="document.getElementById('key').value=this.value;">D* en-de</option>
+<optgroup label="Session D">
+  <option value="product_de-en_beta_test_D" onclick="document.getElementById('key').value=this.value;">de-en</option>
+  <option value="product_en-de_beta_test_D" onclick="document.getElementById('key').value=this.value;">en-de</option>
+<!--  <option value="product_de-en_beta_test_1_D" onclick="document.getElementById('key').value=this.value;">D* de-en</option>
+  <option value="product_en-de_beta_test_1_D" onclick="document.getElementById('key').value=this.value;">D* en-de</option>-->
 </optgroup>
 
 </select>
diff --git a/js/debug.js b/js/debug.js
index 3fab4c7..cd5882b 100644
--- a/js/debug.js
+++ b/js/debug.js
@@ -1,5 +1,15 @@
-$().ready(function()
+var poll = function (url_prefix)
 {
+  setTimeout(function(){
+     $.get(url_prefix+"/status_debug").done(function(response){
+       $("#status").text(response);
+       poll(url_prefix);
+     });
+  }, 3000);
+}
+
+$().ready(function() {
+
   // send commands using ajax
   $(".ajax").each(function(x) {
     $(this).click(function() {
@@ -10,15 +20,19 @@ $().ready(function()
   })
   $("#features").click(function() { if (this.value=="Feature") this.value = ""; });
   $("#features").focusout(function() { if (this.value == "") this.value = "Feature"; });
+  $("#features_get").click(function() { if (this.value=="Feature") this.value = ""; });
+  $("#features_get").focusout(function() { if (this.value == "") this.value = "Feature"; });
   $("#features_value").click(function() { this.value = ""; });
   $("#features_value").focusout(function() { if (this.value == "") this.value = "1e-05"; });
+  $("#features_value_get").click(function() { this.value = ""; });
+  $("#features_value_get").focusout(function() { if (this.value == "") this.value = "1e-05"; });
   $("#feature_groups_value").click(function() { this.value = ""; });
   $("#feature_groups_value").focusout(function() { if (this.value == "") this.value = "1e-05"; });
   // set all sorts of learning rates
   $("#set_features").click(function() {
     k = $("#features").val();
     v = $("#features_value").val();
-    if (k=="Feature" || k=="" || v=="" || !parseFloat(v)) {
+    if (k=="Feature" || k=="" || v=="" || isNaN(parseFloat(v))) {
       alert("Malformed request.");
     } else {
       url_prefix = $("#features_type").val();
@@ -73,5 +87,27 @@ $().ready(function()
     $('<svg width="100%">'+d+'</svg>')
   );
   $("#original_svg").width($("#original_svg").children()[0].getBBox().width+"px");
+
+  // mark stale content
+  if ($("#up_val").text() != "true") {
+    $(".updated").each(function(i,el){$(el).addClass("stale")})
+  }
+
+  poll("http://"+document.URL.split("/")[2]);
+
+  $("#get_features").click(function () {
+    $.get("http://"+document.URL.split("/")[2]+$("#features_type_get").val()+"/"+$("#features_get").val(),
+    function (data) {
+      $("#features_value_get").val(data);
+    });
+  });
+
+  $("#get_feature_groups").click(function () {
+    $.get("http://"+document.URL.split("/")[2]+"/get_rate/"+$("#feature_groups_get").val(),
+    function (data) {
+      $("#feature_groups_value_get").val(data);
+    });
+  });
+
 });
 
diff --git a/js/interface.js b/js/interface.js
index 781dfde..3341575 100644
--- a/js/interface.js
+++ b/js/interface.js
@@ -1,5 +1,5 @@
 /*
- * (common) global vars
+ * Global vars
  *
  */
 var data,    // global data object
@@ -244,7 +244,7 @@ var next =  function ()
     send_data["original_svg"] = document.getElementById("original_svg").value;
   } else {
     post_edit = $.trim(target_textarea.value);
-    send_data["post_edit"] = post_edit;
+    send_data["post_edit"] = encodeURIComponent(post_edit);
     send_data['type'] = 't';
   }
 
diff --git a/server.rb b/server.rb
index ad52b62..5fa291a 100755
--- a/server.rb
+++ b/server.rb
@@ -266,6 +266,9 @@ def process_next reply
         s = splitpipe(rs)[1..2].map{|i|i.strip.lstrip}.join(" ||| ")
         new_rules_ids.has_key?(s)
       }
+      f = WriteFile.new "#{WORK_DIR}/#{$db['progress']}.known_rules"
+      f.write add_known_rules.join "\n"
+      f.close
       $known_rules += add_known_rules
       $known_rules.uniq! { |rs|
         splitpipe(rs)[1..2].map{|i|i.strip.lstrip}.join(" ||| ")
@@ -324,7 +327,8 @@ def process_next reply
       a = send_recv :atools, "#{a_fwd} ||| #{a_back}"
                                                           # 5d actual extractor
       $status = "Updating grammar extractor"                           # status
-      send_recv :extractor, "default_context ||| #{source} ||| #{post_edit} ||| #{a}"
+      msg = "default_context ||| #{source} ||| #{post_edit} ||| #{a}"
+      send_recv :extractor, msg
                                                            # 6. update database
       $db['updated'] << true
       `cp #{WORK_DIR}/dtrain.debug.json \
@@ -544,6 +548,13 @@ get '/status' do                                                 # check status
   return $status
 end
 
+get '/status_debug' do                                                 # check status
+  cross_origin
+  logmsg :server, "status: #{$status}"
+  return "[##{$db["progress"]}] Locked" if $locked
+  return "[##{$db["progress"]}] #{$status}"
+end
+
 get '/confirm' do                        # client confirms received translation
   cross_origin
   $confirmed = true
@@ -572,6 +583,24 @@ get '/set_learning_rates/:name/:rate' do                   # set learning rates
   return msg
 end
 
+get '/get_weight/:name' do
+  name = params[:name].gsub " ", "_"
+  logmsg :server, "getting weight for '#{name}'"
+  return "locked" if $lock
+  msg = send_recv :dtrain, "get_weight #{name}"
+
+  return msg
+end
+
+get '/get_rate/:name' do
+  name = params[:name].gsub " ", "_"
+  logmsg :server, "getting rate for '#{name}'"
+  return "locked" if $lock
+  msg = send_recv :dtrain, "get_rate #{name}"
+
+  return msg
+end
+
 get '/reset_weights' do                                         # reset weights
   logmsg :server, "reset weights"
   return "locked" if $lock
@@ -622,6 +651,8 @@ end
 get '/reset_new_rules' do                               # removed learned rules
   $new_rules.clear
   $known_rules.clear
+  `rm #{WORK_DIR}/*.*_rules`
+  `rm #{WORK_DIR}/g/*`
 
   return "reset new rules: done"
 end
diff --git a/static/debug.css b/static/debug.css
index d24fb5b..ff66a7c 100644
--- a/static/debug.css
+++ b/static/debug.css
@@ -14,13 +14,14 @@ td.right { text-align:right }
 .black { color:#000 }
 .red { color:red }
 .green { color:green }
-.small { font-size:.8em }
-.tiny { font-size:.6em!important }
+.small { font-size:.9em }
+.tiny { font-size:.8em!important }
 .big { font-size:1.1em }
 .mono { font-family:monospace }
 .even { background:#fff }
 .odd { background: #ffffe0  }
 .up { text-align:right;margin-right:1em }
+.stale { opacity:.125 }
 #svg_b64,#original_svg_b64 { display:none }
 #floater {
   float:right;
diff --git a/views/debug.haml b/views/debug.haml
index 0bf56ce..4ebb0a0 100644
--- a/views/debug.haml
+++ b/views/debug.haml
@@ -2,7 +2,7 @@
 !!!
 %html
   %head
-    %title Debug view (Session ##{session_key})
+    %title Debug view (Session #{session_key})
   %link(rel="stylesheet" type="text/css" href="debug.css")
   %script{:src => "http://postedit.cl.uni-heidelberg.de/js/jquery.min.js"}
   %script{:src => "http://postedit.cl.uni-heidelberg.de/js/jquery.tablesorter.min.js"}
@@ -25,7 +25,10 @@
         %li
           %a{ :href => "#kbest" } K-best
 
-      %p.tiny Session: ##{session_key}
+      %p
+        Status:
+        %span#status ...
+      %p.tiny Session: #{session_key}
 
     /=#########################################################################
     %h2#controls Controls
@@ -44,23 +47,42 @@
         %a.ajax{:tgt => "/reset_extractor",       :href => "#controls"} Reset extractor
       %li
         %a.ajax{:tgt => "/reset_new_rules",       :href => "#controls"} Reset new rules
-      %li
-        %a.ajax{:tgt => "/shutdown",              :href => "#controls"} Initiate shutdown
+      /
+        %li
+          %a.ajax{:tgt => "/shutdown",              :href => "#controls"} Initiate shutdown
     %h3 Learning rates &amp; weights
-    %p Set default weight or rate for specific features:
+    %p Get current weight/rate for specific features:
+    %input#features_get{ :value => "Feature", :style => "text-align:center; width:20em" }
+    %select#features_type_get
+      %option{ :value => "/get_weight" } Weight
+      %option{ :value => "/get_rate" } Rate
+    %input#features_value_get{ :value => " ", :style => "text-align:right; width:10em" }
+    %button#get_features
+      Get
+    %p Set weight/rate for specific features:
+    %input#features{ :value => "Feature", :style => "text-align:center; width:20em" }
     %select#features_type
       %option{ :value => "/set_weights" } Weight
       %option{ :value => "/set_learning_rates" } Rate
-    %input#features{ :value => "Feature", :style => "text-align:center; width:20em" }
-    %input#features_value{ :value => "1e-05", :style => "text-align:right; width:5em" }
+    %input#features_value{ :value => "1e-05", :style => "text-align:right; width:10em" }
     %button#set_features
       Set
-    %p Set rate for feature groups:
+    %p
+      Set rate for feature groups:
+      %span.tiny (Features unseen in training/tuning have a common default rate.)
+    %select#feature_groups_get
+      %option{ :value => "R" } rule ids
+      %option{ :value => "RB" } rule bigrams
+      %option{ :value => "Shape" } rule shapes
+    %input#feature_groups_value_get{ :value => "1e-05", :style => "text-align:right; width:10em" }
+    %button#get_feature_groups
+      Get rate
+    %br
     %select#feature_groups
       %option{ :value => "R" } rule ids
       %option{ :value => "RB" } rule bigrams
       %option{ :value => "Shape" } rule shapes
-    %input#feature_groups_value{ :value => "1e-05", :style => "text-align:right; width:5em" }
+    %input#feature_groups_value{ :value => "1e-05", :style => "text-align:right; width:10em" }
     %button#set_feature_groups
       Set rate
 
@@ -81,7 +103,7 @@
       %tr
         %td.noborder
           %strong Progress:
-        %td.left.noborder #{[0,progress].max}
+        %td.left.noborder ##{[0,progress].max}
       %tr
         %td.noborder
           %strong MT Input:
@@ -109,14 +131,16 @@
       %tr
         %td.noborder
           %strong Best match (BLEU=#{(pairwise_ranking_data["best_match_score"]*100).round(2)}%):
-        %td.left.noborder #{pairwise_ranking_data["best_match"]}
+        %td.left.noborder.updated #{pairwise_ranking_data["best_match"]}
 
     %h3 Metadata
-    %p <strong>K:</strong>                      #{pairwise_ranking_data["samples_size"]}
-    %p <strong>Number of updates:</strong>      #{pairwise_ranking_data["num_up"]}
-    %p <strong>Updated features:</strong>       #{pairwise_ranking_data["updated_features"]}
-    %p <strong>Duration:</strong>               #{data["durations"][progress]}ms
-    %p <strong>Updated:</strong>                #{data["updated"][progress]}
+    %p
+      <strong>Updated:</strong>
+      %span#up_val #{data["updated"][progress]}
+    %p.updated <strong>K:</strong>                  #{pairwise_ranking_data["samples_size"]}
+    %p.updated <strong>Number of updates:</strong>  #{pairwise_ranking_data["num_up"]}
+    %p.updated <strong>Updated features:</strong>   #{pairwise_ranking_data["updated_features"]}
+    %p <strong>Duration:</strong>                   #{data["durations"][progress]}ms
 
     %h3 Derivation
     %p
@@ -138,6 +162,8 @@
     /=#########################################################################
     %h2#grammar Grammar
 
+    %p <strong>Notes:</strong> In addition to dynamically adding each source/post-edit instance to the suffix array extractor, the system additionally uses the provided phrase alignments to extract new rules. The extraction follows the original Hiero grammar extraction, but using phrases instead of words and using only a single binary feature: 'NewRule=1'. Extracted rules that already exist in a grammar are annotated with an additional feature: 'KnownRules=1'. OOVs are avoided by asking the user for translations of unknown words prior to translation. These are added to the grammars as new rules ('OOVFix=1').
+
     %h3
       New Rules
     %p Including OOV fixes.
@@ -175,11 +201,13 @@
     /=#########################################################################
     %h2#weights Weights
 
-    %p N.B. that updates are only done if the post-edit is different from the original 1-best output of the decoder.
+    %p <strong>Notes:</strong> If the user modifies the string of the original one-best output, a standard pairwise ranking update is made, using the post-edit as reference translation. No update is performed if the string is not modified. The k-best list is not modified, as the original grammar is used (leave-one-out).
+    /
+      %p N.B. that updates are only done if the post-edit is different from the original 1-best output of the decoder.
     %p Update of a single row is: After = Before - (Rate * -Raw Diff.) = Before + (Rate * Raw. Diff.). The 'Raw Diff.' column is the sum of the difference vectors of the misranked pairs.
 
     %h3 Weight updates
-    %table.sortable
+    %table.sortable.updated
       %thead
         %tr
           %th Feature
@@ -218,8 +246,8 @@
                     ERROR: unknown key #{k}
 
     %p
-      %a{ :onclick => "$('#features').toggle()" } Explanations
-    %table#features{ :style => "display:none" }
+      %a{ :onclick => "$('#features_expl').toggle()" } Explanations
+    %table#features_expl{ :style => "display:none" }
       %tr
         %td EgivenFCoherent
         %td.left -log10[ c(e, f) / sample c(f) ]
@@ -273,7 +301,7 @@
         %td.left Indicator features for rule shapes (39 in total)
       %tr
         %td IsSupportedOnline
-        %td.left Rules with support from local context (Denkowski)
+        %td.left Rules with support from local context (added by Denkowski's online suffix array extractor)
 
     %p.up
       %a{ :href => "#" } ^ up
@@ -281,9 +309,10 @@
     /=#########################################################################
     %h2#kbest K-best list
     %p
+      <strong>Notes:</strong>
       %span.red In red:
-      %span Update needed, i.e. "any of the above hypotheses has a lower model score" (in default order).
-    %table.sortable
+      %span Update needed, i.e. "any of the above hypotheses has a lower model score" (if the list is ordered by BLEU score in descending order).
+    %table.sortable.updated
       %thead
         %tr
           %th.center{:style => "width:5%"}  Rank