Roy Tang

Programmer, engineer, scientist, critic, gamer, dreamer, and kid-at-heart.

Blog Notes Photos Links Archives About

All entries tagged questions.

You can subscribe to an RSS feed of this list.

Sep 2019

  • I have a GeForce 970, it’s a bit old, I got it in 2015. I’ve been having an issue where sometimes in some games, my displays (I have two monitors) will die (go completely black), but the game goes on running for a while (I can hear the audio), then after a couple of minutes or so the audio also stops so I can’t really do anything except do a hard reboot. I’ve tried VNCing into the PC to see what’s going on after this time, but it’s not accessible over the network so I assume it’s dead.

    The crashes happen most often while playing Magic Arena (says a lot about the game!), but I’ve also had it happen once or twice on Starcraft 2 and Borderlands 2. I don’t play very demanding games graphically. I can’t identify any specific action that causes the crashes.

    Some other things I’ve tried:

    • attach my 2nd monitor to onboard graphics, keeping the main monitor connected to the vidcard. In this case when the crash happens, the main monitor goes black, while the 2nd monitor still has a display, but is frozen
    • remove my 2nd monitor (I thought I might be overtaxing the graphics card by having two of them??)
    • checking for memory (RAM) issues. I did have some RAM issues a couple of months ago, but that turned up to be due to a dirty connector, and it was causing crashes not just in games but in general Windows usage, so I think this is a different issue now
    • checking for disk issues. I actually found some bad sectors on my secondary drive and ended up replacing it. But even if I ran everything from my primary (SSD) drive, the graphics crashes still occur
    • reinstalling/updating device drivers
    • resetting Windows 10 (yup, I went that far)

    After all of these, I’m still encountering these crashes so I’m seriously considering replacing the graphics card now, but I’d like the be sure that it’s a graphics card issue and not something wonky with the motherboard, as my purchasing decisions will be different if the motherboard was the problem (I might as well get a new PC then). I’m looking for any advice to help me diagnose or confirm the problem.

    For reference, my current PC specs:

    • Case Silverstone Precision 10SST-PS10B
    • Fan 120mm internal aux fan
    • PSU Cooler Master B600 V2 600W
    • CPU Intel Core i5-4460
    • MB ASUS H97ME
    • RAM 2x DDR3 Kingston HyperX Fury 8gb 1600
    • VC Asus GTX970 STRIX OC 4gb
    • SSD Samsung 850Evo 250gb
    • SSHD Seagate Firecuda 2tb

Feb 2018

Jan 2018

  • A client updated their EC2 CentOS 6.4 instance to 6.9 using yum update. After the update and a reboot, the instance wouldn’t start up anymore (became unreachable)

    The system log (recovered from the AWS console) goes something like this:

    Xen Minimal OS!
      start_info: 0x10d3000(VA)
        nr_pages: 0xe4f53
      shared_inf: 0xeea31000(MA)
         pt_base: 0x10d6000(VA)
    nr_pt_frames: 0xd
        mfn_list: 0x9ab000(VA)
       mod_start: 0x0(VA)
         mod_len: 0
           flags: 0x300
        cmd_line: root=/dev/sda1 ro console=hvc0 4
      stack:      0x96a100-0x98a100
    MM: Init
          _text: 0x0(VA)
         _etext: 0x7b824(VA)
       _erodata: 0x97000(VA)
         _edata: 0x9cce0(VA)
    stack start: 0x96a100(VA)
           _end: 0x9aa700(VA)
      start_pfn: 10e6
        max_pfn: e4f53
    Mapping memory range 0x1400000 - 0xe4f53000
    setting 0x0-0x97000 readonly
    skipped 0x1000
    MM: Initialise page allocator for 1807000(1807000)-e4f53000(e4f53000)
    MM: done
    Demand map pfns at e4f54000-20e4f54000.
    Heap resides at 20e4f55000-40e4f55000.
    Initialising timer interface
    Initialising console ... done.
    gnttab_table mapped at 0xe4f54000.
    Initialising scheduler
    Thread "Idle": pointer: 0x20e4f55050, stack: 0xe4810000
    Thread "xenstore": pointer: 0x20e4f55800, stack: 0xe4820000
    xenbus initialised on irq 3 mfn 0xfeffc
    Thread "shutdown": pointer: 0x20e4f55fb0, stack: 0xe4830000
    Dummy main: start_info=0x98a200
    Thread "main": pointer: 0x20e4f56760, stack: 0xe4840000
    "main" "root=/dev/sda1" "ro" "console=hvc0" "4" 
    vbd 2048 is hd0
    ******************* BLKFRONT for device/vbd/2048 **********
    backend at /local/domain/0/backend/vbd/2334/2048
    629145600 sectors of 512 bytes
    vbd 2128 is hd1
    ******************* BLKFRONT for device/vbd/2128 **********
    backend at /local/domain/0/backend/vbd/2334/2128
    314572800 sectors of 512 bytes
        GNU GRUB  version 0.97  (3751244K lower / 0K upper memory)
        Use the ^ and v keys to select which entry is highlighted.
        Press enter to boot the selected OS, 'e' to edit the
        commands before booting, or 'c' for a command-line.  CentOS (2.6.32-696.18.7.el6.x86_64)                                      CentOS (2.6.32-573.el6.x86_64)                                           CentOS (2.6.32-358.18.1.el6.x86_64)                                      CentOS-6.4-x86_64-GA-03 2.6.32-358.el6.x86_64                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     The highlighted entry will be booted automatically in 1 seconds.     Booting 'CentOS (2.6.32-696.18.7.el6.x86_64)'
    root (hd0)
     Filesystem type is ext2fs, using whole disk
    kernel /boot/vmlinuz-2.6.32-696.18.7.el6.x86_64 root=/dev/xvde ro crashkernel=a
    uto LANG=en_US.UTF-8 KEYTABLE=us
    initrd /boot/initramfs-2.6.32-696.18.7.el6.x86_64.img
    ============= Init TPM Front ================
    Tpmfront:Error Unable to read device/vtpm/0/backend-id during tpmfront initialization! error = ENOENT
    Tpmfront:Info Shutting down tpmfront
    close blk: backend=/local/domain/0/backend/vbd/2334/2048 node=device/vbd/2048
    close blk: backend=/local/domain/0/backend/vbd/2334/2128 node=device/vbd/2128

    We’re all not too experienced with AWS so we don’t have much clue what to do. Any idea what we should be looking at or anything to point us in the right direction? The instance was actually a clone of production that we tested the upgrade on before doing it on actual production, so we have a chance to try it again and adjust any settings or whatever before or after the update.

Dec 2017

  • I have a query using MarkLogic node.js that basically boils down to something like this:

    .on('data', function(row) {
        console.log("Stream on data");
    .on('end', function() {
        console.log("Stream on end");
    .on('error', function(error) {

    Now, for a certain collection we have in our database, the ‘end’ function doesn’t fire, i.e. I never see “Stream on end” appear in the log. There’s no error or anything, processing just stops. It’s only for this particular collection, other collections seem fine.

    If I query documents in that collection directly using other methods such as qb.value() without using qb.collection(), the end event fires correctly. But once I add qb.collection() into the mix (using qb.and), the end event doesn’t fire.

    I’m unsure how to debug this, as this is my first time trying to use streams in the nodejs client library. Any advice as to what I can check?


  • How to specify that a column in the schema should be nullable?

    I tried adding a nullable attribute:

    var myFirstTDE = xdmp.toJSON(
        "template": {
          "context": "/match",
          "collections": ["source1"],
          "rows": [
              "schemaName": "soccer",
              "viewName": "matches",
              "columns": [
                  "name": "id",
                  "scalarType": "long",
                  "val": "id",
                  "nullable": 0
                  "name": "document",
                  "scalarType": "string",
                  "val": "docUri"
                  "name": "date",
                  "scalarType": "date",
                  "val": "match-date"
                  "name": "league",
                  "scalarType": "string",
                  "val": "league"

    But this gave me a template error:

    "message": "TDE-INVALIDTEMPLATENODE: Invalid extraction template node: fn:doc('')/template/array-node('rows')/object-node()/array-node('columns')/object-node()[1]/number-node('nullable')"

    For a template defined using XQuery, adding nullable to the column works:


    How to do the same thing using JS/Json?

Nov 2017

  • This is a follow-up to my question here:

    So let’s say I have a number of documents structured like this:

             content: {
               name:'Joe Parent',
               children: [
                   name: 'Bob Child'
                   name: 'Sue Child'
                   name: 'Guy Child'
           {permissions : xdmp.defaultPermissions(),
            collections : ['test']})

    I want to define a template that would extract triples from these documents defining sibling relationships between the children. For the above example, I would want to extract the following triples (the relationship is two-way):

    Bob Child sibling-of Sue Child
    Bob Child sibling-of Guy Child
    Sue Child sibling-of Bob Child
    Sue Child sibling-of Guy Child
    Guy Child sibling-of Bob Child
    Guy Child sibling-of Sue Child

    How can i set up my template to accomplish this?


  • I’ve been studying the examples here:

    I have a set of documents that are structured with a parent name and an array of children nodes with their own names. I want to create a template that generates triples of the form “name1 is-a-parent-of name2”. Here’s a test I tried, with a sample of the document structure:

             content: {
               name:'Joe Parent',
               children: [
                   name: 'Bob Child'
                   name: 'Sue Child'
           {permissions : xdmp.defaultPermissions(),
            collections : ['test']})
    var tde = require("/MarkLogic/tde.xqy");
    // Load the user template for user profile rows
    var template = xdmp.toJSON(
        "collections": [
            "subject": {
              "val": "xs:string(name)"
            "predicate": {
              "val": "sem:iri('is-parent-of')"
            "object": {
              "val": "xs:string(children/name)"
    tde.templateInsert("/templates/test.tde", template);
      [cts.doc( '/test/tde.json' )]

    However, the above throws an Exception:

    [javascript] TDE-EVALFAILED: tde.nodeDataExtract([cts.doc("/test/tde.json”)]) – Eval for Object='xs:string(children/name)’ returns TDE-BADVALEXPRESSION: Invalid val expression: XDMP-CAST: (err:FORG0001) Invalid cast: (fn:doc("/test/tde.json”)/content/array-node(“children”)/object-node()[1]/text(“name”), fn:doc("/test/tde.json”)/content/array-node(“children”)/object-node()[2]/text(“name”)) cast as xs:string?

    What is the proper syntax for extracting array nodes into a triple?

    2nd somewhat related question: say I also wanted to have triples of the form “child1 is-sibling-of child2”. For the example above it would be “Bob Child is-sibling-of Sue Child”. What would be the proper syntax for this? I’m not even sure how to begin with this one.

    Is TDE even the way to go here? Or is it better to do this programmatically? i.e. on document ingestion, generate those triples inside the document directly?

    (If it’s relevant, the ML version being used is 9.)

  • I’ve been testing migrating one of our systems to Marklogic 9 and using the Optics API.

    One of our functions involves grouping claims by member_id, member_name and getting the sums and counts, so I did something like this:

    var results = op.fromView('test', 'claims')
      .groupBy(['member_id', 'member_name'], [
             op.count('num_claims', 'claim_no'),
             op.sum('total_amount', 'claim_amount')

    Above works fine. The results are of the form

        member_id: 1, 
        member_name: 'Bob', 
        num_claims: 10, 
        total_amount: 500

    However, we also have a field “company”, where each claim is filed under a different company. Basically the relevant view columns are claim_no, member_id, member_name, company, claim_amount

    I would like to be able to show a column that list the different companies for which the member_id/member_name has filed claims, and how many claims for each company.

    i.e. I want my results to be something like:

        member_id: 1, 
        member_name: 'Bob', 
        num_claims: 10, 
        total_amount: 500,
        companies: [
            company: 'Ajax Co',
            num_claims: 8
            company: 'Side Gig',
            num_claims: 2

    I tried something like this:

    results =, index, array) => {
      var companies = op.fromView('test', 'claims')
        .where(op.eq(op.col('member_id'), member.member_id))
        .groupBy('company', [
          op.count('num_claims', 'claim_no')      
      member.companies = companies;
      return member;

    And the output seems correct, but it also executes quite slowly - almost a minute (total number of claim documents is around 120k)

    In our previous ML8 implementation, we were pre-generating summary documents for each member - so retrieval was reasonably fast with the downside that whenever we got a bunch of new data, all of the summary documents had to be re-generated. I was hoping that ML9’s optic API would make it easier to do the retrieval/grouping/aggregates on the fly so we wouldn’t have to do that.

    In theory, I could just add company to the groupBy fields, then merge the rows in the result query as needed. But the problem with that approach is that I can’t guarantee I’ll get the top 200 by total amount (as was my original query)

    So, the question is: Is there a better way of doing this with a reasonable execution time? Or should I just stick to pre-generating the summary documents?

  • I have a Marklogic 9 project that I’m configuring with Roxy. I’ve been following these examples:

    Basically, I have a server-side JS function that I want to call after deploy content. I have something like this:

    then you would define your new method

      def deploy_content
        # you can optionally call the original
        # do your stuff here
          xquery version "1.0-ml";
          xdmp:javascript-eval('var process = require("/ingestion/process.sjs"); process.postDeployContent();')
        :db_name => @properties[""] + "-content")

    The xquery being called here evaluates fine when executed via query console. But when I call ml local deploy content, I get the following error:

    ERROR: 500 "Internal Server Error"
    ERROR: <html xmlns="">
        <title>500 Internal Server Error</title>
        <meta name="robots" content="noindex,nofollow"/>
        <link rel="stylesheet" href="/error.css"/>
        <span class="error">
          <h1>500 Internal Server Error</h1>
            <dt>XDMP-MODNOTFOUND: var process = require("/ingestion/process.sjs"); process.postDeployContent(); -- Module /ingestion/process.sjs not found</dt>
            <dt>in [anonymous], at 1:14 [javascript]</dt>
            <dt>at 3:6,
    in xdmp:eval("var process = require(&amp;quot;/ingestion/process.sjs&amp;quot;); proce...") [javascript]</dt>
            <dt>in /eval, at 3:6 [1.0-ml]</dt>

    Why is the module not found when running via xquery from app_specific.rb?

    Or… is there a better way to call a JS module function from here. Sorry, I’m not too familiar with the xquery side, so I just called a JS function instead.

  • Basically the title. The client is complaining that when he zooms in, the text labels for the nodes are quite large. Is there a way to keep the node labels at a fixed font size even when zooming in or out?

    From the nodes documentation (, there’s a scaling.label option, but it doesn’t seem to work. I think this is only relevant if I’m using values to scale the nodes.

Oct 2017

  • I’m trying out the Roxy deployer. The Roxy app was created using the default app-type. I setup a new ML 9 database, and I ran “ml local bootstrap” using the default ports (8040 and 8041)

    Then I setup a node application. I tried the following (sample code from

    var marklogic = require('marklogic');
    var conn = {
        host: '',  
        port: 8040,
        user: 'admin',
        password: 'admin',
        authType: 'DIGEST'
    var db = marklogic.createDatabaseClient(conn);
      {author: 'Beryl Markham'},
      {author: 'WG Sebald'}
    .result(function(response) {
        console.log(JSON.stringify(response, null, 2));
      }, function (error) {
        console.log(JSON.stringify(error, null, 2));

    Running the script gave me an error like:

    $ node test.js
      "message": "write document list: cannot process response with 500 status",
      "statusCode": 500,
      "body": "<error:error xsi:schemaLocation=\" error.xsd\" xmlns:error=\"\" xmlns:xsi=\"\">\n  <error:code>XDMP-IMPMODNS</error:code>\n  <error:name>err:XQST0059</error:name>\n  <error:xquery-version>1.0-ml</error:xquery-version>\n  <error:message>Import module namespace mismatch</error:message>\n  <error:format-string>XDMP-IMPMODNS: (err:XQST0059) Import module namespace does not match target namespace of imported module /MarkLogic/rest-api/endpoints/config.xqy</error:format-string>\n  <error:retryable>false</error:retryable>\n  <error:expr/>\n  <error:data>\n    <error:datum></error:datum>\n    <error:datum></error:datum>\n    <error:datum>/MarkLogic/rest-api/endpoints/config.xqy</error:datum>\n  </error:data>\n  <error:stack>\n    <error:frame>\n      <error:uri>/roxy/lib/rewriter-lib.xqy</error:uri>\n      <error:line>5</error:line>\n      <error:column>0</error:column>\n      <error:xquery-version>1.0-ml</error:xquery-version>\n    </error:frame>\n  </error:stack>\n</error:error>\n"

    If I change the port to 8000 (the default appserver that inserts into Documents), the node function executes correctly as expected. I’m not sure if I need to configure anything else with the Roxy-created appserver so that it works with the node.js application.

    I’m not sure where the “DELETE_IF_UNUSED” part in the error message is coming from either. There doesn’t seem to be any such text in the configuration files generated by Roxy.

    Edit: When accessing via the browser, I get a an xml with a similar error:

    <error:error xsi:schemaLocation=" error.xsd" xmlns:error="" xmlns:xsi="">
      <error:message>Import module namespace mismatch</error:message>
      <error:format-string>XDMP-IMPMODNS: (err:XQST0059) Import module namespace does not match target namespace of imported module /MarkLogic/rest-api/endpoints/config.xqy</error:format-string>

    If it matters, MarkLogic version is 9.0-3.1. It’s a fresh install too.

    Any advice?

  • I’m trying to migrate one of my dev envts from ML8 to ML9. I have an import script that successfully works on the ML8 version, but there’s an error when I try running it against the ML9 database. The ML9 version is The MLCP version is 9.0.3

    My MLCP options file is as follows:


    Here’s the output of a test run with only 2 records in the test CSV file:

    17/10/30 14:07:33 INFO contentpump.LocalJobRunner: Content type: JSON
    17/10/30 14:07:33 INFO contentpump.ContentPump: Job name: local_455168344_1
    17/10/30 14:07:33 INFO contentpump.FileAndDirectoryInputFormat: Total input paths to process : 1
    17/10/30 14:07:38 WARN contentpump.TransformWriter: Failed document /test/records/31.json
    17/10/30 14:07:38 WARN contentpump.TransformWriter: <error:format-string xmlns:error="" xmlns:xsi="">XDMP-UNEXPECTED: (err:XPST0003) Unexpected token syntax error, unexpected QName_, expecting $end or SemiColon_</error:format-string>
    17/10/30 14:07:38 WARN contentpump.TransformWriter: Failed document /test/records/32.json
    17/10/30 14:07:38 WARN contentpump.TransformWriter: <error:format-string xmlns:error="" xmlns:xsi="">XDMP-UNEXPECTED: (err:XPST0003) Unexpected token syntax error, unexpected QName_, expecting $end or SemiColon_</error:format-string>
    17/10/30 14:07:38 INFO contentpump.LocalJobRunner:  completed 100%
    17/10/30 14:07:38 INFO contentpump.LocalJobRunner: com.marklogic.mapreduce.MarkLogicCounter:
    17/10/30 14:07:38 INFO contentpump.LocalJobRunner: INPUT_RECORDS: 2
    17/10/30 14:07:38 INFO contentpump.LocalJobRunner: OUTPUT_RECORDS: 2
    17/10/30 14:07:38 INFO contentpump.LocalJobRunner: OUTPUT_RECORDS_COMMITTED: 0
    17/10/30 14:07:38 INFO contentpump.LocalJobRunner: OUTPUT_RECORDS_FAILED: 2
    17/10/30 14:07:38 INFO contentpump.LocalJobRunner: Total execution time: 5 sec

    If I remove the transform params, the import works fine.

    I thought it might be a parsing issue with my transform module itself, so I tried replacing it with the following example from the documentation:

    // Add a property named "NEWPROP" to any JSON input document.
    // Otherwise, input passes through unchanged.
    function addProp(content, context)
      const propVal = (context.transform_param == undefined)
                     ? "UNDEFINED" : context.transform_param;
      if (xdmp.nodeKind(content.value) == 'document' &&
          content.value.documentFormat == 'JSON') {
        // Convert input to mutable object and add new property
        const newDoc = content.value.toObject();
        newDoc.NEWPROP = propVal;
        // Convert result back into a document
        content.value = xdmp.unquote(xdmp.quote(newDoc));
      return content;
    exports.addProp = addProp;

    (Of course I changed the params in the MLCP options file accordingly)

    The issue still persists even with just this test function.

    Any advice?

  • I’ve been testing out the WebView component, but I can’t seem to get it to render things.

    Sample here:

    export default class App extends Component {
      render() {
        return (
          <View style={styles.container}>
            <Text style={styles.paragraph}>
              Change code in the editor and watch it change on your phone!
              Save to get a shareable url. You get a new url each time you save.
            <WebView source={{html: '<p>Here I am</p>'}} />        
            <WebView source={{ uri: ''}} />

    When running the above example in Expo, neither of the two WebView components seem to render. What am I doing wrong?

  • I’m trying out vis.js for generating graph visualization. For every edge in my graph, I have a number that describes the strength of the connection between two nodes. I’d like to render the vis.js graph such that the nodes that have stronger relationships (higher edge values) are closer together (edge length is shorter).

    I’ve set the relationship strength (an integer) as the “value” attribute for each edge, but this only seems to make the edge lines slightly thicker for higher values.

    What options should I be looking at? I’m not sure if this is supposed to be a function of vis’s physics-based stabilization.

    Thanks for advice!

Sep 2017

  • I have an HTML page with around 10 charts generated by chart.js (so these are canvas elements). I want to be able to export the page content into a PDF file.

    I’ve tried using jsPDF’s .fromHTML function, but it doesn’t seem to support exporting the canvas contents. (Either that or I’m doing it wrong). I just did something like:

        $(".test").click(function() {
      var doc = new jsPDF()

    Any alternative approaches would be appreciated.

  • I’m using ML8. I have a bunch of json documents in the database. Some documents have a certain property “summaryData”, something like:

    ...(other stuff)...
      summaryData: {
        count: 100,
        total: 10000,
        summaryDate: (date value)

    However, not all documents have this property. I’d like to construct an SJS query to retrieve those documents that don’t have this property defined. If it was SQL, I guess the equivalent would be something like “WHERE summaryData IS NULL”

    I wasn’t sure what to search for in the docs. Any advise would be helpful.

  • I’m using ML8 and Node.js. The documentation here: describes how to do conditional updates in ML using the versionId field.

    But for example if I want to do a conditional update on a different field, is it possible?

    My scenario is: I have JSON documents with elements assignedTo and assignDate (where assignDate is set to current date every time a new value is set to assignedTo)

    Now, for my “Assign” operation, I would like to make sure that no one else has changed the assignedTo/assignDate fields between the time I read the document and when I perform the update. I don’t care if other fields in the same document have been updated or not - if other fields have been updated, I can still proceed with the Assign operation (hence I cannot use the versionId approach, since that covers the whole document)

    How can this be done?

Aug 2017

  • Sorry if the title is unclear, but I’m finding my problem hard to explain concisely. I have a number of JSON documents with structure like this:

      "count": 100,
      "groups": [
        "name": "group A",
        "count": 12
        "name": "group B",
        "count": 22
        "name": "group C",
        "count": 7

    Basically, the document has an item count plus a breakdown of that count into smaller groups. So this record represents a collection of 100 items, of which 12 are from group A, 22 are from group B, 7 are from group C.

    Now, I have an element range index on “count” and a bunch of such documents. I’d like to be able to sort by any of the options:

    • sort by total count (descending or ascending)
    • sort by group A count (descending or ascending)
    • sort by group B count (descending or ascending)
    • sort by group C count (descending or ascending)

    I’ve tried

    .orderBy(qb.sort("count", "descending"))

    This seems to be sort by the total cost (the one in the document root), but I’m not sure if that’s always true or I need to specify something else to guarantee it always gets that particular one.

    For sorting by a specific group, I have no idea how to specify it.

    Any advice?

  • For the past 2 weeks or so now, I’ve been having issues with my ISP (I’m not in the US, so it’s not Comcast or whatever).

    Specifically, I often lose access to Google-related sites/services: the search engine itself, Gmail, Google Docs/Drive, Youtube, Google Keep are the ones I use most often. I’ve restarted my router multiple times already over the past couple of weeks. Same issue is encountered on my desktop (wired), laptop, iPad, phone (when connected over wifi) etc.

    I know it’s related to my ISP because (a) I’ve read some other users experiencing similar issues on social media; (b) if I tether to my phone’s data connection, the problem isn’t encountered.

    I don’t have a proxy server configured. I used to be using Google DNS but I’ve also tried switching back to my ISP’s default DNS with no luck.

    What’s the best way for me to figure out what the issue is? (Preferably in a way that I can bring it up to my ISP so that they know specifically what they need to fix - talking to their customer service is a tedious experience so I’ve decided to figure out as much as possible before going back to them)

    Screenshot of the error from the browser: enter image description here

    Sample tracert: enter image description here

    Pings seem ok right now (although if I leave ping -t running for a while, there’s an occasional “Request timed out”): enter image description here

  • We’ve been trying to set up flexible replication in our system which uses a MarkLogic database. We followed the instructions from and have been able to set up flexible replication between two MarkLogic servers. We have verified that new documents created in the master are copied over to the replica. However, the master database currently has more than 47 million records that were there before we configured the replication. Once the replication process was triggered, we observed that the documents are being replicated to the replica very slowly. Roughly 20,000 documents were replicated within the first two hours. The rate is very slow, it would take months for the old records to be fully replicated.

    Our questions are:

    1. We are looking into increasing the hardware specs of the two servers, but aside from that would anyone have any advice or documentation as to how we could speed up the replication? I couldn’t find any existing documentation regarding this?

    2. Failing that, would it be possible to set up flexible replication without needing to replicate the initial data set? FYI, we also tried to clone the master database and use the clone as the replica. (We thought this might mean that the older records don’t have to be replicated.) However, in this case we encountered XDMP-NEWSTAMP and XDMP-EXTIME errors on the replica server, so we gave up on this approach. A sample of the errors encountered on the replica is below:

    2017-08-03 18:45:04.376 Notice: exp-rest-content-flexrep: XDMP-NEWSTAMP: Timestamp too new for forest exp-rest-content-001-1 (15017569242290900) 2017-08-03 18:45:04.376 Notice: exp-rest-content-flexrep: in /apply.xqy [1.0-ml] 2017-08-03 18:45:04.379 Notice: TaskServer: XDMP-EXTIME: try { let $raw-module-name := module-path($action-to-execute/p:module) let $module-kind := module-kind($raw-module-name) let $module-name := if ($module-kind = “xquery” or $module-kind = “javascript”) then $raw-module-name else $cpfi:xslt-action return if ($module-name = “") then fn:error((), “CPF-ACTIONNOTFOUND”, “Default success”) else if ($module-kind = “javascript”) then (xdmp:trace(“CPF Action Invoke”, fn:string-join(($caller, xdmp:get-current-user(), $uri, $state-or-status, $raw-module-name), " “)), xdmp:invoke($module-name, (fn:QName("”,“uri”), $uri, xs:QName(“cpf:document-uri”), $uri, fn:QName("”,“transition”), $chosen-transition, options-var-js($action-to-execute)), $invoke-options)) else (xdmp:trace(“CPF Action Invoke”, fn:string-join(($caller, xdmp:get-current-user(), $uri, $state-or-status, $raw-module-name), " “)), xdmp:invoke($module-name, ($vars, xs:QName(“cpf:transition”), $chosen-transition, options-var($action-to-execute), if ($module-kind = “xslt”) then (xs:QName(“cpf:stylesheet-uri”), $raw-module-name) else ()), $invoke-options)) } catch ($e) { let $trace := let $context := fn:concat($caller, " “, $uri, " action failed”) return (cpf:log(fn:string-join(($context, $e/err:format-string), " “), “error”), cpf:log(($context, $e), “fine”)) let $failure-action := ($pipelines/p:failure-action)[1] let $raw-failure-module := module-path($failure-action/p:module) let $failure-kind := module-kind($raw-failure-module) let $failure-module := if ($failure-kind = “xquery” or $failure-kind = “javascript”) then $raw-failure-module else $cpfi:xslt-action return if ($failure-module = “") then fn:error((), “CPF-ACTIONNOTFOUND”, “Default failure action”) else xdmp:invoke($failure-module, ($vars, xs:QName(“cpf:transition”), $chosen-transition, options-var($failure-action), xs:QName(“cpf:exception”), $e, if ($failure-kind = “xslt”) then (xs:QName(“cpf:stylesheet-uri”), $raw-failure-module) else ()), $invoke-options) } – Time limit exceeded 2017-08-03 18:45:04.379 Notice: TaskServer: in /MarkLogic/cpf/triggers/internal-cpf.xqy, at 213:4, 2017-08-03 18:45:04.379 Notice: TaskServer: in execute-action(“on-state-enter”, “", “/_smslogs/5849823.xml”, (xs:QName(“trgr:uri”), “/_smslogs/5849823.xml”, xs:QName(“trgr:trigger”), …), different-transactiont…, (fn:doc(“")/p:pipeline, fn:doc(“")/p:pipeline, fn:doc(“")/p:pipeline, …), fn:doc(“")/p:pipeline/p:state-transition[3]/p:default-action, fn:doc(“")/p:pipeline/p:state-transition[3]) [1.0-ml] 2017-08-03 18:45:04.379 Notice: TaskServer: $caller = “on-state-enter” 2017-08-03 18:45:04.379 Notice: TaskServer:
    $state-or-status = “" 2017-08-03 18:45:04.379 Notice: TaskServer: $uri = “/_smslogs/5849823.xml” 2017-08-03 18:45:04.379 Notice: TaskServer: $vars = (xs:QName(“trgr:uri”), “/_smslogs/5849823.xml”, xs:QName(“trgr:trigger”), …) 2017-08-03 18:45:04.379 Notice: TaskServer: $invoke-options = different-transactiont… 2017-08-03 18:45:04.379 Notice: TaskServer: $pipelines = (fn:doc(“")/p:pipeline, fn:doc(“")/p:pipeline, fn:doc(“")/p:pipeline, …) 2017-08-03 18:45:04.379 Notice: TaskServer: $action-to-execute = fn:doc(“")/p:pipeline/p:state-transition[3]/p:default-action 2017-08-03 18:45:04.379 Notice: TaskServer: $chosen-transition = fn:doc(“")/p:pipeline/p:state-transition[3] 2017-08-03 18:45:04.379 Notice: TaskServer: $e = <error:error xsi:schemaLocation=" error.xsd” xmlns:xsi="” xmlns:error="">error:codeXDMP-NEWSTAMP</error:code>error:name/<error:xquery…</error:error> 2017-08-03 18:45:04.379 Notice: TaskServer: in /MarkLogic/cpf/triggers/internal-cpf.xqy, at 342:6, 2017-08-03 18:45:04.379 Notice: TaskServer: in execute-transition(“on-state-enter”, “", “/_smslogs/5849823.xml”, (xs:QName(“trgr:uri”), “/_smslogs/5849823.xml”, xs:QName(“trgr:trigger”), …), <trgr:trigger xmlns:trgr="">trgr:trigger-id6551367241994447650</trgr:trigger-id><trgr:trig…</trgr:trigger>, (fn:doc(“")/p:pipeline, fn:doc(“")/p:pipeline, fn:doc(“")/p:pipeline, …), (fn:doc(“")/p:pipeline/p:state-transition[2], fn:doc(“")/p:pipeline/p:state-transition[3], fn:doc(“")/p:pipeline/p:state-transition[1], …), <p:null-transition xmlns:p=""><p:state></p:state></p:null-transition>) [1.0-ml] 2017-08-03 18:45:04.379 Notice: TaskServer: $caller = cpf:state(“") 2017-08-03 18:45:04.379 Notice: TaskServer: $state-or-status = () 2017-08-03 18:45:04.379 Notice: TaskServer: $uri = (xs:QName(“trgr:uri”), “/_smslogs/5849823.xml”, xs:QName(“trgr:trigger”), …) 2017-08-03 18:45:04.379 Notice: TaskServer: in /MarkLogic/cpf/triggers/internal-cpf.xqy, at 358:3, 2017-08-03 18:45:04.379 Notice: TaskServer: in int:execute-state-transition(“on-state-enter”, cpf:state(“"), “/_smslogs/5849823.xml”, (xs:QName(“trgr:uri”), “/_smslogs/5849823.xml”, xs:QName(“trgr:trigger”), …), <trgr:trigger xmlns:trgr="">trgr:trigger-id6551367241994447650</trgr:trigger-id><trgr:trig…</trgr:trigger>) [1.0-ml] 2017-08-03 18:45:04.379 Notice: TaskServer: $caller = cpf:state(“") 2017-08-03 18:45:04.379 Notice: TaskServer: $state = () 2017-08-03 18:45:04.379 Notice: TaskServer: $uri = (xs:QName(“trgr:uri”), “/_smslogs/5849823.xml”, xs:QName(“trgr:trigger”), …) 2017-08-03 18:45:04.379 Notice: TaskServer: in /MarkLogic/cpf/triggers/on-state-enter.xqy, at 41:6 [1.0-ml] 2017-08-03 18:45:04.379 Notice: TaskServer: $state = cpf:state(“") 2017-08-03 18:45:04.379 Notice: TaskServer: $trace = () 2017-08-03 18:45:04.379 Notice: TaskServer: $vars = (xs:QName(“trgr:uri”), “/_smslogs/5849823.xml”, xs:QName(“trgr:trigger”), …) 2017-08-03 18:45:04.379 Notice: TaskServer: XDMP-NEWSTAMP: Timestamp too new for forest exp-rest-content-001-1 (15017569242290900) 2017-08-03 18:45:04.379 Notice: exp-rest-content-flexrep: XDMP-NEWSTAMP: Timestamp too new for forest exp-rest-content-001-1 (15017569242290900) 2017-08-03 18:45:04.379 Notice: exp-rest-content-flexrep: in /apply.xqy [1.0-ml]

  • We have a search application using MarkLogic node.js. We use parsedQuery like this:

                    qb.word('name', qb.bind('name')),
                    qb.word('birthdate', qb.bind('birthdate')),
                    qb.range('count', qb.datatype('float'), qb.bind('count'))

    The above currently supports search syntax like “count GT 50”, etc. We would like to support searching using a derived value such as age. That is, we want to support a search syntax like “age GT 10”, where the age value is not stored in the documents in the database but rather needs to be computed from the birthdate on the fly. We can’t store the age in the documents since the age changes depending on the current date.

    Is this possible and if so, how? If it matters, we are using ML8

Jul 2017

  • I’m using the demo of jsPlumb here:

    In this demo, there’s no way to move an existing connection to a different target node. Any idea how to do it?

    Some of the other examples have movable connections, but they also use specific endpoints on the nodes. I like this particular example where I can drag the connection endpoint to any point of the target node.

  • I’m trying out jsplumb. I have a copy of this demo:

    In this demo, when I drag one of the nodes outside of the canvas boundary, a scrollbar appears to indicate the canvas area has expanded. However, I still have to manually scroll the view to see the dragged node.

    I would like the view to automatically scroll when I drag a node beyond the edge. Same thing when dragging a new connector, I would like to automatically scroll the view - so I can choose to connect to a node currently outside the visible area. Any advice on how to do this?

    Secondary question: In the demo above the scrollbars appear as expected when I drag elements off the right or bottom of the canvas, but not when I drag them off the left edge or off the top edge. That is, if I drag a node upward off the canvas and drop it there, I no longer have any way of viewing that node or dragging it back down. Is there a way around this?

  • I have a number of documents from different sources. Many of them reference a company name, but may have stored the information slightly differently. The name is a field in the documents.

    I’d like to be able to detect variations on the same name, something like:

    • Ajax Company Incorporated
    • Ajax Co. Inc.
    • Ajax Company Inc.
    • Ajax Company
    • Ajax Company (formerly Ajax Unlimited)
    • etc

    Does MarkLogic have any facility to query documents that have “similar” name as above? I’m not sure if there’s a more technical term that I should be searching for. Preferably for either the node client API or server-side js.

Jun 2017

Dec 2016

  • I tried to insert the following test document:

            uri: "/test/doc1.json",
            contentType: "application/json",
            collections: "test",
            content: {
                name : "Peter",
                hobby: "Sleeping",
                other: "Some other info",
                "triple": {
                    "subject": {   
                        "datatype": "",  
                        "value": "Peter"   
                    "predicate": {     
                        "datatype": "",  
                        "value": "livesin"   
                    "object": {     
                        "datatype": "",  
                        "value": "Paris"   
        console.log("Done loading");

    Then I queried as follows:

    var query = [
      'SELECT ?s ?p ?o' ,
      'WHERE { ?s ?p ?o }' ,
    db.graphs.sparql('application/sparql-results+json', query.join('\n')
    ).result(function (result) {
      console.log(JSON.stringify(result, null, 2));
    }, function(error) {
      console.log(JSON.stringify(error, null, 2));

    The results showed me the values of the triple, but what if I also want to get the entire document where the triple was embedded? Is it also possible to filter by other fields in the document?

  • I inserted a test document using the Node.js API:

            uri: "/test/doc1.json",
            contentType: "application/json",
            collections: "test",
            content: {
                name : "Peter",
                "triple": {
                    "subject": {   
                        "datatype": "",  
                        "value": "Peter"   
                    "predicate": {     
                        "datatype": "",  
                        "value": "livesin"   
                    "object": {     
                        "datatype": "",  
                        "value": "Paris"   
        console.log("Done loading");

    Then I tried querying it via SPARQL on the Marklogic console webapp:

    SELECT ?s ?p ?o
      WHERE { ?s ?p ?o }

    The query above works fine, it retrieves the test triple I inserted above

    However, if I want to filter by the literal value “Paris”, I tried the following:

    PREFIX loc: <>
    SELECT ?s ?p 
      WHERE { ?s ?p loc:Paris }

    But in this case I got zero results. Is this syntax incorrect? I was just following how the queries looked in

    What is the proper way to do it?

Nov 2016

  • I want to retrieve all documents in my MarkLogic db that have month=November and also group them by name, and get the count of records per name. I know I can get the frequency per name by using valuesBuilder with a range index on the name field, but how can I filter this result so that I only get the count of records for the month of November?

    Supposedly valuesBuilder.fromIndexes().where() can do the filtering, but I don’t know what to pass here and examples online seem to be sparse.

Mar 2016

  • I basically never want to use the feature. How to? There’s another question here that answers how to disable it in Linux via a command-line option in .bashrc, but I don’t think that can be done in Windows?

    Edit: Additional information:

    • I am on Windows 10

    • No Recovery.xcu anywhere on my C drive.

    • even if I discard recovery data, then the next time I open the same file there is no recovery prompt. But when I shutdown Windows without shutting down LibreOffice first, then I get the prompt the next time I open the file.

Sep 2015

  • I have installed an SSL certificate on my Apache server, but when I access the site via URL from a different machine, an HTTPS error is shown and viewing the certificate details says “this certificate has an invalid digital signature”

    If I view the same URL from within the server itself, the certificate is fine and there is no HTTPS error.

    I’m not sure what to look for in httpd.conf. Any advice?


Jun 2014

May 2014

Feb 2014

Nov 2013

  • I want to add focus and blur handlers to all CKEditor instances in our web application. I would like to add the handlers in one place, instead of hunting down every part where we instantiate a CKEditor. Can this be done, like maybe in the config.js editorConfig setup?

    I can’t do something like “on document ready, add handlers to all CKEditor instances on the page” either, since additional editor instances may be created dynamically.

Oct 2013

Jul 2013

Apr 2013

Mar 2013

Dec 2012

  • When I came home today, my video card started displaying graphic artifacts, and logging in to windows 7 would cause the computer to reboot. The computer was working fine when i last shut it down last night.

    Here’s a video of what’s happening. I have two monitors, in the middle of the video I pan to the second monitor. At about 0:24, I attempt to login and the machine reboots. My video card is an NVidia GeForce GTX 460.

    Some more symptoms I’ve encountered and stuff I’ve tried over the course of several reboots:

    • was able to login normally once, but after launching a few apps (chrome, tweetdeck), graphic artifacts started appearing again and the machine eventually froze
    • am able to login successfully in safe mode
    • sometimes the display is fine during the start up, and only starts getting wonky a few seconds after I go to the login screen by selecting a user to login with
    • have encountered Windows failing to start up at all a couple of times (blue screen during startup)
    • have tried System restore/repair a few times. i’ve already tried all the restore points available
    • have tried uninstalling the vid card driver; machine boots and lets me login and use apps as normal but of course i’m at a low resolution (and won’t be able to play games). tried to reinstall the vid card driver using automatic update; after the reinstall, same problem occurred

    Now I’m not sure how to proceed; all signs point to the video card being damaged and probably in need of replacement; my question is are there any other things I can try first?

    Like, is it possible there’s just something loose in the connections internally and I can try to remove and reconnect the card first? Or some alternative video driver to try? Or would a shop tech be able to do something to save the card?

    I don’t have time to take apart the computer at the moment, probably I’ll do it this weekend, but I’d like to know my options ahead of time.

Nov 2012

  • It’s happened a few times, I’m not sure how to diagnose/debug, so any advise would be appreciated.


    • sometimes the router will randomly disconnect; the connection icon on my desktop (wired to the router) gets that yellow “!” symbol that tells me my connection just went down. At this point I’m unable to ping the router.
    • afterwards I try to reset the router by removing then reconnecting the power jack on the router side (this is the fastest way as I can’t reset the power strip it’s connected to without rebooting my desktop. the router has a reset thingy, but it’s one of those things where i have to find a pin to stick into the hole, and when I get disconnected I usually need to get reconnected immediately so I just pull and put back the power jack), but even after that the connection has the same state.
    • after the router reboots, if I try to connect to it using a wifi device like my ipad, the ipad prompts me for the wifi password even though it had already “remembered” all the settings for this router forever
    • after i finally decide to reboot the power strip, and my desktop and the router boot up again, the connection returns to its normal state somewhat and i’m able to connect to it as normal using the desktop and wifi devices.

    What do I need to check the next time this happens so I can figure out the problem? Is it possibly because we’ve been using the power jack on the router as an easier way to reboot it? Should I be shopping around for a new router?

    If it helps, the router is a DLink DIR-300

Aug 2012

  • I’m being asked to assess whether we can safely upgrade the java version on one of our production-deployed webapps. The codebase is fairly large and we want to avoid having to regression test everything (no automated tests sadly), but we’ve already encountered at least one problem during some manual testing (XmlStringReader.getLocalName now throws an IllegalStateExeption when it just used to return null) and higher-ups are pretty nervous about the upgrade.

    The current suggested approach is to do a source compare of the JDK sources for each version and assess those changes to see which ones might have impact, but it seems there’s a lot of changes to go through (and as mentioned the codebase is kinda large). Is it safe and easier to just review the java version changes for each version? Or is there an easier way to conduct this assessment?

    Edit: I forgot to mention the version upgrade being considered is a minor version upgrade, i.e. 1.6.10 to 1.6.33