HPCC
  1. HPCC
  2. HPCC-12395 Add a plugin to support redis as a key/value store
  3. HPCC-13158

Correct behaviour of GetOrLock on an empty/non-existent key and SetAndPublish to only transmit data once

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 5.2.0
    • Fix Version/s: 5.2.0
    • Component/s: Plugins
    • Labels:
      None

      Issue Links

        Activity

        Hide
        Jamie Noss added a comment - - edited

        At present, a GetOrLock conducts the following logic:

        if (SET NX) // Attempt lock
        return; // Return if non existent key that is now locked.
        SUB;
        GET;
        assertKey(); // If the key does not exist fail. This conforms with the behaviour of the non-locking Get counterparts.
        if (replyFromGETIsNotLocked)
        return reply;
        listenForMessageViaSUB
        If (messageReceived)
        return message;
        rtlFail();

        In the locking SET GET paradigm, upon a cache miss, the key SET is passed to waiting SUbscribers via a PUBlish rather than have them GET.
        1) This prevents a further race condition in getting the correct value, that explicitly SET at the time that the GET encountered the lock.
        2) That the GETting SUBscriber receives the value even if the SETter fails to complete the SET. (It may be thought that if the SET fails then the GETer should take on the role
        of the SETter, except that this may be poor behaviour when the value that was trying to be SET was successfully retrieved from either an expensive computation or monetary exchange and only the cache SET failed. Therefore, it is not desirable to have a new SETter that would repeat the evaluation process. There is no trivial way to relay this to SUBscribers and have them continue from there, accounting for this. It can be done but involves further messaging schemas and race conditions i.e. who out of the SUBscribers is now the new SETter?)

        The behaviour I intended and documented in the README would only retun if the key was not locked and not empty, yet fails if the value published to it is empty representing a non existent key and thus conforming with the behaviour of the other GETs.

        Show
        Jamie Noss added a comment - - edited At present, a GetOrLock conducts the following logic: if (SET NX) // Attempt lock return; // Return if non existent key that is now locked. SUB; GET; assertKey(); // If the key does not exist fail. This conforms with the behaviour of the non-locking Get counterparts. if (replyFromGETIsNotLocked) return reply; listenForMessageViaSUB If (messageReceived) return message; rtlFail(); In the locking SET GET paradigm, upon a cache miss, the key SET is passed to waiting SUbscribers via a PUBlish rather than have them GET. 1) This prevents a further race condition in getting the correct value, that explicitly SET at the time that the GET encountered the lock. 2) That the GETting SUBscriber receives the value even if the SETter fails to complete the SET. (It may be thought that if the SET fails then the GETer should take on the role of the SETter, except that this may be poor behaviour when the value that was trying to be SET was successfully retrieved from either an expensive computation or monetary exchange and only the cache SET failed. Therefore, it is not desirable to have a new SETter that would repeat the evaluation process. There is no trivial way to relay this to SUBscribers and have them continue from there, accounting for this. It can be done but involves further messaging schemas and race conditions i.e. who out of the SUBscribers is now the new SETter?) The behaviour I intended and documented in the README would only retun if the key was not locked and not empty, yet fails if the value published to it is empty representing a non existent key and thus conforming with the behaviour of the other GETs.
        Hide
        Jamie Noss added a comment -

        GetOrLock will now only return an empty string if the initial lock attempt succeeded on an non existent key. This is imperative for the locking logic within the ECL, such that all other empty string cases either recover or result in an exception.

        SetAndPublish now optimizes the data value transmission to the server, either by sending it once with a Lua script to SET & PUBLISH server-side, or if the data is small enough, send twice as an atomic SET-PUBLISH.

        Show
        Jamie Noss added a comment - GetOrLock will now only return an empty string if the initial lock attempt succeeded on an non existent key. This is imperative for the locking logic within the ECL, such that all other empty string cases either recover or result in an exception. SetAndPublish now optimizes the data value transmission to the server, either by sending it once with a Lua script to SET & PUBLISH server-side, or if the data is small enough, send twice as an atomic SET-PUBLISH.

          People

          • Assignee:
            Jamie Noss
            Reporter:
            Jamie Noss
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: