Versioning ignores versions_retention_obligation `D, auto`

[GAS85]

Steps to reproduce

1. Configure versions_retention_obligation to .e.g. "versions_retention_obligation": "14, auto", as described here https://github.com/nextcloud/server/blob/master/config/config.sample.php#L603.

 * * ``D, auto``
 *     keep versions at least for D days, apply expire rules to all versions
 *     that are older than D days

2. Create file, e.g. todo.txt
3. Update this file few times
4. See that older version are deleted, but they must be kept for a configured (here 14) amount of days.

Expected behaviour

Older files versions must be handled as it is configured in config.php and as it is described in a Doku.

Actual behaviour

Older files versions has being deleted independent of settings in config.php:

Server configuration

Operating system: Ubuntu 18.04

Web server: Apache 2.4.41

Database: 10.1.44-MariaDB

PHP version: 7.3.15

Nextcloud version: 17.0.3

Updated from an older Nextcloud/ownCloud or fresh install: update

Where did you install Nextcloud from: Official

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled:
  - accessibility: 1.3.0
  - activity: 2.10.1
  - admin_audit: 1.7.0
  - audioplayer: 2.9.0
  - bruteforcesettings: 1.5.0
  - calendar: 2.0.2
  - checksum: 0.4.4
  - cloud_federation_api: 1.0.0
  - comments: 1.7.0
  - data_request: 1.4.0
  - dav: 1.13.0
  - deck: 0.8.0
  - federatedfilesharing: 1.7.0
  - federation: 1.7.0
  - files: 1.12.0
  - files_automatedtagging: 1.7.0
  - files_external: 1.8.0
  - files_pdfviewer: 1.6.0
  - files_retention: 1.6.0
  - files_rightclick: 0.15.1
  - files_sharing: 1.9.0
  - files_trashbin: 1.7.0
  - files_versions: 1.10.0
  - files_videoplayer: 1.6.0
  - firstrunwizard: 2.6.0
  - flowupload: 0.1.8
  - gallery: 18.4.0
  - gpxpod: 4.1.1
  - keeweb: 0.6.2
  - logreader: 2.2.0
  - lookup_server_connector: 1.5.0
  - mail: 1.1.3
  - maps: 0.1.5
  - nextcloud_announcements: 1.6.0
  - notes: 3.1.5
  - notifications: 2.5.0
  - oauth2: 1.5.0
  - ocdownloader: 1.7.6
  - password_policy: 1.7.0
  - phonetrack: 0.6.2
  - polls: 1.3.0
  - previewgenerator: 2.2.0
  - privacy: 1.1.0
  - provisioning_api: 1.7.0
  - radio: 0.6.6
  - recommendations: 0.5.0
  - serverinfo: 1.7.0
  - sharebymail: 1.7.0
  - survey_client: 1.5.0
  - systemtags: 1.7.0
  - text: 1.1.1
  - theming: 1.8.0
  - twofactor_backupcodes: 1.6.0
  - twofactor_totp: 4.1.2
  - unsplash: 1.1.5
  - updatenotification: 1.7.0
  - viewer: 1.2.0
  - weather: 1.7.1
  - workflowengine: 1.7.0
Disabled:
  - encryption
  - impersonate
  - sharerenamer
  - support
  - user_ldap

Nextcloud configuration:

Config report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": {
            "0": "***REMOVED SENSITIVE VALUE***",
            "2": "***REMOVED SENSITIVE VALUE***"
        },
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/***REMOVED SENSITIVE VALUE***\/nextcloud",
        "dbtype": "mysql",
        "version": "17.0.3.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "filesystem_check_changes": 0,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "timeout": 1.5
        },
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "theme": "",
        "logfile": "\/var\/nextcloud\/data\/nextcloud.log",
        "loglevel": 1,
        "trashbin_retention_obligation": "14, auto",
        "versions_retention_obligation": "14, auto",
        "data-fingerprint": "***REMOVED SENSITIVE VALUE***",
        "enable_previews": true,
        "enabledPreviewProviders": [
            "OC\\Preview\\PNG",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\GIF",
            "OC\\Preview\\BMP",
            "OC\\Preview\\XBitmap",
            "OC\\Preview\\Movie",
            "OC\\Preview\\PDF",
            "OC\\Preview\\MP3",
            "OC\\Preview\\TXT",
            "OC\\Preview\\MarkDown"
        ],
        "preview_max_x": 1080,
        "preview_max_y": 1920,
        "auth.bruteforce.protection.enabled": true,
        "simpleSignUpLink.shown": false,
        "mail_smtpsecure": "tls",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [
            "admin"
        ],
        "twofactor_enforced_excluded_groups": [],
        "has_rebuilt_cache": true,
        "updater.release.channel": "stable",
        "app_install_overwrite": [
            "keeweb",
            "radio"
        ]
    }
}

Are you using external storage, if yes which one: local smb

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Nextcloud log (data/nextcloud.log)

Nextcloud log

{"reqId":"KiDyFYmuoD4HjXfIhfs7","level":1,"time":"2020-03-05T10:00:39+00:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Mark to expire \/todo.txt next version should be 1583401846 or smaller. (prevTimestamp: 1583401906; step: 60","userAgent":"--","version":"17.0.3.1"}
{"reqId":"KiDyFYmuoD4HjXfIhfs7","level":1,"time":"2020-03-05T10:00:39+00:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Mark to expire \/todo.txt next version should be 1583401846 or smaller. (prevTimestamp: 1583401906; step: 60","userAgent":"--","version":"17.0.3.1"}
{"reqId":"KiDyFYmuoD4HjXfIhfs7","level":1,"time":"2020-03-05T10:00:39+00:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Mark to expire \/todo.txt next version should be 1583401846 or smaller. (prevTimestamp: 1583401906; step: 60","userAgent":"--","version":"17.0.3.1"}
{"reqId":"KiDyFYmuoD4HjXfIhfs7","level":1,"time":"2020-03-05T10:00:39+00:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Expire: \/todo.txt.v1583401886","userAgent":"--","version":"17.0.3.1"}
{"reqId":"KiDyFYmuoD4HjXfIhfs7","level":1,"time":"2020-03-05T10:00:40+00:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Expire: \/todo.txt.v1583401876","userAgent":"--","version":"17.0.3.1"}
{"reqId":"KiDyFYmuoD4HjXfIhfs7","level":1,"time":"2020-03-05T10:00:40+00:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Expire: \/todo.txt.v1583401859","userAgent":"--","version":"17.0.3.1"}

[Peac]

Same here on 18.0.3.

I configured my Nextcloud to keep all versions for a month and apply the auto rule afterwards. When the day came, that I needed a specific version, it was gone. Quite a problem, if you trust that a documented feature works as it should.

[szaimen]

Is this Issue still valid in NC21.0.2? If not, please close this issue. Thanks! :)

[GAS85]

For NC 21 please check it, I did not perform update. For NC 20.0.10 still the Problem.
E.g. 10 Days ago was 3 updates performed:

But only 1 Version saved:

According to config, all versions for last 14 days should be saved:

[szaimen]

Hey @icewind1991,
for me the reported bug looks like documentation is missing about the versions_retention_obligation setting because it seems to work correctly but missing some details: For ‘D, auto‘ it seems to clear some versions in between (probably due to background jobs) and all remaining versions after the D setting expires. If this should be true, I would create a PR that makes this clear. Would be great if you could reply if this is working as intended or a bug.
Thanks! :)

[GAS85]

it seems to clear some versions in between (probably due to background jobs) and all remaining versions after the D setting expires

Actually not, you can see on screenshot that for the rule 14, auto I have versions that are not saved during last 14 days but versions beyond working fine.
On a new screenshot you can see (I hope, have to unzoom it to make sceenshot) that even versions around 1 year old are presented, but "keep it all for at least 14 days" not respected.

Screenshot

What you are pointing is a valid use case, but should be configured via auto, D instead, as per https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/config_sample_php_parameters.html#file-versions

[szaimen]

Thanks for the clarification! So yes, this smells like a bug.