Notes are being deleted

[genuser121]

Operating system

Windows

Joplin version

3.2.11

Desktop version info

Joplin 3.2.11 (prod, win32)

Client ID: 4c9585ebda424a3e98d5e0f94ef69d5a
Sync Version: 3
Profile Version: 47
Keychain Supported: Yes

Revision: 2fba101

Backup: 1.4.2
Freehand Drawing: 2.14.0
Markdown Table: Colorize: 1.1.4
Quick HTML tags: 0.2.0
Rich Markdown: 0.13.2
Spoilers: 1.0.6
turnToChart: 1.9.3

Current behaviour

1. I edit a note on my laptop
2. I edit the same note on my android phone
3. Instead of creating a conflict, it deletes the older version of the note without telling you

I caught this happening twice now. Once I caught it happening right in front of me where the text I had just written disappeared. There were a bunch of conflicts but the text I had written didn't appear in any of them. Then I caught it happening again when I was searching the app for some notes that I had written days ago. When they didn't show up in the search results, I knew it had done it again. Both times I found the notes only in the note history function. There's no knowing how many times it's done this or if it's happening to other people because it's something I only noticed and stumbled upon by chance.

Android version is 3.4.7 / 2097780.

Expected behaviour

Create conflicts. The whole point of the conflict function is to prevent precisely this from happening. My notes are super important to me which is why I used an open source app with a cloud function, to ensure my notes are absolutely secure. This sync conflict issue is thus a major bug for me and I imagine it would be so for others too.

Logs

No response

[mrjo118]

What sync target are you using?
Also, if you manually click the sync button (and wait for it to complete) on both clients after making changes on both devices, is a conflict created then on either device? Conflicts should appear available in a 'Conflicts' notebook at the top of the notebook list

Additionally, your mobile app version is recent, but your windows app is 2 major versions behind. It may help to update to the latest stable version (3.4.12)

[dejafait]

Silent loss of conflicting changes has been randomly happening a few times to me as well. Using one 3.4.12 MacOS device and several 3.4.7 Android devices. Issue is flaky, not easily reproducible. Just attempted to reproduce it without success.

When bug does not happen: conflicting changes appear as expected in "Conflicts" folder at the top of notebooks.

When bug does happen: no Conflicts folder. One of the changes is silently lost.

Recent posts about it:

https://www.reddit.com/r/joplinapp/s/qGW1V7Dkhp

https://www.reddit.com/r/joplinapp/s/2Dz3mShs9r

Ftr everytime it happened I was able to recover lost input in note revision history.

[dejafait]

@genuser121 Just to clarify, "Notes are being deleted" is not technically correct, no note is deleted per se (by note I mean this: you have several notebooks/folders, each one has several notes, each note has a title and a content/body), however what is deleted (silently lost) is some conflicting changes and inputs made to the content of a note. Do we agree?

[mrjo118]

@dejafait FYI both Reddit posts were created by the author of this issue. What sync target are you using by the way? Also when you notice this happening, is the content missing on the desktop client or the Android client? It was only in 3.4 that the conflicts folder was moved to the top of the notebook list on Android. There has been a few times I noticed a change missing, but on every occasion a conflict was created but I just hadn't noticed it, or possibly it didn't get created or shown immediately, potentially due to a notebook refresh issue.

Note that the sync logic in Joplin does time adjustment to cater for variation with device time across devices. However the conflict detection does not do any time adjustment, and it makes use of the updated_time which was stored within the sqlite database (which is derived using the local device time) to determine when the note was modified. So if you make changes to the same note on a couple of devices within a few seconds of each other (regardless of whether sync has completed), and the variation of the local device time on the 2 devices is greater than the time between modifications, then the conflict resolution is going to think the newer change is actually older.

It's not likely there will be a huge time variation of your devices if they sync to the internet time, but it's not uncommon there might be discrepencies of up 10 seconds or so of variation. There are apps available for Android which allow you to see the device time including seconds

[dejafait]

@mrjo118

What sync target are you using by the way?

Joplin Server v3.4.3

Also when you notice this happening, is the content missing on the desktop client or the Android client?

All devices show the same exact content after sync. Assuming I did change A on one device and change B on another device, in the end A ends up on all devices and B is silently lost (instead of being put in a Conflicts folder). B can still be found in the note revision history though.

It was only in 3.4 that the conflicts folder was moved to the top of the notebook list on Android. There has been a few times I noticed a change missing, but on every occasion a conflict was created but I just hadn't noticed it, or possibly it didn't get created or shown immediately, potentially due to a notebook refresh issue.

When it happened, there was no Conflicts folder anywhere, top or bottom, on any device. This is the bug. I am very familiar with the Conflicts folder which works... most of the time. I get one once or twice a week and it works smoothly.

So if you make changes to the same note on a couple of devices within a few seconds of each other (regardless of whether sync has completed), and the variation of the local device time on the 2 devices is greater than the time between modifications, then the conflict resolution is going to think the newer change is actually older.

Wow thanks for explaining this. As long as update times are used here I don't see any solution. May I suggest a more resilient strategy if it makes sense, which keeps working even when device times are messed up. After all, update times are not really needed nor reliable here.

Let's assume each revision of my note knows its parent revision. Usually the revision tree is a trivially linear chain, but sometimes it can "fork" due to conflicts.

When any device creates a new revision, it knows and records in it the parent revision it was based on.

Conflicts detection becomes quite simple and reliable., and does not need update times at all. When gathering latest revisions from devices, usually the revision tree is trivially linear. As soon as it no longer is (e.g. two revisions with the same parent, or a new revision whose parent is not the tip of the chain but an older revision which already had one child and a grand child) then consider there is a conflict and trigger the existing Conflicts folder process, "better safe than sorry".

False negatives are extremely expensive here, they can seriously undermine user trust of the app. False positives are no problem, they even feel a little reassuring, as in "cool, nice to know the conflicts detection is working".

My 2cts.

[genuser121]

@genuser121 Just to clarify, "Notes are being deleted" is not technically correct, no note is deleted per se (by note I mean this: you have several notebooks/folders, each one has several notes, each note has a title and a content/body), however what is deleted (silently lost) is some conflicting changes and inputs made to the content of a note. Do we agree?

yes and no. yes, i've witnessed content from the notes being removed but it's via means of one version of a note being chosen over another and the other version being deleted. But yes, an entire note is not being deleted, just a version of it with content unique to it. I'm glad someone else has witnessed it bc I thought i was all alone here.

[genuser121]

What sync target are you using?

I use dropbox currently.

Also, if you manually click the sync button (and wait for it to complete) on both clients after making changes on both devices, is a conflict created then on either device? Conflicts should appear available in a 'Conflicts' notebook at the top of the notebook list

Actually yes. It's an issue I've mentioned before like a few months back on the subreddit. I will get conflicts when there is no reason for there to be conflicts. It just creates them without provocation. This is essentially why I only noticed the bug recently. Because I knew this problem existed and it was tedious to deal with, I would only edit certain notes on one device and other notes on the other so there was no possibility of overlap and conflict. But that in itself was very tedious so I stopped doing it. I started using both devices to edit notes, not at the same time but whenever it was convenient, and then I noticed this newfound bug.

[mrjo118]

Also, if you manually click the sync button (and wait for it to complete) on both clients after making changes on both devices, is a conflict created then on either device? Conflicts should appear available in a 'Conflicts' notebook at the top of the notebook list

Actually yes. It's an issue I've mentioned before like a few months back on the subreddit. I will get conflicts when there is no reason for there to be conflicts. It just creates them without provocation.

@genuser121 Are you saying this happens when you sync both devices after the old note contents are 'lost', or are you referring to an issue that happens on separate occasions?

If you mean the former, then there's not really anything which the Joplin app can do better. Say you make a change on device A, then make a change on device B when the sync has not yet completed on device A. If you then sync device B only, there is no way it can know there is a conflict because the change on device A has not yet been synced. When you later sync device A however, it should then create a conflict at that point, and therefore the note changes previously made are not lost, because they are available in the conflicts folder.

Also, do you experience the issue (I mean the issue in your OP, if these are in fact 2 separate issues) when using multiple Joplin clients simultaneously, or do you have only 1 Joplin client open at a time?

And when you make the changes to the same note on 2 devices, are the changes made close together within a few seconds, or is it minutes / hours later?

[mrjo118]

Wow thanks for explaining this. As long as update times are used here I don't see any solution. May I suggest a more resilient strategy if it makes sense, which keeps working even when device times are messed up. After all, update times are not really needed nor reliable here.

@dejafait I think the reason the updated_time metadata for a note uses the device time is because Joplin is an offline first and sync agnostic app. So it can't rely on a time server, and if it you do time adjustment using the sync target, you will be coupling the timestamp to the sync target currently in use, and then if you decided to change sync target, all of the timestamps could be out of sync. For sync, it is not a problem to do time adjustment coupled to the sync target, because all the timestamps which will be compared against are stored separetely for the sync target in use.

When any device creates a new revision, it knows and records in it the parent revision it was based on.

You can't rely on revisions for the conflict logic, as they can only be created maximum every 10 minutes for the current client.

Do you think that editing the same note across separate devices within a few seconds of each other is likely the case for when you experienced the issue (and therefore it may have been a device time drift issue), or were you just commenting on this point due to surprise about this being the case?

EDIT: I did a bit of research and it seems it's mostly pc's that are likely to get significant time drift, while with mobile devices it's not really an issue because they sync to the time server so regularly by default. According to Google, Windows 10 will sync with the time server by default once a week and on every boot. According to ChatGPT, a typical quartz drift for a computer is ~1 second per day. So if you boot your pc every time you use it and don't use it for long periods, then you'll probably never experience the issue, but if you leave your pc on all the time, you could experience significant time drift until the next time sync. A simple solution to ensure you never face an issue related to this would be to configure any desktop OS you use to sync to the time server more frequently, eg. daily or hourly

[genuser121]

@genuser121 Are you saying this happens when you sync both devices after the old note contents are 'lost', or are you referring to an issue that happens on separate occasions?

No this is a separate and different bug also to do with conflicts where it will create fake "conflicts" for no reason. There are no actual conflicts happening but it creates conflicts anyway.

And when you make the changes to the same note on 2 devices, are the changes made close together within a few seconds, or is it minutes / hours later?

Well it's hard to say exactly what happened because it's something that I stumbled onto days after the missing content was written. But I don't recall ever using the two devices simultaneously, especially to edit the same note. I just wouldn't have any reason to do that and I already know that that would create problems. I might edit them close to each other such as within a few minutes of each other. But not at the same time.

Also, do you experience the issue (I mean the issue in your OP, if these are in fact 2 separate issues) when using multiple Joplin clients simultaneously, or do you have only 1 Joplin client open at a time?

It's impossible for me to remember that but it very well could be possible. I could definitely see me having both clients open simultaneously. When my laptop is on, obviously Joplin is always open. But on my phone, it's obviously closed most of the time. But sometimes I might just leave it open and using RAM, but my phone would likely be off.

[mrjo118]

@genuser121 Thank you for your reply.

I just reread your OP and I think what you are decribing there is also different to what @dejafait has described.

So to clarify, you're saying you always DO get a conflict created when you expected there to be a conflict, but the conflict does not contain the text you expected?

[genuser121]

So to clarify, you're saying you always DO get a conflict created when you expected there to be a conflict, but the conflict does not contain the text you expected?

No that's a different and separate bug I was mentioning. In this bug, i'm not sure if conflicts are being created or not. all i know for sure is that conflicts aren't being created where they should be. i edit a note on one device. i later edit the same note on another device. and instead of creating a conflict, it chooses one version of the note over another and deletes the other without creating a conflict.

when i said it's a different and separate bug and explained it, i was talking about another bug, not this bug. i believe this bug that this thread is about is the same bug that @dejafait has also witnessed. a question you asked me earlier made me talk about another bug i was experiencing that was a different and separate bug that's also to do with conflicts. bit confusing, i know

[mrjo118]

it chooses one version of the note over another.

@genuser121 As far as I can tell from looking at the code, it is intended behaviour for the original note to be replaced with the remote version when a conflict occurs, even if you happen to be editing the note when that happens. But when working correctly, there should also be a note with the same name created within a 'Conflicts' notebook. If you're not actually sure whether that is true or not because you did not pay specific attention to this, I can only assume that it probably working ok, as I can't see a way in the code which would allow the content to get replaced in front of your eyes AND not create a conflict (with the exception of the time drift scenario).

However...

There were a bunch of conflicts but the text I had written didn't appear in any of them

You mentioned this in your OP which suggests at least on one occasion you did notice a 'Conflicts' notebook and you looked in it. After looking at the code, I think there is a possibility that there is a bug which would cause a conflict to be created with outdated contents, but I need to have a deeper look and prove this before coming to that conclusion