/
NEWS
9190 lines (6975 loc) · 403 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
SyncEvolution 1.5.3 -> 2.0.0, 21.03.2021
========================================
This release modernizes the code base and removes usage of out-dated
libraries and APIs. All Python scripts require Python 3. The major
version bump reflects that this release is not just a minor bug fix.
However, no new features were added.
Binaries on syncevolution.org get built for distros >= Ubuntu Bionic
and Debian Buster. Testing is now based on Docker containers instead
of a custom schroot solution, so adding testing against other distros
will be easier in the future. Compilation on Fedora Rawhide was
already added.
Some features are no longer built and thus untested:
- ActiveSync
- KDE
The code is still there, but needs help from interested developers to
ensure that it really works. It may get removed in a future version if
there is no interest.
SyncEvolution 1.5.2 -> 1.5.3, 03.01.2018
========================================
Maintenance release. syncevolution.org binaries are now getting
compiled for distros >= Ubuntu Xenial 16.04 LTS. Usage of deprecated
libraries (GNOME keyring) and APIs (SoupAsyncSession) was
replaced. libical v3 is supported.
The code now compiles more cleanly with recent compilers and depends
on C++11 support.
Details:
* EDS: more generic open retry handling
Recent EDS started to exhibit race conditions when opening a database (for
example, https://bugzilla.gnome.org/show_bug.cgi?id=791306). Opening was
already tried again for a certain known error in some old EDS version. Now it
is tried five times with a delay of one second for all errors.
* SoupTransportAgent: require libsoup 2.42, no deprecated methods
This allows us to get rid of deprecated function calls. We no longer
need to set a default proxy either, the newer libsoup does that itself
by default.
* C++: replace auto_ptr with unique_ptr, require C++11
auto_ptr has been deprecated for a while now. unique_ptr can
be taken for granted now, so use that instead.
* testing: work around Google CalDAV RECURRENCE-ID
Stand-alone events with RECURRENCE-ID get mangled by the server:
it converts the RECURRENCE-ID time to UTC. Reported in:
https://stackoverflow.com/questions/47811670/detached-recurrence-without-parent-event
* GNOME: replace gnome-keyring with libsecret (FDO #104219)
The GNOME keyring library has been obsoleted for a long time now,
long enough that the replacement libsecret is available on all
supported distros. Therefore we can switch unconditionally.
* libical: support libical v3 (FDO #104220)
libical v3 removes some deprecated functions (like icaltime_from_timet)
and removes the "is_utc" member from icaltimetype. The replacement
code works with old and new libical and thus needs no ifdefs.
Original author: Milan Crha
* syncevolution.org: fixed packaging (FDO #98014, FDO #100549)
The activesyncd package missing dependencies on libgnome-keyring0 and
libglib2.0-bin and therefore failed to work when installed on a minimal
system without those.
* various build and test fixes/workarounds
SyncEvolution 1.5.1 -> 1.5.2, 08.11.2016
========================================
Maintenance release. syncevolution.org binaries are now getting
compiled for distros >= Ubuntu Trusty 14.04 LTS, which allowed
removing several hacks that were needed when building binaries that
also had to run on older distros. Compilation from source for old
distros should still work as before, but is not getting tested
anymore.
Compile problems with recent libraries (libical v2) and tools (GCC v6)
were resolved. Syncing via Bluetooth with certain phones now should
work reliably in incremental mode.
New backends for the Trinity Desktop Environment (TDE) were added to
the source code.
Details:
* ObexTransportAgent.cpp: properly shut down connection (FDO #91485)
Apparently there's a race condition in the OBEX transport that causes the
connection to phones via Bluetooth to be shut down prematurely. Some phones
react by doing a slow sync instead of an incremental sync the next time.
* support non-readable parent directories (FDO #91000)
The previous mkdir_p() walked down top to bottom and checked each path
entry as it went along. That approach failed unnecessarily when some
existing parent directory could not be read (non-readable /home, for
example).
* avoid using dbus-launch (Debian #836399)
dbus-launch is considered deprecated because of the X11 dependency.
See https://lists.debian.org/debian-devel/2016/08/msg00554.html "Mass bug
filing: use and misuse of dbus-launch (dbus-x11)"
The dbus-session.sh script still needs to start the D-Bus daemon when used in the nightly
testing, so the code now does it by invoking the dbus-daemon directly.
syncevo-http-server still has some usage of dbus-launch left, but that's
strictly for systems which don't have the more modern D-Bus.
* syncevo-dbus-server integrates better with systemd (FDO #92164)
A .service file allows the D-Bus daemon to start the service via systemd,
thus ensuring that the process environment is correct. Patch from Simon
McVittie. Auto-starting as part of the desktop login uses D-Bus activation
if the "dbus-send" tool is installed.
* syncevolution.org: compile on Ubuntu Trusty, libical v1/v2 compatibility
syncevolution.org binaries are now getting compiled on Ubuntu Trusty and thus
no longer support distros with older EDS. The code should still compile
against older EDS (for example, for Maemo), but that is not getting tested
anymore.
This allows removing the dynamic linker hacks related to older libraries,
which was only used in those binaries. Instead, backends using libical or EDS
get compiled on Ubuntu Trusty and then the soname of those libs get patched to
make the backend module usable in combination with a different set of
libs. That patching is part of a script maintained in the syncevolution.org
build infrastructure.
This approach was already used before to generate different EDS backends
for EDS versions with the newer EClient API, because that turned out to be
easier than the dynamic loading approach. It works because none of the methods
used by SyncEvolution changed their ABI, only some other parts of the
libraries did. Should there ever be a situation again that cannot be handled
like this, then backends also get compiled on different distros than
Ubuntu Trusty (for example, the ActiveSync backend for Debian Stretch
is built on Debian Stretch).
libical still requires one special hack: system time zone loading in
libical v1 (and only in that version, v2 has builtin support again) must
be overridden such that time zones are generated with rules instead
of transitions because that is more compatible with the peers that
SyncEvolution exchanges data with.
That hack now relies on overriding the two relevant functions inside the main
binaries (has to be there, otherwise libical still ends up calling its own
internal implementation). The overriding code is in
libsyncevo-icaltz-util.so.0 and depends on libical.so.1. If
libsyncevo-icaltz-util.so.0 can be loaded, the wrappers in the main binary use
it, otherwise they fall through to the code from the current libical.so, which
then should be libical.so.2 or more recent.
This hack is active by default when libical v1 is detected during configuration.
* optionally show debug output in --version output
SYNCEVOLUTION_DEBUG=1 syncevolution --daemon=no --version now
dumps also the debug information gathered by the binary
compatibility code. It was only available in sync logs before.
* various build fixes for libical v2, GCC v6/C++14
SyncEvolution 1.5 -> 1.5.1, 05.06.2015
======================================
Maintenance release. Binaries now also get compiled for Debian 8.0
"Jessie".
Details:
* avoid time zone issue with Funambol server
The Funambol iCalendar 2.0 parser fails to handle time zones
with quotation marks around the TZID value, which is something
that SyncEvolution started to add in 1.4.99.3. While it is valid
to quote like that, it is not necessary, so avoid quoting in
this case to restore interoperability.
* syncevo-http-server: stop using deprecated twisted.web.error (FDO #90419)
This has become a real problem for example on Fedora 22 where the
old name is no longer available.
* syncevo-http-server: use TLS instead of SSLv3
This fixes a potential security risk and connection problems with clients
that don't support SSLv3 anymore.
* syncing: avoid segfault for invalid text inside items (FDO #90118)
As reported by Canonical, syncing fails if data items contain
text which is not correct UTF-8 in one of the fields that
SyncEvolution logs in the command line output (like SUMMARY of
a calendar event).
That is because the byte string coming from the item is passed
unchecked to the D-Bus implementation for transmission via D-Bus. But
D-Bus strings must be correct UTF-8, so depending on the D-Bus library
in use, one gets a segfault (GIO D-Bus, due to an unchecked NULL
pointer access) or an "out of memory" error (libdbus, which checks for
NULL).
SyncEvolution now replaces invalid bytes with a question mark in its
output while preserving the rest of the text.
* file backend: log item manipulation
Extracting a meaningful description of each item from the Synthesis
engine when updating and adding items is easy to do for items of
certain known types (contacts and calendar items).
* command line: preserve log prefix of target side of local sync
In some cases, the prefix which was supposed to be embedded
in the log messages from the target side of a local sync got
lost on the way to the command line tool.
Primarily this affected the added/updated/deleted messages, as in:
[INFO remote@client] @client/addressbook: started
[INFO remote@client] updating "Joan Doe"
[INFO remote@client] @client/addressbook: received 1/1
* compile fix: use ${PKG_CONFIG} instead of pkg-config.
This fixes the build on Exherbo that only has prefixed versions of
pkg-config.
* WebDAV: handle 403 during Google OAuth authentication
When sending an access token with insufficient scope (for example,
because the Ubuntu Online Accounts service definition was incomplete,
as documented in FDO #86824), Google responds with a 403 "service
denied" error.
This is now dealt with by retrying, just as for a transient 401 error.
* CalDAV: more efficient "is empty" check (FDO #86335)
Since 1.4.99.4, syncing WebDAV collections always checks first
whether there are items in the collections. This was partly done for
slow sync prevention (which is not necessary for empty collections),
partly for the "is the datastore usable" check.
However, this did not take into account that for CalDAV collections,
the entire content gets downloaded for this check. That is because
filtering by item type (VEVENT vs. VJOURNAL) is not implemented
correctly by all servers. So now all CalDAV syncs, whether incremental
or slow, always transfered all items, which is not the
intention (incremental syncs should be fast and efficient).
This release adds a more efficient isEmpty() check: for simple CardDAV
collections, only luid and etag get transferred, as in
listAllItems(). This is the behavior from 1.5.
For CalDAV, a report with a filter for the content type is used and
the transfer gets aborted after the first item, without actually
double-checking the content of the item. This is different from
listAllItems(), which really transfers the content. This extra content
check would only be needed for some old servers (Radical 0.7) and is
not essential, because reporting "not empty" even when empty is safe.
* WebDAV: send Basic Auth via http in some cases (FDO #57248)
It turned out that finding databases on an Apple Calendar server accessed via
http depends on sending Basic Auth even when the server does not ask for it:
without authentication, there is no information about the current principal,
which is necessary for finding the user's databases.
To make this work again, sending the authentication header is now forced for
plain http if (and only if) the request which should have returned the
principal URL fails to include it. This implies sending the same request
twice, but as this scenario should be rare in practise (was only done for
testing), this is acceptable.
* Ubuntu Online Accounts: support plain text credentials
The backend for UOA was rewritten by Alberto Mardegan and now also
can use plain username/password credentials stored in UOA.
* various compiler error and warning fixes
SyncEvolution 1.4.1 -> 1.5, 31.10.2014
======================================
Based on community feedback and discussions, the terminology used in
SyncEvolution for configuration, local sync and database access was
revised. Some usability issues with setting up access to databases
were addressed.
Interoperability with WebDAV servers and in particular Google Contacts
was enhanced considerably. Access to iCloud contacts was reported as
working when using username=foobar@icloud.com and password, but is not
formally tested. Syncing with iCloud calendars ran into a server
limitation (reported as 17001498 "CalDAV REPORT drops calendar data")
and needs further work (FDO #72133).
Contact data gets converted to and from the format typically used by
CardDAV servers, so now anniversary, spouse, manager, assistant and
instant message information are exchanged properly. Custom labels get
stored in EDS as extensions and no longer get lost when updating some
other aspects of a contact. However, Evolution does not show custom
labels and removes them when editing a property which has a custom
label (BGO #730636).
Scanning for CardDAV/CalDAV resources was enhanced. It now finds
additional calendars with Google CalDAV. For Google, the obsolete
SyncML config template was removed and CalDAV/CardDAV were merged into
a single "Google" template.
Using Google Calendar/Contacts with OAuth2 authentication on a
headless server becomes a bit easier: it is possible to set up access
on one system with a GUI using either gSSO or GNOME Online Accounts,
then take the OAuth2 refresh token and use it in SyncEvolution on a
different system. See
http://cgit.freedesktop.org/SyncEvolution/syncevolution/tree/src/backends/oauth2/README
The PIM Manager API also supports Google Contact syncing. Some
problems with suspending a PBAP sync were fixed. Suspend/abort can
be tested with the sync.py example.
Performance is better for local syncs and PBAP caching. The most
common case, a two-way sync with no changes on either side, no longer
rewrites any meta data files. CPU consumption during local sync was
reduced to one third by exchanging messages via shared memory instead
of internal D-Bus. Redundant vCard decode/encode on the sending side
of PBAP and too agressive flushing of meta data during a normal sync
were removed.
The EDS memo backend is able to switch between syncing in plain
text and iCalendar 2.0 VJOURNAL automatically.
Details:
* source -> datastore rename, improved terminology
The word "source" implies reading, while in fact access is read/write.
"datastore" avoids that misconception. Writing it in one word emphasizes
that it is single entity.
While renaming, also remove references to explicit --*-property
parameters. The only necessary use today is "--sync-property ?"
and "--datastore-property ?".
--datastore-property was used instead of the short --store-property
because "store" might be mistaken for the verb. It doesn't matter
that it is longer because it doesn't get typed often.
--source-property must remain valid for backward compatility.
As many user-visible instances of "source" as possible got replaced in
text strings by the newer term "datastore". Debug messages were left
unchanged unless some regex happened to match it.
The source code will continue to use the old variable and class names
based on "source".
Various documentation enhancements:
Better explain what local sync is and how it involves two sync
configs. "originating config" gets introduces instead of just
"sync config".
Better explain the relationship between contexts, sync configs,
and source configs ("a sync config can use the datastore configs in
the same context").
An entire section on config properties in the terminology
section. "item" added (Todd Wilson correctly pointed out that it was
missing).
Less focus on conflict resolution, as suggested by Graham Cobb.
Fix examples that became invalid when fixing the password
storage/lookup mechanism for GNOME keyring in 1.4.
The "command line conventions", "Synchronization beyond SyncML" and
"CalDAV and CardDAV" sections were updated. It's possible that the
other sections also contain slightly incorrect usage of the
terminology or are simply out-dated.
* local sync: allow config name in syncURL=local://
Previously, only syncURL=local://@<context name> was allowed and used
the "target-config@context name" config as target side in the local
sync.
Now "local://config-name@context-name" or simply "local://config-name"
are also allowed. "target-config" is still the fallback if only a
context is give.
It also has one more special meaning: "--configure
target-config@google" will pick the "Google" template automatically
because it knows that the intention is to configure the target side
of a local sync. It does not know that when using some other name
for the config, in which case the template (if needed) must be
specified explicitly.
The process name in output from the target side now also includes the
configuration name if it is not the default "target-config".
* command line: revise usability checking of datastores
When configuring a new sync config, the command line checks whether a
datastore is usable before enabling it. If no datastores were listed
explicitly, only the usable ones get enabled. If unusable datastores
were explicitly listed, the entire configure operation fails.
This check was based on listing databases, which turned out to be too
unspecific for the WebDAV backend: when "database" was set to some URL
which is good enough to list databases, but not a database URL itself,
the sources where configured with that bad URL.
Now a new SyncSource::isUsable() operation is used, which by default
just falls back to calling the existing Operations::m_isEmpty. In
practice, all sources either check their config in open() or the
m_isEmpty operation, so the source is usable if no error is
enountered.
For WebDAV, the usability check is skipped because it would require
contacting a remote server, which is both confusing (why does a local
configure operation need the server?) and could fail even for valid
configs (server temporarily down). The check was incomplete anyway
because listing databases gave a fixed help text response when no
credentials were given. For usability checking that should have
resulted in "not usable" and didn't.
The output during the check was confusing: it always said "listing
databases" without giving a reason why that was done. The intention
was to give some feedback while a potentially expensive operation
ran. Now the isUsable() method itself prints "checking usability" if
(and only if!) such a check is really done.
Sometimes datastores were checked even when they were about to be
configure as "disabled" already. Now checking such datastores is
skipped.
* command line: fix --update from directory
The "--update <dir name>" operation was supposed to take the
item luids from the file names inside the directory. That part
had not been implemented, turning the operation accidentally
into an "--import".
Also missing was the escaping/unescaping of luids. Now the
same escaping is done as in command line output and command
line parsing to make the luids safe for use as file name.
* sync output: hide "<source>: started" INFO messages
These messages get printed at the start of processing each
SyncML message. This is not particularly useful and just
adds noise to the output.
* config: allow storing credentials for email address
When configuring a WebDAV server with username = email address and no
URL (which is possible if the server supports service discovery via
the domain in the email address), then storing the credentials in the
GNOME keyring used to fail with "cannot store password in GNOME
keyring, not enough attributes".
That is because GNOME keyring seemed to get confused when a network
login has no server name and some extra safeguards were added to
SyncEvolution to avoid this.
To store the credentials in the case above, the email address now gets
split into user and domain part and together get used to look up the
password.
* config: ignore unnecessary username property
A local sync or Bluetooth sync do not need the 'username' property.
When it is set despite that, issue a warning.
Previously, the value was checked even when not needed, which
caused such syncs to fail when set to something other than a plain
username.
* config templates: Funambol URLs
Funambol turned of the URL redirect from my.funambol.com to
onemedia.com. The Funambol template now uses the current URL. Users
with existing Funambol configs must updated the syncURL property
manually to https://onemediahub.com/sync
Kudos to Daniel Clement for reporting the change.
* EDS: memo syncing as iCalendar 2.0 (FDO #52714)
When syncing memos with a peer which also supports iCalendar 2.0 as
data format, the engine will now pick iCalendar 2.0 instead of
converting to/from plain text. The advantage is that some additional
properties like start date and categories can also be synchronized.
The code is a lot simpler, too, because the EDS specific iCalendar 2.0
<-> text conversion code can be removed.
* SoupTransport: drop CA file check
It used to be necessary to specify a CA file for libsoup to enable SSL
certificate checking. Nowadays libsoup uses the default CA store
unless told otherwise, so the check in SyncEvolution became
obsolete. However, now there is a certain risk that no SSL checking is
done although the user asked for it (when libsoup is not recent enough
or compiled correctly).
* CardDAV: use Apple/Google/CardDAV vCard flavor
In principle, CardDAV servers support arbitrary vCard 3.0
data. Extensions can be different and need to be preserved. However,
when multiple different clients or the server's Web UI interpret the
vCards, they need to agree on the semantic of these vCard extensions.
In practice, CardDAV was pushed by Apple and Apple clients are
probably the most common clients of CardDAV services. When the Google
Contacts Web UI creates or edits a contact, Google CardDAV will
send that data using the vCard flavor used by Apple.
Therefore it makes sense to exchange contacts with *all* CardDAV
servers using that format. This format could be made configurable in
SyncEvolution on a case-by-case basis; at the moment, it is
hard-coded.
During syncing, SyncEvolution takes care to translate between the
vCard flavor used internally (based on Evolution) and the CardDAV
vCard flavor. This mapping includes:
X-AIM/JABBER/... <-> IMPP + X-SERVICE-TYPE
Any IMPP property declared as X-SERVICE-TYPE=AIM will get
mapped to X-AIM. Same for others. Some IMPP service types
have no known X- property extension; they are stored in
EDS as IMPP. X- property extensions without a known X-SERVICE-TYPE
(for example, GaduGadu and Groupwise) are stored with
X-SERVICE-TYPE values chosen by SyncEvolution so that
Google CardDAV preserves them (GroupWise with mixed case
got translated by Google into Groupwise, so the latter is used).
Google always sends an X-ABLabel:Other for IMPP. This is ignored
because the service type overrides it.
The value itself also gets transformed during the mapping. IMPP uses
an URI as value, with a chat protocol (like "aim" or "xmpp") and
some protocol specific identifier. For each X- extension the
protocol is determined by the property name and the value is the
protocol specific identifier without URL encoding.
X-SPOUSE/MANAGER/ASSISTANT <-> X-ABRELATEDNAMES + X-ABLabel
The mapping is based on the X-ABLabel property attached to
the X-ABRELATEDNAMES property. This depends on the English
words "Spouse", "Manager", "Assistant" that Google CardDAV
and Apple devices seem to use regardless of the configured
language.
As with IMPP, only the subset of related names which have
a corresponding X- property extension get mapped. The rest
is stored in EDS using the X-ABRELATEDNAMES property.
X-ANNIVERSARY <-> X-ABDATE
Same here, with X-ABLabel:Anniversary as the special case
which gets mapped.
X-ABLabel parameter <-> property
CardDAV vCards have labels attached to arbitrary other properties
(TEL, ADR, X-ABDATE, X-ABRELATEDNAMES, ...) via vCard group tags:
item1.X-ABDATE:2010-01-01
item1.X-ABLabel:Anniversary
The advantage is that property values can contain arbitrary
characters, including line breaks and double quotation marks,
which is not possible in property parameters.
Neither EDS nor KDE (judging from the lack of responses on the
KDE-PIM mailing list) support custom labels. SyncEvolution could
have used grouping as it is done in CardDAV, but grouping is not
used much (not at all?) by the UIs working with the vCards in EDS
and KDE. It seemed easier to use a new X-ABLabel parameter.
Characters which cannot be stored in a parameter get converted
(double space to single space, line break to space, etc.) during
syncing. In practice, these characters don't appear in X-ABLabel
properties anyway because neither Apple nor Google UIs allow entering
them for custom labels.
The "Other" label is used by Google even in case where it adds no
information. For example, all XMPP properties have an associated
X-ABLabel=Other although the Web UI does not provide a means to edit
or show such a label. Editing the text before the value in the UI
changes the X-SERVICE-TYPE parameter value, not the X-ABLabel as for
other fields.
Therefore the "Other" label is ignored by removing it during syncing.
X-EVOLUTION-UI-SLOT (the parameter used in Evolution to determine the
order of properties in the UI) gets stored in CardDAV. The only exception
is Google CardDAV which got confused when an IMPP property had both
X-SERVICE-TYPE and X-EVOLUTION-UI-SLOT parameters set. For Google,
X-EVOLUTION-UI-SLOT is only sent on other properties and thus ordering
of chat information can get lost when syncing with Google.
* synccompare: support grouping and quoted parameter strings
Grouped properties are sorted first according to the actual property
name, then related properties are moved to the place where their group
tag appears first. The first grouped property gets a "- " prefix, all
following ones are just indended with " ". The actual group tag is not
part of the normalized output, because its value is irrelevant:
BDAY:19701230
- EMAIL:john@custom.com
X-ABLabel:custom-label2
...
FN:Mr. John 1 Doe Sr.
- IMPP;X-SERVICE-TYPE=AIM:aim:aim
X-ABLabel:Other
...
- X-ABDATE:19710101
X-ABLabel:Anniversary
Redundant tags (those set for only a single property, X-ABLabel:Other)
get removed as part of normalizing an item.
* WebDAV: use server's order when listing collections
When doing a recursive scan of the home set, preserve the order of
entries as reported by the server and check the first one first. The
server knows better which entries are more relevant for the user (and
thus should be the default) or may have some other relevant
order. Previously, SyncEvolution replaced that order with sorting by
URL, which led to a predictable, but rather meaningless order.
For example, Google lists the users own calendar first, followed by
the shared calendars sorted alphabetical by their name. Now
SyncEvolution picks the main calendar as default correctly when
scanning from https://www.google.com/calendar/dav/.
* WebDAV: improved database search (Google, Zimbra)
Zimbra has a principal URL that also serves as home set. When using it
as start URL, SyncEvolution only looked the URL once, without listing
its content, and thus did not find the databases.
When following the Zimbra principal URL indirectly, SyncEvolution did
check all of the collections there recursively. Unfortunately that
also includes many mail folders, causing the scan to abort after
checking 1000 collections (an internal safe guard).
The solution for both includes tracking what to do with a URL. For the
initial URL, only meta data about the URL itself gets
checked. Recursive scanning is only done for the home set. If that
home set contains many collections, scanning is still slow and may run
into the internal safe guard limit. This cannot be avoided because the
CalDAV spec explicitly states that the home set may contain normal
collections which contain other collections, so a client has to do the
recursive scan.
When looking at a specific calendar, Google CalDAV does not report
what the current principal or the home set is and therefore
SyncEvolution stopped after finding just the initial calendar. Now it
detects the lack of meta information and adds all parents also as
candidates that need to be looked at. The downside of this is that it
doesn't know anything about which parents are relevant, so it ends up
checking https://www.google.com/calendar/ and
https://www.google.com/.
In both cases Basic Auth gets rejected with a temporary redirect to
the Google login page, which is something that SyncEvolution must
ignore immediately during scanning without applying the resend
workaround for "temporary rejection of valid credentials" that can
happen for valid Google CalDAV URLs.
* WebDAV: enhanced database search (Google Calendar)
Additional databases where not found for several
reasons. SyncEvolution ignored all shared calendars
(http://calendarserver.org/ns/shared) and Google marks the additional
calendars that way. The other problem was that the check for leaf
collections (= collections which cannot contain other desired
collections) incorrectly excluded those collections instead of only
preventing listing of their content.
With this change,
https://www.google.com/calendar/dav/?SyncEvolution=Google can be used
as starting point for Google Calendar.
* WebDAV: fix database scan on iCloud
The calendar home set URL on iCloud (the one ending in /calendars/) is
declared as containing calendar data. That was enough for
SyncEvolution to accept it incorrectly as calendar. However, the home
set only contains calendar data indirectly.
* WebDAV: support redirects between hosts and DNS SRV lookup based on URL
When finding a new URL, we must be prepared to reinitialize the Neon
session with the new host settings.
iCloud does not have .well-known support on its www.icloud.com
server. To support lookup with a non-icloudd.com email address, we
must do DNS SRV lookup when access to .well-known URLs fails. We do
this without a www prefix on the host first, because that is what happens
to work for icloud.com.
With these changes it becomes possible to do database scans on Apple
iCloud, using syncURL=https://www.icloud.com or
syncURL=https://icloud.com. Giving the syncURL like this is only
necessary for a username that does not end in @icloud.com. When
the syncURL is not set, the domain for DNS SRV lookup is taken
from the username.
* WebDAV: more efficient item creation
PUT has the disadvantage that a client needs to choose a name and then
figure out what the real name on the server is. With Google CardDAV that
requires sending another request and only works because the server happens
to remember the original name (which is not guaranteed!).
POST works for new items without a name and happens to be implemented
by Google such that the response already includes all required
information (new name and revision string).
POST is checked for as described in RFC 5995 once before creating a new
item. Servers which don't support it continue to get a PUT.
* WebDAV: send "User-Agent: SyncEvolution"
Apple iCloud servers reject requests unless they contain a User-Agent
header. The exact value doesn't seem to matter. Making the string
configurable might be better, but can still be done later when it
is more certain whether and for what it is needed.
* WebDAV: refactor and fix DNS SRV lookup
The syncevo-webdav-lookup script was not packaged. It did not report
"not found" DNS results correctly and the caller did not check for
this either, so when looking up the information for a domain which
does not have DNS SRV entries, SyncEvolution ended up retrying for
while as if there had been a temporary lookup problem.
* Google: remove SyncML template, combine CalDAV/CardDAV
Google has turned off their SyncML server, so the corresponding
"Google Contacts" template became useless and needs to be removed. It
gets replaced by a "Google" template which combines the three
different URLs currently used by Google for CalDAV/CardDAV.
This new template can be used to configure a "target-config@google"
with default calendar and address book database already enabled. The
actual URL of these databases will be determined during the first
sync using them.
The template relies on the WebDAV backend's new capability to search
multiple different entries in the syncURL property for databases. To
avoid listing each calendar twice (once for the legacy URL, once with
the new one) when using basic username/password authentication, the
backend needs a special case for Google and detect that the legacy URL
does not need to be checked.
* Google Calendar: remove child hack, improve alarm hack (FDO #63881)
Google recently enhanced support for RECURRENCE-ID, so SyncEvolution
no longer needs to replace the property when uploading a single
detached event with RECURRENCE-ID. However, several things are still
broken in the server, with no workaround in SyncEvolution:
- Removing individual events gets ignored by the server;
a full "wipe out server data" might work (untested).
- When updating the parent event, all child events also get
updated even though they were included unchanged in the data
sent by SyncEvolution.
- The RECURRENCE-ID of a child event of an all-day recurring event
does not get stored properly.
- The update hack seems to fail for complex meetings: uploading them
once and then deleting them seems to make uploading them again
impossible.
All of these issues were reported to Google and are worked on there,
so perhaps the situation will improve. In the meantime, syncing with
Google CalDAV should better be limited to:
- Downloading a Google calendar in one-way mode.
- Two-way syncing of simple calendars without complex meeting
serieses.
While updating the Google workarounds, the alarm hack (sending a
new event without alarms twice to avoid the automatic server side
alarm) was simplified. Now the new event gets sent only once with a
pseudo-alarm.
* CardDAV: implement read-ahead
Instead of downloading contacts one-by-one with GET, SyncEvolution now
looks at contacts that are most likely going to be needed soon and
gets all of them at once with addressbook-multiget REPORT.
The number of contacts per REPORT is 50 by default, configurable by
setting the SYNCEVOLUTION_CARDDAV_BATCH_SIZE env variable.
This has two advantages:
- It avoids round-trips to the server and thus speeds up a large
download (100 small contacts with individual GETs took 28s on
a fast connection, 3s with two REPORTs).
- It reduces the overall number of requests. Google CardDAV is known
to start issuing intermittent 401 authentication errors when the
number of contacts retrieved via GET is too large. Perhaps this
can be avoided with addressbook-multiget.
* oauth2: new backend using libsoup/libcurl
New backend implements identity provider for obtaining OAuth2 access
token for systems without HMI support.
Access token is obtained by making direct HTTP request to OAuth2 server
and using refresh token obtained by user in some other way.
New provider automatically updates stored refresh token when OAuth2
server is issuing new one.
* signon: make Accounts optional
The new "signon" provider only depends on lib[g]signon-glib. It uses
gSSO if found, else UOA. Instead of pulling parameters and the
identity via libaccounts-glib, the user of SyncEvolution now has to
ensure that the identity exists and pass all relevant parameters
in the "signon:" username.
* gSSO: adapt to gSSO >= 2.0
* signon: fix build
Static build was broken for gSSO and UOA (wrong path name to .la file)
and gSSO was not enabled properly (wrong condition check).
* datatypes: raw text items with minimal conversion (FDO #52791)
When using "raw/text/calendar" or "raw/text/vcard" as SyncEvolution
"databaseFormat", all parsing and conversion is skipped. The backend's
data is identical to the item data in the engine. Finding duplicates
in a slow sync is very limited when using these types because the entire
item data must match exactly.
This is useful for the file backend when the goal is to store an exact copy
of what a peer has or for limited, read-only backends (PBAP). The downside
of using the raw types is that the peer is not given accurate information
about which vCard or iCalendar properties are supported, which may cause
some peers to not send all data.
* datatypes: text/calendar+plain revised heuristic
When sending a VEVENT, DESCRIPTION was set to the SUMMARY if empty. This may
have been necessary for some peers, but for notes (= VJOURNAL) we don't know
that (hasn't been used in the past) and don't want to alter the item
unnecessarily, so skip that part and allow empty DESCRIPTION.
When receiving a plain text note, the "text/calendar+plain" type
used to store the first line as summary and the rest as description.
This may be correct in some cases and wrong in others.
The EDS backend implemented a different heuristic: there the first
line is copied into the summary and stays in the description. This
makes a bit more sense (the description alone is always enough to
understand the note). Therefore and to avoid behavioral changes
for EDS users when switching the EDS backend to use text/calendar+plain,
the engine now uses the same approach.
* datatypes: avoid PHOTO corruption during merge (FDO #77065)
When handling an update/update conflict (both sides of the sync have an
updated contact) and photo data was moved into a local file by EDS, the engine
merged the file path and the photo data together and thus corrupted the photo.
The engine does not know about the special role of the photo property.
This needs to be handled by the merge script, and that script did not
cover this particular situation. Now the loosing side is cleared,
causing the engine to then copy the winning side over into the loosing
one.
Found by Renato Filho/Canonical when testing SyncEvolution for Ubuntu 14.04.
* vcard profile: avoid data loss during merging
When resolving a merge conflict, repeating properties were taken
wholesale from the winning side (for example, all email addresses). If
a new email address had been added on the loosing side, it got lost.
Arguably it is better to preserve as much data as possible during a
conflict. SyncEvolution now does that in a merge script by checking
which properties in the loosing side do not exist in the winning side
and copying those entries.
Typically only the main value (email address, phone number) is checked
and not the additional meta data (like the type). Otherwise minor
differences (for example, both sides have same email address, but with
different types) would lead to duplicates.
Only addresses are treated differently: for them all attributes
(street, country, city, etc.) are compared, because there is no single
main value.
* engine: UID support in contact data
Before, the UID property in a vCard was ignored by the engine.
Backends were responsible for ensuring that the property is
set if required by the underlying storage. This turned out to be
handled incompletely in the WebDAV backend.
This change moves this into the engine:
- UID is now field. It does not get used for matching
because the engine cannot rely on it being stored
by both sides.
- It gets parsed if present, but only generated if
explicitly enabled (because that is the traditional
behavior).
- It is never shown in the DevInf's CtCap
because the Synthesis engine would always show it
regardless whether a rule enabled the property.
That's because rules normally only get triggered
after exchanging DevInf and thus DevInf has to
be rule-independent. We don't want it shown because
then merging the incoming item during a local sync
would use the incoming UID, even if it is empty.
- Before writing, ensure that UID is set.
When updating an existing item, the Synthesis engine reads
the existing item, preserves the existing UID unless the peer
claims to support UID, and then updates with the existing UID.
This works for local sync (where SyncEvolution never claims
to support UID when talking to the other side). It will break
with peers which have UID in their CtCap although they
rewrite the UID and backends whose underlying storage cannot
handle UID changes during an update (for example, CardDAV).
* engine: flush map items less frequently
The Synthesis API does not say explicitly, but in practice all map
items get updated in a tight loop. Rewriting the m_mappingNode (case
insensitive string comparisons) and serialization to disk
(std::ostrstream) consume a significant amount of CPU cycles and cause
extra disk writes that can be avoided by making some assumptions about
the sequence of API calls and flushing only once.
* local sync: exchange SyncML messages via shared memory
Encoding/decoding of the uint8_t array in D-Bus took a surprisingly
large amount of CPU cycles relative to the rest of the SyncML message
processing. Now the actual data resides in memory-mapped temporary
files and the D-Bus messages only contain offset and size inside these
files. Both sides use memory mapping to read and write directly.
For caching 1000 contacts with photos on a fast laptop, total sync
time roughly drops from 6s to 3s.
To eliminate memory copies, memory handling in libsynthesis or rather,
libsmltk is tweaked such that it allocates the buffer used for SyncML
message data in the shared memory buffer directly. This relies on
knowledge of libsmltk internals, but those shouldn't change and if they
do, SyncEvolution will notice ("unexpected send buffer").
* local sync: avoid updating meta data when nothing changed
The sync meta data (sync anchors, client change log) get updated after
a sync even if nothing changed and the existing meta data could be
used again. This can be skipped for local sync, because then
SyncEvolution can ensure that both sides skip updating the meta
data. With a remote SyncML server that is not possible and thus
SyncEvolution has to update its data.
It is based on the observation that when the server side calls
SaveAdminData, the client has sent its last message and the sync is
complete. At that point, SyncEvolution can check whether anything has
changed and if not, skip saving the server's admin data and stop the
sync without sending the real reply to the client.
Instead the client gets an empty message with "quitsync" as content
type. Then it takes shortcuts to close down without finalizing the
sync engine, because that would trigger writing of meta data
changes. The server continues its shutdown normally.