1 The upload process converts count files into reports, and
2 uploads reports. There will be only one report, named YYYY-MM-DD.json,
3 for a given day.
4
5 First phase. Look at the localdir (os.UserConfigdir()/go/telemetry/local)
6 and find all .count and .json files. Find the count files that are no
7 longer active by looking at their metadata.
8
9 Second phase. Group the inactive count files by their expiry date, and
10 for each date generate the local report and the upload report. (The upload
11 report only contains the counters in the upload configuration.) The upload
12 report is saved in the local directory with a name like YYYY-MM-DD.json, if
13 there is no file already existing with that name.
14 If the local report is different, it is saved in the local directory
15 with a name like local.YYYY-MM-DD.json. The new upload report is
16 added to the list of .json files from the first phase. At this point
17 the count files are no longer needed and can be deleted.
18
19 Third phase. Look at the .json files in the list from the first phase.
20 If the name starts with local, skip it. If there is a file with the
21 identical name in the upload directory, remove the one in the local directory.
22 Otherwise try to upload the one in the local directory,
23 If the upload succeeds, move the file to the uploaded directory.
24
25
26 There are various error conditions.
27 1. Several processes could look at localdir and see work to do.
28 1A. They could see different sets of expired count files for some day.
29 This could happen if another process is removing count files. In this
30 case there is already a YYYY-MM-DD.json file either in localdir
31 or updatedir, so the process seeing fewer count files will not generate
32 a report.
33 1B. They could see the same count files, and no report in either directory.
34 They will both generate (in memory) reports and check to see if there
35 is a YYYY-MM-DD.json file in either directory. They could both then
36 write two files with the same name, but different X values, but
37 otherwise the same contents. The X values are very close to the front
38 of the file. Assuming reasonable file system semantics one version of
39 the file will be written. To minimize this, just before writing reports
40 the code checks again to see if they exist.
41 1C. Once there is an existing well-formed file YYYY-MM-DD.json in localdir
42 eventually the upload will succeed, and the file will be moved to updatedir.
43 It is possible that other processes will not see the file in updatedir and
44 upload it again and also move it to uploaddir. This is harmless as all
45 the uploaded files are identical.
46
View as plain text