Preparation for work on the synchronisation and lag prediction filters that jano has underway - firstly by adding a property to indicate the mode of the clock being used. Pre 2018.1 will be mode 0.
Automatic selection in the launcher is disabled for now, since
it needs more testing before release, but the basic UI for selection
is straightforward enough to throw in.
The add-on framework now uses the following files in each add-on
directory:
- addon-config.xml (previously: config.xml)
- addon-main.nas (previously: main.nas)
This is consistent with the addon-metadata.xml file that is already part
of the interface between FG core and add-ons. The goal is to make it
clearer, when browsing an add-on directory, which files belong to the
"FG core <-> add-on" interface and which files belong to the add-on
proper. This will be beneficial also when more files are added to the
"FG core <-> add-on" interface, such as possibly addon-events.xml in the
future.
This change is incompatible, thus it is the right time to do *before*
2018.2.1 is out, especially considering that this upcoming release
already has incompatible changes in the add-on API, namely the
requirement of the addon-metadata.xml file and the type of the argument
passed to each add-on's main() function. We'll try harder not to break
compatibility in the add-on API once 2018.2.1 is out. For now, it is
still a good time to try to get the API as clean as possible.
Due to some misleading 'url' variable name, network error messages used
to contain things such as '/scenery/Airports/N/E/4/.dirindex' (i.e., the
path on the server) instead of the full URL. For the same reason, the
callback function of HTTPGetCallback objects was passed this
path-on-server instead of the URL. This should all be fixed now.
Option --only-subdir allows one to restrict terrasync.py processing[1]
to a chosen subdirectory of the TerraSync repository. Example:
terrasync.py --target=/your/TerraSync/repo --only-subdir="Airports/L/F/P"
[1] This works in both 'check' and 'sync' modes.
Add classes VirtualPath and MutableVirtualPath (the latter derived from
the former) to manipulate slash-separated paths where the root '/'
represents the TerraScenery root. This makes it clear what a function
expects when you see that one of its arguments is a VirtualPath
instance: you don't have to ask yourself whether it can start or end
with a slash, how to interpret it, etc. Operating on these paths is also
easy[1], be it to assemble URLs in order to retrieve files or to join
their relative part with a local directory path in order to obtain a
real (deeper) local path.
VirtualPath and MutableVirtualPath are essentially the same; the former
is hashable and therefore has to be immutable, whereas the latter can be
modified in-place with the /= operator (used to append path components),
and therefore can't be hashable. As a consequence, MutableVirtualPath
instances can't be used as dictionary keys, elements of a set or
frozenset, etc.
VirtualPath and MutableVirtualPath use the pathlib.PurePath API where
applicable (part of this API has been implemented in
[Mutable]VirtualPath; more can be added, of course). These classes have
no assumptions related to TerraSync and thus should be fit for use in
other projects.
To convert a [Mutable]VirtualPath instance to a string, just use str()
on it. The result is guaranteed to start with a '/' and not to end with
a '/', except for the virtual root '/'. Upon construction, the given
string is interpreted relatively to the virtual root, i.e.:
VirtualPath("") == VirtualPath("/")
VirtualPath("abc/def/ghi") == VirtualPath("/abc/def/ghi")
etc.
VirtualPath and MutableVirtualPath instances sort like the respective
strings str() converts them too. The __hash__() method of VirtualPath is
based on the type and this string representation, too. Such objects can
only compare equal (using ==) if they have the same type. If you want to
compare the underlying virtual paths inside a VirtualPath and a
MutableVirtualPath, use the samePath() method of either class.
For more info, see scripts/python/TerraSync/terrasync/virtual_path.py
and unit tests in scripts/python/TerraSync/tests/test_virtual_path.py.
[1] Most useful is the / operator, which works as for SGPath:
VirtualPath("/abc/def/ghi") == VirtualPath("/abc") / "def" / "ghi"
VirtualPath("/abc/def/ghi") == VirtualPath("/abc") / "def/ghi"
- New directory scripts/python/TerraSync/terrasync.
- Move scripts/python/terrasync.py to
scripts/python/TerraSync/terrasync/main.py (main module in the new
structure).
- Add empty __init__.py file to scripts/python/TerraSync/terrasync/ to
make this directory a Python package.
- Wrap the main code from previous terrasync.py in a main() function of
the terrasync.main module. Also move command-line arguments parsing to
a separate parseCommandLine() function.
- Add an executable script scripts/python/TerraSync/terrasync.py for end
users, that just calls terrasync.main.main().
For end users, the only difference is that they now have to use
scripts/python/TerraSync/terrasync.py instead of
scripts/python/terrasync.py (which doesn't exist anymore, since all this
lives under the scripts/python/TerraSync directory from now on).
This structure will allow to cleanly split the code into modules and to
add unit tests.
Prevent FDM-derived properties being sent at full speed (120Hz) which
overloads telnet connections. Instead track dirty properties and
send them at the protocol’s update rate (which is presumably what
the user expects)
Using os.path.abspath() here in TerraSync.setTarget() adds a safety
layer in case the process later calls os.chdir() or similar[1], which
would change the meaning of the "." directory. Also remove the strip()
call which I don't consider useful here, see my message at:
https://sourceforge.net/p/flightgear/mailman/message/36208140/
[1] Not the case currently, but who knows what will happen in the
future...
You may now call terrasync.py with --mode=sync or --mode=check. 'sync'
mode is the default and corresponds to terrasync.py's usual behavior.
In 'check' mode, terrasync.py never writes to disk and aborts at the
first mismatch between local and remote data. The exit status in 'check'
mode is:
- 0 if the program terminated successfully and no mismatch was found
between the local and remote repositories;
- 1 in case an error was encountered;
- 2 if there was a mismatch between local and remote data.
In 'sync' mode, the exit status is:
- 0 if the program terminated successfully;
- 1 in case an error was encountered.
A mismatch in 'check' mode is *not* an error, it is just one of the two
expected results. An error is a worse condition (uncaught exception,
network retrieval aborted after retrying failed, stuff like that).
Additionally, calling terrasync.py with --report causes it to print
lists of:
- files and dirs that were missing or had mismatching hashes (this is
okay in 'sync' mode: these things have been "fixed" in the target
directory before the report was printed);
- files and dirs that have been found to be orphaned (i.e., found
under the target directory but not mentioned in the corresponding
.dirindex file). These are the ones removed in 'sync' mode when
--remove-orphan is passed.
- Add computeHash() utility function that can work with any file-like
object (e.g., a connected socket).
- Rename hash_of_file() to hashForFile(), and of course implement it
using our new computeHash().
- Add class HTTPSocketRequest derived from HTTPGetCallback. It allows
one to process data from the network without storing it to a file (it
uses the file-like interface provided by http.client.HTTPResponse).
The callback returns the http.client.HTTPResponse object, which can be
conveniently used in a 'with' statement.
- Simplify the API of TerraSync.updateDirectory(): its 'dirIndexHash'
argument must now be a hash (a string); the None object is not allowed
anymore (with the soon-to-come addition of --mode=check, having to
deal with this special case in updateDirectory() would make the logic
too difficult to follow, or we would have to really completely
separate check-only mode from update mode, which would entail code
duplication).
Since TerraSync.updateDirectory() must now always have a hash to work
with, compute the hash of the root '.dirindex' file from the server in
TerraSync.start(), using our new HTTPSocketRequest class---which was
written for this purpose, since that will have to work in check-only
mode (but not only), where we don't want to write any file to disk.
- TerraSync.updateFile(): correctly handle the case where a directory
inside the TerraSync repository is (now) a file according to the
server: the directory must be recursively removed before the file can
be downloaded in the place formerly occupied by the directory.
- Add stub class Report. Its methods do nothing for now, but are already
called in a couple of appropriate places. The class will be completed
in a future commit, of course.
The goal of removeDirectoryTree() is to provide a safety net around
recursive directory removal with shutil.rmtree(), in order to prevent
user or bug-caused catastrophic events such as /, /home /home/joeuser or
C:\ being recursively erased.
- Add method assembleUrl() to HTTPGetter.
- Raise a NetworkError exception with the particular URL and number of
retries when it has been exhausted.
- Number of retries is now trivial to expose as a parameter, and set to
5 in HTTPGetter.
- Sleep for one second between self.httpConnection.close() and
self.httpConnection.connect() when retrying a failed HTTP request.
- Apply DRY principle.
- New generic exception class TerraSyncPyException.
- Add subclass NetworkError of TerraSyncPyException.
- Raise a NetworkError exception when the HTTP return code is not 200.
- hash_of_file() does not silently ignore errors anymore; exceptions
should be dealt with wherever appropriate by the callers.
Whenever hash_of_file() returns, its return value is now the SHA-1
hash of the specified file. This is less error-prone IMHO than
returning None. Otherwise, calling code could erroneously conclude
that there is a matching hash when the file to check is actually
missing. For a concrete example, see the 'dirIndexHash' parameter of
TerraSync.updateDirectory(), which so far is used precisely with the
value None to express that "we are just starting the recursion and
have no hash from the server to compare to".
When called, the callback passed to HTTPGetter.get() is now explicitly
passed the URL and the http.client.HTTPResponse instance.
Remove the HTTPGetCallback.result attribute (not needed anymore, leaves
more freedom when implementating HTTPGetCallback subclasses...).
the current timestamp used in mp protocol and in AImultiplayer is not a good one:
it can pause, or even change speed if we change warp value.
we want it to be used for network protocol lag and jitter estimation, and
a time flowing linearly on both side is needed, here's a first introduction
of this timestamp relates to real elapsed time.
here it's initialised to the system clock, then follow the monotonic clock.
in future improvement, it will allow time synchronisation betwen mp players,
to have a very good close formation flight experience.
