For performance replay, we need to ensure that replaying recordings will
replay changes to the view and main window size, if they are present in the
recording. [Replay of these changes can be disabled in the "File/Flight
Recorder Control" dialogue.]
scripts/python/performance_replay.py
New, script for replaying a recording in Flightgear and gathering
statistics.
scripts/python/recordreplay.py
Minor change.
src/Aircraft/replay.cxx
src/Aircraft/replay.hxx
If /sim/replay/log-frame-times is true, populate
/sim/replay/log-frame-times/dt[] with frame times while replaying.
Set telnet_hz to 100 to reduce delay when making a recording - this avoids
problem where telnet round-trip time could result in recording being 1-2s
longer than we expected, leading to test failures.
New properies so recordreplay can know extent of current in-memory normal
recording:
sim/replay/record-normal-begin
sim/replay/record-normal-end
This is required since scripts/python/recordreplay.py improved checking of
recording length when replaying.
scripts/python/recordreplay.py
src/Aircraft/replay.cxx
src/Aircraft/replay.hxx
src/Aircraft/replay.*:
If /sim/replay/record-continuous-compression is true, we compress each
frame's data as a separate raw zlib stream.
Requires latest simgear's simgear/io/iostreams/zlibstream.cxx for
decompression with ZLibCompressionFormat::ZLIB_RAW. Haven't figured out how
to extend simgear's code to provide a compressing ostream so for now we
have our own local compression code.
We open popup and set sim/replay/replay-error=true if we fail to read
compressed data.
scripts/python/recordreplay.py:
Added test of compressed continuous recordings.
docs-mini/README-recordings.md:
Added information about compressed format.
Also fixed uneven MP carrier motion.
If /ai/models/carrier[]/ai-latch is true (e.g. by MPCarrier.nas), we
set /position/* and /orientation/* in C++ every frame instead of in
nasal. For external multiplayer carriers the values are copied from
/ai/models/multiplayer[]/*, so the AI carrier follows the corresponding MP
carrier exactly. For this to be useful, multiplayer motion needs be smooth,
e.g. with /sim/time/simple-time/enabled=true.
scripts/python/recordreplay.py
Added --carrier test - checks that multiplayer carrier moves with even
speed.
src/AIModel/AIBase.hxx
Added speed_fps, to be kept up to date and tied to velocities/uBody-fps
when we are a carrier. Previously this was set from Nasal which doesn't
do what is required when all updates happen in C++.
src/AIModel/AICarrier.cxx
src/AIModel/AICarrier.hxx
If is-user-craft is true, we directly update /position/* every frame.
If ai-latch is true, we don't call TurnToLaunch(), TurnToRecover(),
ReturnToBox(), TurnToBase() etc, because our position and orientation is
determined only by multiplayer packets.
src/AIModel/AIMultiplayer.cxx
src/AIModel/AIMultiplayer.hxx
If an mp craft is a carrier, MP packets define velocities/speed-kts but
set ecLinearVel to all-zeros. So we now copy across to ensure that the
extrapolation algorithm has a velocity to work with. Previously the zero
velocity caused very uneven motion.
If ai-latch is true we set AI craft's position+velocities directly from
the equivalent (extrapolated or interpolated) MP position, every frame. We
also set AI craft's orientation. And we copy MP's uBody-fps to AI's
velocities/speed-kts which ensures that friction works between carrier deck
and aircraft undercarriage.
Added logging of raw speeds implied by multiplayer packets, activated by
/sim/log-multiplayer-callsign; used by scripts/python/recordreplay.py's
--carrier test.
src/AIModel/AIShip.cxx
Tie velocities/uBody-fps to new speed_fps member and set speed_fps in
update() along with members that are tied to properties.
Replaced code that calculated new position using heading and speed:
Previously the new position after dt was calculated using
ft_per_deg_lat, ft_per_deg_lon, speed_north_deg_sec and
speed_east_deg_sec. But this was moving slightly faster than the
specified speed.
This was leading to incremental errors when a different Flightgear
instance extrapolated the multiplayer position from the information in
multiplayer packets, because the specified velocity was too small, so
we jumped forwards when extrapolation moved to a new packet.
The fix is to use a Quaternion-based calculation to calculate movement
in the direction specified by (heading, pitch, roll), as done by other
code such as the view code.
src/Main/fg_init.cxx
Moved FGAIManager to just before FGMultiplayMgr so we send latest info in
mp packets.
src/MultiPlayer/multiplaymgr.cxx
Fixed minimum transmit rate calculation - if transmit rate is less than 1,
default to 1, not 10.
src/Network/props.cxx
Use more precision when sending double-precision values e.g. to telnet
client. Otherwise for example UTC times don't have sufficient resolution.
src/Viewer/viewmgr.cxx
Generate internal logs of multiplayer position and speed (after
interpolation/extrapolation) if /sim/log-multiplayer-callsign is set. Used
by scripts/python/recordreplay.py's --carrier test.
FGReplay was using time t, AIMultiplayer was using t+dt. The fix is to make
FGReplay also use t+dt.
scripts/python/recordreplay.py
Make --test-motion-mp test for this problem - we check that user and mp
aircraft are a constant distance apart when replaying.
src/AIModel/AIMultiplayer.cxx
Create properties showing distance moved since start by user and mp
aircraft, which can then be tested by recordreplay.py.
src/Aircraft/replay.cxx
Increment current_time by dt before calling replay(current_time), to
ensure that replay() sees the time as is later used by AIMultiplayer (via
/sim/replay/time).
test_motion() now sets /sim/time/simple-time/enabled=true.
Also show any description items - these are generate by related changes to
src/AIModel/AIMultiplayer.cxx.
This is in addition to mp speed in mp packets. Looks like this final mp speed
is very uneven.
Have changed the test slightly to move both UFOs at the same speed so replaying
e.g. with Helicopter View shows them incorrectly moving relative to each other.
Setting sim/replay/log-raw-speed logs raw speed to
/sim/replay/log-raw-speed-values/value[].
'scripts/python/recordreplay.py --test-motion' uses this to test for a bug
where the user aircraft simtime and signals information don't quite match in
continuous recordings.
Recording of extra properties is only supported in Continuous recordings.
Modified Continuous recording format to allow future forwards
compatibility. See docs-mini/README-recordings.md for details. This breaks
compatibility with previously-generated Continuous recordings, but this only
affects next.
To reduce overhead we record all extra property values in the first frame and
then later frames contain only extra property changes. When replaying, if the
user jumps backwards we replay all extra property changes since the start of
the recording. Similarly if the user jumps forwards, we replay any intervening
extra property changes.
Recording extra properties:
This is enabled by:
/sim/replay/record-extra-properties
The extra properties that are recorded are identified by the property
paths in the values of /sim/replay/record-extra-properties-paths/path[]
properties. We record the entire tree for each specified path.
Recording of main window position size:
We have specific support for record and replay of main window position/size.
This is enabled by:
/sim/replay/record-main-window
Recording of main window view:
We have specific support for recording the view type and direction/zoom
settings.
This is enabled by:
/sim/replay/record-main-view
We record the /sim/current-view/ property tree, excluding some subtrees
that continuously vary but are not required for replaying of the view.
When replaying, we allow separate control of what extra property changes are
replayed, with:
/sim/replay/replay-extra-property-changes
/sim/replay/replay-extra-property-removal
/sim/replay/replay-main-window-position
/sim/replay/replay-main-window-size
/sim/replay/replay-main-view
We work around some problems caused by the use of tied properties when
replaying changes to view-number.
Window position issue:
When replaying window position and size changes, things get a little tricky
because osgViewer::GraphicsWindow::setWindowRectangle() takes a position
for the entire window, but osgGA::GUIEventAdapter::RESIZE events contain
the position of the interior of the window; for example the y values will
differ by the height of the window's titlebar. This can cause windows to
move progressively further down each time they are positioned or resized.
There doesn't seem to be a way of finding the size of a window's
furniture directly. So instead this commit adds a new method
osgGA::GUIEventAdapter::setWindowRectangleInteriorWithCorrection() which
wraps osgViewer::GraphicsWindow::setWindowRectangle(). We listen for the
following osgGA::GUIEventAdapter::RESIZE event and calculate corrections
for x and y position that are used for subsequent calls.
docs-mini/README-recordings.md:
Updated to document new Continuous format.
scripts/python/recordreplay.py:
New script to test various aspects of record/replay.
Other:
We now create convenience softlink to most recent continuous recording, using
SGPath::makeLink(). Note that SGPath::makeLink() currently does nothing on
Windows.
Changed format of Continuous recordings to contain a single property tree
in header. This is much simpler than having separate Config and Meta trees.
- Instantiate an HTTPSConnection object when the URL scheme is 'https'.
- Clarify and simplify the initialization of HTTPSocketRequest and
HTTPDownloadRequest:
+ clarify initialization of their 'callback' attribute (it's the
method of the same name; make it clear that the base class
constructor, namely HTTPGetCallback.__init__(), doesn't modify the
'callback' attribute when an object of class HTTPSocketRequest or
HTTPDownloadRequest is initialized);
+ HTTPDownloadRequest doesn't need access to the TerraSync object
-> remove the corresponding instance attribute and constructor
argument.
- Don't use super() when initializing HTTPDownloadRequest objects
(see [1]).
[1] https://fuhm.net/super-harmful/
- Refuse to recursively delete a directory that does not contain a
.dirindex file. This will protect users against data loss in case they
inadvertently use the --remove-orphan option with the wrong target
directory.
- Correctly handle the case where we have a file on disk that is now
listed as a directory on the server: remove the file if we are in
'sync' mode, so that the directory can be created and sync'ed from the
server.
- only accept ASCII-encoded .dirindex files (this is guaranteed to work
fine "everywhere");
- reject .dirindex files with a 'path' entry that contains a backslash
or starts with a slash;
- reject .dirindex files with a 'path' entry that contains a '..'
component;
- reject .dirindex files with an 'f', 'd' or 't' entry whose name field
contains a slash or a backslash;
- reject .dirindex files with an 'f', 'd' or 't' entry whose name field
is '..';
- add comment lines (starting with '#') in the sample good .dirindex
file used by unit tests.
Remove the 'if __name__ == "__main__": unittest.main()'. Indeed, the
module can't be run this way due to its imports. Tests from this module
can be run with:
cd scripts/python/TerraSync
python3 -m unittest tests.test_virtual_path
In Python, common usage is not to define accessors, but to directly use
class or instance attributes (especially when the associated data is
constant after instance creation). If it later happens that a given
attribute needs getter or setter logic, this can always be done via the
@property decorator, and doesn't affect calling code at all. See for
instance:
https://docs.python.org/3/library/functions.html#propertyhttps://mail.python.org/pipermail/tutor/2012-December/thread.html#92990
Apply this to the DirIndex class and rename the following attributes for
better readability: f -> files, d -> directories, t -> tarballs.
The tests can be run from directory 'scripts/python/TerraSync' using:
python3 -m unittest tests.test_dirindex
(or just 'python3 -m unittest' to run all tests pertaining to
terrasync.py).
python3 has a default implementation for __ne__ when __eq__ is defined. The opposite is not true -- having only __ne__ does not have a default __eq__ implementation.
Also note that there are cases when eq/ne will both be True or both False, therefore, developers are encouraged to explicitly define these methods in pairs.
During the recent SourceForge migration, the TerraSync server hosted
there used to send .dirindex files with the following contents:
Project web is currently offline pending the final migration of its
data to our new datacenter.
Make sure terrasync.py aborts with a user-understandable error in such a
case.
Add more functions and properties to VirtualPath that directly
correspond to functions and properties of pathlib.PurePath, except that
types are adapted of course, and that for API consistency, VirtualPath
methods use mixedCaseStyle whereas those of pathlib.PurePath use
underscore_style.
Due to some misleading 'url' variable name, network error messages used
to contain things such as '/scenery/Airports/N/E/4/.dirindex' (i.e., the
path on the server) instead of the full URL. For the same reason, the
callback function of HTTPGetCallback objects was passed this
path-on-server instead of the URL. This should all be fixed now.
Option --only-subdir allows one to restrict terrasync.py processing[1]
to a chosen subdirectory of the TerraSync repository. Example:
terrasync.py --target=/your/TerraSync/repo --only-subdir="Airports/L/F/P"
[1] This works in both 'check' and 'sync' modes.
Add classes VirtualPath and MutableVirtualPath (the latter derived from
the former) to manipulate slash-separated paths where the root '/'
represents the TerraScenery root. This makes it clear what a function
expects when you see that one of its arguments is a VirtualPath
instance: you don't have to ask yourself whether it can start or end
with a slash, how to interpret it, etc. Operating on these paths is also
easy[1], be it to assemble URLs in order to retrieve files or to join
their relative part with a local directory path in order to obtain a
real (deeper) local path.
VirtualPath and MutableVirtualPath are essentially the same; the former
is hashable and therefore has to be immutable, whereas the latter can be
modified in-place with the /= operator (used to append path components),
and therefore can't be hashable. As a consequence, MutableVirtualPath
instances can't be used as dictionary keys, elements of a set or
frozenset, etc.
VirtualPath and MutableVirtualPath use the pathlib.PurePath API where
applicable (part of this API has been implemented in
[Mutable]VirtualPath; more can be added, of course). These classes have
no assumptions related to TerraSync and thus should be fit for use in
other projects.
To convert a [Mutable]VirtualPath instance to a string, just use str()
on it. The result is guaranteed to start with a '/' and not to end with
a '/', except for the virtual root '/'. Upon construction, the given
string is interpreted relatively to the virtual root, i.e.:
VirtualPath("") == VirtualPath("/")
VirtualPath("abc/def/ghi") == VirtualPath("/abc/def/ghi")
etc.
VirtualPath and MutableVirtualPath instances sort like the respective
strings str() converts them too. The __hash__() method of VirtualPath is
based on the type and this string representation, too. Such objects can
only compare equal (using ==) if they have the same type. If you want to
compare the underlying virtual paths inside a VirtualPath and a
MutableVirtualPath, use the samePath() method of either class.
For more info, see scripts/python/TerraSync/terrasync/virtual_path.py
and unit tests in scripts/python/TerraSync/tests/test_virtual_path.py.
[1] Most useful is the / operator, which works as for SGPath:
VirtualPath("/abc/def/ghi") == VirtualPath("/abc") / "def" / "ghi"
VirtualPath("/abc/def/ghi") == VirtualPath("/abc") / "def/ghi"
