Containers for families of geometric objects

[mkoeppe]

We define APIs for static and dynamic containers storing finite families of geometric objects, extending Container, Set, MutableSet, Mapping, MutableMapping https://docs.python.org/3/library/collections.abc.html

The family must define a collection of "measurement maps" B_i, each sending an object to an InternalRealInterval.

The product of these intervals can be thought of as a "generalized bounding box".

Ideally the maps would be inclusion-preserving.

The implementation is provided by rtree/libspatialindex (#32000).

The Sage-specific class will take care of:

• replacing an InternalRealInterval by a rescaling or inclusion-preserving overestimation, making the coordinates suitable for the underlying library.
• providing accelerated set operations such as:
  • a fast path for computing empty intersection when the result will be empty,
  • a fast path for __contains__ when the result will be False.

Geometric lookup operations to be supported:

• ...

Applications:

• RationalPolyhedralFan
• PolyhedralComplex (PolyhedralComplex #31748)
• Families/posets of (open) subsets of manifolds (Meta-ticket: Families, posets, complexes of manifold subsets #31740): A natural choice of the maps B_i comes from the coordinates of covering charts, using (-oo, oo) for coordinates for which no coordinate definition for the subset is known

Depends on #34277

CC: @DRKWang

Component: geometry

Work Issues: Rebase on #34277

Author: Binshuai Wang

Branch/Commit: public/32170 @ 05ea99d

Issue created by migration from https://trac.sagemath.org/ticket/32170

[DRKWang]

comment:1

The precision of interval values in Package libspatialindex is double.

They did not claim it explicitly, but I found it from

• https://rtree.readthedocs.io/en/latest/class.html#rtree.index.Index
• https://github.com/libspatialindex/libspatialindex/blob/master/include/spatialindex/capi/DataStream.h.

[mkoeppe]

comment:2

OK, the next step would be to find out whether the library does any arithmetic with these double values (in this case we would have to check whether they apply directed rounding correctly), or whether they only use comparisons on these double values.

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-We define APIs for static and dynamic containers storing finite families of geometric objects.
+We define APIs for static and dynamic containers storing finite families of geometric objects, extending `Container`, `Set`, `MutableSet`, `Mapping`, `MutableMapping` https://docs.python.org/3/library/collections.abc.html
 
 The family must define a collection of "measurement maps" `B_i`, each sending an object to an `InternalRealInterval`. 
 

[DRKWang]

comment:4

The library libspatialindex uses arithmetic operations, including +, - and *, with those double values in order to support some functions, such as

• LineSegment::Collinear( https://libspatialindex.org/en/latest/doxygen/LineSegment_8cc_source.html#l00439),

• LineSegment::getArea(https://github.com/libspatialindex/libspatialindex/blob/master/src/spatialindex/Region.cc)

and etc.

Besides, the library libspatialindex also uses std::numeric_limits<double>::epsilon() to deal with machine precision issue, such as Region::touchesPoint(const Point& p) (https://github.com/libspatialindex/libspatialindex/blob/master/src/spatialindex/Region.cc)

std::numeric_limits<double>::max();(https://github.com/libspatialindex/libspatialindex/blob/master/src/spatialindex/Point.cc).

[DRKWang]

comment:6

A few test codes have been made for testing arithmetic precision. The conclusions are as follows, more details and codes can be found from this(https://github.com/DRKWang/Data-structure-for-piecewise-linear-function/blob/main/log35.ipynb).

• The dimension of the index database should always be greater than 1. (0 or 1 dimension will fail).

• The arithmetic precision is 16 decimal digits, which comes from the double type's precision.

• The minimal capturing of arithmetic precision is 324 decimal digits, which means the minimal difference to distinguish two numbers is 1*10^{-324}. otherwise, they will be viewed as the same number.

• The intersection for retrieving is a closed intersection. (-0.000) is also considered as the same number as 0, but both representations will be displayed at the same time.

• Rounding. The round processing is a little complex and untraceable, it is neither rounding up or rounding down.

[DRKWang]

comment:7

I have made some debugs for this library. I would like to list the key points below. The entire log file can be found here https://github.com/DRKWang/Data-structure-for-piecewise-linear-function/blob/main/logsfile/log36.ipynb.

1. The python wrapper part for rtree was only used for passing the values and data to 2 basic functions, core.rt.Index_InsertData() and core.rt.Index_Intersects_obj(). Those two functions are stored here https://github.com/Toblerity/rtree/blob/master/rtree/core.py. Thus, there is no arithmetic error in this part.

2. The corresponding part in libspatialindex for core.rt.Index_InsertData() and core.rt.Index_Intersects_obj() are placed in line 499 and line 678 in https://github.com/libspatialindex/libspatialindex/blob/master/src/capi/sidx_api.cc .

3. In the Index_InsertData() in C++, the author uses the concepts "points" and "boxes" to refine the input information, and then call the function index().insertData(**) to compute. As we discussed last time, the author mistakenly uses the std::numeric_limits<double>::epsilon() as the range of approximation, that is, if the bound difference is within this range, then they will be seen as a single point instead of a box. According to this note, https://stackoverflow.com/questions/48133572/what-can-stdnumeric-limitsdoubleepsilon-be-used-for, we may correct it by multiplying the magnitudes of the comparison. However, because it is for scientific computation, we should get rid of approximation as much as possible.

4. Except for the above place, the author also mistakenly has used std::numeric_limits<double>::epsilon() in many other places. I think we could inform the author to check them deeply.

[DRKWang]

comment:8

• The extensions, .idx and .dat, are two files used to serialize the rtree to disk, according to https://rtree.readthedocs.io/en/latest/tutorial.html.

• Those two files are created by #include <fstream> internally, based on this source code, https://github.com/libspatialindex/libspatialindex/blob/master/src/storagemanager/DiskStorageManager.cc.

• The .idx is used to store vital information like the page size, the next available page, a list of empty pages, and the sequence of pages associated with every entity ID.There is no note for the detail or purpose of .dat file. Thus, I made a test where putting 50 boxes into this library. The shows that the .dat is way larger than .idx, which indicates the .dat is used to store the real data, whereas .idx is used for storage the rest of it.

• The correct way to close and reopen those serialized files can be found here, https://gis.stackexchange.com/questions/254781/saving-python-rtree-spatial-index-to-file

• Except for DiskStorageManager, it also has a MemoryStorageManager. The detail of it is that "Everything is stored in main memory using a simple vector. No properties are needed to initialize a MemoryStorageManager object. When a MemoryStorageManager instance goes out of scope, all data that it contains are lost."

• The .dat_extension = xxx and .idx_extension = xxx can be used to change the names of those extensions.

[DRKWang]

comment:9

I am trying to build a subclass called rtree_based_realset(), which is inherited from the class RealSet(). But got stuck on the purpose of this subclass. I would like to list my chain of thoughts below and hope to get a verification.

The intention of building this subclass is to keep the interface of this class, (I feel basically the interface can be considered as functions), the same and use rtree data structure to be a preliminary filter for overestimating in order to reduce the computation time. But since rtree only serves as an overestimate from one side, in order to be accurate, this subclass, which we want to define, still needs the finite unit intervals structure inside. In that case, seemingly, the subclass does not have a big difference from the RealSet() class. And the difference between those two classes only comes from whether we use rtree as an auxiliary tools or not.

[mkoeppe]

comment:10

Replying to @DRKWang:

since rtree only serves as an overestimate from one side, in order to be accurate, this subclass, which we want to define, still needs the finite unit intervals structure inside.

That's right, we are not replacing the internal representation of RealSet instances.

We are only adding an index data structure to it.

[DRKWang]

comment:11

Since the rtree.count(), which will return the number of objects whose boxes have intersections with the target point, is not exactly suitable for our expected function, I try to figure out some other functions to replace it. In my opinion, the best function would be rtree.getfirst(), if we should have, which ideally can assist with both RealSet.contains() and RealSet.interesction(). However, I did not get such a function, instead, I found rtree.nearest(), which will return the item whose box has minimal distance to the target point. However, after comparing rtree.nearest() with rtree.count(), the test results shows the rtree.nearest() is slower than rtree.count().

[mkoeppe]

comment:12

As the boxes overestimate the true objects, a count result of 0 is useful.

[mkoeppe]

comment:13

For intersection, can you propose a function for rtree that would help?

[DRKWang]

comment:14

Replying to @mkoeppe:

For intersection, can you propose a function for rtree that would help?

Yes, I think rtree.count() also can help with that when there is a non-intersection between RealSet A and RealSet B, And I have implemented the code.

[DRKWang]

comment:15

I am thinking about changing the internal code of function RealSet.union(). Instead of ordinary function union(A,B), which has two equally operands, the function RealSet.union() has already self, and this format kind of indicates the self set is dominating and the other set will be added to self. In this case, I feel when we are trying to extend this function onto the rtree_bssed_Realset class, it is better to return a rtree_based_RealSet subclass instead of a RealSet class, since the programmer kind of wants to maintain the internal rtree data structure. And since this function only covers the + operation, which correspondingly is also easy to implement with rtree, thus, I feel I can try to achieve that.

[mkoeppe]

comment:16

Replying to @DRKWang:

I feel when we are trying to extend this function onto the rtree_bssed_Realset class, it is better to return a rtree_based_RealSet subclass instead of a RealSet class,

Yes, I agree. This is a general change that should be made throughout the RealSet class. For example, in RealSet.complement, it should probably use self.__class__ instead of RealSet when it constructs the result.