LS Background shift

Kelly Bellis

ME PLS 2099
5PLS
3-COLLECT_20170121-15.19.37.png

Recently I noticed a problem with what appears to be JField shifting the background image. It is not known how long this issue has been present. To illustrate this problem I've created a background map in Global Mapper in the RMaps SQLite .sqlitedb format which included as part of it two surveyed points shown here as 2 small bright green blurry squares labeled in black lettering Stone_32_AVG and Stone_35_AVG. Note that they fall most reasonably on top of the 2 even blurrier gravestones. After importing this into JField, the problem becomes clear, there is a shift between the actual surveyed points (vector) shown with its labels in light blue and the raster background image.

The magnitude of the horizontal shift is about 3.5' in the northing and about 0.4' in the easting.
 

Kelly Bellis

ME PLS 2099
5PLS
ITRF vs NAD83, maybe?
This is most plausible: JField isn't reading the background map correctly.

Using Blue Marble's Geographic Calculator 2016 SP2, I used the OPUS geographic values given for IGS08 (EPOCH:2017.0541) to be converted to NAD83(2011) Epoch 2010.0000 UTM Zone 19, meters. However it complained about HTDP models so had to make epochs on par with each other; i.e., 2010.0000. The result is reasonably close to my earlier guesstimated values, namely 1.171m in the northing and -0.324m in the easting.
 

Sean Joyce

Active Member
Just wondered if there was there an apparent shift or are the surveyed points in the right position on the image in Global Mapper on the computer?
 

Kelly Bellis

ME PLS 2099
5PLS
The surveyed points (shown above as the 2 small bright green blurry squares labeled in black lettering Stone_32_AVG and Stone_35_AVG) in Global Mapper are spot on.

Prove this to yourself by using only correctly produced ortho imagery - 6" imagery or better is ideal, surveying a point on the ground that's recognizable in the photo like an intersection of two paint stripes, import the survey data into Global Mapper and then export that image including the displayed vector data in the RMaps SQLite .sqlitedb format. Lastly, edit your JField project to use that map as a background map.
 

John Evers

Well-Known Member
5PLS
Here is my current solution which does indicate that we either have a problem in Global Mapper with the creation of the sqlitedb file, or we are interpreting the sqlitedb file incorrectly in J-Field.

The first image is the result of a second iteration of sorts. I set photo targets, obtained coordinates on them with the LS on Ohio South NAD83(2011). I flew the drone, used Pix4d to create the image. I then used Global Mapper to create the sqlitedb file, and opened as a background image on the LS. My photo targets needed to move North 2.9 feet, and West 1.6 feet to match my coordinate. I then manually edited the tfw (world file) and added 2.9 feet to the north coordinate, and subtracted 1.6 feet from the East coordinate. I then opened the image again in Global Mapper, and exported out the sqlitedb file. Everything now nicely lines up.
00269_3_COLLECT_FULL_20170121-15.12.01.png


For comparison purposes, and since I live in Ohio, I then downloaded our freely available State Wide Imagery for our farm. The first thing I did was manually edit the tfw world file, moving the coordinates North 2.9 feet. and West 1.6 feet. I then opened the images in Global Mapper, created the sqlitedb files, and dropped onto the LS. I went out and mapped a few things, and was very pleased with the result. I am only seeing at most 1 foot of possible disagreement. I am also certain that this imagery utilized no ground control, only airborn GNSS. The shift I am applying matches he difference between WGS83 and NAD83(2011).

00269_2_REVIEW_COLLECT_20170122-15.06.09.png
00269_2_REVIEW_COLLECT_20170122-15.06.25.png
00269_2_REVIEW_COLLECT_20170122-15.06.37.png
 

Kelly Bellis

ME PLS 2099
5PLS
Here is my current solution which does indicate that we either have a problem in Global Mapper with the creation of the sqlitedb file, or we are interpreting the sqlitedb file incorrectly in J-Field.
John - Screwing with the world file to make up for the shift doesn't seem like a solution to me but rather instead is creating bigger headaches for yourself. And should the format be something other than one using an external world file; e.g., a georeferenced PDF, etc., the compounded errors would further confound you and other users of your work product.
 

John Evers

Well-Known Member
5PLS
Kelly,

I agree that manipulating the world file is a bit cumbersome.
I wanted to:
A: Quantify that the shift matches the WGS83 NAD83 difference.
B: Come up with a temporary workaround.
C: Post some screen caps, enticing our users to take advantage of a very nice feature.
:)
 

Vladimir Prasolov

Well-Known Member
5PLS
View attachment 6002
Recently I noticed a problem with what appears to be JField shifting the background image. It is not known how long this issue has been present. To illustrate this problem I've created a background map in Global Mapper in the RMaps SQLite .sqlitedb format which included as part of it two surveyed points shown here as 2 small bright green blurry squares labeled in black lettering Stone_32_AVG and Stone_35_AVG. Note that they fall most reasonably on top of the 2 even blurrier gravestones. After importing this into JField, the problem becomes clear, there is a shift between the actual surveyed points (vector) shown with its labels in light blue and the raster background image.

The magnitude of the horizontal shift is about 3.5' in the northing and about 0.4' in the easting.
Hi Kelly,
Please send us a project with 4 well recognizable points on Google satellite imaginary. I think it could be coordinate system issue in RMaps. Really J-Filed supposes the RMaps tiles are represented in WGS84 at epoch 2005.0. If Global Mapper uses different assumption then that huge horizontal shift could be easily explained.
 

John Evers

Well-Known Member
5PLS
Vladimir,

In case it would be useful, I have attached a project with my own imagery with clearly visible photo targets. The Page named "AVG PTS" has the control points I shot at the photo targets. In Global mapper, these match up well. I had everything set to WGS84 in Global Mapper when I created the included sqlitedb file in Global Mapper. The file I included named Hole Four WGS.zip is the sqlitedb file.
 

Attachments

Kelly Bellis

ME PLS 2099
5PLS
Hillside Cemetery before 1860 Overview x 750.jpg

The recent discovery of the shifted background map displayed by JField is from a project involving the mapping of gravestones dated prior to 1860 in a particular cemetery. This was the first time that I had observed this issue; however, in conversations with another JField user, the problem has apparently been present for quite some time.

While the Hillside Cemetery project is my first observed instance of the issue; it is admittedly, not the best test case for proving that the background map is being misplaced by JField due to 1) the surveyed point for each gravestone is some cases is offset due to the respective leaning of each stone and 2) the resolution of the ortho imagery is 15cm.

Polly Mitchell 1848 20170120_143648 x 750.jpg



I have uploaded to my web server at http://panocea.us/1543HillsideCemetery/ the following items:
  • The original 15cm geotiff and its accompanying world file provided courtesy of the Maine Office of GIS and USGS (UTM, Zone 19 North, meters), flown May 2014
  • The archived JField project
  • Hillside-170121.txt NAD83(2011), Maine2000, East, U.S. Survey Feet
These items should be sufficient in recreating the issue being discussed, but if not, please let me know.


In general, Google imagery should not be used because it comes from any number of uncontrolled and undocumented sources (i.e., no metadata) for aerial imagery including satellites and has therefore never been correctly rectified. I've routinely seen horizontal errors in Google imagery in the range of 3 to 5 meters when ground truthing.
 

Kelly Bellis

ME PLS 2099
5PLS
Really J-Filed supposes the RMaps tiles are represented in WGS84 at epoch 2005.0. If Global Mapper uses different assumption then that huge horizontal shift could be easily explained.
Hello Vladimir,

I've gotten a little more information on the subject from BMG's Mike Childs:

Global Mapper creates the RMaps/MBTiles data using the Web Mercator projection. This is World Mercator with the pseudo-WGS84 datum that Google Maps uses. It is the original WGS84 with the odd behavior of doing the ellipsoid part of the conversion using a sphere of radius 6378137.0m rather than the ellipsoid. This results in a large Y delta with a normal WGS84 datum.

See http://www.gdal.org/frmt_mbtiles.html for GDAL that discusses the projection specification.

If JField is treating the datum as some variation of a normal WGS84 datum (regardless of epoch) there will be a large shift in position in the Y direction.

See https://alastaira.wordpress.com/2011/01/23/the-google-maps-bing-maps-spherical-mercator-projection/ for a good description of this odd Web Mercator system.
 

John Evers

Well-Known Member
5PLS
Kelly,

Sounds like you nailed it. It is them...not us.

I guess if "odd" behavior happens before J-Field gets a hold of it, we should expect to see odd results in J-Field.

I wonder if Vladimir can accommodate the "odd WGS84 sphere", and work it back to ellipsoid?

I hope :)
 

Vladimir Prasolov

Well-Known Member
5PLS
Hello Vladimir,

I've gotten a little more information on the subject from BMG's Mike Childs:

Global Mapper creates the RMaps/MBTiles data using the Web Mercator projection. This is World Mercator with the pseudo-WGS84 datum that Google Maps uses. It is the original WGS84 with the odd behavior of doing the ellipsoid part of the conversion using a sphere of radius 6378137.0m rather than the ellipsoid. This results in a large Y delta with a normal WGS84 datum.

See http://www.gdal.org/frmt_mbtiles.html for GDAL that discusses the projection specification.

If JField is treating the datum as some variation of a normal WGS84 datum (regardless of epoch) there will be a large shift in position in the Y direction.

See https://alastaira.wordpress.com/2011/01/23/the-google-maps-bing-maps-spherical-mercator-projection/ for a good description of this odd Web Mercator system.
Hi Kelly,
Thank you for detailed information from BMG's Mike Childs. We are using absolutely same projection as Google described it. Projection is not issue, because it would create mostly North-South shift, but not East-West. If Global Mapper has no HTDP inside, it could not take into account time-related shifts. The difference could be quite a big. Currently we make assumption regarding reference frame that it is WGS84 epoch 2005.0. If Global Mapper puts epoch information into R-Maps, we could try to use it. If not, we have no option, but manual input from user. In any case could you send us the map please. Also you can try to convert points into WGS84@2005.0 by our or any NGS software and use those coordinates in Global Mapper. In that case we assured about epoch.
 
Top