Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Some images cropped/cut off #39

Open
jywarren opened this issue May 23, 2019 · 19 comments
Open

Some images cropped/cut off #39

jywarren opened this issue May 23, 2019 · 19 comments
Labels
bug Something isn't working

Comments

@jywarren
Copy link
Member

I see a couple edges which are not fuzzy, and so that means they were cut off partway through their composite processes.

image

image

https://mapknitter.org/maps/otter-view-park-west

I'm wondering how it happened, but it could be either a crash partway through, OR it could be that their new bounding boxes are not properly calculated and they're just cropped wrong. I'll have to check the intermediate images to be sure.
@jywarren jywarren added the bug Something isn't working label May 23, 2019
@jywarren jywarren mentioned this issue May 23, 2019
Open
2 tasks
@jywarren
Copy link
Member Author

This is one that got cut off: 47243/OVPKAP-20111028-85-170_medium.jpg

I'll try to make a special warpable.json for just that

@jywarren
Copy link
Member Author

jywarren commented May 24, 2019 •
edited 

[{"cm_per_pixel":9.71209,"created_at":"2011-10-29T01:30:36Z","deleted":true,"height":2448,"history":"","id":47243,"image_content_type":"image/jpeg","image_file_name":"OVPKAP-20111028-85-170.jpg","image_file_size":971714,"locked":false,"map_id":662,"nodes":[{"author":"662","body":null,"color":"black","created_at":"2011-10-29T01:33:00Z","description":"","id":98866,"lat":"44.0182263551","lon":"-73.178511401","map_id":0,"name":"","order":0,"updated_at":"2011-10-29T01:33:00Z","way_id":0,"way_order":0},{"author":"662","body":null,"color":"black","created_at":"2011-10-29T01:33:00Z","description":"","id":98867,"lat":"44.0161785757","lon":"-73.1784863821","map_id":0,"name":"","order":0,"updated_at":"2011-10-29T01:33:00Z","way_id":0,"way_order":0},{"author":"662","body":null,"color":"black","created_at":"2011-10-29T01:33:00Z","description":"","id":98868,"lat":"44.0163197466","lon":"-73.1806368023","map_id":0,"name":"","order":0,"updated_at":"2011-10-29T01:33:00Z","way_id":0,"way_order":0},{"author":"662","body":null,"color":"black","created_at":"2011-10-29T01:33:00Z","description":"","id":98869,"lat":"44.0184715903","lon":"-73.1804661828","map_id":0,"name":"","order":0,"updated_at":"2011-10-29T01:33:00Z","way_id":0,"way_order":0}],"parent_id":null,"thumbnail":null,"updated_at":"2011-10-29T01:36:19Z","width":3264,"src":"https://s3.amazonaws.com/grassrootsmapping/warpables/47243/OVPKAP-20111028-85-170.jpg","srcmedium":"https://s3.amazonaws.com/grassrootsmapping/warpables/47243/OVPKAP-20111028-85-170_medium.jpg"}]

@jywarren
Copy link
Member Author

It's not cut off!

image

@jywarren
Copy link
Member Author

jywarren commented May 24, 2019 •
edited

So, i'll run this again and let's see how it goes - if it fails in the same places, or if this was a memory error and they'll get cut off differently:

http://export.mapknitter.org/export?url=https://mapknitter.org/maps/otter-view-park-west/warpables.json&scale=30

http://export.mapknitter.org/id/1558733043/status.json

@jywarren
Copy link
Member Author

jywarren commented May 25, 2019 •
edited

Uh oh, that's still stuck at 3 of 25. Not sure why. Will restart it.

{"status_url":"https://mapknitter-exports-warps.storage.googleapis.com/1558733043/status.json","status":"warping 3 of 25","tms":false,"geotiff":false,"zip":false,"jpg":false,"export_id":1558733043,"user_id":null,"size":null,"width":null,"height":null,"start_time":null,"run_time":null,"gem_version":"1.0.6","cm_per_pixel":null}

Here's the restarted one:

https://mapknitter-exports-warps.storage.googleapis.com/1558800253/status.json

@jywarren
Copy link
Member Author

{"status_url":"https://mapknitter-exports-warps.storage.googleapis.com/1558800253/status.json","status":"warping 3 of 25","tms":false,"geotiff":false,"zip":false,"jpg":false,"export_id":1558800253,"user_id":null,"size":null,"width":null,"height":null,"start_time":null,"run_time":null,"gem_version":"1.0.7","cm_per_pixel":null}

So, on the new version. Already at number 3!

@jywarren
Copy link
Member Author

OK, here - for comparison with above image:

image

@jywarren
Copy link
Member Author

Earlier and later images, left to right; the new one is better, but still not complete:

image

@jywarren
Copy link
Member Author

From notes here: #23 (comment)

It looks like it happens on or just before this step, because images like this are copied in wrong, and then distorted, masked, and used in the final map:

https://github.com/publiclab/mapknitter-exporter/blob/c767780392342c109b26a887971b3ab6bcd98c94/lib/mapknitterExporter.rb#L158-L173

@jywarren
Copy link
Member Author

Here's the start of the processing for the above image: https://console.cloud.google.com/logs/viewer?authuser=1&project=public-lab&minLogLevel=0&expandAll=false&timestamp=2019-05-27T23%3A11%3A47.765000000Z&customFacets&limitCustomFacetWidth=true&dateRangeStart=2019-05-20T23%3A13%3A27.528Z&interval=P7D&resource=k8s_container%2Fcluster_name%2Fmapknitter&scrollTimestamp=2019-05-26T04%3A59%3A26.881730593Z&dateRangeEnd=2019-05-27T23%3A13%3A27.528Z&advancedFilter=resource.type%3D%22k8s_container%22%0Aresource.labels.namespace_name%3D%22default%22%0Aresource.labels.location%3D%22us-east1%22%0Aresource.labels.project_id%3D%22public-lab%22%0Aresource.labels.cluster_name%3D%22mapknitter%22%0Aresource.labels.pod_name%3D%22mapknitter-exporter-kubes-6754bb8c54-q6qwt%22%0Aresource.labels.container_name%3D%22mapknitter-exporter-kubes%22%0Atimestamp%3D%222019-05-26T04%3A59%3A25.242514883Z%22%0AinsertId%3D%22f5rcvu9l8f5t5y0yt%22

132014

@jywarren
Copy link
Member Author 

E  - -> /working/1558800531/w132014.tif 
convert -contrast-stretch 0 public/warps/1558824229-working/w132014-LPV-0483.JPG -crop 3648x3648+0+0! -flatten -distort Perspective '0,3648 0,522  0,0 227,0  2736,0 786,438  2736,3648 596,794' -flatten -crop 794x794+0+0! +repage public/warps/1558824229/w132014.png
convert +antialias -size 794x794  xc:none -draw "fill black stroke red stroke-width 30 polyline 0,522 227,0 786,438 596,794 0,522"  -alpha set -channel A -transparent red -blur 0x8 -channel R -evaluate set 0 +channel public/warps/1558824229/w132014-mask.png 
composite public/warps/1558824229/w132014-mask.png public/warps/1558824229/w132014.png -compose DstIn -alpha Set public/warps/1558824229/w132014-masked.png
gdal_translate -of GTiff -a_srs EPSG:4326  -gcp 0.0, 522.4933237396181, -73.1653081419, 44.0562634908 -gcp 227.05951564013958, 0.0, -73.1646962288, 44.0572754172 -gcp 786.0191691294312, 438.93269662186503, -73.1631898628, 44.056425326 -gcp 596.7493552267551, 794.3575743623078, -73.1636999347, 44.0557369574  -co "TILED=NO" public/warps/1558824229/w132014-masked.png public/warps/1558824229/w132014-geo-unwarped.tif
gdalwarp -of GTiff -t_srs EPSG:4326 public/warps/1558824229/w132014-geo-unwarped.tif public/warps/1558824229/w132014.tif

@jywarren
Copy link
Member Author

jywarren commented May 27, 2019 •
edited

So, i'd guess it's at this step, before mask generation and mask application:

2019-05-26 00:59:25.242 EDT

convert -contrast-stretch 0 public/warps/1558824229-working/w132014-LPV-0483.JPG -crop 3648x3648+0+0! -flatten -distort Perspective '0,3648 0,522 0,0 227,0 2736,0 786,438 2736,3648 596,794' -flatten -crop 794x794+0+0! +repage public/warps/1558824229/w132014.png

It's 20 mins until the next command, but i couldn't find any errors between them, besides a lot of garbage collection:

2019-05-26 01:20:52.329 EDT convert +antialias -size 794x794 xc:none -draw "fill black stroke red stroke-width 30 polyline 0,522 227,0 786,438 596,794 0,522" -alpha set -channel A -transparent red -blur 0x8 -channel R -evaluate set 0 +channel public/warps/1558824229/w132014-mask.png

@jywarren
Copy link
Member Author

Maybe we can fetch this file next time: http://export.mapknitter.org/public/warps/1558824229/w132014.png

@jywarren
Copy link
Member Author

Actually wait. Maybe we are seeing 2 types of cutoffs here. The one in the last image is diagonal bc the error was pre-distortion, i think? But the cutoff images in the big export outputs in this comment are all completely horizontal, as if they got cut off during the final compositing step.

@jywarren
Copy link
Member Author

That final compositing step is:

gdalwarp -s_srs EPSG:3857 -te -73.1669712067 44.053772949 -73.1613063812 44.0582295713 public/warps/1558800531/w131950.tif public/warps/1558800531/w131954.tif public/warps/1558800531/w131958.tif public/warps/1558800531/w131962.tif public/warps/1558800531/w131966.tif public/warps/1558800531/w131970.tif public/warps/1558800531/w131974.tif public/warps/1558800531/w131978.tif public/warps/1558800531/w131982.tif public/warps/1558800531/w131986.tif public/warps/1558800531/w131990.tif public/warps/1558800531/w131994.tif public/warps/1558800531/w131998.tif public/warps/1558800531/w132002.tif public/warps/1558800531/w132006.tif public/warps/1558800531/w132010.tif public/warps/1558800531/w132014.tif public/warps/1558800531/w132018.tif public/warps/1558800531/w132022.tif public/warps/1558800531/w132030.tif public/warps/1558800531/w132034.tif public/warps/1558800531/w132038.tif public/warps/1558800531/w132042.tif public/warps/1558800531/w132046.tif public/warps/1558800531/w132118.tif public/warps/1558800531/w132150.tif public/warps/1558800531/w132154.tif public/warps/1558800531/w132158.tif public/warps/1558800531/w132162.tif public/warps/1558800531/w132166.tif public/warps/1558800531/w132170.tif public/warps/1558800531/w132178.tif public/warps/1558800531/w132182.tif public/warps/1558800531/w132186.tif public/warps/1558800531/w132190.tif public/warps/1558800531/w132194.tif public/warps/1558800531/w132198.tif public/warps/1558800531/w132202.tif public/warps/1558800531/w132206.tif public/warps/1558800531/w132210.tif public/warps/1558800531/w132214.tif public/warps/1558800531/w132218.tif public/warps/1558800531/w132222.tif public/warps/1558800531/w132226.tif public/warps/1558800531/w132230.tif public/warps/1558800531/w132234.tif public/warps/1558800531/w132238.tif public/warps/1558800531/w132242.tif public/warps/1558800531/1558800531.tif

here: https://console.cloud.google.com/logs/viewer?authuser=1&project=public-lab&minLogLevel=0&expandAll=false&timestamp=2019-05-27T23%3A11%3A47.765000000Z&customFacets&limitCustomFacetWidth=true&dateRangeStart=2019-05-20T23%3A21%3A11.415Z&interval=P7D&resource=k8s_container%2Fcluster_name%2Fmapknitter&scrollTimestamp=2019-05-26T07%3A15%3A23.632082370Z&dateRangeEnd=2019-05-27T23%3A21%3A11.415Z&filters=text%3A132014&advancedFilter=resource.type%3D%22k8s_container%22%0Aresource.labels.container_name%3D%22mapknitter-exporter-kubes%22%0Aresource.labels.namespace_name%3D%22default%22%0Aresource.labels.location%3D%22us-east1%22%0Aresource.labels.project_id%3D%22public-lab%22%0Aresource.labels.pod_name%3D%22mapknitter-exporter-kubes-6754bb8c54-q6qwt%22%0Aresource.labels.cluster_name%3D%22mapknitter%22%0Atimestamp%3D%222019-05-26T07%3A15%3A23.632082370Z%22%0AinsertId%3D%22f5rcvu9l8f5t5y6sr%22

@jywarren
Copy link
Member Author

This is the big image cut off in the final composite: https://s3.amazonaws.com/grassrootsmapping/warpables/47243/OVPKAP-20111028-85-170_medium.jpg

So, we should restart and try to catch that image in .tif before it's composited to confirm it's at the compositing step that it fails.

@jywarren
Copy link
Member Author

We can try increasing memory at https://github.com/publiclab/mapknitter-exporter-sinatra/blob/main/nolimit.xml

And/or set up a try/retry block?

@jywarren
Copy link
Member Author

We could use this as a template: https://github.com/publiclab/mapknitter/blob/c391beb4e4b27a92bb0a2c11c68bba782145416a/nolimit.xml

@jywarren
Copy link
Member Author

Oh, actually we've already applied that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jywarren