-
Notifications
You must be signed in to change notification settings - Fork 4
/
CHANGELOG
356 lines (300 loc) · 14.4 KB
/
CHANGELOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
2024-02-06 Release 2.2.1:
- require libhxl==5.2.1:
- make logup log level configurable via environment variable
2024-01-18 Release 2.2:
- fix pcodes urls
- require libhxl==5.2:
- update URLs for test files on GitHub
- fixed bug in aggregators that excluded tag patterns ending in "!"
2023-09-26 Release 2.1:
- remove all SQL support
- require libhxl==5.1 to reduce logging output
2023-07-13 Release 2.0.2:
- update to libhxl==5.0.2 to support the .hxl extension for CSV files
2023-06-01 Release 2.0.1:
- tweak dependency versions to avoid requests-cache conflicts
- move setup info from setup.py to setup.cfg
- add more dependency versions to about.html
- ensure that the HXL Proxy can read from itself
2023-05-18 Release 2.0
- from libhxl:
- /api/source-info now returns columns for headers and HXL hashtags
- URL munging fixed for direct-download HXL Proxy URLs
2023-04-06 Release 1.29
- from libhxl:
- /api/source-info now supports all formats (not just XLSX and XLS)
- /api/source-info now returns headers and HXL hashtags
- libhxl fixes for API-munging bug
- renamed 'hashtag_hash' property from /api/source-info to 'hxl_header_hash'
2023-03-29 Release 1.28.1
- temporarily deactivate Python-layer timeout (causing random request failures)
2023-03-20 Release 1.28
- remove controller support for adding or editing saved recipes
- request code stops running after a configurable timeout (default: 30 seconds)
- remove skip-verify-ssl option (SSL certificates for remote data must always be valid)
- add support for structured logs (e.g. JSON)
- add support for Visual Studio
- remove all frozen version requirements (except urllib3 for requests), so that we'll pull the latest packages on deployment
- get rid of warnings during user testing
2022-12-12 Release 1.27.3
- interim release to upgrade libhxl to 4.27.3 (decreased logging levels)
2022-10-07 Release 1.27.1
- interim release to address performance issues
- change Docker config to use threads with nginx
- make root logging level configurable via ROOT_LOGGING_LEVEL config property (and default to INFO instead of DEBUG)
2022-10-03 Release 1.27
- add REMOTE_ACCESS_LOGGING_LEVEL to filter "hxl.REMOTE_ACCESS" separately to show remote file access (currently,
all messages will be at the INFO level)
- show the same error message for a non-existant remote data file, or for a URL that contains HTML markup; prevents
bad actors from using the HXL Proxy to ping HTML websites
- don't trap 404 and HTML markup exceptions in the /data/edit endpoint
- forbid HTML markup or URLs in saved-recipe titles, descriptions, and stubs to cut down on spam attempts
2022-08-11 Release 1.26.1
- downgrade to mysql-connector-python==8.0.29 to avoid bug in 8.0.30
2022-08-08 Release 1.26
- add MAX_REQUEST_TIMEOUT configuration option, to place a cap on the maximum allowable timeout for reading external datasets
- add LOGGING_LEVEL configuration option (defaults to "ERROR")
- log warnings for exceptions caught and returned as web pages with 400/500-series status codes
2022-07-18 Release 1.25.2
- update to libhxl==4.25.2 to block tagger exploit
2022-07-14 Release 1.25.1
- improve logging output
2022-06-29 Release 1.25
- add /api/source-info to get information about a data source (Excel-only for now)
- add support for filling merged areas in Excel to UI and data-preview API
- change references to hxl.io to hxl.input for libhxl-python 1.25+
- refactored to support new InputOptions class in libhxl-python: all input options are now available for all API calls
- all unit tests passing
2022-03-16 Release 1.24.4
- fix cache threshold type
2022-03-16 Release 1.24.3
- added warning banner to admin pages, per OICT requirements (code committed for 1.25)
- increase threshold for cache
2021-04-29 Release 1.24.2
- bugfix release: change debug handling in config for data preview (to detect non-existant sheet in an Excel workbook)
- add PyPi publish target to Makefile
- fix unit tests to work in non-debug mode
2021-04-23 Release 1.24
- fix data-preview api call with sheet param 0
- cache improvements
- tagger fix from libhxl
2021-02-08 Release 1.23
- incorporate release 4.23 of libhxl-python, with stability
improvements for sorting and character-encoding errors
- improved support for redis caching
- disabled HID controller tests (not currently in use)
2021-01-15 Release 1.22
- consolidate recent bugfix releases
- update to libhxl=4.22 for security and Excel improvements and bug fixes
- fix redis caching for input and refactor/simplify input caching code
- add additional security unit tests
2020-12-08 Release 1.21.3
- update to libhxl=4.21.2 to fix rare Unicode encoding error for CSV lookahead buffers
2020-08-18 Release 1.21.2
- update to libhxl==4.21.1 to fix URL munging for HXL Proxy source links
2020-08-06 Release 1.21.1
- restored default Flask logging to stdout/stderr
2020-07-23 Release 1.21
- added a new "Logs" tab to show logs from HXL processing
- added UI support for Expand lists filter
- make tagger UI smarter about avoiding duplicates
- remove "whitelist" and "blacklist" throughout code and comments
- fix saved-recipe bug that prevented disabling cloning or removing a description
- added HTML form to set up data-preview API call interactively, and added objects.json format
- ensure that user always returns to select page after login diversion
- added /api/from-spec endpoint to read a full JSON spec (including input and tagger)
- updated to libhxl-python 4.21
- hide unusable data tabs
2020-05-21 Release 1.20
- fixed bug with JSONPath &selector
- added long description in setup.py for PyPi (from README.md)
- added link files for in-browser tests
- remove unneeded system library from DockerFile
2020-04-27 Release 1.19
- add UI support for labeling steps in a recipe
- added field on data-source page to specify a JSONpath selector to find the HXL data
2020-03-16 Release 1.18
- add UI support for new implode filter
- add UI support for flattening JSON lists
- add optional header field for rename-column filter
- add optional &filename= param for /api/data-preview
2020-02-18 Release 1.17
- added UI support for new Implode filter
- extended Rename column filter to optionally use headers as well as hashtags
- added optional configuration for redis caching (input and output); more needed
- added API endpoint /api/data-preview-sheets to fetch a list of sheet titles from an Excel workbook
- Docker fixes
2019-12-27 Release 1.16.1
- minor update: fix HXL_WORKERS check for Docker deployment
2019-12-09 Release 1.16
- update to libhxl v4.16
- add an admin interface under /admin (need to set a local admin password)
- add an endpoint /api/data-preview to do a raw (no-HXL) preview of tabular data sources
- migrate non-core API calls for P-codes, hash, and hxl_test—to under /api (old URLs still work)
- stay on the recipe page after updating a saved recipe
- fix bug with saving clone permission
- fix bugs with /data/advanced page (for direct JSON entry)
- fix bug in download stub filename (Flask wasn't escaping it)
- major code refactoring with new Recipe class
- remove obsolete iati2hxl endpoint
- improve organisation and documentation for controllers
- added more controller unit tests
- change Dockerfile tag for the base Python image
2019-04-04 Release 1.15.1
- fix caching for validation requests sent via POST
- removed obsolete basic chart and map demo
2019-03-29 Release 1.15
- allow user to provide authorisation token for a private CKAN
dataset or other link that returns 403 Forbidden
- proxy will automatically redirect and start a private dataset
when it detects an authorisation error
- fix 404 error for non-existant P-codes
2019-03-12 Release 1.14.1
- fix typo in config file template
- switch to Flask 1.x and Flask-Caching packages
- fix bug in caching for POST validation request
- increase gunicorn max GET request line to 8K characters
2019-03-04 Release 1.14
- longer cache for P-codes (1 week by default)
- cache results of POST validation service (will fill up cache
quickly with big files)
- bump to libhxl 4.14 for new features and bug fixes
2019-02-06 Release 1.13.1
- update to libhxl 4.13.2 for bug fixes and new round() formula
function
2019-01-31 Release 1.13
- update to libhxl 4.13, which adds support for row formulas and
fixes some bugs
2018-12-03 Release 1.12
- add Concatenate option to Count filter
- add /hash endpoint for generating a JSON dataset hash
- support zipped CSV files (via libhxl-python)
- fixed parsing of YYYY-MM-DD hh:mm:ss SQL dates (via libhxl-python)
2018-08-31 Release 1.11
- add special user-agent strings to help with analytics and traffic management
2018-07-31 Release 1.10
- add UI support for libhxl append-filter improvements (external list
and row filters)
- allow saved-recipe editing page to work even when there is a HXL error
(to fix broken recipes)
- updated libhxl to 4.10 (includes better date cleaning and CSV
separator detection, as well as the append-filter improvements)
2018-07-05 Release 1.9.1 (bug fix)
- fix bug in form-validation regex for tag patterns
2018-06-29 Release 1.9
- added IATI support with /iati2hxl?url=<url> endpoint
- added POST endpoint /actions/json-recipe for submitting JSON
recipes transactionally
- update forms validation to support new tag patterns ending in "!"
- refactored validation support to use JSON report from libhxl
2018-06-14 Release 1.8.3
- add UI support for JSON filter
- require libhxl==4.8.4 for JSON filter support
2018-06-05 Release 1.8.2
- require libhxl==4.8.3 for critical features and fixes
2018-05-31 Release 1.8.1
- hotfix for installation problem with 1.8 (in a clean install)
- require libhxl==4.8.2
- DOCKERFILE updates
2018-05-31 Release 1.8
- add __version__ attribute to module
- add /about.html
- change format of HTML validation error report
- when the validation error format is JSON, report other errors in
JSON as well
- add new POST endpoint /actions/validate (allows either
uploading or providing a URL for both the schema and the
content). Details: https://github.com/HXLStandard/hxl-proxy/wiki/Validation-service
- fixed bug preventing changing custom validation schema in
/data/validate
- added PyCharm project file
2018-05-11 Release 1.7.1
- require libhxl==4.7.1 for date-parsing hotfix
2018-04-30 Release 1.7
- add a /pcodes endpoint to download lists of P-codes
- update to support new libhxl-python hxl.datatypes module (remove
references to hxl.common)
- add JSON output option for data validation (/data/validate.json)
- show expected value in a validation error if available
- fix bug with logging into saved recipes from some pages
- allow sending an inline schema with the schema_content param
(GET or POST)
2018-03-31 Release 1.6
- add lat/lon to clean-data filter
- add purge option to clean-data filter
- support using MySQL as an alternative to SQLite3 for saved recipes
- default to an in-memory database for unit testing
- fix form for advanced JSON recipes
- fix template error for updating tagger in a saved recipe
2018-02-05 Release 1.5.1
- bug-fix release: update to libhxl 4.5.1 so to avoid crashes when
parsing a bad date
- use DEBUG config parameter to decide whether to catch exceptions
and display human-readable error messages
2018-01-31 Release 1.5
- added option to cut-column filter to skip untagged columns
- better error report for a private Google Sheet (or other cloud
file)
- added files from @teodorescuserban for building a HXL
Proxy Docker image
- fixed updating source for a saved recipe (short URL)
- support saving data as a JSON list-of-objects (as well as
list-of-lists)
- allow users to specify number format for clean filter
2017-11-22 Release 1.4
- send JSON error messages instead of HTML when the user is trying to download a
JSON or CSV file
- add subpath support, so that URLs will be correct when the Proxy
is mounted in a subpath in WSGI
- remove H.ID login link for now, since it doesn't do anything,
and is potentially confusing
- require libhxl-python v4.4 for improved exception handling and
JSON support
2017-06-13 Release 1.3
- require libhxl-python v4.3 for bug fixes
- fixed reference to old HDX URL (leading to SSL error)
2017-06-06 Release 1.2
- change SSL verification logic to skip verifying SSL
- redirect back to source page if any URL fails SSL cert check
- function changing changes
2017-06-06 Release 1.1
- add a verify checkbox to the source page; if unchecked, don't
try to verify SSL certs (checked by default)
- add UI support for the new fill-data filter
- added support for new date_format field in clean filter
- support reading new JSON-based input (via libhxl-python)
- use MIME type and extension where avialable to help choose input
type (via libhxl-python)
- use the character encoding specified in the HTTP response when
reading a dataset (via libhxl-python)
- more flexible handing of invalid SSL certs (via libhxl-python)
- new "is (not)" operator for row queries (via libhxl-python)
2016-12-02 Release 1.0
- added support for JSON recipes via a &recipe HTTP parameter
- add row queries for clean, count, deduplicate, replace, and
replace-map filters
- use "recipe" and "step" (instead of "filters") on data recipe page
- allow sorting filters/steps
- allow deleting filters/steps
- remove type selector for an already-defined filter/step
- &force=on works for input as well as output
- limit HTML table display to 5,000 rows to avoid timeouts (full
dataset still available via CSV or JSON download)
- add a max_rows parameter to limit the number of rows returned.
- add a /hxl-test page (available in JSON as well)
- modify the append-filter form to add URLs on demand
- support the new multi-URL append filter from libhxl-python
2016-08-30 Release 0.6
- Add support for requests-cache, so that different views of the same
dataset won't require constant redownloading.
- Add support for a local copy of the p-code GeoJSON shapes tree, for
offline demos (or other purposes).
- Make the list of countries presented in the mapping dropdown menu
configurable.
- Fix exception handling around a missing data source, to avoid
unpredictable behaviour (and incorrect claims of missing HXL hashtags
on first load from Google Drive).
- Start adding scaffolding for jquery-sortable to rearrange filters, but
not supported yet in the backend.
- Remove the sidebar (very slow to download the PNGs in a field
environment).