Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Enable automated CoreNLP, Senna, Prover9/Mace4, Megam, MaltParser CI …
…tests (#2820) * Re-enabled CoreNLP tests, added CoreNLP zip download to CI * Specified JDK version 16 * Specified CLASSNAME environment variable in a different way * Run pytest with CLASSNAME env, and JDK set up * Renamed incorrect CLASSNAME env to CLASSPATH * Output the core_nlp folder contents and try to set up the CoreNLPServer * Also ls the core_nlp folder after creation * Download zip to relative path * Use full paths instead of relative ones * Make directory using full path * Load third party download cache in main pytest job * Output third party folder * Rename stanford-corenlp-\d.\d.\d to remove the version specifier * Ignore exit code 1 * Fixed input to execdir * Pipe guaranteed error warning to null, load cache before Python * Revert to using a third-party.sh script * Output more before attempting to run third-party script * Attempt to get more information * Added third-party.sh * Create envs.sh before filling it * Create envs.sh in a new way * Specified bash shell * Force envs.sh to give a good exit code * Remove printenv * Add to GITHUB_ENV instead of setting env vars some other way * Print CLASSPATH in multiple ways * Hardcode environment variable paths * Use of pwd instead of /home/runner/ * Use ~ instead of pwd * Output folder that should have the Stanford CoreNLP jar * Modified used environment variables * Rename instead of sym-link downloaded folders * Remove ls for CORENLP * Output a tree structure of the cached third party folder * Add os.path.expanduser to find_jar_iter * Removed tree call * More outputting * Cache third party downloads on windows too * Fixed runs-on parameter * Specified sh to third-party call * Compromise by only downloading third party data for Linux tests * Split test step * Add automatic Prover9-Mace4 download * Use .envs.sh instead of hardcoded environment variables * Add environment variables to .bashrc directly * Use export in the same step for env variables * Revert to using echo >> ~/.bashrc * Specify third party env's as top-level * Replace $ {HOME} with ~ * Use /third instead of ~/third * Revert to ~/third, prepend /home/runner * Resolve replace-all error from c2988cc * Attempting to fix broken Prover9 test * Added automatic download for MEGAM and TADM * Set MEGAM and TADM executables to chmod 711 * Updated TADM environment variable to link to the bin folder instead * Remove TADM, add MaltParser * Revert to using MaltParser 1.7.2 * Gave MaltParser test mandatory 'parser_dirname' var * Initialize MaltParser with tagger function, not tagger class instance * Set exe mode in third-party.sh, try to enable another MEGAM test * Use folder name instead of file name for MEGAM change mode * Modified verbosity, changed chmod file * Only download third party, and modify debug logs * Move megam executable to megam directory * Make directory before moving MEGAM into it * Re-enable full third party & nltk_data download and tests * Remove unnecessary, broken megam config check * Removed BFGS as algorithm in rte_classifier, as MaxentClassifier doesn't support it * Removed unnecessary 'import os' * Modified docstring in third-party bash script * Only set up Java if the runner is Linux
- Loading branch information
Showing
9 changed files
with
184 additions
and
26 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,112 @@ | ||
| #!/bin/bash | ||
| # This install script is used in our GitHub Actions CI. | ||
| # See .github/workflows/ci.yaml | ||
|
|
||
| # Installing the third-party software and the appropriate env variables. | ||
| pushd ${HOME} | ||
| [[ ! -d 'third' ]] && mkdir 'third' | ||
| pushd 'third' | ||
|
|
||
| # Download nltk stanford dependencies | ||
| # Downloaded to ~/third/stanford-corenlp | ||
| # stanford_corenlp_package_zip_name=$(curl -s 'https://stanfordnlp.github.io/CoreNLP/' | grep -o 'stanford-corenlp-full-.*\.zip' | head -n1) | ||
| stanford_corenlp_package_zip_name="stanford-corenlp-full-2017-06-09.zip" | ||
| [[ ${stanford_corenlp_package_zip_name} =~ (.+)\.zip ]] | ||
| stanford_corenlp_package_name=${BASH_REMATCH[1]} | ||
| if [[ ! -d ${stanford_corenlp_package_name} ]]; then | ||
| curl -L "https://nlp.stanford.edu/software/$stanford_corenlp_package_zip_name" -o ${stanford_corenlp_package_zip_name} | ||
| # wget -nv "http://nlp.stanford.edu/software/$stanford_corenlp_package_zip_name" | ||
| unzip -q ${stanford_corenlp_package_zip_name} | ||
| rm ${stanford_corenlp_package_zip_name} | ||
| mv ${stanford_corenlp_package_name} 'stanford-corenlp' | ||
| fi | ||
|
|
||
|
|
||
| # Downloaded to ~/third/stanford-parser | ||
| #stanford_parser_package_zip_name=$(curl -s 'https://nlp.stanford.edu/software/lex-parser.shtml' | grep -o 'stanford-parser-full-.*\.zip' | head -n1) | ||
| stanford_parser_package_zip_name="stanford-parser-full-2017-06-09.zip" | ||
| [[ ${stanford_parser_package_zip_name} =~ (.+)\.zip ]] | ||
| stanford_parser_package_name=${BASH_REMATCH[1]} | ||
| if [[ ! -d ${stanford_parser_package_name} ]]; then | ||
| curl -L "https://nlp.stanford.edu/software/$stanford_parser_package_zip_name" -o ${stanford_parser_package_zip_name} | ||
| # wget -nv "https://nlp.stanford.edu/software/$stanford_parser_package_zip_name" | ||
| unzip -q ${stanford_parser_package_zip_name} | ||
| rm ${stanford_parser_package_zip_name} | ||
| mv ${stanford_parser_package_name} 'stanford-parser' | ||
| fi | ||
|
|
||
| # Downloaded to ~/third/stanford-postagger | ||
| #stanford_tagger_package_zip_name=$(curl -s 'https://nlp.stanford.edu/software/tagger.shtml' | grep -o 'stanford-postagger-full-.*\.zip' | head -n1) | ||
| stanford_tagger_package_zip_name="stanford-postagger-full-2017-06-09.zip" | ||
| [[ ${stanford_tagger_package_zip_name} =~ (.+)\.zip ]] | ||
| stanford_tagger_package_name=${BASH_REMATCH[1]} | ||
| if [[ ! -d ${stanford_tagger_package_name} ]]; then | ||
| curl -L "https://nlp.stanford.edu/software/$stanford_tagger_package_zip_name" -o ${stanford_tagger_package_zip_name} | ||
| # wget -nv "https://nlp.stanford.edu/software/$stanford_tagger_package_zip_name" | ||
| unzip -q ${stanford_tagger_package_zip_name} | ||
| rm ${stanford_tagger_package_zip_name} | ||
| mv ${stanford_tagger_package_name} 'stanford-postagger' | ||
| fi | ||
|
|
||
| # Download SENNA to ~/third/senna | ||
| senna_file_name=$(curl -s 'https://ronan.collobert.com/senna/download.html' | grep -o 'senna-v.*.tgz' | head -n1) | ||
| senna_folder_name='senna' | ||
| if [[ ! -d $senna_folder_name ]]; then | ||
| curl -L "https://ronan.collobert.com/senna/$senna_file_name" -o ${senna_file_name} | ||
| # wget -nv "https://ronan.collobert.com/senna/$senna_file_name" | ||
| tar -xzf ${senna_file_name} | ||
| rm ${senna_file_name} | ||
| fi | ||
|
|
||
| # Download PROVER9 to ~/third/prover9 | ||
| prover9_file_name="p9m4-v05.tar.gz" | ||
| [[ ${prover9_file_name} =~ (.+)\.tar\.gz ]] | ||
| prover9_folder_name=${BASH_REMATCH[1]} | ||
| if [[ ! -d ${prover9_folder_name} ]]; then | ||
| curl -L "https://www.cs.unm.edu/~mccune/prover9/gui/$prover9_file_name" -o ${prover9_file_name} | ||
| tar -xzf ${prover9_file_name} | ||
| mv ${prover9_folder_name} 'prover9' | ||
| rm ${prover9_file_name} | ||
| fi | ||
|
|
||
| # Download MEGAM to ~/third/megam | ||
| megam_file_name="megam_i686.opt.gz" | ||
| [[ ${megam_file_name} =~ (.+)\.gz ]] | ||
| megam_folder_name=${BASH_REMATCH[1]} | ||
| if [[ ! -d ${megam_folder_name} ]]; then | ||
| curl -L "http://hal3.name/megam/$megam_file_name" -o ${megam_file_name} | ||
| gunzip -vf ${megam_file_name} | ||
| mkdir -p "megam" | ||
| mv ${megam_folder_name} "megam/${megam_folder_name}" | ||
| chmod -R 711 "megam/$megam_folder_name" | ||
| fi | ||
|
|
||
| # TADM requires `libtaopetsc.so` from PETSc v2.3.3, and likely has more | ||
| # tricky to install requirements, so we don't run tests for it. | ||
|
|
||
| # Download TADM to ~/third/tadm | ||
| # tadm_file_name="tadm-0.9.8.tgz" | ||
| # [[ ${tadm_file_name} =~ (.+)\.tgz ]] | ||
| # tadm_folder_name=${BASH_REMATCH[1]} | ||
| # if [[ ! -d ${tadm_folder_name} ]]; then | ||
| # curl -L "https://master.dl.sourceforge.net/project/tadm/tadm/tadm%200.9.8/$tadm_file_name?viasf=1" -o ${tadm_file_name} | ||
| # tar -xvzf ${tadm_file_name} | ||
| # rm ${tadm_file_name} | ||
| # chmod -R 711 "./tadm/bin/tadm" | ||
| # fi | ||
|
|
||
| # Download MaltParser to ~/third/maltparser | ||
| malt_file_name="maltparser-1.7.2.tar.gz" | ||
| [[ ${malt_file_name} =~ (.+)\.tar\.gz ]] | ||
| malt_folder_name=${BASH_REMATCH[1]} | ||
| if [[ ! -d ${malt_folder_name} ]]; then | ||
| curl -L "http://maltparser.org/dist/$malt_file_name" -o ${malt_file_name} | ||
| tar -xzf ${malt_file_name} | ||
| mv ${malt_folder_name} 'maltparser' | ||
| rm ${malt_file_name} | ||
| fi | ||
|
|
||
| ls ~/third | ||
|
|
||
| popd | ||
| popd |