usage-upload, trouble : make install

Susan and Galen,

Thanks for your patience and persistence in trying to get this working. I'm sorry I didn't make this more robust from the start I put some effort last night and this morning into making it more robust, and tested it on Stampede 2 using their Python 2.7 and other environments using Python 3.

Please try the latest which is currently 1.6-1 at: https://software.xsede.org/development/usage-analysis/. Also, please start in a pristine environment and directory that doesn't have previous failed installations.

NOTE: The first Python 2.7 or 3 in your PATH must satisfy the pre-requisites and include Python virtualenv.

Thanks,

JP

p.s. this is what it looks like when it works on Stampede 2:

login4(333)$ wget https://software.xsede.org/development/usage-analysis/usage-upload-1.6-1.tgz

--2019-01-23 09:08:52--  https://software.xsede.org/development/usage-analysis/usage-upload-1.6-1.tgz

Resolving software.xsede.org (software.xsede.org)... 149.165.157.137

Connecting to software.xsede.org (software.xsede.org)|149.165.157.137|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 9599 (9.4K) [application/x-gzip]

Saving to: ‘usage-upload-1.6-1.tgz’


100%[=============================================================================================>] 9,599       --.-K/s   in 0.001s  


2019-01-23 09:08:52 (8.93 MB/s) - ‘usage-upload-1.6-1.tgz’ saved [9599/9599]


login4(334)$ ls -al

total 9

drwx------ 2 tg455671 G-80236 4096 Jan 23 09:08 .

drwx------ 3 tg455671 G-80236 4096 Jan 23 09:08 ..

-rw------- 1 tg455671 G-80236 9599 Jan 23 09:05 usage-upload-1.6-1.tgz

login4(335)$ tar -xzvf usage-upload-1.6-1.tgz 

usage-upload-1.6-1/

usage-upload-1.6-1/bin/

usage-upload-1.6-1/Makefile

usage-upload-1.6-1/etc/

usage-upload-1.6-1/RELEASE

usage-upload-1.6-1/README

usage-upload-1.6-1/VERSION

usage-upload-1.6-1/etc/upload_glue2.conf

usage-upload-1.6-1/etc/upload_template.conf.in

usage-upload-1.6-1/etc/upload_rabbitmq.conf

usage-upload-1.6-1/bin/upload_usage.py.in

usage-upload-1.6-1/bin/upload_glue2.sh

usage-upload-1.6-1/bin/usage_sync.py

usage-upload-1.6-1/bin/rabbitmq_parser.py.in

usage-upload-1.6-1/bin/upload_glue2.cron.in

usage-upload-1.6-1/bin/upload_template.cron.in

usage-upload-1.6-1/bin/usage_splitter.py

usage-upload-1.6-1/bin/glue2_history_parser.py

login4(336)$ cd usage-upload-1.6-1

login4(339)$ export PREFIX=/home1/00296/tg455671/usage-upload

login4(340)$ python -c "import virtualenv"

login4(341)$ make install

** Initializing Python Virtualenv in /home1/00296/tg455671/usage-upload/python

virtualenv /home1/00296/tg455671/usage-upload/python

New python executable in /home1/00296/tg455671/usage-upload/python/bin/python2.7

Also creating executable in /home1/00296/tg455671/usage-upload/python/bin/python

Installing setuptools, pip, wheel...done.

( source /home1/00296/tg455671/usage-upload/python/bin/activate ; \

    /home1/00296/tg455671/usage-upload/python/bin/pip --disable-pip-version-check install --upgrade pip )

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.

Requirement already up-to-date: pip in /home1/00296/tg455671/usage-upload/python/lib/python2.7/site-packages (19.0.1)

( source /home1/00296/tg455671/usage-upload/python/bin/activate ; \

    /home1/00296/tg455671/usage-upload/python/bin/pip install numpy )

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.

Collecting numpy

  Using cached https://files.pythonhosted.org/packages/9f/85/163127d3fb0573deb9eca947cfc73aa3618eaaf8656501460574471d114a/numpy-1.16.0-cp27-cp27mu-manylinux1_x86_64.whl

h5py 2.7.0 requires six, which is not installed.

Installing collected packages: numpy

Successfully installed numpy-1.16.0

( source /home1/00296/tg455671/usage-upload/python/bin/activate ; \

    /home1/00296/tg455671/usage-upload/python/bin/pip install paramiko )

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.

Collecting paramiko

  Using cached https://files.pythonhosted.org/packages/cf/ae/94e70d49044ccc234bfdba20114fa947d7ba6eb68a2e452d89b920e62227/paramiko-2.4.2-py2.py3-none-any.whl

Collecting cryptography>=1.5 (from paramiko)

  Using cached https://files.pythonhosted.org/packages/17/fd/4c2c8953a9dfe38fbe0c3adafb6355540bd98cef70cc82734acb0a4c0e2f/cryptography-2.5-cp27-cp27mu-manylinux1_x86_64.whl

Collecting pynacl>=1.0.1 (from paramiko)

  Using cached https://files.pythonhosted.org/packages/b3/25/e605574f24948a8a53b497744e93f061eb1dbe7c44b6465fc1c172d591aa/PyNaCl-1.3.0-cp27-cp27mu-manylinux1_x86_64.whl

Collecting pyasn1>=0.1.7 (from paramiko)

  Using cached https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl

Collecting bcrypt>=3.1.3 (from paramiko)

  Using cached https://files.pythonhosted.org/packages/50/d8/95f7cb04344033bf9d1a12c5a7969a15999b6a710fbe1969c517333d9a62/bcrypt-3.1.6-cp27-cp27mu-manylinux1_x86_64.whl

Collecting enum34; python_version < "3" (from cryptography>=1.5->paramiko)

  Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl

Collecting asn1crypto>=0.21.0 (from cryptography>=1.5->paramiko)

  Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl

Collecting cffi!=1.11.3,>=1.8 (from cryptography>=1.5->paramiko)

  Using cached https://files.pythonhosted.org/packages/14/dd/3e7a1e1280e7d767bd3fa15791759c91ec19058ebe31217fe66f3e9a8c49/cffi-1.11.5-cp27-cp27mu-manylinux1_x86_64.whl

Collecting six>=1.4.1 (from cryptography>=1.5->paramiko)

  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl

Collecting ipaddress; python_version < "3" (from cryptography>=1.5->paramiko)

  Using cached https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl

Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=1.5->paramiko)

Installing collected packages: enum34, asn1crypto, pycparser, cffi, six, ipaddress, cryptography, pynacl, pyasn1, bcrypt, paramiko

Successfully installed asn1crypto-0.24.0 bcrypt-3.1.6 cffi-1.11.5 cryptography-2.5 enum34-1.1.6 ipaddress-1.0.22 paramiko-2.4.2 pyasn1-0.4.5 pycparser-2.19 pynacl-1.3.0 six-1.12.0

( source /home1/00296/tg455671/usage-upload/python/bin/activate ; \

    /home1/00296/tg455671/usage-upload/python/bin/pip install pytz )

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.

Collecting pytz

  Using cached https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl

Installing collected packages: pytz

Successfully installed pytz-2018.9

** Installing usage-upload-1.6-1

sed -e "s|%PREFIX%|/home1/00296/tg455671/usage-upload|g" \

   bin/upload_usage.py.in >/home1/00296/tg455671/usage-upload/bin/upload_usage.py

chmod 755 /home1/00296/tg455671/usage-upload/bin/upload_usage.py

login4(342)$ make APP=jptest uploadapp

** Generating /home1/00296/tg455671/usage-upload/bin/upload_jptest.cron

** Generating /home1/00296/tg455671/usage-upload/etc/upload_jptest.conf

login4(343)$ /home1/00296/tg455671/usage-upload/bin/upload_jptest.cron

Traceback (most recent call last):

  File "/home1/00296/tg455671/usage-upload/bin/upload_usage.py", line 228, in <module>

    rc = task.upload_directory(task.config.get('LOCAL_DIRECTORY'), task.config.get('LOCAL_GLOB'))

  File "/home1/00296/tg455671/usage-upload/bin/upload_usage.py", line 98, in upload_directory

    matching_files = [f for f in fnmatch.filter(os.listdir(log_path), log_glob) if os.path.isfile(os.path.join(log_path, f))]

OSError: [Errno 2] No such file or directory: '<USAGE_FILE_DIRECTORY>'