Uploading release asset to Github is corrupting file - python-requests

I am trying to automate our Github release process, and as part of it I am uploading release artifacts to Github release. Artifact I am uploading is a tar.gz file. I am using following python3.7 code to upload the artifact:
with open(filepath, 'rb') as file:
response = self._request_session.post(endpoint,
params={'name': local_artifact.filename},
files={local_artifact.filename: file})
Response status code I am getting is 201, which is expected as per Github upload release asset api documentation.
However, when I am downloading artifacts from Github, I am neither able to validate sha256 of the file, nor I am able open the tarball. On running tar -zxvf test.tar.gz I am getting following error:
tar: Unrecognized archive format
tar: Error exit delayed from previous errors.
I have already tested the file on local which I am uploading and that is working fine.
Can anyone please help me in identifying the problem here?
Here is response text I get after uploading file:
{
"url":"https://api.github.com/repos/c2tarun/upload_test_repo/releases/assets/17244134",
"id":17244134,
"node_id":"MDEyOlJlbGVhc2VBc3NldDE3MjQ0MTM0",
"name":"test.tar.gz",
"label":"",
"uploader":{
"login":"c2tarun",
"id":1129670,
...truncating for readability.
"type":"User",
"site_admin":false
},
"content_type":"multipart/form-data; boundary=1a13638ee5f5f57d303508eea4e64211",
"state":"uploaded",
"size":11969682,
"download_count":0,
"created_at":"2020-01-09T23:49:54Z",
"updated_at":"2020-01-09T23:50:18Z",
"browser_download_url":"https://github.com/c2tarun/upload_test_repo/releases/download/untagged-1baa5c7dd7f5a8d506cc/test.tar.gz"
}
Interesting update: Uploading same file multiple times, gives me different sha on downloading.
More Update: I thought that may be I am providing wrong Content-Type header for tar.gz file, so I created a zip file and used Content-Type as application/zip. I am still seeing same issue. Here are my request headers for zip file:
{
'User-Agent':'python-requests/2.22.0',
'Accept-Encoding':'gzip, deflate',
'Accept':'*/*',
'Connection':'keep-alive',
'Content-Type':'application/zip',
'Content-Length':'332',
'Authorization':'Basic xxxxxxxxx'
}
Thanks

You should set the Content-Type is 'application/tar+gzip' or debug with
file --mime-type -b test.tar.gz
command.

Related

Error with install.packages using renv|knit|rmarkdown

I'm updating the renv folder from a project in order to adjust the libraries, but it seems I'm having a permission problem. After running renv::init() and trying to installing manually the remaining libraries using install.packages() I always get the message
Error: failed to retrieve 'https://cran.rstudio.com/bin/windows/contrib/4.2/ipeadatar_0.1.6.zip' [error code 23]
1: curl: (23) Failure writing output to destination
2: curl: (23) Failure writing output to destination
Using .libPath() I can see that the renv was created in the "AppData" hidden folder
1] "C:/Users/André Ferreira/AppData/Local/R/cache/R/renv/library/MacroBRA_Wrld-09789847/R-4.2/x86_64-w64-mingw32"
So checking my permissions, I couldn't see anything wrong. Any thoughts about this problem? The thing it's that when I open my .Rmd file and try to knit, I receive the same message "1: curl: (23) Failure writing output to destination", now from rmarkdown retrieve installation, so it may be a configuration/permission problem.
Adding "C:\rtools42\usr\bin" and "C:\Program Files\R\R-4.2.1\bin" in the environment variable didn't help.
As I could see, opening an empty file from rstudio, I could use install.packages() without problem.
Although this doesn't solve the problem directly, you can also instruct renv to use a different library path with something like:
# use a project-local library path
RENV_PATHS_LIBRARY = renv/library
in your project's .Renviron file. Depending on your environment, you might also consider placing the library path in an alternate location.
See https://rstudio.github.io/renv/articles/packages.html#r-cmd-build-and-the-project-library for more details.

nbsphinx causes build to fail when building Jupyter Notebooks

Details
I am getting a build failure of my read-the-docs that I don't understand. The assertion of "verbatim" in line 2151 of nbsphinx.py is causing the build failure. So the build fails when I try to include the Jupyter Notebook tutorials I created. I compared current versions of the tutorials to previous versions which had not caused the build to fail, and I can't find a difference that could account for the current failure.
Read the Docs project URL: lofti_gaia
Build URL: https://github.com/logan-pearce/lofti_gaia
Read the Docs username: logan-pearce
Expected Result
A passing build including *.ipynb files
Actual Result
Build failed at line 2151 of nbsphinx.py due to assertion of 'Verbatim' failing.
Terminal output:
Running Sphinx v4.1.2
loading translations [en]... done
making output directory... done
WARNING: html_static_path entry '_static' does not exist
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [latex]: all documents
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
processing lofti_gaia.tex... index installation tutorials/QuickStart tutorials/Tutorial api lofti loftitools
resolving references...
done
writing... failed
Exception occurred:
File "/home/docs/checkouts/readthedocs.org/user_builds/lofti-gaia/conda/latest/lib/python3.7/site-packages/nbsphinx.py", line 2151, in depart_codearea_latex
assert 'Verbatim' in lines[0]
AssertionError
The full traceback has been saved in /tmp/sphinx-err-x1h83s3m.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
According to the github issue 584 for nbsphinx (https://github.com/spatialaudio/nbsphinx/issues/584), this is due to a compatibility issue with sphinx 4.1.0. It can be gotten around by requiring sphinx version 4.0.2.
So in my requirements.txt file, I included sphinx==4.0.2, after which the build passes. So now my requirements.txt file looks like:
numpy
matplotlib
astropy>=4.0.1.post1
astroquery>=0.4
sphinx==4.0.2
ipython==7.19.0
nbsphinx>=0.8.6
and the build passes.
I have encountered the same issue. I did not solve it with .ipynb format, but converting the jupyter notebook to .rst format works.
May it helps

Downloading NetCDF files with R: Manually works, download.file produces error

I am trying to download a set of NetCDF files from: ftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/nwm/prod/nwm.20180425/medium_range/
When I manually download the files I have no issues connecting, but when I use download.file and attempt to connect I get the following error:
Assertion failed!
Program: C:\Program Files\Rstudio\bin\rsession.exe
File: nc4file.c, Line 2771
Expression: 0
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
I have attempted to run the code in R without R studio and got the same result.
My abbreviated code is as followed:
library("ncdf4")
library("ncdf4.helpers")
download.file("ftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/nwm/prod/nwm.20180425/medium_range/nwm.t00z.medium_range.channel_rt.f006.conus.nc","c:/users/nt/desktop/nwm.t00z.medium_range.channel_rt.f006.conus.nc")
temp = nc_open("c:/users/nt/desktop/nwm.t00z.medium_range.channel_rt.f006.conus.nc")
Adding mode = 'wb' to the download.file arguments solves the issue for me. I've had the same problem when downloading PDFs
download.file("ftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/nwm/prod/nwm.20180425/medium_range/nwm.t00z.medium_range.channel_rt.f006.conus.nc","C:/teste/teste.nc", mode = 'wb')

Firebase error: symbolFileMappings:upsert: The uploaded file is not a valid Breakpad Symbol file

I am trying to upload a dSYM file to Firebase using this command:
./Pods/FirebaseCrash/batch-upload -i ./Info.plist -p ./GoogleService-Info.plist ./service-accounts/mtb.json 78*****C-5**4-3***-***C-00*********7
But each time I run this, I get back the following errors
./Pods/FirebaseCrash/upload-sym-util.bash:377: error: symbolFileMappings:upsert: The uploaded file is not a valid Breakpad Symbol file.
./Pods/FirebaseCrash/upload-sym-util.bash:378: note: symbolFileMappings:upsert: The metadata for the symbol file failed to update.
I have tried this link https://groups.google.com/forum/#!msg/firebase-talk/4829Sp1_uKY/IEC_T4-VBAAJ but haven't had any luck as it errors out to
usage: batch-upload [-hv] [-p google-service] [-i info] service-account-file {mach-o file|uuid}
Can someone help me on this one please?
Thanks
Apparently this bug relates to batch-upload script having problems with archives, so you would need to patch it manually.
Try the following:
Change extract_symbols_and_upload "$EXE" "$ARCH" "$BUNDLE" to
extract_symbols_and_upload "$EXE" "$ARCH" "$BUNDLE/$BNDL_PATH"
The dump_syms utility understands dSYM bundles but not archives,
curiously enough. This fix will go out with the next release.
Source

Hosting Mercurial on IIS7

Note, this might perhaps be best suited on serverfault.com, but since it is about hosting a programmer source code repository, I am not entirely sure. I'm posting here first, trusting that it'll be migrated if necessary.
I'm attempting to host clones of my Mercurial repositories on my own server (I have the main repo somewhere else), and I'm attempting to set up Mercurial under IIS.
I followed the guide here, but I get an error message.
Solved: See bottom of this question for details.
The error message is:
mercurial.error.RepoError: repository /path/to/repo/or/config not found
Here's what I did.
I installed Mercurial 1.5.2
I created c:\inetpub\hg
I downloaded the hg source as per the instructions of the webpage, and copied the hgweb.cgi file into c:\inetpub\hg (note, the webpage says hgwebdir.cgi, but this particular file does not exist, hgweb.cgi does, however, can this be the source of the problem?)
I added a hgweb.config, with the following contents:
[paths]
repo1 = C:/hg/**
[web]
style = monoblue
I created c:\hg, created a sub-directory test, and created a repository inside it
I installed python 2.6.5, latest 2.6 version from the website (the webpage mentions I need to install the correct version or I'll get a specific error message, since I don't get an error message that looks remotely like the one mentioned, I assume that 2.6.5 is not the problem)
I added a new virtual host hg.vkarlsen.no, pointing it to c:\inetpub\hg
For this host, I added a script mapping under the Handler Mappings section, mapping *.cgi to c:\python26\python.exe -u %s %s as per the instructions on the website.
I then tested it by navigating to http://hg.vkarlsen.no/hgweb.cgi, but I get an error message.
To make it easier to test, I dropped to a command prompt, navigated to c:\inetpub\hg, and executed the following command (error message is part of the text below):
C:\inetpub\hg>c:\python26\python.exe -u hgweb.cgi
Traceback (most recent call last):
File "hgweb.cgi", line 16, in <module>
application = hgweb(config)
File "mercurial\hgweb\__init__.pyc", line 12, in hgweb
File "mercurial\hgweb\hgweb_mod.pyc", line 30, in __init__
File "mercurial\hg.pyc", line 82, in repository
File "mercurial\localrepo.pyc", line 2221, in instance
File "mercurial\localrepo.pyc", line 62, in __init__
mercurial.error.RepoError: repository /path/to/repo/or/config not found
Does anyone know what I need to look at in order to fix this?
Edit: Ok, I think I managed to get one step closer to the solution, but I'm still stumped.
I realized the .cgi file is a python script file, and not something compile, so I opened it for editing, and these lines was sitting in it:
# Path to repo or hgweb config to serve (see 'hg help hgweb')
config = "/path/to/repo/or/config"
So this was my source for the specific error message.
If I change the line to this:
config = "c:\\hg\\test"
Then I can navigate the empty repository through the Mercurial web interface.
However, I want to host multiple repositories, and seeing as the line says that I can also link to a hgweb config file, I tried this:
config = "c:\\inetpub\\hg\\hgweb.config"
But then I get the following error message:
mercurial.error.Abort: c:\inetpub\hg\hgweb.config: not a Mercurial bundle file
Exception ImportError: 'No module named shutil' in <bound method bundlerepository.__del__
of <mercurial.bundlerepo.bundlerepository object at 0x0260A110>> ignored
Nothing I've tried for the config variable seems to work:
config = "hgweb.config"
config = "c:\\hg\\hgweb.config"
various other variations I don't remember.
So, still stumped, pointers anyone?
Hopefully this question will give others some information if they're stumped as I was.
I ended up having to edit the hgweb.cgi file:
from: from mercurial.hgweb import hgweb, wsgicgi
application = hgweb(config)
to: from mercurial.hgweb import hgweb, hgwebdir, wsgicgi
application = hgwebdir(config)
Note the added hgwebdir parts there. Here's my hgweb.config file, located in the same directory as hgweb.cgi file:
[collections]
C:/hg/ = C:/hg/
[web]
style = gitweb
This now serves my repositories successfully.

Resources