Halp! I accidentally deleted my ~/.julia/logs folder.
BUT the good news is I have several running julia sessions up, which seem to have some memory of the julia REPL history.
Does anyone know if there's a way to reverse-dump a history file from the running julia process's state somehow? Many thanks in advance!
This should store it back to a file which can then be put in .julia/logs/repl_history.jl:
function restore_repl_history()
hist = Base.active_repl.interface.modes[1].hist
open("restored_history.jl", "w") do io
for (mode, cont) in zip(hist.modes, hist.history)
println(io, "# time: $(Libc.strftime("%Y-%m-%d %H:%M:%S %Z", time()))")
println(io, "# mode: $mode")
s = join(split(cont, '\n'), "\n\t")
println(io, "\t", s)
end
end
end
Related
I'm trying to upload large files to a remote SFTP-server. The regular OpenSSH sftp-client averages 4-5 Mb/second.
My code is:
inp = open(fName, "rb")
bsize = os.stat(inp.fileno()).st_blksize
out = SFTP.open(os.path.split(fName)[-1], "w",
bsize * 4) # SFTP is a result of paramiko.SFTPClient()
out.set_pipelined()
while True:
buf = inp.read(bsize)
if not buf:
break
out.write(buf)
inp.close()
out.close()
Averages 40-180Kb -- even if I artificially raise the bsize. One could blame the fact, that Paramiko is a "pure Python" implementation, but the difference should not be this huge...
There is no significant CPU-load on my machine, which runs FreeBSD-11, python-3.6, Paramiko-2.7.1
What's going on?
Update: adding out.set_pipelined() helps raise the throughput to 1-2Mb/s, but it still lags behinds that of the OpenSSH sftp-client by a lot...
Update: adding an explicit buffer-size to the SFTP.open() call -- as suggested by Martin in a comment -- had no perceptible effect. (I suspect, Paramiko already uses some buffering by default.)
In REPL mode, Julia lets you type a semicolon and run shell commands, i.e.
;
cd ~
And then to return to Julian REPL
Is there a way to do something similar in a .jl file? The closest I found was run(…) and that has many caveats. This is a Linux environment, so I'm not concerned about the caveats of shell mode on Windows machines.
The broader topic of interest is in doing this for other REPL modes, like the R one provided by using RCall
As you mentioned, the default way to do is via the run command. If you have not already, check out the docs on this https://docs.julialang.org/en/v1/manual/running-external-programs/#Running-External-Programs which go into some of the caveats.
I am not sure I follow what you are getting at with RCall but it may perhaps be worth opening a separate question for that.
You can find the code for this at https://github.com/JuliaLang/julia/tree/master/stdlib/REPL/test.
Seems there is no API, just lots of typing.
Here is a minimal working example (the codes are mostly copied from different places from the folder above):
using REPL
mutable struct FakeTerminal <: REPL.Terminals.UnixTerminal
in_stream::Base.IO
out_stream::Base.IO
err_stream::Base.IO
hascolor::Bool
raw::Bool
FakeTerminal(stdin,stdout,stderr,hascolor=true) =
new(stdin,stdout,stderr,hascolor,false)
end
REPL.Terminals.hascolor(t::FakeTerminal) = t.hascolor
REPL.Terminals.raw!(t::FakeTerminal, raw::Bool) = t.raw = raw
REPL.Terminals.size(t::FakeTerminal) = (24, 80)
input = Pipe()
output = Pipe()
err = Pipe()
Base.link_pipe!(input, reader_supports_async=true, writer_supports_async=true)
Base.link_pipe!(output, reader_supports_async=true, writer_supports_async=true)
Base.link_pipe!(err, reader_supports_async=true, writer_supports_async=true)
repl = REPL.LineEditREPL(FakeTerminal(input.out, output.in, err.in, false), false)
repltask = #async REPL.run_repl(repl)
Now you can do:
julia> println(input,";ls -la *.jld2")
-rw-r--r-- 1 pszufe 197121 5506 Jul 5 2020 file.jld2
-rw-r--r-- 1 pszufe 197121 5506 Jul 5 2020 myfile.jld2
I am new to programming and i am running this script to clean a large text file (over 12000 lines) and write it to another .txt file. The problem is when a run this with a smaller file (roughly around 500 line) it executes fast, therefore my conclusion was it is taking time due to the size of the file. So if someone can guide me to make this code efficient it will be highly appreciated.
input_file = open('bNEG.txt', 'rt', encoding='utf-8')
l_p = LanguageProcessing()
sentences=[]
for lines in input_file.readlines():
tokeniz = l_p.tokeniz(lines)
cleaned_url = l_p.clean_URL(tokeniz)
remove_words = l_p.remove_non_englishwords(cleaned_url)
stopwords_removed = l_p.remove_stopwords(remove_words)
cleaned_sentence=' '.join(str(s) for s in stopwords_removed)+"\n"
output_file = open('cNEG.txt', 'w', encoding='utf-8')
sentences.append(cleaned_sentence)
output_file.writelines(sentences)
input_file.close()
output_file.close()
EDIT: Below is the corrected code as mentioned in the answer with few other alteration to suit my requirements
input_file = open('chromehistory_log.txt', 'rt', encoding='utf-8')
output_file = open('dNEG.txt', 'w', encoding='utf-8')
l_p = LanguageProcessing()
#sentences=[]
for lines in input_file.readlines():
#print(lines)
tokeniz = l_p.tokeniz(lines)
cleaned_url = l_p.clean_URL(tokeniz)
remove_words = l_p.remove_non_englishwords(cleaned_url)
stopwords_removed = l_p.remove_stopwords(remove_words)
#print(stopwords_removed)
if stopwords_removed==[]:
continue
else:
cleaned_sentence=' '.join(str(s) for s in stopwords_removed)+"\n"
#sentences.append(cleaned_sentence)
output_file.writelines(cleaned_sentence)
input_file.close()
output_file.close()
To have the discussion as answer:
Two problems are here:
You open / create the outputfile and write the data in the loop - for every line of the input file. Additional you are collection all data in an array (sentences).
You have two possibilities:
a) Create the file before the loop, and write in the loop just "cleaned_sentence" (and delete the collecting "sentences").
b) Collect everything in "sentences" and write "sentences" at once after the loop.
Disadvantage of a) is: this is a bit slower than b) (as long as the OS di not have to swap memory for b). But the advantage is: This is much less memory consuming and will work no matter how big the file is and how less memory in the computer is installed.
I'm having issues when trying to read in a binary file I've previously written into another program. I have been able to open it and read it to an array with out compilation errors, however, the array is not populated (all 0's). Any suggestions or thoughts would be great. Here is the open/read statement I'm using:
allocate(dummy(imax,jmax))
open(unit=io, file=trim(input), form='binary', access='stream', &
iostat=ioer, status='old', action='READWRITE')
if(ioer/=0) then
print*, 'Cannot open file'
else
print*,'success opening file'
end if
read(unit=io, fmt=*, iostat=ioer) dummy
j=0
k=0
size: do j=1, imax
do k=1, jmax
if(dummy(j,k) > 0.) print*,dummy(j,k)
end do
end do size
Please let me know if you need more info.
Here is how the file is originally written:
out_file = trim(output_dir)//'SEVIRI_FRP_.08deg_'//trim(season)//'.bin'
print*, out_file
print*, i_max,' i_max,',j_max,' j_max'
open (io, file = out_file, access = 'direct', status = 'replace', recl = i_max*j_max*4)
write(io, rec = 1) sev_frp
write(io, rec = 2) count_sev_frp
write(io, rec = 3) sum_sev_frp
check: do n=1, i_max
inna: do m=1, j_max
!if (sev_frp(n,m) > 0) print*, count_sev_frp(n,m)
end do inna
end do check
print*,'n-',n,'m-',m
close(io)
First of all the form takes two possible values as far as I know: "FORMATTED" or "UNFORMATTED".
Second, to read, you should use a open that is symmetric to the open statement that you used to write the file, Unless you know exactely what you are doing. I suggest that for reading, you open with:
open(unit=io, file=trim(input), access='direct', &
iostat=ioer, status='old', action='READ', recl = i_max*j_max*4)
That corresponds to the open statement that you used to save the file.
As innoSPG says, you have a mismatch in the way the file is written and how it is read.
An external file may be connected with one of three access methods: sequential; direct; stream. Further, a connection may be formatted or unformatted.
When the file is opened for writing it uses the direct access method with unformatted records. The records are unformatted because this is the default (in the abscence of the form= specifier).
When you open the file for reading you use the non-standard extension of form="binary" and stream access. There is possibly nothing wrong with this, but it does require care.
However, with the read statements you are using formatted (list-directed) input. This will not be allowed.
The way suggested in the previous answer, of using a similar access method and record length will require a further change to the code. [You'll also need to set the value of the record length somehow.]
Not only will you need to remove the format, to match the unformatted records written, but you'll want to use the rec= specifier to access the records of the file.
Finally, if you are using the iostat= specifier you really should check the resulting value.
i wrote the following code to output source address and destination address of all packets that are in a .pcap file to a text file using lua and tshark.
#!/usr/bin/lua
do
local file = io.open("luawrite", "w")
local function init_listener()
local tap = Listener.new("ipv6")
function tap.packet(pinfo, tvb)
local srcadd = pinfo.src
local dstadd = pinfo.dst
file:write(tostring(srcadd), "\t", tostring(dstadd)"\n")
end
end
end
I am running this script using the following command:
tshark -r wireless.pcap -xlua_script:MyScript.lua
Why is nothing being written in my text file? Is there something wrong on the code? Help is very much appreciated. Thanks!
Probably because you are missing a comma before "\n":
---------------------------------------------------vv-----
file:write(tostring(srcadd), "\t", tostring(dstadd), "\n")
It may be useful to check for file value returned by the open call.
I don't see any other problems with the script; if you still have issues, I have a page on debugging Wireshark Lua scripts that may help.