I do have a final issue to my project and I just can't see clear enought to find out what the hell is wrong here.
When I run my code with this, everything is just perfect:
Dim p As Process = New Process()
With p.StartInfo
.WorkingDirectory = Environment.GetEnvironmentVariable("ProgramFiles") & "\rest_of_my_path\"
.FileName = "ping"
.Arguments = "192.168.0.24"
.CreateNoWindow = True
.UseShellExecute = False
.RedirectStandardOutput = True
.RedirectStandardError = True
End With
p.Start()
However, when I run this, it throughs an error during runtime and crashes on p.start()
Dim p As Process = New Process()
With p.StartInfo
.WorkingDirectory = Environment.GetEnvironmentVariable("ProgramFiles") & "\rest_of_my_path\"
.FileName = "myextprogram.exe"
.Arguments = "-n Unnamed -f file.abc"
.CreateNoWindow = True
.UseShellExecute = False
.RedirectStandardOutput = True
.RedirectStandardError = True
End With
p.Start()
I tried adding spaces, quotation marks, just name it but ALWAYS getting an Exception has been thrown by the target of an invocation.
I believe it's complaining about the path.
I'm almost sure it's simple but just cannot put my finger on it.
Any help is appreciated.
ok, found a fix...if anybody is interested...
Rather than having
.FileName = "myextprogram.exe"
.Arguments = "-n Unnamed -f file.abc"
I now have
.FileName = "cmd"
.Arguments = "myextprogram.exe -n Unnamed -f file.abc"
Everything is working as expected now.
Related
I am a newbie to web scraping. I am currently working on a project where I want to scrape all the photos of an instagram user. The user has 521 posts in total due to which I had used selenium to scroll down till the bottom of the profile. But I am still able to scrape only the first 37 photos. After further inspecting, I found that as the browser scrolls up or down, only the first few rows of img tags are visible in the source code. As I scroll more, the previously visible img tags disappear and only the next rows are visible. So only a certain no. of rows are visible in the html code at any instant. I doubt it to be the reason why I am able to scrape only the first 37 photos.
I want to know how I can scrape all the photos of the profile. Below I have mentioned my current code using Beautiful Soup and Selenium. Here, "scroll_down" function uses selenium to scroll down till the bottom of the profile. I am trying to scrape all the 'img' tags in the function "downloading_images", but as already mentioned, I am able to scrape only first 37 photos.
def downloading_images(self):
soup = BeautifulSoup(self.driver.page_source,'html.parser')
self.all_images = soup.findAll('img')
print(len(self.all_images))
for index,image in enumerate(self.all_images):
filename = "image_" + str(index) + ".jpg"
image_path = os.path.join(self.path,filename)
link = image['src']
print("Downloading image ", index)
response = requests.get(link,stream = True)
try:
with open(image_path,'wb') as file:
shutil.copyfileobj(response.raw,file)
except Exception as e:
print(e)
print('Could not download image no.', index)
print('Image link',link)
def scroll_down(self):
sleep(3)
try:
num_posts = self.driver.find_element_by_xpath('//span[text()[contains(.," posts")]]/span[#class="g47SY "]')
str_num_posts = str(num_posts.text).replace(',','')
self.int_num_posts = int(str_num_posts)
if self.int_num_posts > 12:
num_scrolls = int(self.int_num_posts/12) + 3
print(num_scrolls)
sleep(3)
try:
for win in range(num_scrolls):
print(win)
self.driver.execute_script('window.scrollTo(0,document.body.scrollHeight);')
sleep(3)
except Exception as e:
self.error = True
print(e)
except Exception as e:
self.error = True
print(e)
I searched for all relevant questions here, but none of them could help me understand how I can fetch those images from the code which keeps disappearing upon scrolling.
Hope my question is clear. Thanks in advance.
Edit: Ok, I tried to scrape upon every scroll and it seems to work. Here is my new code.
def downloading_images(self):
print(len(self.all_images))
for index,image in enumerate(self.all_images):
filename = "image_" + str(index) + ".jpg"
image_path = os.path.join(self.path,filename)
link = image['src']
print("Downloading image ", index)
response = requests.get(link,stream = True)
try:
with open(image_path,'wb') as file:
shutil.copyfileobj(response.raw,file)
except Exception as e:
print(e)
print('Could not download image no.', index)
print('Image link',link)
def scroll_down(self):
sleep(3)
try:
num_posts = self.driver.find_element_by_xpath('//span[text()[contains(.," posts")]]/span[#class="g47SY "]')
str_num_posts = str(num_posts.text).replace(',','')
self.int_num_posts = int(str_num_posts)
if self.int_num_posts > 12:
num_scrolls = int(self.int_num_posts/12) + 1
else:
num_scrolls = self.int_num_posts
print(num_scrolls)
sleep(3)
try:
soup = BeautifulSoup(self.driver.page_source,'html.parser')
images = soup.findAll('img')
self.all_images = images
last_height = self.driver.execute_script("return document.body.scrollHeight")
for win in range(num_scrolls):
print(win)
self.driver.execute_script('window.scrollTo(0,document.body.scrollHeight);')
sleep(3)
new_height = self.driver.execute_script("return document.body.scrollHeight")
if new_height == last_height:
break
soup = BeautifulSoup(self.driver.page_source,'html.parser')
images = soup.findAll('img')
self.all_images.extend(images[-12:])
last_height = new_height
print(len(self.all_images))
except Exception as e:
self.error = True
print(e)
except Exception as e:
self.error = True
print(e)
def search_target(self):
try:
search_bar = self.driver.find_element_by_xpath('//input[#class="XTCLo x3qfX "]')
search_bar.send_keys(self.target_username)
taget_profile_url = self.main_url + '/' + self.target_username + '/'
self.driver.get(taget_profile_url)
except Exception as e:
self.error = True
print(e)
def close_notify_box(self):
try:
sleep(3)
not_now_button = self.driver.find_element_by_xpath('//button[#class="aOOlW HoLwm "]')
not_now_button.click()
except Exception:
pass
def log_in(self):
try:
log_in_button = self.driver.find_element_by_xpath('//p[#class="izU2O"]/a')
log_in_button.click()
sleep(3)
user_name_input = self.driver.find_element_by_xpath('//input[#aria-label="Phone number, username, or email"]')
user_name_input.send_keys(self.username)
password_input = self.driver.find_element_by_xpath('//input[#aria-label="Password"]')
password_input.send_keys(self.password)
password_input.submit()
except Exception as e:
self.error = True
print(e)
I would like to know if there are any alternative solutions to this. And whether it is an efficient solution. Thank you.
Coding in VS-2012 Express for Web -- VB.Net with this code...
Diagnostics.Debug.WriteLine(vbTab + ".prpUID_REPORT = [{0}]", .prpUID_REPORT)
Diagnostics.Debug.WriteLine(vbTab + ".prpRV_HeaderDSName = [{0}]", .prpRV_HeaderDSName)
Diagnostics.Debug.WriteLine(vbTab + ".prpRV_HeaderDSId = [{0}]", .prpRV_HeaderDSId)
Diagnostics.Debug.WriteLine(vbTab + ".prpRV_ReportDSName = [{0}]", .prpRV_ReportDSName)
Diagnostics.Debug.WriteLine(vbTab + ".prpRV_ReportDSId = [{0}]", .prpRV_ReportDSId)
Diagnostics.Debug.WriteLine(vbTab + ".prpRV_ReportPath = [{0}]", .prpRV_ReportPath)
Results in this in the Immediate-window:
.prpUID_REPORT = [22]
dsetCustHeader: .prpRV_HeaderDSName = [{0}]
SDS_RptHeader: .prpRV_HeaderDSId = [{0}]
dsetReportContentAll: .prpRV_ReportDSName = [{0}]
SDS_RptData: .prpRV_ReportDSId = [{0}]
ssrsFleetCostSummary_FLA.rdlc: .prpRV_ReportPath = [{0}]
Notice that the first debug-line shows the text correctly .prpUID_Report = [22]. However, the next debug-lines show the "value" followed by part of the source code line. It appears that the substitution into {0} is faulty.
Any clues as to what may be causing this? I think the debug-source code is syntactically correct, since the first line (= [22]) works as expected but the other lines do not.
Your comments are welcome.
Debug.WriteLine(string,string) method call is different from
Debug.WriteLine(string,object[])
which may be why the first WriteLine statement is working while the rest isnt as the second parameter is possibly a string.
Please check the documentation here
Have you tried using an explicit String.Format() call to see if that makes any difference?
Diagnostics.Debug.WriteLine(String.Format("{0} .prpUID_REPORT = [{1}]",vbTab, .prpUID_REPORT))
Diagnostics.Debug.WriteLine(String.Format("{0} .prpRV_HeaderDSName = [{1}]",vbTab, .prpRV_HeaderDSName))
` Other code omitted for brevity `
It may simply be an issue of the improper method call of the Debug.WriteLine() method (i.e. expecting an array or an object and just receiving a string, which is triggering the wrong functionality).
I have a problem running phantomjs.exe binary in my QProcess class. Consider this code:
QString program = "phantomjs.exe";
QProcess *process = new QProcess(this);
process->start(program, QStringList() << "test.js");
When I start the app main process loads up and nothing happens after that, just hundreds of other phantomjs.exe are created (checking it in TaskManager) as well as conhost.exe processes.
I tried other exe files, like notepad.exe, and it works just fine. Notepad window appears.
Did you encounter this problem?
Do you call phantom.exit() in your test script?
https://github.com/ariya/phantomjs/wiki/Quick-Start
console.log('Hello, world!');
phantom.exit();
Hope that helps.
After checking I found that there is a problem with QProcess. I used SHELLEXECUTEINFO instead. This code works for me well. No recursive calls of phantomjs.exe here:
SHELLEXECUTEINFO shExecInfo;
shExecInfo.cbSize = sizeof(SHELLEXECUTEINFO);
shExecInfo.fMask = NULL;
shExecInfo.hwnd = NULL;
shExecInfo.lpVerb = L"runas";
shExecInfo.lpFile = L"phantomjs.exe";
shExecInfo.lpParameters = L"test.js";
shExecInfo.lpDirectory = NULL;
shExecInfo.nShow = SW_NORMAL;
shExecInfo.hInstApp = NULL;
ShellExecuteEx(&shExecInfo);
I need to pass the "" to the mm.exe that run with nativeprocess. When I pass "In From MIDI Yoke: 1" even using \" to the nativeprocess, it won't launch the application properly and caused it to crash. What wrong this the code?
private function soundbank():void {
var argMidi5:Vector.<String> = new Vector.<String>;
var file:File = File.applicationDirectory.resolvePath("java/mm.exe");
argMidi5.push('-g 0 -m winmidi -o midi.winmidi.device="In From MIDI Yoke: 1"');
npSB = new NativeProcessStartupInfo();
npSB.workingDirectory = File.applicationDirectory;
direct = String(File.applicationDirectory);
npSB.executable = file;
npSB.arguments = argMidi5;
npSBOut = new NativeProcess();
npSBOut.addEventListener(ProgressEvent.STANDARD_OUTPUT_DATA, onSoundbank);
npSBOut.start(npSB);
trace(argMidi5);
}
try this:
argMidi5.push("-g");
argMidi5.push("0");
argMidi5.push("-m");
argMidi5.push("winmidi");
argMidi5.push("-o");
argMidi5.push("midi.winmidi.device=\"In From MIDI Yoke: 1\"");
Arguments are passed as individual strings in a vector, not a single string.
I need help with the code below.
I try to convert a AutoCAD file from the format dwg to the format dwf.
Then, the dwf file is downloaded and opened on the client computer using a java applet.
The command used to convert the dwg file on the command-line is:
C:\inetpub\wwwroot\COR-Basic\cadviewer\converter\ax2008.exe -i="C:\inetpub\wwwroot\test\container\DU38_EG00_070116.dwg" -o="C:\inetpub\wwwroot\COR-Basic\cadviewer\files\DU38_EG00_070116.dwf" -f=dwf -model -text
this works when I enter the command text in cmd.exe.
But when I call it from my asp.net application, it only starts the process, but the process never ends...
I've tried adding an additional user, have given this user full permission, and full permissions on wwwroot, but it still doesn't work.
Anybody knows what I'm doing wrong, or how I could do it in another way?
If System.IO.File.Exists(strDWGlocation) Then
Dim psiProcessSettings As Diagnostics.ProcessStartInfo = New Diagnostics.ProcessStartInfo
psiProcessSettings.FileName = strApplicationPath
psiProcessSettings.Arguments = " -i=""" & strDWGlocation & """ -o=""" & strOutputLocation & """ -f=dwf -model -text"
'ST-LAPTOP\converter
psiProcessSettings.UserName = "converter"
psiProcessSettings.Password = secureString
'StefanSteiger.Debug.MsgBox("Input location:" + strDWGlocation)
'StefanSteiger.Debug.MsgBox("Output location:" + strOutputLocation)
Response.Write("<h1>Argument1: " + psiProcessSettings.Arguments + "</h1>")
Response.Write("<h1>Pfad1: " + psiProcessSettings.FileName + "</h1>")
'psiProcessSettings.RedirectStandardInput = True
psiProcessSettings.RedirectStandardError = True
psiProcessSettings.RedirectStandardOutput = True 'Redirect output so we can read it.
psiProcessSettings.UseShellExecute = False 'To redirect, we must not use shell execute.
'psiProcessSettings.CreateNoWindow = True ' don't create a window
Dim pConverterProcess As Diagnostics.Process = New Diagnostics.Process
pConverterProcess = Diagnostics.Process.Start(psiProcessSettings) 'Create the process.
pConverterProcess.Start() 'Execute the process.
'Response.Write("<h1>" + Replace(pConverterProcess.StandardOutput.ReadToEnd(), vbCrLf, "<BR />") + "</h1>") 'Send whatever was returned through the output to the client.
'pConverterProcess.CancelOutputRead()
'pConverterProcess.CancelErrorRead()
'pConverterProcess.StandardInput.Close()
'Wait for the process to end.
'pConverterProcess.WaitForExit()
pConverterProcess.Close()
'Dim iExitCode As Integer = pConverterProcess.ExitCode()
pConverterProcess.Dispose()
Else
MyNamespace.Debug.MsgBox("No such file.")
End If
This is my code that does a similar thing, and it works!
process.StartInfo.FileName = toolFilePath;
process.StartInfo.Arguments = parameters;
process.StartInfo.UseShellExecute = false; // needs to be false in order to redirect output
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardError = true;
process.StartInfo.RedirectStandardInput = true; // redirect all 3, as it should be all 3 or none
process.StartInfo.WorkingDirectory = Path.GetDirectoryName(toolFilePath);
process.StartInfo.Domain = domain;
process.StartInfo.UserName = userName;
process.StartInfo.Password = decryptedPassword;
process.Start();
output = process.StandardOutput.ReadToEnd(); // read the output here...
process.WaitForExit(); // ...then wait for exit, as after exit, it can't read the output
returnCode = process.ExitCode;
process.Close(); // once we have read the exit code, can close the process
Why have you commented out the WaitForExit()?
You could try setting EnableRaisingEvents to true as well.
In my experience, the Process class is quite difficult to work with when reading the standard output, try removing any code that attempts to redirect and read output