I am trying to scrape data form a public forum for a school project, but every-time I run the code, the resulting .csv file shows multiple rows for the text variable instead of just one.
from bs4 import BeautifulSoup as soup
from urllib.request import urlopen as uReq
my_url = 'https://www.emimino.cz/diskuse/1ivf-repromeda-56566/'
uClient = uReq(my_url)
page_soup = soup(uClient.read(),"html.parser")
uClient.close()
containers = page_soup.findAll("div",{"class":"discussion_post"})
out_filename = "Repromeda.csv"
headers = "text,user_name,date \n"
f = open(out_filename, "w")
f.write(headers)
for container in containers:
text1 = container.div.p
text = text1.text
user_container = container.findAll("span",{"class":"user_category"})
user_id = user_container[0].text
date_container = container.findAll("span",{"class":"date"})
date = date_container[1].text
print("text: " + text + "\n" )
print("user_id: " + user_id + "\n")
print("date: " + date + "\n")
# writes the dataset to file
f.write(text.replace(",", "|") + ", " + user_id + ", " + date + "\n")
f.close()
Ideally I am trying to create a row for each data entry (ie. text, user_id, date in one row), but instead I get multiple rows for one text entry and only one row for user_id and date entry.
this is the actual output
this is the expected output
Just replace the new line with blank string.
for container in containers:
text1 = container.div.p
text = text1.text.replace('\n', ' ')
In the generated reports I cannot go to a new line. I can add only 4 fields side by side but I want to add them in a new line.
If you are just printing to your printer (LPT1) as a device, after entering the code to switch devices from the screen to the printer just reference what line number you want to print on. Here's some code from an old program I used to print the page header, and subsequent headers as needed.
Early in your code:
SET CONSOLE OFF && so your output doesn't echo to the screen while printing.
SET PRINTER ON
SET PRINTER TO LPT1
Then call the Prt_Header() function to print the first page header. You must keep up with the line numbers as you print your detail records, and when you get to the bottom of the page use the EJECT command to kick out that page and send another call to Prt_Header().
****************************
STATIC FUNCTION Prt_Header()
****************************
nPage += 1
# 1, 4 SAY DATE()
# 1, 55 SAY "MyCompany INTERNATIONAL, INC."
# 1,121 SAY "Page " + STR( nPage, 4, 0)
# 2, 51 SAY "MY Report Name"
# 3, 4 SAY "Pay Group: " + cPayGroup
# 3, 58 SAY "For Period: " + cPeriodMon + "/" + cPeriodYr
# 4, 4 SAY cLines
# 5, 4 SAY "EXECUTIVE " + "(" + cParTitle + "): " + cName
# 5, 70 SAY "Member #:" + cDist
# 5,100 SAY "Sponsored: " + STR( nNoSponsored, 5, 0 )
# 6, 21 SAY cAddress
# 6,100 SAY "Qualified: " + STR( nQualified, 5, 0 )
if .not. empty( cAddress2 )
# 7, 21 SAY cAddress2
nLine_no := 8
else
nLine_no := 7
endif
# nLine_no, 21 SAY TRIM(cCity) + ", "+ cState + " " + cZip + " " + =
cFullName
nLine_no += 2
# nLine_no, 4 SAY "LN LEVEL I. D. NAME"
# nLine_no, 70 SAY "SALES BONUS PCT"
# nLine_no, 93 SAY "PHONE LAST ORDER STATUS"
# nLine_no + 1, 4 SAY cLines
nLine_no += 2
nItem := 0
RETURN NIL
* EOP: Prt_Header()
But, if you're using a report generator this is not what you're looking for.
I could use some help performing a database correction, in regards to date and time of pictures were taken.
Essentially, the research we perform entails taking many pictures and entering the picture information into a database (we automated this with Microsoft Access). However, I performed a random check of our database and found that several dates and times of the photos were incorrect, and I am attempting to correct this via R as I was unable to correct it via Access.
What I need to do is to is write a script that reads these data, and compiles a list of the date taken information for all photos (there are several thousand). So far the best thing that I've found is
file.info(list.files(
"E:/Whatcom Creek Project/Data/Seal photos/Discovery/Catalog/Phoca vitulina",
recursive = T))
However this returns a list of NA NA for all information. Also, if I manually select one image to run file.info on, it doesn't return date taken (see the picture for the data I am attempting to retrieve)
If anyone has any suggestions I am all ears. Thanks in advance!!
-Ian
enter image description here
maybe still of any use?
i saw this message. I solved this a long time ago (2011) with a very basic VB6 code. The job is still working, but it is not applicable to all '.jpg files'. It depends with what camera the picture has been taken, but mostly pictures taken from a smartphone and classic camera's are returning the date the picture has been taken (not usable for whatsapp images or edited images).
The code is here below, and it is very basic code (but i hope it still can help or give idea's, and ... any new idea's are welcome too) :
'Reading of "Date Picture Taken" from .jpg file
'-----------------------------------------------
Debug.Print ">>>>>==== Start Read of .jpg-file ===== "; Date; " ===== "; Time; " ====<<<<<"
X = 0
DatePictureTaken = ""
TimePictureTaken = ""
FOUNDdatum = False
SWdatum = False
TELdatum = 0
Foto = FOLDER & "\" & tabFileName(FotoNr)
Open Foto For Binary Access Read As #1
'get startposition of the field "date picture taken"
Do
X = X + 1
Get #1, X, MyByte ': Debug.Print Chr(MyByte);
DoEvents
If Chr(MyByte) = ":" Then
'Debug.Print
SWdatum = True
Get #1, X + 3, MyByte
'Debug.Print "x+03="; Chr(MyByte)
If Chr(MyByte) <> ":" Then SWdatum = False 'Debug.Print "x+03="; Chr(MyByte)
Get #1, X + 9, MyByte
'Debug.Print "x+09="; Chr(MyByte)
If Chr(MyByte) <> ":" Then SWdatum = False 'Debug.Print "x+09="; Chr(MyByte)
Get #1, X + 12, MyByte
'Debug.Print "x+12="; Chr(MyByte)
If Chr(MyByte) <> ":" Then SWdatum = False 'Debug.Print "x+12="; Chr(MyByte)
'if a ':' is on the 3 locations (found above) then it is a date!
If SWdatum _
Then
TELdatum = TELdatum + 1
End If
'the 3e date is the date the picture has been taken
If TELdatum = 3 _
Then
FOUNDdatum = True
X = X - 4
Exit Do
End If
X = X + 12
End If
Loop Until EOF(1) 'Or X = 32765
If FOUNDdatum = False Then Close 1: End
BPdatum = X
EPdatum = X + 9
BPuur = X + 11
EPuur = X + 15
For X = BPdatum To EPdatum
Get #1, X, MyByte
'Debug.Print Chr(MyByte)
If Chr(MyByte) <> ":" _
Then
' DatePictureTaken = DatePictureTaken & "/"
'Else
DatePictureTaken = DatePictureTaken & Chr(MyByte)
End If
Next X
For X = BPuur To EPuur
Get #1, X, MyByte
'Debug.Print Chr(MyByte)
If Chr(MyByte) <> ":" _
Then
' DatePictureTaken = DatePictureTaken & "/"
'Else
TimePictureTaken = TimePictureTaken & Chr(MyByte)
End If
Next X
'Debug.Print "Date Picture Taken = "; DATUM
tbxDatum.Text = DatePictureTaken
tbxUur.Text = TimePictureTaken
Close 1
End Sub
i have a small problem i need to count words inside the console to read doc, docx, pptx, ppt, xls, xlsx, odt, pdf ... so don't suggest me | wc -w or grep because they work only with text or console output and they count only spaces and in japanese, chinese, arabic , hindu , hebrew they use diferent delimiter so the word count is wrong and i tried to count with this
pdftotext file.pdf -| wc -w
/usr/local/bin/docx2txt.pl < file.docx | wc -w
/usr/local/bin/pptx2txt.pl < file.pptx | wc -w
antiword file.doc -| wc -w
antiword file.word -| wc -w
in some cases microsoft word , openoffice sad 1000 words and the counters return 10 or 300 words if the language is ( japanese , chinese, hindu ect... ) , but if i use normal characters then i have no issue the biggest mistake is in some case 3 chars less witch is "OK"
i tried to convert with soffice , openoffice and then try WC -w but i can't even convert ,
soffice --headless --nofirststartwizard --accept=socket,host=127.0.0.1,port=8100; --convert-to pdf some.pdf /var/www/domains/vocabridge.com/devel/temp_files/23/0/东京_1000_words_Docx.docx
OR
openoffice.org --headless --convert-to ........
OR
openoffice.org3 --invisible
so if someone know any way to count correctly or display document statistic with openoffice or anything else or linux with the console please share it
thanks.
If you have Microsoft Word (and Windows, obviously) you can write a VBA macro or if you want to run straight from the command line you can write a VBScript script with something like the following:
wordApp = CreateObject("Word.Application")
doc = ... ' open up a Word document using wordApp
docWordCount = doc.Words.Count
' Rinse and repeat...
If you have OpenOffice.org/LibreOffice you have similar (but more) options. If you want to stay in the office app and run a macro you can probably do that. I don't know the StarBasic API well enough to tell you how but I can give you the alternative: creating a Python script to get the word count from the command line. Roughly speaking, you do the following:
Start up your copy of OOo/LibO from the command line with the appropriate parameters to accept incoming socket connections. http://www.openoffice.org/udk/python/python-bridge.html has instructions on how to do that. Go there and use the browser's in-page find feature to search for `accept=socket'
Write a Python script to use the OOo/LibO UNO bridge (basically equivalent to the VBScript example above) to open up your Word/ODT documents one at a time and get the word count from each. The above page should give you a good start to doing that.
You get the word count from a document model object's WordCount property: http://www.openoffice.org/api/docs/common/ref/com/sun/star/text/GenericTextDocument.html#WordCount
Just building on to what #Yawar wrote. Here is is more explicit steps for how to word count with MS word from the console.
I also use the more accurate Range.ComputeStatistics(wdStatisticWords) instead of the Words property. See here for more info: https://support.microsoft.com/en-za/help/291447/word-count-appears-inaccurate-when-you-use-the-vba-words-property
Make a script called wc.vbs and then put this in it:
Set word = CreateObject("Word.Application")
word.Visible = False
Set doc = word.Documents.Open("<replace with absolute path to your .docx/.pdf>")
docWordCount = doc.Range.ComputeStatistics(wdStatisticWords)
word.Quit
Dim StdOut : Set StdOut = CreateObject("Scripting.FileSystemObject").GetStandardStream(1)
WScript.Echo docWordCount & " words"
Open powershell in the directory you saved wc.vbs and run cscript .\wc.vbs and you'll get back the word count :)
I think this may do what you are aiming for
# Continuously updating word count
import unohelper, uno, os, time
from com.sun.star.i18n.WordType import WORD_COUNT
from com.sun.star.i18n import Boundary
from com.sun.star.lang import Locale
from com.sun.star.awt import XTopWindowListener
#socket = True
socket = False
localContext = uno.getComponentContext()
if socket:
resolver = localContext.ServiceManager.createInstanceWithContext('com.sun.star.bridge.UnoUrlResolver', localContext)
ctx = resolver.resolve('uno:socket,host=localhost,port=2002;urp;StarOffice.ComponentContext')
else: ctx = localContext
smgr = ctx.ServiceManager
desktop = smgr.createInstanceWithContext('com.sun.star.frame.Desktop', ctx)
waittime = 1 # seconds
def getWordCountGoal():
doc = XSCRIPTCONTEXT.getDocument()
retval = 0
# Only if the field exists
if doc.getTextFieldMasters().hasByName('com.sun.star.text.FieldMaster.User.WordCountGoal'):
# Get the field
wordcountgoal = doc.getTextFieldMasters().getByName('com.sun.star.text.FieldMaster.User.WordCountGoal')
retval = wordcountgoal.Content
return retval
goal = getWordCountGoal()
def setWordCountGoal(goal):
doc = XSCRIPTCONTEXT.getDocument()
if doc.getTextFieldMasters().hasByName('com.sun.star.text.FieldMaster.User.WordCountGoal'):
wordcountgoal = doc.getTextFieldMasters().getByName('com.sun.star.text.FieldMaster.User.WordCountGoal')
wordcountgoal.Content = goal
# Refresh the field if inserted in the document from Insert > Fields >
# Other... > Variables > Userdefined fields
doc.TextFields.refresh()
def printOut(txt):
if socket: print txt
else:
model = desktop.getCurrentComponent()
text = model.Text
cursor = text.createTextCursorByRange(text.getEnd())
text.insertString(cursor, txt + '\r', 0)
def hotCount(st):
'''Counts the number of words in a string.
ARGUMENTS:
str st: count the number of words in this string
RETURNS:
int: the number of words in st'''
startpos = long()
nextwd = Boundary()
lc = Locale()
lc.Language = 'en'
numwords = 1
mystartpos = 1
brk = smgr.createInstanceWithContext('com.sun.star.i18n.BreakIterator', ctx)
nextwd = brk.nextWord(st, startpos, lc, WORD_COUNT)
while nextwd.startPos != nextwd.endPos:
numwords += 1
nw = nextwd.startPos
nextwd = brk.nextWord(st, nw, lc, WORD_COUNT)
return numwords
def updateCount(wordCountModel, percentModel):
'''Updates the GUI.
Updates the word count and the percentage completed in the GUI. If some
text of more than one word is selected (including in multiple selections by
holding down the Ctrl/Cmd key), it updates the GUI based on the selection;
if not, on the whole document.'''
model = desktop.getCurrentComponent()
try:
if not model.supportsService('com.sun.star.text.TextDocument'):
return
except AttributeError: return
sel = model.getCurrentSelection()
try: selcount = sel.getCount()
except AttributeError: return
if selcount == 1 and sel.getByIndex(0).getString == '':
selcount = 0
selwords = 0
for nsel in range(selcount):
thisrange = sel.getByIndex(nsel)
atext = thisrange.getString()
selwords += hotCount(atext)
if selwords > 1: wc = selwords
else:
try: wc = model.WordCount
except AttributeError: return
wordCountModel.Label = str(wc)
if goal != 0:
pc_text = 100 * (wc / float(goal))
#pc_text = '(%.2f percent)' % (100 * (wc / float(goal)))
percentModel.ProgressValue = pc_text
else:
percentModel.ProgressValue = 0
# This is the user interface bit. It looks more or less like this:
###############################
# Word Count _ o x #
###############################
# _____ #
# 451 / |500| #
# ----- #
# ___________________________ #
# |############## | #
# --------------------------- #
###############################
# The boxed `500' is the text entry box.
class WindowClosingListener(unohelper.Base, XTopWindowListener):
def __init__(self):
global keepGoing
keepGoing = True
def windowClosing(self, e):
global keepGoing
keepGoing = False
setWordCountGoal(goal)
e.Source.setVisible(False)
def addControl(controlType, dlgModel, x, y, width, height, label, name = None):
control = dlgModel.createInstance(controlType)
control.PositionX = x
control.PositionY = y
control.Width = width
control.Height = height
if controlType == 'com.sun.star.awt.UnoControlFixedTextModel':
control.Label = label
elif controlType == 'com.sun.star.awt.UnoControlEditModel':
control.Text = label
elif controlType == 'com.sun.star.awt.UnoControlProgressBarModel':
control.ProgressValue = label
if name:
control.Name = name
dlgModel.insertByName(name, control)
else:
control.Name = 'unnamed'
dlgModel.insertByName('unnamed', control)
return control
def loopTheLoop(goalModel, wordCountModel, percentModel):
global goal
while keepGoing:
try: goal = int(goalModel.Text)
except: goal = 0
updateCount(wordCountModel, percentModel)
time.sleep(waittime)
if not socket:
import threading
class UpdaterThread(threading.Thread):
def __init__(self, goalModel, wordCountModel, percentModel):
threading.Thread.__init__(self)
self.goalModel = goalModel
self.wordCountModel = wordCountModel
self.percentModel = percentModel
def run(self):
loopTheLoop(self.goalModel, self.wordCountModel, self.percentModel)
def wordCount(arg = None):
'''Displays a continuously updating word count.'''
dialogModel = smgr.createInstanceWithContext('com.sun.star.awt.UnoControlDialogModel', ctx)
dialogModel.PositionX = XSCRIPTCONTEXT.getDocument().CurrentController.Frame.ContainerWindow.PosSize.Width / 2.2 - 105
dialogModel.Width = 100
dialogModel.Height = 30
dialogModel.Title = 'Word Count'
lblWc = addControl('com.sun.star.awt.UnoControlFixedTextModel', dialogModel, 6, 2, 25, 14, '', 'lblWc')
lblWc.Align = 2 # Align right
addControl('com.sun.star.awt.UnoControlFixedTextModel', dialogModel, 33, 2, 10, 14, ' / ')
txtGoal = addControl('com.sun.star.awt.UnoControlEditModel', dialogModel, 45, 1, 25, 12, '', 'txtGoal')
txtGoal.Text = goal
#addControl('com.sun.star.awt.UnoControlFixedTextModel', dialogModel, 6, 25, 50, 14, '(percent)', 'lblPercent')
ProgressBar = addControl('com.sun.star.awt.UnoControlProgressBarModel', dialogModel, 6, 15, 88, 10,'' , 'lblPercent')
ProgressBar.ProgressValueMin = 0
ProgressBar.ProgressValueMax =100
#ProgressBar.Border = 2
#ProgressBar.BorderColor = 255
#ProgressBar.FillColor = 255
#ProgressBar.BackgroundColor = 255
addControl('com.sun.star.awt.UnoControlFixedTextModel', dialogModel, 124, 2, 12, 14, '', 'lblMinus')
controlContainer = smgr.createInstanceWithContext('com.sun.star.awt.UnoControlDialog', ctx)
controlContainer.setModel(dialogModel)
controlContainer.addTopWindowListener(WindowClosingListener())
controlContainer.setVisible(True)
goalModel = controlContainer.getControl('txtGoal').getModel()
wordCountModel = controlContainer.getControl('lblWc').getModel()
percentModel = controlContainer.getControl('lblPercent').getModel()
ProgressBar.ProgressValue = percentModel.ProgressValue
if socket:
loopTheLoop(goalModel, wordCountModel, percentModel)
else:
uthread = UpdaterThread(goalModel, wordCountModel, percentModel)
uthread.start()
keepGoing = True
if socket:
wordCount()
else:
g_exportedScripts = wordCount,
Link for more info
https://superuser.com/questions/529446/running-word-count-in-openoffice-writer
Hope this helps regards tom
EDIT : Then i found this
http://forum.openoffice.org/en/forum/viewtopic.php?f=7&t=22555
wc can understand Unicode and uses system's iswspace function to find whether the unicode character is whitespace. "The iswspace() function tests whether wc is a wide-character code representing a character of class space in the program's current locale." So, wc -w should be able to correctly count words if your locale (LC_CTYPE) is configured correctly.
The source code of the wc program
The manual page for the iswspace function
I found the answer create one service
#!/bin/sh
#
# chkconfig: 345 99 01
#
# description: your script is a test service
#
(while sleep 1; do
ls pathwithfiles/in | while read file; do
libreoffice --headless -convert-to pdf "pathwithfiles/in/$file" --outdir pathwithfiles/out
rm "pathwithfiles/in/$file"
done
done) &
then the php script that i needed counted everything
$ext = pathinfo($absolute_file_path, PATHINFO_EXTENSION);
if ($ext !== 'txt' && $ext !== 'pdf') {
// Convert to pdf
$tb = mktime() . mt_rand();
$tempfile = 'locationofpdfs/in/' . $tb . '.' . $ext;
copy($absolute_file_path, $tempfile);
$absolute_file_path = 'locationofpdfs/out/' . $tb . '.pdf';
$ext = 'pdf';
while (!is_file($absolute_file_path)) sleep(1);
}
if ($ext !== 'txt') {
// Convert to txt
$tempfile = tempnam(sys_get_temp_dir(), '');
shell_exec('pdftotext "' . $absolute_file_path . '" ' . $tempfile);
$absolute_file_path = $tempfile;
$ext = 'txt';
}
if ($ext === 'txt') {
$seq = '/[\s\.,;:!\? ]+/mu';
$plain = file_get_contents($absolute_file_path);
$plain = preg_replace('#\{{{.*?\}}}#su', "", $plain);
$str = preg_replace($seq, '', $plain);
$chars = count(preg_split('//u', $str, -1, PREG_SPLIT_NO_EMPTY));
$words = count(preg_split($seq, $plain, -1, PREG_SPLIT_NO_EMPTY));
if ($words === 0) return $chars;
if ($chars / $words > 10) $words = $chars;
return $words;
}
I'm scratching my head on how to completely parse this line,
I'm having trouble with the '( 4801)' part, every other elements are being grabbed OK.
# MAIN_PROG ( 4801) Generated at 2010-01-25 06:55:00
This is what I have so far
from pyparsing import nums, Word, Optional, Suppress, OneOrMore, Group, Combine, ParseException
unparsed_log_data = "# MAIN_PROG ( 4801) Generated at 2010-01-25 06:55:00.007 Type: Periodic"
binary_name = "# MAIN_PROG"
pid = Literal("(" + nums + ")")
report_id = Combine(Suppress(binary_name) + pid)
year = Word(nums, max=4)
month = Word(nums, max=2)
day = Word(nums, max=2)
yearly_day = Combine(year + "-" + month + "-" + day)
clock24h = Combine(Word(nums, max=2) + ":" + Word(nums, max=2) + ":" + Word(nums, max=2) + Suppress("."))
timestamp = Combine(yearly_day + White(' ') + clock24h).setResultsName("timestamp")
time_bnf = report_id + Suppress("Generated at") + timestamp
time_bnf.searchString(unparsed_log_data)
EDIT:
Paul, if you have the patience,
how would I filter
unparsed_log_data =
"""
# MAIN_PROG ( 4801) Generated at 2010-01-25 06:55:00
bla bla bla
multi line garbage
bla bla
Efficiency | 38 38 100 | 3497061 3497081 99 |
more garbage
"""
time_bnf = report_id + Suppress("Generated at") + timestamp
partial_report_ignore = Suppress(SkipTo("Efficiency"))
efficiency_bnf = Suppress("|") + integer.setResultsName("Efficiency") + Suppress(integer) + integer.setResultsName("EfficiencyPercent")
Both
efficiency_bnf.searchString(unparsed_log_data) and
report_and_effic.searchString(unparsed_log_data)
return data as expected,
but if I try
report_and_effic = report_bnf + partial_report_ignore + efficiency_bnf
report_and_effic.searchString(unparsed_log_data)
returns ([], {})
EDIT2:
one should read in the code,
partial_report_ignore = Suppress(SkipTo("Efficiency", include=True))
pid = Literal("(" + nums + ")")
should be
pid = "(" + Word(nums) + ")"
Pyparsing allows you to add strings to expression objects using '+', like:
expr + "some string"
Which gets interpreted as:
expr + Literal("some string")
You wrote Literal("(" + nums + ")"). nums is the string "0123456789", to be used as part of creating Word's, like Word(nums). So what you were trying to match was not "left-paren followed by a word composed of nums followed by right-paren", you were trying to match the literal string "(0123456789)".