Dynamics AX : error while importing data from excel on client - axapta

I did a custom X++ code to import data from Excel to General Ledger, however the import works very well on the server directly but while running it from the end user(client) it imports several records (like 24 records) then throw an error
the number of argument provided is different from the number of argument provided to the method
it is obvious that the error is related to connectivity issue since I have tried the same Excel file on the on the server and successfully imported.
in order to prevent this issue I was thinking to alternative solution rather than looping the through the excel file and do the business and insert the records, instead I thought it might be useful if I save the file directly/ bulk save in a table or something else then try to loop through the table to prevent the connectivity issue.
note: several solution are available on google such as windows ghosting but none works for me
could anyone please advise about that or suggest the suitable solution

I would recommend you save the Excel file as tab-separated text, then do the import using the TextIO class.
You will also benefit from a +10 times increase in performance!
static void ExcelTest(Args _args)
{
#Excel
FilePath excelFile = #'C:\Users\user\Documents\MyExcelFile.xlsx';
FilePath textFile = #'C:\Users\user\Documents\MyTextFile.txt';
Microsoft.Office.Interop.Excel.Application application = new Microsoft.Office.Interop.Excel.ApplicationClass();
Microsoft.Office.Interop.Excel.Workbooks workBooks = application.get_Workbooks();
Microsoft.Office.Interop.Excel.Workbook workBook;
// Save the excel file as tab-separated text
application.set_DisplayAlerts(false);
application.set_Visible(false);
new FileIOPermission(excelFile, 'r').assert();
workBooks.Open(excelFile, 0, true, 5, '', '', true, #xlWindows, '', false, false, 1, false, false, 1);
CodeAccessPermission::revertAssert();
workBook = workBooks.get_Item(1);
new FileIOPermission(textFile, 'w').assert();
CodeAccessPermission::revertAssert();
workBook.SaveAs(textFile, #xlTextWindows, '', '', false, false, null, #xlLocalSessionChanges, false, null, null, false);
workBooks.Close();
application.Quit();
// Now read the text file
new FileIOPermission(textFile, 'r').assert();
io = new TextIo(textFile, 'r');
if (!io)
throw error("#SYS18447");
io.inFieldDelimiter('\t');
for (con = io.read(); io.status() == IO_Status::Ok; con = io.read())
{
info(con2str(con));
}
}

Related

Import and parse a file to fill the form

Currently, I'm developing a custom app. So far I got the DocType ready to be filled in manually. We got files (SQLite3) that I'd like to upload, parse, extract the necessary fields of it and fill in the form. Basically like the import data tool. In my case, no bulk operation is needed and if possible do the extraction part server-side.
What I tried so far
I added a Server Action to call a whitelisted method of my app. I can get the current doc with:
#frappe.whitelist()
def upload_data_and_extract(doc: str):
"""
Uploads and processes an existing file and extracts data from it
"""
doc_dict = json.loads(doc)
custom_dt = frappe.get_doc('CustomDT', doc_dict['name'])
# parse data here
custom_dt.custom_field = "new value from parsed data"
custom_dt.save()
return doc # How do I return a JSON back to the website from the updated doc?
With this approach, I only can do the parsing when the document has been saved before. I'd rather update the fields of the form when the attach field gets modified. Thus, I tried the Server Side Script approach:
frappe.ui.form.on('CustomDT', {
original_data: function(frm, cdt, cdn) {
if(original_data) {
frappe.call({
method: "customapp.customapp.doctype.customdt.customdt.parse_file",
args: {
"doc": frm.doc
},
callback: function(r) {
// code snippet
}
});
}
}
});
Here are my questions:
What's the best approach to upload a file that needs to be parsed to fill the form?
How to access the uploaded file (attachment) the easiest way. (Is there something like frappe.get_attachment()?)
How to refresh the form fields in the callback easily?
I appreciate any help on these topics.
Simon
I have developed the same tool but that was for CSV upload. I am going to share that so it will help you to achieve your result.
JS File.
// Copyright (c) 2020, Bhavesh and contributors
// For license information, please see license.txt
frappe.ui.form.on('Car Upload Tool', {
upload: function(frm) {
frm.call({
doc: frm.doc,
method:"upload_data",
freeze:true,
freeze_message:"Data Uploading ...",
callback:function(r){
console.log(r)
}
})
}
});
Python Code
# -*- coding: utf-8 -*-
# Copyright (c) 2020, Bhavesh and contributors
# For license information, please see license.txt
from __future__ import unicode_literals
import frappe
from frappe.model.document import Document
from carrental.carrental.doctype.car_upload_tool.csvtojson import csvtojson
import csv
import json
class CarUploadTool(Document):
def upload_data(self):
_file = frappe.get_doc("File", {"file_url": self.attach_file})
filename = _file.get_full_path()
csv_json = csv_to_json(filename)
make_car(csv_json)
def csv_to_json(csvFilePath):
jsonArray = []
#read csv file
with open(csvFilePath, encoding='latin-1') as csvf:
#load csv file data using csv library's dictionary reader
csvReader = csv.DictReader(csvf,delimiter=";")
#convert each csv row into python dict
for row in csvReader:
frappe.errprint(row)
#add this python dict to json array
jsonArray.append(row)
#convert python jsonArray to JSON String and write to file
return jsonArray
def make_car(car_details):
for row in car_details:
create_brand(row.get('Marke'))
create_car_type(row.get('Fahrzeugkategorie'))
if not frappe.db.exists("Car",row.get('Fahrgestellnr.')):
car_doc = frappe.get_doc(dict(
doctype = "Car",
brand = row.get('Marke'),
model_and_description = row.get('Bezeichnung'),
type_of_fuel = row.get('Motorart'),
color = row.get('Farbe'),
transmission = row.get('Getriebeart'),
horsepower = row.get('Leistung (PS)'),
car_type = row.get('Fahrzeugkategorie'),
car_vin_id = row.get('Fahrgestellnr.'),
licence_plate = row.get('Kennzeichen'),
location_code = row.get('Standort')
))
car_doc.model = car_doc.model_and_description.split(' ')[0] or ''
car_doc.insert(ignore_permissions = True)
else:
car_doc = frappe.get_doc("Car",row.get('Fahrgestellnr.'))
car_doc.brand = row.get('Marke')
car_doc.model_and_description = row.get('Bezeichnung')
car_doc.model = car_doc.model_and_description.split(' ')[0] or ''
car_doc.type_of_fuel = row.get('Motorart')
car_doc.color = row.get('Farbe')
car_doc.transmission = row.get('Getriebeart')
car_doc.horsepower = row.get('Leistung (PS)')
car_doc.car_type = row.get('Fahrzeugkategorie')
car_doc.car_vin_id = row.get('Fahrgestellnr.')
car_doc.licence_plate = row.get('Kennzeichen')
car_doc.location_code = row.get('Standort')
car_doc.save(ignore_permissions = True)
frappe.msgprint("Car Uploaded Successfully")
def create_brand(brand):
if not frappe.db.exists("Brand",brand):
frappe.get_doc(dict(
doctype = "Brand",
brand = brand
)).insert(ignore_permissions = True)
def create_car_type(car_type):
if not frappe.db.exists("Vehicle Type",car_type):
frappe.get_doc(dict(
doctype = "Vehicle Type",
vehicle_type = car_type
)).insert(ignore_permissions = True)
So for this upload tool, I created one single doctype with the below field:
Attach File(Field Type = Attach)
Button (Field Type = Button)

xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 1, column 0

I'm trying to parse a directory with a collection of xml files from RSS feeds.
I have a similar code for another directory working fine, so I can't figure out the problem. I want to return the items so I can write them to a CSV file. The error I'm getting is:
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 1, column 0
Here is the site I've collected RSS feeds from: https://www.ba.no/service/rss
It worked fine for: https://www.nrk.no/toppsaker.rss and https://www.vg.no/rss/feed/?limit=10&format=rss&categories=&keywords=
Here is the function for this RSS:
import os
import xml.etree.ElementTree as ET
import csv
def baitem():
basepath = "../data_copy/bergens_avisen"
table = []
for fname in os.listdir(basepath):
if fname != "last_feed.xml":
files = ET.parse(os.path.join(basepath, fname))
root = files.getroot()
items = root.find("channel").findall("item")
#print(items)
for item in items:
date = item.find("pubDate").text
title = item.find("title").text
description = item.find("description").text
link = item.find("link").text
table.append((date, title, description, link))
return table
I tested with print(items) and it returns all the objects.
Can it be how the XML files are written?
Asked a friend and said to test with a try except statement. Found a .DS_Store file, which only applies to Mac computers. I'm providing the solution for those who might experience the same problem in the future.
def baitem():
basepath = "../data_copy/bergens_avisen"
table = []
for fname in os.listdir(basepath):
try:
if fname != "last_feed.xml" and fname != ".DS_Store":
files = ET.parse(os.path.join(basepath, fname))
root = files.getroot()
items = root.find("channel").findall("item")
for item in items:
date = item.find("pubDate").text
title = item.find("title").text
description = item.find("description").text
link = item.find("link").text
table.append((date, title, description, link))
except Exception as e:
print(fname, e)
return table

ScalikeJDBC + SQlite: Cannot change read-only flag after establishing a connection

Trying to get working ScalikeJDBC and SQLite. Have a simple code based on provided examples:
import scalikejdbc._, SQLInterpolation._
object Test extends App {
Class.forName("org.sqlite.JDBC")
ConnectionPool.singleton("jdbc:sqlite:test.db", null, null)
implicit val session = AutoSession
println(sql"""SELECT * FROM kv WHERE key == 'seq' LIMIT 1""".map(identity).single().apply()))
}
It fails with exception:
Exception in thread "main" java.sql.SQLException: Cannot change read-only flag after establishing a connection. Use SQLiteConfig#setReadOnly and QLiteConfig.createConnection().
at org.sqlite.SQLiteConnection.setReadOnly(SQLiteConnection.java:447)
at org.apache.commons.dbcp.DelegatingConnection.setReadOnly(DelegatingConnection.java:377)
at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.setReadOnly(PoolingDataSource.java:338)
at scalikejdbc.DBConnection$class.readOnlySession(DB.scala:138)
at scalikejdbc.DB.readOnlySession(DB.scala:498)
...
I've tried both scalikejdbc 1.7 and 2.0, error remains. As sqlite driver I use "org.xerial" % "sqlite-jdbc" % "3.7.+".
What can I do to fix the error?
The following will create two separate connections, one for read-only operations and the other for writes.
ConnectionPool.add("mydb", s"jdbc:sqlite:${db.getAbsolutePath}", "", "")
ConnectionPool.add(
"mydb_ro", {
val conf = new SQLiteConfig()
conf.setReadOnly(true)
val source = new SQLiteDataSource(conf)
source.setUrl(s"jdbc:sqlite:${db.getAbsolutePath}")
new DataSourceConnectionPool(source)
}
)
I found that the reason is that you're using "org.xerial" % "sqlite-jdbc" % "3.7.15-M1". This version looks still unstable.
Use "3.7.2" as same as #kawty.
Building on #Synesso's answer, I expanded slightly to be able to get config value from config files and to set connection settings:
import scalikejdbc._
import scalikejdbc.config.TypesafeConfigReader
case class SqlLiteDataSourceConnectionPool(source: DataSource,
override val settings: ConnectionPoolSettings)
extends DataSourceConnectionPool(source)
// read settings for 'default' database
val cpSettings = TypesafeConfigReader.readConnectionPoolSettings()
val JDBCSettings(url, user, password, driver) = TypesafeConfigReader.readJDBCSettings()
// use those to create two connection pools
ConnectionPool.add("db", url, user, password, cpSettings)
ConnectionPool.add(
"db_ro", {
val conf = new SQLiteConfig()
conf.setReadOnly(true)
val source = new SQLiteDataSource(conf)
source.setUrl(url)
SqlLiteDataSourceConnectionPool(source, cpSettings)
}
)
// example using 'NamedDB'
val name: Option[String] = NamedDB("db_ro") readOnly { implicit session =>
sql"select name from users where id = $id".map(rs => rs.string("name")).single.apply()
}
This worked for me with org.xerial/sqlite-jdbc 3.28.0:
String path = ...
SQLiteConfig config = new SQLiteConfig();
config.setReadOnly(true);
return DriverManager.getConnection("jdbc:sqlite:" + path, config.toProperties());
Interestingly, I wrote a different solution on the issue on the xerial repo:
PoolProperties props = new PoolProperties();
props.setDriverClassName("org.sqlite.JDBC");
props.setUrl("jdbc:sqlite:...");
Properties extraProps = new Properties();
extraProps.setProperty("open_mode", SQLiteOpenMode.READONLY.flag + "");
props.setDbProperties(extraProps);
// This line can be left in or removed; it no longer causes a problem
// as long as the open_mode code is present.
props.setDefaultReadOnly(true);
return new DataSource(props);
I don't recall why I needed the second, and was then able to simplify it back to the first one. But if the first doesn't work, you might try the second. It uses a SQLite-specific open_mode flag that then makes it safe (but unnecessary) to use the setDefaultReadOnly call.

PHPExcel fails to read .xlsx

So, i have a form which able to read excel files and save to database by uploading an excel file. It is fine when i upload .xls file, but not with the .xlsx. Here is the code:
echo ' File berhasil diupload => '.$dok['file_name']; //success to echo uploaded .xlsx file
//EXCEL READING, load library
$this->load->library('excel');
//Identify the type of $inputFileName
$inputFileType = PHPExcel_IOFactory::identify(FCPATH."/asset/files/uploads/".$dok['file_name']);
//Create a new Reader of the type that has been identified
$objReader = PHPExcel_IOFactory::createReader($inputFileType);
//Failed to echo (if i upload .xlsx file)
echo 'input file type => '.$inputFileType;
//begin to read excel file
$this->excel = $objReader->load(FCPATH."/asset/files/uploads/".$dok['file_name']);
$objWorksheet=$this->excel->getActiveSheet();
$highestRow=$objWorksheet->getHighestRow();
$highestColumm = $objWorksheet->getHighestColumn();
$highestColumm++;
//jika option timpa, maka delete semua data
if($timpa == 'y')
{
$this->_model->del_table_mcactivity();
}
//read per row and save to database
foreach($this->excel->getWorksheetIterator() as $worksheet){
$worksheetTitle = $worksheet->getTitle();
$highestRow = $worksheet->getHighestRow();
$highestColumn = $worksheet->getHighestColumn();
$highestColumnIndex = PHPExcel_Cell::columnIndexFromString($highestColumn);
$nrColumns= ord($highestColumn) - 64;
for ($row = 2; $row <= $highestRow; ++ $row) {
$kodemak = $objWorksheet->getCell("A$row")->getValue();
$tahun = $objWorksheet->getCell("B$row")->getValue();
$unitkerja = $objWorksheet->getCell("C$row")->getValue();
$nomorkegiatan = $objWorksheet->getCell("D$row")->getValue();
$kegiatan = $objWorksheet->getCell("E$row")->getValue();
$noskkegiatan = $objWorksheet->getCell("F$row")->getValue();
$jenis = $objWorksheet->getCell("G$row")->getValue();
$pagu = $objWorksheet->getCell("H$row")->getValue();
//get for refworkingunit id
$idrefworkingunit = $this->_model->get_workingunit_by_name($unitkerja);
//filtering for budget
$budget = $this->filter_budget($pagu);
//grouping variable to save in database
$activity_data = array('year'=>$tahun, 'idrefworkingunit'=>$idrefworkingunit, 'activitynumber'=>$nomorkegiatan, 'activityname'=>$kegiatan, 'activityreference'=>$noskkegiatan, 'accountnumber'=>$kodemak, 'traveltype'=>$jenis, 'budget'=> $budget);
//insert to database
$this->db->insert('mcactivity', $activity_data);
}
}
I use PHPExcel_IOFactory::identify() to identify the reader should be used. But i think it's failed.I've read this question , then i replace
$objReader = PHPExcel_IOFactory::createReader($inputFileType);
with:
$objReader = PHPExcel_IOFactory::createReader('Excel2007'); but it still doesn't work. Can somebody help me, please ?
XLS and XLSX are two different files types.
XLS
$this->load->library('Excel5');
XLSX
$this->load->library('Excel2007');
Consult the documentation "PHPExcel User Documentation - Reading Spreadsheet Files.doc" that comes with PHPExcel. Page 1 explains the formats, and Page 4 explains different ways of telling PHPExcel what file type to handle.
There are multiple lines where the reference to "Excel2007" and "Excel5" might occur.
Do a search on 'Excel' and not over-look any possibility.
e.g. $objReader = new PHPExcel_Reader_Excel2007(); //was Excel5
Change to _Excel2007 did allow me to read a .xlsx file, but did not break a script that read a .xls file. So there might be backwards compatibility.

POI - unable to create custom validation for a range of cells

I need to check if the user has entered a valid number in a cells A1:A10. In Excel i would choose the cells and then create a custom validator and set the formula to =isNumber("$A$1:$A10")
Trying do this using POI is getting me all tied in knots:
Here is what i have tried:
CellRangeAddressList addressList = new CellRangeAddressList(0, 10, 0, 0);
XSSFDataValidationHelper dvHelper = new XSSFDataValidationHelper(sheet);
DataValidationConstraint customConstraint = dvHelper.createCustomConstraint("isNumber(\"$A$0:$A$10\"");
XSSFDataValidation validation = (XSSFDataValidation)dvHelper.createValidation(customConstraint, addressList);
validation.setShowErrorBox(true);
sheet.addValidationData(validation);
When i try and open this in Excel, i get an error and Excel disables the validation
thanks in advance
-anish
CellReference crStartCell = new CellReference(startRow, column, true, true); // 0-based row and column index
CellReference crEndCell = new CellReference(endRow, column, true, true);
XSSFDataValidationConstraint dvConstraint = (XSSFDataValidationConstraint) dvHelper.createCustomConstraint("ISNUMBER("+crStartCell.formatAsString()+":"+crEndCell.formatAsString() +")");
You can convert that excel to csv than perform your validations using SuperCSV. It would be length but more easier.

Resources